Computo ergo sum

Recovering Datasets from Broken ZFS raidz Pools

Sun, 09 Jul 2017 20:15:00 +0200

There are generally two kinds of people–those who’ve suffered a severe data loss and those who are about to suffer a severe data loss. I repeatedly jump back and forth between the two kinds.

Recently, a combination of hardware defects and a series of power outages rendered the raidz pool of the NAS of my previous research group unreadable. The OS, an old Solaris 10 x86, would not import the pool with a dreaded I/O error message. We tried importing in various modern OpenSolaris-based live distributions, even forcing the kernel to try and fix errors when possible, to no success. Perhaps disabling the ZIL (because of performance problems with NFS clients) wasn’t that good idea after all. The lack of resources for proper preventive maintenance meant that there were no real backups to restore from. Gone were a lot of research data, source codes, PhD theses, mails, and web content. In the face of the growing despair, as it all happened in the middle of several ongoing project calls, and the rapidly approaching need to accept that the data is most likely gone for good and one has to start anew, I got curious–what could really break in the “unbreakable” ZFS? Previous to that moment, ZFS was to me just a magical filesystem that can do all those things such as cheaply creating multiple filesets and instantaneous snapshots, and I never had real interest in learning how is this all implemented. This time my curiosity won and I asked the sysadmin to wait a while before wiping the disks and let me first poke around the filesystem and see if I could make it readable again. In the end, what started as a set of Python scripts to read and display on-disk data structures quickly grew into a very functional minimalistic ZFS implementation capable of reading and exporting entire datasets.

It turned out that a fundamental structure of the ZFS pool known as Meta Object Set (MOS) was badly damaged, therefore the pool couldn’t be restored to an importable state, at least not without a more than considerable amount of effort, but reading what was readable brought back to life 1.4 TiB of data, among which all the research data and theses. Most of the hard work in restoring the data happened automatically using a Python tool I assembled in the process. It wasn’t easy to develop, especially given the poor state of implementation information about ZFS and the differences between the on-disk format and the one described in the only official documentation I was able to find online, therefore I’m making it available under open-source license in hope that it could help someone else too. Most kudos go to Max Bruning for his conference talks on YouTube and especially the ZFS Raidz Data Walk blog article and to the FreeBSD project for their excellent port of ZFS, which served as reference source code for some of the modules in my implementation.

The tool–aptly named py-zfs-rescue–is available under the 3-clause BSD license in its GitHub repo.

Technical Details

py-zfs-rescue is not for the faint of heart as there are no command-line options of any kind and the configuration is performed exclusively by altering the source code, therefore some knowledge of Python 3 is required. Also, one has to have a good idea of how ZFS is structured internally, thus I provide here a quick overview of the ZFS on-disk format.

ZFS is an incredibly complex filesystem, but fundamentally it is just a huge tree of blocks with the leaf blocks containing data and the rest containing various types of metadata. The tree is rooted in what is known as uberblock, which serves the same purpose as the superblock in most filesystems. The uberblock itself (actually an entire array of uberblocks) is part of the ZFS label, four copies of which are found on any device, disk partition or file that are part of a ZFS vdev and contains besides the uberblock array a collection of key-value pairs with information about the type of the vdev and its constituent elements. A typical label (in a 6-dev raidz1) looks like this:

txg: 8510825
name: pool
version: 10
guid: 6106808927530115088
vdev_tree:
  children[0]:
    guid: 6106808927530115088
    id: 0
    type: disk
    path: /dev/dsk/c3t0d0s7
    devid: id1,sd@f0000000049be3b9a000ea8d90002/h
    whole_disk: 0
    phys_path: /pci@0,0/pci15d9,d280@1f,2/disk@0,0:h
    DTL: 35
  ... two children omitted for brevity ...
  children[3]:
    guid: 9245908906035854570
    id: 3
    type: disk
    path: /dev/dsk/c3t3d0s7
    faulted: 1
    devid: id1,sd@f0000000049bbf10a000ac4500003/h
    whole_disk: 0
    phys_path: /pci@0,0/pci15d9,d280@1f,2/disk@3,0:h
    DTL: 33
  ... two children omitted for brevity ...
  guid: 14559490109128549798
  asize: 2899875201024
  nparity: 1
  id: 0
  metaslab_array: 14
  metaslab_shift: 34
  is_log: 0
  ashift: 9
  type: raidz
hostid: 237373741
pool_guid: 1161904676014256579
hostname: spof
state: 0
top_guid: 14559490109128549798

The label contains the pool name (pool in our case), the guid of the component, the ZFS version, the list of vdevs in the pool, their type and constituent devices (vdev_tree), the pool state, and information about the host that the pool belongs to (ours is named spof for Single Point Of Failure, which it indeed proved to be…) By matching the GUIDs of the individual vdev components with the GUIDs in the vdev_tree list the OS is capable of assembling the pool even if the device names/paths change. Faulty components are marked accordingly like the fourth child (/dev/dsk/c3t3d0s7) in this case. There are two copies of the label at the beginning of each component and two copies at the end.

Each and every I/O operation in ZFS is performed in the context of a specific transaction, which groups a set of modifications to the data stored on the disk. When a ZFS object is written to the disk, the transaction number is recorded as part of the metadata. Unlike most other filesystems, ZFS stores data and metadata in blocks of varying sizes. Each block is located by its block pointer, which holds the type of the block, checksum of its contents, the location and physical (eventually compressed) size (collectively known as DVA) of up to three copies of the block data, the logical size of the data, and the compression type.

ZFS is organised as a set of objects with each object represented by a dnode (equivalent to the inode in Unix filesystems) containing pointers to up to three associated groups of data blocks. For some really small objects the data is stored within the free space of the dnode block itself and there are no associated data blocks. dnodes are organised in arrays called Object Sets with the notable exception of the dnode for the top-level object set (the MOS), a pointer to which is located in the uberblock. By default, there are three copies of the MOS, two copies of the other metadata objects including the object sets and directory objects, and a single copy of the file data blocks. The metadata blocks are usually compressed with LZJB (LZ4 in newer ZFS versions) while the file data blocks are uncompressed unless the dataset is configured accordingly. There is a maximum block size of 128 KiB and larger objects are stored using block trees with blocks at the nodes containing arrays of block pointers to the lower levels. Some simple modulo and integer division arithmetic is used to figure out which intermediate (indirect in ZFS terminology) block at each level of the tree contains the relevant pointer. The depth of the block tree is stored in the dnode. All top-level objects such as datasets, dataset property lists, space maps, snapshots, etc., are stored in the MOS.

Datasets are implemented as separate object sets consisting of all files and directories in a given dataset plus two (or more in newer ZFS implementations) special ZFS objects–the master node of the dataset and the delete queue. A typical dataset looks like this:

[ 0] <unallocated dnode>
[ 1] [ZFS master node] 1B 1L/16384 blkptr[0]=<[L0 ZFS master node] 200L/200P DVA[0]=<0:92e4c6400:600> DVA[1]=<0:b1ca2b1800:600> birth=448585 fletcher4 off LE contiguous fill=1>
[ 2] [ZFS delete queue] 1B 1L/16384 blkptr[0]=<[L0 ZFS delete queue] 200L/200P DVA[0]=<0:b2024e5c00:600> DVA[1]=<0:13110532c00:600> birth=449850 fletcher4 off LE contiguous fill=1>
[ 3] [ZFS directory] 1B 1L/16384 blkptr[0]=<[L0 ZFS directory] 200L/200P DVA[0]=<0:b2024e5800:600> DVA[1]=<0:13110532800:600> birth=449850 fletcher4 off LE contiguous fill=1> bonus[264]
[ 4] [ZFS plain file] 1B 1L/16384 blkptr[0]=<[L0 ZFS plain file] 200L/200P DVA[0]=<0:92e4ff400:600> birth=448593 fletcher2 off LE contiguous fill=1> bonus[264]
[ 5] [ZFS plain file] 1B 1L/16384 blkptr[0]=<[L0 ZFS plain file] 200L/200P DVA[0]=<0:92e4ffc00:600> birth=448593 fletcher2 off LE contiguous fill=1> bonus[264]
[ 6] [ZFS plain file] 1B 1L/16384 blkptr[0]=<[L0 ZFS plain file] 200L/200P DVA[0]=<0:92e500000:600> birth=448593 fletcher2 off LE contiguous fill=1> bonus[264]
[ 7] [ZFS plain file] 1B 1L/16384 blkptr[0]=<[L0 ZFS plain file] 200L/200P DVA[0]=<0:b0c5cd3400:600> birth=450114 fletcher2 off LE contiguous fill=1> bonus[264]
[ 8] [ZFS directory] 1B 1L/16384 blkptr[0]=<[L0 ZFS directory] 200L/200P DVA[0]=<0:95e52f000:600> DVA[1]=<0:b1ce66a400:600> birth=449176 fletcher4 off LE contiguous fill=1> bonus[264]
... (many) more dnodes ...

Directories are implemented simply as key-value pair collections with the file name being the key and a bit field of the index in the object set and the file type the value and are stored in so-called ZAPs (ZAP stands for ZFS Attribute Processor). The master node of each dataset (always at index 1 in the object set) contains the index of the root directory’s ZAP, which index tends to be always equal to 3. File metadata such as owner, permissions, ACLs, timestamps, etc. is stored in the file’s dnode. In order to reach the content of a specific file in a given dataset, the following has to be done:

Locate the dataset’s dnode in the MOS
Read the content of the dataset’s object set
Read the master node to find the root directory’s index
Read the root directory to find the index of the next directory in the file path
Repeat recursively the directory traversal until the index of the file object is found
Walk the associated block tree to find pointers to all the file data blocks

The Python tool is capable of recursively following the root directory of a given dataset and either producing a CSV file with the content of the fileset (similar to ls -lR) or creating a tar file of the content with the associated metadata (owner, timestamps, and permissions). It keeps symbolic links but ignores device nodes and special files. It is possible to configure it to skip certain objects (provided as lists of IDs), which is useful when working with really large datasets. The current version performs caching of certain objects, most notably the block trees, and achieves about 11 MiB/s read speed on our faulted server without any read-ahead optimisations. A peculiar feature is the ability to access the pool remotely via a simple binary TCP protocol, e.g., over an SSH tunnel, which is exactly how I was using it throughout the entire development process. This was more a result of the way the program was developed than a deliberate design decision, but I think it’s pretty nifty. ZFS mirror and raidz1 vdevs as implemented in ZFS version 10 (the one that an ancient Solaris 10 x86 comes with) are supported. For raidz1 the tool is able to recover information on faulty devices using the checksum. Up to date status information is available on the project’s GitHub page.

I really hope nobody will ever need to use this tool.

Recipe: Obtaining Peak VM Size in Pure Fortran

Thu, 22 May 2014 12:34:30 +0200

Often in High Performance Computing one needs to know about the various memory metrics of a given program with the peak memory usage probably being the most important one. While the getrusage(2) syscall provides some of that information, it’s use in Fortran programs is far from optimal and there are lots of metrics that are not exposed by it.

On Linux one could simply parse the /proc/PID/status file. Being a simple text file it could easily be processed entirely with the built-in Fortran machinery as shown in the following recipe:

program test
  integer :: vmpeak

  call get_vmpeak(vmpeak)
  print *, 'Peak VM size: ', vmpeak, ' kB'
end program test

!---------------------------------------------------------------!
! Returns current process' peak virtual memory size             !
! Requires Linux procfs mounted at /proc                        !
!---------------------------------------------------------------!
! Output: peak - peak VM size in kB                             !
!---------------------------------------------------------------!
subroutine get_vmpeak(peak)
  implicit none
  integer, intent(out) :: peak
  character(len=80) :: stat_key, stat_value
  !
  peak = 0
  open(unit=1000, name='/proc/self/status', status='old', err=99)
  do while (.true.)
    read(unit=1000, fmt=*, err=88) stat_key, stat_value
    if (stat_key == 'VmPeak:') then
      read(unit=stat_value, fmt='(I)') peak
      exit
    end if
  end do
88 close(unit=1000)
  if (peak == 0) goto 99
  return
  !
99 print *, 'ERROR: procfs not mounted or not compatible'
  peak = -1
end subroutine get_vmpeak

The code accesses the status file of the calling process /proc/self/status. The unit number is hard-coded which could present problems in some cases. Modern Fortran 2008 compilers support the NEWUNIT specifier and the following code could be used instead:

integer :: unitno 

open(newunit=unitno, name='/proc/self/status', status='old', err=99)
! ...
close(unit=unitno)

With older compilers the same functionality could be simulated using the following code.

MPI Programming Basics

Tue, 18 Mar 2014 17:38:34 +0100

Embracing the current development in educational technologies, the IT Center of the RWTH Aachen University (former Center for Computing and Communication) makes available online the audio recordings of most tutorials delivered during this year’s PPCES seminar. Participation in PPCES is for free and course materials are available online, but this is the first time when proper audio recordings were taken.

All videos (presentation slides + audio) are available on the PPCES YouTube channel under Creative Commons Attribution license. Course materials are available in the PPCES 2014 archive under unclear (read: do not steal blatantly) license.

My own contribution to PPCES - as usual - consists of:

Message passing with MPI, part 1: Basic concepts and point-to-point communication
Message passing with MPI, part 2: Collective operations and often-used patterns
Tracing and profiling MPI applications with VampirTrace and Vampir

Big thanks to all the people who made recording and publishing the sessions possible.

Linear Congruency Considered Harmful

Sun, 15 Dec 2013 21:43:32 +0100

Recently I stumbled upon this Stack Overflow question. The question author was puzzled with why he doesn’t see any improvement in the resultant value of $\pi$ approximated using a parallel implementation of the well-known Monte Carlo method when he increase the number of OpenMP threads. His expectation was that, since the number of Monte Carlo trials that each thread performs was kept constant, adding more threads would increase linearly the sample size and therefore improve the precision of the approximation. He did not observe such improvement and blamed it on possible data races although all proper locks were in place. The question seems to be related to an assignment that he got at his university. What strikes me is the part of the assignment, which requires that he should use a specific linear congruential pseudo-random number generator (LCPRNG for short). In his case a terrible LCPRNG.

An inherent problem with all algorithmic pseudo-random number generators is that they are deterministic and only mimic randomness since each new output is a well-defined function of the previous output(s) (thus the pseudo- prefix). The more previous outputs are related together, the better the “randomness” of the output sequence could be made. Since the internal state can only be of a finite length, every now and then the generator function would map the current state to one of the previous ones. At that point the generator starts repeating the same output sequence again and again. The length of the unique part of the sequence is called the cycle length of the generator. The longer the cycle length, the better the PRNG.

Linear congruency is the worst method for generating pseudo-random numbers. The only reason it is still used is that it is extremely easy to be implemented, takes very small amount of memory, and it works acceptably well in some cases if the parameters are chosen wisely. It’s just that Monte Carlo simulations are rarely that cases. So what is the problem with LCPRNGs? The problem is that their output depends solely on the previous one as the congruential relation is

$$p_{i+1} \equiv (A \cdot p_i + B),(mod,C),$$

where $A$, $B$ and $C$ are constants. If the initial state (the seed of the generator) is $p_0$, then the i-th output is the result of $i$ applications of the generator function $f$ to the initial state, $p_i = f^i(p_0)$. When it happens that an output repeats the initial state, i.e., $p_N = p_0$ for some $N > 0$, the generator loops since

$$p_{N+i} = f^{N+i}(p_0) = f^i(f^N(p_0)) = f^i(p_N) = f^i(p_0) = p_i.$$

As is also true with the human society, short memory leads to history repeating itself in (relatively short) cycles.

The generator from the question uses $C = 741025$ and therefore it produces pseudo-random numbers in the range $[0, 741024]$. For each test point two numbers are sampled consecutively from the output sequence, therefore a total of $C^2$ or about 550 billion points are possible. Right? Wrong! The choice of parameters results in this particular LCPRNG having a cycles length of 49400, which is orders of magnitude worse than the otherwise considered bad ANSI C pseudo-random generator rand(). Since the cycle length is even, once the sequence folds over, the same set of 24700 points is repeated over and over again. The unique sequence covers $49400/C$ or about 6,7% of the output range (which is already quite small).

A central problem in Monte Carlo simulations is the so called ergodicity or the ability of the simulated system to pass through all possible states. Because of the looping character of the LCPRNG and the very short cycle length, there are many states that remain unvisited and therefore the simulation exhibits really bad ergodicity. Not only this, but the output space is partitioned into 16 ($\lceil C/49400\rceil$) disjoint sets and there are only 16 unique initial values (seeds) possible. Therefore only 32 different sets of points can be drawn from that generator (why 32 and not 16 is left as an exercise to the reader).

How this relates to the bad approximation of $\pi$? The method used in the question is a geometric approximation based on the idea that if a set of points ${ P_i }$ is drawn randomly and uniformly from $ [0, 1) \times [0, 1) $, the probability that such a point lies inside a unit circle centred at the origin of the coordinate system is $\frac{\pi}{4}$. Therefore:

$$\pi \approx 4\frac{\sum_{i=1}^N \theta{}(P_i)}{N},$$

where $\theta{}(P_i)$ is an indicator function that has a value of 1 for all points ${ P(x,y): x^2+y^2 \leq 1}$ and 0 for all other points and $N$ is the number of trials. Now, it is well known that the precision of the approximation is proportional to $1/\sqrt{N}$ and therefore more trials give better results. The problem in this case is that due to the looping nature of the LCPRNG, the sum in the nominator is simply $m \times S_0$, where

$$S_0 = \sum_{i=1}^{24700} \theta{}(P_i).$$

For large $N$ we have $m \approx N/24700$ and therefore the approximation is stuck at the value of:

$$\tilde{\pi} = 4 \frac{\sum_{i=1}^{24700} \theta(P_i)}{24700}.$$

It doesn’t matter if one samples 24700 points or if one samples 247000000 points. The result is going to be the same and the precision in the latter case is not going to be 100 times better but rather exactly the same as in the former case with 9999 times the computational resources used in the former case now effectively wasted.

Adding more threads could improve the precision if:

each thread has its own PRNG, i.e. the generator state is thread-private and not globally shared, and
the seed in each thread is chosen carefully so not to reproduce some other thread’s generator output.

It was already shown that there are at most 32 unique sets of points and therefore using only up to 32 threads makes sense with an expected 5,7-fold increase of the precision of the approximation (less than one decimal digit).

This leaves me scratching my head: was his docent grossly incompetent or did s/he deliberately give him an exercise with such a bad PRNG so that he could learn how easily beautiful Monte Carlo methods are spoiled by bad pseudo-random generators?

It should be noted that having a cyclic PRNG is not necessarily a bad thing. Even if two different seed values result in the same unique sequence, they usually start the generator output at different positions in the sequence. And if the sample size is small relative to the cycle length (or respectively the cycle length is huge relative to the sample size), it would appear as if two independent sequences are being sampled. Not in this case though.

Some final words. Never use linear congruential PRNGs for Monte Carlo simulations! Ne-ver! Use something like Mersenne twister MT19937 instead. Also don’t try to reinvent RANDU with all its ill consequences to the simulation science. Thank you!