PolITiGenomics » storage

Next-Generation Sequencing Informatics Update

dd — Fri, 19 Feb 2010 21:55:23 +0000

I updated the Next-Generation Sequencing Informatics table a few weeks ago but forgot to mention it on the blog. The main update was the 50G configuration of the Illumina GA IIx. Also, the Sides & Associates blog linked to my table and referred to it as a “somewhat dated comparison of next-generation sequencing platforms.” Just to clarify, this table represents average throughput for production systems; not vendor claims about throughput, not future vaporware (and Alejandro Gutierrez corrected his description in the post once I pointed this out). As new systems come online and further improvements are made to existing platforms, the table will be updated.

What’s in an Illumina GA run directory?

dd — Wed, 28 Oct 2009 21:46:40 +0000

One of the main things that differentiates genomics from other endeavors that use a lot of disk space is that genomics file systems tend to have a lot of files (millions). This was true with Sanger sequencing, and it seems to be even more true with next-generation sequencing technologies, especially Illumina/Solexa and AB SOLiD. This large number of files and the parallel access of these files by large computational clusters tends to give most storage solutions great difficulty.

So what, exactly, is in an Illumina run directory? Well, to get breakdowns of file statistics there is a nifty little tool called fsstats. It is just a simple Perl script that crawls through a directory stat’ing files and reporting metrics. For example, when you run it on an Illumina GA IIx 2×100, high cluster density run after the primary analysis has completed, you get the following information about the distribution of file sizes. (I have rearranged and condensed the information to make it fit.)

total 7.46 TB used to store 7.46 TB user data, overhead 0.04%
  count=991227 avg=8076.50 KB
  min=0.00 KB max=13128679.30 KB
           size range    count   %tot  %tot cum       total size   %tot  %tot cum
[       0-       2 KB):   4019 ( 0.41) (  0.41)       3009.03 KB ( 0.00) (  0.00)
[       2-       4 KB):      2 ( 0.00) (  0.41)          6.99 KB ( 0.00) (  0.00)
[       4-       8 KB):    981 ( 0.10) (  0.50)       5964.82 KB ( 0.00) (  0.00)
[       8-      16 KB): 193351 (19.51) ( 20.01)    2588619.88 KB ( 0.03) (  0.03)
[      16-      32 KB):   2656 ( 0.27) ( 20.28)      58586.79 KB ( 0.00) (  0.03)
[      32-      64 KB):    901 ( 0.09) ( 20.37)      31369.79 KB ( 0.00) (  0.03)
[      64-     128 KB):   2893 ( 0.29) ( 20.66)     303872.38 KB ( 0.00) (  0.04)
[     128-     256 KB):      2 ( 0.00) ( 20.66)        345.34 KB ( 0.00) (  0.04)
[     256-     512 KB):      4 ( 0.00) ( 20.66)       1222.53 KB ( 0.00) (  0.04)
[     512-    1024 KB):      1 ( 0.00) ( 20.66)        622.26 KB ( 0.00) (  0.04)
[    1024-    2048 KB):      2 ( 0.00) ( 20.66)       3199.89 KB ( 0.00) (  0.04)
[    2048-    4096 KB):     12 ( 0.00) ( 20.66)      41779.69 KB ( 0.00) (  0.04)
[    4096-    8192 KB): 776654 (78.35) ( 99.02) 5863161178.18 KB (73.24) ( 73.28)
[   16384-   32768 KB):     21 ( 0.00) ( 99.02)     487156.46 KB ( 0.01) ( 73.28)
[   32768-   65536 KB):   3856 ( 0.39) ( 99.41)  163552521.17 KB ( 2.04) ( 75.32)
[   65536-  131072 KB):   3825 ( 0.39) ( 99.79)  307535341.32 KB ( 3.84) ( 79.17)
[  131072-  262144 KB):    133 ( 0.01) ( 99.81)   32458046.12 KB ( 0.41) ( 79.57)
[  262144-  524288 KB):   1787 ( 0.18) ( 99.99)  658830514.46 KB ( 8.23) ( 87.80)
[ 2097152- 4194304 KB):     16 ( 0.00) ( 99.99)   47898262.36 KB ( 0.60) ( 88.40)
[ 4194304- 8388608 KB):     64 ( 0.01) (100.00)  432084134.39 KB ( 5.40) ( 93.80)
[ 8388608-16777216 KB):     47 ( 0.00) (100.00)  496603147.67 KB ( 6.20) (100.00)

So the total size of the run directory is nearly 7.5 TB and there are almost one million files. The average size of a file in the run directory is about 8 MB and the maximum size is over 13 GB. The images (represented in the 4096-8192 KB range), comprise over 78% of the files and 73% of the total size of the run directory. This significant penalty can be avoided by using RTA and not transferring image files. The largest files are the alignment (ELAND) outputs and the FASTQ files in the GERALD directory. Speaking of directories, here is a breakdown by number of files in each directory.

  count=1652 avg=601.02 ents
  min=0.00 ents max=24720.00 ents
              range   count   %tot  %tot cum total ent   %tot  %tot cum
  [    0-    1 ents]:     4 ( 0.24) (  0.24)      0.00 ( 0.00) (  0.00)
  [    2-    3 ents]:     1 ( 0.06) (  0.30)      2.00 ( 0.00) (  0.00)
  [    8-   15 ents]:     3 ( 0.18) (  0.48)     26.00 ( 0.00) (  0.00)
  [   16-   31 ents]:     2 ( 0.12) (  0.61)     44.00 ( 0.00) (  0.01)
  [  128-  255 ents]:     9 ( 0.54) (  1.15)   1826.00 ( 0.18) (  0.19)
  [  256-  511 ents]:  1616 (97.82) ( 98.97) 775680.00 (78.12) ( 78.32)
  [  512- 1023 ents]:     3 ( 0.18) ( 99.15)   2920.00 ( 0.29) ( 78.61)
  [ 1024- 2047 ents]:     4 ( 0.24) ( 99.39)   7845.00 ( 0.79) ( 79.40)
  [ 2048- 4095 ents]:     2 ( 0.12) ( 99.52)   6775.00 ( 0.68) ( 80.08)
  [16384-32767 ents]:     8 ( 0.48) (100.00) 197760.00 (19.92) (100.00)

The picture for directory entries is a bit muddled since most of the directories are organized around a small multiple of the number of tiles per lane, falling in the 256-511 entries range. The directories in the 16384-32767 entries range? The image analysis (Firecrest) Temp/L00[1-8] directories, each with 24,720 entries (four clu.txt per tile (one per color) and one qcm.xml (XML!) file for each cycle for each tile in a lane).

Next-Generation Sequencing Informatics table update

dd — Mon, 05 Oct 2009 14:45:02 +0000

I have made some updates to the Next-Generation Sequencing Informatics table. Specifically, I have updated the numbers for 454 Ti, including paired-end information, and added information on the Illumina GA IIx. If anyone that is not employed by AB has real-world numbers for SOLiD 3, I’d appreciate you passing them along to me (I’m looking at you drd).

Update: I received some SOLiD 3 number from Nicholas Socci (thanks Nicholas!).

Update2: I received a fuller set of numbers from drd and the SOLiD 3 column is complete (thanks drd!).

Expansion

dd — Fri, 02 Oct 2009 19:04:04 +0000

The Genome Data Center has received a Gold LEED Certification from the U.S. Green Building Council. This is in addition to the Keystone Award from the St. Louis Association of General Contractors. It is quite an achievement for a power hungry data center to receive a LEED certification, much more a Gold Certification, but the WUSM Design and Construction team along with the architects, engineers, and contractors were able to pull it off.

Recently the final phase of construction at the Genome Data Center was completed. The initial build out had enough power and cooling for about 40 racks of equipment. Now at full capacity, the data center is capable of supplying 4 MW of power (about the amount used by 800 homes on a hot day) and the requisite cooling to the equipment housed within it. This will support over 100 racks worth of high-density computational (blades) and storage equipment and its supporting infrastructure (chilled water plants, air handlers, humidity control, office space, etc.). The electrical system is completely redundant, all the way to the double-ended substation of our electrical utility. That means even if we lose one entire electrical feed, we can still operate on utility power. If we lose both electrical feeds, we have battery and fly-wheel UPS systems to carry us until the two 2 MW diesel generators start (under 10 seconds). The building is about 1480 m² while the actual data center is about 288 m² (as they shrink computing equipment, the required electrical and cooling equipment keeps increasing in size). The data center is arranged in a standard hot aisle/cold aisle layout with cooling delivered from below through floor grates (perf plates did not provide enough airflow) via a 1.2 m raised floor. We currently have about 3,000 cores in our computational cluster and over 3 PB (3,000,000 GB) of storage online. When full of equipment in a few years, the data center will likely house tens of thousands of cores and on the order of 100 PB of storage.

There are more pictures of the Genome Data Center on Flickr.

Data intensive science

dd — Fri, 06 Mar 2009 23:00:12 +0000

There is an interesting article, Beyond the Data Deluge, on Science that discusses how various scientific disciplines are facing massive increases in data generation rates and how they are dealing with it. The article has several useful references for those interested in learning more. It seems the authors worked closely with Jim Gray, who went missing while sailing about two years ago.

Ensembl on Amazon

dd — Fri, 05 Dec 2008 15:46:08 +0000

The Amazon Web Services (AWS) blog has an entry on using Amazon’s Elastic Compute Cloud (EC2) to host and access public data sets, including Ensembl release 51. The data are stored as Amazon Elastic Block Store (Amazon EBS) snapshots. Anyone using EC2 can then create their own EBS using the public data EBS as a starting point. The data are then available to the user to modify, update, and perform calculations using the cloud. You can find more information on how to use the available public data sets and even upload your own data sets at Public Data Sets on AWS.

Next-Generation Sequencing Informatics

dd — Thu, 04 Dec 2008 22:15:59 +0000

I have put together a table with a bunch of important metrics for the major next-generation sequencing platforms: Next-Generation Sequencing Informatics (there is also a link on the left-hand side of the page). It includes number of reads, read length, data sizes, computational time, etc. I will try to keep it as up to date as I can and add new platforms and revisions as they become available. Consider it an early Christmas present.

Living data storage

dd — Tue, 08 Jul 2008 13:22:23 +0000

Researchers in Japan have created the first DNA molecule made from entirely artificially created bases. While some are trying to manufacture artificial life by creating DNA molecules in the lab, the aim of this research is to take advantage of, and in this case expand on, DNA’s high information density to create a very dense data storage platform. Natural DNA has four different bases (A, G, C, and T) it can use to encode information. To expand DNA’s ability to encode information, these researchers created four new bases and integrated them in to a DNA molecule.

N Genomes

dd — Fri, 09 May 2008 20:21:23 +0000

Earlier this week there were several meetings about the 1000 Genomes Project at Cold Spring Harbor Labs. The first meeting Monday morning was about data flow and data repositories. NCBI’s Short Read Archive (SRA) and the equivalent at EBI (which should be ready in a month or two) will house all the data. The pilot projects for the 1000 Genomes Project just started less than two months ago and have already generated as much sequence data as half of the entire trace archive (which contains the sequence data for all publicly funded genome projects over the last 10 years). In other words, this project is going to generate a lot of sequence data (not to mention all the data generated by analysis of the sequence). Paul Flicek from EBI estimates the pilot projects alone will generate about 1 PT (1,000,000 GB) of sequence data. Moving that much data from site to site will be a challenge. Normal solutions, e.g., FTP, rsync, and shipping hard drives, can’t seem to keep up with the data generation rates. NCBI, EBI, and the sequencing centers are testing a high-speed data transfer solution called Aspera scp. It has impressive transfer rates, but seems to stall after a while for no discernible reason. We’ll see if we can get it to work reliably over the coming weeks.

After the data flow meeting was a meeting of the 1000 Genomes Steering Committee. The day and a half that ensued was filled with a lot of lively discussion. When all was said and done, one thing was clear: there are a lot of questions that need to be answered. The analysis group presented convincing results from simulations that indicated 2× coverage in a large number of individual genomes (Pilot 1) is probably not sufficient to detect the rare variants the project is going after (present in 1-2% of the population). The simulations indicated that the power of the study to detect such variants (at a constant cost, i.e., constant total amount of sequence generated) would be greatly enhanced by sequencing half as many people at 4× coverage. There was no firm decision on how to change the pilot (if at all), but going forward it is likely that some of the individuals in Pilot 1 will be sequenced up to 4× or even 8×. Thus, while the project may be named 1000 Genomes, exactly how many genomes we are going to sequence is yet to be determined.

Another issue that arose was the rapid development of the massively parallel sequencing technologies. These platforms (454 FLX, Illumina Genome Analyzer, and AB SOLiD) increase their throughput, improve their data quality, improve analysis software, etc. several times each year. Such dynamic platforms make the development of tools to analyze their data, e.g., align the data to a reference genome and detect variants, very difficult. The right platforms and tools today may not be the best next month or next year when the main project gets underway. This causes two major needs to come to the fore. First, experimental design will not end when the project starts. The experiment will need to be adjusted as capabilities and capacities change. Second, we will not only have to continually develop and refine tools throughout the project, we will need to develop frameworks to continually test and compare the tools that are available. It’s always fun to hit a moving target.

The meeting also discussed the ethical, legal, and social implications (ELSI) of the project. This discussion largely focused on which populations to sample for the project. Should we deepen our knowledge of individuals of Central European, African, and East Asian ancestry to aid in methodology development? Or should we broaden our knowledge of overall human variation by including fewer individuals from a larger number of populations? To be determined…

Lustre bluster

admin — Mon, 18 Feb 2008 19:18:00 +0000

What is Lustre? It is a cluster file system developed by Cluster File Systems, Inc., which was recently purchased by Sun. We do not have much experience with Lustre but we have played around a bit with it in house and I have talked a lot with other centers who do use it (like Sanger). The main advantages of Lustre are that it is very scalable and can sustain very high performance. The two main problems with Lustre is that it is extremely difficult to get working and it is very difficult to back up your data on a Lustre file system. The difficulty in getting it to work seems to be a business strategy of Cluster File Systems. It is hard to get paid for installation and configuration consulting if you have a well-documented, easy to perform process. It will be interesting to see if this will change given Sun CEO Jonathan Schwartz‘s well-publicized commitment to open source software. The second difficulty, back up, may be a bit of a red herring. Do you really want to put archive-worthy data on a high-performance, clustered, 20 TB file systems that depends on a metadata server for file system integrity? Lustre file systems should really be used for high-performance scratch space. Once the calculations are done, move the data somewhere else. Of course, if you have a lot of data, this may take a while. So you may want to move it two places: on-line and off-line storage. Because if you have a lot of data to move, it will also be a lot of data to back up. So you may just want to dump it to tape one time while you are dumping it to your on-line storage.