Update: I received some SOLiD 3 number from Nicholas Socci (thanks Nicholas!).

Update2: I received a fuller set of numbers from drd and the SOLiD 3 column is complete (thanks drd!).

Here are the links that appear on the last slide.

• The Genome Center
• PolITiGenomics (this) blog
• LIMS Paper
• YAPC UR presentation

The Odyssey

So you may be asking yourself, how did he generate a movie of his talk? Even if you are not asking yourself that, I am going to tell you so that if you need to do it (or if I need to do it again), you can avoid a lot of hassle. This was all done on a MacBook using a slide deck created in MS PowerPoint 2008. First I tested screen capture using VLC. It took a few tries with the video settings to make it not look terrible (use H.264 at 1024 kb/s bitrate, 25 or more fps, MPEG 4 encapsulation), but the audio capture did not work. To compensate for the lack of audio, I did audio capture using GarageBand (podcast project type) at the same time I did the video capture. I then loaded the audio and the video into iMovie HD, synchronized the audio to the video, clipped the beginning and the end, and exported. This resulted in a fair quality product. Being stupid, I thought I could do better. I next tested recording narration in MS PowerPoint, but there didn’t seem to be a good way to export both a video of the slides with the recorded timings and the audio into a single format (not to mention it stupidly saves the audio in seemingly uncompressed files without an extension, one file per slide). So I imported the slides into Keynote 08, cleaned up the messes created by the import, recorded the slide show, and exported to QuickTime at high quality. This looked and played great in QuickTime so I went to upload it to YouTube. Sorry, YouTube has a ridiculous and arbitrary limit of 10 minutes for a video. Moving on to blip.tv (which, nicely, also supports Creative Commons licensing), I uploaded the video. After waiting for it to convert, I played it and noticed that although the audio was fine, the slides did not advance. Thinking something was wrong with the blip.tv Flash converter, I moved on to Vimeo. Slides didn’t advance there either. So next I checked the video by watching it using VLC and mplayer. No slide advancement. Trying to fix it, I went into the custom video settings when exporting in KeyNote, trying all sorts of combinations to generate a video that played well in VLC. Using various settings, I was able to get the slides to advance for a while, but eventually they would stop. Audio was always fine. I tried upgrading to Mac OS X 10.5. Still no luck (although in 10.5, VLC can capture from the iSight camera, so maybe audio capture will work in VLC now). I then recalled that the MPEG-2 on Mac was a little quirky (QuickTime won’t played decoded Tivo Series 2 video without first converting them to MPEG-4 using ffmpeg). As a last ditch effort, I imported the KeyNote exported movie into iMovie HD as an MPEG-4 project. I check to make sure iMovie HD played it correctly, then exported at full quality as an MPEG-4. This ballooned the size of the video from around 20 MB to over 110 MB, lessened the quality, and introduced a strange pulsing phenomena in the slides (probably due to compression degradation being corrected by key frames every second or so), but it seemed to play correctly in VLC and was higher quality than the VLC capture video. This video, uploaded to blip.tv, is what you see above.

Update: Here is how Lawrence Lessig screencasts.

Later in the program there was also an interesting segment on geneticist and rocker, Pardis Sabeti, who pioneered a statistical approach to determine if mutations in a population were random or enriched due to natural selection.

Speaking of Neil deGrasse Tyson, check out the tour of the Hayden Planetarium he gives to Stephen Colbert so Stephen can become an astrophysicist if the Colbert Report doesn’t pan out.

The SRA is currently accepting 454 data in standard flowgram format (SFF) and Solexa in SRF format. Soon 454 and AB SOLiD will support the SRF format and submissions will commence in that format for those platforms. The SFF format contains the flowgrams (intensity per cycle at each spot), base calls, and base quality values. In other words, the SFF is very similar to the SCF format used for capillary sequencing data (except flowgrams are discrete whereas chromatograms are continuous). Also, NCBI (as recently discussed) has developed their own storage format for massively parallel sequencing data that they will also be accepting as a submission format within the next few months.

So what is an SRF? Well, it is basically just a container format, i.e., what you store in it is up to the implementation. Thus far, SRF has only been implemented for Illumina/Solexa data; so the rest of this post is specific to that platform and the data types that its implementation of the SRF format contains. The Solexa SRF implementation was done largely by James Bonfield at Sanger and is distributed as part of the io_lib package (now distributed separately from the Staden package). I would imagine that the SOLiD implementation will be very similar to the Solexa implementation. The 454 implementation will likely be very similar to the SFF already in wide use.

For the 1000 Genomes pilot projects, the 1000 Genomes Data Collection Center (DCC) is asking that we submit the “raw”, “processed”, and “base” data for each spot. Raw data are the intensity values (int) and noise (nse) values. Processed data are the processed intensity values (sig2) and four-channel quality values (prb). Base data are the base calls (the quality value is gotten from the prb for the called base). This results in about 50 bytes per base for the SRF. Compared to 2 bits per base, the minimum possible for DNA’s four letter alphabet, this is a 200-fold increase. So not only do these instruments generate a lot more data, we are storing more information per base now too. The average submission for an Solexa run is about 100 GB.

Why store all this extra information? Essentially, people do not trust/believe the data at this point. The quality values provided by these pipelines are not as reliable as those generated for capillary sequence data. Some people want the raw data so that they can develop and improve base calling/quality algorithms. Clearly you would not need all the 1000 Genomes data to develop such algorithms (although the technology changes at such a rate that you would likely want some rolling subset of the latest runs). Others want the raw data because they think they may want to go back and re-analyze data when better algorithms become available. For a wide variety of reasons (disk space, computational cost, network bandwidth, keeping pace with newly generated data), I doubt any such massive re-analysis will ever take place.

This increase in data does not come without a price. Up until now, the primary analysis (image processing and base calling) of 454 data was able to be performed in a few hours on a moderately powerful computer. With the increased data output, primary analysis requires a small cluster: 20 cores with 1 GiB RAM per core having shared access to 1-2 TB of disk space. While those are the minimal requirements, 10 cores per run region seem to be the sweet spot for best performance. The initial production release will support Red Hat-compatible GNU/Linux distributions (RHEL, CentOS, and Fedora). Previous releases also only officially supported Red Hat-like operating systems but we have not had a problem running them on Debian GNU/Linux (454 also indicated they are pushing toward LSB3 compliance). Fortunately, 454 is eliminating the hard-coded dependence that the software be installed and the analysis processes have write access to /usr/local/rig. This will make installation across a cluster much easier. They are also abandoning their custom IPC implementation in favor of the “standard” MPI, specifically OpenMPI or MPICH2. While it is good that they are using a standard IPC implementation, it is unfortunate that MPI implementations are so fragmented and often incompatible, i.e., if one vendor uses MPICH2 and another uses LAM, you need to set up different systems to support each because they cannot coexist on the same system without problems.

I know this is unrelated to informatics, but if you will allow me to journey back to my transport phenomena days as a chemical engineer, the new picotiter plate requires much smaller beads, about 1 micron in diameter. At these length scales, transport phenomena, specifically boundary affects and polymer diffusion, may become important during the emulsion PCR and sequencing. Someone needs to calculate a Reynolds number.

Oh, one more thing, there is talk of paired-end reads with 20 kb inserts.

After the data flow meeting was a meeting of the 1000 Genomes Steering Committee. The day and a half that ensued was filled with a lot of lively discussion. When all was said and done, one thing was clear: there are a lot of questions that need to be answered. The analysis group presented convincing results from simulations that indicated 2× coverage in a large number of individual genomes (Pilot 1) is probably not sufficient to detect the rare variants the project is going after (present in 1-2% of the population). The simulations indicated that the power of the study to detect such variants (at a constant cost, i.e., constant total amount of sequence generated) would be greatly enhanced by sequencing half as many people at 4× coverage. There was no firm decision on how to change the pilot (if at all), but going forward it is likely that some of the individuals in Pilot 1 will be sequenced up to 4× or even 8×. Thus, while the project may be named 1000 Genomes, exactly how many genomes we are going to sequence is yet to be determined.

Another issue that arose was the rapid development of the massively parallel sequencing technologies. These platforms (454 FLX, Illumina Genome Analyzer, and AB SOLiD) increase their throughput, improve their data quality, improve analysis software, etc. several times each year. Such dynamic platforms make the development of tools to analyze their data, e.g., align the data to a reference genome and detect variants, very difficult. The right platforms and tools today may not be the best next month or next year when the main project gets underway. This causes two major needs to come to the fore. First, experimental design will not end when the project starts. The experiment will need to be adjusted as capabilities and capacities change. Second, we will not only have to continually develop and refine tools throughout the project, we will need to develop frameworks to continually test and compare the tools that are available. It’s always fun to hit a moving target.

The meeting also discussed the ethical, legal, and social implications (ELSI) of the project. This discussion largely focused on which populations to sample for the project. Should we deepen our knowledge of individuals of Central European, African, and East Asian ancestry to aid in methodology development? Or should we broaden our knowledge of overall human variation by including fewer individuals from a larger number of populations? To be determined…