For analyzing a single genome, you really can’t beat that price. Of course, at the rate next-generation sequencing instruments are generating data, most people are not going to want to analyze just one genome. So the question becomes, what is the break even point? That is, how many genomes do you have to sequence to make buying compute resources cheaper than renting them from Amazon? We currently estimate that the fully loaded (node, chassis, rack, networking, etc.) cost of a single computational core is about $500. Thus, to purchase 320 cores would cost you about $160,000. It’s going to take a lot (1280) genomes to hit that break even point. But, do you really need to analyze a genome in three hours? With the current per run throughput of a single Illumina GA IIx, it would take about four ten-day runs (40 days) to generate 38× coverage of a human genome. After each run, you could align the sequence data from that run. Each lane of data would take 8-12 core·hours to align, so a whole run’s (eight lanes’) worth of data would take about 80 core·hours. Therefore, even if you had just one core, you could align all the data before the next run completed. The consensus calling and variant detection portions of the pipeline typically take a handful of core·hours and therefore do not change the economics; they too can be completed before the first run of the next genome is completed. Thus, with a $500 investment in computational resources, you can more than keep pace with the Illumina instrument. Note that I am completely excluding the cost of storage, as that will be needed for the data and results regardless of where the computation is done. Of course, you probably wouldn’t buy just one core. Checking over at the Dell Higher Education web site, you can get a Quad Core Precision T3500n with 4 GiB of RAM (more RAM per core than the Amazon EC2 Extra Large Instance used in the paper) and 750 GB local storage capacity (about the same storage per core as the Extra Large Instance) for $1700. You would need less than one core’s (25%) of that workstation’s capacity dedicated to alignment of and variant detection on data from a single Illumina GA IIx (thanks to Burrows-Wheeler Transform aligners like bowtie and bwa). Using the single core numbers, the break even point for purchase versus cloud is less than five whole genomes. Using the entire cost of the Dell workstation (even though you require less than 25% of its computational capacity), the break even point is about 14 genomes. It would take about 1.5 years (about half the expected life of IT hardware) at current throughput to sequence 14 genomes with a single Illumina GA IIx. At data rates expected in January 2010, it would take less than a year to break even.

These numbers indicate that unless you are just sequencing a few genomes, you are probably better off purchasing a (possibly single node) cluster. With the proliferation of sequencing applications and publications in the last couple years, not many researchers will fall into the “few genomes” bin. Our experience has been that the more sequencing data people get, the more they want. Another way to look at this is that the entire analysis computational hardware costs (<$1700) is less than 1% of the sequencing instrument cost; or the computational cost to analyze a whole genome (<$500) is less than 1% of the total data generation costs (reagents, flow cells, instrument depreciation, technician time, etc.). This is all not to say that there is not a place for cloud and other distributed computing frameworks in bioinformatics, but that's the topic of a future post.

More and more data are being submitted to the NCBI Short Read Archive (SRA). So you may ask yourself, “How am I going to download all that data?” Well, as luck would have it, you can download it using the same high-speed network protocol that we use to upload it, Aspera. You can download the Aspera Connect browser plugin (it is offered at not cost, but sadly is not free), install it, and then begin downloading data at near line speed in no time. Of course, if your line speed is not so hot, Aspera cannot help you much.

In a previous post, I mentioned some of the difficulties in using the Aspera scp client. The president of Aspera, Michelle Munson, posted a good retort to my musings, which I reproduce below for ease of viewing. Basically, to avoid problems, don’t allow Aspera scp to transfer data faster than your system can provide it. If you do so, I can report that Aspera scp behaves quite reliably (and still speedily). Well, except for the time NCBI overrode on the server side the bandwidth limit we set on the client side, increasing it beyond what the back-end disk systems were happy with. After contacting NCBI, we were told they wouldn’t do that any more.

Here is Ms. Munson’s comment.

Hello all,

On the use of Aspera Scp, the stalling behavior described is a result of artificially induced heavy packet loss for the FASP protocol, usually due to setting a target transfer rate that significantly exceeds the throughput to the storage system on the receiver side. The other cause is bandwidth shaping/artificial dropping of UDP traffic along the transmission path.

The Aspera transfer logs (routed to syslog on Unix systems) have detailed statistics that we can interpret for you which will indicate the root cause.

Assuming that the receiver side I/O throughput is overdriven, you can verify this for yourselves by running a 3rd party disk benchmarking utility such as bonnie++. Use bonnie to measure the write throughput for blocks of 64K and 1 MB (Aspera software uses a configurable block size, 64K by default).

Once you know the disk throughput bottleneck, you can either set a target rate that does not exceed, or better yet, as of our 2.2 release (available as of April 2009) you can configure on the storage rate control option, which will automatically adapt the transmission rate to the storage throughput. This is much like network congestion control extended to the storage systems (a patent-pending innovation by our company).

If you have any questions or problems on the above, be glad to help over here at Aspera. You can reach us at support@asperasoft.com or email me directly, michelle@asperasoft.com.

Thank you,
Michelle Munson
President, Aspera, Inc.

Update: The story has been posted on Slashdot.

1. SGE 6.2 design goal includes supporting a single array job with 500,000 tasks and hundreds of thousands of concurrent jobs
2. People have been running hundreds of thousands of SGE jobs per week since the SGE 5.3 days many years ago
3. I personally know of several sites pushing hundreds of thousands of heavy SGE jobs per week through their systems right now
4. SGE 6.2 runs a 62,000 core cluster in Texas (RANGER) and has been for some time

“tens of thousands of jobs” is actually pretty easy with Grid Engine and has been for some time, scaling issues encountered in this range have more to do with bad spooling decisions, filesystem design and occasionally an overwhelmed qmaster host. The developers have worked quite a bit this year to improve threading performance, reduce memory footprints and remove things like external RSH methods that consumed system resources like filehandles and TCP ports etc.

This is especially evident in the SGE 6.2 and 6.2u1 release series where speed and scaling were specifically addressed as part of the design effort (6.2u3 and 6.3 will introduce new features). This is the reason why the SGE scheduler is now a thread within the qmaster – one of the more obvious user-visible changes made recently. (emphasis mine – dd)

There are many reasons why one would chose between LSF vs SGE (I have used both for years now) but scaling is not one of the significant selection factors. Features, price, APIs and quality of documentation are far more important along with community adoption/support.

I would guess breaking out the scheduler into its own thread is a major factor in SGE’s ability to manage so many jobs. This was the major deficiency of SGE and other batch schedulers we tested at the time. Several systems designed their schedulers to automatically run through the list of jobs a certain intervals. With a lot of jobs in the queue, the scheduler would not finish its previous traversal before the new one was scheduled to start. Depending on the design implementation this meant that either the original scheduling was killed and the scheduler never processed some jobs or that scheduler threads kept spawning until the resources were exhausted on the master node (that’s bad).

(A couple asides here, since GAPipeline is built on Makefile’s, another option that came up in the thread was parallel execution across an LSF cluster using distmake. Because of our interest in grid computing, we are currently investigating replacing LSF with Condor.)

Of course, with the roll out of SCS2.4 with RTA (real-time analysis), most of the primary analysis is now done on the instrument control computer. Thus, all of this talk about the requirements to produce sequence from the machine are made much less important. Now there is only one stage of the pipeline, the alignment and reporting (called GERALD), now run off the instrument computer. The most computationally intensive part of this stage of the pipeline is the alignment (ELAND and its post-processing) and it can only be made parallel on a per lane basis, i.e., eight ways.

Of course, there is also the specter of the requirements for sequence analysis at Illumina GA IIx scale, but that’s a whole other post…

Hello David,

We are extending early registration until June 23, 2009 for OSCON. A savings of $250 off standard registration. Because you have been accepted as a speaker for this year’s show, we would like to extend an additional savings for your friends, family and co-workers. If you have a blog, newsletter or website, please feel free to use and distribute this 20% off code: os09fos.

Act now while early registration is still in effect.

So head on over the the OSCON registration site and enter the discount code os09fos. I hope to see you there!

Update: Updated links to latest released version (0.6).