More and more data are being submitted to the NCBI Short Read Archive (SRA). So you may ask yourself, "How am I going to download all that data?" Well, as luck would have it, you can download it using the same high-speed network protocol that we use to upload it, Aspera. You can download the Aspera Connect browser plugin (it is offered at not cost, but sadly is not free), install it, and then begin downloading data at near line speed in no time. Of course, if your line speed is not so hot, Aspera cannot help you much.
In a previous post, I mentioned some of the difficulties in using the Aspera scp client. The president of Aspera, Michelle Munson, posted a good retort to my musings, which I reproduce below for ease of viewing. Basically, to avoid problems, don't allow Aspera scp to transfer data faster than your system can provide it. If you do so, I can report that Aspera scp behaves quite reliably (and still speedily). Well, except for the time NCBI overrode on the server side the bandwidth limit we set on the client side, increasing it beyond what the back-end disk systems were happy with. After contacting NCBI, we were told they wouldn't do that any more.
Here is Ms. Munson's comment.
On the use of Aspera Scp, the stalling behavior described is a result of artificially induced heavy packet loss for the FASP protocol, usually due to setting a target transfer rate that significantly exceeds the throughput to the storage system on the receiver side. The other cause is bandwidth shaping/artificial dropping of UDP traffic along the transmission path.
The Aspera transfer logs (routed to syslog on Unix systems) have detailed statistics that we can interpret for you which will indicate the root cause.
Assuming that the receiver side I/O throughput is overdriven, you can verify this for yourselves by running a 3rd party disk benchmarking utility such as bonnie++. Use bonnie to measure the write throughput for blocks of 64K and 1 MB (Aspera software uses a configurable block size, 64K by default).
Once you know the disk throughput bottleneck, you can either set a target rate that does not exceed, or better yet, as of our 2.2 release (available as of April 2009) you can configure on the storage rate control option, which will automatically adapt the transmission rate to the storage throughput. This is much like network congestion control extended to the storage systems (a patent-pending innovation by our company).
President, Aspera, Inc.