Comments on: N Genomes http://www.politigenomics.com/2008/05/n-genomes.html Politics, Information Technology, and Genomics Mon, 31 Oct 2011 00:27:27 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Michelle Munson http://www.politigenomics.com/2008/05/n-genomes.html/comment-page-1#comment-9839 Michelle Munson Wed, 29 Apr 2009 20:49:14 +0000 http://www.politigenomics.com/?p=74#comment-9839 Hello all, On the use of Aspera Scp, the stalling behavior described is a result of artificially induced heavy packet loss for the FASP protocol, usually due to setting a target transfer rate that significantly exceeds the throughput to the storage system on the receiver side. The other cause is bandwidth shaping/artificial dropping of UDP traffic along the transmission path. The Aspera transfer logs (routed to syslog on Unix systems) have detailed statistics that we can interpret for you which will indicate the root cause. Assuming that the receiver side I/O throughput is overdriven, you can verify this for yourselves by running a 3rd party disk benchmarking utility such as bonnie++. Use bonnie to measure the write throughput for blocks of 64K and 1 MB (Aspera software uses a configurable block size, 64K by default). Once you know the disk throughput bottleneck, you can either set a target rate that does not exceed, or better yet, as of our 2.2 release (available as of April 2009) you can configure on the storage rate control option, which will automatically adapt the transmission rate to the storage throughput. This is much like network congestion control extended to the storage systems (a patent-pending innovation by our company). If you have any questions or problems on the above, be glad to help over here at Aspera. You can reach us at support@asperasoft.com or email me directly, michelle@asperasoft.com. Thank you, Michelle Munson President, Aspera, Inc. Hello all,

On the use of Aspera Scp, the stalling behavior described is a result of artificially induced heavy packet loss for the FASP protocol, usually due to setting a target transfer rate that significantly exceeds the throughput to the storage system on the receiver side. The other cause is bandwidth shaping/artificial dropping of UDP traffic along the transmission path.

The Aspera transfer logs (routed to syslog on Unix systems) have detailed statistics that we can interpret for you which will indicate the root cause.

Assuming that the receiver side I/O throughput is overdriven, you can verify this for yourselves by running a 3rd party disk benchmarking utility such as bonnie++. Use bonnie to measure the write throughput for blocks of 64K and 1 MB (Aspera software uses a configurable block size, 64K by default).

Once you know the disk throughput bottleneck, you can either set a target rate that does not exceed, or better yet, as of our 2.2 release (available as of April 2009) you can configure on the storage rate control option, which will automatically adapt the transmission rate to the storage throughput. This is much like network congestion control extended to the storage systems (a patent-pending innovation by our company).

If you have any questions or problems on the above, be glad to help over here at Aspera. You can reach us at support@asperasoft.com or email me directly, michelle@asperasoft.com.

Thank you,
Michelle Munson
President, Aspera, Inc.

]]> By: dd http://www.politigenomics.com/2008/05/n-genomes.html/comment-page-1#comment-3305 dd Tue, 18 Nov 2008 18:42:08 +0000 http://www.politigenomics.com/?p=74#comment-3305 To the best of my knowledge, none of the formats have been published. In truth, I am trying to track down those formats as well. There are currently two: Sequence Assembly and Mapping (SAM) format and Genotype Likelihood Format (GLF). I'll post more information as I find it. To the best of my knowledge, none of the formats have been published. In truth, I am trying to track down those formats as well. There are currently two: Sequence Assembly and Mapping (SAM) format and Genotype Likelihood Format (GLF). I’ll post more information as I find it.

]]> By: David S http://www.politigenomics.com/2008/05/n-genomes.html/comment-page-1#comment-3246 David S Mon, 17 Nov 2008 21:50:17 +0000 http://www.politigenomics.com/?p=74#comment-3246 David, I went to a talk at ASHG by David Altschuler stating that the project was developing formats to transfer intermediary data types between the sites (i.e formats for analyzed data, raw data, and other steps in the process). Have any of those formats been released or published? When will they be published and will you be including the community in this process? Thanks, David Sexton Center for Human Genetics Research Vanderbilt University David,

I went to a talk at ASHG by David Altschuler stating that the project was developing formats to transfer intermediary data types between the sites (i.e formats for analyzed data, raw data, and other steps in the process). Have any of those formats been released or published? When will they be published and will you be including the community in this process?

Thanks,

David Sexton
Center for Human Genetics Research
Vanderbilt University

]]> By: dd http://www.politigenomics.com/2008/05/n-genomes.html/comment-page-1#comment-56 dd Mon, 30 Jun 2008 14:00:15 +0000 http://www.politigenomics.com/?p=74#comment-56 You can get the FASTQ sequence from the first two data freezes here: ftp://ftp-trace.ncbi.nih.gov/1000genomes/ You can get the FASTQ sequence from the first two data freezes here: ftp://ftp-trace.ncbi.nih.gov/1000genomes/

]]> By: dd http://www.politigenomics.com/2008/05/n-genomes.html/comment-page-1#comment-51 dd Tue, 24 Jun 2008 20:50:16 +0000 http://www.politigenomics.com/?p=74#comment-51 The first data from both Pilot 1 (2×) and 2 (trios) will be available from both NCBI and EBI in FASTQ format. You can get some of the raw data from the NCBI SRA <a href="http://www.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=tracking&m=main&s=tracking&cmd=tracking&pages=12&prev=%3C&page=12" rel="nofollow">tracking page</a>. Quality can mean a lot of things. Thus far, the data quality seems typical of the platforms they are being generated on. Each data generating center is undoubtedly mapping reads back to the human reference, but these alignments are not available currently. I believe the plan is for the 1000 Genomes DCC to provide mapping files once a standard alignment file format for massively parallel sequencing data is settled upon. The first data from both Pilot 1 (2×) and 2 (trios) will be available from both NCBI and EBI in FASTQ format. You can get some of the raw data from the NCBI SRA tracking page. Quality can mean a lot of things. Thus far, the data quality seems typical of the platforms they are being generated on. Each data generating center is undoubtedly mapping reads back to the human reference, but these alignments are not available currently. I believe the plan is for the 1000 Genomes DCC to provide mapping files once a standard alignment file format for massively parallel sequencing data is settled upon.

]]> By: Yong Zhang http://www.politigenomics.com/2008/05/n-genomes.html/comment-page-1#comment-49 Yong Zhang Mon, 23 Jun 2008 16:06:40 +0000 http://www.politigenomics.com/?p=74#comment-49 Hi David, Nice to read this blog discussing 1000 genome project. It seems that NCBI SRA does not house the corresponding 2X sequences as of now. I am also just wondering when these data will get publicly available. Also, how about the sequencing quality? Have these sequence been mapped to the reference genome now? Many thanks ahead. Yong Zhang University of Chicago Hi David,

Nice to read this blog discussing 1000 genome project. It seems that NCBI SRA does not house the corresponding 2X sequences as of now. I am also just wondering when these data will get publicly available. Also, how about the sequencing quality? Have these sequence been mapped to the reference genome now?

Many thanks ahead.

Yong Zhang
University of Chicago

]]>