The files start to download, but then get a permission denied error on hitting the file ". That is fine since I don't want that file, but ascp will error and not continue onto the next file. Morgan, Ric Mackie in our support team did in fact tell you to use the -Q flag when downloading to engage adaptive rate control. Regarding the man page inclusion with 'ascp', we do not include the man page with the Connect client installer because it is intended to be used as a browser plug-in application.
You are using the contained 'ascp' binary which is fine but that is typically done when a user has installed our desktop client package, Aspera client. This package rpm or deb includes the man page. Regarding the permission denied errors when transferring these files, ascp uses the native file system permissions to determine if it can read or write files. We are actually adding support for that in our 2.
I completely agree ascp should support it. Regarding -k2 depending on the use of the -l option, I haven't noticed that myself, but if -l is not specified at the command line, ascp uses a default target rate of 10 Mbps which limits the transfer rate to 10 Mbps on top of the automatically adapted rate -- probably not what you want.
You will want to include a "-l" in your command line options that is as high as you would ever want the transfer to be, e. I realize there are a number of command line options, but it is worth learning them to get the results you want. Users of our GUI client products don't have to worry about any of these -- they are built in to the application. Thanks for all of your feedback. Morgan, 'ascp' does in fact resume a directory download even if no -l is specified, for example try the following: ascp -TQ -k2 asperaweb demo.
Terminate the transfer part-way through. Then repeat the command. The transfer will pick up from where it left off. Any files previously transferred are "skipped" and the transfer resumes from within the file where it was interrupted. Ric Mackie in our support team did in fact tell you to use the -Q flag when downloading to engage adaptive rate control. No, he didn't. I told me to make sure I set the -L limit to the max bandwidth or just below to make sure the packets don't get backed up due to me not having enough bandwidth.
No mention of -Q. So why not allow everyone to download this client? Would have saved me quite a bit of time. Yes, I probably don't have permission to read that particular file. That is fine. The problem is that ascp seems to have problems continuing on afterwards Regarding -k2 depending on the use of the -l option, I haven't noticed that myself, but if -l is not specified at the command line, ascp uses a default target rate of 10 Mbps which limits the transfer rate to 10 Mbps on top of the automatically adapted rate -- probably not what you want.
Actually suggested that the setting of -l M was the reason -K2 wasn't working. The whole point of me making this blog post was to provide information about how to use Aspera to get data from NCBI.
Considering there is no documentation anywhere about the command line program, I thought the default settings would at least allow me to download a few files without problems 'ascp' does in fact resume a directory download even if no -l is specified, for example try the following: ascp -TQ -k2 asperaweb demo. This did not work for me. I have attached a picture to my blog to show my problem.
Morgan, In reply, not necessarily in order Ric Mackie feels very concerned because he did tell you to use the -Q option, but the important point is simply to note the option for your usage going forward. Please feel free to add this info for your users. Regarding your problem with the download not resuming when specifying -k2, I can not see the error picture. It is possible that if you used fixed rate no -Q and a Mbps target rate on 35 Mbps connection that the severe overdrive would cause the transfer to fail rather than complete the resume checks.
That said, by adding -Q to engage adaptive rate control, the transfer session will not overdrive and there will be no problem in resuming transfers. Regarding the permission denied error causing ascp to not continue transferring, that is actually not the behavior at all. Thanks, Michelle. Morgan, I was able to zoom in on the picture you provided and can see the error.
This sort of "Connection lost" error shortly into the transfer precisely means that one end of the transfer was not able to receive UDP traffic on the FASP port from the other end for 10 seconds if the session has not been fully established or for more than 60 seconds if the session has been fully established.
This error is terminating the connection and preventing the resume check and the progress of the transfer. There is more than one possible root cause, and can be determined from the transfer log files. Please email me once available and I will review and reply asap. Michelle, I think many of these problems could have been avoided if NCBI or Aspera posted documentation for the ascp program. If the documentation is not freely available maybe there should be a warning that it is not for command line use?
I think many users would like to use the Aspera software to speed up their downloads, but for some reason NCBI is not providing much information about it or that it even exists. Maybe NCBI is still in a "test" phase of using your software?
Anyway, thanks for your help so far, I think we are making some forward progress. Morgan, I am glad we were able to get to the bottom of the three issues you were facing. Hopefully they can be of help to other users: 1.
The use of the skip special files option at the command line made it appear as if the transfer was not making progress when in fact it was actually skipping multiple special files in a row. The permission denied error caused by the source file permissions being propagated to the destination files, unless you request an alternate directory mask through the ascp configuration file.
Please let us know if you have any other questions. Hopefully you will be able to share your ongoing experiences with your readers. After about 5. A day later, after reading this blog, I restarted the download using the -k 2 option. Hi Alex, 12TB, that is a ton of data. I have been trying to just get all of GenBank downloaded GB and have been failing miserably. I have tried FTP and also using Aspera web plugin and ascp. Aspera is faster, but I will still lose my connection at some point.
The problem with resuming either FTP or Aspera is that it takes so long for the file checking, that often by the time to starts to download again I get another disconnect. As to your specific error with Aspera, I have no idea since I don't work for them.
If you happen to have a recent download of GenBank still compressed and untouched please add it to BioTorrents. Thanks for this post, I have been having similar issues with getting data out of dbGAP.
Unfortunately your post did not help address what I needed but it was informative and I did learn quite a bit about aspera which hopefully will help me in the future.
More than anything, it was good to see that I am not the only person dealing with similar issues. It is horrible that NCBI uses aspera. My tax money is wasted totally. Staying in status 'connecting' forever, no explaining for any error. Learn something from wget please!!! I just gave ascp a spin and it worked on first try. At least 20 times faster than wget, wow! Worked great on first try. At least 20 times faster than wget. This link contains vital information that will assist you to resolve all outstanding issues with ASCP and Aspera line of products.
It is now quite a bit after this original thread, but dealing with Aspera is still difficult. I tried convincing both the Broad and my client that they'd be better off buying 10 3TB disks and re-using them for such transfer but no go.. Inspired by reading this blog Thanks! It appears to be doing something like an rsync, which may be the only way to get it to work, but that's a big chunk of disk activity..
For some reason the obvious increment '-C -O ' would not work - the ssh connection was refused. They all appear to be working and all commands claim to be working on the same file 'MH So, parts of it work. It has the above mentioned parallel option that I've been trying for about a week and the main problem is that as others have noted it keeps dropping the connection.
Not only does it drop the connection, but the ascp process keeps running so a monitor script cannot tell that the process has dropped altho I recently discovered that if you add add a few [Enter]s after the one for the password, it will drop out of the process so a monitor script can now tell that one process has stopped.
The transfer speed seems to be about 10MBytes per connection. Not bad, but not much better than my experience with bbcp , which is both free, quite reliable, and whose author is quite responsive to suggestions. If it keeps working and it often doesn't , it will transfer data, but I don't see much improvement over transfer rates that bbcp or better, gridFTP and especially its consumer interface Globus Online which can move data extraordinarily fast and with less fuss.
Why the Broad Institute and especially the NIH would use a commercial 'solution' that seems to still be in an alpha stage when there are free solutions that demonstrably work better, I'm not sure.
I have incorporated a short segment about Aspera's ascp into my "How to transfer large amounts of data via network. A good alternative to Aspera is a tiny open source program called xc.
NCBI says aspera is free but in aspera site it is asking for subscription. Can you please direct me where I can download aspera ascp for free? Many thanks. Check out SuperTCP supertcp. They sound like Aspera, but can operate directly on TCP, so you don't have to totally change your workflow to get faster transfers!
Hi Morgan I am a new user of Aspera, and thanks for your blog. But I have copied "ascp-license" file from. If you have some suggestion, please let me know.
Thanks again. It's and Aspera still sucks. No globbing. No sensible logs or error messages. Still difficult to get hold of. Switches still are overly complicated and not particularly powerful.
What a shame that transferring data is still a thing bioinformatics struggles with. Thank you.. This is very helpful. Tableau Online Training. I am a regular reader of your blog and I find it really informative. Hope more Articles From You. Best Tableau tutorial videos available Here. You re in point of fact a just right webmaster. The website loading speed is amazing.
It kind of feels that you're doing any distinctive trick. Moreover, The contents are masterpiece. Hi, Thanks for sharing nice stuff Data Science Training in Hyderabad. The development of artificial intelligence AI has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning.
Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point.
First Choice — Aspera Connect. It is a commercial high speed file transfer software produced by IBM. Many sites can transfer data at Mbps. At last, please try fastq-dump and sam-dump in sratoolkit. If the connection of fastq-dump is unstable, I would suggest the wonderdump script in Biostar Handbook. Warning : Try not to use wget or curl to download, it might cause incompletion in downloaded sra files.
Firstly, go to Aspera Connect , choose the linux version and copy link address. The default rate is rather low, you would better declare it explicitly.
0コメント