Differences

This shows you the differences between two versions of the page.

--- data:retrieval [2024/04/09 10:49] – removed - external edit (Unknown date) 127.0.0.1
+++ data:retrieval [2025/08/20 10:04] (current) – Ania Piskorz
@@ Line 1: / Line 1: @@
+====== Retrieving Your Sequencing Data ======
+There are two methods we employ to allow users of the sequencing service to retrieve their sequencing data. Persons working inside CRUK-CI should use our data download tool; everyone else must fetch their data from our FTP site.
+==== CRUK-CI Researchers ====
+We provide a tool for downloading files for projects, libraries and runs that you can use from the command line. The full user manual for the download tool can be found [[https://internal-bioinformatics.cruk.cam.ac.uk/docs/clarity/downloadtool/userguide.html|on this internal web page]]. Please visit the page to download the tool, find instructions for how to use it and also how to install Java on your personal machine (link requires you to be in the building or running the VPN).
+==== External Sequencing Service Users ====
+Outside of CRUK-CI, data is delivered through the FTP site: ''ftp1.cruk.cam.ac.uk''.
+This is a vanilla FTP site that should be accessible by any FTP client you choose. Your group will have been provided a user name and password for accessing the site when your group arranged to use the CRUK-CI sequencing service. Your data will be in a private region of the server only accessible with your group's credentials. The site is read only.
+**Files are available on the FTP site for a guaranteed thirty (30) days after sequencing. You MUST fetch the files in this time period. The files will be removed from the FTP site once this time has elapsed.**
+It is your responsibility thereafter to look after the files and store them as you see fit. We do not hold external groups' data in the CRUK-CI archives.
+It is possible that the files do not transfer correctly from the FTP site on the first go. We've never really known of corruption during a transfer but truncated files do happen, where the connection is dropped before the transfer is complete. **External users of the service must check the transfer is accurate after taking a copy.** We provide checksums for the FASTQ files, and one can make sure that the local copy of the files match the original by running the md5sum command once the download has finished.
+The checksum files have the suffix "''.md5sums.txt''" and can be used to check the FASTQ by running:
+<code>
+md5sum -c *.md5sums.txt
+</code>
+Any files that are not reported as "OK" should be downloaded again and rechecked.
+=== FTP Clients ===
+ There are many FTP clients available on the web one can use to fetch files from our FTP site. Some clients are:
+  * [[https://filezilla-project.org|Filezilla]] (desktop)
+  * [[https://cyberduck.io|Cyberduck]] (desktop)
+  * [[https://www.coffeecup.com/free-ftp|CoffeeCup Free FTP]] (desktop)
+  * [[https://lftp.yar.ru|LFTP]] (command line)
+  * [[https://www.ncftp.com/ncftp|NcFTP]] (command line)
+On Linux, you might find that these programs are available through the platform's package management system. Most web browsers will also allow you to navigate the FTP site if you use the FTP protocol in the address bar: [[ftp://ftp1.cruk.cam.ac.uk/]]. The browser will prompt for your user name and password. The web browser is handy for having a look at your area of the server and previewing the reports but a proper FTP program is recommended for fetching the files.
+We have become aware of some users using the Mac's //Finder// application to connect to the FTP server and copy the files. While convenient, it appears that //Finder// can silently truncate files while copying if the connection to the FTP server drops. Thus we do not recommend using //Finder// or //Windows Explorer// to copy the files: use a proper FTP program that will report errors. Above all, and regardless of the program used, **you must check your files against the checksums after downloading** as described above.
+=== Troubleshooting ===
+Our FTP server is pretty much as vanilla as they come and should not cause any problems. Nonetheless, occasionally people do tell us they cannot connect, and so far this has always been problems at the client end. Here are some things to check.
+  - Make sure any encryption option is turned off. Some clients have encryption, sometimes called SSL or FTPS, turned on by default.
+  - Make sure your computer can see our server. Try "pinging" the FTP server with "''ping ftp1.cruk.cam.ac.uk''" from the command line. The server will echo back a reply if your pings are getting through. If ping says the packets are not being returned, please check with **your** IT department to check network connectivity. The problems have never yet been at the CRUK-CI end; if our FTP site does need to be taken out of commission for a while we will let all our collaborators know beforehand.