Uploading data into COSMIC²

Please note that there are two ways to upload data:

  1. Single file uploads (files <200 MB)
  2. Dataset uploads (image datasets for analysis)

You’ll that there are both tools for single-file analysis (e.g. cryoSPARC to RELION conversion; validation of reconstructions using cryoEF) as well as single-particle analysis (e.g. submitting RELION jobs).

Dataset uploads using Globus file transfers

Globus is a big data transfer service that is a global leader in data movement and distributed user networks. We are using Globus for 1) user authentication and 2) big data transfer for ingesting cryo-EM data into COSMIC². Globus allows users to upload terabytes (or more) through a web-interface, a feat that would be impossible using normal file uploads. As such, Globus is the perfect fit for moving cryo-EM data into our gateway.

Globus FAQs

Installing Globus on your local machine

A big choice to make for installing Globus is whether to have an IT admin install Globus Connect Server onto your data storage machines, or if you want to just install Globus Connect Personal, software that does not require IT support.

Globus Connect Server

Installed by an IT admin, this will ensure that your Globus ‘endpoint’ on your data storage drive is always open and available. It will also allow users to authenticate via Globus.org.

Globus Connect Personal

Easy to install on an individual user basis – can be installed within 5 minutes. The downside is that if you close the program, the endpoint is turned off. Link for learning more about Globus Connect Personal (video on bottom left).

Uploading datasets with Globus into COSMIC²

Outline of process for uploading data into COSMIC² using Globus:

  1. Add Globus endpoint to COSMIC2
  2. Navigate to directory containing your data
  3. Select directory for transfer
  4. Wait for transfer to complete – email will notify successful transfer

Video demo for login & data upload:

1.  Select ‘Data’ tab within Folder on COSMIC²

2. Select Globus for uploading all input particles (also fast for uploading 3D models)

3. Add Globus endpoint to COSMIC²

4. Select endpoint from list of endpoints matching search text

5. Enter path to location of data to be uploaded

Note: This should be the in directory above the directory that you want to upload. This means if you want to upload directory ‘Particles’ and it is in the directory: ‘/home/test/Data/Particles’, then you would enter ‘/home/test/Data’ into the search bar  

 

6. Select directory to be uploaded into COSMIC²

7. Monitor upload status and wait for completion

Under ‘Transfer status’ page, you will see full list of transfers

Once finished, status will be updated to ‘Succeeded’: 

AND, Globus will email you to notify that your transfer completed: 

8. You will now see the file / directory displayed in the ‘Data’ tab

Need help?

Please email cosmic2support@umich.edu.