Transferring Files to and from Quest

Quest and Kellogg Linux Cluster Downtime, June 8th-14th, 2024.

Quest, including the Quest Analytics Nodes, the Genomics Compute Cluster (GCC), the Kellogg Linux Cluster (KLC), and Quest OnDemand, will be unavailable for scheduled maintenance starting at 7 A.M. on Saturday, June 8, and ending approximately at 5 P.M. on Friday, June 14. During the maintenance window, you will not be able to login to Quest, Quest Analytics Nodes, the GCC, KLC, or Quest OnDemand submit new jobs, run jobs, or access files stored on Quest in any way including Globus. For details on this maintenance, please see the Status of University IT Services page.

This page provides information on transferring files to and from Quest.

Quest supports several secure file transfer protocols and systems for transferring files to and from Quest. We strongly recommend using the Globus client if you want to transfer files to or from a Windows, Linux, or Mac computer.

Contact Northwestern data management specialists by emailing globus-help@northwestern.edu for help with using Globus. For further information, learn more about what Globus is and about how to transfer files using Globus.

Transferring Data To and From Quest and ....

Globus

Personal Laptop or Workstation

Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to your own Personal Globus Endpoint.

Research Data Storage Service or FSMRESFILES

Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern RDSS/FSMRES Endpoint.

OneDrive/SharePoint

Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern OneDrive/Sharepoint Endpoint.

Amazon S3

Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern AWS Endpoint.

Google Cloud:

Solution: The Google Cloud SDK is installed system-wide on Quest. To load this package, run:

module load gcloud/322.0.0

Following the instructions on this page, Google Cloud SDK Quickstart Linux, you can configure the Google Cloud SDK with your credentials, etc.

Other Data Transfer Options

Web Browser

For example, an option for transferring Quest storage and OneDrive would be to connect to Quest with the FastX client and start a Gnome desktop session or a Gnome terminal session. Then launch a terminal (or use the terminal that is already launched, if you pick a Gnome terminal session) and type

firefox

to launch the firefox browser. In that browser you can log into OneDrive and transfer files.

Documentation on how to use FastX to connect to Quest is available here: Connecting to Quest with FastX.

We recommend that for large data transfers or repeated transfers to use the Globus OneDrive connector when it becomes available.

Secure Copy (SCP)

You can use Secure Copy to transfer individual files to or from Quest using the Linux or Mac OSX Terminal application on the computer you want to transfer files to/from. If you are using a Windows system, you will need PuTTY; Windows users substitute the command pscp for scp in the examples below.

These commands are executed from your computer, NOT from Quest (your Terminal prompt should reference your computer, not Quest).

To transfer files from your computer to Quest:
scp /PathToSourceFile/file NetID@quest.it.northwestern.edu:/PathToTargetDir/file
Example:
scp /Users/willie/myfile.txt abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt

To transfer files from Quest to your computer:
scp NetID@quest.it.northwestern.edu:/PathToSourceDir/file /PathToTargerDir/file
Example:
scp abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt .
This would copy the file /home/abc123/myfile.txt from Quest to the current working directory on your computer with the same name (myfile.txt).

For more examples of Secure Copy, see In Unix, how do I use SCP to securely transfer files between two computers? Note that scp will overwrite existing files without prompting.

wget

You can download datasets directly to Quest from external ftp and web servers using the wget command (see the Wget manual).

While connected to Quest, you could use commands like:
wget http://www.nber.org/nberces/bbg96_87.dta
wget ftp://ftp.bls.gov/pub/doc/ap.txt
To copy files into your current working directory on Quest.

With additional options wget can also download whole directories of files recursively and download from password-protected servers.

 

Was this helpful?
33% helpful - 3 reviews
Print Article

Details

Article ID: 1535
Created
Thu 5/12/22 12:39 PM
Modified
Wed 5/1/24 9:18 AM