Quest and Kellogg Linux Cluster Downtime, December 14 - 18.
Quest, including the Quest Analytics Nodes, the Genomics Compute Cluster (GCC), the Kellogg Linux Cluster (KLC), and Quest OnDemand, will be unavailable for scheduled maintenance starting at 8 A.M. on Saturday, December 14, and ending approximately at 5 P.M. on Wednesday, December 18. During the maintenance window, you will not be able to login to Quest, Quest Analytics Nodes, the GCC, KLC, or Quest OnDemand submit new jobs, run jobs, or access files stored on Quest in any way including Globus. For details on this maintenance, please see the Status of University IT Services page.
Quest RHEL8 Pilot Environment - November 18.
Starting November 18, all Quest users are invited to test and run their workflows in a RHEL8 pilot environment to prepare for Quest moving completely to RHEL8 in March 2025. We invite researchers to provide us with feedback during the pilot by contacting the Research Computing and Data Services team at quest-help@northwestern.edu. The pilot environment will consist of 24 H100 GPU nodes and seventy-two CPU nodes, and it will expand with additional nodes through March 2025. Details on how to access this pilot environment will be published in a KB article on November 18.
This page provides information on transferring files to and from Quest.
Quest supports several secure file transfer protocols and systems for transferring files to and from Quest. We strongly recommend using the Globus client if you want to transfer files to or from a Windows, Linux, or Mac computer.
Contact Northwestern data management specialists by emailing globus-help@northwestern.edu for help with using Globus. For further information, learn more about what Globus is and about how to transfer files using Globus.
Transferring Data To and From Quest and ....
Globus
Personal Laptop or Workstation
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to your own Personal Globus Endpoint.
Research Data Storage Service or FSMRESFILES
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern RDSS/FSMRES Endpoint.
OneDrive/SharePoint
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern OneDrive/Sharepoint Endpoint.
Amazon S3
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern AWS Endpoint.
Google Cloud:
Solution: The Google Cloud SDK is installed system-wide on Quest. To load this package, run:
module load gcloud/322.0.0
Following the instructions on this page, Google Cloud SDK Quickstart Linux, you can configure the Google Cloud SDK with your credentials, etc.
Other Data Transfer Options
Web Browser
For example, an option for transferring Quest storage and OneDrive would be to connect to Quest with the FastX client and start a Gnome desktop session or a Gnome terminal session. Then launch a terminal (or use the terminal that is already launched, if you pick a Gnome terminal session) and type
firefox
to launch the firefox browser. In that browser you can log into OneDrive and transfer files.
Documentation on how to use FastX to connect to Quest is available here: Connecting to Quest with FastX.
We recommend that for large data transfers or repeated transfers to use the Globus OneDrive connector when it becomes available.
Secure Copy (SCP)
You can use Secure Copy to transfer individual files to or from Quest using the Linux or Mac OSX Terminal application on the computer you want to transfer files to/from. If you are using a Windows system, you will need PuTTY; Windows users substitute the command pscp for scp in the examples below.
These commands are executed from your computer, NOT from Quest (your Terminal prompt should reference your computer, not Quest).
To transfer files from your computer to Quest:
scp /PathToSourceFile/file NetID@quest.it.northwestern.edu:/PathToTargetDir/file
Example:
scp /Users/willie/myfile.txt abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt
To transfer files from Quest to your computer:
scp NetID@quest.it.northwestern.edu:/PathToSourceDir/file /PathToTargerDir/file
Example:
scp abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt .
This would copy the file /home/abc123/myfile.txt from Quest to the current working directory on your computer with the same name (myfile.txt).
For more examples of Secure Copy, see In Unix, how do I use SCP to securely transfer files between two computers? Note that scp will overwrite existing files without prompting.
wget
You can download datasets directly to Quest from external ftp and web servers using the wget command (see the Wget manual).
While connected to Quest, you could use commands like:
wget http://www.nber.org/nberces/bbg96_87.dta
wget ftp://ftp.bls.gov/pub/doc/ap.txt
To copy files into your current working directory on Quest.
With additional options wget can also download whole directories of files recursively and download from password-protected servers.