Quest RHEL8 Pilot Environment
The RHEL8 Pilot Environment is available for use now.
Starting Tuesday, December 10, seventy of the latest Intel Emerald Rapids CPU nodes (128 cores and 512GB of RAM) will be available to the RHEL8 Pilot, significantly increasing the compute capacity of this environment. With this expansion, the pilot environment will consist of twenty-four NVIDIA H100 GPU nodes (totaling ninety-six H100 cards) and 140 CPU nodes totaling 12,600 cores. Quest users are encouraged to test their workflows in RHEL8 Pilot environment to prepare for Quest moving completely to RHEL8 in March 2025. Detailed instructions are available on how to submit jobs for the new Operating System in the Knowledge Base article, RHEL8 Pilot Environment.
This page provides information on transferring files to and from Quest.
Quest supports several secure file transfer protocols and systems for transferring files to and from Quest. We strongly recommend using the Globus client if you want to transfer files to or from a Windows, Linux, or Mac computer.
Contact Northwestern data management specialists by emailing globus-help@northwestern.edu for help with using Globus. For further information, learn more about what Globus is and about how to transfer files using Globus.
Transferring Data To and From Quest and ....
Globus
Personal Laptop or Workstation
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to your own Personal Globus Endpoint.
Research Data Storage Service or FSMRESFILES
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern RDSS/FSMRES Endpoint.
OneDrive/SharePoint
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern OneDrive/Sharepoint Endpoint.
Amazon S3
Solution: Transfer data to and from the Northwestern Quest Globus Endpoint to the Northwestern AWS Endpoint.
Google Cloud:
Solution: The Google Cloud SDK is installed system-wide on Quest. To load this package, run:
module load gcloud/322.0.0
Following the instructions on this page, Google Cloud SDK Quickstart Linux, you can configure the Google Cloud SDK with your credentials, etc.
Other Data Transfer Options
Web Browser
For example, an option for transferring Quest storage and OneDrive would be to connect to Quest with the FastX client and start a Gnome desktop session or a Gnome terminal session. Then launch a terminal (or use the terminal that is already launched, if you pick a Gnome terminal session) and type
firefox
to launch the firefox browser. In that browser you can log into OneDrive and transfer files.
Documentation on how to use FastX to connect to Quest is available here: Connecting to Quest with FastX.
We recommend that for large data transfers or repeated transfers to use the Globus OneDrive connector when it becomes available.
Secure Copy (SCP)
You can use Secure Copy to transfer individual files to or from Quest using the Linux or Mac OSX Terminal application on the computer you want to transfer files to/from. If you are using a Windows system, you will need PuTTY; Windows users substitute the command pscp for scp in the examples below.
These commands are executed from your computer, NOT from Quest (your Terminal prompt should reference your computer, not Quest).
To transfer files from your computer to Quest:
scp /PathToSourceFile/file NetID@quest.it.northwestern.edu:/PathToTargetDir/file
Example:
scp /Users/willie/myfile.txt abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt
To transfer files from Quest to your computer:
scp NetID@quest.it.northwestern.edu:/PathToSourceDir/file /PathToTargerDir/file
Example:
scp abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt .
This would copy the file /home/abc123/myfile.txt from Quest to the current working directory on your computer with the same name (myfile.txt).
For more examples of Secure Copy, see In Unix, how do I use SCP to securely transfer files between two computers? Note that scp will overwrite existing files without prompting.
wget
You can download datasets directly to Quest from external ftp and web servers using the wget command (see the Wget manual).
While connected to Quest, you could use commands like:
wget http://www.nber.org/nberces/bbg96_87.dta
wget ftp://ftp.bls.gov/pub/doc/ap.txt
To copy files into your current working directory on Quest.
With additional options wget can also download whole directories of files recursively and download from password-protected servers.