Body
Quest and Kellogg Linux Cluster Downtime, December 14 - 18.Quest, including the Quest Analytics Nodes, the Genomics Compute Cluster (GCC), the Kellogg Linux Cluster (KLC), and Quest OnDemand, will be unavailable for scheduled maintenance starting at 8 A.M. on Saturday, December 14, and ending approximately at 5 P.M. on Wednesday, December 18. During the maintenance window, you will not be able to login to Quest, Quest Analytics Nodes, the GCC, KLC, or Quest OnDemand submit new jobs, run jobs, or access files stored on Quest in any way including Globus. For details on this maintenance, please see the Status of University IT Services page.
Quest RHEL8 Pilot Environment - November 18.Starting November 18, all Quest users are invited to test and run their workflows in a RHEL8 pilot environment to prepare for Quest moving completely to RHEL8 in March 2025. We invite researchers to provide us with feedback during the pilot by contacting the Research Computing and Data Services team at quest-help@northwestern.edu. The pilot environment will consist of 24 H100 GPU nodes and seventy-two CPU nodes, and it will expand with additional nodes through March 2025. Details on how to access this pilot environment will be published in a KB article on November 18.
Overview
The Quest Analytics nodes allow users with active Quest allocations to launch Jupyter notebooks from a web browser. See Research Computing: Quest Analytics Nodes for an overview of the system.
Note: you cannot submit jobs to the Quest job scheduling system using the Quest Analytics nodes.
Connecting
Users must already have a Quest account and an active Quest allocation.
Connections to the Quest Analytics Nodes are limited to the Northwestern network. If you are connecting from off-campus, you must use the Northwestern VPN.
Using any browser, connect to: https://jupyter.questanalytics.northwestern.edu and sign in with your NetID and password.
Note: notebook servers are automatically killed after 18 hours of inactivity.
Accessing JupyterLab Instead of Notebook
The default behavior for JupyterHub will be to launch a Jupyter Notebook instance. The URL for the notebook will end as follows:
/home/<netid>/tree?
To instead run JupyterLab after logging into JupyterHub, you can change the URL to
/home/<netid>/lab
which will trigger the creation of a JupyterLab instance.
Transferring and Accessing Files
All of the Quest filesystem is accessible from the Quest Analytics nodes. You can transfer files using the methods to transfer files to Quest more generally.
You can also transfer files smaller than 500MB using the the Jupyter web interface. There is an Upload button that will allow you to select files from your computer to transfer.
When you connect to Jupyter, you will initially be in your home directory. This is /home/<netid> on Quest. You cannot navigate to a directory "outside" of your home directory (e.g., to a project directory) via the Jupyter interface. However, you can SSH to Quest and use the terminal to create a symlink to a project directory within your home directory, allowing you to navigate there. For example:
ln -s /projects/<projectID> ~/<projectID>
This will create a symlink <projectID> in your home directory, and when you click on it in the Jupyter interface you will be taken to the project directory itself.
To transfer files larger than 500MB to Quest, please use SFTP or another transfer method.
Environments and Packages
The default kernel/environment contains the following packages: Analytics-Node-Default-Enviroment.yml.
If you need other packages (or versions of packages) installed, you can create a conda environment and add it to your list of available environments using the commands below (Note you must SSH or use FastX to access Quest and type these commands in the terminal):
Create a iPython Kernel
module purge all
module load python-miniconda3/4.12.0
conda create --name jupyter-kernel-py38 -c conda-forge python=3.8 numpy pandas matplotlib ipykernel --yes
source activate jupyter-kernel-py38
python -m ipykernel install --user --name jupyter-kernel-py38 --display-name "Python (jupyter-kernel-py38)"
The last command adds the environment to your list of available environments in Jupyter, and once you have done this it will be available for you under the "New" menu.
Create a R Kernel
module purge all
module load python-miniconda3/4.12.0
conda create -c conda-forge --name r-kernel r-irkernel r=4.1 jupyterlab
source activate r-kernel
R -e "IRkernel::installspec(name = 'ir41', displayname = 'R4.1')"
The last command adds the environment to your list of available environments in Jupyter, and once you have done this it will be available for you under the "New" menu.
In addition, to list all of the available kernels that you have installed, you can run
module load anaconda3/2018.12
jupyter kernelspec list
For more information about managing environments with conda, see Using Python on Quest.
System Details
The Quest Analytics nodes are shared by many researchers. Please be aware of your memory use when analyzing large data sets. Users utilizing a large amount of memory, especially those using over 60GB of RAM, may be asked to move their analysis to other systems. Multicore and parallel processes should not be run on the Analytics nodes. Users needing to run computationally intensive jobs should schedule interactive or batch jobs on Quest instead of using the Analytics nodes. Please contact Research Computing at quest-help@northwestern.edu with questions about memory use or analyzing large data sets.
Issues or Problems with the Quest Analytics nodes
To report issues or problems with the Quest Analytics nodes, please email quest-help@northwestern.edu with information on which service you were using (Jupyter), the error you received or problem you encountered, and what you were doing prior to the problem occurring. Please note that you were using the Quest Analytics nodes.