Summary
Are you familiar with the base level Linux commands access Great lakes, your storage location and need assistance checking or installing software? You've come to the right place. This is Phase 2.b/3 of the self-guided practice. Within this Phase there are 3 sections in total and you can find them within the Index for the HPC self guided practice.
As always, if you have questions or concerns while going through the learning material, please don't hesitate to stop by Office Hours (Monday's 2-3pm and Thursday's 3-4pm), our HPC related workshops, which will be linked below, or submit a ticket to arc-support@umich.edu.
"Go forth and compute!" - Dr. Charles Antonelli
Environment
High Performance Computing (HPC) on the Great Lakes cluster (non-sensitive data)
Directions
Please do not only copy and paste the commands here. Any <value> placed within a greater and lesser than symbol will need to be replaced and the symbols removed. For example, if your uniqname is ‘umstudent’ then if you see ‘/home/<uniqname>’ in the instructions, write ‘/home/umstudent’.
How to load or install software on the cluster:
If you need assistance installing software within your home environment or have any questions and concerns, please contact us at arc-support@umich.edu.
- Lmod: is a Lua based module system that easily handles the MODULEPATH hierarchical problem. Allowing users to add or remove directories to the PATH environment variable.
- Basic commands:
- module keyword <string>
- Searches for <string> in module descriptions
- module spider <name>
- Searches for <name> in module list
- module load <name>
- Loads module <name> for use
- module list OR ml
- module unload <name>
- module purge
- Commands for Module Sets:
- module save
- Creates a module set to be loaded each time you log in
- module save <myset>
- Saves a group of modules named <myset>
- module savelist
- List all module sets you've defined
- module restore <myset>
- Restores the modules from <myset>
- UM ARC Lmod Documentation
- Conda: Is an open source package management system and environment system for installing multiple versions of software packages and their dependencies while being able to easily switch between them.
- You may use conda with the module load command to load the Anaconda python distribution or by installing conda separately into your home directory
- Please note: When loading the Anaconda module, the best practice here is to create separate environments for each type of session you want to have. When you create a conda environment, then only load the packages you need. Installing packages with pip increases the risk of package conflicts with Anadaconda because of it loading packages into ~.local instead of ~ .conda/envs.
- Load Anaconda, which includes conda, into your environment with Lmod:
- module load python3.11-anaconda/2024.02
- Conda Cheat Sheet
- Please note: If this is your first time running conda you’ll need to initialize the environment by typing:
conda init bash
Then log out and back into the cluster. Then run:
conda config –set auto_activate_base false
conda deactivate
Once deactivated after this change, you can easily create new environments or use the ARC modules.
- Basic commands:
- conda create -n <environment_name> python=<#.#>
- Creates a conda environment with a given name and if needed specifies a python version
e.g python=3.7
- conda activate <environment_name>
- Activates newly created environment
- conda install <package> OR pip3 install <package>
- You can use Conda or Pip to install specific packages within your environment, but as a rule of thumb conda should be used first
- After the package name you can add an operator to specify an exact version “==”, or less than “<=”, or greater than “>=”
- conda deactivate
- Exit activated environment
- To activate an environment in the batch script after the Slurm preamble:
module load python3.<version of conda you are using>
source ~/.bashrc
conda activate <environment_name>
- Initialization of a conda environment with Open OnDemand interactive sessions may vary per application.
- Mamba: is a fast, robust, and cross-platform package manager. It runs on Windows, OS X, and Linux (ARM64 and PPC64LE included) and is fully compatible with conda packages and supports most of conda's commands.
- You may install mamba locally to your home directory or by using the module load command.
- Load mamba into your environment with Lmod:
- Please note: If this is your first time running mamba you'll need to initialize the environment by typing:
mamba init bash
Then log out and back into the cluster. Then run:
mamba deactivate
Once deactivated after this change, you can easily create new environments or use the ARC modules.
- Basic commands:
- mamba create -n <environment_name> python=<#.#>
- Creates a mamba environment with a given name and if needed specifies a python version
e.g python=3.7
- mamba activate <environment_name>
- Activates newly created environment
- mamba install <packageName>
- Install specific packages within your environment
- After the package name you can add an operator to specify an exact version “==”, or less than “<=”, or greater than “>=”
- mamba deactivate
- Exit activated environment
- To activate an environment in the batch script after the Slurm preamble:
module load mamba/<version you are using>
source ~/.bashrc
mamba activate <environment_name>
- Initialization of a mamba environment with Open OnDemand interactive sessions may vary per application.
- Python and Pip: pip is a package management system written in Python that allows users to install and manage software packages.
- Pip can be used within Python, Conda and Mamba environments.
- Great Lakes is a shared cluster, meaning that depending on the version of pip. Installs will need to specify the user with the –user option.
- pip install --user <nameOFpackage>
- Versions of pip 23.1 and above can automatically fall back to a user install if it detects that it doesn't have the necessary write permissions for a system-wide install.
External resources
As this document evolves. Will be adding links here.