Attention

This documentation is under active development, meaning that it can change over time as we refine it. Please email help@massive.org.au if you require assistance, or have suggestions to improve this documentation.

Running Anaconda on M3#

M3 has different versions of Anaconda (conda) installed as modules. To list them, use the command module avail anaconda:

module avail anaconda

---------------------------- /usr/local/Modules/modulefiles -----------------------------
anaconda/2018.12-Python3.7-gcc6        anaconda/5.0.1-Python2.7-gcc5
anaconda/2019.03-Python3.7-gcc5        anaconda/5.0.1-Python3.5-gcc5
anaconda/2020.07-Python3.8-gcc8        anaconda/5.0.1-Python3.6-gcc5
anaconda/4.3.1-Python3.5               anaconda/5.1.0-Python3.6-gcc5(default)
anaconda/4.3.1-Python3.5-gcc5

The listed Anaconda versions have a standard set of modules installed. As a user, unfortunately you are unable to update or install your own modules into the supplied Anaconda software on M3.

There are two main alternatives:

  • Miniconda, our recommended method, which you can install by following the instructions here.

  • Anaconda environments, which you can install by following the instructions below.

Instructions for Anaconda on M3#

A conda environment effectively creates your own copy of conda by cloning the Anaconda module on M3. This will allow you to install and update any conda and Python modules you require. The instructions below focus on creating an Anaconda environment using the module anaconda/2019.03-Python3.7-gcc5 which will be stored in your project /scratch space.

Note

On M3, DO NOT run conda init. After completing the following steps, you may be advised to run conda activate and conda init to activate your environment. Running this command will update your .bashrc file. This will prevent your Desktop session from starting. Please follow the “Activating your conda environment on M3” instructions.

Install Anaconda and create your environments#

To install Anaconda, you will load the desired Anaconda module and clone your own copy.

myPythonEnv represents your environment name.

# Load the anaconda module
module load anaconda/2019.03-Python3.7-gcc5

# Change myID to your project ID, i.e. nq46
export PROJECT=myID
export CONDA_ENVS=/scratch/$PROJECT/$USER/conda_envs

# Change the location where conda stores environments and packages
# This prevents your $HOME directory filling up.
conda config --add envs_dirs $CONDA_ENVS/conda/envs
conda config --add pkgs_dirs $CONDA_ENVS/conda/pkgs

There are three installation options.

  1. Create the conda environment, cloning existing packages;

  2. Create the conda environment with no packages installed, not even Python;

  3. Create the conda environment with packages from a .yml file.

All three are covered below.

# Option 1. Create the conda environment - cloning the already installed packages
# Change myPythonEnv to your environment name
conda create --yes -p $CONDA_ENVS/myPythonEnv --clone base

# Option 2. Create the conda environment with no packages installed, not even Python
# Change myPythonEnv to your environment name
conda create --yes -p $CONDA_ENVS/myPythonEnv

# Option 3. Create the conda environment with packages from a .yml file
# Change myPythonEnv to your environment name
# Change environment.yml to the name of your yml file
conda env create -p $CONDA_ENVS/myPythonEnv -f environment.yml

Conda will then create the environment. To activate the environment, continue to the next section on this page. Do not follow the instructions conda prints out, which will look like:

Executing transaction: done
#
# To activate this environment, use:
# > conda activate myPythonEnv
#
# To deactivate an active environment, use:
# > conda deactivate
#

If you run conda activate on M3, you will receive a message like this advising you to use conda-init which you should NOT follow.

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Running conda init on M3 will update your .bashrc file. This will prevent your Desktop session from starting. Please follow the instructions provided in “Activating your conda environment on M3” instead.

Activating your conda environment on M3#

This will look different to what you are used to on your local workstation, using source activate instead of conda activate. You need to source activate every time you start a new shell in order to use your conda environment or install packages.

Assuming you are in a new shell with no modules loaded:

# Load the anaconda module
module load anaconda/2019.03-Python3.7-gcc5

# Change this to your project ID, i.e nq46
export PROJECT=myID
export CONDA_ENVS=/scratch/$PROJECT/$USER/conda_envs

# To use the environment
source activate $CONDA_ENVS/myPythonEnv

You will notice that when conda is activated, (/scratch/$PROJECT/$USER/conda_envs/myPythonEnv) will appear in the command prompt. For example:

(/scratch/nq46/user/conda_envs/myPythonEnv) [user@m3-login]$

Installing and updating Python packages in your conda environment#

Check that your conda environment has been properly activated.

conda env list

# conda environments:

                         *  /scratch/PROJECT_ID/USER_ID/myPythonEnv
base                     /usr/local/anaconda/2019.03-Python3.7-gcc5
illumina-utils-2.6       /usr/local/anaconda/2019.03-Python3.7-gcc5/envs/illumina-utils-2.6
pytorch-1.0-cuda10       /usr/local/anaconda/2019.03-Python3.7-gcc5/envs/pytorch-1.0-cuda10
pytorch-1.0-cuda9        /usr/local/anaconda/2019.03-Python3.7-gcc5/envs/pytorch-1.0-cuda9

The ‘*’ indicates the active environment. Ensure the correct one is indicated. If not, refer above to correctly activate your environment.

Use the following sample commands to install or update packages.

# To install packages to your conda environment
conda install packageName
# To install a specific version of a package to your conda environment
conda install packageName=versionNumber

# To update a package in your conda environment to the newest version
conda update packageName
# To update a package in your conda environment to a specific version
conda update packageName=versionNumber

You can find other conda commands in this conda cheat sheet.

Deactivating your conda environment#

To stop using Anaconda, you will need to deactivate the environment you’re using by running the following:

$ conda deactivate

You will notice the command prompt stops displaying the path to your environment (/scratch/$PROJECT/$USER/conda_envs/myPythonEnv) indicating that conda is not activated.

You may also receive the following warning, which you can ignore:

DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.

Troubleshooting#

I ran conda init and my Desktop session no longer starts#

Resolve this by following steps in our Frequently Asked Questions.

You can find other conda commands in this conda cheat sheet.

If you encounter any problems when installing or using Miniconda, refer to our FAQs as you may be able to resolve them independently. If you are unable to resolve your problem or wish to give feedback on these instructions, please contact us at help@massive.org.au.