Attention

This documentation is under active development, meaning that it can change over time as we refine it. Please email help@massive.org.au if you require assistance, or have suggestions to improve this documentation.

Running Python and conda on M3

There are a few differences between installing Python on your local workstation and M3. The HPC is a shared system, and as an end user, you are unable to install software directly or have access to sudo (admin) privileges. This is so you don’t impact other users utilising the same software.

This page provides general guidance on the different methods to access Python on M3, including links to more detailed instructions and an FAQ.

Note

You should NOT use the default system Python. It is subject to change and not suitable for reproducible research. Please use one of the methods listed below.

Miniconda (conda)

Miniconda allows you to manage multiple Python environments with conda. Detailed instructions to install Miniconda on M3 can be found at Running Miniconda on M3.

Miniconda is our preferred Python install method on M3 because:

  • We provide a user-friendly script to install your own copy of Miniconda on M3, minimising installation difficulty.

  • Installing Miniconda using our script automatically creates an environment called jupyterlab which can be used with the JupyterLab option in Strudel2.

  • Miniconda installs a reduced set of initial packages, ensuring you only install packages you intend to use.

Using our Miniconda install script also ensures that if you need any assistance, our team has more clarity on how best to help.

You may consider using Anaconda instead if you want more packages upon installation.

You may consider using a virtual environment if you don’t need an environment manager and only intend to use Python.

Anaconda (conda)

Anaconda allows you to manage multiple Python environments with conda. Detailed instructions to install Anaconda are found at Running Anaconda on M3.

Anaconda installs more initial packages than Miniconda, and we do not provide an installation script for Anaconda.

You may consider using a virtual environment if you don’t need an environment manager and only intend to use Python.

Virtual Environments (venv)

M3 offers static Python builds which can be used to create a virtual environment (venv). A Python virtual environment effectively creates your own copy of Python, allowing you to install any Python packages you require. You are able to have multiple virtual environments, each with different packages installed; pip will manage the packages. Detailed instructions to set up a virtual environment are found at Running Python Virtual Environments on M3.

The difference between a virtual environment and a conda install is that conda will manage your packages and environments, and allows you to use languages other than Python. You may find virtual environments offer everything you need, or prefer conda for its additional features.

Python Modules

M3 has a number of different versions of Python installed as modules. Each module will contain a standard set of Python packages installed using pip. As a user, you are unable to update or install your own Python packages into the supplied Python modules on MASSIVE. If each and every user was able to pip update the Python modules that they use, repeatability and management of these modules would be impractical.

If you require additional Python packages, we recommend you consider a Miniconda, Anaconda, or virtual environment installation of Python which you can customise.

To see the versions of Python available as a module on M3, use the command module avail python:

# To see which python modules are available
module avail python

---------------------------------------------------------------------- /usr/local/Modules/modulefiles ----------------------------------------------------------------------
python/2.7.11-gcc           python/2.7.15-gcc5          python/2.7.18-ucs4-gcc5     python/3.5.2-gcc5           python/3.6.6-gcc5           python/3.8.5
python/2.7.12-gcc4(default) python/2.7.17-gcc8          python/3.5.2-gcc            python/3.6.2                python/3.7.2-gcc6           python/3.8.5-gcc8
python/2.7.12-gcc5          python/2.7.18-gcc5          python/3.5.2-gcc4           python/3.6.2-static         python/3.7.3-system         python/3.8.5-gcc8-static

# To load a python module
module load python/3.6.2-static

# To see which modules you already have loaded
module list

You will need to repeat the module load command each time you start a new shell. The default Python module is subject to change, so please specify which version of Python you want to use to ensure reproducibility.

Python and conda: Frequently Asked Questions (FAQ)

The Python and conda FAQ may assist you to independently solve Python related problems on M3. If you would like to add to this FAQ, or are unable to resolve your problem, please email us at help@massive.org.au.