Attention

This documentation is under active development, meaning that it can change over time as we refine it. Please email help@massive.org.au if you require assistance.

Python and conda on M3: Frequently Asked Questions (FAQ)#

If you are unable to resolve your problem independently or don’t see your question answered here, please contact the helpdesk at help@massive.org.au so we can investigate further.

My $HOME directory is full of conda packages and my desktop won’t start.#

You can see how much of your quota has been used by running

user_info
Home directory usage
+----------------+---------+---------+----------+
| Path           |   Usage |   Quota |   % Used |
|----------------+---------+---------+----------|
| /home/user     |    6 GB |   10 GB |     60 % |
+----------------+---------+---------+----------+

If your home directory is full, you may experience unexpected behaviour including desktops failing to start. Sometimes this happens as a result of conda packages being installed in a .conda directory in your $HOME folder by default.

If you have already installed conda, you can remedy this issue with the following steps.

  1. Export your conda environment to a yaml file. You will need to do this for each environment you have created and intend to keep. To do this:

# Activate your conda environment as usual
# Insert environment activation here

# Once your environment is activated, run:
conda env export > environment.yml

  1. Remove your current install of conda, including the ~/.conda directory.

  2. Reinstall conda following the Anaconda or Miniconda install instructions under “create the conda environment with packages from a .yml file”.

Note: In our Anaconda instructions we include two commands which ensure you packages are not installed in your $HOME folder. These lines are also included within the Miniconda install script that we provide per the instructions. Specifically, these are the lines to pay attention to:

# These commands prevent your environments and packages being stored in $HOME
conda config --add envs_dirs $CONDA_ENVS/conda/envs
conda config --add pkgs_dirs $CONDA_ENVS/conda/pkgs

If you need any assistance with this process, please contact the helpdesk at help@massive.org.au

How do I share Python environments?#

We don’t recommend sharing Python environments on M3 for a few reasons:

  1. Sharing your environment with someone else means it may be altered in ways that impact your work, preventing reproducible results and hindering your research.

  2. Setting file permissions for shared Python environments isn’t straightforward.

  3. This can cause unforeseen issues if one person has been maintaining the environment for the group and becomes unavailable.

Instead, if you want to replicate a conda or Python virtual environment your colleague is using, follow these instructions.

Miniconda and Anaconda (conda)#

  1. Ask your colleague to export their conda environment to a yaml file. They can do this by:

# Activate your conda environment as usual
# Insert environment activation here

# Once your environment is activated, run:
conda env export > environment.yml

Then get a copy of this file from them, for example by placing it in a shared space on M3 like /scratch or /projects.

  1. Install your own copy of Miniconda, or Anaconda depending on your preference, following the instructions under “create the conda environment with packages from a .yml file”.

    This should create an identical environment to the one your colleague has.

Virtual Environment (venv)#

  1. Ask your colleague to activate their virtual environment as usual, and then export their environment to a requirements.txt file. You can do this by following:

# Activate the virtual environment as usual
# Change myPythonEnv to the environment name
source myPythonEnv/bin/activate

# Copy the packages installed to a requirements.txt
pip freeze > requirements.txt

Then get a copy of this file from them, for example by placing it in a shared space on M3 like /scratch or /projects.

  1. Create and activate your own virtual environment.

  2. Once your virtual environment is activated, run the following command:

# The following command will copy the packages from requirements.txt into your environment
pip install -r requirements.txt

This should create an identical environment to the one your colleague has.

I ran conda init and my desktop session no longer starts#

To resolve this problem, follow these instructions.

  1. ssh into M3.

  2. Edit your .bashrc file using your favourite editor.

  3. Comment out the ‘conda’ lines using a # at the start of the lines. Below is a sample .bashrc with the comments

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
 . /etc/bashrc
fi

# User specific aliases and functions
# Give bash history a decent size
export HISTFILESIZE=10000
# Helpful alias for users to quickly see their queue status
alias squ='squeue -u $USER'

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
#__conda_setup="$('/usr/local/anaconda/2019.03-Python3.7-gcc5/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
#if [ $? -eq 0 ]; then
#    eval "$__conda_setup"
#else
#    if [ -f "/usr/local/anaconda/2019.03-Python3.7-gcc5/etc/profile.d/conda.sh" ]; then
#        . "/usr/local/anaconda/2019.03-Python3.7-gcc5/etc/profile.d/conda.sh"
#    else
#        export PATH="/usr/local/anaconda/2019.03-Python3.7-gcc5/bin:$PATH"
#    fi
#fi
#unset __conda_setup
# <<< conda initialize <<<

  1. Save the file and exit your ssh session.

  2. Restart your Desktop session.

Note

If you are not comfortable updating your .bashrc file, please see Help and Support to obtain assistance.