Welcome to the M3 user guide

Attention

MASSIVE website issues

The main massive.org.au website is experiencing issues with an external hosting provider. We have temporarily redirected traffic to docs.massive.org.au while we are working on a long term fix.

Important

Hardware Refresh Plan 2021 – Update: 17 May 2021**

Please be advised of the following M3 hardware refresh schedule for 2021. These servers are now coming into end-of-life and will be retired this year.

While this will result in a reduction of total CPU capacity for 2021, retiring these servers is necessary to make room for new and faster compute nodes. The 2021 round of procurement for new hardware is expected to be provisioned during the middle of this year and this includes new remote desktop servers to replace those being decommissioned.

Compute Nodes	Capability	To be retired by the
m3a[000-021]	Intel Xeon-E5-2680-v3 CPU	end of May 2021
m3d[000-012]
m3f[000-031]	GRID K1 Desktop	end of August 2021
m3c[000-013]	Intel Xeon-E5-2680-v3 CPU NVIDIA K80 GPUs	middle of November 2021
m3h[001,015] m3h[003-008] m3h[010,011]	Intel Xeon-E5-2680-v4 CPU NVIDIA P100 GPUs	middle of November 2021

We will be enabling the appropriate mechanisms (e.g., SLURM reservation) to ensure that these nodes will be idle of running jobs prior to their retirement. Please check your job scripts to ensure they do not specify these nodes using --nodelist.

Important

Planned maintenance outages

We have scheduled quarterly outages for the M3 cluster. This is to ensure we communicate scheduled outages to our HPC users in advance. Where possible, we perform rolling upgrades with the cluster online. However, sometimes we have to perform upgrades that require the cluster to be taken offline. These include:

• system software upgrades;

• network maintenance;

• bug and security patches; and

• hardware maintenance

This site contains the documentation for the MASSIVE HPC systems. M3 is the newest addition to the facility.

Help and Support

• Help and Support
• Scheduled Maintenance
• MASSIVE System Status and Known Issues
• MASSIVE Terms of Use

Using M3

• About M3
  • System configuration
  • Compute
  • GPU Compute
  • Remote desktops through the CVL
• Requesting an account
  • Overview of the process
    • 1. Log in to the HPC ID system
    • 2. Create an M3 account
    • 3. Create or join an M3 project
    • 4. Set your M3 account password
    • 5. Log in to M3
    • Request an M3 account via email
• Requesting help on M3
  • Software issues
  • System issues
  • Contacting us
    • Help Desk
    • General Enquiries
    • Postal Address
• Connecting to M3
  • Connecting to M3 via ssh
    • Linux and OS X Users
    • Windows Users
  • Connecting to M3 via the MASSIVE desktop
    • MASSIVE Desktop (Strudel)
    • MASSIVE Desktop in browser (Strudel Web)
  • Connecting to M3 via Strudel2
    • Using the Desktop Application in Strudel2
    • Using the JupyterLab Application in Strudel2
    • Using the Terminal Application in Strudel2
• File Systems on M3
  • What to put on each file system?
    • Home directory (~10GB)
    • Project directory (Shared with everyone on your project)
    • Scratch directory (Shared with everyone on your project)
  • Disk Quotas
  • Default Quotas for New Projects
  • Scratch Usage Policies
  • System Backups and File Recovery
    • File Recovery Procedure
  • Information for Desktop Users
    • Thumbnails Generating Too Much Data
    • Remember to empty your trash folder
    • Already over quota?
  • Storage outside of M3
    • Instructions to access Market share
    • Instructions to access Vault
• Copying files to and from M3
  • GUI Tool - Windows, Mac OS, and Linux Users
    • FileZilla
    • WinSCP - Windows
  • Globus
    • Getting Started
    • Personal Globus Endpoint
    • Transferring Data
  • Globus - Command Line Interface
  • Globus - Platform as a Service
    • Globus Juptyer Notebooks
    • Data Portals
  • Command Line Interface - Linux and OS X Users
    • rsync
• Software on M3
  • Modules
    • Requesting an install
  • Containers and Docker on M3
  • Python and conda on M3
  • Software for the Monash Bioinformatics Platform (MBP)
  • Software Deprecation: Java
• Running jobs on M3
  • Partitions Available
  • Other partitions
  • Checking the status of M3
    • The STATUS field explained
  • Slurm Accounts
    • Default accounts
    • Setting the account for a job
    • Questions about slurm accounts
  • Getting started with job submission scripts
  • Running Simple Batch Jobs
    • An example Slurm job script
    • Cancelling jobs
  • Running MPI Jobs
    • An example Slurm MPI job script
  • Running Multi-threading Jobs
    • An example Slurm Multi-threading job script
  • Running Interactive Jobs
    • Submitting an Interactive Job
    • How long do interactive jobs last for?
    • Reconnecting to/Disconnecting from an Active Interactive Job
  • Running GPU Jobs
    • Running GPU Batch Jobs
    • Compiling your own CUDA or OpenCL codes for use on M3
  • Running Array Jobs
    • An example of Slurm Array job script
  • QoS (Quality of Service)
    • How to run jobs with QoS
    • Explanation
  • Features & Constraints
    • An example of using constraint in the job script
    • Features Available
  • Checking job status
    • Method 1: show_job
    • Method 2: Slurm commands
  • Project Allocation
    • Project Space
    • Questions about allocation
• Lustre File System Quickstart Guide
  • About Lustre
• GPUs on M3
  • Starter Guide: GPUs on M3
    • GPUs on HPC (remote) vs. Laptop/Workstation (local)
    • How do I choose a GPU?
    • Questions to ask yourself when choosing a GPU
  • GPU Look-Up Tables
    • How do I choose a GPU?
    • Look-Up Tables
  • Frequently Asked Questions About GPUs on M3
    • How long will I wait for a GPU?
    • Why am I waiting so long for a GPU?
    • How do I know if my job is using the GPU?
    • I’m in the Machine Learning Community, do you have any specific advice for me?

Communities

• MX2 Eiger
  • The Big Picture
  • Authorising MASSIVE to download your data (CAP Leaders only)
  • Creating research projects (research lab leaders only)
  • Requesting access to M3
  • Getting started on M3
  • Connecting to M3
  • Accessing your MX data
  • Reprocessing your MX data
  • Recommended Settings
  • Known Issues
    • Cannot open or read filename.tmp error
  • Need help?
• Machine Learning
  • Software
    • Caffe
    • TensorFlow
    • PyTorch
    • Keras
    • Scikit-learn
  • Reference datasets
  • Quick guide for checkpointing
    • Why checkpointing?
• Neuroimaging
  • Using SLURM to submit a simple FSL job
    • Background
    • Data and scripts
• Cryo EM
  • Relion
    • The Graphical Interface
    • Using the Queue
    • Motion Correction
    • 2d/3d Classification & Refinement
    • Particle Polishing
  • Cryo EM Pre-Processing Tool
  • Cryo EM Benchmarking and Optimisation
    • Cryo EM Benchmarking and Optimisation
• Bioinformatics
  • Requesting an account on M3
• The Genomics partition
  • Getting started with the Bioinformatics module
    • Importing the Bioinformatics module environment
    • Installing additional software with Bioconda
  • “Pipelines” and workflow managers on M3
    • Running NextFlow on M3
  • FAQ
• DGX
  • Hardware
  • How to access the DGX hardware
  • What jobs are suitable?
    • How do you demonstrate a suitable project/job?
• Data Collections
  • Machine learning
    • ImageNet 2012 (ILSVRC2012)
    • ImageNet (Fall 2011)
    • International Skin Imaging Collaboration 2019 (ISIC 2019)
    • NIH Chest X-ray Dataset (NIH CXR-14)
    • Stanford Natural Language Inference (SNLI) Corpus
    • COCO (Common Objects in Context) 2017
  • Neuroimaging
    • Human Connectome Project Dataset (HCP): HCP-1200
    • Brain Genomics Superstruct Project (GSP)
    • Nathan Kline Institute Rockland Sample (NKI-RS): Neuroimaging Release
  • Genomes
    • BlastDB
  • Requesting a data collection
• XNAT
  • Create an account at Monash-XNAT
  • Request to create a project at Monash-XNAT
  • Mirroring data from Alfred-XNAT to Monash-XNAT
  • Pulling Data from Monash-XNAT to MASSIVE M3

FAQs

• Frequently asked questions
  • Accounts
  • Jobs
  • M3, desktops and Strudel
• About Field of Research (FOR) and Socio-Economic Objective (SEO) codes
• Miscellaneous
• CloudStor Plus
  • Getting Started
  • CloudStor Plus on the CVL Desktop
  • Deactivating your CloudStor Plus Account on the CVL Desktop