Description

The UMCG cluster is composed of:

• 1 head node
• 10 compute nodes each with:
  • 48 cores
  • 256GB RAM
  • 2.3TB of local storage
  • 10Gb network connection to storage
• 2 PB GPFS storage (only 1.1PB mounted at time of writing)

The 10 nodes are dedicated for the GCC group at UMCG. GoNL being the most compute intensive project at GCC, most of the cluster can be used for it. The storage is shared by different groups in Groningen but there is currently no "hard limit" on how much space GoNL can use on the storage; this of course will only work as long as there is sufficient space for everyone.

Access

Access to the UMCG cluster is done via SFTP (application20.target.rug.nl) (data access only, see SFTP page about this) or SSH (application30.target.rug.nl; only works from registered IP addresses!). There is no public access to the UMCG cluster. Additional personal ssh or sftp accounts can be requested via Morris who keeps the list of all users that have full data access.

Usage

First of all here are a few important things to know about the cluster and using it efficiently:

• Head Node: The head node should NOT be used to run any job directly nor any intensive process. Here intensive means either CPU, RAM or I/O intensive (i.e. large copying jobs should be run as jobs and queued appropriately). Note that loading the head node can cause the scheduler to crash which is harmful for everyone running jobs on the cluster!
• Storage: The block size on the storage is 6MB, which means that each file -regardless of its real size- will occupy at least 6MB on the file system. This means that data should rather be kept in big files rather than a multitude of small files whenever possible. Typically things like logs, old submit scripts, etc. should be compressed into 1 file for archiving.
• I/O: While 10Gb network connection per node is fast, typical GoNL jobs use large files and consumes lots of I/O. Therefore, I/O should be kept minimal and if a job can be parallelized on multiple cores (i.e. load data once in memory, process it on multiple cores, push it back), it is typically preferred as having separate processes all loading the same data in memory.
• Local Storage: In order to reduce I/O, temporary files (and eventually other heavily used resources) should be stored directly on the local node; the local storage on each node is mounted in /local. Note that:
  • Any data on the local storage that you want to keep after the job terminates should be copied to the general storage as the local storage is periodically cleaned and any data that is not in use by currently running job will be deleted.
  • Even if the local storage is periodically cleaned, if you store large files on a node while running a job you should clean afterwards. Small temp files are fine.
• Data Management: Please read thoroughly the Data Management section of this wiki and respect the structure and conventions described there when using data outside your home directory.

Scheduler

Application30 uses the ​Portable Bash System (PBS) scheduling system. You can find the full documentation in this ​PBS guide. However, here are a few basic commands and tips:

• qstat -u username
  • Shows a list of your jobs along with information and status
• showq [-u username]
  • Shows the list of all jobs (if you use the -u flag, only the user's jobs) running on the cluster along with information and status.
• checkjob jobid
  • Shows in-depth information about a specific job.
• qsub jobScript
  • Submit a new job to the cluster. Note that is is important to submit your jobs with the appropriate options; See the qsub flags section below for a quick overview of the common options.
• qdel jobid
  • Removes a job from the queue, killing the process if it was already started
  • "qdel all" can be used to purge all of your jobs

qsub options

Jobs to be submitted via PBS qsub can specify a number of options to claim resources, report status, etc. These options can either be specified in the qsub command or in your job script. The latter is usually preferred as all information about the job including memory requirements, etc. stay with the script, below is an example header with some commonly used options, followed by a list of some commonly used flags and their meaning.

Example script header:

#!/bin/bash
#PBS -N JobName
#PBS -q gcc
#PBS -l nodes=1:ppn=1
#PBS -l mem=4gb
#PBS -l walltime=12:00:00
#PBS -o /target/gpfs2/gcc/home/lfrancioli/output.log
#PBS -e /target/gpfs2/gcc/home/lfrancioli/error.log

#Here comes your bash script commands

echo "Hello World!"

Commonly used options:

• -q queueName
  • Selects which queue the job should be put in. The only queue available at the moment is 'gcc'
• -N jobName
  • Set the job name
• -l nodes=X:ppn=Y
  • Requests X nodes and Y cores per node
• -l mem=Xgb
  • Requests X GB RAM
• -l walltime=12:00:00
  • Sets the walltime to the specified value (here 12hrs). This flag should be set.
• -j oe
  • Redirects all error output to standard output
• -o outputLog
  • Redirects the standard output to the desired file. Note that using '~' in the path for you home directory does not work.
  • Note that the standard output is first written on the local node and only copied once the job terminates (regardless of the reason of the job termination).
• -e errorLog
  • Redirects the error output to the desired file. Note that using '~' in the path for you home directory does not work.
  • `Note that the error output is first written on the local node and

only copied once the job terminates (regardless of the reason of the

job termination).`