Description

The UMCG cluster is composed of:

• 1 head node
• 10 compute nodes each with:
  • 48 cores
  • 256GB RAM
  • 2.3TB of local storage
  • 10Gb network connection to storage
• 2 PB GPFS storage (only 1.1PB mounted at time of writing)

The 10 nodes are dedicated for the GCC group at UMCG. GoNL being the most compute intensive project at GCC, most of the cluster can be used for it. The storage is shared by different groups in Groningen but there is currently no "hard limit" on how much space GoNL can use on the storage; this of course will only work as long as there is sufficient space for everyone.

Access

Access to the UMCG cluster is done via:

• SFTP via sftp.gcc.rug.nl - data access only, see SFTP page for details
• SSH via cluster.gcc.rug.nl - to submit and monitor jobs. Connections from outside the UMCG/RUG network require a double hop via proxy.gcc.rug.nl, which can be automated as described in ​TransparentMultiHopSSH

Additional accounts can be requested via Morris who keeps the list of all users that have full data access.

Usage

First of all here are a few important things to know about the cluster and using it efficiently:

• Storage: The block size on the storage is 6MB, which means that each file -regardless of its real size- will occupy at least 6MB on the file system. This means that data should rather be kept in big files rather than a multitude of small files whenever possible. Typically things like logs, old submit scripts, etc. should be compressed into 1 file for archiving.
• I/O: While 10Gb network connection per node is fast, typical GoNL jobs use large files and consumes lots of I/O. Therefore, I/O should be kept minimal and if a job can be parallelized on multiple cores (i.e. load data once in memory, process it on multiple cores, push it back), it is typically preferred as having separate processes all loading the same data in memory.
• Local Storage: In order to reduce I/O, temporary files (and eventually other heavily used resources) should be stored directly on the local node; the local storage on each node is mounted in /local. Note that:
  • Any data on the local storage that you want to keep after the job terminates should be copied to the general storage as the local storage is periodically cleaned and any data that is not in use by currently running job will be deleted.
  • Even if the local storage is periodically cleaned, if you store large files on a node while running a job you should clean afterwards. Small temp files are fine.
• Data Management: Please read thoroughly the Data Management section of this wiki and respect the structure and conventions described there when using data outside your home directory.

Scheduler

Cluster.gcc.rug.nl uses the ​Portable Batch System (PBS) scheduling system. You can find the full documentation in this ​PBS guide. However, here are a few basic commands and tips:

• qstat -u username
  • Shows a list of your jobs along with information and status
• showq [-u username]
  • Shows the list of all jobs (if you use the -u flag, only the user's jobs) running on the cluster along with information and status.
• checkjob jobid
  • Shows in-depth information about a specific job.
• qsub jobScript
  • Submit a new job to the cluster. Note that is is important to submit your jobs with the appropriate options; See the qsub flags section below for a quick overview of the common options.
• qdel jobid
  • Removes a job from the queue, killing the process if it was already started
  • "qdel all" can be used to purge all of your jobs

Available queues

In order to quickly test jobs you are allowed to run the directly on cluster.gcc.rug.nl outside the scheduler. Please think twice though before you hit enter: if you crash cluster.gcc.rug.nl others can no longer submit or monitor their jobs, which is pretty annoying. On the other hand it's not a disaster as the scheduler and execution daemons run on physically different servers and hence are not affected by a crash of cluster.gcc.rug.nl.

To test how your jobs perform on an execution node and get an idea of the typical resource requirements for your analysis you should submit a few jobs to the test queues first. The test queues run on a dedicated execution node, so in case your jobs make that server run out of disk space, out of memory or do other nasty things accidentally, it will not affect the production queues and ditto nodes.

Once you've tested your job scripts and are sure they will behave nice & perform well, you can submit jobs to the production queue named gcc.

Queue	Job type	Limits
test-short	debugging	10 minutes max. walltime per job; limited to a single test node / 48 cores
test-long	debugging	max 4 jobs running simultaneously per user; limited to half the test node / 24 cores
gcc	production - default prio	none
gaf	production - high prio	only available to users from the gaf group

Nodes

It is not allowed to directly run tasks on the nodes. Always use the scheduler. If you want to test then use the test queues.

Local data

If for some reason it is necessary to use local disk space instead of the GPFS then you need to request local disk space like this:

#PBS -l file=10mb

A private temp folder will be created. This folder will be removed after your script is finished. You can access this folder using this environment variable: $TMPDIR. Please make sure you do not use more disk space then requested.

File created on local disk without using this system can be deleted without notice.

qsub options

Jobs to be submitted via PBS qsub can specify a number of options to claim resources, report status, etc. These options can either be specified in the qsub command or in your job script. The latter is usually preferred as all information about the job including memory requirements, etc. stay with the script, below is an example header with some commonly used options, followed by a list of some commonly used flags and their meaning.

Example script header:

#!/bin/bash
#PBS -N JobName
#PBS -q gcc
#PBS -l nodes=1:ppn=1
#PBS -l mem=10mb
#PBS -l mem=4gb
#PBS -l walltime=12:00:00
#PBS -o /target/gpfs2/gcc/home/lfrancioli/output.log
#PBS -e /target/gpfs2/gcc/home/lfrancioli/error.log

#Here comes your bash script commands

echo "Hello World!"

Commonly used options:

• -q queueName
  • Selects which queue the job should be put in. The only queue available at the moment is 'gcc'
• -N jobName
  • Set the job name
• -l nodes=X:ppn=Y
  • Requests X nodes and Y cores per node
• -l mem=Xgb
  • Requests X GB RAM
• -l walltime=12:00:00
  • Sets the walltime to the specified value (here 12hrs). This flag should be set.
• -j oe
  • Redirects all error output to standard output
• -o outputLog
  • Redirects the standard output to the desired file. Note that using '~' in the path for you home directory does not work.
  • Note that the standard output is first written on the local node and only copied once the job terminates (regardless of the reason of the job termination).
• -e errorLog
  • Redirects the error output to the desired file. Note that using '~' in the path for you home directory does not work.
  • `Note that the error output is first written on the local node and

only copied once the job terminates (regardless of the reason of the

job termination).`