| 4 | | * 1 head node |
| 5 | | * 10 compute nodes each with: |
| 6 | | * 48 cores |
| 7 | | * 256GB RAM |
| 8 | | * 2.3TB of local storage |
| 9 | | * 10Gb network connection to storage |
| 10 | | * 2 PB GPFS storage (only 1.1PB mounted at time of writing) |
| 11 | | |
| 12 | | The 10 nodes are dedicated for the GCC group at UMCG. GoNL being the most compute intensive project at GCC, most of the cluster can be used for it. The storage is shared by different groups in Groningen but there is currently no "hard limit" on how much space GoNL can use on the storage; this of course will only work as long as there is sufficient space for everyone. |
| 13 | | |
| 14 | | == Access == |
| 15 | | Access to the UMCG cluster is done via: |
| 16 | | * SFTP via sftp.gcc.rug.nl - data access only, see [wiki:DataManagement/SftpServer SFTP page] for details |
| 17 | | * SSH via cluster.gcc.rug.nl - to submit and monitor jobs. Connections from outside the UMCG/RUG network require a double hop via proxy.gcc.rug.nl, which can be automated as described in [http://wiki.gcc.rug.nl/wiki/TransparentMultiHopSSH TransparentMultiHopSSH] |
| 18 | | |
| 19 | | Additional accounts can be requested via Morris who keeps the list of all users that have full data access. |
| | 4 | Access to the default *gcc* queue is available to all GoNL members. If you do not have an account yet, you can request one via Morris who keeps the list of all users that have full data access. |
| 30 | | == Scheduler == |
| 31 | | Cluster.gcc.rug.nl uses the [http://doesciencegrid.org/public/pbs/ Portable Batch System (PBS)] scheduling system. You can find the full documentation in this [http://doesciencegrid.org/public/pbs/ PBS guide]. However, here are a few basic commands and tips: |
| 32 | | |
| 33 | | * qstat -u username |
| 34 | | * Shows a list of your jobs along with information and status |
| 35 | | * showq [-u username] |
| 36 | | * Shows the list of all jobs (if you use the -u flag, only the user's jobs) running on the cluster along with information and status. |
| 37 | | * checkjob jobid |
| 38 | | * Shows in-depth information about a specific job. |
| 39 | | * qsub jobScript |
| 40 | | * Submit a new job to the cluster. Note that is is important to submit your jobs with the appropriate options; See the qsub flags section below for a quick overview of the common options. |
| 41 | | * qdel jobid |
| 42 | | * Removes a job from the queue, killing the process if it was already started |
| 43 | | * "qdel all" can be used to purge all of your jobs |
| 44 | | == Available queues == |
| 45 | | In order to quickly test jobs you are allowed to run the directly on cluster.gcc.rug.nl outside the scheduler. Please think twice though before you hit enter: if you crash cluster.gcc.rug.nl others can no longer submit or monitor their jobs, which is pretty annoying. On the other hand it's not a disaster as the scheduler and execution daemons run on physically different servers and hence are not affected by a crash of cluster.gcc.rug.nl. |
| 46 | | |
| 47 | | To test how your jobs perform on an execution node and get an idea of the typical resource requirements for your analysis you should submit a few jobs to the test queues first. The test queues run on a dedicated execution node, so in case your jobs make that server run out of disk space, out of memory or do other nasty things accidentally, it will not affect the production queues and ditto nodes. |
| 48 | | |
| 49 | | Once you've tested your job scripts and are sure they will behave nice & perform well, you can submit jobs to the production queue named ''gcc''. |
| 50 | | |
| 51 | | ||**Queue**||**Job type**||**Limits**|| |
| 52 | | ||test-short||debugging||10 minutes max. walltime per job; limited to a single test node / 48 cores|| |
| 53 | | ||test-long||debugging||max 4 jobs running simultaneously per user; limited to half the test node / 24 cores|| |
| 54 | | ||gcc||production - default prio||none|| |
| 55 | | ||gaf||production - high prio||only available to users from the gaf group|| |
| 56 | | |
| 57 | | == Nodes == |
| 58 | | |
| 59 | | It is not allowed to directly run tasks on the nodes. Always use the scheduler. If you want to test then use the test queues. |
| 60 | | |
| 61 | | == Local data == |
| 62 | | |
| 63 | | If for some reason it is necessary to use local disk space instead of the GPFS then you need to request local disk space like this: |
| 64 | | {{{ |
| 65 | | #PBS -l file=10mb |
| 66 | | }}} |
| 67 | | A private temp folder will be created. This folder will be removed after your script is finished. You can access this folder using this environment variable: $TMPDIR. Please make sure you do not use more disk space then requested. |
| 68 | | |
| 69 | | File created on local disk without using this system can be deleted without notice. |
| 70 | | |
| 71 | | === qsub options === |
| 72 | | Jobs to be submitted via PBS qsub can specify a number of options to claim resources, report status, etc. These options can either be specified in the qsub command or in your job script. The latter is usually preferred as all information about the job including memory requirements, etc. stay with the script, below is an example header with some commonly used options, followed by a list of some commonly used flags and their meaning. |
| 73 | | |
| 74 | | '''`Example script header:`''' |
| 75 | | |
| 76 | | {{{ |
| 77 | | #!/bin/bash |
| 78 | | #PBS -N JobName |
| 79 | | #PBS -q gcc |
| 80 | | #PBS -l nodes=1:ppn=1 |
| 81 | | #PBS -l mem=10mb |
| 82 | | #PBS -l mem=4gb |
| 83 | | #PBS -l walltime=12:00:00 |
| 84 | | #PBS -o /target/gpfs2/gcc/home/lfrancioli/output.log |
| 85 | | #PBS -e /target/gpfs2/gcc/home/lfrancioli/error.log |
| 86 | | |
| 87 | | #Here comes your bash script commands |
| 88 | | |
| 89 | | echo "Hello World!" |
| 90 | | }}} |
| 91 | | '''Commonly used options:''' |
| 92 | | |
| 93 | | * -q queueName |
| 94 | | * Selects which queue the job should be put in. The only queue available at the moment is 'gcc' |
| 95 | | * -N jobName |
| 96 | | * Set the job name |
| 97 | | * -l `nodes=X:ppn=Y` |
| 98 | | * `Requests X nodes and Y cores per node` |
| 99 | | * `-l mem=Xgb` |
| 100 | | * `Requests X GB RAM` |
| 101 | | * `-l walltime=12:00:00` |
| 102 | | * `Sets the walltime to the specified value (here 12hrs). This flag should be set.` |
| 103 | | * `-j oe` |
| 104 | | * `Redirects all error output to standard output` |
| 105 | | * `-o outputLog` |
| 106 | | * `Redirects the standard output to the desired file. Note that using '~' in the path for you home directory does not work.` |
| 107 | | * `Note that the standard output is first written on the local node and only copied once the job terminates (regardless of the reason of the job termination).` |
| 108 | | * `-e errorLog` |
| 109 | | * `Redirects the error output to the desired file. Note that using '~' in the path for you home directory does not work.` |
| 110 | | * `Note that the error output is first written on the local node and |
| 111 | | only copied once the job terminates (regardless of the reason of the |
| 112 | | job termination).` |
