Changes between Version 7 and Version 8 of ComputeResources/UMCGCluster


Ignore:
Timestamp:
Aug 26, 2011 2:05:59 PM (13 years ago)
Author:
laurent
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ComputeResources/UMCGCluster

    v7 v8  
    1818First of all here are a few '''important''' things to know about the cluster and using it efficiently:
    1919
     20 * '''Head Node''': The head node should NOT be used to run any job directly nor any intensive process. Here intensive means either CPU, RAM or I/O intensive (i.e. large copying jobs should be run as jobs and queued appropriately). Note that loading the head node can cause the scheduler to crash which is harmful for everyone running jobs on the cluster!
    2021 * '''Storage''': The block size on the storage is 6MB, which means that each file -regardless of its real size- will occupy at least 6MB on the file system. This means that data should rather be kept in big files rather than a multitude of small files whenever possible. Typically things like logs, old submit scripts, etc. should be compressed into 1 file for archiving.
    2122 * '''I/O''': While 10Gb network connection per node is fast, typical GoNL jobs use large files and consumes lots of I/O. Therefore, I/O should be kept minimal and if a job can be parallelized on multiple cores (i.e. load data once in memory, process it on multiple cores, push it back), it is typically preferred as having separate processes all loading the same data in memory.
    … …  
    2425   * Even if the local storage is periodically cleaned, if you store large files on a node while running a job you should clean afterwards. Small temp files are fine.
    2526 * '''Data Management''': Please read thoroughly the [wiki:DataManagement Data Management] section of this wiki and respect the structure and conventions described there when using data outside your home directory.
    26 
    2727== Scheduler ==
    2828Application30 uses the [http://doesciencegrid.org/public/pbs/ Portable Bash System (PBS)] scheduling system. You can find the full documentation in this [http://doesciencegrid.org/public/pbs/ PBS guide]. However, here are a few basic commands and tips: