NAF2.0 for ILC users

Why and how to maintain (change/update) my password by myself from inside and outside DESY

# Or you can change your password in this way.

Why and How to run my jobs on working nodes (Marlin ... ...)

ssh -X -l yourusername
condor_submit myjob.submit

This part about "BIRD/SGE" is going to be removed after migrating to HTCondor completely.

#(execute my job from the current directory)
#$ -cwd
#(the same OS as login machine)
#$ -l arch=amd64
#$ -l os=sld6
#(the cpu time for my job)
#$ -l h_rt=23:59:00
#(the maximum memory for my job)
#$ -l h_vmem=5G
#(send email when my job ends)
#$ -m ae
#(send to this email address)
##$ -M myemail@my.server
#(paste my init soft here)

#(paste my program here)


Getting a NAF2.0 account

 [flc desktop] id <your_accout>
 uid=NNNNN(<your_account>)gid=NNN(flc) groups=1417(flc),5295(af-ilc)

Login to a NAF2.0 ILC Workgroupserver

There are two work group servers:

  • SL6
  • SL6

nafhh-ilc01 has been recently migrated to sl6, and has the identical configuration as nafhh-ilc02.

Now both could access the CVMFS ilcsoft installation, GRID storage element dCache, and the new NAF2 scratch storage space DUST.

For login, simply do an ssh to your favourite work group server.

ssh -X -l yourusername
ssh -X -l yourusername

nafhh-ilc01 and nafhh-ilc02 are both rather powerful machines with which you can test your jobs without causing any issues. However, please do not copy/move large or many data files on these machines. This consumes the complete bandwidth and slows down the machines dramatically. Instead, use the qrsh command (see further down below) to access a bird machine and run your copy-job there.

Batch system - BIRD cluster

The NAF2.0 uses the general purpose batch system BIRD (Batch Infrastructure Resource at DESY).

To get details about your jobs (and only yours), use


A summary of current queue usage can be produced by doing

qstat -g c

Start the GUI:

qmon &

Direct login: Sometimes it's useful to run some commands interactively on the node similar to the one you plan to submit your jobs to.

qlogin -q login.q

or, if you wish to specify a particular node

qlogin -q

or, if you want to request specify resource node

qrsh -l distro=sld6 -l arch=amd64 -l h_vmem=2G -l h_rt=12:00:00

Information on how to use it can be found at

Note: The BIRD cluster have the same configuration as our work group server. All nodes could access the CVMFS ilcsoft installation, GRID storage element dCache, and the new NAF2 scratch storage space DUST.

When you find your jobs can run on most of the BIRD nodes, but have problem on one specify node, please send email to bird.service<at>, stating the BIRD node name, and the problem.

Note In order to be able to send your jobs to BIRD cluster, you need permission to access resource "batch(IT)". You may check it by yourself by login onto with your desy account, and looking at the column "Current resource access" to verify the content "batch(IT)". If it did not exist, you may ask one administrator to help you. You will find the administrator list by clicking on "Administrators" at the left column on your login screen.

Storage systems

The main scratch space is DUST, Please note that this is scratch space, i.e. there is no backup!

Report Storage Issues

If you feel a problem about the storage, and want to know "How to report storage issues?". Please read this documentation


AFS: The AFS cell /afs/ is used to provide each user with a home directory.

To learn about your current quota usage,

fs lq


Large scratch space: The technology DUST is used to provide fast scratch space. To access your working space, do

cd /nfs/dust/ilc/user/<yourspace>

The scratch has been mounted on the BIRD cluster nodes, your working space can be accessed from BIRD cluster nodes, too.

For both users and administrators.

As a registry namespace administrator, you will see an additional button "GPFS Management".

The following DUST related tasks can be managed via Amfora:



Access to experiments data on dCache: Fast access is provided to the DESY dCache systems where experiments data is hosted.

The ILC users may access the data at /pnfs/

All the BIRD working nodes and WGS could access the data directly with the full path. It is NOT necessary to copy them to any DUST space!

Grid Certificates

ssh -X -l yourusername
# DO NOT need any ini, or script, the grid envoriment is ready for the user.
# currently, please do "export X509_USER_PROXY=$HOME/k5-ca-proxy.pem", or put it into your "$HOME/.zshrc" if you use zsh.
export X509_USER_PROXY=$HOME/k5-ca-proxy.pem
# If your grid certificate has been initialized, you can read it, or re-initialize it. It will be saved into the same "$HOME/k5-ca-proxy.pem".
# Which can be accessed from AFS, NAF2, BIRD working nodes
voms-proxy-info -all
voms-proxy-init -voms ilc -valid 24:00

ILC/CALICE software

Example for "zsh" and "bash" shell users. You may find out what kind of shell you are using by the command "echo $SHELL".

echo $SHELL

echo $SHELL

ILCsoft CVMFS installation

[@nafhh-ilc02] ls /cvmfs/

[@nafhh-ilc02] ls /cvmfs/

ILC specifics

For the NAF2.0 ILC users, (NAF2.0 are 64bit machine now), please use:

qlogin -q login.q
source /cvmfs/

Please checkout ILDConfig. And following the to run the sim/reco jobs.

git clone
less ILDConfig/StandardConfig/production/

Additional information can be found here:, and

CALICE specifics

For the CALICE, the DESY HCAL group provides the CALICE software.


more information about calice software:

Useful Tricks

  • How to know you are properly registered for access the BIRD resource?

Please check if you are properly registered by doing qconf -suserl | grep yourusername. If nothing shows up, you need to add the resource batch in the registry. Please contact FLC namespace administrator or UCO for this.

More information about UCO, please checkout this link:

More Documentations

  • Please find the following links, if you want to read something more.

Modules on SLD6:

DESY IT: Software_Env_with_Modules.

NAF2Start (last edited 2018-04-04 09:56:13 by ShaojunLu)