NAF2.0 for ILC users

1. Basics

1.1. DESY User Consulting Office (UCO)

{i} If you have problems finding the right mailing list in the following documentation for reporting, you may try to send an e-mail to DESY UCO at least:

1.2. Password Change

ssh USERNAME@passwd.desy.de

1.3. Getting Access to NAF2.0

1.3.1. Access to the ILC Ressources

 [flc desktop] id <your_accout>
 uid=NNNNN(<your_account>)gid=NNN(flc) groups=1417(flc),5295(af-ilc)

1.3.2. Batch System

1.4. NAF2.0 ILC Workgroupservers

Name

OS

Group

Scheduler

Cores

naf-ilc11.desy.de

SL6

af-ilc

HTCondor

20

naf-ilc12.desy.de

SL6

af-ilc

HTCondor

12

naf-ilc13.desy.de

SL6

af-ilc

HTCondor

12

naf-ilc-el7.desy.de (VM)

EL7

af-ilc

HTCondor

2

nafhh-ilc01.desy.de, phased out

SL6

af-ilc

SGE

12

ssh -X yourusername@naf-ilc.desy.de

The Workgroupservers are both rather powerful machines with which you may compile and test your programs without causing any issues. However:

/!\ Please do not copy or move several data files in parallel on these machines! This consumes the complete bandwidth and slows down the machines dramatically.

1.5. Where and how to run programs

######################################################
# HTCondor Submit Description File. COMMON TEMPLATE
#     my_htc_job.submit
# Next to submit it to BIRD:
#     condor_submit my_htc_job.submit
######################################################

# Default Universe for normal jobs
Universe = vanilla
+MyProject = "af-ilc"
Requirements = OpSysAndVer=="SL6"

InitialDir = $ENV(PWD)

# Here is for your Executable file
Executable = run_MyApps_with_MyArguments.sh
transfer_executable = True
Should_Transfer_Files = IF_NEEDED

# Infos from your job: Name them in your way
Output = $(Cluster).$(Process).out
Error = $(Cluster).$(Process).err
Log = $(Cluster).$(Process).log

# Mailing requests: Update as you need
Notification = Always

# BIRD resources requests: Update as you need
+RequestRuntime = 3600 * 3
RequestMemory = 1024 * 2
RequestDisk = 2048 * 3
RequestCpus = 1

# This will submit 100 BIRD jobs,
# with different input from "Arguments"
# Process: from 0 to 99
N = 100
Arguments = "MyInput_$(Process).slcio 30"
Queue $(N)

#FYI: you can use the "Arguments" in command line too.

{i} Examples can be found in NAF2 ILD group directory: /nfs/dust/ilc/group/ild/NAF2_HTCondor

2. Batch system - BIRD cluster

2.1. Overview

The NAF2.0 uses the general purpose batch system BIRD (Batch Infrastructure Resource at DESY). The BIRD cluster nodes have the same configuration as our work group server. They can access the CVMFS ilcsoft installation, GRID storage element dCache, and the NAF2 scratch storage space DUST.

When you find your jobs can run on most of the BIRD nodes, but have a problem on one specify node, please send an email to bird.service<at>desy.de, stating the BIRD node name, and the problem.

2.2. Scheduling jobs

condor_submit myjob.submit

2.3. Interactive job

condor_submit -i
hostname

condor_submit -i interactive.submit
hostname

######################################################
# HTCondor Submit Description File. COMMON TEMPLATE
#     interactive.submit
# Next to login onto one BIRD WN:
#     condor_submit -i interactive.submit
######################################################
# condor_config_val MaxJobRetirementTime : 3600 * 24 * 7

+MyProject = "af-ilc"
Requirements = OpSysAndVer=="SL6"

+RequestRuntime = 3600 * 3

queue 1


{i} We are currently in the process of migrating to the new batch and scheduling software, HTCondor. The old "BIRD/SGE" is going to be removed after migrating to HTCondor completely. For the time being, you may still use it on nafhh-ilc01.

3. Storage Systems

Storage System

Size

Backup

Read/Write

Type

AFS user directory

16 GB

yes (multiple)

yes

Disk

DUST

1TB

no

yes

Disk

dCache

huge

yes

read only

Disk&Tape

CVMFS

O(GB)

yes, versioning

read only

Disk

3.1. AFS

AFS: The AFS cell /afs/desy.de is used to provide each user with a home directory.

To learn about your current quota usage,

fs lq [--human]

3.2. DUST

Large scratch space: The technology DUST is used to provide fast scratch space. As there is no backup you should use it for big data that can be easily reproduced.

To access your working space, do

cd /nfs/dust/ilc/user/<yourspace>

Your working space can be accessed from the BIRD cluster nodes, too.

Checking DUST quota:

Administration:

/nfs/dust/ilc/group/ild
/nfs/dust/ilc/group/flctpc
/nfs/dust/ilc/group/flchcal

3.3. dCache

Access to experiments data on dCache: Fast access is provided to the DESY dCache systems where experiments data is hosted.

The ILC users may access the data at /pnfs/desy.de/ilc/.This mount is read-only.

/!\ All the BIRD working nodes and WGS can access the data directly with the full path. It is NOT necessary to copy them to any DUST space!

3.4. Report Storage Issues

If you experience a problem with the storage you can report it, following these instructions: https://naf-wiki.desy.de/ReporaboutStorageIssues

4. Grid User Interface Tools

5. ILC/CALICE software

Example for "zsh" and "bash" shell users. You may find out what kind of shell you are using by the command "echo $SHELL".

echo $SHELL
/bin/zsh

echo $SHELL
/bin/bash

5.1. ILCsoft CVMFS installation

[@naf-ilc12] ls /cvmfs/ilc.desy.de/sw/x86_64_gcc49_sl6/v02-00-01/

[@naf-ilc12] ls /cvmfs/ilc.desy.de/sw/ILDConfig/v02-00-01/

You also have change to access one nightly build ilcsoft from cvmfs server.
[@naf-ilc12] ls /cvmfs/clicdp.cern.ch/iLCSoft/builds/nightly/

5.2. ILC specifics

For the NAF2.0 ILC users, (NAF2.0 are 64bit machine now), please use:

source /cvmfs/ilc.desy.de/sw/x86_64_gcc49_sl6/v02-00-01/init_ilcsoft.sh

Please checkout ILDConfig. And following the README.md to run the sim/reco jobs.

git clone https://github.com/iLCSoft/ILDConfig.git
less ILDConfig/StandardConfig/production/README.md

Additional information can be found here: http://ilcsoft.desy.de, and https://confluence.desy.de/display/ILD+Software+Working+Group

5.3. CALICE specifics

For the CALICE, the DESY HCAL group provides the CALICE software.

/afs/desy.de/group/flc/hcal/calice_soft/v04-08-03

more information about calice software: https://twiki.cern.ch/twiki/bin/view/CALICE/SoftwareNews

6. Modules on SLD6

[@naf-ilc11] module avail git
--------------------------------------------------------------------- /etc/modulefiles ---------------------------------------------------------------------
git/1.9

[@naf-ilc11] module load git
[@naf-ilc11] git --version
git version 1.9.0

[@naf-ilc11] which git
/opt/git/1.9/bin/git

DESY IT: Software_Env_with_Modules.

http://modules.sourceforge.net/.

NAF2Start (last edited 2018-08-22 11:36:27 by OliverSchaefer)