Using resources effectively
Overview
Teaching: 15 min
Exercises: 10 minQuestions
How do we monitor our jobs?
How can I get my jobs scheduled more easily?
Objectives
Understand how to look up job statistics and profile code.
Understand job size implications.
We now know virtually everything we need to know about getting stuff on a cluster. We can log on, submit different types of jobs, use preinstalled software, and install and use software of our own. What we need to do now is use the systems effectively.
Estimating required resources using the scheduler
Although we covered requesting resources from the scheduler earlier, how do we know how much and what type of resources we will need in the first place?
Answer: we don’t. Not until we’ve tried it ourselves at least once. We’ll need to benchmark our job and experiment with it before we know how much it needs in the way of resources.
The most effective way of figuring out how much resources a job needs is to submit a test job, and then ask the scheduler how many resources it used. A good rule of thumb is to ask the scheduler for more time and memory than your job can use. This value is typically two to three times what you think your job will need.
Resources for Computational Fluid Dynamics (CFD)
Copy the Python 2D CFD application from the course website to the HPC system using the following command:
[yourUsername@cirrus-login0 ~]$ wget https://archer-cse.github.io/2018-12-06-turing-hpcintro/files/cfd.tar.gzThen unpack it using
[yourUsername@cirrus-login0 ~]$ tar -xvf cfd.tar.gzCreate a job that runs the following commands in the directory containing the
cfd.pyprogram.module load anaconda/python2 python cfd.py 3 20000You’ll need to figure out a good amount of resources to ask for for this first “test run”. You might also want to have the scheduler email you to tell you when the job is done.
Hint: the job only needs 1 cpu and not too much time. The trick is figuring out just how much you’ll need!
Do not forget to check the
.efile produced by the job to make sure there are no errors! You should also check the.ofile produced by the job to make sure it contains the output from the CFD program.
Once the job completes (note that it takes much less time than expected), we can query the scheduler
to see how long our job took and what resources were used. We will use qstat -x to get statistics
about our job.
[yourUsername@cirrus-login0 ~]$ qstat -x -u yourUsername
indy2-login0:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
324396.indy2-lo user workq test1 57348 1 1 -- 00:01 F 00:00
324397.indy2-lo user workq test2 57456 1 1 -- 00:01 F 00:01
324401.indy2-lo user workq test3 58159 1 1 -- 00:00 F 00:00
324410.indy2-lo user workq test4 34027 1 1 -- 00:05 F 00:05
324418.indy2-lo user workq test5 35243 1 1 -- 00:05 F 00:01
This shows all the jobs we ran recently. To get detailed info about a job, we change our command slightly:
[yourUsername@cirrus-login0 ~]$ qstat -x -f 1965
It will show a lot of info, in fact, every single piece of info collected on your job by the
scheduler. It may be useful to redirect this information to less to make it easier to view (use
the left and right arrow keys to scroll through fields).
[yourUsername@cirrus-login0 ~]$ qstat -x -f 1965 | less
Some interesting fields include the following:
- exec_vnode - Where did your job run?
- resources_used.walltime - How long did the job take?
- comment - Any notes on success or errors in the job
- Output_Path - The file that stdout from the job was sent to
- Resource_List. - Set of resources requested by the job
- resources_used. - Set of resources used by the job
Measuring the statistics of currently running tasks
We can also check on stuff running on the login node right now.
top
The best way to check current system stats is with top.
Some sample output from my laptop might look like the following (Ctrl + c to exit):
top
op - 12:54:24 up 70 days, 12:36, 59 users, load average: 3.04, 2.78, 2.53
Tasks: 1526 total, 4 running, 1495 sleeping, 8 stopped, 19 zombie
%Cpu(s): 2.4 us, 1.6 sy, 0.0 ni, 95.8 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 26377216+total, 11843416+free, 10668532 used, 13466947+buff/cache
KiB Swap: 2097148 total, 105600 free, 1991548 used. 22326803+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21917 root 20 0 374324 233452 6584 R 55.6 0.1 308:13.19 pbs_server.bin
30680 marius 20 0 152436 20772 5872 R 17.8 0.0 0:00.08 cc1
27287 aturner 20 0 157312 3768 1600 R 8.9 0.0 0:00.59 top
30681 kfindlay 20 0 16744 2176 932 S 4.4 0.0 0:00.02 pbsnodes
2765 root 20 0 20940 32 0 S 2.2 0.0 5:59.78 aksusbd
7361 root 20 0 0 0 0 S 2.2 0.0 36:53.49 ptlrpcd_35
26386 hallen 20 0 4321956 123520 6740 S 2.2 0.0 0:03.81 conda
30830 pcerro 20 0 117344 1656 1312 S 2.2 0.0 0:05.70 deployer_ooj.sh
1 root 20 0 196108 3932 1644 S 0.0 0.0 82:49.29 systemd
2 root 20 0 0 0 0 S 0.0 0.0 6:14.69 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:06.40 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
8 root rt 0 0 0 0 S 0.0 0.0 0:46.32 migration/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4
Overview of the most important fields:
PID- What is the numerical id of each process?USER- Who started the process?RES- What is the amount of memory currently being used by a process (in bytes)?%CPU- How much of a CPU is each process using? Values higher than 100 percent indicate that a process is running in parallel.%MEM- What percent of system memory is a process using?TIME+- How much CPU time has a process used so far? Processes using 2 CPUs accumulate time at twice the normal rate.COMMAND- What command was used to launch a process?
free
Another useful tool is the free -h command. This will show the currently used/free amount of
memory.
$ free -h
total used free shared buff/cache available
Mem: 251G 10G 112G 4.2G 128G 212G
Swap: 2.0G 1.9G 103M
The key fields here are total, used, and available - which represent the amount of memory that the machine has in total, how much is currently being used, and how much is still available. When a computer runs out of memory it may attempt to use “swap” space on your hard drive instead. Swap space is very slow to access - a computer may appear to “freeze” if it runs out of memory and begins using swap. However, compute nodes on HPC systems usually have swap space disabled so when they run out of memory you usually get an “Out Of Memory (OOM)” error instead.
ps
To show all processes from your current session, type ps.
$ ps
PID TTY TIME CMD
15113 pts/5 00:00:00 bash
15218 pts/5 00:00:00 ps
Note that this will only show processes from our current session. To show all processes you own
(regardless of whether they are part of your current session or not), you can use ps ux.
$ ps ux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
auser 67780 0.0 0.0 149140 1724 pts/81 R+ 13:51 0:00 ps ux
auser 73083 0.0 0.0 142392 2136 ? S 12:50 0:00 sshd: auser@pts/81
auser 73087 0.0 0.0 114636 3312 pts/81 Ss 12:50 0:00 -bash
This is useful for identifying which processes are doing what.
Killing processes
To kill all of a certain type of process, you can run killall commandName. killall rsession
would kill all rsession processes created by RStudio, for instance. Note that you can only kill
your own processes.
You can also kill processes by their PIDs using kill 1234 where 1234 is a PID. Sometimes
however, killing a process does not work instantly. To kill the process in the most hardcore manner
possible, use the -9 flag. It’s recommended to kill using without -9 first. This gives a process
the chance to clean up child processes, and exit cleanly. However, if a process just isn’t
responding, use -9 to kill it instantly.
Key Points
The smaller your job, the faster it will schedule.