Wednesday, April 30, 2014

Using qselect to make job management easy

How many times have you wanted to delete all your queued jobs, but not your running ones?  What about putting holds on all your queued jobs so that the next job you submit jumps them all in line?

There is an easy way to do this.  A combination of qselect to filter your jobs, and subshells makes this a breeze.  For the most recent qselect documentation run man qselect on the login node.

Below is a selection of example qselects. All the options can be combined to make a very flexible set of job selection.

#all jobs for a user|s
qselect -u user
qselect -u user1,user2
#all jobs owned by me in a given PBS account
qselect -u $USER -A account_flux
#all jobs in state Q
# options E: exiting H: held Q: queued R: Running T:trasiting W: Waiting
qselect -s Q
#all jobs Queued and Held
qselect -s QH
#all jobs with walltimes under 24 hours
qselect -l walltime.lt.24:00:00
view raw qselect.sh hosted with ❤ by GitHub


Now that qselect has given us all the job id's we want we can use the sub shell.  A subshell, evaluates the command in the subshell (our qselect) first and takes the output of that command and feeds it to the second.  To create a subshell,  wrap the command you want to run first in $( ). The options below is some of our most common requests.
#delete all my jobs,
# only do this if you have fewer than a few hundred jobs.
qdel $(qselect -u $USER)
#delete all my queued jobs
qdel $(qselect -u $USER -s Q)
#delete all my jobs submiting with the wrong account
qdel $(qselect -u $USER -A wrongaccount)
view raw qselect-qdel.sh hosted with ❤ by GitHub

Users need not stop here, combine qselect to hold all your jobs so that other jobs from your Flux account can run.  In this case assume you have jobs queued in two Flux accounts, and I only want todo this in one of them.

#hold all my queued jobs in account_flux
qhold $(qselect -A account_flux -u $USER -s Q)
#release the holds so they can now run
qrls $(qselect -A account_flux -u $USER -s H)


Qselect should make your life very easy for mass job changes. There are a few commands (qmove for one) that does not take a list of jobs and thus this method wont work. Contact us if you are in need of one of these commands.