Skip to content
Snippets Groups Projects
Commit 91e64eb1 authored by David Verelst's avatar David Verelst
Browse files

updated how-to-dlcs docs

parent 4756e669
No related branches found
No related tags found
No related merge requests found
Pipeline #
...@@ -389,24 +389,53 @@ number of cpu's requested (using ```-c``` or ```--nr_cpus```) and minimum ...@@ -389,24 +389,53 @@ number of cpu's requested (using ```-c``` or ```--nr_cpus```) and minimum
of required free cpu's on the cluster (using ```--cpu_free```, 48 by default). of required free cpu's on the cluster (using ```--cpu_free```, 48 by default).
Jobs will be launched after a predefined sleep time (as set by the Jobs will be launched after a predefined sleep time (as set by the
```--tsleep``` option, and set to 5 seconds by default). After the initial sleep ```--tsleep``` option, and set to 5 seconds by default). After the initial sleep
time a new job will be launched every 0.1 second. If the launch condition is not time a new job will be launched every 0.5 second. If the launch condition is not
met (```nr_cpus > cpu's used by user AND cpu's free on cluster > cpu_free```), met:
the program will wait 5 seconds before trying to launch a new job again.
```
nr_cpus > cpu's used by user
AND cpu's free on cluster > cpu_free
AND jobs queued by user < cpu_user_queue)
```
the program will sleep 5 seconds before trying to launch a new job again.
Depending on the amount of jobs and the required computation time, it could Depending on the amount of jobs and the required computation time, it could
take a while before all jobs are launched. When running the launch script from take a while before all jobs are launched. When running the launch script from
the login node, this might be a problem when you have to close your ssh/putty the login node, this might be a problem when you have to close your ssh/putty
session before all jobs are launched. In that case the user should use a session before all jobs are launched. In that case the user can use the
dedicated compute node for launching jobs. To run the launch script on a ```--crontab``` argument: it will trigger the ```launch.py``` script every 5
compute instead of the login node, use the ```--node``` option. You can inspect minutes to check if more jobs can be launched until all jobs have been
the progress in the ```launch_scheduler_log.txt``` file. executed. The user does not need to have an active ssh/putty session for this to
work. You can follow the progress and configuration of ```launch.py``` in
crontab mode in the following files:
* ```launch_scheduler_log.txt```
* ```launch_scheduler_config.txt```: you can change your launch settings on the fly
* ```launch_scheduler_state.txt```
* ```launch_pbs_filelist.txt```: remaining jobs, when a job is launched it is
removed from this list
You can check if ```launch.py``` is actually active as a crontab job with:
```
crontab -l
```
```launch.py``` will clean-up the crontab after all jobs are launched, but if
you need to prevent it from launching new jobs before that, you can clean up your
crontab with:
```
crontab -r
```
The ```launch.py``` script has some different options, and you can read about The ```launch.py``` script has various different options, and you can read about
them by using the help function (the output is included for your convenience): them by using the help function (the output is included for your convenience):
```bash ```bash
g-000 $ launch.py --help g-000 $ launch.py --help
usage: Usage:
launch.py -n nr_cpus launch.py -n nr_cpus
...@@ -414,7 +443,7 @@ launch.py --crontab when running a single iteration of launch.py as a crontab jo ...@@ -414,7 +443,7 @@ launch.py --crontab when running a single iteration of launch.py as a crontab jo
File list is read from "launch_pbs_filelist.txt", and the configuration can be changed on the fly File list is read from "launch_pbs_filelist.txt", and the configuration can be changed on the fly
by editing the file "launch_scheduler_config.txt". by editing the file "launch_scheduler_config.txt".
options: Options:
-h, --help show this help message and exit -h, --help show this help message and exit
--depend Switch on for launch depend method --depend Switch on for launch depend method
-n NR_CPUS, --nr_cpus=NR_CPUS -n NR_CPUS, --nr_cpus=NR_CPUS
...@@ -425,7 +454,11 @@ options: ...@@ -425,7 +454,11 @@ options:
full pbs file path. Escape backslashes! By default it full pbs file path. Escape backslashes! By default it
will select all *.p files in pbs_in/. will select all *.p files in pbs_in/.
--dry dry run: do not alter pbs files, do not launch --dry dry run: do not alter pbs files, do not launch
--tsleep=TSLEEP Sleep time [s] after qsub command. Default=5 seconds --tsleep=TSLEEP Sleep time [s] when cluster is too bussy to launch new
jobs. Default=5 seconds
--tsleep_short=TSLEEP_SHORT
Sleep time [s] between between successive job
launches. Default=0.5 seconds.
--logfile=LOGFILE Save output to file. --logfile=LOGFILE Save output to file.
-c, --cache If on, files are read from cache -c, --cache If on, files are read from cache
--cpu_free=CPU_FREE No more jobs will be launched when the cluster does --cpu_free=CPU_FREE No more jobs will be launched when the cluster does
...@@ -436,15 +469,19 @@ options: ...@@ -436,15 +469,19 @@ options:
No more jobs will be launched after having No more jobs will be launched after having
cpu_user_queue number of jobs in the queue. This cpu_user_queue number of jobs in the queue. This
prevents users from filling the queue, while still prevents users from filling the queue, while still
allowing to aim for a high cpu_free target. allowing to aim for a high cpu_free target. Default=5
Default=500
--qsub_cmd=QSUB_CMD Is set automatically by --node flag --qsub_cmd=QSUB_CMD Is set automatically by --node flag
--node If executed on dedicated node. Although this works, --node If executed on dedicated node. Although this works,
consider using --crontab instead. Default=False consider using --crontab instead. Default=False
--sort Sort pbs file list. Default=False --sort Sort pbs file list. Default=False
--crontab Crontab mode. Implies --cache, and not compatible with --crontab Crontab mode: %prog will check every 5 (default)
--node. When all jobs are done, crontab -r will remove minutes if more jobs can be launched. Not compatible
all the user's current crontab jobs. Default=False with --node. When all jobs are done, crontab -r will
remove all existing crontab jobs of the current user.
Use crontab -l to inspect current crontab jobs, and
edit them with crontab -e. Default=False
--every_min=EVERY_MIN
Crontab update interval in minutes. Default=5
--debug Debug print statements. Default=False --debug Debug print statements. Default=False
``` ```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment