sanitize paths and do not use rsync --remove-source-files
Make sure all paths are quoted in the generated cluster batch scripts (*.p
files), and perform some additional sanity checks to avoid rsync-ing on paths that are obviously wrong. If users would input spaces at unexpected places this could lead to unintended side affects such as destruction of data. Mainly wetb/prepost/Simulations.py and wetb/prepost/simchunks.py are affected.
Please limit comments and constructive feedback in this issue regarding a more robust way of handling user defined paths. Architectural changes to the toolbox are planned (with a complete revised template engine for the bash script generation) but although very interesting, should take place elsewhere.
Update: --remove-source-files
is required in order to prevent the node scratch disk from running out of space if a users performs a decent amount of simulations. Due to the context (moving files one directory level up on the scratch disk) of that command an additional check on the path can be done to make sure it does not escape the context of the scratch directory.