Job caching

If cache set to False (detected in the sequence of configuration files, Pipen constructor, and process definition), the job is running anyway regardless of previous runs.

If a previous run of a job fails, the job will be running anyway.

If a job is done successfully, a signature file will be generated for the job. When we try to run the job again, the signature will be used to check if we can skip running the job again but to use the results generated by previous run.

We can also do a force-cache for a job by setting cache to "force". This make sure of the results of previous successful run regardless of input or script changes. This is useful for the cases that, for example, you make some changes to input/script, but you don't want them to take effect immediately, especially when the job takes long time to run.

Job signature

The signature of a job (job.signature.toml) consists of the input types and data, the output types and data, and ctime — the maximum modification time over the rendered job script, the declared file/directory inputs and the declared outputs. Validation is modification-time based: pipen compares mtimes and never hashes file contents. So these situations will make job-cache checking fail (job will start over):

  1. Any changes in input or output types
  2. Any changes in input or output data
  3. Any changes to script
  4. Any touches to input files — including a bare touch with no content change, which invalidates exactly the same jobs as editing the file's content (only the mtime is compared)
  5. Any touches to input directories
  6. Use dirsig as the depth to check the files under the directories
  7. Otherwise if it is 0, only the directories themselves are checked. Note that modify a file inside a directory may not change the last modified time of the directory itself.

  8. Any deletions to the output files/directories Note that only the files/directories specified by output are checked. Files or subdirectories in the output directories will NOT be checked.

  9. Because only mtimes are compared, touching a declared input invalidates the same job scope as a real content edit — the file's bytes are never looked at.

  10. ctime also covers the rendered job script, so a value interpolated into the script that is not a modelled input (for example an absolute path taken from envs) invalidates every job of that process when it changes, even though no declared input or output changed.