Skip to content

pipen.pluginmgr

module

pipen.pluginmgr

</>

Define hooks specifications and provide plugin manager

Classes
Functions
function

pipen.pluginmgr.on_setup(config)

</>

Setup for plugins, primarily used for the plugins tosetup some default configurations.

This is only called once for all pipelines.

Parameters
  • config (Dict) — The configuration dictionaryplugin options should be defined under "plugin_opts" One should define a configuration item either with a prefix as the identity for the plugin or a namespace inside the plugin config
function

pipen.pluginmgr.on_init(pipen)

</>

When the pipeline is initialized, and default configs are loaded

Parameters
  • pipen (Pipen) — The Pipen object
function

pipen.pluginmgr.on_start(pipen)

</>

Right before the pipeline starts running.

Process relationships are inferred.

Parameters
  • pipen (Pipen) — The Pipen object
function

pipen.pluginmgr.on_complete(pipen, succeeded)

</>

The the pipeline is completed.

Parameters
  • pipen (Pipen) — The Pipen object
  • succeeded (bool) — Whether the pipeline has successfully completed.
function

pipen.pluginmgr.on_proc_create(proc)

</>

Called Proc constructor when a process is created.

Enables plugins to modify the default attributes of processes

Parameters
  • proc (Proc) — The Proc object
function

pipen.pluginmgr.on_proc_init(proc)

</>

Called when a process is initialized.

Allows plugins to modify the process attributes after initialization, but before the jobs are initialized.

Parameters
  • proc (Proc) — The Proc object
function

pipen.pluginmgr.on_proc_input_computed(proc)

</>

Called after process input data is computed.

Parameters
  • proc (Proc) — The Proc object
function

pipen.pluginmgr.on_proc_script_computed(proc)

</>

Called after process script is computed.

The script is computed as a string that is about to compiled into a template.

Parameters
  • proc (Proc) — The Proc object
function

pipen.pluginmgr.on_proc_start(proc)

</>

When a process is starting

Parameters
  • proc (Proc) — The process
function

pipen.pluginmgr.on_proc_shutdown(proc, sig)

</>

When pipeline is shutting down, by Ctrl-c for example.

Return False to stop shutting down, but you have to shut it down by yourself, for example, proc.xqute.task.cancel()

Only the first return value will be used.

Parameters
  • sig (signal.Signals) — The signal. None means a natural shutdown
  • pipen — The xqute object
function

pipen.pluginmgr.on_proc_done(proc, succeeded)

</>

When a process is done

Parameters
  • proc (Proc) — The process
  • succeeded (bool | str) — Whether the process succeeded or not. 'cached' if all jobsare cached.
function

pipen.pluginmgr.on_job_init(job)

</>

When a job is initialized

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_queued(job)

</>

When a job is queued in xqute. Note it might not be queued yet inthe scheduler system.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_submitting(job)

</>

When a job is submitting.

The first plugin (based on priority) have this hook return False will cancel the submission

Parameters
  • job (Job) — The job
Returns (bool)

False to cancel submission

function

pipen.pluginmgr.on_job_submitted(job)

</>

When a job is submitted in the scheduler system.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_started(job)

</>

When a job starts to run in then scheduler system.

Note that the job might not be running yet in the scheduler system.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_polling(job)

</>

When status of a job is being polled.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_killing(job)

</>

When a job is being killed.

The first plugin (based on priority) have this hook return False will cancel the killing

Parameters
  • job (Job) — The job
Returns (bool)

False to cancel killing

function

pipen.pluginmgr.on_job_killed(job)

</>

When a job is killed

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_succeeded(job)

</>

When a job completes successfully.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_cached(job)

</>

When a job is cached.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.on_job_failed(job)

</>

When a job is done but failed.

Parameters
  • job (Job) — The job
function

pipen.pluginmgr.norm_inpath(job, inpath, is_dir)

</>

Normalize the input path

Parameters
  • job (Job) — The job
  • inpath (str | os.pathlike) — The input path
  • is_dir (bool) — Whether the path is a directory
Returns (str)

The normalized path

function

pipen.pluginmgr.norm_outpath(job, outpath, is_dir)

</>

Normalize the output path

Parameters
  • job (Job) — The job
  • outpath (str) — The output path
  • is_dir (bool) — Whether the path is a directory
Returns (str)

The normalized path

function

pipen.pluginmgr.get_mtime(job, path, dirsig)

</>

Get the mtime of a path, either a file or a directory

Parameters
  • job (Job) — The job
  • path (str | os.pathlike) — The path to get mtime
  • dirsig (int) — The depth of the directory to check the last modification time
Returns (float)

The last modification time

function

pipen.pluginmgr.clear_path(job, path, is_dir)

</>

Clear the path, either a file or a directory

Parameters
  • job (Job) — The job
  • path (str | os.pathlike) — The path to clear
  • is_dir (bool) — Whether the path is a directory
Returns (bool)

Whether the path is cleared successfully

function

pipen.pluginmgr.output_exists(job, path, is_dir)

</>

Check if the output exists

Parameters
  • job (Job) — The job
  • path (str) — The path to check
  • is_dir (bool) — Whether the path is a directory
Returns (bool)

Whether the output exists

function

pipen.pluginmgr.on_jobcmd_init(job)

</>

When the job command wrapper script is initialized before the prescript is run

This should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available: status and job. status is the class JobStatus from xqute.defaults.py and job is the Job instance.

For multiple plugins, the code will be inserted in the order of the plugin priority.

The code will replace the #![jobcmd_init] placeholder in the wrapped job script. See also https://github.com/pwwang/xqute/blob/master/xqute/defaults.py#L95

Parameters
  • job (Job) — The job object
Returns (str)

The bash code to be inserted

function

pipen.pluginmgr.on_jobcmd_prep(job)

</>

When the job command right about to be run

This should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available: status and job. status is the class JobStatus from xqute.defaults.py and job is the Job instance.

The bash variable $cmd is accessible in the context. It is also possible to modify the cmd variable. Just remember to assign the modified value to cmd.

For multiple plugins, the code will be inserted in the order of the plugin priority. Keep in mind that the $cmd may be modified by other plugins.

The code will replace the #![jobcmd_prep] placeholder in the wrapped job script. See also https://github.com/pwwang/xqute/blob/master/xqute/defaults.py#L95

Parameters
  • job (Job) — The job object
Returns (str)

The bash code to be inserted

function

pipen.pluginmgr.on_jobcmd_end(job)

</>

When the job command finishes and after the postscript is run

This should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available: status and job. status is the class JobStatus from xqute.defaults.py and job is the Job instance.

The bash variable $rc is accessible in the context, which is the return code of the job command.

For multiple plugins, the code will be inserted in the order of the plugin priority.

The code will replace the #![jobcmd_end] placeholder in the wrapped job script. See also https://github.com/pwwang/xqute/blob/master/xqute/defaults.py#L95

Parameters
  • job (Job) — The job object
Returns (str)

The bash code to be inserted

class

pipen.pluginmgr.PipenMainPlugin()

</>

The builtin core plugin, used to update the progress bar andcache the job

class

pipen.pluginmgr.XqutePipenPlugin()

</>

The plugin for xqute working as proxy for pipen plugin hooks