module

pipen.pipen

</>

Main entry module, provide the Pipen class

Classes
  • Pipen — The Pipen class provides interface to assemble and run the pipeline</>
Functions
  • run(name, starts, data, desc, outdir, profile, **kwargs) (bool) — Shortcut to run a pipeline</>
class

pipen.pipen.Pipen(name=None, desc=None, outdir=None, **kwargs)

</>

The Pipen class provides interface to assemble and run the pipeline

Attributes
  • PIPELINE_COUNT — How many pipelines are loaded
  • SETUP — Whether the one-time setup hook is called
  • _kwargs — The extra configrations passed to overwrite the default ones
  • config — The configurations
  • desc — The description of the pipeline
  • name — The name of the pipeline
  • outdir — The output directory of the results
  • pbar — The progress bar
  • procs — The processes
  • profile — The profile of the configurations to run the pipeline
  • starts — The start processes
  • workdir — The workdir for the pipeline
Parameters
  • name (str | none, optional) — The name of the pipeline
  • desc (str | none, optional) — The description of the pipeline
  • outdir (str | os.pathlike, optional) — The output directory of the results
  • **kwargs — Other configurations
Methods
  • __init_subclass__() — This method is called when a class is subclassed.</>
  • async_run(profile) (bool) — Run the processes one by one</>
  • build_proc_relationships() — Build the proc relationships for the pipeline</>
  • run(profile) (bool) — Run the pipeline with the given profileThis is just a sync wrapper for the async async_run function using asyncio.run() </>
  • set_data(*indata) (Pipen) — Set the input_data for start processes</>
  • set_starts(*procs, clear) — Set the starts</>
classmethod

__init_subclass__()

</>

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

method

async_run(profile='default')

</>

Run the processes one by one

Parameters
  • profile (str, optional) — The default profile to use for the run
Returns (bool)

True if the pipeline ends successfully else False

method

run(profile='default')

</>

Run the pipeline with the given profileThis is just a sync wrapper for the async async_run function using asyncio.run()

Parameters
  • profile (str, optional) — The default profile to use for the run
Returns (bool)

True if the pipeline ends successfully else False

method

set_data(*indata)

</>

Set the input_data for start processes

Parameters
  • *indata (Any) — The input data for the start processesThe data will set for the processes in the order determined by set_starts(). If a process has input_data set, an error will be raised. To use that input_data, set None here in the corresponding position for the process
Raises
  • ProcInputDataError — When trying to set input data toprocesses with input_data already set
Returns (Pipen)

self to chain the operations

method

set_starts(*procs, clear=True)

</>

Set the starts

Parameters
  • *procs (Union) — The processes to set as starts of the pipeline.
  • clear (bool, optional) — Wether to clear previous set starts
Raises
  • ProcDependencyError — When processes set as starts repeatedly
Returns

self to chain the operations

method

build_proc_relationships()

</>

Build the proc relationships for the pipeline

function

pipen.pipen.run(name, starts, data=None, desc=None, outdir=None, profile='default', **kwargs)

</>

Shortcut to run a pipeline

Parameters
  • name (str) — The name of the pipeline
  • starts (Union) — The start processes
  • data (Iterable, optional) — The input data for the start processes
  • desc (str, optional) — The description of the pipeline
  • outdir (PathType | None, optional) — The output directory of the results
  • profile (str, optional) — The profile to use
  • **kwargs — Other options pass to Pipen to create the pipeline
Returns (bool)

True if the pipeline ends successfully else False