module

pipen.pipen

Main entry module, provide the Pipen class

Classes
  • Pipen — The Pipen class provides interface to assemble and run the pipeline</>
Functions
  • async_run(name, starts, data, desc, outdir, profile, **kwargs) (bool) — Shortcut to run a pipeline</>
  • run(name, starts, data, desc, outdir, profile, **kwargs) (bool) — Shortcut to run a pipeline</>
class

pipen.pipen.Pipen(name=None, desc=None, outdir=None, **kwargs)

The Pipen class provides interface to assemble and run the pipeline

Attributes
  • PIPELINE_COUNT — How many pipelines are loaded
  • SETUP — Whether the one-time setup hook is called
  • _kwargs — The extra configrations passed to overwrite the default ones
  • config — The configurations
  • desc — The description of the pipeline
  • name — The name of the pipeline
  • outdir — The output directory of the results
  • pbar — The progress bar
  • procs — The processes
  • profile — The profile of the configurations to run the pipeline
  • starts — The start processes
  • workdir — The workdir for the pipeline
Parameters
  • name (str | none, optional) — The name of the pipeline
  • desc (str | none, optional) — The description of the pipeline
  • outdir (str | pathlib.path | none, optional) — The output directory of the results
  • **kwargs — Other configurations
Methods
  • __init_subclass__() — This method is called when a class is subclassed.</>
  • async_run(profile) (bool) — Run the processes one by one</>
  • build_proc_relationships() — Build the proc relationships for the pipeline</>
  • run(profile) (bool) — Run the pipeline with the given profileThis is just a sync wrapper for the async async_run function using asyncio.run() </>
  • set_data(*args, **kwargs) (Pipen) — Set the input_data for start processes</>
  • set_starts(*procs, clear) — Set the starts</>
classmethod

__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

method

async_run(profile='default')

Run the processes one by one

Parameters
  • profile (str, optional) — The default profile to use for the run
Returns (bool)

True if the pipeline ends successfully else False

method

run(profile='default')

Run the pipeline with the given profileThis is just a sync wrapper for the async async_run function using asyncio.run()

Parameters
  • profile (str, optional) — The default profile to use for the run
Returns (bool)

True if the pipeline ends successfully else False

method

set_data(*args, **kwargs)

Set the input_data for start processes

Parameters
  • *args (Any) — The unnamed input data for the start processes
  • **kwargs (Any) — The named input data for the start processesThe data will be distributed to the start processes like a python function is called. The function is like def func(start1, start2, ...), where startN is the name of the N-th start process. set_data(data1, start2=data2, ...) will set data1 to the first start process, and data2 to the start process named start2. set_data(data1, start1=data2, ...) will error, since multiple data are set for the same start process start1.
Raises
  • PipenSetDataError — When the number of input data does not matchthe number of start processes
Returns (Pipen)

self to chain the operations

method

set_starts(*procs, clear=True)

Set the starts

Parameters
  • *procs (Union) — The processes to set as starts of the pipeline.
  • clear (bool, optional) — Wether to clear previous set starts
Raises
  • ProcDependencyError — When processes set as starts repeatedly
Returns

self to chain the operations

method

build_proc_relationships()

Build the proc relationships for the pipeline

function

pipen.pipen.async_run(name, starts, data=None, desc=None, outdir=None, profile='default', **kwargs)

Shortcut to run a pipeline

Parameters
  • name (str) — The name of the pipeline
  • starts (Union) — The start processes
  • data (Iterable, optional) — The input data for the start processes
  • desc (str, optional) — The description of the pipeline
  • outdir (str | pathlib.path | none, optional) — The output directory of the results
  • profile (str, optional) — The profile to use
  • **kwargs — Other options pass to Pipen to create the pipeline
Returns (bool)

True if the pipeline ends successfully else False

function

pipen.pipen.run(name, starts, data=None, desc=None, outdir=None, profile='default', **kwargs)

Shortcut to run a pipeline

Parameters
  • name (str) — The name of the pipeline
  • starts (Union) — The start processes
  • data (Optional, optional) — The input data for the start processes
  • desc (str | none, optional) — The description of the pipeline
  • outdir (str | pathlib.path | none, optional) — The output directory of the results
  • profile (str, optional) — The profile to use
  • **kwargs — Other options pass to Pipen to create the pipeline
Returns (bool)

True if the pipeline ends successfully else False