Introduction

Documentation | ChangeLog | Examples | API
Features
- Easy to use
- Nearly zero-configuration
- Nice logging
- Highly extendable
- Cloud support naively
Installation
pip install -U pipen
Quickstart
example.py
from pipen import Proc, Pipen, run
class P1(Proc):
"""Sort input file"""
input = "infile"
input_data = ["/tmp/data.txt"]
output = "outfile:file:intermediate.txt"
script = "cat {{in.infile}} | sort > {{out.outfile}}"
class P2(Proc):
"""Paste line number"""
requires = P1
input = "infile:file"
output = "outfile:file:result.txt"
script = "paste <(seq 1 3) {{in.infile}} > {{out.outfile}}"
# class MyPipeline(Pipen):
# starts = P1
if __name__ == "__main__":
# MyPipeline().run()
run("MyPipeline", starts=P1)
> echo -e "3\n2\n1" > /tmp/data.txt
> python example.py
04-17 16:19:35 I core _____________________________________ __
04-17 16:19:35 I core ___ __ \___ _/__ __ \__ ____/__ | / /
04-17 16:19:35 I core __ /_/ /__ / __ /_/ /_ __/ __ |/ /
04-17 16:19:35 I core _ ____/__/ / _ ____/_ /___ _ /| /
04-17 16:19:35 I core /_/ /___/ /_/ /_____/ /_/ |_/
04-17 16:19:35 I core
04-17 16:19:35 I core version: 0.17.3
04-17 16:19:35 I core
04-17 16:19:35 I core ╔═══════════════════════════ MYPIPELINE ════════════════════════════╗
04-17 16:19:35 I core ║ My pipeline ║
04-17 16:19:35 I core ╚═══════════════════════════════════════════════════════════════════╝
04-17 16:19:35 I core plugins : verbose v0.14.1
04-17 16:19:35 I core # procs : 2
04-17 16:19:35 I core profile : default
04-17 16:19:35 I core outdir :
/home/pwwang/github/pipen/examples/MyPipeline-output
04-17 16:19:35 I core cache : True
04-17 16:19:35 I core dirsig : 1
04-17 16:19:35 I core error_strategy : ignore
04-17 16:19:35 I core forks : 1
04-17 16:19:35 I core lang : bash
04-17 16:19:35 I core loglevel : info
04-17 16:19:35 I core num_retries : 3
04-17 16:19:35 I core scheduler : local
04-17 16:19:35 I core submission_batch: 8
04-17 16:19:35 I core template : liquid
04-17 16:19:35 I core workdir :
/home/pwwang/github/pipen/examples/.pipen/MyPipeline
04-17 16:19:35 I core plugin_opts :
04-17 16:19:35 I core template_opts : filters={'realpath': <function realpath at
0x7fc3eba12...
04-17 16:19:35 I core : globals={'realpath': <function realpath at
0x7fc3eba12...
04-17 16:19:35 I core Initializing plugins ...
04-17 16:19:36 I core
04-17 16:19:36 I core ╭─────────────────────────────── P1 ────────────────────────────────╮
04-17 16:19:36 I core │ Sort input file │
04-17 16:19:36 I core ╰───────────────────────────────────────────────────────────────────╯
04-17 16:19:36 I core P1: Workdir:
'/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1'
04-17 16:19:36 I core P1: <<< [START]
04-17 16:19:36 I core P1: >>> ['P2']
04-17 16:19:36 I verbose P1: in.infile: /tmp/data.txt
04-17 16:19:36 I verbose P1: out.outfile:
/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1/0/output/intermediate
.txt
04-17 16:19:38 I verbose P1: Time elapsed: 00:00:02.051s
04-17 16:19:38 I core
04-17 16:19:38 I core ╭═══════════════════════════════ P2 ════════════════════════════════╮
04-17 16:19:38 I core ║ Paste line number ║
04-17 16:19:38 I core ╰═══════════════════════════════════════════════════════════════════╯
04-17 16:19:38 I core P2: Workdir:
'/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P2'
04-17 16:19:38 I core P2: <<< ['P1']
04-17 16:19:38 I core P2: >>> [END]
04-17 16:19:38 I verbose P2: in.infile:
/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1/0/output/intermediate
.txt
04-17 16:19:38 I verbose P2: out.outfile:
/home/pwwang/github/pipen/examples/MyPipeline-output/P2/result.txt
04-17 16:19:41 I verbose P2: Time elapsed: 00:00:02.051s
04-17 16:19:41 I core
MYPIPELINE: 100%|██████████████████████████████| 2/2 [00:06<00:00, 0.35 procs/s]
> cat ./MyPipeline-output/P2/result.txt
1 1
2 2
3 3
Examples
See more examples at examples/
and a more realcase example at:
https://github.com/pwwang/pipen-report/tree/master/example
Plugin gallery
Plugins make pipen
even better.
pipen-annotate
: Use docstring to annotate pipen processespipen-args
: Command line argument parser for pipenpipen-board
: Visualize configuration and running of pipen pipelines on the webpipen-diagram
: Draw pipeline diagrams for pipenpipen-dry
: Dry runner for pipen pipelinespipen-filters
: Add a set of useful filters for pipen templates.pipen-lock
: Process lock for pipen to prevent multiple runs at the same time.pipen-log2file
: Save running logs to file for pipenpipen-poplog
: Populate logs from jobs to running log of the pipelinepipen-report
: Generate report for pipenpipen-runinfo
: Save running information to file for pipenpipen-verbose
: Add verbosal information in logs for pipen.pipen-gcs
: A plugin for pipen to handle files in Google Cloud Storage.pipen-cli-init
: A pipen CLI plugin to create a pipen project (pipeline)pipen-cli-ref
: Make reference documentation for processespipen-cli-require
: A pipen cli plugin check the requirements of a pipelinepipen-cli-run
: A pipen cli plugin to run a process or a pipeline