module

biopipen.ns.splitpipe

</>

Split-pipe processes for Parse Biosciences split-pool data

Classes
class

biopipen.ns.splitpipe.SplitPipeRef(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Build an indexed reference genome for split-pipe

Runs split-pipe --mode mkref, which is required before any data can be processed. The output of this process is used by SplitPipeCount with envs.genome_dir. Requires the Parse Biosciences Pipeline (split-pipe) v1.9.0+.

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • output_flatten — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously.The program entrance for some schedulers may take too much resources when submitting a job or checking the job status. So we may use a smaller number here to limit the simultaneous submissions.
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • fasta — The input reference sequence file(s) in fasta formatMultiple files are allowed, e.g. for a species-mix reference. They should correspond to envs.genome_name and in.genes in the same order.
  • genes — The input reference gene annotation file(s) in GTF formatShould correspond to envs.genome_name and in.fasta in the same order.
Output
  • outdir — The output directory, named by envs.genome_name
Envs
  • — Other environment variables required by split-pipe --mode mkrefSee https://support.parsebiosciences.com/hc/en-us/articles/23060102930580 for more details.
  • genome_name (type=list) — The name(s) of the genome(s), e.g. hg38, or["hg38", "GRCm39"] for a species-mix reference. When multiple names are given, the output directory is named after them joined by _.
  • ncores — Alias for nthreads. Takes precedence when both are given.
  • nthreads — Number of threads to use
  • parfile — Path to the parameter file with one NAME VALUE entry perline. Required for small genomes, for which STAR rejects its default genomeSAindexNbases (14): STAR's options are passed through the mkref_star_args parameter as a comma-separated list, e.g. mkref_star_args --genomeSAindexNbases,8,--genomeSAsparseD,2.
  • splitpipe — Path to the split-pipe executable
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, output_flatten, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Init all other properties and jobs</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, output_flatten=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str | none, optional) — The new name of the process
  • desc (str | none, optional) — The new description of the process
  • envs (Optional, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int | none, optional) — How deep to update the envs when subclassed.
  • cache (bool | none, optional) — Whether we should check the cache for the jobs
  • export (bool | none, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • output_flatten (bool | none, optional) — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • error_strategy (str | none, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • num_retries (int | none, optional) — How many times to retry to jobs once error occurs
  • forks (int | none, optional) — New forks for the new process
  • input_data (any | none, optional) — The input data for the process. Only when this processis a start process
  • order (int | none, optional) — The order to execute the new process
  • plugin_opts (Optional, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Optional, optional) — The required processes for the new process
  • scheduler (str | none, optional) — The new shedular to run the new process
  • scheduler_opts (Optional, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int | none, optional) — How many jobs to be submited simultaneously.
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

run()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
class

biopipen.ns.splitpipe.SplitPipeCount(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Run split-pipe on one sublibrary

Runs split-pipe --mode all to pre-process, align and post-process the sequencing data of a single sublibrary, and to generate the expression matrices and the report for it. Requires the Parse Biosciences Pipeline (split-pipe) v1.9.0+.

Terminology: a "sublibrary" is one well-pool of a kit processed by --mode all (i.e. the fastq files of one sample well-pool of one sequencing run). Sublibraries are always processed independently and must never be merged at this step: fastq files from multiple sequencing lanes of the SAME sublibrary are passed together as a list to in.fq1, while fastq files of DIFFERENT sublibraries belong in different jobs of this process. Sublibraries are merged into a single dataset afterwards, using SplitPipeCombine (--mode comb).

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • output_flatten — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously.The program entrance for some schedulers may take too much resources when submitting a job or checking the job status. So we may use a smaller number here to limit the simultaneous submissions.
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • fq1 — The input fastq (R1) file(s) of one sublibraryMultiple files are allowed when the sublibrary was sequenced over multiple lanes, and they are passed together to --fq1.
  • fq2 — The input fastq (R2) file(s) of the sublibraryMultiple files are allowed when the sublibrary was sequenced over multiple lanes, and they are passed together to --fq2. If not given, the R2 file names are derived from the R1 ones by split-pipe, which requires the R1 and R2 files to differ only by R1/R2 in their names.
  • id — The id defining the output directory. If not provided, it isinferred from the fq1 file names, e.g. s1_S1_R1_001.fastq.gz gives s1.
Output
  • outdir — The output directory
Envs
  • — Other environment variables required by split-pipe --mode allSee https://support.parsebiosciences.com/hc/en-us/articles/23060220494228 for more details.
  • bcr_analysis (flag) — Whether to run BCR analysis.
  • chemistry — The chemistry version of the kit, e.g. v1, v2, v3 orv4. Required by split-pipe.
  • crispr (flag) — Whether this is a CRISPR Detect sublibrary. It must beused together with envs.crsp_guides and envs.parent_dir.
  • crsp_guides — Path to the csv file with the guide RNA sequences forCRISPR Detect data.
  • crsp_read_thresh (type=int) — The guide_read_cutoff for CRISPR Detectdata. The default is 3.
  • crsp_tscp_thresh (type=int) — The guide_tscp_cutoff for CRISPR Detectdata. The default is 5.
  • ffpe (flag) — Whether this is an Evercode WT FFPE sublibrary.Only v4 chemistry is supported for FFPE data.
  • genome_dir — Path to the indexed genome directory built bySplitPipeRef (--mode mkref). It can also be passed by the input channel, see SplitPipePipeline.
  • immune_genome — The immune genome to align to for TCR/BCR analysis.human and mouse are supported for TCR, and human, mouse and transgenic_mouse for BCR analysis.
  • kit — The kit name, e.g. WT, WT_mini, WT_mega, WT_mega_384,WT_penta, WT_penta_384, MAX_WT_S, MAX_WT_M, MAX_WT_L or MAX_WT_L_384. Required by split-pipe.
  • min_map_fraction (type=float) — The minimum mapping fraction of thereads to be retained as transcripts. Parse documents this parameter as post_min_map_frac in the --parfile parameter file, so a parameter file is generated for this run; it cannot be used together with envs.parfile.
  • ncores — Alias for nthreads. Takes precedence when both are given.
  • nthreads — Number of threads to use
  • parent_dir — Path to the whole transcriptome (parent) output directoryof the sublibrary. If provided, the parent results are used to filter the cells of the TCR/BCR/CRISPR analysis.
  • parfile — Path to the parameter file with one NAME VALUE entry perline, e.g. post_min_map_frac 0.01.
  • samp_list — Path to the sample specification list (text) file, with oneNAME WELLS entry per line.
  • samp_slt — Alias of samp_sltab in Parse's own documentation, whereboth spellings are used for the SampleLoadingTable. Don't set both of them.
  • samp_sltab — Path to the SampleLoadingTable (excel) file from theSupport Suite.
  • samples (type=list) — The sample definitions, as a list of NAME WELLSentries, e.g. ["xcond_1 A1-A6", "xcond_2 C1-C6"]. Each entry is passed to split-pipe as one --sample NAME WELLS argument. If omitted, the pipeline processes all samples (all-sample).
  • splitpipe — Path to the split-pipe executable
  • targeted_list — Path to the target gene list (csv) for Gene Select data.
  • tcr_analysis (flag) — Whether to run TCR analysis. Immune databases musthave been installed with the pipeline, see split-pipe --immune_check.
  • use_imgt_db (flag) — Whether to use the IMGT reference for theTCR/BCR analysis. The IMGT database must have been installed with bash install_immune_dbs.sh -I -y and followed by pip install --no-cache-dir . in the pipeline directory.
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, output_flatten, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Init all other properties and jobs</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, output_flatten=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str | none, optional) — The new name of the process
  • desc (str | none, optional) — The new description of the process
  • envs (Optional, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int | none, optional) — How deep to update the envs when subclassed.
  • cache (bool | none, optional) — Whether we should check the cache for the jobs
  • export (bool | none, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • output_flatten (bool | none, optional) — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • error_strategy (str | none, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • num_retries (int | none, optional) — How many times to retry to jobs once error occurs
  • forks (int | none, optional) — New forks for the new process
  • input_data (any | none, optional) — The input data for the process. Only when this processis a start process
  • order (int | none, optional) — The order to execute the new process
  • plugin_opts (Optional, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Optional, optional) — The required processes for the new process
  • scheduler (str | none, optional) — The new shedular to run the new process
  • scheduler_opts (Optional, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int | none, optional) — How many jobs to be submited simultaneously.
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

run()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
class

biopipen.ns.splitpipe.SplitPipeCombine(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Combine sublibraries into a single dataset

Runs split-pipe --mode comb to merge the outputs of the sublibrary (one well-pool processed by --mode all) directories of one sequencing run into a single dataset with a new report. Requires the Parse Biosciences Pipeline (split-pipe) v1.9.0+.

Terminology: each sublibrary is one well-pool processed by SplitPipeCount (--mode all). Only sublibraries of the same library preparation can be combined (for example multiple WT sublibraries), and sublibraries must have been processed with the same version of the pipeline as the one used here. Chemistry, kit and sample definitions are parsed from the sublibrary directories, so they are not passed to --mode comb.

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • output_flatten — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously.The program entrance for some schedulers may take too much resources when submitting a job or checking the job status. So we may use a smaller number here to limit the simultaneous submissions.
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • id — The id defining the output directory. If not provided, it isinferred from the sublibrary directories (see SplitPipePipeline).
  • sublibs — The output directories of the sublibraries to combine, fromSplitPipeCount.
Output
  • outdir — The output directory
Envs
  • — Other environment variables required by split-pipe --mode comb.Note that --tcr_analysis/--bcr_analysis/--crispr are not documented for --mode comb; any other option can be passed through here if the installed version of split-pipe supports it. See https://support.parsebiosciences.com/hc/en-us/articles/23060220494228 for more details.
  • immune_genome — The immune genome to align to for the TCR/BCR analysiscombined here, either human or mouse.
  • ncores — Alias for nthreads. Takes precedence when both are given.
  • nthreads — Number of threads to use
  • parent_dir — Path to the combined whole transcriptome (parent) outputdirectory. The parent outputs must be combined before the corresponding TCR/BCR/CRISPR ones, and both must combine the same set of sublibraries.
  • splitpipe — Path to the split-pipe executable
  • sublib_list — Path to a file with one sublibrary directory per line.If provided, it is passed as --sublib_list instead of passing in.sublibs as --sublibraries.
  • use_imgt_db (flag) — Whether to use the IMGT reference for theTCR/BCR analysis. The IMGT database must have been installed with bash install_immune_dbs.sh -I -y and followed by pip install --no-cache-dir . in the pipeline directory.
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, output_flatten, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Init all other properties and jobs</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, output_flatten=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str | none, optional) — The new name of the process
  • desc (str | none, optional) — The new description of the process
  • envs (Optional, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int | none, optional) — How deep to update the envs when subclassed.
  • cache (bool | none, optional) — Whether we should check the cache for the jobs
  • export (bool | none, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • output_flatten (bool | none, optional) — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • error_strategy (str | none, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • num_retries (int | none, optional) — How many times to retry to jobs once error occurs
  • forks (int | none, optional) — New forks for the new process
  • input_data (any | none, optional) — The input data for the process. Only when this processis a start process
  • order (int | none, optional) — The order to execute the new process
  • plugin_opts (Optional, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Optional, optional) — The required processes for the new process
  • scheduler (str | none, optional) — The new shedular to run the new process
  • scheduler_opts (Optional, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int | none, optional) — How many jobs to be submited simultaneously.
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

run()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
class

biopipen.ns.splitpipe.SplitPipeSummary(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Summarize split-pipe metrics

Merges the metrics files of the sublibrary (or combined) output directories of SplitPipeCount/SplitPipeCombine into one table. The names of the metrics files are version-dependent, so they are discovered with envs.metrics_glob and the process fails if nothing matches.

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • output_flatten — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously.The program entrance for some schedulers may take too much resources when submitting a job or checking the job status. So we may use a smaller number here to limit the simultaneous submissions.
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • indirs — The directories containing split-pipe results, fromSplitPipeCount/SplitPipeCombine.
Output
  • outdir — The output directory
Envs
  • metrics_glob (type=str) — The glob pattern of the metrics file names.split-pipe 1.9.0 writes agg_sample_summary.csv at the top level of each output directory (the older metrics_summary.csv name is not written by this version), relative to each input directory, e.g. *metrics_summary.csv. The directories are searched recursively, so metrics files nested in per-sample subdirectories are found as well.
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, output_flatten, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Init all other properties and jobs</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, output_flatten=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str | none, optional) — The new name of the process
  • desc (str | none, optional) — The new description of the process
  • envs (Optional, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int | none, optional) — How deep to update the envs when subclassed.
  • cache (bool | none, optional) — Whether we should check the cache for the jobs
  • export (bool | none, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • output_flatten (bool | none, optional) — Whether to flatten the output when saving to the outputdirectory. Normally, the output will be saved in a subdirectory named after the job index (e.g. <outdir>/0, <outdir>/1, etc.). If output_flatten is True, the output will be saved directly in the output directory without the subdirectories. This is useful when you want the job outputs to be directly revealed in the output directory. Note that this only works for processes with export=True or end processes and make sure the name of the output files won't conflict for jobs with each other when flattening. It takes 3 possible values
    • - None (default): flatten the output for single-job processes only
    • - True: flatten the output for all processes
    • - False: never flatten the output
  • error_strategy (str | none, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
  • num_retries (int | none, optional) — How many times to retry to jobs once error occurs
  • forks (int | none, optional) — New forks for the new process
  • input_data (any | none, optional) — The input data for the process. Only when this processis a start process
  • order (int | none, optional) — The order to execute the new process
  • plugin_opts (Optional, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Optional, optional) — The required processes for the new process
  • scheduler (str | none, optional) — The new shedular to run the new process
  • scheduler_opts (Optional, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int | none, optional) — How many jobs to be submited simultaneously.
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

run()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger