module

biopipen.ns.vcf

</>

Tools to handle VCF files

Classes
class

biopipen.ns.vcf.VcfLiftOver(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Liftover a VCF file using GATK

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • invcf — The input VCF file
Output
  • outvcf — The output VCF file
Envs
  • args — Other CLI arguments for gatk LiftoverVcf
  • chain — The map chain file for liftover
  • gatk — The path to gatk4, which should be installed via conda
  • tmpdir — Directory for temporary storage of working files
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfFilter(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Filter records in vcf file

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • invcf — The input vcf file, could be bgzipped.
Output
  • outfile — The filtered vcf file. If in.invcf is bgzipped, thenthis will be bgzipped.
Envs
  • filter_descs — Descriptions for the filters. Will be saved to the headerof the output vcf file
  • filters — A dict of filters with keys the filter names.

    Typically

    lambda variant: Things to notice 1. Filters should return False to get variant filtered out 2. See https://brentp.github.io/cyvcf2/docstrings.html#cyvcf2.cyvcf2.Variant For what you can do with the variant 3. The filter python functions should be in string representation 4. Builtin filters can have parameters {"QUAL": 30} 5. List of builtin filters. Specify them like: {"FILTER": params} SNPONLY: keeps only SNPs ({"SNPONLY": False} to filter SNPs out) QUAL: keeps variants with QUAL>=param ({"QUAL": (30, False)}) to keep only variants with QUAL<30
  • helper — Some helper code for the filters
  • keep — Keep the variants not passing the filters?
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfIndex(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Index VCF files. If they are already index, use the index files

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
Output
  • outfile — The output VCF file (bgzipped)
  • outidx — The index file of the output VCF file
Envs
  • tabix — Path to tabix
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.Vcf2Bed(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Convert Vcf file to Bed file

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The vcf file
Output
  • outfile — The converted bed file
Envs
  • inbase — The coordinate base of the vcf file
  • outbase — The coordinate base of the base file
Requires
  • cyvcf2 —
    • check: {{proc.lang}} -c "import cyvcf2"
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfDownSample(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Down-sample VCF files to keep only a subset of variants in there

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
Output
  • outfile — The output VCF file with subet variantsGzipped if in.infile is gzipped
Envs
  • n — Fraction/Number of variants to keepIf n > 1, it is the number. If n <= 1, it is the fraction.
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfSplitSamples(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Split a VCF file into multiple VCF files, one for each sample

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
Output
  • outdir — The output directory containing the split VCF files
Envs
  • bcftools — Path to bcftools
  • gz — Gzip the output VCF files? Has to be True if envs.index is True
  • index — Index the output VCF files?
  • ncores — Number of cores, used to extract samples, but not to index
  • private — Keep sites where only the sample carries an non-ref allele.That means, sites with genotypes like 0/0 will be removed.
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfIntersect(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Find variants in both VCF files

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile1 — The first VCF file
  • infile2 — The second VCF file
Output
  • outfile — The output VCF file with subet variants in both files
Envs
  • bcftools — Path to bcftools
  • collapse — How to match the variants in the two files? Will be passed tobcftools isec -c option. See also https://samtools.github.io/bcftools/bcftools.html#common_options
    • - none: only records with identical REF and ALT alleles are
        compatible
    • - some: only records where some subset of ALT alleles match are
        compatible
    • - all: all records are compatible, regardless of whether the ALT
        alleles match or not.
    • - snps: any SNP records are compatible, regardless of whether the
        ALT alleles match or not.
    • - indels: any indel records are compatible, regardless of whether
        the ALT alleles match or not.
    • - both: abbreviates snps and indels
    • - id: only records with identical ID are compatible
  • gz — Gzip the output VCF files? Has to be True if envs.index is True
  • index — Index the output VCF files?
  • keep_as — Keep the variants as presented in the first (0) orthe second (1) file?
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfFix(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Fix some issues with VCF files

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
Output
  • outfile — The output VCF file
Envs
  • fixes — A list of fixes to apply.Each one is a dict with keys kind, id, regex and fix kind: The kind of fix. Including filter the FILTERs in the header, info the info INFOs in the header, contig the contig lines in the header format the FORMATs in the header, colnames the column names in the header header general header item variant the variants None matches everything id: The ID the match. If kind is filter, info, contig or format, then it matches the ID of the item. If kind is variant, then it matches the ID of the variant. If a list is given, then it matches any of the IDs in the list. regex: The regular expression to match. When id is given, this is ignored. append: Whether to append a record instead of to replace an existing one. When it is True, kind has to not be None fix: The fix to apply in the format of a lambda function (in string), with a single argument. The function should either return a string (raw representation) for the record, the record itself, None, False. If None is returned, the original record is used. if False, the record is removed. If append is True, then the function should either return a string or an object. And the argument is None The argument is a different object based on different kinds. When kind is None, the argument is the plain line of the record with line ending. When kind is info or format, the record is a dict with keys ID, Description, Type and Number. When kind is filter, the record is a dict with keys ID and Description. When kind is contig, the record is a dict with keys ID and length. When kind is header, the record is a dict with key the name of the header and value the value of the header. When kind is colnames, the record is a list of column names. When kind is variant, the record is a dict with keys CHROM, POS, REF, ALT, QUAL, FILTER, INFO, FORMAT and SAMPLES. INFO is a dict with key-value pairs and SAMPLES are a list of values for each sample. Each value is also a list of values for each FORMAT. If a record matches multiple fixes, the first one is applied.
  • helpers — raw code the provide some helpers for the fixesThe code will automatically dedented if given as a string. A list of strings is also supported and will be joined with newlines.
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.VcfAnno(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Annotate a VCF file using vcfanno

https://github.com/brentp/vcfanno

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • conffile — The configuration file for vcfanno or configuration dictitself
  • infile — The input VCF file
Output
  • outfile — The output VCF file
Envs
  • args — Additional arguments to pass to vcfanno
  • conffile — configuration file for vcfanno or configuration dict itselfThis is ignored when conffile is given as input
  • ncores — Number of cores to use
  • vcfanno — Path to vcfanno
Requires
  • - name — vcfannocheck: | {{proc.envs.vcfanno}} --help
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.TruvariBench(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Run truvari bench to compare a VCF with CNV calls andbase CNV standards

Requires truvari v4+

See https://github.com/ACEnglish/truvari/wiki/bench

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • basevcf — The VCF file with standard CNVs
  • compvcf — The VCF file with CNV calls to compare
Output
  • outdir — The output directory
Envs
  • `` — Ohter truvari bench arguments
  • truvari — Path to truvari
Requires
  • truvari —
    • check: {{proc.envs.truvari}} version
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.TruvariBenchSummary(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Summarise the statistics from TruvariBench for multiple jobs (VCFs)

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • indirs — The input directories, which should be the output directoriesof TruvariBench
Output
  • outdir — The output directory, including the summary table and plots
Envs
  • devpars — The parameters to use for the plots.
  • plots — The stats to plot with barplots.Candidates are TP-base, TP-call, FP, FN, precision, recall, f1, base cnt, call cnt, TP-call_TP-gt, TP-call_FP-gt, TP-base_TP-gt, TP-base_FP-gt, and gt_concordance See https://github.com/ACEnglish/truvari/wiki/bench
Requires
  • r-dplyr —
    • check: {{proc.lang}} -e "library(dplyr)"
  • r-ggplot2 —
    • check: {{proc.lang}} -e "library(ggplot2)"
  • r-ggprism —
    • check: {{proc.lang}} -e "library(ggprism)"
  • r-rjson —
    • check: {{proc.lang}} -e "library(rjson)"
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.TruvariConsistency(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Run truvari consistency to check consistency of CNV calls

See https://github.com/ACEnglish/truvari/wiki/consistency

Requires truvari v4+

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • vcfs — The vcf files with CNV calls
Output
  • outfile — The output file with the report
Envs
  • heatmap — Whether to generate a heatmap of the consistencySet to False to disable annofile: The annotation file for the heatmap, multiple columns but the first column must be the sample name. Note that the stem of the vcf file name from consistency file will be used. These annotations will be added as row annotations. Other options see also biopipen.ns.plot.Heatmap.
  • truvari — Path to truvari
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.BcftoolsAnnotate(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Add or remove annotations from VCF files

See also: https://samtools.github.io/bcftools/bcftools.html#annotate

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • annfile — The annotation file.Currently only VCF files are supported.
  • infile — The input VCF file
Output
  • outfile — The VCF file with annotations added or removed.
Envs
  • — Other arguments for bcftools annotateSee also https://samtools.github.io/bcftools/bcftools.html#annotate Note that the underscore _ will be replaced with dash - in the argument name.
  • annfile — The annotation file. If in.annfile is provided,this is ignored
  • bcftools — Path to bcftools
  • columns (auto) — Comma-separated or list of columns or tags to carry over fromthe annotation file. Overrides -c, --columns
  • gz (flag) — Whether to gzip the output file
  • header (list) — Headers to be added
  • index (flag) — Whether to index the output file (tbi) (envs.gz forced to True)
  • ncores (type=int) — Number of cores (--threads) to use
  • remove (auto) — Remove the specified columns from the input file
  • tabix — Path to tabix, used to index infile and annfile
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.BcftoolsFilter(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Apply fixed threshold filters to VCF files

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
Output
  • outfile — The filtered VCF file. If the in.infile is gzipped, this isgzipped as well.
Envs
  • — Other arguments for bcftools filterSee also https://samtools.github.io/bcftools/bcftools.html#filter
  • bcftools — Path to bcftools
  • excludes — include/exclude only sites for which EXPRESSION is true.See: https://samtools.github.io/bcftools/bcftools.html#expressions If provided, envs.include/exclude will be ignored. If str/list used, The filter names will be Filter_<type>_<index>. A dict is used where keys are filter names and values are expressions
  • gz (flag) — Whether to gzip the output file
  • includes — and
  • index (flag) — Whether to index the output file (tbi) (envs.gz forced to True)
  • keep — Whether we should keep the filtered variants or not.If True, the filtered variants will be kept in the output file, but with a new FILTER.
  • ncores (type=int) — Number of cores (--threads) to use
  • tabix — Path to tabix, used to index infile/outfile
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.BcftoolsSort(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

Sort VCF files using bcftools sort.

bcftools sort is used to sort VCF files by chromosome and position based on the order of contigs in the header.

Here we provide a chrsize file to first sort the contigs in the header and then sort the VCF file using bcftools sort.

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
Output
  • outfile — The sorted VCF file.
Envs
  • — Other arguments for bcftools sort. For example max_mem.See also https://samtools.github.io/bcftools/bcftools.html#sort
  • bcftools — Path to bcftools
  • chrsize — The chromosome size file, from which the chromosome order is usedto sort the contig in the header first. If not provided, bcftools sort will be used directly.
  • gz (flag) — Whether to gzip the output file
  • index (flag) — Whether to index the output file (tbi) (envs.gz forced to True)
  • ncores (type=int) — Number of cores (--threads) to use
  • notfound (choice) — What if the contig in the VCF file is not found in thechrsize file.
    • - error: Report error
    • - remove: Remove the contig from the header.
        Note that if there are records with the removed contig, an error will
        be raised by bcftools sort
    • - start: Move the contig to the start of the contigs from chrsize
    • - end: Move the contig to the end of the contigs from chrsize
  • tabix — Path to tabix, used to index infile/outfile
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process

class

biopipen.ns.vcf.BcftoolsView(*args, **kwds) → Proc

</>
Bases
biopipen.core.proc.Proc pipen.proc.Proc

View, subset and filter VCF files by position and filtering expression.

Also convert between VCF and BCF.

Attributes
  • cache — Should we detect whether the jobs are cached?
  • desc — The description of the process. Will use the summary fromthe docstring by default.
  • dirsig — When checking the signature for caching, whether should we walkthrough the content of the directory? This is sometimes time-consuming if the directory is big.
  • envs — The arguments that are job-independent, useful for common optionsacross jobs.
  • envs_depth — How deep to update the envs when subclassed.
  • error_strategy — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • export — When True, the results will be exported to <pipeline.outdir>Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • forks — How many jobs to run simultaneously?
  • input — The keys for the input channel
  • input_data — The input data (will be computed for dependent processes)
  • lang — The language for the script to run. Should be the path to theinterpreter if lang is not in $PATH.
  • name — The name of the process. Will use the class name by default.
  • nexts — Computed from requires to build the process relationships
  • num_retries — How many times to retry to jobs once error occurs
  • order — The execution order for this process. The bigger the numberis, the later the process will be executed. Default: 0. Note that the dependent processes will always be executed first. This doesn't work for start processes either, whose orders are determined by Pipen.set_starts()
  • output — The output keys for the output channel(the data will be computed)
  • output_data — The output data (to pass to the next processes)
  • plugin_opts — Options for process-level plugins
  • requires — The dependency processes
  • scheduler — The scheduler to run the jobs
  • scheduler_opts — The options for the scheduler
  • script — The script template for the process
  • submission_batch — How many jobs to be submited simultaneously
  • template — Define the template engine to use.This could be either a template engine or a dict with key engine indicating the template engine and the rest the arguments passed to the constructor of the pipen.template.Template object. The template engine could be either the name of the engine, currently jinja2 and liquidpy are supported, or a subclass of pipen.template.Template. You can subclass pipen.template.Template to use your own template engine.
Input
  • infile — The input VCF file
  • regions_file — The region file used to subset the input VCF file.
  • samples_file — The samples file used to subset the input VCF file.
Output
  • outfile — The output VCF file.
Envs
  • — Other arguments for bcftools view.See also https://samtools.github.io/bcftools/bcftools.html#view Note that the underscore _ will be replaced with dash - in the argument name.
  • bcftools — Path to bcftools
  • gz (flag) — Whether to gzip the output file
  • index (flag) — Whether to index the output file (tbi) (envs.gz forced to True)
  • ncores (type=int) — Number of cores (--threads) to use
  • regions_file — The region file used to subset the input VCF file.If in.regions_file is provided, this is ignored.
  • samples_file — The samples file used to subset the input VCF file.If in.samples_file is provided, this is ignored.
  • tabix — Path to tabix, used to index infile/outfile
Classes
Methods
  • __init_subclass__() — Do the requirements inferring since we need them to build up theprocess relationship </>
  • from_proc(proc, name, desc, envs, envs_depth, cache, export, error_strategy, num_retries, forks, input_data, order, plugin_opts, requires, scheduler, scheduler_opts, submission_batch) (Type) — Create a subclass of Proc using another Proc subclass or Proc itself</>
  • gc() — GC process for the process to save memory after it's done</>
  • init() — Init all other properties and jobs</>
  • log(level, msg, *args, logger) — Log message for the process</>
  • run() — Run the process</>
class

pipen.proc.ProcMeta(name, bases, namespace, **kwargs)

</>
Bases
abc.ABCMeta

Meta class for Proc

Methods
  • __call__(cls, *args, **kwds) (Proc) — Make sure Proc subclasses are singletons</>
  • __instancecheck__(cls, instance) — Override for isinstance(instance, cls).</>
  • __repr__(cls) (str) — Representation for the Proc subclasses</>
  • __subclasscheck__(cls, subclass) — Override for issubclass(subclass, cls).</>
  • register(cls, subclass) — Register a virtual subclass of an ABC.</>
staticmethod
register(cls, subclass)
</>

Register a virtual subclass of an ABC.

Returns the subclass, to allow usage as a class decorator.

staticmethod
__instancecheck__(cls, instance)
</>

Override for isinstance(instance, cls).

staticmethod
__subclasscheck__(cls, subclass)
</>

Override for issubclass(subclass, cls).

staticmethod
__repr__(cls) → str
</>

Representation for the Proc subclasses

staticmethod
__call__(cls, *args, **kwds)
</>

Make sure Proc subclasses are singletons

Parameters
  • *args (Any) — and
  • **kwds (Any) — Arguments for the constructor
Returns (Proc)

The Proc instance

classmethod

from_proc(proc, name=None, desc=None, envs=None, envs_depth=None, cache=None, export=None, error_strategy=None, num_retries=None, forks=None, input_data=None, order=None, plugin_opts=None, requires=None, scheduler=None, scheduler_opts=None, submission_batch=None)

</>

Create a subclass of Proc using another Proc subclass or Proc itself

Parameters
  • proc (Type) — The Proc subclass
  • name (str, optional) — The new name of the process
  • desc (str, optional) — The new description of the process
  • envs (Mapping, optional) — The arguments of the process, will overwrite parent oneThe items that are specified will be inherited
  • envs_depth (int, optional) — How deep to update the envs when subclassed.
  • cache (bool, optional) — Whether we should check the cache for the jobs
  • export (bool, optional) — When True, the results will be exported to<pipeline.outdir> Defaults to None, meaning only end processes will export. You can set it to True/False to enable or disable exporting for processes
  • error_strategy (str, optional) — How to deal with the errors
    • - retry, ignore, halt
    • - halt to halt the whole pipeline, no submitting new jobs
    • - terminate to just terminate the job itself
  • num_retries (int, optional) — How many times to retry to jobs once error occurs
  • forks (int, optional) — New forks for the new process
  • input_data (Any, optional) — The input data for the process. Only when this processis a start process
  • order (int, optional) — The order to execute the new process
  • plugin_opts (Mapping, optional) — The new plugin options, unspecified items will beinherited.
  • requires (Sequence, optional) — The required processes for the new process
  • scheduler (str, optional) — The new shedular to run the new process
  • scheduler_opts (Mapping, optional) — The new scheduler options, unspecified items willbe inherited.
  • submission_batch (int, optional) — How many jobs to be submited simultaneously
Returns (Type)

The new process class

classmethod

__init_subclass__()

</>

Do the requirements inferring since we need them to build up theprocess relationship

method

init()

</>

Init all other properties and jobs

method

gc()

</>

GC process for the process to save memory after it's done

method

log(level, msg, *args, logger=<LoggerAdapter pipen.core (WARNING)>)

</>

Log message for the process

Parameters
  • level (int | str) — The log level of the record
  • msg (str) — The message to log
  • *args — The arguments to format the message
  • logger (LoggerAdapter, optional) — The logging logger
method

run()

</>

Run the process