pipen.scheduler
Provide builting schedulers
LocalJob— Job class for local scheduler</>LocalScheduler— Local scheduler</>SgeJob— Job class for SGE scheduler</>SgeScheduler— SGE scheduler</>SlurmJob— Job class for Slurm scheduler</>SlurmScheduler— Slurm scheduler</>SshJob— Job class for SSH scheduler</>SshScheduler— SSH scheduler</>
get_scheduler(scheduler)(Type) — Get the scheduler by name of the scheduler class itself</>
pipen.scheduler.LocalJob(*args, **kwargs)
Job class for local scheduler
CMD_WRAPPER_SHELL— The shell to run the wrapped scriptCMD_WRAPPER_TEMPLATE— The template for job wrapping_error_retry— Whether we should retry if error happened_num_retries— Total number of retries_rc— The return code of the job_status— The status of the job_wrapped_cmd— The wrapped cmd, used for job submissioncached— Check if a job is cached</>cmd— The commandhook_done— Mark whether hooks have already been. Since we don't havea trigger for job finished/failed, so we do a polling on it. This is to avoid calling the hooks repeatedlyindex— The index of the jobjid— The jid of the job in scheduler systemjid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>metadir— The metadir of the jobrc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>trial_count— The count for re-tries
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — The shebang of the wrapped script</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)
Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → str
repr of the job
shebang(scheduler) → str
The shebang of the wrapped script
clean(retry=False)
Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)
Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
log(level, msg, *args, limit=3, limit_indicator=True, logger=<LoggerAdapter pipen.core (WARNING)>)
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
pipen.scheduler.LocalScheduler(forks, prescript='', postscript='', **kwargs)
Local scheduler
forks(int) — Max number of job forksprescript(str, optional) — The script to run before the commandpostscript(str, optional) — The script to run after the command**kwargs— Other arguments for the scheduler
job_class— The job classname— The name of the scheduler
job_is_running(job)(bool) — Tell if a job is really running, not only the job.jid_file</>job_is_submitted_or_running(job)(bool) — Check if a job is already submitted or running</>kill_job(job)— Kill a job asynchronously</>kill_job_and_update_status(job)— Kill a job and update its status</>kill_running_jobs(jobs)— Try to kill all running jobs</>polling_jobs(jobs,on,halt_on_error)(bool) — Check if all jobs are done or new jobs can submit</>retry_job(job)— Retry a job</>submit_job(job)(int) — Submit a job locally</>submit_job_and_update_status(job)— Submit and update the status</>
submit_job_and_update_status(job)
Submit and update the status
- Check if the job is already submitted or running
- If not, run the hook
- If the hook is not cancelled, clean the job
- Submit the job, raising an exception if it fails
- If the job is submitted successfully, update the status
- If the job fails to submit, update the status and write stderr to the job file
job(Job) — The job
retry_job(job)
Retry a job
job(Job) — The job
kill_job_and_update_status(job)
Kill a job and update its status
job(Job) — The job
polling_jobs(jobs, on, halt_on_error)
Check if all jobs are done or new jobs can submit
jobs(List) — The list of jobson(str) — query on status:can_submitorall_donehalt_on_error(bool) — Whether we should halt the whole pipeline on error
True if yes otherwise False.
kill_running_jobs(jobs)
Try to kill all running jobs
jobs(List) — The list of jobs
job_is_submitted_or_running(job)
Check if a job is already submitted or running
job(Job) — The job
True if yes otherwise False.
submit_job(job)
Submit a job locally
job(Job) — The job
The process id
kill_job(job)
Kill a job asynchronously
job(Job) — The job
job_is_running(job)
Tell if a job is really running, not only the job.jid_file
In case where the jid file is not cleaned when job is done.
job(Job) — The job
True if it is, otherwise False
pipen.scheduler.LocalJob(*args, **kwargs)
Job class for local scheduler
cached— Check if a job is cached</>jid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>rc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — The shebang of the wrapped script</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → strrepr of the job
shebang(scheduler) → strThe shebang of the wrapped script
clean(retry=False)Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
pipen.scheduler.SgeJob(*args, **kwargs)
Job class for SGE scheduler
CMD_WRAPPER_SHELL— The shell to run the wrapped scriptCMD_WRAPPER_TEMPLATE— The template for job wrapping_error_retry— Whether we should retry if error happened_num_retries— Total number of retries_rc— The return code of the job_status— The status of the job_wrapped_cmd— The wrapped cmd, used for job submissioncached— Check if a job is cached</>cmd— The commandhook_done— Mark whether hooks have already been. Since we don't havea trigger for job finished/failed, so we do a polling on it. This is to avoid calling the hooks repeatedlyindex— The index of the jobjid— The jid of the job in scheduler systemjid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>metadir— The metadir of the jobrc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>trial_count— The count for re-tries
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — Make the shebang with options</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)
Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → str
repr of the job
clean(retry=False)
Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)
Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
log(level, msg, *args, limit=3, limit_indicator=True, logger=<LoggerAdapter pipen.core (WARNING)>)
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
shebang(scheduler)
Make the shebang with options
scheduler(Scheduler) — The scheduler
The shebang with options
pipen.scheduler.SgeScheduler(*args, **kwargs)
SGE scheduler
**kwargs— Other arguments for the scheduler
job_class— The job classname— The name of the scheduler
job_is_running(job)(bool) — Tell if a job is really running, not only the job.jid_file</>job_is_submitted_or_running(job)(bool) — Check if a job is already submitted or running</>kill_job(job)— Kill a job on SGE</>kill_job_and_update_status(job)— Kill a job and update its status</>kill_running_jobs(jobs)— Try to kill all running jobs</>polling_jobs(jobs,on,halt_on_error)(bool) — Check if all jobs are done or new jobs can submit</>retry_job(job)— Retry a job</>submit_job(job)(str) — Submit a job to SGE</>submit_job_and_update_status(job)— Submit and update the status</>
submit_job_and_update_status(job)
Submit and update the status
- Check if the job is already submitted or running
- If not, run the hook
- If the hook is not cancelled, clean the job
- Submit the job, raising an exception if it fails
- If the job is submitted successfully, update the status
- If the job fails to submit, update the status and write stderr to the job file
job(Job) — The job
retry_job(job)
Retry a job
job(Job) — The job
kill_job_and_update_status(job)
Kill a job and update its status
job(Job) — The job
polling_jobs(jobs, on, halt_on_error)
Check if all jobs are done or new jobs can submit
jobs(List) — The list of jobson(str) — query on status:can_submitorall_donehalt_on_error(bool) — Whether we should halt the whole pipeline on error
True if yes otherwise False.
kill_running_jobs(jobs)
Try to kill all running jobs
jobs(List) — The list of jobs
job_is_submitted_or_running(job)
Check if a job is already submitted or running
job(Job) — The job
True if yes otherwise False.
submit_job(job)
Submit a job to SGE
job(Job) — The job
The job id
kill_job(job)
Kill a job on SGE
job(Job) — The job
job_is_running(job)
Tell if a job is really running, not only the job.jid_file
In case where the jid file is not cleaned when job is done.
job(Job) — The job
True if it is, otherwise False
pipen.scheduler.SgeJob(*args, **kwargs)
Job class for SGE scheduler
cached— Check if a job is cached</>jid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>rc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — Make the shebang with options</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → strrepr of the job
clean(retry=False)Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
shebang(scheduler)Make the shebang with options
scheduler(Scheduler) — The scheduler
The shebang with options
pipen.scheduler.SlurmJob(*args, **kwargs)
Job class for Slurm scheduler
CMD_WRAPPER_SHELL— The shell to run the wrapped scriptCMD_WRAPPER_TEMPLATE— The template for job wrapping_error_retry— Whether we should retry if error happened_num_retries— Total number of retries_rc— The return code of the job_status— The status of the job_wrapped_cmd— The wrapped cmd, used for job submissioncached— Check if a job is cached</>cmd— The commandhook_done— Mark whether hooks have already been. Since we don't havea trigger for job finished/failed, so we do a polling on it. This is to avoid calling the hooks repeatedlyindex— The index of the jobjid— The jid of the job in scheduler systemjid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>metadir— The metadir of the jobrc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>trial_count— The count for re-tries
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — Make the shebang with options</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)
Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → str
repr of the job
clean(retry=False)
Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)
Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
log(level, msg, *args, limit=3, limit_indicator=True, logger=<LoggerAdapter pipen.core (WARNING)>)
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
shebang(scheduler)
Make the shebang with options
scheduler(Scheduler) — The scheduler
The shebang with options
pipen.scheduler.SlurmScheduler(*args, **kwargs)
Slurm scheduler
**kwargs— Other arguments for the scheduler
job_class— The job classname— The name of the scheduler
job_is_running(job)(bool) — Tell if a job is really running, not only the job.jid_file</>job_is_submitted_or_running(job)(bool) — Check if a job is already submitted or running</>kill_job(job)— Kill a job on Slurm</>kill_job_and_update_status(job)— Kill a job and update its status</>kill_running_jobs(jobs)— Try to kill all running jobs</>polling_jobs(jobs,on,halt_on_error)(bool) — Check if all jobs are done or new jobs can submit</>retry_job(job)— Retry a job</>submit_job(job)(str) — Submit a job to Slurm</>submit_job_and_update_status(job)— Submit and update the status</>
submit_job_and_update_status(job)
Submit and update the status
- Check if the job is already submitted or running
- If not, run the hook
- If the hook is not cancelled, clean the job
- Submit the job, raising an exception if it fails
- If the job is submitted successfully, update the status
- If the job fails to submit, update the status and write stderr to the job file
job(Job) — The job
retry_job(job)
Retry a job
job(Job) — The job
kill_job_and_update_status(job)
Kill a job and update its status
job(Job) — The job
polling_jobs(jobs, on, halt_on_error)
Check if all jobs are done or new jobs can submit
jobs(List) — The list of jobson(str) — query on status:can_submitorall_donehalt_on_error(bool) — Whether we should halt the whole pipeline on error
True if yes otherwise False.
kill_running_jobs(jobs)
Try to kill all running jobs
jobs(List) — The list of jobs
job_is_submitted_or_running(job)
Check if a job is already submitted or running
job(Job) — The job
True if yes otherwise False.
submit_job(job)
Submit a job to Slurm
job(Job) — The job
The job id
kill_job(job)
Kill a job on Slurm
job(Job) — The job
job_is_running(job)
Tell if a job is really running, not only the job.jid_file
In case where the jid file is not cleaned when job is done.
job(Job) — The job
True if it is, otherwise False
pipen.scheduler.SlurmJob(*args, **kwargs)
Job class for Slurm scheduler
cached— Check if a job is cached</>jid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>rc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — Make the shebang with options</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → strrepr of the job
clean(retry=False)Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
shebang(scheduler)Make the shebang with options
scheduler(Scheduler) — The scheduler
The shebang with options
pipen.scheduler.SshJob(*args, **kwargs)
Job class for SSH scheduler
CMD_WRAPPER_SHELL— The shell to run the wrapped scriptCMD_WRAPPER_TEMPLATE— The template for job wrapping_error_retry— Whether we should retry if error happened_num_retries— Total number of retries_rc— The return code of the job_status— The status of the job_wrapped_cmd— The wrapped cmd, used for job submissioncached— Check if a job is cached</>cmd— The commandhook_done— Mark whether hooks have already been. Since we don't havea trigger for job finished/failed, so we do a polling on it. This is to avoid calling the hooks repeatedlyindex— The index of the jobjid— The jid of the job in scheduler systemjid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>metadir— The metadir of the jobrc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>trial_count— The count for re-tries
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — The shebang of the wrapped script</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)
Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → str
repr of the job
shebang(scheduler) → str
The shebang of the wrapped script
clean(retry=False)
Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)
Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
log(level, msg, *args, limit=3, limit_indicator=True, logger=<LoggerAdapter pipen.core (WARNING)>)
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
pipen.scheduler.SshScheduler(forks, prescript='', postscript='', **kwargs)
SSH scheduler
forks(int) — Max number of job forksprescript(str, optional) — The script to run before the commandpostscript(str, optional) — The script to run after the command**kwargs— Other arguments for the scheduler
job_class— The job classname— The name of the scheduler
job_is_running(job)(bool) — Tell if a job is really running, not only the job.jid_file</>job_is_submitted_or_running(job)(bool) — Check if a job is already submitted or running</>kill_job(job)— Kill a job on SSH</>kill_job_and_update_status(job)— Kill a job and update its status</>kill_running_jobs(jobs)— Try to kill all running jobs</>polling_jobs(jobs,on,halt_on_error)(bool) — Check if all jobs are done or new jobs can submit</>retry_job(job)— Retry a job</>submit_job(job)(str) — Submit a job to SSH</>submit_job_and_update_status(job)— Submit and update the status</>
submit_job_and_update_status(job)
Submit and update the status
- Check if the job is already submitted or running
- If not, run the hook
- If the hook is not cancelled, clean the job
- Submit the job, raising an exception if it fails
- If the job is submitted successfully, update the status
- If the job fails to submit, update the status and write stderr to the job file
job(Job) — The job
retry_job(job)
Retry a job
job(Job) — The job
kill_job_and_update_status(job)
Kill a job and update its status
job(Job) — The job
polling_jobs(jobs, on, halt_on_error)
Check if all jobs are done or new jobs can submit
jobs(List) — The list of jobson(str) — query on status:can_submitorall_donehalt_on_error(bool) — Whether we should halt the whole pipeline on error
True if yes otherwise False.
kill_running_jobs(jobs)
Try to kill all running jobs
jobs(List) — The list of jobs
job_is_submitted_or_running(job)
Check if a job is already submitted or running
job(Job) — The job
True if yes otherwise False.
submit_job(job)
Submit a job to SSH
job(Job) — The job
The job id
kill_job(job)
Kill a job on SSH
job(Job) — The job
job_is_running(job)
Tell if a job is really running, not only the job.jid_file
In case where the jid file is not cleaned when job is done.
job(Job) — The job
True if it is, otherwise False
pipen.scheduler.SshJob(*args, **kwargs)
Job class for SSH scheduler
cached— Check if a job is cached</>jid(int | str | none) — Get the jid of the job in scheduler system</>jid_file(Path) — The jid file of the job</>rc(int) — The return code of the job</>rc_file(Path) — The rc file of the job</>retry_dir(Path) — The retry directory of the job</>script_file— Get the path to script file</>signature_file— Get the path to the signature file</>status(int) — Query the status of the job
If the job is submitted, try to query it from the status file Make sure the status is updated by trap in wrapped script </>status_file(Path) — The status file of the job</>stderr_file(Path) — The stderr file of the job</>stdout_file(Path) — The stdout file of the job</>strcmd(str) — Get the string representation of the command</>
__repr__()(str) — repr of the job</>cache()— write signature to signature file</>clean(retry)— Clean up the meta files</>log(level,msg,*args,limit,limit_indicator,logger)— Log message for the jobs</>prepare(proc)— Prepare the job by given process</>shebang(scheduler)(str) — The shebang of the wrapped script</>wrapped_script(scheduler)(PathLike) — Get the wrapped script</>
abc.ABCMeta(name, bases, namespace, **kwargs)Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' -- these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won't show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).
__instancecheck__(cls,instance)— Override for isinstance(instance, cls).</>__subclasscheck__(cls,subclass)— Override for issubclass(subclass, cls).</>register(cls,subclass)— Register a virtual subclass of an ABC.</>
register(cls, subclass)Register a virtual subclass of an ABC.
Returns the subclass, to allow usage as a class decorator.
__instancecheck__(cls, instance)Override for isinstance(instance, cls).
__subclasscheck__(cls, subclass)Override for issubclass(subclass, cls).
__repr__() → strrepr of the job
shebang(scheduler) → strThe shebang of the wrapped script
clean(retry=False)Clean up the meta files
retry(optional) — Whether clean it for retrying
wrapped_script(scheduler)Get the wrapped script
scheduler(Scheduler) — The scheduler
The path of the wrapped script
Log message for the jobs
level(int | str) — The log level of the recordmsg(str) — The message to log*args— The arguments to format the messagelimit(int, optional) — limitation of the log (don't log for all jobs)limit_indicator(bool, optional) — Whether to show an indicator saying the loghas been limited (the level of the indicator will be DEBUG)logger(LoggerAdapter, optional) — The logger used to log
pipen.scheduler.get_scheduler(scheduler)
Get the scheduler by name of the scheduler class itself
scheduler(Union) — The scheduler class or name
The scheduler class