Skip to content

pipen.scheduler

module

pipen.scheduler

</>

Provide builting schedulers

Classes
Functions
  • get_scheduler(scheduler) (Type) — Get the scheduler by name of the scheduler class itself</>
class

pipen.scheduler.SchedulerPostInit()

</>

Provides post init function for all schedulers

class

pipen.scheduler.LocalScheduler(workdir, forks=1, error_strategy='ignore', num_retries=3, prescript='', postscript='', jobname_prefix=None, **kwargs)

</>
Bases
pipen.scheduler.SchedulerPostInit xqute.schedulers.local_scheduler.LocalScheduler xqute.scheduler.Scheduler

Local scheduler

Parameters
  • workdir (str | PathType) — The working directory
  • forks (int, optional) — Max number of job forks
  • error_strategy (str, optional) — The strategy when there is error happened
  • num_retries (int, optional) — Max number of retries when error_strategy is retry
  • prescript (str, optional) — The prescript to run before the job commandIt is a piece of script that inserted into the wrapper script, running on the scheduler system.
  • postscript (str, optional) — The postscript to run when job finishedIt is a piece of script that inserted into the wrapper script, running on the scheduler system.
  • jobname_prefix (str | none, optional) — The prefix for the job name
  • **kwargs — Other arguments for the scheduler
Attributes
  • job_class — The job class
  • jobcmd_wrapper_init — The init script for the job command wrapper
  • jobcmd_wrapper_init (str) — The init script for the job command wrapper</>
  • name — The name of the scheduler
Methods
method

create_job(index, cmd)

</>

Create a job

Parameters
  • index (int) — The index of the job
  • cmd (Union) — The command of the job
Returns (Job)

The job

method

submit_job_and_update_status(job)

</>

Submit and update the status

  1. Check if the job is already submitted or running
  2. If not, run the hook
  3. If the hook is not cancelled, clean the job
  4. Submit the job, raising an exception if it fails
  5. If the job is submitted successfully, update the status
  6. If the job fails to submit, update the status and write stderr to the job file
Parameters
  • job (Job) — The job
method

retry_job(job)

</>

Retry a job

Parameters
  • job (Job) — The job
method

kill_job_and_update_status(job)

</>

Kill a job and update its status

Parameters
  • job (Job) — The job
method

polling_jobs(jobs, on)

</>

Check if all jobs are done or new jobs can submit

Parameters
  • jobs (List) — The list of jobs
  • on (str) — query on status: submittable or all_done
Returns (bool)

True if yes otherwise False.

method

kill_running_jobs(jobs)

</>

Try to kill all running jobs

Parameters
  • jobs (List) — The list of jobs
method

job_is_submitted_or_running(job)

</>

Check if a job is already submitted or running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_shebang(job) → str

</>

The shebang of the wrapper script

method

jobcmd_init(job) → str

</>

The job command init

method

jobcmd_prep(job) → str

</>

The job command preparation

method

jobcmd_end(job) → str

</>

The job command end

method

wrap_job_script(job)

</>

Wrap the job script

Parameters
  • job (Job) — The job
Returns (str)

The wrapped script

method

wrapped_job_script(job)

</>

Get the wrapped job script

Parameters
  • job (Job) — The job
Returns (DualPath)

The path of the wrapped job script

method

submit_job(job)

</>

Submit a job locally

Parameters
  • job (Job) — The job
Returns (int)

The process id

method

kill_job(job)

</>

Kill a job asynchronously

Parameters
  • job (Job) — The job
method

job_is_running(job)

</>

Tell if a job is really running, not only the job.jid_file

In case where the jid file is not cleaned when job is done.

Parameters
  • job (Job) — The job
Returns (bool)

True if it is, otherwise False

class

pipen.scheduler.SgeScheduler(*args, **kwargs)

</>
Bases
pipen.scheduler.SchedulerPostInit xqute.schedulers.sge_scheduler.SgeScheduler xqute.scheduler.Scheduler

SGE scheduler

Parameters
  • **kwargs — Other arguments for the scheduler
Attributes
  • job_class — The job class
  • jobcmd_wrapper_init — The init script for the job command wrapper
  • jobcmd_wrapper_init (str) — The init script for the job command wrapper</>
  • name — The name of the scheduler
Methods
method

create_job(index, cmd)

</>

Create a job

Parameters
  • index (int) — The index of the job
  • cmd (Union) — The command of the job
Returns (Job)

The job

method

submit_job_and_update_status(job)

</>

Submit and update the status

  1. Check if the job is already submitted or running
  2. If not, run the hook
  3. If the hook is not cancelled, clean the job
  4. Submit the job, raising an exception if it fails
  5. If the job is submitted successfully, update the status
  6. If the job fails to submit, update the status and write stderr to the job file
Parameters
  • job (Job) — The job
method

retry_job(job)

</>

Retry a job

Parameters
  • job (Job) — The job
method

kill_job_and_update_status(job)

</>

Kill a job and update its status

Parameters
  • job (Job) — The job
method

polling_jobs(jobs, on)

</>

Check if all jobs are done or new jobs can submit

Parameters
  • jobs (List) — The list of jobs
  • on (str) — query on status: submittable or all_done
Returns (bool)

True if yes otherwise False.

method

kill_running_jobs(jobs)

</>

Try to kill all running jobs

Parameters
  • jobs (List) — The list of jobs
method

job_is_submitted_or_running(job)

</>

Check if a job is already submitted or running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_init(job) → str

</>

The job command init

method

jobcmd_prep(job) → str

</>

The job command preparation

method

jobcmd_end(job) → str

</>

The job command end

method

wrap_job_script(job)

</>

Wrap the job script

Parameters
  • job (Job) — The job
Returns (str)

The wrapped script

method

wrapped_job_script(job)

</>

Get the wrapped job script

Parameters
  • job (Job) — The job
Returns (DualPath)

The path of the wrapped job script

method

jobcmd_shebang(job) → str

</>

The shebang of the wrapper script

method

submit_job(job)

</>

Submit a job to SGE

Parameters
  • job (Job) — The job
Returns (str)

The job id

method

kill_job(job)

</>

Kill a job on SGE

Parameters
  • job (Job) — The job
method

job_is_running(job)

</>

Tell if a job is really running, not only the job.jid_file

In case where the jid file is not cleaned when job is done.

Parameters
  • job (Job) — The job
Returns (bool)

True if it is, otherwise False

class

pipen.scheduler.SlurmScheduler(*args, **kwargs)

</>
Bases
pipen.scheduler.SchedulerPostInit xqute.schedulers.slurm_scheduler.SlurmScheduler xqute.scheduler.Scheduler

Slurm scheduler

Parameters
  • **kwargs — Other arguments for the scheduler
Attributes
  • job_class — The job class
  • jobcmd_wrapper_init — The init script for the job command wrapper
  • jobcmd_wrapper_init (str) — The init script for the job command wrapper</>
  • name — The name of the scheduler
Methods
method

create_job(index, cmd)

</>

Create a job

Parameters
  • index (int) — The index of the job
  • cmd (Union) — The command of the job
Returns (Job)

The job

method

submit_job_and_update_status(job)

</>

Submit and update the status

  1. Check if the job is already submitted or running
  2. If not, run the hook
  3. If the hook is not cancelled, clean the job
  4. Submit the job, raising an exception if it fails
  5. If the job is submitted successfully, update the status
  6. If the job fails to submit, update the status and write stderr to the job file
Parameters
  • job (Job) — The job
method

retry_job(job)

</>

Retry a job

Parameters
  • job (Job) — The job
method

kill_job_and_update_status(job)

</>

Kill a job and update its status

Parameters
  • job (Job) — The job
method

polling_jobs(jobs, on)

</>

Check if all jobs are done or new jobs can submit

Parameters
  • jobs (List) — The list of jobs
  • on (str) — query on status: submittable or all_done
Returns (bool)

True if yes otherwise False.

method

kill_running_jobs(jobs)

</>

Try to kill all running jobs

Parameters
  • jobs (List) — The list of jobs
method

job_is_submitted_or_running(job)

</>

Check if a job is already submitted or running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_init(job) → str

</>

The job command init

method

jobcmd_prep(job) → str

</>

The job command preparation

method

jobcmd_end(job) → str

</>

The job command end

method

wrap_job_script(job)

</>

Wrap the job script

Parameters
  • job (Job) — The job
Returns (str)

The wrapped script

method

wrapped_job_script(job)

</>

Get the wrapped job script

Parameters
  • job (Job) — The job
Returns (DualPath)

The path of the wrapped job script

method

jobcmd_shebang(job) → str

</>

The shebang of the wrapper script

method

submit_job(job)

</>

Submit a job to Slurm

Parameters
  • job (Job) — The job
Returns (str)

The job id

method

kill_job(job)

</>

Kill a job on Slurm

Parameters
  • job (Job) — The job
method

job_is_running(job)

</>

Tell if a job is really running, not only the job.jid_file

In case where the jid file is not cleaned when job is done.

Parameters
  • job (Job) — The job
Returns (bool)

True if it is, otherwise False

class

pipen.scheduler.SshScheduler(*args, **kwargs)

</>
Bases
pipen.scheduler.SchedulerPostInit xqute.schedulers.ssh_scheduler.scheduler.SshScheduler xqute.scheduler.Scheduler

SSH scheduler

Parameters
  • **kwargs — Other arguments for the scheduler
Attributes
  • job_class — The job class
  • jobcmd_wrapper_init — The init script for the job command wrapper
  • jobcmd_wrapper_init (str) — The init script for the job command wrapper</>
  • name — The name of the scheduler
Methods
method

create_job(index, cmd)

</>

Create a job

Parameters
  • index (int) — The index of the job
  • cmd (Union) — The command of the job
Returns (Job)

The job

method

submit_job_and_update_status(job)

</>

Submit and update the status

  1. Check if the job is already submitted or running
  2. If not, run the hook
  3. If the hook is not cancelled, clean the job
  4. Submit the job, raising an exception if it fails
  5. If the job is submitted successfully, update the status
  6. If the job fails to submit, update the status and write stderr to the job file
Parameters
  • job (Job) — The job
method

retry_job(job)

</>

Retry a job

Parameters
  • job (Job) — The job
method

kill_job_and_update_status(job)

</>

Kill a job and update its status

Parameters
  • job (Job) — The job
method

polling_jobs(jobs, on)

</>

Check if all jobs are done or new jobs can submit

Parameters
  • jobs (List) — The list of jobs
  • on (str) — query on status: submittable or all_done
Returns (bool)

True if yes otherwise False.

method

kill_running_jobs(jobs)

</>

Try to kill all running jobs

Parameters
  • jobs (List) — The list of jobs
method

job_is_submitted_or_running(job)

</>

Check if a job is already submitted or running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_shebang(job) → str

</>

The shebang of the wrapper script

method

jobcmd_init(job) → str

</>

The job command init

method

jobcmd_prep(job) → str

</>

The job command preparation

method

jobcmd_end(job) → str

</>

The job command end

method

wrap_job_script(job)

</>

Wrap the job script

Parameters
  • job (Job) — The job
Returns (str)

The wrapped script

method

wrapped_job_script(job)

</>

Get the wrapped job script

Parameters
  • job (Job) — The job
Returns (DualPath)

The path of the wrapped job script

method

submit_job(job)

</>

Submit a job to SSH

Parameters
  • job (Job) — The job
Returns (str)

The job id

method

kill_job(job)

</>

Kill a job on SSH

Parameters
  • job (Job) — The job
method

job_is_running(job)

</>

Tell if a job is really running, not only the job.jid_file

In case where the jid file is not cleaned when job is done.

Parameters
  • job (Job) — The job
Returns (bool)

True if it is, otherwise False

class

pipen.scheduler.GbatchScheduler(*args, project, location, fast_mount=None, **kwargs)

</>
Bases
pipen.scheduler.SchedulerPostInit xqute.schedulers.gbatch_scheduler.GbatchScheduler xqute.scheduler.Scheduler

Google Cloud Batch scheduler

Parameters
  • **kwargs — Other arguments for the scheduler
Attributes
  • jobcmd_wrapper_init — The init script for the job command wrapper
  • name — The name of the scheduler
Methods
method

create_job(index, cmd)

</>

Create a job

Parameters
  • index (int) — The index of the job
  • cmd (Union) — The command of the job
Returns (Job)

The job

method

submit_job_and_update_status(job)

</>

Submit and update the status

  1. Check if the job is already submitted or running
  2. If not, run the hook
  3. If the hook is not cancelled, clean the job
  4. Submit the job, raising an exception if it fails
  5. If the job is submitted successfully, update the status
  6. If the job fails to submit, update the status and write stderr to the job file
Parameters
  • job (Job) — The job
method

retry_job(job)

</>

Retry a job

Parameters
  • job (Job) — The job
method

kill_job_and_update_status(job)

</>

Kill a job and update its status

Parameters
  • job (Job) — The job
method

polling_jobs(jobs, on)

</>

Check if all jobs are done or new jobs can submit

Parameters
  • jobs (List) — The list of jobs
  • on (str) — query on status: submittable or all_done
Returns (bool)

True if yes otherwise False.

method

kill_running_jobs(jobs)

</>

Try to kill all running jobs

Parameters
  • jobs (List) — The list of jobs
method

job_is_submitted_or_running(job)

</>

Check if a job is already submitted or running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_shebang(job) → str

</>

The shebang of the wrapper script

method

jobcmd_init(job) → str

</>

The job command init

method

jobcmd_prep(job) → str

</>

The job command preparation

method

jobcmd_end(job) → str

</>

The job command end

method

wrap_job_script(job)

</>

Wrap the job script

Parameters
  • job (Job) — The job
Returns (str)

The wrapped script

method

wrapped_job_script(job)

</>

Get the wrapped job script

Parameters
  • job (Job) — The job
Returns (DualPath)

The path of the wrapped job script

method

submit_job(job)

</>

Submit a job

Parameters
  • job (Job) — The job
Returns (str)

The unique id in the scheduler system

method

kill_job(job)

</>

Kill a job

Parameters
  • job (Job) — The job
method

job_is_running(job)

</>

Check if a job is really running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

function

pipen.scheduler.get_scheduler(scheduler)

</>

Get the scheduler by name of the scheduler class itself

Parameters
  • scheduler (Union) — The scheduler class or name
Returns (Type)

The scheduler class