Skip to content

xqute

package

xqute

</>

A job management system for python

module

xqute.xqute

</>

The xqute module

Classes
  • Xqute — The main class of the package</>
module

xqute.scheduler

</>

The scheduler to schedule jobs

Classes
  • Scheduler(workdir, forks, error_strategy, num_retries, prescript, postscript, jobname_prefix, **kwargs) — The abstract class for scheduler</>
module

xqute.utils

</>

Utilities for xqute

Classes
  • DuplicateFilter — Filter instances are used to perform arbitrary filtering of LogRecords.</>
module

xqute.job

</>

Job to execute

Classes
  • Job — The class for job</>
module

xqute.plugin

</>

Hook specifications for scheduler plugins

Functions
  • on_init(xqute) — When xqute is initialized</>
  • on_job_failed(scheduler, job) — When the job is failed</>
  • on_job_init(scheduler, job) — When the job is initialized</>
  • on_job_killed(scheduler, job) — When the job is killed</>
  • on_job_killing(scheduler, job) — When the job is being killed</>
  • on_job_polling(scheduler, job) — When the system is polling job status</>
  • on_job_queued(scheduler, job) — When the job is queued</>
  • on_job_started(scheduler, job) — When the job starts to run.</>
  • on_job_submitted(scheduler, job) — When the job is submitted</>
  • on_job_submitting(scheduler, job) — When the job is to be submitted</>
  • on_job_succeeded(scheduler, job) — When the job is succeeded</>
  • on_jobcmd_end(scheduler, job) (str) — When the job command finishes and after the postscript is runThis should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available: status and job. status is the class JobStatus from xqute.defaults.py and job is the Job instance. The bash variable $rc is accessible in the context, which is the return code of the job command. For multiple plugins, the code will be inserted in the order of the plugin priority. </>
  • on_jobcmd_init(scheduler, job) (str) — When the job command wrapper script is initialized before the prescript is runThis should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available: status and job. status is the class JobStatus from xqute.defaults.py and job is the Job instance. For multiple plugins, the code will be inserted in the order of the plugin priority. </>
  • on_jobcmd_prep(scheduler, job) (str) — When the job command right about to be runThis should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available: status and job. status is the class JobStatus from xqute.defaults.py and job is the Job instance. The bash variable $cmd is accessible in the context. It is also possible to modify the cmd variable. Just remember to assign the modified value to cmd. For multiple plugins, the code will be inserted in the order of the plugin priority. Keep in mind that the $cmd may be modified by other plugins. </>
  • on_shutdown(xqute, sig) — When xqute is shutting down</>
module

xqute.defaults

</>

Default settings and utilities for xqute

Attributes
  • DEFAULT_ERROR_STRATEGY (str) — The default strategy when there iserror happened
  • DEFAULT_JOB_CMD_WRAPPER_SHELL — The default shell for job wrapper
  • DEFAULT_NUM_RETRIES (int) — Default number of retries whenDEFAULT_ERROR_STRATEGY is retry
  • DEFAULT_SCHEDULER_FORKS (int) — Default number of job forks for scheduler
  • DEFAULT_SUBMISSION_BATCH (int) — Default consumer workers
  • DEFAULT_WORKDIR — The default work directory for jobs to save the metadata
Classes
Functions
module

xqute.path

</>

Provides the SpecPath and MountedPath classes.

It is used to represent paths of jobs and it is useful when a job is running in a remote system (a VM, a container, etc.), where we need to mount the paths into the remote system (MountedPath).

But in the system where this framework is running, we need to use the paths (specified directly) that are used in the framework, where we also need to carry the information of the mounted path (SpecPath).

The module provides two main abstract base classes:

  • - MountedPath: Represents a path as it appears in the remote execution environment.
  • - SpecPath: Represents a path as it appears in the local environment where the
framework runs.

Both classes have implementations for local paths and various cloud storage paths, including:

  • - Google Cloud Storage
  • - Azure Blob Storage
  • - Amazon S3

These classes maintain the relationship between the local and remote pathrepresentations, allowing transparent path operations while preserving both path contexts.

Classes
  • MountedPath (xqute.path.mountedlocalpath | xqute.path.mountedcloudpath) — A router class to instantiate the correct path based on the path typefor the mounted path. </>
  • MountedLocalPath — A class to represent a mounted local path</>
  • MountedCloudPath(cloud_path, *args, **kwargs) (Union) — A class to represent a mounted cloud path</>
  • MountedGSPath (Union) — A class to represent a mounted Google Cloud Storage path</>
  • MountedAzureBlobPath (Union) — A class to represent a mounted Azure Blob Storage path</>
  • MountedS3Path (Union) — A class to represent a mounted Amazon S3 path</>
  • SpecPath (xqute.path.speclocalpath | xqute.path.speccloudpath) — A router class to instantiate the correct path based on the path typefor the spec path. </>
  • SpecLocalPath — A class to represent a spec local path</>
  • SpecCloudPath(cloud_path, *args, **kwargs) (Union) — A class to represent a spec cloud path</>
  • SpecGSPath (Union) — A class to represent a spec Google Cloud Storage path</>
  • SpecAzureBlobPath (Union) — A class to represent a spec Azure Blob Storage path</>
  • SpecS3Path (Union) — A class to represent a spec Amazon S3 path</>
package

xqute.schedulers

</>

Builtin schedulers

Functions