xqute
xqute
A job management system for python
Utilities for xqute
DuplicateFilter
— Filter instances are used to perform arbitrary filtering of LogRecords.</>
Hook specifications for scheduler plugins
on_init
(
xqute
)
— When xqute is initialized</>on_job_failed
(
scheduler
,job
)
— When the job is failed</>on_job_init
(
scheduler
,job
)
— When the job is initialized</>on_job_killed
(
scheduler
,job
)
— When the job is killed</>on_job_killing
(
scheduler
,job
)
— When the job is being killed</>on_job_polling
(
scheduler
,job
)
— When the system is polling job status</>on_job_queued
(
scheduler
,job
)
— When the job is queued</>on_job_started
(
scheduler
,job
)
— When the job starts to run.</>on_job_submitted
(
scheduler
,job
)
— When the job is submitted</>on_job_submitting
(
scheduler
,job
)
— When the job is to be submitted</>on_job_succeeded
(
scheduler
,job
)
— When the job is succeeded</>on_jobcmd_end
(
scheduler
,job
)
(str) — When the job command finishes and after the postscript is runThis should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available:status
andjob
.status
is the classJobStatus
fromxqute.defaults.py
andjob
is theJob
instance. The bash variable$rc
is accessible in the context, which is the return code of the job command. For multiple plugins, the code will be inserted in the order of the plugin priority. </>on_jobcmd_init
(
scheduler
,job
)
(str) — When the job command wrapper script is initialized before the prescript is runThis should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available:status
andjob
.status
is the classJobStatus
fromxqute.defaults.py
andjob
is theJob
instance. For multiple plugins, the code will be inserted in the order of the plugin priority. </>on_jobcmd_prep
(
scheduler
,job
)
(str) — When the job command right about to be runThis should return a piece of bash code to be inserted in the wrapped job script (template), which is a python template string, with the following variables available:status
andjob
.status
is the classJobStatus
fromxqute.defaults.py
andjob
is theJob
instance. The bash variable$cmd
is accessible in the context. It is also possible to modify thecmd
variable. Just remember to assign the modified value tocmd
. For multiple plugins, the code will be inserted in the order of the plugin priority. Keep in mind that the$cmd
may be modified by other plugins. </>on_shutdown
(
xqute
,sig
)
— When xqute is shutting down</>
Default settings and utilities for xqute
DEFAULT_ERROR_STRATEGY
(str) — The default strategy when there iserror happenedDEFAULT_JOB_CMD_WRAPPER_SHELL
— The default shell for job wrapperDEFAULT_NUM_RETRIES
(int) — Default number of retries whenDEFAULT_ERROR_STRATEGY is retryDEFAULT_SCHEDULER_FORKS
(int) — Default number of job forks for schedulerDEFAULT_SUBMISSION_BATCH
(int) — Default consumer workersDEFAULT_WORKDIR
— The default work directory for jobs to save the metadata
JobErrorStrategy
— The strategy when error happen from jobs</>JobStatus
— The status of a job</>
get_jobcmd_wrapper_init
(
local
,remove_jid_after_done
)
(str) — Get the job command wrapper initialization script</>
Provides the SpecPath and MountedPath classes.
It is used to represent paths of jobs and it is useful when a job is running in a remote system (a VM, a container, etc.), where we need to mount the paths into the remote system (MountedPath).
But in the system where this framework is running, we need to use the paths (specified directly) that are used in the framework, where we also need to carry the information of the mounted path (SpecPath).
The module provides two main abstract base classes:
- -
MountedPath
: Represents a path as it appears in the remote execution environment. - -
SpecPath
: Represents a path as it appears in the local environment where the
Both classes have implementations for local paths and various cloud storage paths, including:
- - Google Cloud Storage
- - Azure Blob Storage
- - Amazon S3
These classes maintain the relationship between the local and remote pathrepresentations, allowing transparent path operations while preserving both path contexts.
MountedPath
(xqute.path.mountedlocalpath | xqute.path.mountedcloudpath) — A router class to instantiate the correct path based on the path typefor the mounted path. </>MountedLocalPath
— A class to represent a mounted local path</>MountedCloudPath
(
cloud_path
,*args
,**kwargs
)
(Union) — A class to represent a mounted cloud path</>MountedGSPath
(Union) — A class to represent a mounted Google Cloud Storage path</>MountedAzureBlobPath
(Union) — A class to represent a mounted Azure Blob Storage path</>MountedS3Path
(Union) — A class to represent a mounted Amazon S3 path</>SpecPath
(xqute.path.speclocalpath | xqute.path.speccloudpath) — A router class to instantiate the correct path based on the path typefor the spec path. </>SpecLocalPath
— A class to represent a spec local path</>SpecCloudPath
(
cloud_path
,*args
,**kwargs
)
(Union) — A class to represent a spec cloud path</>SpecGSPath
(Union) — A class to represent a spec Google Cloud Storage path</>SpecAzureBlobPath
(Union) — A class to represent a spec Azure Blob Storage path</>SpecS3Path
(Union) — A class to represent a spec Amazon S3 path</>
xqute.schedulers
Builtin schedulers
get_scheduler
(
scheduler
)
(Type) — Get the scheduler class</>