Skip to content

xqute.schedulers.gbatch_scheduler

module

xqute.schedulers.gbatch_scheduler

</>
Classes
class

xqute.schedulers.gbatch_scheduler.GbatchScheduler(*args, project, location, mount=None, service_account=None, network=None, subnetwork=None, no_external_ip_address=None, machine_type=None, provisioning_model=None, image_uri=None, entrypoint=None, commands=None, runnables=None, **kwargs)

</>
Bases
xqute.scheduler.Scheduler

Scheduler for Google Cloud Batch

You can pass extra configuration parameters to the constructor that will be used in the job configuration file. For example, you can pass taskGroups to specify the task groups and their specifications.

For using containers, it is a little bit tricky to specify the commands. When no entrypoint is specified, the commands should be a list with the first element being the interpreter (e.g. /bin/bash) and the second element being the path to the wrapped job script. If the entrypoint is specified, we can use the {lang} and {script} placeholders in the commands list, where {lang} will be replaced with the interpreter (e.g. /bin/bash) and {script} will be replaced with the path to the wrapped job script. With entrypoint specified and no {script} placeholder, the joined command will be the interpreter followed by the path to the wrapped job script will be appended to the commands list.

Parameters
  • **kwargs — Other arguments for the scheduler
  • project (str) — GCP project ID
  • location (str) — GCP location (e.g. us-central1)
  • mount (Union, optional) — GCS path to mount (e.g. gs://my-bucket:/mnt/my-bucket)You can pass a list of mounts. You can also use named mount like NAME=gs://bucket/dir then it will be mounted to /mnt/disks/NAME in the container. You can use environment variable NAME in your job scripts to refer to the mounted path.
  • service_account (str | none, optional) — GCP service account email (e.g. test-account@example.com)
  • network (str | none, optional) — GCP network (e.g. default-network)
  • subnetwork (str | none, optional) — GCP subnetwork (e.g. regions/us-central1/subnetworks/default)
  • no_external_ip_address (bool | none, optional) — Whether to disable external IP address
  • machine_type (str | none, optional) — GCP machine type (e.g. e2-standard-4)
  • provisioning_model (str | none, optional) — GCP provisioning model (e.g. SPOT)
  • image_uri (str | none, optional) — Container image URI (e.g. ubuntu-2004-lts)
  • entrypoint (str, optional) — Container entrypoint (e.g. /bin/bash)
  • commands (Union, optional) — The command list to run in the container.There are three ways to specify the commands: 1. If no entrypoint is specified, the final command will be [commands, wrapped_script], where the entrypoint is the wrapper script interpreter that is determined by JOBCMD_WRAPPER_LANG (e.g. /bin/bash), commands is the list you provided, and wrapped_script is the path to the wrapped job script. 2. You can specify something like "-c", then the final command will be ["-c", "wrapper_script_interpreter, wrapper_script"] 3. You can use the placeholders {lang} and {script} in the commands list, where {lang} will be replaced with the interpreter (e.g. /bin/bash) and {script} will be replaced with the path to the wrapped job script. For example, you can specify ["{lang} {script}"] and the final command will be ["wrapper_interpreter, wrapper_script"]
  • runnables (Optional, optional) — Additional runnables to run before or after the main job.Each runnable should be a dictionary that follows the GCP Batch API specification. You can also specify an "order" key in the dictionary to control the execution order of the runnables. Runnables with negative order will be executed before the main job, and those with non-negative order will be executed after the main job. The main job runnable will always be executed in the order it is defined in the list.
Attributes
  • jobcmd_wrapper_init — The init script for the job command wrapper
  • jobcmd_wrapper_init (str) — The init script for the job command wrapper</>
  • name — The name of the scheduler
Methods
method

create_job(index, cmd, envs=None)

</>

Create a job

Parameters
  • index (int) — The index of the job
  • cmd (Union) — The command of the job
Returns (Job)

The job

method

submit_job_and_update_status(job)

</>

Submit and update the status

  1. Check if the job is already submitted or running
  2. If not, run the hook
  3. If the hook is not cancelled, clean the job
  4. Submit the job, raising an exception if it fails
  5. If the job is submitted successfully, update the status
  6. If the job fails to submit, update the status and write stderr to the job file
Parameters
  • job (Job) — The job
method

retry_job(job)

</>

Retry a job

Parameters
  • job (Job) — The job
method

kill_job_and_update_status(job)

</>

Kill a job and update its status

Parameters
  • job (Job) — The job
method

polling_jobs(jobs, on, polling_counter)

</>

Check if all jobs are done or new jobs can submit

Parameters
  • jobs (List) — The list of jobs
  • on (str) — query on status: submittable or all_done
  • polling_counter (int) — The polling counter, used to limit the number of polls orskip some polls if the scheduler is busy.
Returns (bool)

True if yes otherwise False.

method

kill_running_jobs(jobs)

</>

Try to kill all running jobs

Parameters
  • jobs (List) — The list of jobs
method

job_is_submitted_or_running(job)

</>

Check if a job is already submitted or running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_shebang(job) → str

</>

The shebang of the wrapper script

method

jobcmd_prep(job) → str

</>

The job command preparation

method

jobcmd_end(job) → str

</>

The job command end

method

wrap_job_script(job)

</>

Wrap the job script

Parameters
  • job (Job) — The job
Returns (str)

The wrapped script

method

wrapped_job_script(job)

</>

Get the wrapped job script

Parameters
  • job (Job) — The job
Returns (SpecPath)

The path of the wrapped job script

method

submit_job(job)

</>

Submit a job

Parameters
  • job (Job) — The job
Returns (str)

The unique id in the scheduler system

method

kill_job(job)

</>

Kill a job

Parameters
  • job (Job) — The job
method

job_fails_before_running(job)

</>

Check if a job fails before running.

For some schedulers, the job might fail before running (after submission). For example, the job might fail to allocate resources. In such a case, the wrapped script might not be executed, and the job status will not be updated (stays in SUBMITTED). We need to check such jobs and mark them as FAILED.

For the instant scheduler, for example, the local scheduler, the failure will be immediately reported when submitting the job, so we don't need to check such jobs.

Parameters
  • job (Job) — The job to check
Returns (bool)

True if the job fails before running, otherwise False.

method

job_is_running(job)

</>

Check if a job is really running

Parameters
  • job (Job) — The job
Returns (bool)

True if yes otherwise False.

method

jobcmd_init(job) → str

</>

The job command init