Running work
Jobs
What a job is, the fields it takes, its limits and how it ends.
A job is one container run on 1 to 16 GPUs in one forge, under one plan, with a time limit. You start it with forge run or POST /jobs, and it ends when the command exits, the limit is reached, you stop it, or (on Ember) the GPU is needed back.
Fields
| Field | Required | Meaning |
|---|---|---|
image | Yes | OCI image reference, public or from a registry you added in the console. |
command | Yes | Array of strings. Replaces the image entrypoint. |
gpus | Yes | 1 to 8 on one node, or 16 for both nodes. |
plan | Yes | strike, bank, anvil or ember. |
limit | Yes | Duration such as 45m, 6h or 3d. Up to 14d, or to the end of your Anvil term. |
env | No | Object of environment variables. Secrets go in the console instead. |
ports | No | Ports to expose over HTTPS, for inference servers. See Network. |
resume | No | Ember only. Requeue after preemption. Default false. |
tags | No | Free form object, returned unchanged and shown on invoices. |
Placement
A job that asks for 8 GPUs or fewer always lands on one node, on one NVLink fabric. A 16 GPU job takes both nodes and gets NCCL over InfiniBand between them. A job never starts partly: if it asked for 4 GPUs and 3 are free, it waits. See Scheduling.
Limits
limit is required so a forgotten job cannot run up a bill. When it is reached the container gets SIGTERM, then SIGKILL 20 seconds later, and the result is stopped.
Results
| Result | When | Billed |
|---|---|---|
succeeded | Container exited 0. | Yes |
failed | Container exited non zero. | Yes |
stopped | You stopped it, or the limit was reached. | Yes, to the stop |
preempted | Ember job gave its GPU back. | Yes, to the notice |
hardware | A GPU or node fault on our side ended it. | No, and bank hours are returned |
Stopping and retrying
forge stop job_id or POST /jobs/{id}/stop ends a job in any stage. A job in queue is removed and never billed. We do not retry failed jobs for you, except Ember jobs with resume: true.
History
Jobs and their logs stay visible in the console and API for 60 days. Heat samples for a job stay for as long as the feed keeps them, see Telemetry.