The forge
Scheduling
Who goes first, where jobs land, and how to predict when yours starts.
The scheduler has one queue per forge. These are all of its rules, so you can predict what it will do.
Order
| Order | Plans | Rule |
|---|---|---|
| First | Anvil | Runs at once on the team's own node. Other teams never see that queue. |
| Second | Strike and Hour bank | Share one queue, first in, first out, on GPUs that are not Anvils. |
| Last | Ember | Runs on any GPU that is idle right now. Gives way to everything above. |
All at once
A job starts only when every GPU it asked for is free together. A 4 GPU job does not start on 3. The job behind it does not jump ahead, even if it would fit: strict order means a big job is never starved by small ones.
Where a job lands
Up to 8 GPUs, one node, packed onto GPUs that share an NVSwitch path. 16 GPUs, both nodes, which only happens when neither node is an Anvil.
Fair share
No team holds more than 12 of the 16 GPUs on Strike and Hour bank at once. Jobs past that wait, even if GPUs are free, so one team cannot fill the forge with a sweep. Anvils do not count toward this.
Preemption
Only Ember is preempted: SIGTERM, 30 seconds, SIGKILL. Strike, Bank and Anvil jobs end only when they exit, hit their limit or are stopped.
Maintenance
Before a window the forge drains: no new jobs that would run into it, Ember gets its notice at the start, and Strike and Bank jobs whose limit ends before the window keep running.
Predicting a start
GET /forges shows free GPUs and an estimated wait per plan. GET /jobs/{id} on a queued job shows queue_ahead, the number of GPU requests in front of it.