Ember jobs

Ember jobs run on GPUs that are idle right now: free capacity, and Anvil GPUs their owners lend out. They cost less than half of Strike. The catch is that they give the GPU back when someone paying more needs it.

#Preemption

When an Ember GPU is needed, PID 1 in your container gets SIGTERM. You have 30 seconds to save state, then SIGKILL. The result is preempted and billing stops at the moment of the notice.

#Write for it

import os, signal, sys

def save_and_exit(*_):
    trainer.save_checkpoint("/shared/ckpt/latest.pt")
    sys.exit(0)

signal.signal(signal.SIGTERM, save_and_exit)

if os.environ.get("BF_RESUME") == "1":
    trainer.load_checkpoint("/shared/ckpt/latest.pt")

#Resume

With resume: true (CLI --resume), a preempted job goes back to the front of the Ember queue and starts again from the image with BF_RESUME=1 set. It keeps the same job id, and its billed seconds add up across runs.

#What fits

Evaluation sweeps, embedding large corpora, hyperparameter searches, data processing on GPU, anything that checkpoints at least every few minutes. Do not put an inference endpoint or a run with no checkpoints on Ember.

#How often it happens

It depends on how full the forge is. The heat ledger publishes the median Ember run length between preemptions each month, so you can plan around it.