Telemetry

Every GPU in every forge reports once a second. That one feed lights the bars on our home page, the status page, your console, the ledger and the API. There is no second set of numbers.

#What each sample holds

#Messages

{ "type": "heat", "forge": "f1", "ts": 1792695600000,
  "gpus": [
    { "id": "gpu3", "node": "a", "util": 94.0, "mem_used": 152.4,
      "temp_c": 69.5, "power_w": 948, "sm_clock": 1965,
      "nvlink_gbps": 712, "job": "job_2a4k1" }
  ] }

{ "type": "job.start", "job": "job_2a4k1", "gpus": ["gpu0", "gpu1"], "ts": 1792695601000 }
{ "type": "job.end",   "job": "job_2a4k1", "result": "succeeded", "ts": 1792702800000 }
{ "type": "gpu.fault", "gpu": "gpu11", "reason": "fell off bus", "ts": 1792703000000 }

Timestamps are Unix milliseconds. Other teams' GPUs appear without a job id.

#Reading it

GET /forges/f1/heat?window=10m for the forge, GET /jobs/{id}/heat for one job. Windows up to 1 hour return every second; up to 7 days, one sample per minute (mean); longer, one per 10 minutes.

#Streaming

websocat -H "Authorization: Bearer $BF_KEY" wss://feed.useblackforge.com/f1

The stream sends the messages above as they happen, with no averaging.

#How it is collected

A small agent on each node reads NVML directly once a second and pushes to the feed service. If the link drops it buffers 10 minutes and replays. It never looks inside containers.

#Retention

Per second samples for 14 days, per minute for 18 months. Job events for as long as the ledger, 18 months.