projects / gpu-fleet

Replay Systems 2026

gpu-fleet

Node lifecycle over Redfish, topology-aware reservations, tenant attestation and automated remediation. Go control plane, Rust node agent, Postgres.

gpu-fleet architecture: Rust node agents, Go control plane, Postgres, Prometheus and Alertmanager
  • 88 s Fault to quarantine Recorded chaos run
  • 2 s Replacement in the same pod Recorded chaos run
  • 50 Simulated nodes Docker Compose demo
  • 5 / 10 min Quarantine rate limit Remediation policy

Overview

Problem

Racks of GPU servers are not something a tenant can reserve until software provisions, places, isolates and repairs them.

Approach

A Go control plane drives node lifecycle over Redfish, a Rust agent on each node exports GPU metrics, and Postgres holds the concurrency rules. Simulators stand in for the hardware.

Result

In a recorded chaos run a GPU fell off the bus under a running tenant. The node was quarantined 88 seconds later and a replacement placed in the same pod 2 seconds after that, with no operator.

A kernel implant left by a previous tenant survives a disk wipe, but not attestation.

On tenant isolation

Decisions · why this stack

  1. Go for the control plane

    The control plane is mostly network I/O and concurrency: fanning out to hundreds of BMCs, talking to Postgres and serving an API. Goroutines fit that shape, and the infrastructure ecosystem (Kubernetes client-go, controller-runtime, Terraform SDK, Redfish libraries) is Go first.

  2. Rust for the node agent

    The agent runs on every GPU host next to latency-sensitive training jobs, so it must be small, predictable and hard to crash. Rust gives a single static binary with no GC pauses, and unsafe code is confined to the NVML FFI boundary and forbidden elsewhere by crate lints.

  3. Postgres as the concurrency engine

    Instead of a separate lock service, correctness lives in the database: row locks around every lifecycle event, SKIP LOCKED leases for workers, and an exclusion constraint that rejects overlapping reservations. Losing a placement race becomes a retry, not a bug.

  4. Prometheus and Alertmanager for remediation

    The agent speaks the dcgm-exporter metric format, so the same dashboards and alerts would work against real NVIDIA hardware. Alertmanager already handles grouping and retries, so remediation is just an authenticated webhook into the control plane.

  5. Simulators instead of hardware

    A Redfish BMC simulator, a fake GPU backend and a chaos service let the whole system run and be tested without GPUs. Each has a written contract, so swapping in real hardware means implementing an interface, not rewriting the platform.

  6. One repository, two languages

    Both sides change together while the API between them is still moving, and one CI pipeline keeps them consistent. The cost is two toolchains and an explicit contract, since types cannot be shared across Go and Rust.

Stack

  • Go
  • Rust
  • axum
  • tokio
  • PostgreSQL
  • pgx
  • Prometheus
  • Alertmanager
  • Grafana
  • Redfish
  • Ed25519
  • Docker Compose
  • GitHub Actions

Demo

Numbers

Incident timeline

A GPU fell off the bus under a running tenant

  1. 0 s GPU XID fault
  2. 88 s Node quarantined
  3. 90 s Replacement in the same pod
Recorded chaos run