projects / gpu-fleet
gpu-fleet
Node lifecycle over Redfish, topology-aware reservations, tenant attestation and automated remediation. Go control plane, Rust node agent, Postgres.
- 88 s Fault to quarantine Recorded chaos run
- 2 s Replacement in the same pod Recorded chaos run
- 50 Simulated nodes Docker Compose demo
- 5 / 10 min Quarantine rate limit Remediation policy
Overview
Problem
Racks of GPU servers are not something a tenant can reserve until software provisions, places, isolates and repairs them.
Approach
A Go control plane drives node lifecycle over Redfish, a Rust agent on each node exports GPU metrics, and Postgres holds the concurrency rules. Simulators stand in for the hardware.
Result
In a recorded chaos run a GPU fell off the bus under a running tenant. The node was quarantined 88 seconds later and a replacement placed in the same pod 2 seconds after that, with no operator.
A kernel implant left by a previous tenant survives a disk wipe, but not attestation.
On tenant isolation
Decisions · why this stack
-
Go for the control plane
The control plane is mostly network I/O and concurrency: fanning out to hundreds of BMCs, talking to Postgres and serving an API. Goroutines fit that shape, and the infrastructure ecosystem (Kubernetes client-go, controller-runtime, Terraform SDK, Redfish libraries) is Go first.
-
Rust for the node agent
The agent runs on every GPU host next to latency-sensitive training jobs, so it must be small, predictable and hard to crash. Rust gives a single static binary with no GC pauses, and unsafe code is confined to the NVML FFI boundary and forbidden elsewhere by crate lints.
-
Postgres as the concurrency engine
Instead of a separate lock service, correctness lives in the database: row locks around every lifecycle event, SKIP LOCKED leases for workers, and an exclusion constraint that rejects overlapping reservations. Losing a placement race becomes a retry, not a bug.
-
Prometheus and Alertmanager for remediation
The agent speaks the dcgm-exporter metric format, so the same dashboards and alerts would work against real NVIDIA hardware. Alertmanager already handles grouping and retries, so remediation is just an authenticated webhook into the control plane.
-
Simulators instead of hardware
A Redfish BMC simulator, a fake GPU backend and a chaos service let the whole system run and be tested without GPUs. Each has a written contract, so swapping in real hardware means implementing an interface, not rewriting the platform.
-
One repository, two languages
Both sides change together while the API between them is still moving, and one CI pipeline keeps them consistent. The cost is two toolchains and an explicit contract, since types cannot be shared across Go and Rust.
Stack
- Go
- Rust
- axum
- tokio
- PostgreSQL
- pgx
- Prometheus
- Alertmanager
- Grafana
- Redfish
- Ed25519
- Docker Compose
- GitHub Actions
Demo
Numbers
Incident timeline
A GPU fell off the bus under a running tenant
- 0 s GPU XID fault
- 88 s Node quarantined
- 90 s Replacement in the same pod