Want to know if this job is worth applying to?
Global Remote / San Francisco, CA
Hybrid
About Andromeda
Andromeda gives AI companies access to the kind of scaled compute once reserved for hyperscalers. Our platform connects 100+ AI customers to 50+ global providers, with billions of GPU-hours supported, and those numbers are all rapidly growing. We combine enterprise-grade reliability with the speed and economics of an open market, serving teams running everything from large-scale training to production inference.
Nat Friedman (former CEO of GitHub) and Daniel Gross (former head of AI at Apple, YC partner) started Andromeda in 2023 with a single GPU cluster. It filled almost immediately. Three years later, we're a $1.5B company, profitable since day one, with a Series A from Paradigm to scale the platform globally. The global flow of compute is already a multi-trillion dollar market, and our team is building the infrastructure that enables it to continue to scale.
The problem is deceptively hard. Not all compute is equal: interconnect, networking, OEM, firmware, and cluster age all vary across providers, and the differences matter at scale. Our platform benchmarks and validates capacity, takes positions, structures contracts, and operates clusters globally, delivering a consistent product regardless of where it runs. No one else has built this layer, and the AI industry can't scale without it.
The Role
Our customers run large AI training and inference workloads on GPU clusters we source from providers worldwide. When a node goes dark or a job dies eight hours into a run, the Customer Reliability Engineer is who they hear from, and who gets it sorted.
The job has three parts. You triage incoming issues and debug them at the Linux and Kubernetes layer. You work provider-side to figure out whose fault something actually is and push external providers to fix it. And you build the monitoring and scripts that catch problems before a customer has to tell us.
You need to be comfortable in a Linux shell and know how Kubernetes works. You don't need GPU or HPC experience. Most people pick that up here.
What You’ll Do
Triage and fix customer issues
Own issues start to finish: reproduce, diagnose, fix or escalate, close the loop
Debug at the Linux layer: processes, networking, storage, kernel logs, resource contention, systemd, journald
Dig into Kubernetes problems like pods stuck pending or crash-looping, node conditions, scheduling failures, resource limits
Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, nodes that need cordoning or draining, jobs failing across multiple nodes
Escalate when you're past your depth, with the evidence already gathered
Handle incidents
Take part in a 24/7 on-call rotation
First response on alerts and customer-reported outages: assess impact, set severity, pull in the right people
Keep customers updated during incidents. Clear status, honest unknowns, no silence
Write up what happened, then turn it into a runbook, an alert, or a fix so it costs less next time
Push providers to resolution
Work out whether a fault is provider-side, ours, or the customer's before it gets handed anywhere
Open tickets with compute providers and chase them down rather than waiting
Track recurring provider failures and flag the patterns to the people making sourcing decisions
Build the tooling
Write Python or Bash to automate the checks you'd otherwise run by hand
Build and improve monitoring: cluster and node health checks, GPU telemetry, dashboards, alerts that fire on real problems
Keep runbooks and customer docs current as you go
What We’re Looking For
Real Linux troubleshooting ability from the command line. You can work a problem through logs, processes, networking, and disk without a script to follow
Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and how to investigate when one of those breaks
Can write a script in Python or Bash to automate something repetitive
Strong writing. You can explain a technical problem to a frustrated customer clearly and without condescension
Good judgment under pressure. You know what to check first, when to escalate, and how to keep people informed while you're still working it out
Willing to join a 24/7 on-call rotation
Strong Candidates May Have
Hands-on time with NVIDIA GPUs in production: drivers, CUDA, DCGM, the Kubernetes device plugin
Experience with high-performance networking (InfiniBand, RoCE) or NCCL
Experience with HPC or batch schedulers like Slurm
A previous customer-facing technical role: support engineering, TAM, solutions, professional services
Knowledge of Prometheus, Grafana, Datadog, or similar
IAC: Terraform, Ansible, or Helm
Genuine interest in AI infrastructure and how big training jobs behave
Why You’ll Love It Here
High-growth environment: Get in early at a company at the center of the AI infrastructure boom
Competitive compensation: + meaningful equity
Comprehensive benefits: for you and your dependents, including healthcare, dental, and vision coverage, 401(k), and unlimited PTO
Andromeda Cluster is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.