Want to know if this job is worth applying to?
United States
Remote only
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Staff Software Engineer, Tech Foundations based in the United States.
As a Staff Software Engineer, you will shape the technical direction of a large-scale Observability platform supporting critical production systems. You will lead the design, development, and operation of distributed infrastructure that processes millions of telemetry events every second. Working across metrics, logs, and distributed tracing, you will help engineering teams gain deeper insights into system performance, reliability, and operational costs. You will also contribute to building AI-ready observability capabilities, enabling autonomous agents to access operational data safely and efficiently. In this highly collaborative environment, you will influence architectural decisions, mentor engineers, and drive innovation across infrastructure teams. This is an opportunity to solve complex technical challenges while improving developer productivity and system resilience at global scale.
Technical Leadership & Architecture: Define the technical vision and architecture for major areas of the Observability platform, leading initiatives from initial design and prototyping through production deployment and ongoing operations.
Distributed Systems Development: Design, build, and maintain highly scalable and reliable distributed systems capable of processing tens of millions of events per second.
Metrics Infrastructure: Scale and optimize the metrics pipeline using technologies such as OpenTelemetry, vmagent, and Grafana Mimir, while managing infrastructure costs through aggregation, sampling, and unused-metric filtering.
Logging & Tracing Modernization: Lead the transition from legacy log aggregation systems to OpenTelemetry-based pipelines and improve log and trace storage capabilities using technologies such as OpenSearch.
Query Infrastructure: Develop fast, reliable, and secure query systems for engineers and AI agents, incorporating rate limiting, access controls, isolation, and caching to protect critical incident-response workflows.
Telemetry Cost Optimization: Build self-service tools that provide engineering teams with visibility into telemetry costs, including cost attribution by team and service, as well as controls for data granularity, sampling, and retention.
Observability & Reliability: Improve incident detection through anomaly detection, automated signal correlation, and enhanced monitoring capabilities.
Cross-functional Collaboration: Align technical roadmaps with infrastructure and product teams, influence build-versus-buy decisions, and communicate architectural tradeoffs and investment priorities to engineering leadership.
Engineering Excellence: Write high-quality, maintainable, and well-tested code, participate in design reviews, improve operational readiness, and contribute to relevant open-source projects.
Mentorship & Knowledge Sharing: Mentor engineers, strengthen technical practices across the team, and help developers instrument services and independently investigate production issues.
Professional Experience: At least 9 years of software engineering experience, with a proven track record of designing, building, and operating large-scale distributed systems in production environments.
Technical Leadership: Demonstrated experience leading the technical direction of complex systems or multi-team initiatives, including architecture design, stakeholder alignment, delivery, and production operations.
Observability Expertise: Deep experience in at least one major telemetry domain at scale, including metrics platforms such as Prometheus, Grafana Mimir, or VictoriaMetrics; logging technologies such as OpenSearch or Kafka; or distributed tracing solutions such as OpenTelemetry, Jaeger, or ClickHouse.
Programming Skills: Strong software development capabilities in Go or Java/Kotlin, with the ability to navigate unfamiliar codebases and contribute to upstream open-source projects.
Cloud Infrastructure: Hands-on experience with cloud-native technologies, particularly Kubernetes, AWS, and service mesh solutions such as Istio or Envoy.
Communication & Collaboration: Excellent written and verbal communication skills, including the ability to produce clear technical design documents, explain architectural tradeoffs, and present investment recommendations to engineering leadership.
Problem-solving: Strong analytical and architectural thinking, with the ability to address complex scalability, reliability, performance, and cost challenges.
Open-source Contributions (Preferred): Experience contributing to OpenTelemetry, Prometheus, Grafana Mimir, VictoriaMetrics, OpenSearch, Jaeger, or comparable projects.
Advanced Technical Knowledge (Preferred): Familiarity with time-series storage internals, high-cardinality datasets, query optimization, anomaly detection, automated correlation, or SLO-based alerting.
Additional Expertise (Preferred): Experience with telemetry cost attribution, large-scale infrastructure cost optimization, AI-agent-facing APIs or tools (including MCP servers), and columnar analytics platforms such as ClickHouse.
Competitive annual base salary ranging from $212,000 to $265,000 USD.
Potential eligibility for performance-based bonuses.
Equity compensation opportunities.
Comprehensive employee benefits package.
Employee Travel Credits.
Remote-first working arrangement for eligible US locations.
Flexibility to work remotely, with occasional office visits or team offsites as agreed with management.
Opportunities to contribute to large-scale engineering initiatives and open-source projects.
Collaborative environment focused on technical innovation, professional development, and knowledge sharing.
Inclusive workplace culture that values diverse perspectives and equal employment opportunities.
