Sr. Observability Engineering

POSITION SUMMARY 

McKesson is seeking a Senior Observability Engineer to join our Platform Engineering team. In this role, you will serve as a key engineer and practitioner of McKesson’s enterprise observability strategy, supporting the design, deployment, and continuous improvement of our monitoring and telemetry platforms across cloud and on-premises infrastructure. You will operate with a high degree of autonomy, applying SRE principles and an engineering-first mindset to ensure the reliability, performance, and availability of mission-critical healthcare technology systems. 

You will influence observability standards across the organization, mentor colleagues, and drive innovation without direct management responsibilities. Promotions beyond this level are selective and driven by demonstrated business impact. 

KEY RESPONSIBILITIES 

Observability Platform Ownership 

• Engineer, implement, and operate enterprise-grade observability platforms including Dynatrace, LogicMonitor, Grafana, and Prometheus across multi-cloud and hybrid environments. 

• Lead the adoption and integration of OpenTelemetry (OTel) standards for distributed tracing, metrics collection, and log correlation across engineering teams. 

• Manage and optimize Cisco ThousandEyes for network path visibility, internet performance monitoring, and end-user experience insights. 

• Define and enforce observability-as-code practices, managing configurations through version-controlled pipelines (e.g., Terraform, Helm, GitOps). 

• Evaluate, recommend, and pilot emerging observability technologies and methodologies to continuously advance McKesson’s monitoring capabilities. 

SRE & Engineering Excellence 

• Apply Site Reliability Engineering (SRE) practices including error budgets, toil reduction, and chaos engineering to enhance system resilience. 

• Define, implement, and report on Service Level Objectives (SLOs), Service Level Agreements (SLAs), and Key Performance Indicators (KPIs) in partnership with engineering and business stakeholders. 

• Develop and maintain alerting frameworks that minimize noise and maximize signal fidelity, reducing mean time to detect (MTTD) and mean time to resolve (MTTR). 

• Build self-service observability tooling and dashboards that enable engineering teams to own their own reliability metrics. 

• Lead blameless post-incident reviews (PIRs) and drive remediation actions to prevent recurrence. 

Cloud & Infrastructure Monitoring 

• Design comprehensive observability  strategies for cloud-native workloads across AWS, Azure, and/or GCP, including containers (Kubernetes/EKS/AKS), serverless, and microservices architectures. 

• Instrument infrastructure, application, and business-layer telemetry (logs, metrics, traces) to provide end-to-end visibility. 

• Manage network performance monitoring and synthetic testing via ThousandEyes to proactively identify and resolve connectivity and latency issues impacting end users. 

• Partner with DevOps, NetOps, and Security teams to integrate observability signals into CI/CD pipelines, ITSM workflows, and SOC operations. 

AI & LLM Observability (Emerging Capability) 

• Extend the organization’s existing observability platforms to instrument AI-powered applications and autonomous agent workflows, capturing the telemetry unique to non-deterministic systems. 

• Implement OpenTelemetry GenAI semantic conventions to standardize how AI agent telemetry (traces, spans, token usage, tool invocations) is collected across frameworks and services. 

• Design and maintain tracing pipelines that capture the full agent execution chain from user intent and planner decisions through tool calls, retrieval steps, and model responses enabling root cause analysis beyond traditional request/response tracing. 

• Define AI-specific SLOs and cost budgets covering token consumption, hallucination rate, tool invocation success rate, and agent task completion rate. 

• Build dashboards and alerting for AI behavioral signals: model drift, performance degradation, prompt/context version changes, and guardrail violations. 

• Collaborate with data science, MLOps, and application teams to integrate observability signals into AI evaluation pipelines and continuous improvement feedback loops. 

• Implement security-focused AI monitoring including prompt injection detection, PII exposure in logs, and anomalous tool call patterns in alignment with OWASP Top 10 for LLMs. 

Collaboration & Technical Leadership 

• Act as an observability subject-matter expert and trusted resource for engineers, architects, and product teams across McKesson. 

• Develop and maintain internal observability standards, runbooks, and best-practice documentation. 

• Mentor and guide junior and mid-level engineers on observability tooling, instrumentation techniques, and SRE principles. 

• Anticipate organizational scaling needs and proactively direct engineering efforts to stay ahead of capacity, reliability, and visibility challenges. 

• Contribute to the development of new frameworks, methodologies, and tooling that advance McKesson’s engineering maturity. 

REQUIRED QUALIFICATIONS 

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience. 

• 7+ years of experience in observability, monitoring, SRE, or platform engineering roles within complex, enterprise-scale environments. 

• Hands-on expertise with two or more of the following platforms: Dynatrace, LogicMonitor, Grafana,ThousandEyes and Prometheus. 

• Demonstrated experience implementing OpenTelemetry (OTel) for distributed tracing and metrics instrumentation in polyglot service environments. 

• Proficiency with Cisco ThousandEyes or similar platforms for network monitoring, synthetic testing, and internet intelligence. 

• Deep understanding of cloud infrastructure monitoring across AWS, Azure, and/or GCP, including IaaS, PaaS, and container platforms. 

• Strong working knowledge of SRE principles: SLOs, SLAs, error budgets, alerting philosophy, and incident management. 

• Scripting/automation proficiency in Python, Go, Bash, or equivalent, with experience building or extending monitoring integrations. 

• Experience with infrastructure-as-code and GitOps practices (Terraform, Helm, Ansible, etc.) as applied to observability configuration management. 

• Demonstrated ability to work independently, set direction, and deliver results in a complex, matrixed organization. 

PREFERRED QUALIFICATIONS 

• Experience in healthcare IT, regulated industries, or large-scale enterprise environments. 

• Familiarity with AIOps platforms and ML-based anomaly detection capabilities within Dynatrace or similar tooling. 

• Knowledge of eBPF-based observability approaches and agent-less instrumentation techniques. 

• Experience integrating observability data into ITSM platforms (ServiceNow, PagerDuty) and security workflows (SIEM).