Head of Observability Engineering

The TP ICAP Group is a world leading provider of market infrastructure.

Our purpose is to provide clients with access to global financial and commodities markets, improving price discovery, liquidity, and distribution of data, through responsible and innovative solutions.

Through our people and technology, we connect clients to superior liquidity and data solutions.

The Group is home to a stable of premium brands. Collectively, TP ICAP is the largest interdealer broker in the world by revenue, the number one Energy & Commodities broker in the world, the world’s leading provider of OTC data, and an award winning all-to-all trading platform.

Founded in London in 1866, the Group operates from more than 60 offices in 27 countries.  We are 5,200 people strong.  We work as one to achieve our vision of being the world’s most trusted, innovative, liquidity and data solutions specialist.

 

Role Overview

We are establishing a dedicated Observability Engineering function to create a step change in how TP ICAP understands, manages and improves the health, performance and resilience of its technology estate. Reporting to and working closely with the Group Head of Production Engineering, the Head of Observability Engineering will define the enterprise direction, standards, tooling and engineering practices for observability across applications and infrastructure. The role will lead the evolution of the strategic Grafana LGTM platform from a like-for-like monitoring capability into a data-led service that provides end-to-end insight across complex business and technology value streams.

The successful candidate will establish consistent standards for metrics, logs and traces, define reusable engineering patterns and best practices, and partner with Infrastructure and Production Engineering teams across multiple lines of business to raise observability maturity. The team will have direct engineering and operational ownership of the observability tooling stack, including OpenTelemetry collectors, the Grafana LGTM stack and other enterprise observability tooling. This role will help define standards that include consistent and decision-focused dashboards, richer instrumentation, event correlation across dependent applications and infrastructure, end-to-end visibility of order and trade flows across multiple systems and components, and monitoring of low-latency market data feeds.

This is a hands-on engineering leadership role. Alongside setting direction and leading two Observability Engineers, you will design, implement, operate and support solutions; establish a trusted data foundation; engineer automated health and latency checks; capture key service and business metrics in time-series repositories; and build baselines that define normal behaviour. The team will apply analytics and AI to accelerate issue detection and evidence-based hypotheses, improve capacity forecasting and enable safe, increasingly-automated remediation.

This is a high-impact opportunity to shape a strategically important capability at a pivotal stage of its development. The organisation has a strong platform foundation and a clear mandate to raise standards significantly; the successful candidate will have the scope to influence enterprise practices, introduce modern observability engineering and help move Production Engineering from reactive monitoring towards predictive, resilient and AI-enabled operations.

 

Role Responsibilities

• Strategy, Standards & Technical Leadership: Work with the Group Head of Production Engineering to define the enterprise observability strategy, standards, tooling, governance and best practices, while remaining hands-on in solution design, implementation and operational support.
• Team Leadership: Lead, mentor and coordinate the work of the Observability Engineers, building a strong engineering culture and ensuring consistent delivery across the observability roadmap.
• Observability Engineering with Grafana: Design, implement, and optimize observability solutions built on the Grafana LGTM stack — Loki (logs), Grafana (visualisation and dashboarding), Mimir (metrics), and Tempo (distributed tracing) — that give teams clear, actionable insight into system health and performance.
• Observability Strategy & Instrumentation: Establish best practices across the three pillars of observability — logs, metrics, and traces — including managing ingestion rates, cardinality and retention, and using annotations to correlate deployments and incidents across dashboards.
• Platform Engineering in AWS: Design, implement, and optimize AWS-based platform solutions that host and support the observability stack, aligned with application requirements and industry best practices.
• Containerisation & Infrastructure Support: Build and maintain the containerised infrastructure the observability stack runs on, using Docker and Kubernetes / Amazon EKS, ensuring the platform is scalable and resilient.
• Automation & Infrastructure as Code (IaC): Develop and maintain infrastructure and observability automation using scripting and IaC tooling (e.g. Terraform, AWS CDK), enabling repeatable, self-service deployment of monitoring and alerting.
• Escalation & Troubleshooting: Serve as an escalation point for complex technical and observability issues, providing guidance and support to other team members.
• Collaboration & Mentorship: Work closely with cross-functional teams, sharing knowledge and best practices to foster continuous improvement

 

Experience / Competences

Essential

• Engineering leadership experience: defining observability strategy, operating models, standards and governance across large-scale technology estates. Strong awareness of industry trends, operational resilience requirements, cloud-native observability practices and AI-driven operations, with a track record of translating best practices into measurable improvements in service availability, performance and operational efficiency

• Strong observability expertise within the Grafana LGTM stack — hands-on experience with Loki, Grafana, Mimir and Tempo for logs, metrics, dashboarding and distributed tracing.
• Solid understanding of the observability pillars — logs, metrics and traces — including ingestion rates, cardinality, retention and the use of annotations to correlate events across dashboards.
• Containerisation knowledge to support the infrastructure on which the observability stack runs, using Docker and Kubernetes.
• Scripting and automation proficiency, with a demonstrated history of developing and maintaining custom automated solutions — strong experience with Python, Go and PowerShell.
• Key AWS concepts and hands-on AWS experience, including cloud automation, security best practices, cost optimisation and architectural design.
• Experience with DevOps methodologies and CI/CD pipelines.
• Familiarity with security and compliance within cloud environments.
• Excellent problem-solving skills and the ability to troubleshoot complex cloud-based and observability environments.
• Strong leadership capabilities, with experience leading technical projects and mentoring team members.
• Ability to effectively communicate technical concepts to both technical and non-technical stakeholders

Desired

• Current or recent experience working within an equivalent Financial Services organisation, with a strong understanding of the operational resilience, control and availability demands of business-critical technology services.
• Experience of Geneos observability platform, including low latency solutions
• AWS certifications such as AWS Certified Solutions Architect – Professional, AWS Certified DevOps Engineer – Professional, AWS Certified AI Practitioner, or AWS Certified Developer – Associate.
• Familiarity with other monitoring/logging solutions such as CloudWatch, Prometheus or the ELK Stack.
• Proven track record of managing and transforming observability across large, complex system architectures and technology landscapes.

 

Job Band & Level

• Function Head / 8

 

#LI-Hybrid #LI-DIR #LI-LH

Not The Perfect Fit?

Concerned that you may not meet the criteria precisely? At TP ICAP, we wholeheartedly believe in fostering inclusivity and cultivating a work environment where everyone can flourish, regardless of your personal or professional background. If you are enthusiastic about this role but find that your experience doesn't align perfectly with every aspect of the job description, we strongly encourage you to apply. You may be the ideal candidate for this position or another opportunity within our organisation. Our dedicated Talent Acquisition team is here to assist you in recognising how your unique skills and abilities can be a valuable contribution. Don't hesitate to take the leap and explore the possibilities. Your potential is what truly matters to us.

Company Statement

We know that the best innovation happens when diverse people with different perspectives and skills work together in an inclusive atmosphere. That's why we're building a culture where everyone plays a part in making people feel welcome, ready and willing to contribute. TP ICAP Accord - our Employee Network - is a central to this. As well as representing specific groups, TP ICAP Accord helps increase awareness, collaboration, shares best practice, and holds our firm to account for driving continuous cultural improvement. 

Location

Philippines - Ecoprime Building - Taguig City