SIte Reliability Engineer, AI Inference

The AI Inference Engineer plays a critical role in the AI lifecycle by bridging the gap between high-performance model development and optimized deployment environments. This position focuses on optimizing Large Language Models (LLMs) for inference, serving diverse environments—from GPU-rich data centers to resource-constrained edge devices—with a strong emphasis on maximizing throughput, minimizing latency, and maintaining model accuracy.  

This role is pivotal in advancing F5’s AI capabilities, ensuring enterprise-grade reliability by leveraging hardware acceleration, designing scalable infrastructure, and monitoring system performance. 

Key Responsibilities 

High-Performance AI Serving 

• Build and maintain robust inference engines using tools like vLLM, TGI (Text Generation Inference), and NVIDIA Triton, ensuring high performance at scale.  

• Handle deployment optimizations to deliver low-latency AI serving solutions for multiple business applications. 

Hardware Acceleration and Optimization 

• Profile and optimize models for specialized hardware backends, including NVIDIA GPUs (CUDA/TensorRT), Apple Silicon (CoreML), and AI accelerators like TPUs and LPUs.  

• Collaborate with hardware teams to maximize utilization and performance across various computational environments. 

Inference Orchestration and Scalability 

• Design and implement auto-scaling architectures for online (real-time) and batch inference pipelines, leveraging Kubernetes for inference routing and orchestration.  

• Ensure software solutions are optimized for peak performance during traffic spikes, maintaining reliability and scalability. 

Performance Monitoring and Observability 

• Establish robust observability frameworks to monitor Time to First Token (TTFT), tokens per second, and memory bandwidth utilization against service-level agreements (SLAs).  

• Build and execute performance and load testing suites to identify bottlenecks and ensure consistent reliability at scale. 

Technical Requirements 

Required Skills: 

• Programming Languages: Proficiency in programming languages such as Python, C++, Rust, or Golang specifically for high-performance AI workflows.  

• Inference Tools: Proven hands-on experience with tools like vLLM, TensorRT, Llama.cpp, and Ollama for inference development and optimization.  

• Infrastructure Expertise: Strong familiarity with infrastructure technologies, including Docker, Kubernetes, and cloud platforms such as AWS, GCP, and Azure.  

• Hardware Optimization Expertise: Comprehensive understanding of GPU and AI hardware, including techniques for profiling and optimizing performance for accelerators like NVIDIA GPUs and TPUs. 

Preferred Experience: 

• Prior experience deploying Large Language Models (LLMs) with advanced techniques like Speculative Decoding or PagedAttention.  

• Contributions to open-source inference libraries or hardware-level kernel development (e.g., CUDA, Triton kernels).  

• Background in MLOps or SRE roles focused on high-performance AI endpoints and reliability during demand surges.  

• Proficiency in designing scalable solutions for high-throughput inference environments optimized for traffic bursts. 

Success Metrics (KPIs): 

• Latency Reduction: Continuously improve inference latency metrics, ensuring minimal Time to First Token (TTFT) and maximum tokens per second.  

• Cost Efficiency: Achieve lower "Cost per 1K Tokens" through better resource utilization and hardware optimization.  

• Scalability: Maintain system stability and reliability during traffic spikes, ensuring performance consistency across environments.  

• Throughput Maximization: Deploy models optimized for peak hardware usage and maximized process throughput. 

Why Join F5? 

F5 empowers you to push boundaries in AI optimization and high-performance engineering. Joining our team means:  

• Collaborating with cutting-edge technologies and hardware solutions to support real-time AI applications.  

• Advancing your career in a fast-paced, multidisciplinary environment focused on innovation, scalability, and problem-solving.  

• Driving transformative projects that deliver real-time AI reliability to global customers while maintaining cost and efficiency standards.  

• Working on advanced MLOps solutions that seamlessly scale enterprise AI systems and shape the future of intelligent deployment. 

What Success Looks Like: 

As an AI Inference Engineer at F5, success is measured by your ability to:  

• Combine technical expertise and problem-solving skills to deliver low-latency, scalable, and high-performing AI prediction systems.  

• Collaborate efficiently across cross-functional teams, participating in knowledge sharing and system refinement.  

• Demonstrate initiative by driving optimizations across hardware, tools, and orchestration processes, balancing immediate solutions with long-term architectural goals.  

• Translate complex AI and inference workflows into practical solutions that align with F5's strategic objectives. 

#LI-AK1