Want to know if this job is worth applying to?
San Jose, California, United States
Onsite
NVIDIA is recruiting a Senior Inference Engineer to advance AIConfigurator (https://github.com/ai-dynamo/aiconfigurator), a system that automatically discovers high-performance deployment configurations for large-scale LLM inference. This role integrates GPU systems, model serving, performance modeling, and production software engineering. The work directly aids users in deploying models on NVIDIA platforms by optimizing efficiency, latency, parallelism, and resource utilization across both aggregated and disaggregated serving architectures. The team partners closely with Dynamo, TensorRT-LLM, vLLM, SGLang, benchmarking, and platform teams to translate sophisticated performance data into useful deployment mentorship. This is a high-impact IC role for someone who enjoys owning deep technical systems and making them practical for real developers and customers.
What you'll be doing:
Build and evolve AIConfigurator's core optimization engine for LLM serving, including configuration search, SLA-aware ranking, efficiency and latency estimation, and Pareto frontier analysis.
Build production-quality Python/Rust APIs, CLIs, SDK surfaces, and web workflows that help users generate strong deployment configurations for NVIDIA GPU clusters.
Develop configuration generation systems that emit backend-specific artifacts for Dynamo, Kubernetes, TensorRT-LLM, vLLM, and SGLang deployments.
Collaborate with inference runtime, performance, benchmarking, and product groups to ensure simulated results correspond with actual deployment performance on H100, H200, B200, GB200, and upcoming NVIDIA platforms.
Improve model, hardware, and backend support by integrating performance databases, profiling data, support matrices, and validation tools.
Drive software quality through maintainable architecture, schema development, tests, documentation, and automation suitable for open-source and production users.
Convert intricate inference ideas like prefill/decode disaggregation, tensor parallelism, pipeline parallelism, expert parallelism, batching, and KV cache behavior into dependable software abstractions.
What we need to see:
BS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering, Applied Math, or a related field, or equivalent experience.
10+ years of relevant software engineering experience.
Strong Python/Rust engineering skills, including production APIs, CLI tools, packaging, testing, debugging, and maintainable software development.
Experience with GPU computing, distributed systems, ML infrastructure, or high-performance model serving.
Understanding of LLM inference concepts such as batching, latency, efficiency, memory constraints, parallelism strategies, and serving SLAs.
Experience working with data-driven performance analysis, benchmarking, simulation, optimization, or managing resource needs.
Ability to collaborate across research, runtime, platform, and customer-facing engineering teams.
Strong written and verbal communication skills, with the ability to explain sophisticated technical tradeoffs clearly.
Ways to stand out from the crowd:
Practical experience working directly with TensorRT-LLM, vLLM, SGLang, Triton Inference Server, Dynamo, Kubernetes, or comparable serving platforms.
Experience improving LLM deployments on NVIDIA GPUs, especially H100, H200, B200, GB200, or multi-node GPU clusters.
Familiarity with disaggregated serving, prefill/decode separation, KV cache management, NCCL/NIXL/NVSHMEM communication, or expert-parallel MoE inference.
Open-source project experience, technical writing experience, or prior ownership of developer-facing tools.
Agentic AI solutions to solve sophisticated technical problems.
#LI-Hybrid
Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.You will also be eligible for equity and benefits.
This posting is for an existing vacancy.
NVIDIA uses AI tools in its recruiting processes.
NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.