Want to know if this job is worth applying to?
Palo Alto, California, United States
Onsite
About the Team
Enterprise AI is a platform that provides end-to-end machine learning tooling experience to support and accelerate machine learning development, including autonomous driving and other related projects. Our platform serves customers as a standardized machine learning platform within Woven by Toyota as the larger Toyota Group companies.
The Enterprise AI ML Training Infrastructure team builds and operates the infrastructure that enables engineers and researchers to train, evaluate, and iterate on machine learning models at scale. Our customers include teams working on AD/ADAS and Woven City, where large-scale machine learning workloads, simulation, and data processing are critical to developing the next generation of mobility technologies.
The team operates at the intersection of machine learning, distributed systems, cloud infrastructure, and developer platforms. Our systems include large-scale GPU compute environments, Kubernetes-based cluster infrastructure, and workflow and pipeline orchestration technologies such as Ray, Airflow, and Temporal.
We are looking for an Engineering Manager to lead a team to help shape the technical direction, engineering culture, and long-term evolution of our ML training infrastructure platform.
WHO ARE WE LOOKING FOR?
As the Engineering Manager, ML Training Infrastructure, you will lead the team responsible for building and operating the infrastructure that powers machine learning training workloads across Woven by Toyota.
You will combine engineering leadership with strong technical judgment. You will work closely with engineering, research, and product stakeholders to understand customer needs, translate them into scalable technical solutions, and ensure that our platform is reliable, efficient, secure, and easy to use.
This role is particularly suited to an engineering leader who enjoys working on complex infrastructure problems and is comfortable operating across Kubernetes, distributed systems, GPU infrastructure, cloud platforms, and machine learning workloads.
You will also play an important role in building a strong engineering culture, developing engineers, establishing effective engineering practices, and partnering with our customers to continuously improve the platform.
What You Will Enable
In this role, you will help build the infrastructure that enables teams across Woven by Toyota to train and operate increasingly large and sophisticated machine learning workloads.
Your work will directly support engineers and researchers working on AD/ADAS, Woven City, and other mobility initiatives by making compute resources easier to access, workloads more reliable, and ML development faster and more scalable.
You will have the opportunity to shape both the technical architecture of a critical ML infrastructure platform and the engineering culture of the team building it.
Lead engineers and foster a collaborative and innovative environment.
Manage and nurture a group of around 7–8 engineers, offering technical vision, mentorship, professional growth, and effective project delivery.
Establish and articulate the technical vision and future roadmap for machine learning training infrastructure.
Direct the architecture, construction, and operation of resilient systems supporting massive ML model training workloads.
Construct and maintain Kubernetes-driven compute clusters and container ecosystems tailored for machine learning workloads.
Oversee and optimize GPU fleets, addressing cluster scheduling, resource utilization, system stability, and cost performance.
Advance distributed model training capabilities and workload management leveraging frameworks like Ray and adjacent tools.
Enhance workflow orchestration platforms by deploying and supporting solutions such as Airflow, Temporal, and similar pipeline frameworks.
Elevate the system resilience, monitoring insights, elasticity, and operational productivity of our ML training stack.
Work closely alongside AD/ADAS, Woven City, and internal platform partners to gather feature requests and ship infrastructure solutions that accelerate development.
Partner with cross-functional engineering groups to specify platform interfaces, APIs, system architectures, and operational standards.
Translate high-level organizational objectives and user requirements into concrete, executable engineering milestones with product leads.
Institute rigorous software practices across implementation, code reviews, automated testing, continuous deployment, and live site management.
Engage in production support workflows, lead post-incident reviews, and champion system refinements based on operational findings.
Balance immediate client feature requests against long-term architectural stability, technical hygiene, and system maintainability.
Foster an inclusive, highly collaborative, and excellence-driven engineering culture throughout Woven by Toyota.
8+ years of experience in software engineering, with at least 2 years in a leadership role.
Proven track record mentoring and managing high-performing software or platform teams.
Demonstrated expertise in architecting, deploying, and maintaining large-scale mission-critical infrastructure.
Deep technical mastery of Kubernetes orchestration alongside containerized runtime systems.
Hands-on exposure to cloud platform architectures and highly resilient distributed systems.
Background engineering platform foundation layers designed for machine learning model development or data pipeline intensive workloads.
Practical understanding of massive GPU cluster management, hardware scheduling algorithms, and capacity efficiency tuning.
Operational background overseeing high-availability environments, leading post-incident reviews, and advancing monitoring capabilities.
Excellent interpersonal abilities to align technical roadmaps across diverse engineering partners and leadership stakeholders.
Experience working with Japanese customers or clients.
Experience with Golang development (especially for cloud services)
Experience in the automotive industry or knowledge of automotive software development (e.g., ASPICE, V-model).
Experience working in environments with a focus on security and safety.
English/Japanese bilingual ability