What if the work you did every day could impact the lives of people you know? Or all of humanity?
At Illumina, we are expanding access to genomic technology to realize health equity for billions of people around the world. Our efforts enable life-changing discoveries that are transforming human health through the early detection and diagnosis of diseases and new treatment options for patients.
Working at Illumina means being part of something bigger than yourself. Every person, in every role, has the opportunity to make a difference. Surrounded by extraordinary people, inspiring leaders, and world changing projects, you will do more and become more than you ever thought possible.
Position Summary:
The Senior Data Engineer will be part of the Data Engineering and Analytics team, designing, building, validating, and supporting enterprise data pipelines that integrate SAP S/4HANA data into the Databricks Lakehouse through SAP Datasphere and Amazon S3. The engineer owns assigned SAP datasets end to end, from source analysis and ingestion through reconciliation, testing, release, and production support, within approved architecture, security, and change-control standards. The role partners with SAP, platform, architecture, governance, and business teams, leads complex troubleshooting and continuous improvement, and provides technical guidance to other engineers.
Responsibilities:
- Own end-to-end onboarding of assigned SAP objects following the standard onboarding runbook, including source analysis, CDS wrapper view creation, Datasphere replication flow setup, Bronze ingestion configuration, validation, and sign-off.
- Design, build, and maintain Bronze ingestion pipelines in Databricks (Auto Loader, Lakeflow Declarative Pipelines, PySpark, Delta Lake) for full, delta, CDC, and watermark loads, handling restartability, deletes, and schema changes within approved design patterns.
- Operate and monitor Datasphere replication flows into Amazon S3, recover failed runs, and work with the platform administration team to resolve connectivity and capacity issues.
- Run source-to-Bronze reconciliation as a continuous control; review PASS/FAIL results, investigate aggregate and record-level discrepancies, and document corrective actions.
- Plan and execute SIT/UAT scenarios covering full loads, deltas, deletes, and field additions or removals; reconcile SAP validation output against Bronze data and compile acceptance evidence.
- Monitor pipeline observability dashboards and alerts; resolve ingestion failures and schema drift, lead root cause analysis (RCA) for complex issues, and implement preventive actions.
- Promote all changes through Git and CI/CD, review peer code and configurations, and apply approved Unity Catalog access controls for assigned datasets.
- Develop automation and reusable components that improve onboarding, reconciliation, monitoring, and recovery.
- Maintain the SAP object inventory (extraction method, load pattern, Bronze location, delivery wave, status, and exceptions) along with per-object operational documentation, recovery procedures, and knowledge-transfer material.
- Resolve Jira and ServiceNow tickets for SAP data pipelines, record hypercare incidents and resolutions, mentor junior engineers, and provide after-hours on-call support on a rotating basis.
Listed responsibilities are an essential, but not exhaustive list, of the usual duties associated with the position. Changes to individual responsibilities may occur due to business needs.
Requirements:
Required
- 6+ years of enterprise data engineering or data integration experience, including hands-on Databricks delivery and production SAP data integration.
- Hands-on Databricks experience with PySpark, Spark SQL, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines (DLT), Workflows, Databricks SQL, and Unity Catalog.
- Advanced SQL and Python, including reconciliation queries, aggregate comparisons, record-level validation, and automation scripting.
- SAP S/4HANA knowledge, including core table structures, CDS view development, ODP delta extraction, and record-level verification (SE16).
- Strong understanding of full, delta, pseudo-delta, and CDC loads, including watermarks, delete handling, restartability, error handling, and schema drift.
- Practical experience with Git, code review, testing, and CI/CD deployment across environments.
- Experience with pipeline monitoring and incident management using Databricks dashboards, Grafana, Jira, and ServiceNow.
- Proven ability to troubleshoot complex production issues, lead root cause analysis, and maintain clear technical documentation.
- Strong communication and collaboration skills across SAP, platform, and business teams, with experience mentoring junior engineers.
- Flexibility to work occasional non-standard hours and participate in a rotating on-call schedule; international travel may be needed.
Preferred
- Experience with SAP Datasphere / Business Data Cloud replication flows, or replication tools such as SLT, HVR, or Fivetran.
- SIT/UAT, cutover, and hypercare experience on an ERP program.
- Experience using AI-enabled engineering tools, such as Cursor, Claude, ChatGPT, or GitHub Copilot.
Experience/Education:
- Typically requires a minimum of 6 years of related experience with a Bachelor’s degree; or 3 years and a Master’s degree; or a PhD without experience; or equivalent work experience.
We are a company deeply rooted in belonging, promoting an inclusive environment where employees feel valued and empowered to contribute to our mission. Built on a strong foundation, Illumina has always prioritized openness, collaboration, and seeking alternative perspectives to propel innovation in genomics. We are proud to confirm a zero-net gap in pay, regardless of gender, ethnicity, or race. We also have several Employee Resource Groups (ERG) that deliver career development experiences, increase cultural awareness, and offer opportunities to engage in social responsibility. We are proud to be an equal opportunity employer committed to providing employment opportunity regardless of sex, race, creed, color, gender, religion, marital status, domestic partner status, age, national origin or ancestry, physical or mental disability, medical condition, sexual orientation, pregnancy, military or veteran status, citizenship status, and genetic information. Illumina conducts background checks on applicants for whom a conditional offer of employment has been made. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable local, state, and federal laws. Background check results may potentially result in the withdrawal of a conditional offer of employment. The background check process and any decisions made as a result shall be made in accordance with all applicable local, state, and federal laws. Illumina prohibits the use of generative artificial intelligence (AI) in the application and interview process. If you require accommodation to complete the application or interview process, please contact accommodations@illumina.com. To learn more, visit: https://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf. The position will be posted until a final candidate is selected or the requisition has a sufficient number of qualified applicants. This role is not eligible for visa sponsorship.