Lead Data Engineer

Key Responsibilities

Design and modeling

•    Lead the design of data products end to end: what gets ingested, how it is cleansed and conformed, how it is modeled, and what the Gold layer looks like to the people querying it.
•    Own dimensional design — grain, natural and surrogate keys, Type 2 history, facts, bridges, and conformed dimensions shared across the pod’s products.
•    Set the pod’s position on where logic belongs: what is cleaned in Silver, what is business logic in Gold, and what is a consumer’s own concern.
•    Keep models as simple as the questions require, and push back on designs that will not hold.
•    Partner with ML engineering where the pod’s Gold layer is the training or feature source for a model, so those datasets are contracted, versioned, and reproducible like any other consumer-facing product.

Data contracts and consumer compatibility

•    Own the pod’s ODCS data contracts as real interfaces: named owners, named consumers, enforceable quality rules, freshness and update expectations, and an explicit breaking-change policy.
•    Make the compatibility call on every proposed contract change, and drive consumer notification when a change is genuinely breaking.
•    Represent the pod’s contracts in cross-domain conversations, where one pod’s Gold layer is another team’s dependency.
Standards, quality, and operations
•    Set and hold the pod’s engineering standards for Python, SQL, dbt, testing, model structure, naming, and repository conventions — consistent with the platform’s paved paths and Azure DevOps CI gates rather than in competition with them.
•    Lead code review. Be the reviewer who catches the modeling mistake, the missing test, and the change that will break a consumer, and who explains why so the pod learns it.
•    Ensure quality rules are enforced in tests and asset checks rather than asserted in documentation, and that pipeline health is observable without someone going to look — freshness, volume, latency, and cost instrumented, and alerting set against the SLAs and SLOs the pod’s contracts commit to.
•    Own the pod’s operational posture for its own pipelines: failure diagnosis, data-issue triage, backfills, cost and performance tuning, on-call coverage and escalation, root-cause analysis, and runbooks someone other than the author can execute.
•    Keep the pod’s delivery inside the control expectations of a regulated carrier: change management through pull request and pipeline, segregation of duties between authoring and deploying, least-privilege access, and audit evidence that falls out of CI/CD rather than being assembled for an auditor.
•    Develop reusable frameworks, templates, and patterns that raise the pod’s consistency and delivery speed.

Delivery leadership

•    Partner with the Product Owner and Scrum Master on decomposition and refinement: turn use cases and features into estimable, grounded stories with testable acceptance criteria and an identified target layer and repository.
•    Hold the Definition of Ready before the pod commits and the Definition of Done before the pod calls something finished — merged and approved code, passing CI and coverage gates, and evidence that the outcome is real.
•    Identify unknowns that need a spike rather than an estimate, and say so during planning rather than mid-sprint.

Mentorship and collaboration

•    Grow the engineers on the pod through design review, pairing, and code review rather than by taking the hard work yourself.
•    Bring new engineers up to productive speed on the platform’s conventions and tooling, and reduce single points of knowledge — no data product that only one person understands.
•    Work with the platform team on capability gaps: when the pod needs something the platform does not yet offer, raise it as a demand signal rather than building a private workaround.
•    Partner with the DataOps/MLOps Lead on the shared CI/CD, orchestration, and observability standards — adopt and strengthen the paved path rather than forking it, and be the pod’s voice on what it is still missing.
•    Work with data architecture and governance on solution shape, Unity Catalog placement, and access requirements.

Qualifications

Required Qualifications

•    Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent practical experience considered.
•    6+ years building and operating production data pipelines and consumer-facing data models, from source ingestion through to published data products.
•    Strong hands-on Python and SQL, with the credibility to make design calls and the willingness to still write and review code.
•    Hands-on experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake, MERGE, incremental processing, and performance tuning.
•    Deep dimensional modeling experience — grain, keys, slowly changing dimensions, facts and dimensions, conformed dimensions.
•    Demonstrated technical leadership: setting standards, leading design, and raising other engineers’ work, whether or not the role carried a lead title.
•    Experience owning data that other teams depend on, including handling breaking changes and production data incidents.
•    Experience with orchestration (Dagster, Databricks Workflows, Airflow, or similar), Git-based collaborative development, code review, and CI/CD — Azure DevOps or comparable.
•    Experience setting observability and SLA/SLO expectations for data other teams depend on, and running the incident and communication path when they are missed.
•    Ability to explain trade-offs clearly to engineers, product owners, and business stakeholders, and to say no to a design that will not hold.

Preferred Qualifications

•    Databricks certification (Data Engineer Professional or equivalent demonstrated depth).
•    Unity Catalog at multi-team scale: catalogs, schemas, external locations, permissions, and lineage.
•    dbt at scale on Databricks, and Python-based modeling frameworks over Delta Lake.
•    Dagster and Dagster Cloud, including assets, asset checks, and branch deployments.
•    Practical experience with data contracts, ODCS, or data-mesh style data product ownership.
•    Experience with a declarative Python ingestion framework such as dlt (dltHub) or comparable.
•    Data quality and observability tooling such as Great Expectations, Monte Carlo, or similar.
•    Familiarity with MLOps practice — MLflow, model registries, and model serving — sufficient to design data products that ML systems can depend on.
•    Azure and Azure DevOps.
•    Financial services, insurance, or another regulated industry, including data access, lineage, and audit expectations.
•    Experience introducing AI-assisted development into a team’s normal workflow in a disciplined way.