Senior Data Engineer

Position Summary:

The Senior Data Engineer will design, build, and maintain serverless data pipelines and data models on AWS that make high-quality, analytics-ready data available to our AI and Analytics teams. This role will own ingestions, transformation, and storage patterns using services such as Lambda, Glue, Athena, and S3, ensuring data is reliable, well-documented, and aligned with business and model-training needs. The role will also partner with Data Science to support MLOps, including reproducible machine-learning training and scoring pipeline, automated feature generation, deployment, and monitoring. It will power ML model workflows using sensitive healthcare data and support model training and scoring, production monitoring, and business feedback loops.

Key Responsibilities:

• Design, implement, and maintain scalable, fully serverless data pipelines on AWS using Lambda, Glue, Athena, Step Functions, and S3 to support reporting, analytics, and AI use cases.
• Build and evolve data models and schemas that enable performant querying and downstream consumption by AI, Analytics and engineering teams. Build versioned data models, feature stores, with semantics, reproducible backfills, and safeguards against leakage or inconsistent definitions.
• Develop ETL/ELT workflows to ingest, cleanse, transform, and load data from internal applications, third‑party sources, and event streams into our data lake and analytical layers.
• Partner closely with Product, Underwriting, Analytics, and AI stakeholders to understand data requirements and translate them into robust data structures, contracts, and SLAs.
• Implement data quality controls, monitoring, and alerting to ensure accuracy, completeness, timeliness, and lineage of critical datasets and features used by models. Include automated controls for schema change, validity, reconciliation, claims maturity, and model leakage.
• Optimize serverless workloads for cost, performance, and scalability, including query tuning in Athena and efficient storage formats/partitioning in S3.
• Contribute to and enforce data engineering best practices, including version control, code review, CI/CD for data pipelines, and Infrastructure as Code with AWS CDK.
• Collaborate with AI and Analytics teams to design and maintain feature stores and other reusable data assets that accelerate experimentation and model deployment. Partner on batch or API scoring and capture model/data versions, recommendations, actions, overrides, and claims to close the learning loop.
• Experience designing and operating event-driven or streaming data pipelines for near-real-time scoring or decisioning, using Kafka, Amazon Kinesis, or comparable technologies; strong understanding of event schemas, idempotency, ordering, retries, observability, and failure recovery.
• Troubleshoot pipeline issues, resolve data‑related incidents, and provide ongoing support for production data workflows and model‑driven applications.
• Document data models, pipelines, and data contracts, and help evangelize data literacy and self‑service analytics across the organization.

Required Skills & Qualifications:

• Strong experience with AWS data and serverless services (Lambda, Glue, Athena, S3, Step Functions or similar orchestration tools).
• Advanced proficiency in Python (required) and SQL, with experience in relational and NoSQL databases and in building reusable ETL/ELT libraries. Hands-on experience with PySpark for distributed data processing is strongly preferred; TypeScript experience for AWS CDK required.
• Experience with MLOps practices, including versioned feature and training datasets, orchestration of training and scoring workflows, CI/CD, model/data monitoring, and reproducible deployments in partnership with AI.
• Solid understanding of data modeling principles for AI workloads. Apply schema evolution, incremental processing, partitioning, and columnar or open-table formats such as Parquet and Iceberg.
• Experience designing and operating data pipelines at scale, including batch and near‑real‑time ingestion. Hands-on experience with Kafka is preferred.
• Familiarity with SQL and query optimization in columnar data stores and engines.
• Knowledge of data quality, governance, and security for sensitive healthcare and financial data, including least-privilege access, retention, classification, tokenization, and de-identification.
• Hands‑on experience with Infrastructure as Code, preferably AWS CDK, for provisioning and managing data infrastructure.
• Ability to partner with AI and Analytics teams to design data solutions for model experimentation and production; Kedro or similar workflow experience is a plus.
• Knowledge of responsible AI and data-use practices, including permitted-use controls, privacy, fairness considerations, documentation, and governance for sensitive healthcare and financial data.
• Strong communication skills to explain complex data concepts clearly to business stakeholders; regulated-domain experience, preferably healthcare claims or insurance, is a plus.

Minimum Requirements: 

• Bachelor's degree in Computer Science, Engineering, Mathematics, Data Science, or a related field, or equivalent practical experience; a master's degree is preferred.
• 5+ years in data engineering or a related role, with significant ownership of production data pipelines and analytics platforms; hands-on ML/AI experience supporting feature preparation, model training, deployment, or monitoring is strongly preferred.
• Experience with AWS data services in a production environment (for example, Lambda, Glue, Athena, and S3) is preferred; equivalent production experience on another cloud data platform will be considered.
• Experience building and maintaining data solutions that support reporting, BI, and/or AI/ML initiatives.

Perks & Benefits:

• Fully paid medical, dental, and vision benefits.
• Flexible PTO
• 401k company contribution
• Tuition reimbursement
• Professional development allowance
• Transportation allowance and daily parking reimbursement 
• Engaging hybrid work environment