This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Engineer, Big Data - Databricks based in United States.
This is a senior technical leadership role focused on building enterprise-scale data and analytics solutions in a healthcare environment.
You will lead the design and delivery of cloud-based data architectures supporting complex Medicare and Medicaid operations.
The role combines hands-on engineering with strategic partnership, translating business challenges into scalable, data-driven solutions.
You’ll work extensively with Azure Databricks, Data Factory, Synapse, Data Lake, and machine learning services.
A strong emphasis is placed on AI-first development, real-time data processing, automation, data quality, and advanced analytics.
You will also mentor engineers, influence technical direction, and collaborate with business and executive stakeholders across a highly matrixed organization.
This opportunity is ideal for an experienced data engineer who wants to connect modern cloud technology with meaningful healthcare outcomes.
Accountabilities
- Lead end-to-end data engineering initiatives as a Forward Deployed Engineer, taking ownership of technical solutions and mentoring team members in business-focused problem solving.
- Provide technical leadership and subject matter expertise across Azure Databricks, Azure Synapse, Azure Data Factory, and Azure Data Lake Storage.
- Design and implement scalable, high-performance data architectures for large healthcare datasets, with an emphasis on reliability and maintainability.
- Champion AI-first development practices, including AI-assisted coding workflows and Azure Machine Learning solutions for predictive healthcare analytics.
- Develop business intelligence solutions using tools such as Power BI and Tableau, integrated with the broader Azure data ecosystem.
- Analyze business processes and identify opportunities to apply Azure data, AI, and platform services to improve Medicare and Medicaid operations.
- Build reusable data-quality frameworks and automated validation processes using Azure DevOps and quality-assurance automation tools.
- Lead integration initiatives connecting the Azure data platform with low-code and enterprise platforms such as Pega and Salesforce.
- Translate business requirements into technical architecture, detailed designs, and production-ready implementations for healthcare analytics use cases.
- Design analytical data models that enable self-service reporting for clinical, financial, and operational stakeholders.
- Implement batch and real-time data pipelines supporting claims processing, quality measurement, and population health analytics.
- Partner with business leaders to identify, evaluate, and prioritize high-value data opportunities.
- Lead technical issue resolution and coordinate with cloud support, infrastructure, and external vendor teams when required.
- Support project planning, scheduling, and delivery using Agile methodologies.
- Maintain compliance with HIPAA requirements and applicable healthcare data governance standards.
- Train, mentor, and support new and existing team members across Azure data technologies and Forward Deployed Engineering practices.
Requirements
- At least 6 years of experience in data management or data engineering across cloud data platforms, or an equivalent combination of relevant education and experience.
- Bachelor's degree in information technology, computer science, statistics, business, healthcare, engineering, or a related discipline.
- Strong hands-on experience with Microsoft Azure data services, particularly Azure Databricks, Azure Data Factory, Azure Synapse, Azure Data Lake Storage, and Azure Machine Learning.
- Demonstrated experience implementing AI-assisted development practices and machine learning solutions.
- Strong proficiency in Python, PySpark, Scala, and SQL, together with experience in automated quality assurance and data validation.
- Experience designing and implementing real-time streaming architectures using technologies such as Kafka, Azure Event Hubs, or Spark Streaming.
- Experience integrating data platforms with enterprise or low-code systems such as Pega and Salesforce.
- Healthcare payer experience, particularly with Medicare/Medicaid claims, eligibility, HEDIS or Stars quality metrics, and regulatory reporting, is highly valued.
- Solid understanding of HIPAA requirements and healthcare data governance frameworks.
- Experience with Azure DevOps CI/CD pipelines and Infrastructure as Code tools such as Terraform or ARM templates.
- Strong critical-thinking, analytical, and problem-solving abilities with a focus on measurable business outcomes.
- Ability to mentor engineers and promote an ownership-oriented, business-focused engineering mindset.
- Strong communication skills, with the ability to explain complex technical concepts and translate business requirements for technical and executive audiences.
- Demonstrated ability to collaborate effectively across teams in a complex, highly matrixed environment.
- Experience with Microsoft 365/O365, Power BI, and Azure DevOps.
- Azure Data Engineer certification and additional healthcare industry experience are preferred.
Benefits
- Annual salary range of $96,325.57–$208,705.40 USD, with actual compensation determined by factors such as geographic location, experience, education, and skill level.
- Competitive overall compensation package.
- Comprehensive employee benefits package.
- Full-time remote work opportunity within the United States.
- Opportunity to work with modern Azure cloud, Databricks, AI, machine learning, and big data technologies.
- Opportunity to lead high-impact healthcare data initiatives supporting clinical, financial, operational, and population health outcomes.
- Strong opportunities for technical leadership, mentoring, and professional development.
- Collaborative environment involving engineering, business, healthcare, infrastructure, and executive stakeholders.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
