Job Description:
Title: Lead data engineer
Experience - 7-10 Years
About the Role
We are seeking a highly skilled and delivery-focused Lead GCP Data Engineer to support the design, development, and implementation of next-generation enterprise data and AI platforms on Google Cloud Platform (GCP).
This role will work closely with Enterprise Architects, platform leaders, and cross-functional engineering teams to build scalable, reusable, and AI-ready data foundations that enable advanced analytics, intelligent automation, and enterprise AI adoption.
The ideal candidate combines strong hands-on expertise in cloud-native data engineering, modern data platform development, semantic data enablement, and scalable pipeline engineering with the ability to lead engineering teams and drive high-quality delivery across multiple initiatives.
This role is expected to play a critical leadership position within the engineering organization by driving implementation excellence, mentoring teams, and operationalizing modern data architecture patterns.
Key Responsibilities
1. Enterprise Data Platform Engineering
- Design, develop, and optimize scalable cloud-native data platforms and pipelines on GCP.
- Implement robust batch, streaming, and event-driven data processing solutions supporting enterprise analytics and AI use cases.
- Collaborate with Enterprise Architects to translate target-state architecture into scalable engineering implementations.
- Contribute to modernization of legacy data ecosystems into reusable, governed, and AI-ready cloud platforms.
- Support implementation of scalable ingestion, transformation, serving, and orchestration frameworks.
2. Data Product Engineering
- Develop reusable and domain-oriented data products aligned with data mesh and data-as-a-product principles.
- Implement scalable and modular data pipelines supporting multiple downstream consumers including analytics, AI/ML, and operational applications.
- Contribute to implementation of:
- Data contracts
- Schema management
- Metadata enrichment
- Data quality frameworks
- Reusable transformation patterns
- Enable discoverability, trust, and operational reliability of enterprise data assets.
3. Semantic Layer & Consumption Enablement
- Support implementation of semantic and business-consumption layers that simplify enterprise data access.
- Collaborate with analytics and BI teams to enable standardized business metrics, reusable dimensions, and governed KPI definitions.
- Contribute to semantic modeling and metadata integration initiatives supporting self-service analytics and AI consumption.
- Assist in improving enterprise data usability, consistency, and discoverability across platforms.
4. GCP-Native Engineering & Development
- Develop and optimize solutions leveraging GCP-native services including:
- BigQuery
- Dataflow
- Dataproc
- DBT
- Pub/Sub
- Cloud Storage
- Cloud Composer (Airflow)
- Cloud SQL
- Build scalable ETL/ELT frameworks and real-time streaming pipelines.
- Optimize data processing performance, reliability, scalability, and cost efficiency.
- Implement CI/CD pipelines and engineering automation for data platform delivery.
5. AI/ML & GenAI Data Enablement
- Build AI-ready data pipelines and scalable feature engineering workflows supporting enterprise AI initiatives.
- Support integration with:
- Vertex AI
- BigQuery ML
- Vector databases
- LangChain
- Generative AI Studio
- Contribute to implementation of RAG architectures, semantic search, and AI-assisted data interaction patterns.
- Partner with AI/ML teams to operationalize scalable ML and GenAI workflows.
6. Engineering Leadership & Delivery Excellence
- Lead day-to-day engineering activities across multiple data engineering workstreams.
- Guide and mentor junior and mid-level data engineers on modern engineering best practices.
- Ensure adherence to coding standards, architecture guidelines, and operational best practices.
- Drive engineering quality through automated testing, observability, monitoring, and performance optimization.
- Collaborate with architects, product owners, analysts, and client stakeholders to ensure successful delivery outcomes.
7. Governance, Reliability & Observability
- Implement data governance, lineage, monitoring, and observability frameworks.
- Support enforcement of enterprise standards around security, reliability, scalability, and operational readiness.
- Contribute to platform monitoring, incident management, and continuous improvement initiatives.
- Ensure production readiness of pipelines and data services through robust testing and validation processes.
Technical Expertise Required
Area
Skills / Technologies
Cloud Data Engineering
GCP, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, Cloud SQL
Data Transformation
DBT, PySpark, SQL, ETL/ELT frameworks
Streaming & Pipelines
Apache Beam, real-time processing, event-driven architectures
Semantic Layer & Modeling
Semantic modeling concepts, Looker modeling, business metrics standardization
AI/ML Enablement
Vertex AI, BigQuery ML, LangChain, Vector Databases, GenAI integration
Orchestration & Automation
Cloud Composer (Airflow), CI/CD, Workflows
Metadata & Governance
Data Catalog, lineage, metadata management, observability frameworks
Programming
Python, SQL, PySpark
Qualifications
- Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or related field.
- 7+ years of experience in data engineering and cloud-native data platform development.
- Minimum 4+ years of hands-on experience delivering enterprise-scale solutions on GCP.
- Strong expertise in building scalable batch and streaming data pipelines.
- Experience working on modern enterprise data platforms supporting analytics, AI/ML, and GenAI use cases.
- Good understanding of semantic layer concepts, reusable data models, and governed data consumption patterns.
- Experience working within large-scale data modernization and cloud transformation initiatives.
- Strong problem-solving, debugging, and performance optimization skills.
- Proven ability to lead engineering teams and collaborate across architecture, product, and business functions.
- Excellent communication and stakeholder management skills.
- GCP certifications such as Professional Data Engineer preferred.
Location:
DGS India - Mumbai - Thane Ashar IT Park
Brand:
Merkle
Time Type:
Full time
Contract Type:
Permanent
