This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Data Engineer / Data Scientist based in India.
This role offers the opportunity to build scalable data and machine learning solutions within a technology-driven, collaborative environment. You will design and optimize batch and streaming data pipelines using Python, PySpark, and Azure Databricks. The position combines data engineering with machine learning, MLOps, and model deployment to support impactful analytics and AI initiatives. You will work extensively with Azure services, Spark, Delta Lake, APIs, and cloud-based data architectures. As an individual contributor, you will take ownership of assigned deliverables while partnering closely with technical and business teams. The role is well suited to an experienced data professional who enjoys solving complex data challenges and working across modern cloud and ML technologies.
Accountabilities:
You will develop, optimize, and support modern data processing and machine learning solutions across cloud-based environments. The role requires strong technical ownership, practical problem-solving, and collaboration across engineering, data science, and business teams.
- Develop scalable data processing solutions using Python, PySpark, and Azure Databricks.
- Build, maintain, and optimize batch and real-time streaming data pipelines.
- Develop Spark DataFrame-based transformations and data processing workflows.
- Debug, troubleshoot, and optimize Spark applications and Databricks jobs.
- Implement Delta Lake solutions to improve data reliability, versioning, and query performance.
- Develop APIs using Python or Scala for data and machine learning applications.
- Support machine learning initiatives, MLOps workflows, and model deployment activities.
- Work with Azure services for data ingestion, storage, security, integration, and processing.
- Configure and manage Databricks job clusters, compute environments, and notebook workflows.
- Build and execute DataFrame-based data validation and quality checks.
- Develop pipelines using Event Hubs, Kafka, IoT sources, or other real-time data technologies.
- Support data quality monitoring and production troubleshooting.
- Implement secure integrations between Azure services using managed identities and secrets.
- Contribute to CI/CD practices for data engineering and machine learning workloads.
- Collaborate with technical and business stakeholders while independently managing assigned deliverables.
- Apply performance tuning techniques to Spark applications and Databricks workloads.
Requirements
The ideal candidate brings 5–8 years of relevant experience across data engineering, data science, machine learning, or cloud analytics, with strong hands-on capabilities in Python, PySpark, Azure, and Databricks. You should be comfortable developing production-ready data solutions, troubleshooting distributed processing workloads, and contributing to machine learning and MLOps initiatives.
- Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, Information Technology, or a related discipline.
- 5–8 years of relevant professional experience in data engineering, data science, machine learning, or cloud analytics.
- Strong hands-on expertise in Python and PySpark.
- Good knowledge of Microsoft Azure and Azure Databricks.
- Hands-on experience with MLOps practices and tools.
- Practical experience supporting machine learning projects.
- Basic understanding of machine learning model deployment.
- Strong experience developing and debugging Spark-based applications.
- Hands-on experience with Databricks notebook development.
- Strong knowledge of Spark DataFrames using PySpark or Scala.
- Experience optimizing Spark jobs and Databricks workloads.
- Experience developing APIs using Python or Scala.
- Working knowledge of Azure Event Hubs, Storage Accounts, Key Vault, Service Bus, Azure Functions, and Azure Data Lake Storage.
- Understanding of Databricks job clusters and compute configurations.
- Experience implementing cloud-based data solutions on Azure.
- Knowledge of real-time streaming technologies such as Kafka.
- Experience developing batch and streaming pipelines using Event Hubs, Kafka, or IoT data sources.
- Hands-on experience implementing Delta Lake solutions.
- Working knowledge of GitHub or similar version-control platforms.
- Exposure to MLflow or comparable tools for experiment tracking and model lifecycle management is beneficial.
- Experience with CI/CD for data and machine learning workloads is a plus.
- Knowledge of data quality validation, monitoring, and production support is advantageous.
- Strong analytical and problem-solving abilities.
- Ability to work independently while collaborating effectively with cross-functional project teams.
Benefits
- Full-time position.
- Remote work arrangement.
- Immediate requirement with an opportunity to join a technology-focused data and AI environment.
- Opportunity to work with modern cloud technologies including Microsoft Azure and Azure Databricks.
- Hands-on exposure to Python, PySpark, Spark, Delta Lake, streaming, and MLOps.
- Opportunity to contribute to machine learning projects and model deployment initiatives.
- Exposure to real-time data technologies such as Kafka, Event Hubs, and IoT data sources.
- Opportunities to work across data engineering, machine learning, and cloud analytics.
- Collaboration with technical and business teams on impactful data initiatives.
- Scope for continued development in cloud, data engineering, and machine learning technologies.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
