Sr Data Engineer-Noida(Hybrid)
We are looking for a highly skilled Senior Data Engineer to design, build, and maintain scalable data pipelines and integrations within our Databricks-based data platform. The role will focus on expanding our data ecosystem by onboarding new data sources, integrating enterprise applications, and delivering reliable, secure, and analytics-ready datasets.
The ideal candidate will have strong hands-on expertise in Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake, along with experience building production-grade data pipelines and enterprise integrations. Working knowledge of AWS infrastructure and cloud services is important for collaborating with Engineering and other platform teams on connectivity, deployments, and evolving infrastructure requirements.
The candidate will report directly to the Senior Manager - Business Insights & Analytics and collaborate with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders to deliver scalable and reliable data solutions.
Responsibilities:
- Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.
- Design and implement a medallion architecture (Bronze, Silver, and Gold layers) to deliver trusted, well-structured, analytics-ready datasets.
- Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems
- Developing reusable ingestion frameworks to accelerate onboarding of new data sources.
- Build reliable batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution and error recovery.
- Configure and manage Databricks Workflows and implement ingestion and transformation pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines
- Optimize Apache Spark and Delta Lake workloads using execution-plan analysis, efficient join strategies, data-skew handling, partitioning, and SQL query tuning to improve performance and control compute costs.
- Implement Unity Catalog governance, including access controls, lineage, auditing, and sensitive-data protection through masking and row-level security.
- Maintain and support existing data workflows running on Amazon Redshift and Amazon EC2, collaborating with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational issues.
- Follow engineering best practices, including version control, code reviews, automated testing, CI/CD, and technical documentation.
- Partner with Data Analytics and business stakeholders to translate requirements into trusted, well-documented, analytics-ready datasets.
- Provide technical guidance to team members and identify opportunities to improve automation, engineering standards, platform reliability, and resource utilization.
Requirements:
- 5+ years of experience in data engineering or a related technical role, including developing and supporting production data pipelines.
- Strong hands-on expertise in Databricks, including developing, troubleshooting, and optimizing production workloads.
- Strong proficiency in Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation.
- Practical experience with Delta Lake, including table management, incremental processing, and performance optimization.
- Experience with data architecture and modeling, including medallion architecture and the development of analytics-ready datasets.
- Experience with Databricks Workflows or equivalent orchestration solutions.
- Proven experience building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications.
- Understanding of reliable data processing patterns, including retries, idempotency, schema evolution, backfills, and failure recovery.
- Working knowledge of AWS services and cloud infrastructure, particularly IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms.
- Experience with Git/GitHub, code reviews, automated testing, and CI/CD practices.
- Strong problem-solving skills and the ability to independently design, build, deploy, and maintain production data solutions.
- Good communication and collaboration skills to work effectively with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders.
Desirable:
- Experience with Databricks Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), Auto Loader, and advanced incremental ingestion patterns.
- Experience implementing Unity Catalog governance, including fine-grained access controls, data lineage, auditing, and sensitive-data protection.
- Experience with Databricks Genie, Genie Agents, and Genie Code
- Experience in AI/ML, Data Science, LLMs, contextualization, Amazon Bedrock
Sumo Logic, Inc. helps make the digital world secure, fast, and reliable by unifying critical security and operational data through its Intelligent Operations Platform. Built to address the increasing complexity of modern cybersecurity and cloud operations challenges, we empower digital teams to move from reaction to readiness—combining agentic AI-powered SIEM and log analytics into a single platform to detect, investigate, and resolve modern challenges. Customers around the world rely on Sumo Logic for trusted insights to protect against security threats, ensure reliability, and gain powerful insights into their digital environments. For more information, visit www.sumologic.com.
Sumo Logic Privacy Policy. Employees will be responsible for complying with applicable federal privacy laws and regulations, as well as organizational policies related to data protection.
