Data Engineer - Python and Azure Data Bricks

ole Overview

We are looking for a highly skilled Data Engineer with strong expertise in Python and Azure Databricks to design, build, and optimize scalable data pipelines and modern data platforms. The ideal candidate will have hands-on experience in cloud-based data engineering, big data processing, and data transformation using Azure services and Databricks.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Azure Databricks.
• Build and optimize ETL/ELT workflows for ingesting, transforming, and processing large datasets.
• Develop data solutions using Apache Spark within the Databricks environment.
• Integrate data from multiple structured and unstructured sources.
• Ensure data quality, reliability, security, and governance across data platforms.
• Collaborate with Data Scientists, Analysts, Architects, and Business stakeholders to deliver data solutions.
• Implement performance tuning and optimization for large-scale data processing workloads.
• Develop and maintain data models, metadata management, and documentation.
• Monitor and troubleshoot production data pipelines and resolve data-related issues.
• Support CI/CD implementation and deployment automation for data engineering solutions.

Required Skills & Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.
• 4+ years of experience in Data Engineering.
• Strong hands-on programming experience in Python.
• Expertise in:
  • Azure Databricks
  • Apache Spark (PySpark)
  • SQL
  • Delta Lake
• Experience with Azure data services:
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage (ADLS)
  • Azure Synapse Analytics
  • Azure Key Vault
• Strong understanding of data warehousing concepts and dimensional modeling.
• Experience building and maintaining batch and real-time data pipelines.
• Knowledge of Git, CI/CD pipelines, and DevOps practices.
• Experience working in Agile development environments.