Data Engineer (Databricks) | SR

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Engineer (Databricks) | SR based in Brazil.

We are seeking a Senior Data Engineer with deep expertise in Databricks to design, build, and evolve scalable, high-performance data architectures. In this role, you will lead the development of reliable data pipelines and complex processing solutions across batch and streaming environments. You will work with modern technologies such as Apache Spark, PySpark, Delta Lake, and cloud data platforms to ensure efficient data ingestion, transformation, governance, and delivery. Your work will contribute to robust Lakehouse architectures, optimized workloads, and secure, well-governed data ecosystems. As a senior technical reference, you will help guide engineering practices, mentor professionals, and solve complex technical challenges. This is a fully remote opportunity in a technology-driven environment focused on innovation, continuous learning, and advanced data engineering.

Accountabilities

• Define and evolve data architectures using Databricks, ensuring pipeline optimization, processing efficiency, and effective resolution of complex technical challenges.

• Build, maintain, and optimize scalable, reliable data pipelines for data ingestion, transformation, and delivery.

• Design and implement Lakehouse architectures and Medallion structures (Bronze, Silver, and Gold).

• Develop batch and streaming data processing solutions, including Change Data Capture (CDC) and Structured Streaming.

• Build and orchestrate data ingestion and integration processes using Azure Data Factory (ADF), Auto Loader, and Lakeflow.

• Work with Apache Spark, Airflow, Kafka, dbt, SQL and NoSQL databases, and cloud data platforms.

• Implement data governance, security, lineage, and access control practices using Unity Catalog.

• Develop REST APIs, integrations, microservices, and automation solutions using Python.

• Establish CI/CD processes, version control, and automated deployments using Git and Databricks Workflows.

• Optimize Spark performance, compute resource allocation, and processing costs.

• Monitor and maintain production data pipelines, ensuring observability, reliability, and operational continuity.

• Act as a technical reference for the team, mentoring professionals and promoting engineering standards and best practices.

Requirements:

• Advanced expertise in Databricks, PySpark, Spark SQL, and Delta Lake.

• Strong experience with Delta Live Tables (DLT) / Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming.

• Hands-on knowledge of Liquid Clustering, Auto Loader, and Delta table optimization.

• Strong understanding of Unity Catalog, including data governance, security, lineage, and access control.

• Proven experience designing and implementing data architectures based on Lakehouse and Medallion principles (Bronze, Silver, and Gold).

• Practical experience with Azure Data Factory (ADF) for data orchestration, ingestion, and integration.

• Advanced Python programming skills and experience developing REST APIs, integrations, microservices, and automation.

• Advanced knowledge of Spark performance tuning, including Photon, partitioning, shuffle optimization, and execution plan analysis.

• Experience with Databricks Workflows, CI/CD, Git, and Declarative Automation Bundles.

• Ability to optimize workloads, size computing resources, and reduce processing costs.

• Experience with observability, monitoring, and production pipeline support.

• Ability to act as a technical leader, guide other professionals, and establish data engineering best practices.

• Relevant Databricks certifications:

  • Databricks Certified Data Engineer Associate.

  • Databricks Certified Data Engineer Professional.

  Benefits:

  • Meal and/or food allowance.

  • Health insurance and dental insurance.

  • Pet health insurance.

  • Home-office allowance for remote positions.

  • Welcome kit during onboarding.

  • Life insurance.

  • Zenklub access, including psychological and nutritional support.

  • TotalPass membership.

  • Certification and continuous learning program with partners such as Adobe, Anthropic, AWS, Google, Microsoft, and Salesforce.

  • Partnerships and discounts supporting professional development.

  • Access to UOL content.

  • Maternity kit.

  • Extended maternity and paternity leave.

  • Employee referral program.