This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Data Engineer (Azure/Databricks) | SR based in Brazil.
This is a senior-level data engineering role focused on building scalable, reliable, and high-performance data solutions.
You will design and maintain modern data pipelines across Azure and Databricks environments.
The role combines structured and unstructured data processing, data modeling, and cloud-based data architecture.
You will work with large volumes of data from APIs, databases, documents, and other enterprise sources.
You will also contribute to Data Lake and Data Lakehouse initiatives, including advanced use cases involving RAG and AI.
The position requires strong technical ownership, attention to data quality, and the ability to transform complex data into usable assets.
You will collaborate in a technology-driven environment where modern data platforms and emerging AI capabilities are central to innovation.
Accountabilities:
- Build and maintain scalable, reliable data pipelines using modern distributed processing technologies, ensuring efficient data ingestion, transformation, and delivery.
- Design and implement data models using Databricks, SAP Datasphere, Azure Data Factory, Azure Data Lake Storage (ADLS), Python, and relational databases.
- Extract, transform, and organize data to create and maintain Data Lakes, Data Lakehouses, and analytical tables.
- Structure and manage data tables within Databricks, ensuring performance, consistency, and accessibility.
- Develop ETL pipelines for both structured and unstructured data, using APIs and specialized tools for file analysis and data validation.
- Integrate data from APIs, including CXL sources, and transform and process the resulting datasets within Databricks.
- Structure RAG (Retrieval-Augmented Generation) solutions using contracts and other business documents as data sources.
- Apply data engineering best practices to ensure data quality, scalability, reliability, and performance.
- Work with multiple data sources and technologies to support analytical, operational, and AI-driven use cases.
- Contribute to the evolution of cloud-based data architecture and help identify opportunities to improve data processing and integration workflows.
Requirements:
- Advanced professional experience and strong technical knowledge of Databricks.
- Solid experience with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS).
- Proven experience implementing Data Lake and Data Lakehouse architectures.
- Strong programming skills in Python and experience with Apache Spark and distributed data processing.
- Professional experience with Apache Airflow for data pipeline orchestration.
- Experience working with SQL and NoSQL databases, including technologies such as PostgreSQL, MongoDB, and Cassandra.
- Strong understanding of data modeling, ETL/ELT processes, data integration, and large-scale data processing.
- Ability to work with both structured and unstructured datasets and transform complex sources into reliable analytical assets.
- Analytical mindset and strong problem-solving skills, with attention to data quality and technical details.
- Ability to work independently on complex data engineering challenges while collaborating effectively with technical and business stakeholders.
- Desirable: Knowledge of Kafka and event-driven data architectures.
- Desirable: Experience with dbt and modern data transformation practices.
- Desirable: Knowledge of AWS Glue and BigQuery.
- Desirable: Experience with AI, RAG, or document-based data solutions.
- Desirable: Familiarity with SAP Datasphere.
Benefits:
- Opportunity to work with modern Azure and Databricks data technologies.
- Exposure to advanced AI, Generative AI, and RAG use cases.
- Opportunity to work with cloud-native Data Lake and Data Lakehouse architectures.
- Professional development in a technology-focused and innovation-driven environment.
- Access to modern AI-powered tools and technologies.
- Opportunities to work with distributed and borderless teams.
- Exposure to emerging technologies and current industry trends.
- Career development opportunities and continuous learning.
- Opportunity to contribute to projects focused on digital transformation and AI-driven innovation.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
