Want to know if this job is worth applying to?
Brasil
Remote only
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Engineer – Platform Modernization & Migration based in Brazil.
As a Senior Data Engineer, you will play a key role in a large-scale data platform modernization initiative, helping transform legacy data infrastructure into a scalable, high-performance environment. You will execute migration strategies for approximately 1,000 datasets to Snowflake while modernizing orchestration workflows with Apache Airflow 3. Working closely with the Data Platform team, you will contribute to the adoption of Apache Iceberg, the modernization of streaming infrastructure, and the optimization of data pipelines across cloud environments. This role offers the opportunity to tackle complex engineering challenges while ensuring data quality, reliability, and performance at scale. You will work in a collaborative, globally distributed engineering environment that values technical excellence, autonomy, and efficient delivery. The position is fully remote, with regular collaboration with international teams and required availability overlapping with Pacific Time business hours.
Data Pipeline Migration: Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, following established migration blueprints and ensuring reliable, consistent data delivery.
Workflow Modernization: Upgrade orchestration workflows from Apache Airflow 2 to Airflow 3, including adapting, testing, and debugging DAGs to maintain scheduling reliability and operational continuity.
Data Lake Modernization: Migrate legacy Hive-based data structures and datasets to Apache Iceberg, leveraging Snowpark Connect frameworks and connectors where applicable.
Streaming Infrastructure Migration: Reconfigure and migrate Confluent and Apache Kafka streaming connections across cloud regions, ensuring seamless connectivity and minimal disruption to data flows.
Pipeline Adaptation and Optimization: Adapt existing data pipelines for Snowflake environments, optimize SQL queries, and improve processing efficiency using advanced techniques such as common table expressions (CTEs), window functions, and complex joins.
Data Processing and Orchestration: Build, maintain, test, and troubleshoot data workflows using Python, Apache Spark, PySpark, Apache Airflow, and Apache Kafka.
Testing and Quality Assurance: Develop and execute testing strategies, perform hands-on debugging, and validate data accuracy, consistency, and integrity throughout the migration lifecycle.
Performance and Reliability: Identify bottlenecks and optimize distributed data pipelines, storage structures, and query execution to meet performance and reliability standards.
Cloud Data Management: Support data lake operations on AWS S3, including storage organization, bucket partitioning, and lifecycle management.
Cross-Functional Collaboration: Partner with data engineers and platform teams to execute migration plans, resolve technical challenges, and deliver modernization milestones within established timelines.
Independent Delivery: Manage assigned migration activities with a high degree of autonomy, communicating progress, risks, and technical findings clearly to distributed stakeholders.
Professional Experience: At least two years of hands-on data engineering experience working with large-scale datasets and distributed data processing environments.
Programming and SQL: Strong Python development skills and advanced SQL expertise, including complex joins, CTEs, window functions, query optimization, and performance tuning.
Data Warehousing: Proven experience with Snowflake, including data modeling, schema design, and adapting data pipelines to cloud data warehouse environments.
Data Processing and Orchestration: Practical experience developing and testing pipelines with Apache Airflow and Apache Spark or PySpark, along with familiarity with Apache Kafka and streaming data architectures.
Data Formats and Storage: Experience with Apache Iceberg and Apache Parquet, as well as AWS S3 and data lake concepts, including partitioning strategies and storage lifecycle policies.
Testing and Debugging: Ability to design testing approaches, troubleshoot complex distributed systems, and independently resolve data pipeline issues.
Cloud and Platform Modernization: Understanding of data platform migrations, workflow orchestration, and infrastructure modernization. Experience with Snowflake migrations or Apache Airflow version upgrades is a strong advantage.
AI-Assisted Development: Familiarity with AI-powered coding and development tools to improve engineering productivity and accelerate delivery is desirable.
Communication and Collaboration: Strong written and verbal English skills, with the ability to participate in technical discussions, produce documentation, and collaborate effectively with international teams.
Autonomy and Adaptability: Demonstrated ability to work independently, prioritize tasks, and deliver results in a fast-paced, technically demanding environment.
Remote Collaboration: Ability to work remotely while maintaining availability to overlap with the core engineering team until at least 12:00 PM Pacific Time. Candidates in EMEA or APAC are preferred, while the role is open to eligible candidates in Brazil.
AWS WorkSpace Connectivity: Ability to access a client-provisioned AWS WorkSpace with reliable connectivity. Candidates must complete the connectivity test at https://clients.amazonworkspaces.com/Health and submit a screenshot of the results as supporting evidence with their application.
Availability and Contract: Availability for a full-time, permanent position and willingness to collaborate across time zones as required.
Fully Remote Work: Work remotely from Brazil, with flexibility to operate from your preferred location, subject to the role's collaboration and time-zone requirements.
Full-Time, Permanent Position: Opportunity to contribute to a long-term data platform modernization initiative.
International Collaboration: Work alongside a globally distributed engineering team on a technically challenging project.
Modern Data Technology Stack: Gain hands-on exposure to Snowflake, Apache Iceberg, Apache Airflow 3, Apache Spark, Kafka, and AWS cloud infrastructure.
Large-Scale Engineering Impact: Contribute to the migration and modernization of approximately 1,000 datasets, helping improve data platform scalability, reliability, and performance.
Technical Development: Strengthen expertise in cloud data warehousing, distributed processing, streaming infrastructure, data migration, and orchestration modernization.
AI-Enabled Engineering: Opportunity to leverage AI-assisted development tools to improve productivity and streamline engineering workflows.