Want to know if this job is worth applying to?
United States
Remote only
We are looking for a Senior Data Engineer to help build, operate, and continuously improve the data infrastructure that powers analytics and business intelligence across Tala's global markets.
In this individual contributor role, you will take ownership of Tala's data synchronization pipelines, ensuring that data from our market-level databases is reliably captured, processed, and delivered to Snowflake for downstream analytics and business use.
Our data infrastructure uses a modern AWS-based architecture, including AWS Database Migration Service (DMS), Kafka, Spark Structured Streaming, Apache Iceberg, Amazon S3, and Snowflake. You will work across this ecosystem to ensure data is accurate, available, and delivered with minimal latency.
Beyond maintaining existing pipelines, you will identify opportunities to improve performance, scalability, reliability, and cost efficiency. You will also establish and monitor service level objectives (SLOs), strengthen testing and deployment practices, and lead root cause analysis when issues arise.
This is an opportunity for a hands-on engineer who enjoys solving complex distributed data challenges and takes ownership of production systems from development through operation.
Own and operate data pipelines: Maintain and improve Tala's change data capture (CDC) and data synchronization infrastructure, ensuring accurate and timely delivery of data from market-level databases into Snowflake.
Improve pipeline performance and scalability: Identify and implement improvements to reduce latency, optimize infrastructure costs, and support growing data volumes across markets.
Build and evolve data infrastructure: Onboard new data sources, tables, and markets while managing schema evolution and ensuring compatibility across systems.
Drive reliability and data quality: Establish service level objectives (SLOs), implement monitoring and alerting, and build automated data quality checks to ensure data remains accurate and available.
Strengthen engineering practices: Develop and maintain unit and integration tests, CI/CD workflows, infrastructure-as-code, and safe deployment strategies, including dry runs and blue-green deployments.
Lead incident resolution: Troubleshoot production issues, conduct root cause analyses, and translate lessons learned into better monitoring, testing, documentation, and operational runbooks.
Contribute to technical direction: Participate in architecture and design discussions, write and review technical design documents, and provide thoughtful feedback through code reviews.
Collaborate across teams: Partner with Data Science, Analytics, Data Platform, Product, Credit, and Business Development teams to understand their data needs and ensure reliable access to the information they depend on.
4+ years of experience in backend engineering or data engineering, ideally supporting consumer-scale applications.
At least 2 years of hands-on professional Python experience.
Proven experience independently building, operating, and improving production-grade data pipelines.
Hands-on experience with change data capture (CDC) and stream processing technologies such as Kafka, Amazon MSK, AWS DMS, or comparable tools.
Strong proficiency in Apache Spark, particularly Spark Structured Streaming.
Experience with modern data lakehouse architectures and table formats such as Apache Iceberg, Delta Lake, or Apache Hudi, as well as Parquet.
Strong database experience across relational and analytical systems, including MySQL, PostgreSQL, Amazon RDS, and Snowflake.
Hands-on experience with AWS cloud infrastructure, ideally including EMR, S3, MSK, DMS, RDS, and EKS.
Experience with infrastructure-as-code and CI/CD tools such as Terraform, Terragrunt, Helm, and Jenkins.
Familiarity with pipeline orchestration and job scheduling tools such as Airflow or Python-based services running on Kubernetes.
Strong understanding of distributed systems, scalable algorithms, and large-scale data processing.
Ability to communicate complex technical concepts clearly and collaborate effectively with engineering teams and business stakeholders.
Nice to Have
Experience with Trino or Presto.
Experience with Kubernetes and containerized applications.
Familiarity with AWS Glue.
Experience with additional stream processing technologies such as Debezium, Amazon Kinesis, or Apache Flink.
Experience with other modern data lakehouse technologies such as Delta Lake or Apache Hudi.
Advanced degree in Computer Science, Mathematics, or a related technical field.