Senior Data Engineer

Accountabilities

• Own Data Platform Architecture: Design, build, and operate a production-grade data platform from initial architecture through deployment, optimization, and ongoing maintenance.

• Build Multi-Source Ingestion Pipelines: Develop concurrent ingestion systems for provider APIs, webhooks, files, and web scraping, incorporating rate limiting, error handling, and reliable processing at scale.

• Develop Batch and Event-Driven Workflows: Implement scheduled pipelines, historical backfills, and dependency management using orchestration tools such as Airflow, Dagster, or Prefect. Build event-driven processing with messaging and task queue services, including Pub/Sub or Cloud Tasks.

• Ensure Reliable Failure Recovery: Implement retries, dead-letter queues, controlled replay mechanisms, and idempotent upserts to recover from failures without generating duplicate records or corrupting data.

• Establish Raw-to-Curated Data Architecture: Preserve original source payloads, including provider JSON, in Google Cloud Storage (GCS), then transform and standardize data using dbt. Maintain clearly defined raw, standardized, and curated data layers.

• Strengthen Data Quality and Governance: Define data contracts, automated validation rules, pipeline tests, retention policies, and monitoring systems to track data freshness, processing failures, and overall reliability.

• Optimize Database Performance: Design and maintain PostgreSQL schemas, queries, and incremental loading strategies to ensure efficient processing and dependable data delivery to downstream products.

• Integrate Enrichment and AI Extraction: Incorporate third-party enrichment providers and large language model (LLM)-based structured extraction from text into the data platform.

• Ensure Data Traceability: Maintain clear lineage between source data and curated records, enabling safe reprocessing, historical recovery, auditing, and consistent data management.

• Drive Continuous Improvement: Identify opportunities to improve throughput, reliability, maintainability, and infrastructure efficiency while adapting the platform to evolving business and product needs.

Requirements:

• End-to-End Data Engineering Experience: Proven experience building production data platforms from the ground up, including architecture, pipeline implementation, deployment, monitoring, and ongoing operations.

• Workflow Orchestration: Hands-on production experience with Airflow, Dagster, Prefect, or Temporal, including workflow dependencies, retries, scheduling, and failure recovery. Airflow experience is preferred.

• Google Cloud and dbt: Practical production experience with Google Cloud Platform (GCP) and dbt, including data transformation, pipeline management, and cloud-based data infrastructure.

• Event-Driven Architecture: Experience developing asynchronous, event-driven systems using messaging platforms or task queues, with a strong understanding of concurrent processing, retries, and fault tolerance.

• Python Programming: Strong Python skills, particularly asynchronous HTTP requests and concurrent processing using libraries such as asyncio, aiohttp, or HTTPX.

• SQL and PostgreSQL: Advanced SQL knowledge and practical PostgreSQL experience, including data modeling, query optimization, incremental loading, and performance tuning.

• Data Quality and Modeling: Experience designing data contracts, raw and curated storage layers, retention policies, automated validation, and data quality monitoring.

• Web Scraping Expertise: Hands-on experience building and maintaining production scraping pipelines using tools such as Scrapy, Playwright, Selenium, or Beautiful Soup, including concurrency management, rate limiting, and failure handling.

• Independent Ownership: Ability to make sound technical decisions, manage complex projects, and take responsibility for data infrastructure across the full development lifecycle.

• Startup or Small-Team Experience: Experience working in a startup or lean engineering team, where priorities evolve quickly and engineers take broad ownership, is a plus.

• Entity Resolution: Familiarity with matching company and individual records across multiple sources, identifying duplicates, and consolidating information into consistent entities is highly desirable.

• LLM-Based Data Processing: Experience using large language models for structured data extraction, company enrichment, or people data enrichment is an advantage.

• Additional Data Technologies: Familiarity with BigQuery and Dataform is a plus.

• Communication Skills: Ability to communicate technical concepts clearly, collaborate effectively with distributed teams, and work comfortably in an English-speaking professional environment.

Benefits:

• Fully remote work: Work remotely from eligible locations across Europe and Latin America, with flexibility to organize your working environment.

• Technical ownership: Lead the architecture and evolution of a data platform, with meaningful influence over technical decisions and engineering practices.

• Modern technology stack: Work with Python, PostgreSQL, GCP, dbt, event-driven systems, workflow orchestration, web scraping, and AI-powered data extraction.

• End-to-end impact: Build foundational infrastructure that directly supports product functionality, data quality, and business growth.

• Engineering autonomy: Take ownership of technical challenges and shape scalable solutions in a dynamic, product-focused environment.

• Innovation opportunities: Explore advanced data processing, entity resolution, external data enrichment, and LLM-based extraction techniques.

• International collaboration: Work with a distributed team across multiple regions, with opportunities to contribute to evolving technical initiatives.

• Compensation flexibility: Salary expectations are discussed in USD on a monthly basis, with the final package determined through the recruitment process.