Data Engineer

M9 Solutions is seeking a Data Engineer to work onsite in support of a government contract for a client located in Springfield, VA. An active TS/SCI clearance is required.

Responsibilities

• Data Ingestion & Acquisition: Collect and integrate data from a wide variety of structured and unstructured sources, including APIs, RDBMS, file systems, third-party services, and real-time streams.
• Pipeline Development: Design and implement scalable ETL/ELT pipelines to clean, enrich, normalize, and semantically align data (ontology-driven transformations).
• Cloud Deployment: Build and deploy data pipelines and associated infrastructure on AWS or Azure, using managed services like Lambda, Glue, Step Functions, Azure Data Factory, etc.
• Database Architecture: Understand and optimize for different storage engines—relational (PostgreSQL, MySQL), columnar (Redshift, BigQuery), indexing engines (ElasticSearch), key-value stores (DynamoDB, Redis), Object stores (S3 or similar), and caching layers.
• Streaming Data Processing: Work with Apache Kafka (or similar platforms) to handle high-volume, low-latency data streams.
• Workflow Orchestration: Utilize Apache Airflow (or equivalent) to schedule and monitor complex data workflows.
• AI/ML Integration: Collaborate with data scientists to integrate LLMs and ML models into pipelines for inference, tagging, enrichment, or intelligent routing of data.

Required Skills and Qualifications

• Active TS/SCI security clearance.
• Bachelor's or master’s degree in computer science, engineering, or related field.
• 10+ years of experience in data engineering or software development roles.
• Strong proficiency in Python, including experience with libraries like pandas, PySpark, FastAPI, or similar.
• Solid experience with cloud services (AWS or Azure) and Cloud native data engineering tools.
• Proven experience in building and maintaining data pipelines using Kafka, Airflow, and other open-source frameworks.
• Strong grasp of database internals and trade-offs between different storage technologies.
• Familiarity with data governance, lineage, and metadata management concepts.
• Experience or strong interest in integrating LLMs and AI/ML models into production-grade data systems.

Preferred Skills and Qualifications

• Knowledge of data cataloging tools and semantic layer design.
• Experience with containerization (Docker) and orchestration (Kubernetes).
• Familiarity with MLOps tools or platforms (e.g., SageMaker, MLflow).