Senior Solution Architect- Spark & Databricks Expert

Accountabilities

• Lead customer engagements from initial discovery and technical assessment through solution architecture, implementation, and final delivery, taking full ownership of outcomes.

• Design, architect, implement, and optimize large-scale Apache Spark and Databricks workloads, ensuring scalability, reliability, and cost efficiency.

• Apply advanced knowledge of Spark internals to troubleshoot, debug, and optimize distributed data processing jobs and SQL workloads.

• Develop and deliver proofs of concept (PoCs), technical demonstrations, and customer workshops to communicate solution capabilities and validate architectural approaches.

• Act as a trusted technical advisor, proactively identifying customer challenges, anticipating potential issues, and recommending innovative, data-driven solutions.

• Drive technical decision-making and translate complex business requirements into practical, scalable data architectures.

• Collaborate with product and engineering teams to enhance platform capabilities, improve technical solutions, and strengthen the overall customer experience.

• Establish and promote engineering best practices across PySpark, Databricks, and Lakehouse architecture.

• Mentor junior engineers and support the development of technical expertise across data engineering teams.

• Continuously identify opportunities to improve data processing performance, platform governance, and overall solution efficiency.

Requirements:

• Between 5 and 10+ years of professional experience in data engineering, analytics, or enterprise data solution delivery.

• At least 3 years of hands-on experience with Databricks, including workload optimization, debugging, performance tuning, and governance.

• Expert-level understanding of Apache Spark fundamentals and internals, with strong practical experience in PySpark development.

• Proven expertise in Databricks SQL (DBSQL), including query optimization and performance tuning.

• Strong knowledge of modern Lakehouse architectures, Delta Lake, Structured Streaming, and cloud infrastructure.

• Demonstrated ability to architect and deliver complex, large-scale data solutions independently, with minimal supervision.

• Strong analytical and problem-solving skills, with the ability to take initiative, make technical decisions, and manage complex challenges autonomously.

• Excellent communication and stakeholder management skills, particularly the ability to explain sophisticated technical concepts and architectural decisions to business leaders and non-technical audiences.

• Ability to lead customer-facing technical engagements and build trusted relationships through proactive collaboration and solution-oriented thinking.

• Preferred: Databricks Certified Data Engineer Professional certification.

• Experience with advanced Databricks capabilities, including Delta Live Tables (DLT) and Unity Catalog (UC).

• Previous consulting experience involving large-scale data migrations, enterprise analytics transformations, or AI-assisted analytics initiatives is considered an advantage.

Benefits:

• Fully remote working arrangement within the United States.

• Full-time contract opportunity.

• Opportunity to work on complex, enterprise-scale data engineering and analytics projects.

• Exposure to advanced technologies, including Apache Spark, Databricks, PySpark, Delta Lake, and modern Lakehouse architectures.

• Opportunity to lead strategic customer engagements and influence enterprise data solutions.

• Collaborative environment with opportunities to work alongside product, engineering, and customer-facing teams.

• Professional growth through hands-on experience with advanced data platforms, architecture, and cloud technologies.

• Opportunity to mentor other engineers and contribute to technical best practices.