Senior Azure Databricks Architect

Accountabilities:

• Design secure, scalable, and highly available Azure Databricks architectures aligned with enterprise security, connectivity, governance, and operational requirements.
• Architect and deploy Azure Databricks workspaces using Terraform and infrastructure-as-code practices, with direct ownership of implementation rather than simply operating within preconfigured environments.
• Design and implement Azure Private Link and private endpoint connectivity for Databricks, including front-end, back-end, and browser authentication connectivity where required.
• Design customer-managed VNets and VNet-injected Databricks deployments, including Secure Cluster Connectivity and no-public-IP configurations.
• Architect private DNS zones, DNS resolution, routing, network security groups, user-defined routes, and firewall configurations.
• Integrate Databricks environments into hub-and-spoke and other enterprise Azure network architectures.
• Troubleshoot complex DNS, routing, authentication, firewall, and network connectivity issues in collaboration with client security and networking teams.
• Design identity and access management solutions using Microsoft Entra ID, Databricks Unity Catalog, service principals, and related security controls.
• Establish standardized architecture patterns and deployment practices across development, testing, and production environments.
• Implement CI/CD pipelines and infrastructure-as-code processes for Databricks infrastructure and platform configuration.
• Design secure connectivity between Databricks and Azure services such as ADLS Gen2, Key Vault, Azure SQL, Event Hubs, and other enterprise data services.
• Produce architecture diagrams, technical design decisions, deployment documentation, and operational runbooks.
• Provide technical leadership and guidance to data engineers, cloud engineers, security teams, networking teams, and client stakeholders.
• Translate business and security requirements into practical, scalable architecture decisions and guide solutions through implementation and production deployment.

Requirements:

• Significant hands-on experience architecting and implementing Azure Databricks environments in enterprise settings.
• Demonstrated experience personally deploying Azure Databricks using Terraform, including writing, reviewing, and troubleshooting infrastructure-as-code.
• Proven hands-on experience implementing Azure Private Link and private endpoints for Databricks.
• Strong experience with VNet injection, customer-managed VNets, Secure Cluster Connectivity, and no-public-IP Databricks deployments.
• Deep understanding of Azure networking, including private DNS zones, DNS resolution, routing, network security groups, user-defined routes, and Azure firewalls.
• Experience designing and implementing hub-and-spoke Azure network architectures.
• Experience with ExpressRoute and/or VPN connectivity in enterprise environments.
• Strong knowledge of Terraform resources and infrastructure-as-code deployment patterns.
• Experience securing connectivity between Databricks and services such as ADLS Gen2, Key Vault, Azure SQL, Event Hubs, or other Azure data services.
• Strong knowledge of Databricks security, identity and access management, Unity Catalog, Microsoft Entra ID, and service principals.
• Experience establishing CI/CD and infrastructure-as-code practices for enterprise data platforms.
• Ability to diagnose and resolve complex issues involving front-end access, back-end connectivity, browser authentication, private DNS, routing, and cluster connectivity.
• Ability to translate security, technical, and business requirements into clear architecture decisions and practical implementation plans.
• Strong written and verbal communication skills, with the ability to work directly with clients and diverse technical stakeholders.
• Ability to combine strong architectural judgment with a hands-on approach and guide solutions from design through implementation.
• Databricks architecture or engineering certification is preferred.
• Microsoft Azure certification is preferred.
• Experience with Databricks Asset Bundles, Azure DevOps, or GitHub Actions is an advantage.
• Knowledge of data governance, medallion architecture, observability, and cloud cost management is desirable.
• Consulting or professional services experience is a plus.
• Candidates must be based in Canada; the position is fully remote within Canada.

Benefits:

• Remote position within Canada, offering flexibility to work from home.
• Opportunity to work on complex, enterprise-scale Azure, Databricks, and cloud data platform initiatives.
• Hands-on exposure to advanced cloud networking, infrastructure-as-code, data security, governance, and AI/data technologies.
• Opportunity to work directly with enterprise clients and influence critical architecture and technology decisions.
• Collaborative environment involving cloud, data engineering, security, networking, and technology professionals.
• Contract engagement within the technology function.
• Opportunity to take ownership of solutions from initial architecture and design through implementation and production deployment.
• Professional growth through exposure to diverse enterprise cloud and data modernization challenges.