In Data CoE, we are seeking a senior consultant to support our migration to Databricks as a hands-on expert advisor. The consultant will support internal teams by helping to shape and define architecture-related decisions, DevOps and data management cycle standards and best practices, and clarify options with their consequences with emphasis on efficiency and migration time constraints

What will you do?

• Databricks architecture and engineering patterns,
• practical guidance on how to structure workloads, delivery patterns, standards and reusable technical approaches in Databricks.
• Governance/data management lifecycle
• Metadata management, tagging, dictionaries, catalogs, Unity Catalog setup, native options
• Data quality / DQMS approach help evaluate options for data quality management, including native Databricks monitoring/profiling capabilities, process implications, and possible hybrid approaches.
• Airflow and orchestration
• advice on orchestration options and trade-offs, including how Airflow can be used with Databricks jobs + best practices/standards
• DevOps setup
• support in defining practical approaches in terms of CICD, metadata/template driven approach, monitoring and ops, connection to dictionaries and catalogs (incl. tools)
• standards and conventions — support in defining basic standards such as naming conventions, repository structure, deployment principles, and selected governance guardrails.

 

The proposed consultant should have:

• Strong hands-on Databricks experience in real delivery environments
• Practical experience with Airflow orchestration in combination with Databricks
• Strong CI/CD experience for Databricks, including Git-based development, testing, deployment automation, operations
• Experience with data quality / monitoring / DQ frameworks, ideally in Databricks or similar modern data platforms.
• Ability to provide practical recommendations, alternatives, and consequences of choices
• Ability to work in a supporting, collaborative mode