Principal Engineer - Databricks Data Engineer
Hybrid Guadalajara, mx, Mexico Full-time
Summary
This role involves auditing and optimizing Databricks environments, implementing Unity Catalog, defining governance standards, and building automation for data platforms. Key technologies include Azure Data Factory, Databricks, and Unity Catalog.
Job Description :
- Audit the current Databricks workspace state, including environments, clusters, Unity Catalog setup, access controls, and cost configuration.
- Implement and harden Unity Catalog, including namespace structure, access controls, cluster policies, and environment separation.
- Define workspace standards, naming conventions, access request workflows, and user/team onboarding processes.
- Establish data quality frameworks, including SLAs, certification criteria, and data ownership models.
- Build automation templates for routine workspace operations such as cluster lifecycle management, cost alerting, and utilization monitoring.
- Stand up observability dashboards covering platform health, cost, job success rates, and DQ SLA KPIs, and monitor core data asset lineage across workspaces.
- Create runbooks for recurring operational tasks that can be executed by a mid-level/junior engineer without ongoing support from the senior resource.
- Conduct knowledge-transfer sessions with the platform engineering team.
- Provide final documentation covering architecture decisions, operational standards, open items, and recommendations for the permanent hire profile.
Key Responsabilities.
- Evaluate and optimize Databricks environments, including workspaces, clusters, catalogs, permissions, and security configurations.
- Define and implement governance standards to ensure secure, scalable, and compliant platform operations.
- Lead Unity Catalog adoption, designing data structures, access models, and environment segregation strategies.
- Establish RBAC policies, workspace standards, naming conventions, and user access processes.
- Drive data governance initiatives, including data quality frameworks, ownership models, SLAs, lineage, and stewardship practices.
- Develop automation solutions and Infrastructure as Code (IaC) frameworks to streamline platform administration and reduce manual effort.
- Implement cost optimization, monitoring, and operational excellence practices across Databricks environments.
- Design observability and performance dashboards to track platform health, utilization, job success rates, and data quality KPIs.
- Create operational documentation, runbooks, and support procedures to improve platform reliability and team readiness.
- Partner with security, compliance, infrastructure, and data engineering teams to align platform capabilities with business needs.
- Provide training, mentoring, and knowledge transfer sessions to empower platform administrators and engineering teams.
- Recommend platform improvements and long-term strategies to enhance governance, scalability, and organizational maturity.
Must have skills: Azure Data Factory, Databricks, Unity Catalog
Good to have skills: Data platform implementation, Google Workspace Administration