Senior Azure Data Engineer - #35291

Key Accountabilities/BAU Objectives:

• Pipelines from Source to Reporting
• Design, build and maintain the pipelines across the medallion layers, from raw ingestion through to the datasets reporting depends on. You own yours end to end, including how they behave when they fail.
• Pipelines delivered end to end without needing a rebuild
• Refresh schedules met, with failures caught by monitoring rather than by a user
• Pipelines built to the team standard so someone else can pick them up

Engineering Depth and Data Design

• Transformations are production code: readable, tested and maintainable. Idempotency and safe replay, correct incremental logic, modelling for how the data is queried, and the judgement to see that generated SQL is the wrong shape even when it returns the right number today.
• Transformation logic readable, tested and reusable rather than one off
• Pipelines that replay and backfill correctly by design
• Work another engineer can pick up without a handover conversation

Ingestion from Product Sources

• Bring data in from operational systems through gateways or equivalent connectors, handling incremental loads, schema drift and late arriving data without silent loss.
• Source data landed completely and repeatedly, reconciled against the source
• Incremental loads correct on replay an