Senior Data Engineer

Senior Data Engineer

Location: Herndon, VA (Remote Work)

Must have an Public Trust Clearance

KEY RESPONSIBILITIES

•      Provide authoritative expertise on data engineering methods and best practices, including code first development approaches and modern pipeline design patterns.

•      Design, implement, and maintain the data architecture that supports products and end users, with all assets managed under source control.

•      Design, implement, and maintain ELT and ETL pipelines for efficient processing of source data in Azure Synapse and Azure Machine Learning, using both SDK V1 and SDK V2.

•      Migrate source data identified by SBA OIG into Azure Data Lake Storage.

•      Normalize entity attributes such as addresses, phone numbers, and other common fields.

•      Review, maintain, and improve existing architecture and pipelines, including periodic audits addressing bottlenecks, deprecated dependencies, and architecture drift.

•      Establish quality controls across all pipelines and introduce error handling, logging mechanisms, and validation checks.

•      Incorporate source control across all pipelines and analytics codebas