Data Engineer

As a Data Engineer, your main goal is to build and maintain the systems that process and store data for the Events & Exhibitions ecosystem. You will take scattered, vendor-specific data (from registration systems, apps, marketing tools) and transform it into a unified, AI-ready dataset using a Medallion Architecture on Azure.

Think of it as organizing raw data into a structured pipeline that’s ready for analysis and machine learning.

Requirements

1. Data Ingestion & API Integration (Bronze Layer)

• Build and manage robust ETL/ELT pipelines using Azure Data Factory to ingest data from 3rd-party vendors (REST APIs, Webhooks, SFTP).
• Ensure raw data is landed securely in Azure Data Lake Gen2 (Bronze Layer) without data loss.
• Implement error-handling and logging to monitor the health of real-time and batch ingestion jobs.

2. Transformation & Modeling (Silver & Gold Layers)

• Utilize PySpark (Azure Databricks/Synapse) and SQL to clean, deduplicate, and standardize data in the Silver Layer.
• Execute Identity Resolution logic to stitch together visitor and exhibitor profiles from multiple touchpoints into a "Golden Record."
• Develop optimized data sets in