Senior Data Engineer
Remote Warsaw, Masovian Voivodeship, pl, Poland Full-time 3+ yrs
Summary
Design and build scalable cloud-native data platforms, implement near-real-time ingestion pipelines, and ensure data quality. This role involves refactoring Spark scripts, driving AI tooling adoption, and collaborating with customer teams using Python, PySpark, SQL, Databricks, and AWS.
- Design and build scalable cloud-native data platforms from greenfield to production
- Design and implement near-real-time ingestion pipelines using event-driven patterns
- Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts
- Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability
- Introduce best practices for code quality, testing, and CI/CD across data pipelines
- Drive adoption of AI tooling and agentic workflows within the data engineering team
- Ensure data quality, observability, and reliability across pipelines and platforms
- Design and deliver self-service tooling and microservices that abstract platform complexity
- Collaborate directly with the Customer’s scientific and engineering teams
- Monitor platform performance, reliability, and cloud infrastructure costs
- 5+ years of experience in data engineering
- 3+ years of production experience with Databricks and Delta Lake
- Strong expertise in medallion architecture and event-driven asynchronous pipeline design
- Advanced proficiency in Python, PySpark, and SQL
- Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization
- Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway
- Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles
- Practical experience with Git workflows, pull request reviews, and automated testing
- Experience configuring secure CI pipelines using OIDC or service principals
- Practical MLflow experience including experiment tracking, model registration, and reproducibility practices
- Strong understanding of data quality, lineage, provenance, schema contracts, and versioning
- Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX
- Experience with monitoring, observability, and cloud cost optimization
- Upper-Intermediate English level or higher
WILL BE A PLUS
- Experience packaging workloads in Docker containers and running batch jobs on AWS
- Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy
- Experience building feature sets and train/test splits that avoid data leakage
- Experience with regulated or high-security environments including data residency requirements
- Experience disabling or restricting AI features in secured environments
PERSONAL PROFILE
- Strong analytical mindset
- Ownership and accountability
- Ability to communicate complex technical concepts clearly
- Proactive approach to problem-solving
- Attention to scalability and maintainability
- Collaborative and team-oriented attitude