About the Role
You will design, develop, and maintain highly scalable, reliable, and efficient data processing systems with a strong emphasis on code quality and performance. You will collaborate closely with data analysts, software developers, and business stakeholders to deeply understand data requirements and architect robust solutions to address their needs. You will focus on developing and maintaining ETL pipelines that extract, transform, and load data from diverse sources into a Databricks-based data warehouse. You will spearhead the development of real-time data processing systems using technologies like Spark Streaming and Kafka. You will establish rigorous data quality and validation checks, troubleshoot complex data processing issues, and monitor and optimize systems for peak performance and scalability using AWS services such as EC2, S3, Glue, and Databricks. You will implement robust security measures to safeguard sensitive data and stay current with the latest AWS advancements to continually improve data processing capabilities.
Requirements
- Bachelor's or Master's degree in Computer Science or a related field
- Minimum of 5 years of hands-on experience as a Data Engineer
- Good understanding of Databricks platform and Delta Lake
- Familiar with data job scheduler tools such as Dagster
- Proficiency in one or more programming languages such as Scala, Java, or Python
- Deep expertise in big data technologies including Apache Spark for ETL processing and optimization
- Proficient in utilizing BI tools such as Metabase for data visualization and analysis
- Advanced understanding of data modeling, data quality, and data governance best practices
- Outstanding communication and collaboration skills
- Extensive experience in AWS operational management
- Strong understanding of AWS services such as EC2, S3, Glue and EMR
- Proficiency in AWS security best practices
- Hands-on experience with automation and DevOps tools such as Terraform
- Can read/write in English
Responsibilities
- Design, develop, and maintain highly scalable, reliable, and efficient data processing systems
- Collaborate with data analysts, software developers, and business stakeholders to architect data solutions
- Develop and maintain ETL pipelines for extracting, transforming, and loading data into the Databricks data warehouse
- Develop and maintain real-time data processing systems using Spark Streaming and Kafka
- Establish and enforce data quality and validation checks
- Troubleshoot and resolve complex data processing issues across teams
- Monitor and optimize data processing systems for performance, scalability, and reliability
- Utilize AWS services such as EC2, S3, Glue and Databricks to architect and manage data infrastructure
- Implement security measures and access controls to safeguard data assets
- Stay current with AWS technologies and best practices