Keywords: Data Engineering, ETL, Metadata-driven Design, Data Quality Checks, Data Integrity Checks, Workflow Orchestration
Our client is seeking a Principal Data Engineer to join their dynamic team in Singapore. This role is perfect for someone who thrives on developing reusable frameworks and tools that enable teams to build pipelines and workflows efficiently. The successful candidate will focus on designing metadata-driven solutions for ETL, workflows, and data specifications, optimizing data integration processes, and ensuring seamless interaction with the platform’s core components. This is an exciting opportunity to work in a supportive environment where your skills will be valued and nurtured.
- Design and implement flexible, metadata-driven frameworks
- Develop automated data quality checks using tools like Great Expectations or similar solutions
- Build seamless integrations with core platform components like Apache Iceberg, Spark, Kafka, Airflow, and Argo
What you'll do:
As a Principal Data Engineer, you will play a pivotal role in enabling teams to build pipelines and workflows efficiently. Your expertise in designing metadata-driven solutions for ETL, workflows, and data specifications will be crucial in optimizing data integration processes. You will also be responsible for implementing robust data quality and integrity checks. Your ability to seamlessly integrate with the platform’s core components will be key in ensuring consistent processing across batch and streaming workloads. Your commitment to adhering to governance policies and regulatory requirements will ensure the success of our operations.
- Design and implement flexible, metadata-driven frameworks for ETL, workflow orchestration, and data specifications to support efficient pipeline creation
- Establish best practices and reusable components for data ingestion, transformation, and processing to streamline efforts across teams
- Build frameworks leveraging metadata to define and automate ETL processes, workflows, and data specifications
- Develop and integrate automated data quality checks into frameworks using tools like Great Expectations or similar solutions
- Ensure frameworks include mechanisms to validate data integrity across pipelines
- Build seamless integrations with core platform components like Apache Iceberg, Spark, Kafka, Airflow, and Argo
- Develop metadata-driven workflow orchestration frameworks to enable easy configuration and deployment of ETL processes
- Build frameworks with fail-safe mechanisms to handle errors and ensure reliability in pipeline execution
- Ensure frameworks adhere to governance policies and regulatory requirements
What you bring:
As the ideal candidate for the Principal Data Engineer position, you bring a wealth of experience in Python and Java, with a focus on building reusable frameworks. Your expertise in metadata-driven framework design for ETL, workflow orchestration, and data specifications sets you apart. You are proficient in data quality and integrity tools like Great Expectations or similar solutions. Your hands-on experience with Apache Spark, Kafka, and orchestration tools like Airflow or Argo will be invaluable to our team. Your familiarity with governance tools like Apache Ranger, OpenLineage, and DataHub will ensure our compliance with regulatory standards.
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field
- Expertise in Python, Java, with a focus on building reusable frameworks
- Experience with metadata-driven framework design for ETL, workflow orchestration, and data specifications
- Proficiency in data quality and integrity tools like Great Expectations or similar solutions
- Hands-on experience with Apache Spark, Kafka, and orchestration tools like Airflow or Argo
- Familiarity with governance tools like Apache Ranger, OpenLineage, and DataHub
What sets this company apart:
Our client is an international bank that has been making a positive difference for their clients, communities, and employees for more than 100 years. They question the status quo, love a challenge and enjoy finding new opportunities to grow and do better than before. They value difference and advocate inclusion. When you work with them, you'll see how they put the client at the heart of what they do, continuously strive to improve and innovate, keeping things simple and learning from doing well, and not so well. They believe in working collectively to build for the long term.
What's next:
Ready to take your career to the next level? Apply now!
Apply Today by clicking on the link!
Do note that we will only be in touch if your application is shortlisted.
Robert Walters (Singapore) Pte Ltd
ROC No.: 199706961E | EA Licence No.: 03C5451
EA Registration No.: R21100958 Harsh Paras Mehta
