Lead Data EngineerApply Now

About the client:
Our client is based in the northern suburbs of Melbourne and is committed to providing a seamless travel experience and enhancing operational efficiency through innovative technology solutions. Our client is seeking an experienced Lead Data Engineer to join their dynamic team and contribute to our mission of delivering exceptional services.

About the role:
As a Lead Data Engineer you will bring extensive experience in Databricks,
preferably Microsoft Dynamics, and a solid understanding of Microsoft Stack – Fabric to join our growing data engineering team. You will be a strong technical
leader with expertise in building scalable data pipelines, managing big data
environments, and working with a variety of cloud-based data engineering tools.

Key Responsibilities:

• Data Pipeline Development: Design, implement, and maintain robust, scalable, and efficient ETL (Extract, Transform, Load) pipelines using Databricks, Azure Data Factory, and other tools within the Microsoft Stack – Fabric.
  • Cloud Data Infrastructure: Architect, manage, and optimize Azure Data Lake, Azure Synapse Analytics, and Databricks environments to enable seamless data processing at scale.
  • Collaboration with Cross-functional Teams: Work closely with data scientists, analysts, business intelligence teams, and other stakeholders to understand data requirements and ensure high-quality data solutions are delivered.
  • Real-Time Data Processing: Build and optimize real-time data pipelines using Azure Stream Analytics, Event Hubs, or other stream-processing technologies for time-sensitive analytics.
  • Data Governance and Quality: Implement data governance best practices, ensuring data accuracy, consistency, and security across all datasets and pipelines.
  • Optimization and Performance Tuning: Regularly analyze and optimize the performance of data workflows and queries, ensuring scalability and high availability in cloud environments.
  • Mentorship: Provide guidance and mentorship to junior data engineers and help grow the data engineering team’s technical expertise.

Qualifications:

• Experience in data engineering or a related role, with a strong emphasis on working with large-scale data processing and cloud platforms.
• Expertise in Databricks and hands-on experience working with Apache Spark in cloud environments.
• Experience with Azure Data Services, including Azure Data Factory, Azure Synapse Analytics, Azure Data Lake, and other Microsoft Azure-based data engineering tools.
• Familiarity with Microsoft Stack – Fabric and how it integrates with Azure Synapse, Power BI, Azure Machine Learning, and other related technologies.
• Proficiency in SQL and experience working with relational and non-relational databases.
• Solid understanding of data modeling, ETL processes, and data pipeline automation.
• Experience with Microsoft Dynamics (preferred) or similar enterprise resource planning (ERP) systems is a plus.
• Familiarity with big data technologies and distributed computing concepts
• Bachelor’s degree in computer science, Engineering, Information Technology, or a related field.

Benefits:

• Competitive salary and benefits package.
• Opportunities for professional development and career advancement.
• A collaborative and inclusive work environment.
• The chance to contribute to innovative projects 

Does this sound like you? If you have the skills and experience we’re looking for please click ‘Apply Now’, or please send your updated CV to suniva.rego@mtr.com.au Alternatively please call 03 9643 3720 for a confidential discussion about the role.

Apply Now