Databricks Data Engineer Job Vacancy at Intelligent Debt Management

Position: Data Engineer – Intermediate

(Hybrid/Pinelands, Cape Town)

The Intelligent Debt Management (IDM) Group, recognized as South Africa’s largest and award-winning debt management organization, is seeking a competent Data Engineer to join its expanding Data Team.

This role will focus on data engineering design, data ingestion, and support across both on-premise and cloud platforms, ensuring cutting-edge data solutions are delivered effectively. As part of the IDM team, you’ll participate in a growing company that is shaping the future of data engineering.

Why Join IDM?

Working at IDM offers you the chance to leverage current-generation platforms, such as Databricks and Microsoft Fabric. This is not merely a legacy maintenance position; instead, you will engage in the design, construction, and enhancement of innovative data solutions from the ground up.

As part of a talented group of senior data engineers, you’ll benefit from knowledge sharing, where open technical discussions are encouraged, and individual opinions are highly valued. Our team actively engages in AI and machine learning initiatives, expanding your experience beyond conventional data pipelines.

We hold values like clean architecture, thoughtful design, and creativity in high regard. You will have a say in how projects are built, contribute to architectural decisions, and play a crucial role in shaping our data platform’s evolution.

IDM fosters a respectful, diverse, and collaborative workplace. We invest in our people’s growth and provide an environment that enables engineers to develop both their technical expertise and career paths. Whether you aspire to deepen your engineering knowledge or expand your influence throughout the organization, IDM offers opportunities for professional growth.

Your Impact at IDM

At IDM, the data you engineer empowers real South Africans to enhance their financial well-being. This role provides you with a sense of purpose, knowing that your contributions have a tangible impact.

What We Offer

• Access to a vibrant office situated in a lively area, with convenience in proximity to retail outlets.
• Comprehensive company benefits, including medical aid, retirement annuities, bursaries, and insurance options.
• A culture of continuous growth, allowing for vertical or lateral movement within the organization.

Requirements for the Role

To succeed in this position, candidates must demonstrate the following:

• A minimum of 3 years’ experience as a data engineer or in a comparable role, focusing on designing and implementing data pipelines, integrations, and ETL processes. Experience with Azure, Azure Data Factory, and Azure Databricks is preferred.
• Proficiency in Python/PySpark and SQL/SparkSQL programming.
• At least 3 years in IT, specifically in technical data roles such as data engineering, analytics, or data science.
• Understanding of data management and storage solutions like SQL Server, Azure Data Lake Storage, or comparable technologies.
• Good communication skills along with the ability to thrive in a team environment.
• Exceptional attention to detail and strong problem-solving capabilities regarding data and its structures.

Key Responsibilities

Your primary responsibilities will include:

• Contributing to the architecture and design of the IDM Data Platform and its ingestion patterns.
• Implementing and supporting scalable data solutions on the Microsoft Azure cloud platform, particularly leveraging Azure Databricks.
• Developing, maintaining, and optimizing robust production data pipelines to facilitate data ingestion, processing, and transformation for analytics, machine learning, and other data-driven services.
  • Ensuring data security, quality, and integrity across all pipelines and related data processes.
• Collaborating with the data product owner, analysts, data scientists, and business stakeholders to comprehend data requirements and establish effective data workflows.
• Supporting data initiatives and assisting technical teams, including architects, analysts, software developers, and data scientists.

Qualifications

The ideal candidate will possess:

• A Bachelor’s degree in Computer Science, Engineering, or a related field.
• Certifications in Azure, Microsoft Fabric Data Engineering, Azure Databricks Data Engineering, or related fields, or equivalent practical experience.