Intelligent Debt Management
Position: Data Engineer – Intermediate
(Hybrid/Pinelands, Cape Town)
The Intelligent Debt Management (IDM) Group, South Africa’s largest and multi-award winning debt management company, is seeking a talented Data Engineer to become a crucial member of our Data Team. This exciting opportunity entails designing and implementing data engineering solutions while providing essential support across both on-premise and cloud platforms.
What’s in it for You?
IDM is not just the largest company of its type in South Africa; we are committed to growth and innovation without limits. By working with cutting-edge platforms such as Databricks and Microsoft Fabric, you will gain hands-on experience with tools that redefine the future of data engineering. In this role, you won’t be merely maintaining legacy systems; instead, you will actively design, build, and develop new data solutions, allowing for ample creativity and technical judgment.
Our high-caliber team of senior data engineers promotes a culture of knowledge sharing. We encourage technical discussions and respect all opinions, ensuring decisions are made based on merit rather than hierarchy. You’ll also be involved in exciting initiatives related to AI and machine learning, extending your expertise beyond conventional data pipelines.
Why Choose Us?
We value clean architecture, thoughtful design, and creativity. In this collaborative environment, you will have the opportunity to influence architecture decisions and contribute to the evolution of our data platform. Culturally, we pride ourselves on our diverse and respectful team atmosphere. We invest in our employees, champion continuous learning, and provide an environment that facilitates both technical and professional growth.
At IDM, you will find a stable and growing environment. The data solutions you create will help real South Africans improve their financial circumstances, offering a sense of purpose and impact to your work.
Additional Benefits
- Located in a vibrant office within a lively part of Cape Town, providing easy access to amenities and stores.
- Comprehensive company benefits including medical aid, retirement annuity, bursaries, and insurance.
- A culture that fosters professional growth, allowing significant opportunities for upward or lateral movement within the company.
Qualifications and Requirements
To be considered for this role, candidates should demonstrate the following:
| Requirement | Description |
|---|---|
| Experience | Proven background as a data engineer or in similar positions, with skills in designing and implementing data pipelines, data integration, and ETL processes. Preference for hands-on experience with Azure services like Azure Data Factory and Azure Databricks. |
| Programming Skills | Proficient in Python/PySpark, along with strong SQL/SparkSQL programming capabilities. |
| IT Experience | A minimum of 3 years of experience in Information Technology, particularly in data engineering, analytics, or data science roles. |
| Data Management Knowledge | Familiarity with data management and storage solutions, including SQL Server Database and Azure Data Lake Storage or similar technologies. |
| Communication Skills | Ability to engage effectively in a team setting and communicate data requirements clearly. |
| Attention to Detail | Exceptional attention to detail coupled with robust problem-solving skills relevant to data structures and algorithms. |
Key Responsibilities
As a Data Engineer at IDM, your key responsibilities will include:
- Contributing to the architecture and design of the IDM Data Platform and associated data ingestion patterns.
- Implementing and maintaining scalable data solutions on the Microsoft Azure cloud platform, including Azure Databricks.
- Developing, maintaining, and optimizing robust production data pipelines for data ingestion, processing, and transformation, to support analytics, machine learning, and other data-driven services.
- Ensuring data security, quality, and integrity throughout all data pipelines and processes.
- Collaborating with the data product owner, analysts, data scientists, and business stakeholders to ascertain data requirements and establish effective data workflows.
- Supporting data initiatives and assisting technical teams, including architects, analysts, software developers, and data scientists.
Educational Qualifications
Candidates must possess the following educational qualifications:
- A Bachelor’s degree in Computer Science, Engineering, or a related field.
- Relevant certifications in Azure, Microsoft Fabric Data Engineering, Azure Databricks Data Engineering, or similar practical experience.
Was this helpful?
0 / 0