Data Engineer (Databricks)

centurion, Gauteng

Posted 12 days ago

Company:: AICA Consultancy
Company Description:: Founded in 2019 with the core objective and to be one of sub Saharan Africa’s leading business solutions architects, AICA Consultancy is the certified Mict Seta and registered ICT Consultancy that affiliate with different OEMS like Microsoft, Oracle; Cloudera; Hilkvision and Huawei. AICA Consultancy is one the leading ICT Consultancy in the sub-Saharan Africa bridging gaps within Telecommunication and Financial Services space by providing knowledge, skills and experience required, Having seen a big skills shortage in the areas of merging technology, the founders found it critical to meet our clients at their point of need by bridging the skills gap. It started off with training courses in latest technologies that are bringing disruptive innovation in various industries .One of our values is, “nothing beats experience” therefore we pride ourselves with providing professional consultants who are currently practicing in their various areas of expertise thereby exposing our clients to the current, proven and relevant experiences
Contract Type:: Remote
Experience Required:: 5 to 10 years
Education Level:: Bachelor
Number of vacancies:: 3

Job Description

• Data Pipeline Development: o Building efficient ETL/ELT pipelines using Databricks and Delta Lake for structured, semi structured, and unstructured data. o Transforming raw data into consumable datasets for analytics and machine learning.
• Data Optimization: o Improving performance by implementing best practices like partitioning, caching, and Delta Lake optimizations. o Resolving bottlenecks and ensuring scalability.
• Data Integration: o Integrating data from various sources such as APIs, databases, and cloud storage systems (e.g., AWS S3, Azure Data Lake).
• Real-Time Streaming: o Designing and deploying real-time data streaming solutions using Databricks Structured Streaming.
• Data Quality and Governance: o Implementing data validation, schema enforcement, and monitoring to ensure high-quality data delivery. o Using Unity CatLog to manage metadata, access permissions, and data lineage. •
• Collaboration and Documentation: o Collaborating with data analysts, data scientists, and other stakeholders to meet business needs. o Documenting pipelines, workflows, and technical solutions. 3. Deliverables
• Fully functional and documented data pipelines. •
• Optimized and scalable data workflows on Databricks.
• Real-time streaming solutions integrated with downstream systems.
• Detailed documentation for implemented solutions and best practices.