Big Data Engineer

BI Developer

  • AvailabilityPermanent
  • Experience5+ Years
  • SalaryR500,000.00 - R800,000.00
  • Share

Big Data Engineer

BI DeveloperWestern Cape

A leader in the mobile space, based in Johannesburg, is looking for a Senior Specialist - Big Data Engineering. This person delivers through self and others to load the necessary data from local and group sources onto the shared platform that are necessary for analysis and for commercial actions. Building applications that make use of large volumes of data and generate outputs that allow commercial actions that generate incremental value. Support local markets and group functions in obtaining benefiting business value from the operational data


  • Design and develop highly performant and stable applications that processes large volumes of data in a Hadoop environment
  • Build batch and real-time processing applications that ingests source data from the operational systems and output insights back to business systems
  • Integrated applications with business systems to enable value from analytic models to drive rapid decision making
  • Work with the architecture team to evolve the Big Data capabilities (reusable assets/patterns)
  • Participate and contribute in defining technology components to be used on the Big Data Platform
  • Investigating new technologies and methods to improve delivery and sustainability of applications and services

Core competencies, knowledge and experience:

  • Expert level experience in designing, building and managing applications to process large amounts of data in a Hadoop ecosystem
  • Extensive experience with performance tuning applications on Hadoop and configuring Hadoop systems to maximise performance
  • Experience building systems to perform real-time data processing using Spark Streaming or Kafka
  • Experience with managing the development life-cycle for agile software development projects
  • Experience working Hadoop environment;

The ideal candidate for this role will have:

  • Matric / Grade 12 essential
  • 3-year Degree / diploma or Bachelor’s degree in Information Systems, or other related field essential
  • 5+ years BI or related experience essential
  • Working experience with using Yarn, Hive or Oozie
  • Working knowledge of HBase, Solr, Kafka, Nifi or Flume
  • Some experience with other distributed technologies such as Cassandra, Elastic Seatch and Flink would also be desirable
  • Python and Scala programming ability would be an advantage
  • Big Data
  • Hadoop
  • Mobile
  • Digital
  • Spark
  • Kafka
  • Education Tertiary Degree
  • Experience 5+ Years

Similar Jobs