Добрый день, Нужен специалист в Apache Beam (Google Dataflow) и Kafka. TASK 1: We have many Kafka topics with real time data. And we need to archive data from Kafka to json files (archived) once per day. So we need to build a script (Python or Scala) with the logic like next: 1) Script runs once per day. 2) It starts Apache Beam job to read data from hundreds of Kafka topics and put that data to json file and send it to Google Cloud Storage. 3) Each topic has its own separate file for each day of the data TASK 2: Another task is to create Apache Beam job that reads data from few (up to 100) different Kafka topics in realtime and sends it to new Kafka topic synchronised by timestamp.

Похожие

Посмотрите другие вакансии

Вот самые похожие вакансии

Новый поиск