기본 콘텐츠로 건너뛰기

[ROAD TO DATA ENGINEER] 09.24

 What Is Spark?

: Open-source distributed general-purpose cluster-computing framework.

How to start data engineering projects?

  1.   Choose any framework, let's say Kafka.
  2.   Write some codes using that framework.
  3.    Keep Expanding (try adding other technology)

Project Idea : Creating Real Time REST API

  1. crawls data from popular websites like Twitter,Forex.
  2. store them in buffer (by producer in Kafka).
  3. store them in MySQL database (by consumer in Kafka).
  4. Web server provides real-time REST API.

댓글