DBT-Project

Steps to run the code:

Start confluent services using confluent local services start
Open up a new terminal and run python3 twitter_stream.py
Open another terminal and run spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.0 spark_ingest.py
Open a third terminal and run the Kafka consumer using python3 consumer.py
You will be able to observe the consumer reading the spark data. Check the database to see if the data has been entered into the db.

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
0_Kafka_running.png		0_Kafka_running.png
1.2_Twitter_streaming.png		1.2_Twitter_streaming.png
1_twitter_streaming_spark.png		1_twitter_streaming_spark.png
2.5_batch_process.jpeg		2.5_batch_process.jpeg
2_twitter_to_spark.png		2_twitter_to_spark.png
3_Spark_to_kafka_before_5.png		3_Spark_to_kafka_before_5.png
4_kafka_consume_from_spark.png		4_kafka_consume_from_spark.png
README.md		README.md
consumer.py		consumer.py
spark_ingest.py		spark_ingest.py
tokens.txt		tokens.txt
twitter_stream.py		twitter_stream.py