🎬 CineInsight - A Movie Recommendation & Analysis Engine using Apache Spark

📌 Problem Statement

With the rapid growth of online streaming platforms, users are overwhelmed by the sheer number of movies available. Recommender systems help users discover relevant content by analyzing past interactions and preferences.

The goal of this project is to:

• Analyze large-scale movie rating data using Spark SQL
• Build a personalized movie recommendation system using ALS (Alternating Least Squares)
• Store and process data using a distributed HDFS environment
• Demonstrate Big Data concepts such as distributed storage, parallel computation, and ML pipelines

---

🛠 Technologies Used

Category	Technology
Big Data Framework	Apache Spark 3.5.1
Storage	HDFS (Hadoop 3.3.6)
Language	Python (PySpark)
ML Library	Spark MLlib (ALS)
Query Engine	Spark SQL
Dataset	MovieLens (ml-latest-small)
OS	Ubuntu (WSL2)
IDE	IntelliJ IDEA / VS Code

---

🧱 Project Architecture

⚙️ Setup and Installation

1️⃣ Prerequisites

Ensure the following are installed:

• Java 11
• Hadoop 3.3.6
• Spark 3.5.1 (Hadoop-compatible build)
• Python 3.12
• WSL2 (Ubuntu)

---

2️⃣ Set up your Environment Variables

Add the following to your .bashrc or .zshrc:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export HADOOP_HOME=~/hadoop-3.3.6
export SPARK_HOME=~/spark-3.5.1-bin-hadoop3
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip
export PYSPARK_PYTHON=python3.11
export PYSPARK_DRIVER_PYTHON=python3.11

Add these environment variables using the command :

nano ~/.bashrc

After adding it, save it using the command :

Ctrl + O  -> Enter
Ctrl + X

Now, apply the changes made using the command :

source ~/.bashrc

▶️ How to Run the Application

Step 1: Start HDFS

cd ~/hadoop-3.3.6
sbin/start-dfs.sh
jps

Ensure

NameNode, DataNode

, and

SecondaryNameNode

are running.

Step 2: Move Dataset to HDFS

hdfs dfs -mkdir -p /user/<username>/movielens/ml-latest-small
hdfs dfs -put *.csv /user/<username>/movielens/ml-latest-small

Verify:

hdfs dfs -ls /user/<username>/movielens/ml-latest-small

Step 3: Run Spark SQL Analysis

cd ~/IdeaProjects/MovieRecommendationSystem
spark-submit spark/spark_sql_analysis.py

Sample Output (Screenshots) :

Top 10 Most Rated Movies

Top 10 Highest Rated Movies (min 50 ratings)

Genre-wise Average Rating

Step 4: Run ALS Recommendation Engine

spark-submit spark/als_recommender.py

Sample Output (Screenshots) :

Top 10 movie recommendations for a randomly selected user

Evaluation Metrics (RMSE & MAE)

📊 Inference

• The ALS model successfully learns latent user and movie factors
• Popular movies tend to have higher rating counts but not always higher ratings
• RMSE and MAE values indicate acceptable prediction accuracy for a dataset containing 100,000 ratings and 3,600 tag applications applied to 9,000 movies by 600 users.
• Using HDFS enables scalable storage and fault tolerance
• Spark SQL efficiently handles analytical queries over large datasets.

🚀 Future Work

• Scale to MovieLens 20M / 25M dataset
• Add implicit feedback (views, clicks)
• Implement Top-N ranking metrics (Precision@K, Recall@K)
• Add genre-based recommendations.
• Integrate real-time streaming (Kafka + Spark Streaming)
• Build a REST API / Web UI for recommendations.
• Compare ALS with content-based filtering

Author: Shrika Thota