The objective of this project is to scrape BITS Hyderabad's internal Facebook groups and retrieve data of all movie screenings conducted on campus, and consolidate this information cleanly.
I was the Joint Secretary of Movie Club BPHC, and this has always been something we have wanted to track: all the films ever screened on campus. And we know that almost all of this data exists on two Facebook groups, BPHC Shoutbox and BPHC Free Expression Group.
We have previously tried making juniors (slaves) manually go through old posts and update a sheet, but this hasn't been a reliable way of getting this data, especially because the juniors don't find it very enjoyable to go through hundreds of posts and fill sheets with no incentive.
But with a cunnning combination of web scraping techniques and leveraging LLMs, we can finally try to get this data without any slave labour whatsoever.
Facebook doesn't really like it when you use scripts and bots to access their site so every time we login with an automated browser, they throw a captcha at us.
My initial approach was to go through all the posts and specifically only retrieve posts containing certain keywords. This approach didn't work out, mainly because the scraping tools aren't reliable enough to properly scrape the post details.
And our requirements can change frequently.
The better approach in my opinion is to just scrape all the posts. This ensures that we don't have to keep dealing with captchas and only need one proper scraping session to get all the posts. Certain optimisations can be made to improve the efficiency of the scraping as it took almost 4 hours just to scrape 1000 posts.