Social Media Political Analysis (SMPA) Pipeline
This step was guided by my anticipation that the data will be used for trend graphing, sentiment analysis, age inference, and correlation between user characteristics and extent of participation in responding to political events. With this in mind, I plan to optimize query speed via limiting storage by geographic location of both users and events to the US (though this categorization may be loose at times because of US involvement on the world stage). My PostgreSQL database will also be sharded by datetime, as the analytical window references 3 days before and 3 days after the political event of interest.
The article text in the news table will serve as data sources from which to extract our top 3 keywords (per article) using Term Frequency-Inverse Document Frequency (TF-IDF) calculations. These keywords will be applied to our searches for related TikToks and tweets. For some resources on this, check out TF-IDF in the real world and a step-by-step guide by Prachi Prakash.
With data from social media adding a pop-cultural context to political news, we inch closer to an understanding of TikTok and Twitter as novel forms of youth political engagement!
The ETL pipeline code will call multiple APIs (NewsAPI, Tweepy, & TikTok), transform the data in Python, and load that data into Postgres. Because this is pretty niche data, there is a small enough amount of data ingested to justify an ETL approach.
- NewsAPI finds top news by day
- Parse news story titles and their article text into individual words and phrases
- Determine 3 most important individual words and phrases
- Use key words as filters for TikTok and Twitter data
- Count the number of tweets and TikToks mentioning key words and phrases
PgAdmin UI Postgres as DW
<//>
Data is batch-ingested into ...
Data is served through a Flask API
- GraphQL: more efficient than REST due to compound requests pulling data in one lump sum
- Rate-limiting factors
- Firewall
- TLS encryption
- Caching
- Testing in isolation for funtionality, reliability, latency, performance, security, etc.
- Testing with JSON payloads over HTTP, HTTPS, JMS, & MQ
- Unit testing: individual operations of API in logical unit divisions to ID imperfections in early stages
- Functional testing: testing all functions in codebase, using unit tests as building blocks
- Load testing: validating functionality and performance under load to ensure it will work as expected with multiple concurrent users
- Runtime error detection: monitoring API by running it in its entirety to check for errors, exceptions, and prevent resource leaks
- Security testing: checking for external threats, validation, access control, and data encryption
- Penetration testing: to find vulnerabilities in system or codebase that could be exploited by attackers, testing vulnerability of functions and security assets
- Fuzz testing: utilizes random input data (aka fuzz) to rest reliability and ensure reliability in worst-case
- no need to stream updates
- don't need dbt with pyspark sql transformations