Skip to content

07-DataSources.md

Latest commit

 

History

History
259 lines (202 loc) · 16.1 KB

07-DataSources.md

File metadata and controls

259 lines (202 loc) · 16.1 KB
Raw

100 Plus Data Sources Data Science

This is a section with links to data sources. During my data engineer coaching we need to find good data sets to work with. So, I started this section to make it easier to find good sources.

I've taken these links from articles and blog posts. Why not only link the articles? You know, these posts can go away at any time. I want to keep the links to the platforms either way.

I haven't had the chance to check each link myself. Please let me know if something isn't right.

You can find the articles on the bottom of this section to read more. They include even more data sources I haven't had time to add to this list.

Contents

Student Favorites

In my Coaching program my students learn by doing a project. And the foundation of every project is selecting a dataset. That can be an API or a file source, depending a lot on the student's goals and interests.

Working out goals for the dataset, figuring out the data modeling, creating the architecture and building it. It's a fun way to learn and get better at Data Engineering.

Here's a list of my student's favorite datasets and APIs

Building a project? Use my project README template to document it properly: template-readme.md

Datasets

Dataset Description
Fraud detection Simulated credit card transactions labeled as fraudulent or legitimate, ideal for classification and imbalanced-data projects.
Industrial equipment monitoring Sensor readings from industrial equipment (temperature, pressure, vibration) for predictive maintenance and anomaly detection.
Energy demand & generation Four years of hourly electricity consumption, generation, prices and weather data for Spain, great for time-series forecasting.
Online Retail Transactional data from a UK online retailer, useful for customer segmentation, market basket and RFM analysis.
Brazilian E-commerce Olist's public dataset of 100k+ orders with customers, products, payments, reviews and geolocation for end-to-end analytics.
Beijing Air Quality Hourly air pollutant readings across multiple Beijing stations with weather data, good for time-series and environmental analysis.
NYC Taxi New York City taxi trip and revenue data for cab drivers, useful for revenue, tipping and time-series analysis.
E-commerce clickstream & transactions Simulated online shopping clickstream and transaction data — user sessions, page views, clicks and purchases — good for funnel and behavior analysis.
IESO Ontario electricity data Public report files from Ontario's grid operator: electricity demand, generation by fuel type, market prices and more, updated regularly as CSV/XML.
IoT device temperature readings Temperature sensor readings from IoT devices with indoor/outdoor tags and timestamps, great for time-series and anomaly detection.
IoT agriculture 2024 Smart-farming sensor data (soil moisture, temperature, humidity, etc.) for agriculture monitoring and IoT analytics projects.

APIs

API Description
Bike sharing Bluebikes Trip and station data for Boston's Bluebikes bike-share system, including GBFS real-time feeds and downloadable trip history.
Bike sharing Divvy Bikes Trip and station data for Chicago's Divvy bike-share system, with GBFS real-time feeds and historical ride files.
Weather API Weather API with current conditions, forecasts, historical data and astronomy; generous free tier.
Bluesky API API for the Bluesky social network (AT Protocol): posts, profiles, feeds and social graph data.
Guardian news API Access to The Guardian's articles, tags and sections; great for text and news analysis, free with an API key.
Football API Football (soccer) data API: fixtures, live scores, standings, teams, players and statistics across many leagues.
HSL Helsinki public transport Open transit data for the Helsinki region: routes, schedules, GTFS feeds, real-time vehicle positions and a routing API.
Digitransit map API Background map and vector tiles API for building transit and mobility maps (Finland/Helsinki region).
Open-Meteo Free open-source weather API with forecasts, historical data and air quality, no API key required.
Finnish Meteorological Institute open data Weather observations, forecasts and climate data for Finland, served via open WFS/download APIs.
OpenWeatherMap Weather API with current conditions, forecasts and historical data via the One Call API; free tier plus paid plans.
SMARD German electricity market API API for Germany's electricity market data: generation, consumption and prices from the Bundesnetzagentur (documentation in German).

General And Academic

Content Marketing

Crime

Drugs

Education

Entertainment

Environmental And Weather Data

Financial And Economic Data

Government And World

Health

Human Rights

Labor And Employment Data

Politics

Retail

Social

Travel And Transportation

Various Portals

Source Articles and Blog Posts