Project: Build a Full Tools Catalog + Connections Graph for All Content

[kavaivaleri]

📝 Background & Objective

Across our podcasts, Zoomcamp lectures, webinars, and workshops, we mention hundreds of data science, AI, and ML tools. Right now, these mentions are scattered. While podcast transcripts live on our website in the _podcast folder, our webinars and workshops are only stored as video recordings on our YouTube channel.

We want to build a dynamic tools encyclopedia that consolidates all these mentions across all of DataTalks.Club ecosystem.

We want to build an automated system that extracts mentions of these tools, generates structured tool pages, and builds a central catalog on the website.

This catalog will be incredibly valuable for learners navigating the ecosystem, heavily boost our SEO, and create opportunities for partnerships with tool creators.

✅ Core Requirements

You will need to build an extraction and generation pipeline that handles the following:

1. Multi-Source Transcript Processing

Your system needs to handle two different data sources:

• Podcasts: Parse the existing YAML/JSON transcripts located in the _podcast folder of this repository.
• Webinars & Workshops: Fetch and parse automated transcripts directly from our YouTube channel's video recordings.

2. Tool Extraction (LLM Pipeline)

Read the transcripts and accurately identify mentions of specific software tools (both open-source and proprietary).
The system must extract and link: Tool → Source Video → Timestamp → Context Snippet.

3. Generate Structured Tool Profiles

For every extracted tool, compile the data into a structured format (e.g., Markdown or JSON) that includes:

• Name of the tool
• Category / Purpose (e.g., "Vector Database," "Orchestration," "Data Quality")
• Short Description
• Mentions: A list of all appearances across our content, including the Video/Episode Title, Timestamp, Speaker Name, and the exact Quote/Context snippet.
• Metadata (Optional but encouraged): Automatically pull in the tool's Homepage, GitHub repo, or Documentation link.

4. Connections Graph (Optional but Highly Encouraged)

Map the relationships between tools, concepts, and episodes. You can use graph tooling like Cognee (or a similar graph DB/framework) to generate and visualize these connections.

Reference: Check out a demo of Cognee in action on our channel here.