Skip to content

Commit 838fe5e

Browse files
pavanpaikpavanpaik
committed
feat: AI Leela Generator with quality control and re-indexed article IDs
- Add AI Agent API for structured content generation using Gemini - Implement quality assessment to reject low-quality transcripts - Add Admin AI Generator page for on-demand content creation - Add ToastContext for standardized notifications - Update seed.ts to cleanup database before seeding - Re-index leela_articles.json to preserve original IDs (1, 2, 3...) - Remove deprecated Bodhakatha feature and related files - Update Prisma schema with new Leela fields (story, doubt, revelation, scriptural_refs)
1 parent c983054 commit 838fe5e

45 files changed

Lines changed: 3407 additions & 1943 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.DS_Store‎

0 Bytes
Binary file not shown.

‎data/.DS_Store‎

6 KB
Binary file not shown.

‎data/implementations-script-prompt,md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
Gemeni - GEM
22

3-
Knowledge - 
3+
Knowledge -  WIP
4+
45
# Glossary:
56
    Contains all the glossary terms and their definitions.
67
    teachings from transcriptions of discourses given by KrishnaJI.

‎data/raw-upstream-data/.DS_Store‎

6 KB
Binary file not shown.

‎web/.agent/workflows/generate-leela.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
description: Generate structured Leela article content from a YouTube transcript
3+
---
4+
5+
# Generate Leela Content
6+
7+
This workflow generates a structured JSON object for a Leela article by processing a raw transcript associated with a given `youtube_id`.
8+
9+
## Steps
10+
11+
1. **Find Transcript**
12+
Search for the transcript in `data/raw-upstream-data/yt-transcripts-data.json` using the provided `youtube_id`.
13+
14+
2. **Generate Content**
15+
Using the transcript found in step 1, generate the content with the following prompt:
16+
17+
> **Role & Objective**:
18+
> You are an expert editor and spiritual content curator. Your task is to take the provided raw transcript of a talk by KrishnaJi and convert it into a structured article for the "Leela" section of our website.
19+
>
20+
> **Source Material**:
21+
> [Insert Transcript Here]
22+
>
23+
> **Instructions & Constraints**:
24+
> 1. **No Hallucinations**: Stick strictly to the content of the transcript. Do not invent details.
25+
> 2. **Tone**: Devotional yet analytical, clear, and engaging.
26+
> 3. **Structure**:
27+
> - **The Leela (The Story)**: A narrative retelling of the event. Use H3 headers if needed.
28+
> - **The Conflict/Doubt**: Highlight the specific doubt, question, or skepticism that the devotee had.
29+
> - **The Revelation (KrishnaJi's Reasoning)**: The core teaching or explanation given by KrishnaJi/Sai Baba that resolves the doubt. This is the most important part. Use bullet points or H4 headers for clarity.
30+
> - **Scriptural References**: List any specific chapters or verses mentioned (e.g., "Sai Satcharitra Chapter 25").
31+
> 4. **Formatting**: Use Markdown. Bold key terms.
32+
>
33+
> **Output Format**:
34+
> Return ONLY a JSON object with the following keys: `story`, `doubt`, `revelation`, `scriptural_refs`. Do not wrap in markdown code blocks.
35+
36+
3. **Output Result**
37+
Print the generated JSON object.

‎web/package-lock.json‎

Lines changed: 12 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎web/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
"dependencies": {
1414
"@clerk/localizations": "^3.31.0",
1515
"@clerk/nextjs": "^6.36.3",
16+
"@google/generative-ai": "^0.24.1",
1617
"@neondatabase/serverless": "^1.0.2",
1718
"@prisma/adapter-neon": "^7.2.0",
1819
"@prisma/adapter-pg": "^7.2.0",
@@ -42,4 +43,4 @@
4243
"tailwindcss": "^4",
4344
"typescript": "^5"
4445
}
45-
}
46+
}

‎web/prisma/schema.backup.prisma‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
generator client {
2+
provider = "prisma-client-js"
3+
}
4+
5+
datasource db {
6+
provider = "postgresql"
7+
}
8+
9+
model User {
10+
id String @id @default(cuid())
11+
email String @unique
12+
name String?
13+
image String?
14+
createdAt DateTime @default(now())
15+
updatedAt DateTime @updatedAt
16+
tickets Ticket[]
17+
}
18+
19+
model Ticket {
20+
id String @id @default(cuid())
21+
subject String
22+
status Status @default(OPEN)
23+
userId String
24+
createdAt DateTime @default(now())
25+
updatedAt DateTime @updatedAt
26+
messages Message[]
27+
user User @relation(fields: [userId], references: [id])
28+
lastReadMessageId String?
29+
isArchived Boolean @default(false)
30+
31+
@@index([userId])
32+
}
33+
34+
model Message {
35+
id String @id @default(cuid())
36+
text String
37+
sender Sender
38+
ticketId String
39+
createdAt DateTime @default(now())
40+
ticket Ticket @relation(fields: [ticketId], references: [id], onDelete: Cascade)
41+
42+
@@index([ticketId])
43+
}
44+
45+
model Leela {
46+
id String @id @default(cuid())
47+
orderId Int @default(0)
48+
title_english String
49+
title_hindi String
50+
chapter String?
51+
youtube_id String?
52+
description String
53+
keywords String[]
54+
social_tags String[]
55+
createdAt DateTime @default(now())
56+
updatedAt DateTime @updatedAt
57+
}
58+
59+
model Bodhakatha {
60+
id String @id @default(cuid())
61+
orderId Int @default(0)
62+
theme String
63+
title_english String
64+
title_hindi String
65+
description String
66+
youtube_id String?
67+
keywords String[]
68+
social_tags String[]
69+
createdAt DateTime @default(now())
70+
updatedAt DateTime @updatedAt
71+
}
72+
73+
model Glossary {
74+
id String @id @default(cuid())
75+
term String
76+
chapter String?
77+
definition_en String
78+
definition_es String?
79+
definition_hi String?
80+
createdAt DateTime @default(now())
81+
updatedAt DateTime @updatedAt
82+
}
83+
84+
enum Status {
85+
OPEN
86+
ANSWERED
87+
CLOSED
88+
}
89+
90+
enum Sender {
91+
USER
92+
ADMIN
93+
}

‎web/prisma/schema.prisma‎

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,11 @@ model Leela {
4646
id String @id @default(cuid())
4747
orderId Int @default(0)
4848
title_english String
49-
title_hindi String
50-
chapter String?
49+
transcript String?
50+
story String?
51+
doubt String?
52+
revelation String?
53+
scriptural_refs String?
5154
youtube_id String?
5255
description String
5356
keywords String[]
@@ -56,19 +59,7 @@ model Leela {
5659
updatedAt DateTime @updatedAt
5760
}
5861

59-
model Bodhakatha {
60-
id String @id @default(cuid())
61-
orderId Int @default(0)
62-
theme String
63-
title_english String
64-
title_hindi String
65-
description String
66-
youtube_id String?
67-
keywords String[]
68-
social_tags String[]
69-
createdAt DateTime @default(now())
70-
updatedAt DateTime @updatedAt
71-
}
62+
7263

7364
model Glossary {
7465
id String @id @default(cuid())

‎web/scripts/batch-generate-leela.js‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
/**
2+
* Batch Generate Leela Content
3+
*
4+
* This script generates structured Leela content from YouTube transcripts
5+
* using the AI Agent API at /api/agent/generate-leela
6+
*
7+
* Usage: node scripts/batch-generate-leela.js
8+
*/
9+
10+
const fs = require('fs');
11+
const path = require('path');
12+
13+
// Configuration
14+
const API_URL = 'http://localhost:3000/api/agent/generate-leela';
15+
const TRANSCRIPTS_PATH = path.join(__dirname, '../../data/raw-upstream-data/yt-transcripts-data.json');
16+
const OUTPUT_PATH = path.join(__dirname, '../src/data/leela_articles.json');
17+
18+
// Rate limiting: delay between API calls (ms)
19+
const DELAY_BETWEEN_CALLS = 3000;
20+
21+
// Damu Anna entry to preserve (from existing data)
22+
const DAMU_ANNA_ENTRY = {
23+
"id": 100,
24+
"title_english": "The Story of Damu Anna Kasar",
25+
"youtube_id": "test-damu-anna",
26+
"youtube_url": "https://www.youtube.com/watch?v=test-damu-anna",
27+
"description": "The miraculous story of Damu Anna Kasar and his encounter with Sai Baba.",
28+
"keywords": ["Damu Anna", "Kasar", "miracle", "Sai Baba"],
29+
"social_tags": ["#DamuAnna", "#SaiBaba", "#Miracle"],
30+
"story": "## Leela\n\nIn the village of Shirdi, there lived a devout goldsmith named **Damu Anna Kasar**...",
31+
"doubt": "❓ **Doubt**\n\nDamu Anna questioned: *\"How can a simple fakir know the innermost thoughts of my heart?\"*",
32+
"revelation": "💡 **Revelation**\n\n* Baba demonstrated that He is not bound by physical form\n* The Sadguru sees all, knows all",
33+
"scriptural_refs": "📖 Sai Satcharitra Chapter 10"
34+
};
35+
36+
async function sleep(ms) {
37+
return new Promise(resolve => setTimeout(resolve, ms));
38+
}
39+
40+
async function generateContent(youtubeId, title, transcript) {
41+
console.log(`\n🔄 Generating content for: ${title}`);
42+
console.log(` YouTube ID: ${youtubeId}`);
43+
44+
try {
45+
const response = await fetch(API_URL, {
46+
method: 'POST',
47+
headers: { 'Content-Type': 'application/json' },
48+
body: JSON.stringify({
49+
youtube_id: youtubeId,
50+
title: title,
51+
transcript: transcript
52+
})
53+
});
54+
55+
if (!response.ok) {
56+
const error = await response.json();
57+
console.error(` ❌ Error: ${error.error}`);
58+
return null;
59+
}
60+
61+
const content = await response.json();
62+
console.log(` ✅ Generated successfully!`);
63+
return content;
64+
} catch (error) {
65+
console.error(` ❌ Network error: ${error.message}`);
66+
return null;
67+
}
68+
}
69+
70+
function extractYoutubeId(url) {
71+
const match = url.match(/[?&]v=([^&]+)/);
72+
return match ? match[1] : null;
73+
}
74+
75+
async function main() {
76+
console.log('='.repeat(60));
77+
console.log('🚀 Batch Leela Content Generator');
78+
console.log('='.repeat(60));
79+
80+
// Load transcripts
81+
console.log('\n📂 Loading transcripts...');
82+
if (!fs.existsSync(TRANSCRIPTS_PATH)) {
83+
console.error('❌ Transcripts file not found:', TRANSCRIPTS_PATH);
84+
process.exit(1);
85+
}
86+
87+
const transcripts = JSON.parse(fs.readFileSync(TRANSCRIPTS_PATH, 'utf-8'));
88+
console.log(` Found ${transcripts.length} transcripts`);
89+
90+
// Load existing articles to resume
91+
let leelaArticles = [DAMU_ANNA_ENTRY];
92+
if (fs.existsSync(OUTPUT_PATH)) {
93+
const existing = JSON.parse(fs.readFileSync(OUTPUT_PATH, 'utf-8'));
94+
if (existing.length > 1) {
95+
leelaArticles = existing;
96+
console.log(` Resuming from ${leelaArticles.length} existing articles`);
97+
}
98+
}
99+
100+
let nextId = leelaArticles.length > 0 ? Math.max(...leelaArticles.map(a => a.id)) + 1 : 101;
101+
const startIndex = leelaArticles.length - 1; // Subtract 1 (DAMU_ANNA_ENTRY)
102+
103+
// Process each transcript
104+
for (let i = startIndex; i < transcripts.length; i++) {
105+
const t = transcripts[i];
106+
const youtubeId = extractYoutubeId(t.URL);
107+
108+
if (!youtubeId) {
109+
console.log(`⚠️ Skipping (no valid URL): ${t.Title}`);
110+
continue;
111+
}
112+
113+
// Generate content
114+
const content = await generateContent(youtubeId, t.Title, t.Transcript);
115+
116+
if (content && content.rejected) {
117+
console.log(` ⚠️ Skipped: ${content.reason || 'Transcript rejected'}`);
118+
continue;
119+
}
120+
121+
if (content && content.story) {
122+
// Create article entry
123+
const article = {
124+
id: nextId++,
125+
title_english: content.suggested_title || t.Title || `Leela ${nextId}`,
126+
youtube_id: youtubeId,
127+
youtube_url: `https://www.youtube.com/watch?v=${youtubeId}`,
128+
description: (content.story || '').substring(0, 200) + '...',
129+
keywords: content.keywords || ['Sai Baba', 'Shirdi', 'Leela'],
130+
social_tags: content.social_tags || ['#SaiBaba', '#SaiLeela', '#Shirdi'],
131+
story: content.story,
132+
doubt: content.doubt,
133+
revelation: content.revelation,
134+
scriptural_refs: content.scriptural_refs
135+
};
136+
137+
leelaArticles.push(article);
138+
console.log(` 📝 Added article ID ${article.id}: ${article.title_english}`);
139+
140+
// Progress Save (every 2 articles)
141+
if (leelaArticles.length % 2 === 0) {
142+
console.log(` 💾 Auto-saving progress...`);
143+
fs.writeFileSync(OUTPUT_PATH, JSON.stringify(leelaArticles, null, 2), 'utf-8');
144+
}
145+
}
146+
147+
// Rate limiting
148+
if (i < transcripts.length - 1) {
149+
console.log(` ⏱️ Waiting ${DELAY_BETWEEN_CALLS / 1000}s before next call... (${i + 1}/${transcripts.length})`);
150+
await sleep(DELAY_BETWEEN_CALLS);
151+
}
152+
}
153+
154+
// Write output
155+
console.log('\n📝 Writing output file...');
156+
fs.writeFileSync(OUTPUT_PATH, JSON.stringify(leelaArticles, null, 2), 'utf-8');
157+
console.log(` ✅ Saved ${leelaArticles.length} articles to ${OUTPUT_PATH}`);
158+
159+
console.log('\n' + '='.repeat(60));
160+
console.log('✨ Batch generation complete!');
161+
console.log('='.repeat(60));
162+
console.log('\nNext steps:');
163+
console.log('1. Review the generated content in src/data/leela_articles.json');
164+
console.log('2. Run: npx tsx src/scripts/seed.ts');
165+
console.log('3. Verify in browser: http://localhost:3001/leela');
166+
}
167+
168+
main().catch(console.error);

0 commit comments

Comments
 (0)