✅ Steve Jobs Complete Corpus - READY FOR OPENAI

Status: RECOVERED & COMPLETE

The corpus was recovered from git commit e945899 with all data intact.

📊 Final Statistics

Metric	Value
Total Documents	500
Total Words	1,472,740
Cognitive Patterns	10,427
File Size	13.9 MB
Years Covered	1980-2022

📁 File Structure

final_corpus/
├── steve_jobs_complete_corpus.jsonl  (13 MB, 500 lines)
├── steve_jobs_complete_corpus.json   (13 MB, human-readable)
└── split_files/                      (500 individual files)
    ├── keynote/        89 files  (7.7 MB)
    ├── interview/      85 files  (3.7 MB)
    ├── quote/         157 files  (628 KB)
    ├── anecdote/       88 files  (512 KB)
    ├── presentation/   61 files  (624 KB)
    ├── commencement/    7 files  (284 KB)
    ├── meeting/         8 files  (264 KB)
    ├── deposition/      4 files  (164 KB)
    └── book/            1 file   (516 KB)

📦 What's Included

Video Transcripts (254 documents)

89 keynotes (WWDC, Macworld, Special Events)
85 interviews (TV, podcasts, documentaries)
61 presentations (product launches)
8 internal meetings
7 commencements (Stanford 2005, etc.)
4 depositions (court testimonies)

Supplementary Content (246 documents)

157 verbatim quotes (primary sources)
88 anecdotes/stories (about Steve Jobs)
1 book ("Make Something Wonderful" - 244 pages, 46K words)

🧠 Cognitive Enrichment

10,427 cognitive patterns extracted across 406 documents (81%):

Category	Count	%	What It Reveals
Reasoning	2,530	24.3%	How he built arguments
Epistemology	1,824	17.5%	How he formed knowledge
Agency	1,777	17.0%	His decision-making process
Prediction	1,353	13.0%	His vision statements
Values	1,233	11.8%	His core principles
Criticism	647	6.2%	What he rejected
Analogy	538	5.2%	His metaphors & comparisons
Story	525	5.0%	Personal narratives

Book: "Make Something Wonderful"

The richest single source: 549 cognitive patterns in 46K words

130 epistemology markers ("I believe", "I think")
122 reasoning markers ("because", "the reason is")
74 agency markers ("we decided", "I chose")

🚀 Ready to Upload to OpenAI

Method 1: Automated Upload (Recommended)

cd /Users/emanuelprivat/Desktop/DocScraper
source venv/bin/activate

# Upload all 500 files and create assistant
python upload_to_openai.py \
    --input final_corpus/split_files \
    --create-assistant

Method 2: Manual Upload

See example code in previous messages or use the provided scripts.

🎯 Why This Structure is Optimal

Following OpenAI's best practices:

✅ Individual files (not one giant JSONL)

Better citations: "keynote/WWDC_2007.json" vs "row 234"
Better filtering: Query only keynotes, or only from 2007
Easier updates: Replace individual files

✅ Rich metadata per document

Year, event type, era, topics
Cognitive pattern counts
Source URL for attribution

✅ Within all limits

Each file < 512 MB ✓
Each file < 5M tokens ✓
Total files: 500 << 10,000 limit ✓

📝 Document Schema

Each JSON file contains:

{
  "id": "unique_id",
  "title": "Event/Video Title",
  "speaker": "Steve Jobs",
  "year": 2007,
  "event": "Macworld San Francisco",
  "event_type": "keynote",
  "era": "apple_return",
  "topics": ["iPhone", "Innovation"],
  "content": "Steve Jobs' words only",
  "full_transcript": "Complete transcript",
  "cognitive_patterns": {
    "agency": {"count": 15, "examples": ["we decided"]},
    "reasoning": {"count": 8, "examples": ["because"]}
  },
  "source_url": "https://...",
  "word_count": 5000,
  "steve_jobs_percentage": 85.5
}

✅ Ready Checklist

🎉 CORPUS IS COMPLETE

Your Steve Jobs Complete Corpus is ready for OpenAI Assistant upload!

Next step: Run the upload script or manually upload the files to OpenAI.

FilesExpand file tree

FINAL_CORPUS_STATUS.md

Latest commit

History