fix: replace Defuddle API with local library and remove YouTube support - #24
Merged
Conversation
Browser direct calls to defuddle.md are blocked by CORS policy. Route requests through /api/extract server-side proxy instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Cloudflare bot protection may block requests with default server User-Agent. Forward the client's User-Agent to avoid 403 responses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Log upstream status, headers, and body preview to diagnose Cloudflare 403 responses from Vercel deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cloudflare Bot Fight Mode blocks both browser direct calls and server proxy calls to defuddle.md. Documented findings and remaining options. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
YouTube transcript extraction is unreliable due to datacenter IP blocking by InnerTube API. Remove all YouTube-related code from hooks, components, tests, and spec.yaml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace external defuddle.md API proxy with local defuddle/node library for HTML-to-markdown conversion. This avoids Cloudflare Bot Fight Mode blocking and removes external API dependency. - Add defuddle and linkedom dependencies - Create lib/extract.ts with server-side extraction logic - Update API route to call extractContent() directly - Switch hooks/use-extract.ts from frontmatter to JSON responses - Update all test mocks to use JSON response format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documented defuddle library direct usage as the solution. YouTube support removed due to datacenter IP blocking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
toy-crane
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
defuddle.md)가 Cloudflare Bot Fight Mode로 브라우저/서버 모두 차단되는 문제 해결defuddlenpm 패키지를 Next.js API Route에서 직접 사용하여 외부 API 의존성 제거Changes
lib/extract.ts생성:defuddle/node+linkedom기반 서버사이드 추출app/api/extract/route.ts: Defuddle API 프록시 →extractContent()직접 호출hooks/use-extract.ts: frontmatter 파싱 → JSON 응답 처리로 전환Test plan
bun run test— 테스트 통과 (기존 실패 2개 제외)bun run build— 빌드 성공bun run dev→ 웹페이지 URL 추출 동작 확인🤖 Generated with Claude Code