1- # 🕷️ Deepcrawl SDK
1+ # Deepcrawl SDK
22
3- ** World-class TypeScript SDK for the Deepcrawl API** - Professional web scraping and crawling with enterprise-grade error handling.
3+ TypeScript SDK for the Deepcrawl API - Web scraping and crawling with comprehensive error handling.
44
55[ ![ npm version] ( https://badge.fury.io/js/deepcrawl.svg )] ( https://www.npmjs.com/package/deepcrawl )
66[ ![ TypeScript] ( https://img.shields.io/badge/TypeScript-Ready-blue.svg )] ( https://www.typescriptlang.org )
77[ ![ MIT License] ( https://img.shields.io/badge/License-MIT-green.svg )] ( https://opensource.org/licenses/MIT )
88
99## ⚡ ** Why Deepcrawl SDK?**
1010
11- - 🏗️ ** oRPC-Powered** : Built on industry-leading RPC framework for optimal performance
12- - 🔒 ** Type-Safe** : End-to-end TypeScript with intelligent error handling
11+ - 🏗️ ** oRPC-Powered** : Built on oRPC framework for type-safe RPC
12+ - 🔒 ** Type-Safe** : End-to-end TypeScript with error handling
1313- 🖥️ ** Server-Side Only** : Designed for Node.js, Cloudflare Workers, and Next.js Server Actions
1414- 🪶 ** Lightweight** : Minimal bundle size with tree-shaking support
15- - 🛡️ ** World-Class Error Handling** : Comprehensive, typed errors with actionable context
16- - 🔄 ** Smart Retry Logic** : Built-in exponential backoff for transient failures
17- - ⚡ ** Connection Pooling** : Automatic HTTP connection reuse for optimal performance (Node.js)
15+ - 🛡️ ** Error Handling** : Comprehensive, typed errors with context
16+ - 🔄 ** Retry Logic** : Built-in exponential backoff for transient failures
17+ - ⚡ ** Connection Pooling** : Automatic HTTP connection reuse (Node.js)
1818
1919## 📦 ** Installation**
2020
@@ -54,7 +54,6 @@ const result = await deepcrawl.readUrl('https://example.com', {
5454 markdown: true , // Convert to markdown
5555 cleanedHtml: true , // Get sanitized HTML
5656 rawHtml: true , // Get original HTML
57- screenshot: false , // Capture screenshot
5857 metricsOptions: { // Performance tracking
5958 enable: true
6059 }
@@ -67,7 +66,6 @@ interface ReadUrlResponse {
6766 markdown? : string ;
6867 cleanedHtml? : string ;
6968 rawHtml? : string ;
70- screenshot? : string ;
7169 metadata? : {
7270 title? : string ;
7371 description? : string ;
@@ -141,38 +139,53 @@ interface ExtractLinksResponse {
141139```
142140
143141### ** getManyLogs(options?)**
144- Retrieve activity logs with type-safe filtering.
142+ Retrieve activity logs with paginated results and filtering by path, success status, and date range. Returns logs with full type safety through discriminated unions based on the endpoint path .
145143
146144``` typescript
147- const = await deepcrawl .getManyLogs ({
148- userId: ' user123' // Filter by user
149- url: ' https://example.com' // Filter by URL
150- operation: ' getMarkdown' // Filter by operation
151- status: ' success' // Filter by status
152- limit: 50 , // Pagination limit
153- offset: 0 // Pagination offset
145+ const = await deepcrawl .getManyLogs ({
146+ limit: 50 , // Max results (default: 20, max: 100)
147+ offset: 0 , // Skip first N results (default: 0)
148+ path: ' read-getMarkdown' // Filter by endpoint path (optional)
149+ success: true , // Filter by success status (optional)
150+ startDate: ' 2025-01-01T00:00:00Z' // Filter from date (ISO 8601) (optional)
151+ endDate: ' 2025-12-31T23:59:59Z' // Filter to date (ISO 8601) (optional)
152+ orderBy: ' requestTimestamp' // Sort column (default: 'requestTimestamp')
153+ orderDir: ' desc' // Sort direction: 'asc' | 'desc' (default: 'desc')
154154});
155155
156- // Response type
157- interface ActivityLog {
158- id: string ;
159- userId: string ;
160- sessionId: string ;
161- url: string ;
162- operation: ' readUrl' | ' getMarkdown' | ' extractLinks'
163- status: ' success' | ' error'
164- errorType? : ' auth' | ' network' | ' read' | ' links' | ' unknown'
165- errorMessage? : string ;
166- responseTime? : number ;
167- createdAt: string ;
156+ // Response type with discriminated unions
157+ interface GetManyLogsResponse {
158+ logs: ActivityLogEntry []; // Array of log entries with discriminated unions
159+ meta: {
160+ limit: number ; // Effective limit applied
161+ offset: number ; // Effective offset applied
162+ hasMore: boolean ; // More logs available?
163+ nextOffset: number | null ; // Next page offset (null if no more data)
164+ orderBy: string ; // Column used for sorting
165+ orderDir: ' asc' | ' desc' // Sort direction applied
166+ startDate? : string ; // Normalized start date boundary
167+ endDate? : string ; // Normalized end date boundary
168+ };
168169}
170+
171+ // Each log entry uses discriminated union based on 'path' field:
172+ // - 'read-getMarkdown': response is string
173+ // - 'read-readUrl': response is ReadSuccessResponse | ReadErrorResponse
174+ // - 'links-getLinks': response is LinksSuccessResponse | LinksErrorResponse
175+ // - 'links-extractLinks': response is LinksSuccessResponse | LinksErrorResponse
169176```
170177
171- ### ** getOneLog(id)**
172- Get a single activity log by ID.
178+ ### ** getOneLog(options)**
179+
180+ Get a single activity log entry by ID with full type safety through discriminated unions.
173181
174182``` typescript
175- const = await deepcrawl .getOneLog (' log_abc123'
183+ const = await deepcrawl .getOneLog ({
184+ id: ' request-id-123' // Request ID (required)
185+ });
186+
187+ // Response is ActivityLogEntry with discriminated union
188+ // TypeScript automatically narrows types based on log.path
176189```
177190
178191## 🌟 ** Real-World Usage Examples**
@@ -481,7 +494,7 @@ export function ActivityLogsClient() {
481494
482495## 🛡️ ** Error Handling Patterns**
483496
484- Our SDK provides multiple patterns for different coding styles:
497+ The SDK provides multiple patterns for different coding styles:
485498
486499### ** Traditional Try/Catch**
487500``` typescript
@@ -573,7 +586,7 @@ const deepcrawl = new DeepcrawlApp({
573586
574587### ** Connection Pooling (Node.js Only)**
575588
576- The SDK automatically uses HTTP connection pooling in Node.js environments for optimal performance :
589+ The SDK automatically uses HTTP connection pooling in Node.js environments:
577590
578591``` typescript
579592// Automatic configuration (no action needed)
@@ -587,7 +600,7 @@ The SDK automatically uses HTTP connection pooling in Node.js environments for o
587600```
588601
589602Benefits:
590- - ⚡ ** ~ 40% faster ** for concurrent requests
603+ - ⚡ ** Faster ** for concurrent requests
591604- 🔄 ** Connection reuse** reduces handshake overhead
592605- 🎯 ** Auto-cleanup** of idle connections
593606- 📊 ** Optimized for batch operations**
0 commit comments