Skip to content

Commit 657c8e7

Browse files
Merge pull request #10 from alichherawalla/feat/add-support-for-attachments
Feat/add support for attachments
2 parents e3e7204 + 9d1b8e6 commit 657c8e7

33 files changed

Lines changed: 2216 additions & 292 deletions

README.md

Lines changed: 70 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ OffgridMobile is a React Native application that brings large language models, v
2424
| Vision AI | llama.rn multimodal | llama.rn multimodal |
2525
| Image Generation | local-dream (MNN/QNN) | Core ML (ANE + CPU) |
2626
| Voice Transcription | whisper.cpp | whisper.cpp |
27+
| PDF Extraction | PdfRenderer (Kotlin) | PDFKit (Swift) |
28+
| Document Viewer | Intent.ACTION_VIEW | QuickLook |
2729
| Background Downloads | Native DownloadManager | RNFS / URLSession |
2830

2931
---
@@ -35,6 +37,9 @@ OffgridMobile is a React Native application that brings large language models, v
3537
- **Image Generation** - On-device Stable Diffusion (CPU/NPU), real-time preview, background generation
3638
- **AI Prompt Enhancement** - Use text LLM to expand simple prompts into detailed descriptions for better image quality
3739
- **Voice Transcription** - On-device Whisper for speech-to-text, multiple model sizes
40+
- **Document Attachments** - Attach text files, code, CSV, JSON, and PDFs to chat messages
41+
- **PDF Extraction** - Native on-device PDF text extraction (Android + iOS)
42+
- **Message Queue** - Send multiple messages during active generation, auto-processed sequentially
3843
- **GPU Acceleration** - Optional OpenCL GPU offloading for text models
3944
- **Auto/Manual Image Generation** - Automatic intent detection or manual toggle for image generation
4045
- **Dark Mode** - Full light/dark theme with dynamic color palettes and elevation system
@@ -180,6 +185,53 @@ On-device speech recognition using whisper.cpp via whisper.rn native bindings:
180185
- Audio temporarily buffered in native code, cleared after transcription
181186
- Model selection in settings (Tiny: fastest, Base: balanced, Small: most accurate)
182187

188+
### Document Attachments
189+
190+
Attach documents to chat messages for context-aware conversations. Documents are parsed and included in the LLM context window alongside the user's message.
191+
192+
**Supported Formats:**
193+
- **Text files** - `.txt`, `.md`, `.log`
194+
- **Code files** - `.py`, `.js`, `.ts`, `.jsx`, `.tsx`, `.java`, `.c`, `.cpp`, `.h`, `.swift`, `.kt`, `.go`, `.rs`, `.rb`, `.php`, `.sql`, `.sh`
195+
- **Data files** - `.csv`, `.json`, `.xml`, `.yaml`, `.yml`, `.toml`, `.ini`, `.cfg`, `.conf`, `.html`
196+
- **PDF documents** - Native text extraction via platform-specific modules
197+
198+
**Features:**
199+
- **File picker integration** - Uses `@react-native-documents/picker` for native file selection
200+
- **PDF text extraction** - Native modules extract text from PDFs on both platforms
201+
- **Persistent storage** - Attached files are copied to persistent app storage so they survive temp file cleanup
202+
- **Tappable document badges** - Tap any attached document in chat to open it with the system viewer (QuickLook on iOS, Intent viewer on Android)
203+
- **Paste-as-attachment** - Large pasted text can be attached as a document
204+
- **File size limit** - 5MB maximum, with text content truncated to 50K characters for context window management
205+
206+
**PDF Extraction Implementation:**
207+
- Android: `PdfExtractorModule` (Kotlin) uses Android's native `PdfRenderer` API
208+
- iOS: `PDFExtractorModule` (Swift) uses Apple's `PDFKit` framework
209+
- Both extract page-by-page text content with page separators
210+
- Graceful fallback when PDF extraction is unavailable
211+
212+
**Document Viewer:**
213+
- Uses `@react-native-documents/viewer` for cross-platform document opening
214+
- iOS: Opens in QuickLook preview
215+
- Android: Opens with `Intent.ACTION_VIEW` using the appropriate system app
216+
217+
### Message Queue
218+
219+
Send messages while the LLM is still generating a response. Messages are queued and processed automatically after the current generation completes.
220+
221+
**Features:**
222+
- **Non-blocking input** - Send button stays active during generation, alongside a visible stop button
223+
- **Queue indicator** - Shows count and preview of queued messages in the toolbar
224+
- **Clear queue** - Tap the "x" on the queue indicator to discard all queued messages
225+
- **Aggregated processing** - When generation completes, all queued messages are combined into a single prompt and processed together
226+
- **Image bypass** - Image generation requests skip the queue and process immediately via the separate image generation service
227+
- **Stop + Send side-by-side** - Both stop and send buttons visible in the input area during active generation
228+
229+
**Implementation:**
230+
- Queue lives in `generationService.ts` as transient state (not persisted across app restarts)
231+
- User messages are added to chat only when the queue processor picks them up, preserving correct chronology
232+
- ChatScreen registers a `queueProcessor` callback; when generation resets and the queue has items, the service calls this callback
233+
- Multiple queued messages are aggregated: texts joined with `\n\n`, attachments combined
234+
183235
### Image Generation Modes
184236

185237
**Auto Detection:**
@@ -837,6 +889,8 @@ Prevents OOM crashes by blocking loads that would exceed safe RAM limits.
837889
- **React Navigation 7.x** - Native navigation
838890
- **React Native Reanimated 4.x** - Performant native-thread animations
839891
- **React Native Haptic Feedback** - Haptic responses on interactions
892+
- **@react-native-documents/picker** - Native document picker for file attachments
893+
- **@react-native-documents/viewer** - Native document viewer (QuickLook / Intent.ACTION_VIEW)
840894

841895
### Native Modules
842896

@@ -864,6 +918,12 @@ Prevents OOM crashes by blocking loads that would exceed safe RAM limits.
864918
- Safety checker disabled for reduced latency
865919
- Supports palettized (6-bit) and full-precision (fp16) Core ML models
866920

921+
**PdfExtractorModule (Android + iOS):**
922+
- Android: Kotlin module using `android.graphics.pdf.PdfRenderer` for page-by-page text extraction
923+
- iOS: Swift module using Apple's `PDFKit` (`PDFDocument`, `PDFPage`) for text extraction
924+
- Both expose `extractText(filePath)` returning concatenated page text with separators
925+
- Graceful error handling when PDF is corrupted or password-protected
926+
867927
**DownloadManager (Android only):**
868928
- Native Android DownloadManager wrapper
869929
- Background download support
@@ -1045,7 +1105,8 @@ OffgridMobile/
10451105
│ │ ├── localDreamGenerator.ts # local-dream bridge
10461106
│ │ ├── imageGenerator.ts # Image generation utilities
10471107
│ │ ├── hardware.ts # Device info and memory
1048-
│ │ ├── documentService.ts # Document text extraction
1108+
│ │ ├── documentService.ts # Document attachment processing
1109+
│ │ ├── pdfExtractor.ts # Native PDF text extraction bridge
10491110
│ │ ├── authService.ts # Passphrase authentication
10501111
│ │ ├── voiceService.ts # Voice recording
10511112
│ │ ├── whisperService.ts # Whisper transcription
@@ -1079,15 +1140,20 @@ OffgridMobile/
10791140
│ │ ├── DownloadManagerModule.kt
10801141
│ │ ├── DownloadManagerPackage.kt
10811142
│ │ └── DownloadCompleteBroadcastReceiver.kt
1082-
│ └── localdream/ # local-dream native module
1083-
│ ├── LocalDreamModule.kt
1084-
│ └── LocalDreamPackage.kt
1143+
│ ├── localdream/ # local-dream native module
1144+
│ │ ├── LocalDreamModule.kt
1145+
│ │ └── LocalDreamPackage.kt
1146+
│ └── pdf/ # PDF text extraction
1147+
│ ├── PdfExtractorModule.kt
1148+
│ └── PdfExtractorPackage.kt
10851149
├── ios/ # iOS native code
10861150
│ └── OffgridMobile/
10871151
│ ├── AppDelegate.swift # Application delegate
10881152
│ ├── CoreMLDiffusion/ # Core ML image generation
10891153
│ │ ├── CoreMLDiffusionModule.swift
10901154
│ │ └── CoreMLDiffusionModule.m
1155+
│ ├── PDFExtractor/ # PDF text extraction
1156+
│ │ └── PDFExtractorModule.swift
10911157
│ └── Download/ # iOS download manager
10921158
│ ├── DownloadManagerModule.swift
10931159
│ └── DownloadManagerModule.m

__tests__/integration/generation/imageGenerationFlow.test.ts

Lines changed: 209 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,21 +10,25 @@ import { useAppStore } from '../../../src/stores/appStore';
1010
import { imageGenerationService } from '../../../src/services/imageGenerationService';
1111
import { localDreamGeneratorService } from '../../../src/services/localDreamGenerator';
1212
import { activeModelService } from '../../../src/services/activeModelService';
13+
import { llmService } from '../../../src/services/llm';
1314
import {
1415
resetStores,
1516
flushPromises,
1617
getAppState,
1718
getChatState,
1819
setupWithConversation,
1920
} from '../../utils/testHelpers';
20-
import { createONNXImageModel, createGeneratedImage } from '../../utils/factories';
21+
import { createONNXImageModel, createGeneratedImage, createMessage } from '../../utils/factories';
22+
import { Message } from '../../../src/types';
2123

2224
// Mock the services
2325
jest.mock('../../../src/services/localDreamGenerator');
2426
jest.mock('../../../src/services/activeModelService');
27+
jest.mock('../../../src/services/llm');
2528

2629
const mockLocalDreamService = localDreamGeneratorService as jest.Mocked<typeof localDreamGeneratorService>;
2730
const mockActiveModelService = activeModelService as jest.Mocked<typeof activeModelService>;
31+
const mockLlmService = llmService as jest.Mocked<typeof llmService>;
2832

2933
describe('Image Generation Flow Integration', () => {
3034
beforeEach(async () => {
@@ -55,6 +59,11 @@ describe('Image Generation Flow Integration', () => {
5559
});
5660
mockActiveModelService.loadImageModel.mockResolvedValue();
5761

62+
// Default LLM service mocks (for prompt enhancement)
63+
mockLlmService.isModelLoaded.mockReturnValue(false);
64+
mockLlmService.isCurrentlyGenerating.mockReturnValue(false);
65+
mockLlmService.stopGeneration.mockResolvedValue();
66+
5867
// Reset imageGenerationService state by canceling any in-progress generation
5968
await imageGenerationService.cancelGeneration().catch(() => {});
6069
});
@@ -512,4 +521,203 @@ describe('Image Generation Flow Integration', () => {
512521
expect(message?.generationMeta?.resolution).toBe('512x512');
513522
});
514523
});
524+
525+
describe('Prompt Enhancement with Conversation Context', () => {
526+
const setupEnhancement = () => {
527+
const imageModel = setupImageModelState();
528+
mockActiveModelService.getActiveModels.mockReturnValue({
529+
text: { model: null, isLoaded: false, isLoading: false },
530+
image: { model: imageModel, isLoaded: true, isLoading: false },
531+
});
532+
533+
// Enable enhancement and set up LLM as available
534+
useAppStore.setState({
535+
settings: {
536+
...useAppStore.getState().settings,
537+
enhanceImagePrompts: true,
538+
},
539+
});
540+
mockLlmService.isModelLoaded.mockReturnValue(true);
541+
mockLlmService.isCurrentlyGenerating.mockReturnValue(false);
542+
mockLlmService.generateResponse.mockResolvedValue('A beautifully enhanced prompt');
543+
544+
return imageModel;
545+
};
546+
547+
it('should pass conversation history to enhancement when conversationId provided', async () => {
548+
setupEnhancement();
549+
550+
// Set up a conversation with prior messages
551+
const messages: Message[] = [
552+
createMessage({ role: 'user', content: 'Draw me a cat' }),
553+
createMessage({ role: 'assistant', content: 'Here is a cat image' }),
554+
createMessage({ role: 'user', content: 'Make it darker' }),
555+
];
556+
const conversationId = setupWithConversation({ messages });
557+
558+
await imageGenerationService.generateImage({
559+
prompt: 'Make it darker',
560+
conversationId,
561+
});
562+
563+
// Verify generateResponse was called with conversation context
564+
expect(mockLlmService.generateResponse).toHaveBeenCalled();
565+
const callArgs = mockLlmService.generateResponse.mock.calls[0];
566+
const enhancementMessages = callArgs[0] as Message[];
567+
568+
// Should have: system + context messages + user enhance prompt
569+
// system (1) + conversation messages (3) + user enhance (1) = 5
570+
expect(enhancementMessages.length).toBe(5);
571+
expect(enhancementMessages[0].role).toBe('system');
572+
expect(enhancementMessages[0].content).toContain('conversation history');
573+
expect(enhancementMessages[1].content).toBe('Draw me a cat');
574+
expect(enhancementMessages[2].content).toBe('Here is a cat image');
575+
expect(enhancementMessages[3].content).toBe('Make it darker');
576+
expect(enhancementMessages[4].role).toBe('user');
577+
expect(enhancementMessages[4].content).toBe('Make it darker');
578+
});
579+
580+
it('should not include conversation context when no conversationId', async () => {
581+
setupEnhancement();
582+
583+
await imageGenerationService.generateImage({
584+
prompt: 'A sunset',
585+
});
586+
587+
expect(mockLlmService.generateResponse).toHaveBeenCalled();
588+
const callArgs = mockLlmService.generateResponse.mock.calls[0];
589+
const enhancementMessages = callArgs[0] as Message[];
590+
591+
// Should have: system + user enhance prompt only (no context)
592+
expect(enhancementMessages.length).toBe(2);
593+
expect(enhancementMessages[0].role).toBe('system');
594+
expect(enhancementMessages[0].content).not.toContain('conversation history');
595+
expect(enhancementMessages[1].role).toBe('user');
596+
expect(enhancementMessages[1].content).toBe('A sunset');
597+
});
598+
599+
it('should truncate long messages in conversation context', async () => {
600+
setupEnhancement();
601+
602+
const longContent = 'x'.repeat(1000);
603+
const messages: Message[] = [
604+
createMessage({ role: 'user', content: longContent }),
605+
];
606+
const conversationId = setupWithConversation({ messages });
607+
608+
await imageGenerationService.generateImage({
609+
prompt: 'Enhance this',
610+
conversationId,
611+
});
612+
613+
const callArgs = mockLlmService.generateResponse.mock.calls[0];
614+
const enhancementMessages = callArgs[0] as Message[];
615+
616+
// The context message should be truncated to 500 chars
617+
const contextMsg = enhancementMessages.find(m => m.id.startsWith('ctx-'));
618+
expect(contextMsg).toBeDefined();
619+
expect(contextMsg!.content.length).toBe(500);
620+
});
621+
622+
it('should limit conversation context to last 10 messages', async () => {
623+
setupEnhancement();
624+
625+
// Create 15 messages
626+
const messages: Message[] = [];
627+
for (let i = 0; i < 15; i++) {
628+
messages.push(createMessage({
629+
role: i % 2 === 0 ? 'user' : 'assistant',
630+
content: `Message ${i + 1}`,
631+
}));
632+
}
633+
const conversationId = setupWithConversation({ messages });
634+
635+
await imageGenerationService.generateImage({
636+
prompt: 'Generate image',
637+
conversationId,
638+
});
639+
640+
const callArgs = mockLlmService.generateResponse.mock.calls[0];
641+
const enhancementMessages = callArgs[0] as Message[];
642+
643+
// system (1) + last 10 context messages + user enhance (1) = 12
644+
expect(enhancementMessages.length).toBe(12);
645+
// First context message should be message 6 (index 5), not message 1
646+
const firstContextMsg = enhancementMessages[1];
647+
expect(firstContextMsg.content).toBe('Message 6');
648+
});
649+
650+
it('should skip system messages from conversation context', async () => {
651+
setupEnhancement();
652+
653+
const messages: Message[] = [
654+
createMessage({ role: 'user', content: 'Hello' }),
655+
createMessage({ role: 'system', content: 'Model loaded successfully' }),
656+
createMessage({ role: 'assistant', content: 'Hi there' }),
657+
];
658+
const conversationId = setupWithConversation({ messages });
659+
660+
await imageGenerationService.generateImage({
661+
prompt: 'Draw something',
662+
conversationId,
663+
});
664+
665+
const callArgs = mockLlmService.generateResponse.mock.calls[0];
666+
const enhancementMessages = callArgs[0] as Message[];
667+
668+
// system (1) + 2 context (user + assistant, system skipped) + user enhance (1) = 4
669+
expect(enhancementMessages.length).toBe(4);
670+
const contextMessages = enhancementMessages.filter(m => m.id.startsWith('ctx-'));
671+
expect(contextMessages).toHaveLength(2);
672+
expect(contextMessages.every(m => m.role !== 'system')).toBe(true);
673+
});
674+
675+
it('should use original prompt when enhancement is disabled', async () => {
676+
setupImageModelState();
677+
mockActiveModelService.getActiveModels.mockReturnValue({
678+
text: { model: null, isLoaded: false, isLoading: false },
679+
image: { model: setupImageModelState(), isLoaded: true, isLoading: false },
680+
});
681+
682+
// Enhancement disabled (default)
683+
useAppStore.setState({
684+
settings: {
685+
...useAppStore.getState().settings,
686+
enhanceImagePrompts: false,
687+
},
688+
});
689+
690+
const messages: Message[] = [
691+
createMessage({ role: 'user', content: 'Draw a cat' }),
692+
];
693+
const conversationId = setupWithConversation({ messages });
694+
695+
await imageGenerationService.generateImage({
696+
prompt: 'Make it blue',
697+
conversationId,
698+
});
699+
700+
// LLM should not be called for enhancement
701+
expect(mockLlmService.generateResponse).not.toHaveBeenCalled();
702+
});
703+
704+
it('should handle empty conversation gracefully', async () => {
705+
setupEnhancement();
706+
707+
const conversationId = setupWithConversation({ messages: [] });
708+
709+
await imageGenerationService.generateImage({
710+
prompt: 'A landscape',
711+
conversationId,
712+
});
713+
714+
const callArgs = mockLlmService.generateResponse.mock.calls[0];
715+
const enhancementMessages = callArgs[0] as Message[];
716+
717+
// system + user enhance only (no context from empty conversation)
718+
expect(enhancementMessages.length).toBe(2);
719+
expect(enhancementMessages[0].role).toBe('system');
720+
expect(enhancementMessages[0].content).not.toContain('conversation history');
721+
});
722+
});
515723
});

__tests__/integration/stores/chatStoreIntegration.test.ts

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -345,21 +345,22 @@ describe('ChatStore Streaming Integration', () => {
345345
expect(conversation?.messages).toHaveLength(0);
346346
});
347347

348-
it('should create conversation and clear streaming state together', () => {
348+
it('should create conversation and preserve streaming state', () => {
349349
const conversationId = setupWithConversation();
350350
const chatStore = useChatStore.getState();
351351

352352
// Start streaming
353353
chatStore.startStreaming(conversationId);
354354
chatStore.appendToStreamingMessage('Content');
355355

356-
// Create new conversation (should clear streaming)
356+
// Create new conversation (streaming state preserved — scoped by streamingForConversationId)
357357
const newConvId = chatStore.createConversation('model-id', 'New Conv');
358358

359359
const state = getChatState();
360360
expect(state.activeConversationId).toBe(newConvId);
361-
expect(state.streamingMessage).toBe('');
362-
expect(state.isStreaming).toBe(false);
361+
// Streaming state is preserved — UI uses streamingForConversationId to scope display
362+
expect(state.streamingMessage).toBe('Content');
363+
expect(state.isStreaming).toBe(true);
363364
});
364365
});
365366
});

0 commit comments

Comments
 (0)