hotfix(chat): Chat answer streaming and smooth markdown rendering#1515
Merged
Conversation
- Introduced a new test file to validate the LLM bundle construction for streaming flows in chat. - Implemented tests to ensure that both DB-backed and global models enable streaming correctly. - Utilized mocking to isolate dependencies and verify the expected behavior of the LLM constructor.
- Updated the LLM bundle construction to include a streaming option for both DB-backed and global models. - Modified the `litellm_kwargs` to set the streaming parameter to True, enhancing the functionality for chat streaming flows.
- Updated the MarkdownTextPrimitive component to enable smooth rendering by default. - Adjusted the props to streamline the rendering process for improved user experience.
|
@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
AnishSarkar22
changed the title
📝 WalkthroughWalkthrough
ChangesBackend: Force streaming in LLM bundle
Frontend: Enable auto-scroll and smooth markdown rendering
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
surfsense_backend/tests/unit/tasks/chat/streaming/test_llm_bundle.py (1)
75-78: ⚡ Quick winStrengthen forced-streaming override coverage.
Line 75 and Line 132 stub kwargs without a
streamingfield, so these tests don’t prove precedence when upstream providesstreaming=False. Seed that value and keep expectedstreaming=Trueto lock the override contract.Proposed test hardening
monkeypatch.setattr( llm_bundle, "to_litellm", lambda _conn, _model_id: ( "openai/gpt-4o-mini", - {"api_key": "sk-test", "temperature": 0.1}, + {"api_key": "sk-test", "temperature": 0.1, "streaming": False}, ), ) @@ monkeypatch.setattr( llm_bundle, "to_litellm", lambda _conn, _model_id: ( "anthropic/claude-sonnet-4-5", - {"api_key": "sk-ant-test", "temperature": 0.2}, + {"api_key": "sk-ant-test", "temperature": 0.2, "streaming": False}, ), )Also applies to: 132-135
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@surfsense_backend/tests/unit/tasks/chat/streaming/test_llm_bundle.py` around lines 75 - 78, The test stubs at line 75 and line 132 are missing a streaming field in their kwargs dictionaries, which means they don't actually verify the override behavior when upstream explicitly provides streaming=False. Add streaming: False to the kwargs dictionary returned by the lambda functions at both locations, while keeping the expected streaming=True assertion in the test to confirm the override mechanism properly handles and overrides explicit False values from upstream.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@surfsense_backend/tests/unit/tasks/chat/streaming/test_llm_bundle.py`:
- Around line 75-78: The test stubs at line 75 and line 132 are missing a
streaming field in their kwargs dictionaries, which means they don't actually
verify the override behavior when upstream explicitly provides streaming=False.
Add streaming: False to the kwargs dictionary returned by the lambda functions
at both locations, while keeping the expected streaming=True assertion in the
test to confirm the override mechanism properly handles and overrides explicit
False values from upstream.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 018339dc-ff0f-4236-ab23-065a88eeab32
📒 Files selected for processing (4)
surfsense_backend/app/tasks/chat/streaming/flows/shared/llm_bundle.pysurfsense_backend/tests/unit/tasks/chat/streaming/test_llm_bundle.pysurfsense_web/components/assistant-ui/chat-viewport.tsxsurfsense_web/components/assistant-ui/markdown-text.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
streaming=Truefor both DB-backed and global model paths.MarkdownTextPrimitiveso streamed answers reveal progressively instead of appearing in rigid chunks.Motivation and Context
FIX #
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This hotfix resolves chat answer streaming issues by enabling token streaming in the LLM bundle construction for both database-backed and global model configurations. The backend now explicitly passes
streaming=Truewhen creatingSanitizedChatLiteLLMinstances, fixing the default non-streaming behavior. On the frontend, smooth markdown rendering is restored by re-enabling thesmoothproperty inMarkdownTextPrimitiveand restoring auto-scroll behavior in the chat viewport, ensuring streamed responses appear progressively rather than in rigid chunks. The changes include comprehensive unit tests to verify streaming is properly configured for both model loading paths.⏱️ Estimated Review Time: 5-15 minutes
💡 Review Order Suggestion
surfsense_backend/tests/unit/tasks/chat/streaming/test_llm_bundle.pysurfsense_backend/app/tasks/chat/streaming/flows/shared/llm_bundle.pysurfsense_web/components/assistant-ui/markdown-text.tsxsurfsense_web/components/assistant-ui/chat-viewport.tsxSummary by CodeRabbit
New Features
Bug Fixes
Tests