feat: Unify logging and add request middleware#1442
Open
edwinjosechittilappilly wants to merge 3 commits intorelease-saas-0.1from
Open
feat: Unify logging and add request middleware#1442edwinjosechittilappilly wants to merge 3 commits intorelease-saas-0.1from
edwinjosechittilappilly wants to merge 3 commits intorelease-saas-0.1from
Conversation
github-actions
bot
added
backend
Contributor
There was a problem hiding this comment.
Pull request overview
This PR standardizes structured logging across the API/services/connectors, moves logging configuration to the earliest possible point in startup, and adds an ASGI request middleware for correlation IDs + single-line request summaries.
Changes:
- Replaced
traceback.print_exc()/ ad-hoc error logging withlogger.exception()and consistent context prefixes. - Refactored logging configuration (
structlog+ stdlib bridge) and moved initial configuration intobootstrapto run before other imports. - Added a pure-ASGI
RequestLoggingMiddlewarethat bindsrequest_id/method/pathand emits a single[API] Requestlog with duration + status.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/utils/logging_config.py | Expands structlog setup with shared processors, adds stdlib logging bridge, adds header sanitization helper and filtering processors. |
| src/bootstrap.py | Loads .env and immediately configures logging on import to ensure consistent logger setup early in startup. |
| src/main.py | Removes explicit configure_from_env() call, adds ASGI request logging middleware, and prevents uvicorn from overriding logging config. |
| src/dependencies.py | Updates auth-related log prefixes (notably IBM auth paths). |
| src/utils/opensearch_utils.py | Normalizes OpenSearch security/setup logging prefixes and adjusts shutdown logging. |
| src/utils/acl_utils.py | Replaces print(...) error paths with structured logging. |
| src/services/search_service.py | Adjusts search-related log levels and adds [SEARCH] prefixes for key events. |
| src/services/langflow_file_service.py | Removes logging of headers/payload; adds ingestion start/complete logs. |
| src/services/chat_service.py | Removes verbose header logging; adds structured [CHAT] request/response logs. |
| src/services/auth_service.py | Replaces traceback printing with logger.exception for better tracebacks. |
| src/api/upload.py | Adds structured logging for ingestion upload failures and access-denied cases. |
| src/api/settings.py | Improves onboarding logs (structured fields, warning vs info). |
| src/api/provider_validation.py | Promotes a fallback attempt log to warning with [API] prefix. |
| src/api/knowledge_filter.py | Converts webhook failure logging to logger.exception with context fields. |
| src/api/connectors.py | Converts webhook failure logging to logger.exception with context fields; adjusts error levels. |
| src/api/chat.py | Converts error paths to logger.exception and adds missing error logs in history endpoints. |
| src/api/auth.py | Adds module logger and replaces traceback printing with structured exceptions. |
| src/connectors/service.py | Promotes connector document processing logs to structured [CONNECTOR] info events. |
| src/connectors/sharepoint/oauth.py | Replaces traceback printing with logger.exception across SharePoint OAuth flows. |
| src/connectors/sharepoint/connector.py | Replaces traceback printing with logger.exception; improves detect-url warning structure. |
| src/connectors/onedrive/oauth.py | Replaces traceback printing with logger.exception across OneDrive OAuth flows. |
| src/connectors/onedrive/connector.py | Replaces traceback printing with logger.exception; improves detect-url warning structure. |
| src/connectors/ibm_cos/connector.py | Removes __main__ guard/test code from connector module. |
| src/config/settings.py | Adjusts log level for forced key regeneration to warning. |
| src/config/config_manager.py | Reduces config logging payload volume; adds [CONFIG] prefix. |
| src/agent.py | Replaces traceback printing with logger.exception in agent streaming/non-streaming paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replace ad-hoc prints/tracebacks with structured logger.exception/logger.* calls across the codebase and standardize log messages/levels (agent, api, connectors, services, utils). Introduce a redesigned utils/logging_config.py with structured logging processors, JSON/dev renderers, stdlib logging bridge, header sanitization, and environment-driven configuration. Add a pure-ASGI RequestLoggingMiddleware in main.py to bind request_id and emit per-request structured logs (status, duration) and inject x-request-id. Misc: update log contexts, remove test __main__ in IBM COS connector, and improve a few log levels/messages for clarity and observability.
Configure structured logging immediately after loading .env so module-level loggers are correct on import (bootstrap.py). Move get_logger after configuration and emit a startup info log. Remove redundant configure_from_env() from main and prevent Uvicorn from overriding structlog by passing log_config=None to uvicorn.run. Enhance logging_config: add json_logs flag, choose JSON vs console renderer, route uvicorn loggers through our handler, suppress noisy third‑party libraries, capture Python warnings via logging, and tune logger levels. Lowered Opensearch shutdown noise by changing a warning to a debug message in opensearch_utils.
edwinjosechittilappilly
force-pushed
the
remove-log-points
branch
from
9b18b97 to
6aa532b
Compare
github-actions
bot
added
enhancement
Adjust logging and error handling across several modules: - src/dependencies.py: Remove a development debug log that printed IBM auth headers. - src/main.py: Stop importing configure_from_env (only get_logger kept). Ensure a single x-request-id in responses by removing any upstream header values before injecting the generated correlation ID. - src/utils/acl_utils.py: Replace logger.error calls with logger.exception to include tracebacks when ACL checks or updates fail. - src/utils/logging_config.py: Change filter_health_and_metrics to only suppress routine access-log INFO entries for high-frequency endpoints (errors/warnings still surface). Also use dict_tracebacks when JSON logs are enabled to improve traceback formatting. These changes reduce noisy logs, ensure consistent request ID headers, and provide better diagnostic information for ACL-related errors.
github-actions
bot
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request refactors exception handling and logging across the codebase to improve consistency, clarity, and structure. The main changes include replacing
traceback.print_exc()and unstructured error logs withlogger.exception()calls that include context-specific messages and, where appropriate, structured fields. Additionally, logging is now configured immediately on application startup to ensure all modules use the correct logger setup.Logging and Exception Handling Improvements:
Replaced all instances of
traceback.print_exc()and genericlogger.error()calls withlogger.exception()for better stack trace visibility and context-specific error messages throughout the API and connector modules. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]Updated many log messages to include a clear context prefix (e.g.,
[AUTH],[CHAT],[CONNECTOR],[API],[CONFIG],[LF],[INGEST]) for easier log filtering and diagnosis. [1] [2] [3] [4] [5] [6] [7] [8]Application Startup and Logging Configuration:
src/bootstrap.pyto configure environment variables and structured logging as the very first step in application startup, ensuring all loggers are properly set up before any other imports.Additional Logging Enhancements:
logger.warningfor recoverable issues). [1] [2] [3] [4] [5] [6]Code Cleanup:
These changes collectively improve the maintainability, observability, and reliability of the codebase by standardizing how errors and important events are logged.