fix inconsistencies in logging, error handling, correlation id impl

Author: irahardianto

.serena/memories/2026-01-05-bug-fixes-report.md

@@ -0,0 +1,32 @@
+# Bug Fixes & Refactoring Report (Jan 5, 2026)
+
+## Summary
+Executed the plan `docs/plans/2026-01-05-bug-fixes-inconsistencies-2.md` to standardize logging, refactor worker logic, and fix MCP error handling.
+
+## Implemented Changes
+
+### 1. Infrastructure - Context-Aware Logger
+- **New Package:** `apps/backend/internal/logger`
+- **Component:** `ContextHandler` (Decorator for `slog.Handler`)
+- **Functionality:** Automatically extracts `correlation_id` from context (via `middleware.CorrelationKey`) and injects it into every log record.
+- **Integration:** Wired into `apps/backend/main.go` as the default logger.
+
+### 2. Observability - Adapter Logging
+- **Gemini Embedder:** Added `slog.DebugContext` (input stats) and `slog.ErrorContext` (failures) to `Embed`.
+- **Weaviate Store:** Added structured logging to `StoreChunk` and `Search` methods.
+- **Outcome:** Full traceability of ingestion and search operations with correlation IDs.
+
+### 3. Refactor - Link Discovery Pure Function
+- **New File:** `apps/backend/internal/worker/link_discovery.go`
+- **Function:** `DiscoverLinks` (Pure function)
+- **Logic:** Extracted normalization, host matching, and exclusion logic from `ResultConsumer`.
+- **Benefit:** Improved testability (unit tests added) and separation of concerns.
+
+### 4. MCP Error Handling
+- **Fix:** `qurio_search` (and others) now return standard JSON-RPC Error objects (`code`, `message`) instead of embedded success results with `IsError: true`.
+- **Standard:** Compliant with JSON-RPC 2.0.
+
+## Verification
+- **Unit Tests:** `handler_test.go` (Logger), `link_discovery_test.go` (Worker).
+- **Integration Tests:** `go test ./apps/backend/features/mcp/...` passed.
+- **Build:** `go build ./apps/backend/...` successful.

.serena/memories/core_foundation_state.md

@@ -21,6 +21,7 @@
 
 ## Code Quality & Testing
 - **Backend Coverage:** High (>90%) for core adapters (`gemini`, `weaviate`) and repositories (`settings`, `source`).
+- **Observability:** Context-aware JSON logging standardized across Backend and Adapters.
 - **Frontend Quality:** Full store and component testing coverage for `jobs`, `stats`, `sources`, and `settings`.
 - **Standards:** STRICT adherence to `technical-constitution` (Dependency Injection, Interface-based design).
 

.serena/memories/implementation_details.md

@@ -1,8 +1,3 @@
-Plan updated with clean formatting and explicit target descriptions.
-`qurio_list_pages` uses `source_id`.
-`qurio_read_page` uses `url`.
-Full argument guide included for `qurio_search`.
-
-## Ingestion Worker
-- `handle_file_task` now returns `list[dict]` (standardized with `handle_web_task`) to simplify `main.py` logic.
-- Metadata (Title, Author, PageCount) is extracted via Docling v2 and passed to backend.
+- **Logging:** Implemented `apps/backend/internal/logger` package with `ContextHandler` to automatically propagate `correlation_id` from context to JSON logs.
+- **Worker Logic:** Link discovery logic extracted to pure function `DiscoverLinks` in `apps/backend/internal/worker/link_discovery.go` to separate I/O from business logic.
+- **MCP Errors:** `qurio_search` now returns standard JSON-RPC errors (code -32603) for internal failures instead of embedded text errors.

.serena/memories/mcp_tools_spec.md

@@ -51,3 +51,9 @@ Qurio exposes its knowledge base via the Model Context Protocol (MCP).
 
 ## Context Propagation
 All tools propagate `correlationId` from the request to internal services for traceability.
+
+## Error Handling
+Tools return standard JSON-RPC 2.0 errors for internal failures (e.g., database connection issues, search failures).
+- **Code:** -32603 (Internal Error)
+- **Message:** Human-readable error description.
+- **Data:** (Optional) Additional context.

apps/backend/features/mcp/handler.go

@@ -277,14 +277,8 @@ read_page(url="https://docs.stripe.com/webhooks/signatures")`,
 			results, err := h.retriever.Search(ctx, args.Query, opts)
 			if err != nil {
 				slog.Error("search failed", "error", err)
-				return &JSONRPCResponse{
-					JSONRPC: "2.0",
-					ID:      req.ID,
-					Result: ToolResult{
-						Content: []ToolContent{{Type: "text", Text: "Error: " + err.Error()}},
-						IsError: true,
-					},
-				}
+				resp := makeErrorResponse(req.ID, ErrInternal, "Search failed: "+err.Error())
+				return &resp
 			}
 
 			var textResult string

apps/backend/internal/adapter/gemini/embedder.go

@@ -2,6 +2,8 @@ package gemini
 
 import (
 	"context"
+	"log/slog"
+
 	"github.com/google/generative-ai-go/genai"
 	"google.golang.org/api/option"
 )
@@ -20,9 +22,11 @@ func NewEmbedder(ctx context.Context, apiKey string) (*Embedder, error) {
 }
 
 func (e *Embedder) Embed(ctx context.Context, text string) ([]float32, error) {
+	slog.DebugContext(ctx, "embedding content", "model", e.model, "length", len(text))
 	em := e.client.EmbeddingModel(e.model)
 	res, err := em.EmbedContent(ctx, genai.Text(text))
 	if err != nil {
+		slog.ErrorContext(ctx, "embedding failed", "error", err)
 		return nil, err
 	}
 	if res.Embedding != nil {

apps/backend/internal/adapter/weaviate/store.go

@@ -3,6 +3,7 @@ package weaviate
 import (
 	"context"
 	"fmt"
+	"log/slog"
 	"qurio/apps/backend/internal/retrieval"
 	"qurio/apps/backend/internal/worker"
 	"github.com/weaviate/weaviate-go-client/v5/weaviate"
@@ -19,6 +20,7 @@ func NewStore(client *weaviate.Client) *Store {
 }
 
 func (s *Store) StoreChunk(ctx context.Context, chunk worker.Chunk) error {
+	slog.DebugContext(ctx, "storing chunk", "source_id", chunk.SourceID, "chunk_index", chunk.ChunkIndex, "url", chunk.SourceURL)
 	properties := map[string]interface{}{
 		"content":    chunk.Content,
 		"url":        chunk.SourceURL,
@@ -53,6 +55,9 @@ func (s *Store) StoreChunk(ctx context.Context, chunk worker.Chunk) error {
 		WithProperties(properties).
 		WithVector(chunk.Vector).
 		Do(ctx)
+	if err != nil {
+		slog.ErrorContext(ctx, "failed to store chunk", "error", err, "source_id", chunk.SourceID, "chunk_index", chunk.ChunkIndex)
+	}
 	return err
 }
 
@@ -89,6 +94,7 @@ func (s *Store) DeleteChunksBySourceID(ctx context.Context, sourceID string) err
 }
 
 func (s *Store) Search(ctx context.Context, query string, vector []float32, alpha float32, limit int, searchFilters map[string]interface{}) ([]retrieval.SearchResult, error) {
+	slog.DebugContext(ctx, "searching vector store", "query", query, "alpha", alpha, "limit", limit)
 	hybrid := s.client.GraphQL().HybridArgumentBuilder().
 		WithQuery(query).
 		WithVector(vector).
@@ -136,6 +142,7 @@ func (s *Store) Search(ctx context.Context, query string, vector []float32, alph
 
 	res, err := queryBuilder.Do(ctx)
 	if err != nil {
+		slog.ErrorContext(ctx, "search failed", "error", err)
 		return nil, err
 	}
 

apps/backend/internal/logger/handler.go

@@ -0,0 +1,22 @@
+package logger
+
+import (
+	"context"
+	"log/slog"
+	"qurio/apps/backend/internal/middleware"
+)
+
+type ContextHandler struct {
+	slog.Handler
+}
+
+func NewContextHandler(h slog.Handler) *ContextHandler {
+	return &ContextHandler{Handler: h}
+}
+
+func (h *ContextHandler) Handle(ctx context.Context, r slog.Record) error {
+	if id, ok := ctx.Value(middleware.CorrelationKey).(string); ok && id != "" {
+		r.AddAttrs(slog.String("correlation_id", id))
+	}
+	return h.Handler.Handle(ctx, r)
+}

apps/backend/internal/logger/handler_test.go

@@ -0,0 +1,31 @@
+package logger
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"log/slog"
+	"qurio/apps/backend/internal/middleware"
+	"testing"
+)
+
+func TestContextHandler_Handle(t *testing.T) {
+	var buf bytes.Buffer
+	jsonHandler := slog.NewJSONHandler(&buf, nil)
+	h := NewContextHandler(jsonHandler)
+	logger := slog.New(h)
+
+	ctx := context.Background()
+	ctx = middleware.WithCorrelationID(ctx, "test-correlation-id")
+
+	logger.InfoContext(ctx, "test message")
+
+	var logMap map[string]interface{}
+	if err := json.Unmarshal(buf.Bytes(), &logMap); err != nil {
+		t.Fatalf("failed to unmarshal log: %v", err)
+	}
+
+	if logMap["correlation_id"] != "test-correlation-id" {
+		t.Errorf("expected correlation_id 'test-correlation-id', got %v", logMap["correlation_id"])
+	}
+}

apps/backend/internal/worker/link_discovery.go

@@ -0,0 +1,52 @@
+package worker
+
+import (
+	"net/url"
+	"regexp"
+)
+
+func DiscoverLinks(sourceID, host string, links []string, currentDepth, maxDepth int, exclusions []string) []PageDTO {
+	if currentDepth >= maxDepth {
+		return nil
+	}
+
+	var newPages []PageDTO
+	seen := make(map[string]bool)
+
+	for _, link := range links {
+		// 1. External Check
+		linkU, err := url.Parse(link)
+		if err != nil || linkU.Host != host {
+			continue
+		}
+
+		// Normalize: Strip Fragment
+		linkU.Fragment = ""
+		normalizedLink := linkU.String()
+
+		// 2. Exclusion Check
+		excluded := false
+		for _, ex := range exclusions {
+			if matched, _ := regexp.MatchString(ex, normalizedLink); matched {
+				excluded = true
+				break
+			}
+		}
+		if excluded {
+			continue
+		}
+
+		if seen[normalizedLink] {
+			continue
+		}
+		seen[normalizedLink] = true
+
+		newPages = append(newPages, PageDTO{
+			SourceID: sourceID,
+			URL:      normalizedLink,
+			Status:   "pending",
+			Depth:    currentDepth + 1,
+		})
+	}
+	return newPages
+}