Skip to content

TEST_COVERAGE_ANALYSIS.md

Latest commit

 

History

History
273 lines (202 loc) · 10.9 KB

TEST_COVERAGE_ANALYSIS.md

File metadata and controls

273 lines (202 loc) · 10.9 KB
Raw

Test Coverage Analysis for llama_index

Executive Summary

This analysis examines the test coverage across the llama_index monorepo, covering 946 test files across 600+ test directories. The project uses pytest with a 50% minimum coverage threshold enforced in CI. While the core framework has solid coverage foundations, several critical gaps exist in satellite packages, core subsystems, and integration test depth.

Key findings:

  • llama-index-finetuning has zero test coverage (2,236 LOC)
  • llama_index/core/instrumentation/ has no dedicated test directory (786 LOC)
  • Callback integrations are severely undertested (33% have tests, all shallow)
  • response_synthesizers, retrievers, and selectors in core have <10% test coverage
  • Many integration packages have only import-verification tests, not behavioral tests

1. Core Package Gaps (llama-index-core)

1.1 Modules With Zero Dedicated Tests

Module Source Files LOC Impact
core/instrumentation/ 25 786 HIGH - Observability/monitoring backbone
core/composability/ 3 110 LOW - Small but entirely untested
core/settings.py 1 248 MEDIUM - Global configuration singleton
core/bridge/ 5 212 MEDIUM - Pydantic/LangChain interop (heavily used, never directly tested)

core/instrumentation/ is the most critical gap. It contains the event handler system (agent, chat_engine, embedding, LLM, query, rerank, retrieval, synthesis events), span tracking, and the dispatcher. This infrastructure underpins all observability and is completely excluded from coverage via pyproject.toml:

[tool.coverage.run]
omit = [
    "llama_index/core/instrumentation/*",  # explicitly excluded
    "llama_index/core/workflow/*",
]

Recommendation: Remove the coverage exclusion and add unit tests for:

  • Event handler dispatch and registration
  • Span lifecycle (open, close, error)
  • Event payload serialization
  • Dispatcher thread-safety guarantees

1.2 Modules With Minimal Coverage (<10%)

Module Source Files LOC Test Files Test LOC Coverage
core/retrievers/ 6 993 2 23 ~2.3%
core/response_synthesizers/ 13 1,925 3 160 ~8.3%
core/selectors/ 6 633 1 53 ~8.4%
core/base/ 15 3,202 1 510 ~15.9%

core/retrievers/ has only one test file (test_composable_retriever.py, 23 lines) covering 6 modules and nearly 1,000 LOC. The retriever is a fundamental component in every RAG pipeline.

Recommendation:

  • Add tests for BaseRetriever, VectorIndexRetriever, and query transformation logic
  • Test retriever composition and chaining
  • Test async retrieval paths
  • Add edge cases: empty results, malformed queries, metadata filtering

core/response_synthesizers/ has only test_generate.py and test_refine.py for 13 source modules covering tree summarize, compact, accumulate, and other synthesis strategies.

Recommendation:

  • Add tests for TreeSummarize, CompactAndRefine, Accumulate synthesizers
  • Test streaming response synthesis
  • Test token-budget-aware synthesis
  • Test error recovery in multi-step synthesis chains

1.3 Workflow Module - Indirect-Only Coverage

core/workflow/ (14 modules, 196 LOC) has no dedicated test directory. It is tested indirectly via tests/agent/workflow/ (10 files, 2,478 lines), but these are integration tests exercising workflows through the agent layer. There are no isolated unit tests for:

  • context.py / context_serializers.py - Workflow context management
  • retry_policy.py - Retry and error handling policies
  • decorators.py - Workflow step decorators
  • events.py - Workflow event primitives
  • drawing.py - Workflow visualization

Recommendation: Add unit tests for workflow primitives independent of agents, especially context serialization round-trips and retry policy behavior.

2. Satellite Package Gaps

2.1 llama-index-finetuning - Zero Coverage (Critical)

Metric Value
Source files 23
Lines of code 2,236
Test files 0

This package provides fine-tuning engines for OpenAI, Azure OpenAI, MistralAI, embedding adapters, sentence transformers, Cohere rerankers, and cross-encoders. It also includes dataset generation utilities and a fine-tuning callback handler.

Recommendation (Priority: HIGH):

  • Add unit tests for OpenAIFinetuneEngine with mocked API calls
  • Test EmbeddingQAFinetuneDataset generation and serialization
  • Test validate_json.py with valid/invalid training data
  • Test OpenAIFineTuningHandler callback event recording
  • Test dataset generation utilities with controlled inputs

2.2 llama-index-experimental - 19% Coverage

3 test files cover only the Pandas/Polars query engines and exec_utils. Untested:

  • ParamTuner - Parameter tuning utility
  • Nudge - Prompt optimization module
  • JSONAlyzeQueryEngine - JSON query engine
  • All 3 natural language retrievers (NLCSVRetriever, NLDataFrameRetriever, NLJSONRetriever)
  • Output parsers and prompt templates

Recommendation: Add tests for the NL retrievers and JSONAlyze engine since these are user-facing query interfaces.

2.3 llama-index-cli - 21% Coverage

Only 2 test files cover the RAG CLI base. Untested:

  • new_package/ - Package scaffolding from templates
  • upgrade/ - Code migration utilities
  • command_line.py - Main CLI dispatcher and argument parsing

Recommendation: Add tests for package template generation (deterministic output) and CLI argument parsing/dispatch.

3. Integration Package Gaps

3.1 Callbacks - Severely Undertested

Only 4 of 12 callback packages have tests, and all 4 are shallow import-only checks that merely verify class inheritance:

def test_handler_callable():
    names_of_base_classes = [b.__name__ for b in Handler.__mro__]
    assert BaseCallbackHandler.__name__ in names_of_base_classes

Packages without any tests (8): agentops, aim, argilla, arize-phoenix, honeyhive, langfuse, literalai, opik

Recommendation (Priority: HIGH):

  • Add mocked behavioral tests for at least the top 3 most-used callbacks (langfuse, arize-phoenix, wandb)
  • Test event serialization, callback registration/deregistration, error handling when the external service is unavailable
  • Replace import-only tests with actual functionality tests

3.2 Embeddings - 11 Packages Without Tests

Missing: azure-openai, baseten, clarifai, fireworks, gaudi, huggingface-openvino, instructor, ipex-llm, opea, openvino-genai, premai

Recommendation: Prioritize azure-openai (widely used) - add mocked tests for embedding generation, batch operations, and error handling.

3.3 LLMs - 11 Packages Without Tests

Missing: clarifai, gaudi, ipex-llm, maritalk, mlx, mymagic, openvino-genai, openvino, optimum-intel, premai, text-generation-inference

Recommendation: Add at least class instantiation + mocked completion tests for each. Prioritize mlx (growing Apple Silicon adoption).

3.4 Shallow Integration Tests

Many integration packages that nominally "have tests" only test class instantiation or inheritance. A sample analysis found:

Package Tests Depth
Anthropic LLM 30+ Excellent - mocks, streaming, multi-provider, error handling
OpenAI Embeddings 6 Good - retry logic, error exhaustion, batch operations
Cohere Embeddings 13 Good - async, images, batch, serialization
Groq LLM 3 Shallow - only integration tests requiring API key
All 4 callback packages 1 each Trivial - import/inheritance check only

Recommendation: Establish a minimum test standard for integrations:

  1. Class instantiation with required parameters
  2. At least one mocked completion/embedding call
  3. At least one error handling scenario
  4. Serialization round-trip test

4. Structural and Process Improvements

4.1 Coverage Threshold Is Low

The CI enforces only 50% minimum coverage (COV_FAIL_UNDER: 50). For a framework this widely used, consider:

  • Raising to 60% for llama-index-core
  • Requiring 40% minimum for all integration packages
  • Adding differential coverage checks on PRs (already have diff-cover in llama-dev)

4.2 Coverage Exclusions Hide Gaps

The pyproject.toml explicitly excludes instrumentation/* and workflow/* from coverage measurement. These are production-critical subsystems that should be measured and tested.

4.3 Missing Test Categories

The test suite is predominantly unit tests. Consider adding:

  • Contract tests for integration packages: Verify each integration implements the base interface correctly (beyond just inheritance checking)
  • Snapshot/regression tests for response synthesizers: Ensure deterministic outputs don't change unexpectedly
  • Property-based tests (with Hypothesis) for text splitters and node parsers: These have complex edge cases that are well-suited to fuzzing
  • Performance regression tests for hot paths (embedding batching, retrieval)

4.4 Async Test Parity

While pytest-asyncio is configured and used, async code paths don't consistently have test parity with their sync counterparts. Key areas to audit:

  • Async retrieval in core/retrievers/
  • Async streaming in response synthesizers
  • Async workflow execution

5. Prioritized Recommendations

Tier 1 - Critical (High Impact, Currently Zero Coverage)

  1. Add tests for llama-index-finetuning - 2,236 LOC with no tests; mock external API calls and test dataset generation/validation logic
  2. Add tests for core/instrumentation/ - Remove coverage exclusion; test event dispatch, span lifecycle, and handler registration
  3. Improve callback integration tests - Replace trivial inheritance checks with mocked behavioral tests

Tier 2 - Important (Low Coverage on Critical Paths)

  1. Expand core/retrievers/ tests - Only 2.3% coverage on a fundamental RAG component
  2. Expand core/response_synthesizers/ tests - Only 8.3% coverage; test all synthesis strategies
  3. Add core/settings.py tests - Global configuration has no dedicated tests
  4. Add unit tests for core/workflow/ primitives - Context serialization, retry policies, decorators

Tier 3 - Valuable (Breadth Improvements)

  1. Add mocked tests for untested embedding integrations (especially azure-openai)
  2. Add mocked tests for untested LLM integrations (especially mlx)
  3. Improve llama-index-experimental coverage - NL retrievers and JSONAlyze engine
  4. Improve llama-index-cli coverage - Package scaffolding and CLI dispatch
  5. Establish minimum test standard for all integration packages

Tier 4 - Process Improvements

  1. Raise CI coverage threshold from 50% to 60% for core
  2. Remove instrumentation/* and workflow/* from coverage exclusions
  3. Add contract tests for integration base class compliance
  4. Investigate property-based testing for text splitters/node parsers