Proposal: Semantic Kernel RAG failure mode checklist (WFGY 16-problem ProblemMap)

[onestardao]

Hi, and thanks for Semantic Kernel. It has become a key framework for building orchestrated AI apps on top of multiple models and tools.

I maintain an MIT-licensed open-source project called WFGY (~1.5k GitHub stars).
Its main diagnostic component is a 16-problem “ProblemMap” for RAG and LLM pipelines, which organizes failure modes across:

• data ingestion, normalization, and chunking
• embeddings and vector store configuration
• retrievers, ranking, and re-ranking
• planner, tool call, and kernel routing behavior
• evaluation gaps, logging, and guardrails

ProblemMap overview:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

The ProblemMap is already referenced by external research projects and curated lists, including:

• ToolUniverse from Harvard MIMS Lab
• Multimodal RAG Survey from QCRI LLM Lab
• Rankify from University of Innsbruck

Given that Semantic Kernel is used to compose quite complex RAG workflows, many developers end up asking if a failure comes from their data layer, embeddings, planner, or prompt logic. The ProblemMap tries to give a neutral checklist for exactly these questions.

I would like to propose adding WFGY ProblemMap as an optional reference in the docs, for example:

• a short “RAG failure mode checklist” page under guidance or best practices
• a link from any existing troubleshooting sections that talk about retrieval quality

If this is interesting, I would be glad to draft a PR that:

• maps the 16 problems to typical Semantic Kernel patterns
• shows how to use the checklist without coupling to any specific model provider
• keeps content concise and aligned with Microsoft documentation style

Thanks for considering this suggestion, and for all the work you do on Semantic Kernel.

[rajantripathi]

I'd like to contribute to this RAG failure mode checklist. I've been working on RAG evaluation across multiple frameworks and have documented failure patterns from my multilingual healthcare RAG work.

Proposed contributions:

1. Healthcare-specific failure modes (missing from the original WFGY map):

   • PHI/PII leakage during retrieval
   • Medical terminology inconsistency across languages
   • Hallucination of treatment recommendations
   • Citation of outdated clinical guidelines

2. Multilingual RAG failure modes:

   • Script mismatch (Latin queries retrieving CJK documents)
   • Embedding density variance across languages
   • Cross-lingual semantic drift in chunking

3. Mapping to Semantic Kernel components:

   • Kernel → orchestration layer
   • TextSearchPlugin → retrieval layer
   • ChatHistoryPromptTemplate → context assembly

Example checklist entry:

### Failure #1: Retrieval Collapse
**Symptoms**: Irrelevant chunks returned for specific queries
**Semantic Kernel Components**: TextSearchPlugin, VectorStore
**Detection**: Context precision < 0.3
**Fix**: Check embedding model compatibility, verify query transformation

References:

• RAGAS PR Bump @adobe/css-tools from 4.2.0 to 4.3.1 in /samples/apps/chat-summary-webapp-react #2651: fix(testset): add adapt_all_prompts() to TestsetGenerator closes #1970 vibrantlabsai/ragas#2651 (multilingual evaluation fix)
• SOAS RAG Evaluation: https://github.com/rajantripathi/soas-rag-evaluation (benchmark data)

Would the maintainers prefer this as a standalone docs page or integrated into existing RAG documentation?