Brainstorm Roadmap

[joelthe1]

• API
  • Stable auth token
• Support agents
  • need more details
  • Central controlling agent that coordinates actions
• Include images (and other modalities?)!
  • Find datasets
• Multi-language support
  • via MT
  • Target fixed number of languages
  • Add a layer at the user-interaction point
• Setup with more public content
  • CCEL
  • UW data
• STT and TTS
• Reproducible data loading
  • Data partitioning strategies

[ryderwishart]

Some updates:

• Data
  • clean up data?
  • add more data? (ccel data)
  • add richer metadata (markdown splitting)
• Platform functionality
  • stable auth token
  • capture usage (standard monitoring stacks?)
  • add evaluation and feedback (thumbs up or down) (types of human feedback)
  • tests (pytest already implemented; needs to be extended)
  • better source document transparency (e.g., show document snippet or title along with filename)
• Improve QA prompting pipeline
  • simple functions to return information (direct db lookup, maybe alongside vector retrieval?)
  • clarify user query (query rewriting)
  • Other RAG optimizations (various rag prompt styles, making rag work with complex documents)
  • Multimodal retrieval
• review logs to harvest user queries
• add a second LLM (PredictionGuard?)
• Ryder integration with some kind of plugin (vague at this point)
• Add a dev server - Mark/SIL MLops person?
• Improve developer experience

[ryderwishart]

Next priorities (in order):

• stable auth token Create stable auth tokens #118, refs We need some way to check when accessToken has expired #53
• better source document transparency (e.g., show document snippet or title along with filename) Show more information about source documents #119
• add evaluation and feedback (thumbs up or down) (types of human feedback) Example of capturing inputs/completions in LangChain #25
  • allow testers to add comments about a thumbs up or down
  • track 'I don't know' answers
  • come up with a plan for how to leverage this
  • frameworks for capturing feedback? Host an open source one?

@kavitharaju and @alejandroaquintero are adding some additional tests, including an integration test to ensure deployments return 200