-
Notifications
You must be signed in to change notification settings - Fork 14
feat(kb): add production knowledge base ingestion #5174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
rschlaefli
wants to merge
24
commits into
v3-ai
Choose a base branch
from
kb-poc
base: v3-ai
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 4 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
d28a672
docs(project): add KB management POC implementation plan
rschlaefli ffecea3
docs(project): record PR 5174 id in KB POC plan filename
rschlaefli b8e1442
docs(project): revise KB POC plan to package architecture on v3-ai
rschlaefli 09a464f
feat(packages/prisma): add slim KB and KBResource models for KB POC
rschlaefli 6b86f9e
enhance(packages/prisma): support blob and web/URL KB resource kinds
rschlaefli 8df7777
docs(project): add KB production v1 roadmap plan from PR 5182/5078 re…
rschlaefli dcbdbe3
docs(project): record KB v1 grill rulings Q1-Q11 and finalize roadmap
rschlaefli cbb6780
docs(project): move KB roadmap onto integration branch
rschlaefli b66ae01
feat(kb): add management and ingestion POC
jabbadizzleCode d898fad
chore(kb): sync v3-ai integration line
rschlaefli c8ab80b
enhance(kb): align dispatch with ingestion API
rschlaefli fb39440
fix(kb): preserve ingestion retry identity
rschlaefli e7a3df8
enhance(kb): align ingestion status webhook
rschlaefli b646f9c
enhance(kb): finish ingestion contract cleanup
rschlaefli 3ca3490
fix(kb): harden ingestion reconciliation
rschlaefli 0d28539
fix(kb): allow worker startup without ingestion config
rschlaefli d520150
docs(kb): start W3 status history work
rschlaefli 89f2a4e
feat(kb): track ingestion history and serving state
rschlaefli 07bd7f2
feat(kb): show ingestion history and serving state
rschlaefli 519aa2a
docs(kb): close W3 status history milestone
rschlaefli bc368ea
docs(kb): start W4 chatbot retrieval work
rschlaefli 0491fd0
feat(kb): connect chatbots to scoped retrieval
rschlaefli cd4c806
fix(kb): harden scoped retrieval boundary
rschlaefli 69c2a49
docs(kb): recapture W4 chatbot evidence
rschlaefli File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| // ----- KNOWLEDGE BASES ----- | ||
| // #region | ||
|
|
||
| enum KBResourceStatus { | ||
| UPLOADED | ||
| QUEUED | ||
| PROCESSING | ||
| READY | ||
| FAILED | ||
| } | ||
|
|
||
| model KB { | ||
| id String @id @default(uuid()) @db.Uuid | ||
| name String | ||
| description String? | ||
|
|
||
| owner User @relation(fields: [ownerId], references: [id], onDelete: Cascade, onUpdate: Cascade) | ||
| ownerId String @db.Uuid | ||
|
|
||
| resources KBResource[] | ||
|
|
||
| createdAt DateTime @default(now()) | ||
| updatedAt DateTime @updatedAt | ||
|
|
||
| @@index([ownerId]) | ||
| } | ||
|
|
||
| model KBResource { | ||
| id String @id @default(uuid()) @db.Uuid | ||
|
|
||
| title String | ||
| originalFilename String | ||
| mimeType String | ||
| sizeBytes Int | ||
| blobName String | ||
| blobHref String | ||
|
|
||
| status KBResourceStatus @default(UPLOADED) | ||
| statusMessage String? | ||
| ingestedAt DateTime? | ||
|
|
||
| kb KB @relation(fields: [kbId], references: [id], onDelete: Cascade, onUpdate: Cascade) | ||
| kbId String @db.Uuid | ||
|
|
||
| createdAt DateTime @default(now()) | ||
| updatedAt DateTime @updatedAt | ||
|
|
||
| @@index([kbId, status]) | ||
| } | ||
|
|
||
| // #endregion | ||
45 changes: 45 additions & 0 deletions
45
packages/prisma/src/prisma/schema/migrations/20260715211610_kb_poc_schema/migration.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| -- CreateEnum | ||
| CREATE TYPE "public"."KBResourceStatus" AS ENUM ('UPLOADED', 'QUEUED', 'PROCESSING', 'READY', 'FAILED'); | ||
|
|
||
| -- CreateTable | ||
| CREATE TABLE "public"."KB" ( | ||
| "id" UUID NOT NULL, | ||
| "name" TEXT NOT NULL, | ||
| "description" TEXT, | ||
| "ownerId" UUID NOT NULL, | ||
| "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, | ||
| "updatedAt" TIMESTAMP(3) NOT NULL, | ||
|
|
||
| CONSTRAINT "KB_pkey" PRIMARY KEY ("id") | ||
| ); | ||
|
|
||
| -- CreateTable | ||
| CREATE TABLE "public"."KBResource" ( | ||
| "id" UUID NOT NULL, | ||
| "title" TEXT NOT NULL, | ||
| "originalFilename" TEXT NOT NULL, | ||
| "mimeType" TEXT NOT NULL, | ||
| "sizeBytes" INTEGER NOT NULL, | ||
| "blobName" TEXT NOT NULL, | ||
| "blobHref" TEXT NOT NULL, | ||
| "status" "public"."KBResourceStatus" NOT NULL DEFAULT 'UPLOADED', | ||
| "statusMessage" TEXT, | ||
| "ingestedAt" TIMESTAMP(3), | ||
| "kbId" UUID NOT NULL, | ||
| "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, | ||
| "updatedAt" TIMESTAMP(3) NOT NULL, | ||
|
|
||
| CONSTRAINT "KBResource_pkey" PRIMARY KEY ("id") | ||
| ); | ||
|
|
||
| -- CreateIndex | ||
| CREATE INDEX "KB_ownerId_idx" ON "public"."KB"("ownerId"); | ||
|
|
||
| -- CreateIndex | ||
| CREATE INDEX "KBResource_kbId_status_idx" ON "public"."KBResource"("kbId", "status"); | ||
|
|
||
| -- AddForeignKey | ||
| ALTER TABLE "public"."KB" ADD CONSTRAINT "KB_ownerId_fkey" FOREIGN KEY ("ownerId") REFERENCES "public"."User"("id") ON DELETE CASCADE ON UPDATE CASCADE; | ||
|
|
||
| -- AddForeignKey | ||
| ALTER TABLE "public"."KBResource" ADD CONSTRAINT "KBResource_kbId_fkey" FOREIGN KEY ("kbId") REFERENCES "public"."KB"("id") ON DELETE CASCADE ON UPDATE CASCADE; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
blobNameblobNameis server-minted as<uuid>.<ext>inrequestKbFileUpload, but neither the schema nor the plannedconfirmKbFileUploadservice checks whether aKBResourcerow already exists for that blob before inserting. A client that retriesconfirmKbFileUploadafter a network timeout (the blob is already present from the first call, soblobClient.exists()passes) will create a second row pointing to the same Azure blob. When either row is later deleted viadeleteKbResource, the shared blob is removed from storage while the other row remains — producing a danglingKBResourcewhose ingestion or re-download will silently fail.Adding
@@unique([blobName])toKBResource(UUIDs are globally unique, so the column is safe to uniquify without a composite key) gives a DB-level guarantee that theconfirmKbFileUploadmutation is idempotent under retries, with no service-layer coordination needed.