Adding a dynamic RAG system to handle build errors#1216

Draft

wenta0g wants to merge 43 commits intogoogle:mainfrom

Collaborator

wenta0g commented Jan 13, 2026

This PR introduces a RAG-based system to handle build errors more effectively. By retrieving previously successful fix snippets ("patch texts") from a database, the agent can reuse past solutions to avoid recurring build issues.

Implementation details:

Under agent, we add a memory_prototyper agent that extends the general workflow of prototyper, but injects historical fix data into the context when compilation fails.
Infrastructure: We add the memory_helper module with utilities to access and connect to cloudSQL instance.

Current Status and Purpose: This PR is a draft to verify that the memory_helper utilities can successfully authenticate and connect to the Cloud SQL instance from the CI/CD environment. It is not ready for review.

wenta0g added 2 commits

January 13, 2026 17:53


          add memory prototyper and dependencies

1fd1388


          add Cloud SQL Proxy sidecar with IAM auth

4ef8e34

Collaborator Author

wenta0g commented Jan 13, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -b quick-test -ns 1 -- --use-error-memory

github-actions bot commented Jan 13, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-01-13-1216-wentao-quick-test/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-13-1216-wentao-quick-test
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-13-1216-wentao-quick-test


          db proxy less resource

1bee8f1

Collaborator Author

wenta0g commented Jan 14, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -b quick-test -rd -ns 1 -- --use-error-memory

github-actions bot commented Jan 14, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-01-14-1216-wentao-quick-test/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-14-1216-wentao-quick-test
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-14-1216-wentao-quick-test

iany0 and others added 3 commits

January 20, 2026 17:20


          added new db field with filters

5a502a8


          Merge branch 'google:main' into main

d547216


          cloud sql connect add backup

dfbe285

Collaborator Author

wenta0g commented Jan 25, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -b minor-for-ci -rd -ns 1 -- --use-error-memory

github-actions bot commented Jan 25, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-01-25-1216-wentao-minor-for-ci/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-25-1216-wentao-minor-for-ci
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-25-1216-wentao-minor-for-ci

wenta0g added 2 commits

January 27, 2026 15:31


          presubmit format fix

38f161d


          lint fix

40ee45d

Collaborator Author

wenta0g commented Jan 27, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -b quick-test -rd -ns 5 -- --use-error-memory

github-actions bot commented Jan 27, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-01-27-1216-wentao-quick-test/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-27-1216-wentao-quick-test
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-27-1216-wentao-quick-test


          add memory_based enhancer

206f764

Collaborator Author

wenta0g commented Jan 27, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -- --use-error-memory

github-actions bot commented Jan 27, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-01-27-1216-wentao-comparison/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-27-1216-wentao-comparison
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-01-27-1216-wentao-comparison

iany0 and others added 2 commits

February 5, 2026 16:26


          Improved prompt and truncate stderr

730f3d0


          add a longer timeout when gemini 429 error (resource exhausted)

7c65e0f

Collaborator Author

wenta0g commented Feb 6, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -- --use-error-memory

github-actions bot commented Feb 6, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-02-06-1216-wentao-comparison/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-02-06-1216-wentao-comparison
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-02-06-1216-wentao-comparison


          smart truncate for long prompt to fetch true error message

f5f77a9

Collaborator Author

wenta0g commented Feb 10, 2026

/gcbrun exp -n wentao -m vertex_ai_gemini-2-5-pro-chat -ag -- --use-error-memory

github-actions bot commented Feb 10, 2026

Requested GKE Job: ofg-pr-1216-wentao

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wentao
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-02-10-1216-wentao-comparison/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-02-10-1216-wentao-comparison
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-02-10-1216-wentao-comparison


          create a new test set and enable project specific query

22849b6

Collaborator

iany0 commented Feb 23, 2026

/gcbrun exp -n wenqi-include -m vertex_ai_gemini-2-5-pro-chat -ag -b hard -- --use-error-memory --memory-project-filter all

github-actions bot commented Feb 23, 2026

Requested GKE Job: ofg-pr-1216-wenqi-include

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-include
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-02-23-1216-wenqi-include-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-02-23-1216-wenqi-include-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-02-23-1216-wenqi-include-hard


          updated benchmark and logic for query cloudsql

d508653

Collaborator

iany0 commented Mar 17, 2026

/gcbrun exp -n wenqi-rq341 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-2-5-pro-chat

github-actions bot commented Mar 17, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq341

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq341
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-17-1216-wenqi-rq341-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-17-1216-wenqi-rq341-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-17-1216-wenqi-rq341-hard


          fixed argument issue

ea03532

Collaborator

iany0 commented Mar 17, 2026

/gcbrun exp -n wenqi-rq342 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-2-5-pro-chat

github-actions bot commented Mar 17, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq342

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq342
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-17-1216-wenqi-rq342-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-17-1216-wenqi-rq342-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-17-1216-wenqi-rq342-hard


          fixed truncation

72de930

Collaborator

iany0 commented Mar 18, 2026

/gcbrun exp -n wenqi-rq343 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-2-5-pro-chat

github-actions bot commented Mar 18, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq343

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq343
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-18-1216-wenqi-rq343-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-18-1216-wenqi-rq343-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-18-1216-wenqi-rq343-hard


          refined truncation logic

3cbc2d3

Collaborator

iany0 commented Mar 20, 2026

/gcbrun exp -n wenqi-rq344 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-2-5-pro-chat

github-actions bot commented Mar 20, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq344

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq344
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-20-1216-wenqi-rq344-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-20-1216-wenqi-rq344-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-20-1216-wenqi-rq344-hard


          added sleep and refined truncation logic

2b23dc9

Collaborator

iany0 commented Mar 21, 2026

/gcbrun exp -n wenqi-rq345 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-2-5-pro-chat

github-actions bot commented Mar 21, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq345

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq345
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-21-1216-wenqi-rq345-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-21-1216-wenqi-rq345-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-21-1216-wenqi-rq345-hard


          fixed logger

Collaborator

iany0 commented Mar 22, 2026

/gcbrun exp -n wenqi-rq346 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-2-5-pro-chat

github-actions bot commented Mar 22, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq346

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq346
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-22-1216-wenqi-rq346-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-22-1216-wenqi-rq346-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-22-1216-wenqi-rq346-hard


          removed truncation log

206bfc8

Collaborator

iany0 commented Mar 22, 2026

/gcbrun exp -n wenqi-rq35 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-3-1-pro-chat

github-actions bot commented Mar 22, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq35

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq35
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-22-1216-wenqi-rq35-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-22-1216-wenqi-rq35-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-22-1216-wenqi-rq35-hard

Collaborator

iany0 commented Mar 23, 2026

/gcbrun exp -n wenqi-rq351 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-3-1-pro-chat

github-actions bot commented Mar 23, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq351

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq351
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-23-1216-wenqi-rq351-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-23-1216-wenqi-rq351-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-23-1216-wenqi-rq351-hard

Collaborator

iany0 commented Mar 24, 2026

/gcbrun exp -n wenqi-rq352 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory --memory-model-filter vertex_ai_gemini-3-1-pro-chat

github-actions bot commented Mar 24, 2026

Requested GKE Job: ofg-pr-1216-wenqi-rq352

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wenqi-rq352
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-03-24-1216-wenqi-rq352-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-24-1216-wenqi-rq352-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-03-24-1216-wenqi-rq352-hard

Collaborator

iany0 commented Apr 15, 2026

/gcbrun exp -n wqrq1 -m vertex_ai_gemini-2-5-pro-chat -ag -b comparison-new -- --use-error-memory --memory-project-filter only-current

github-actions bot commented Apr 15, 2026

Requested GKE Job: ofg-pr-1216-wqrq1

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wqrq1
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-04-15-1216-wqrq1-comparison-new/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-04-15-1216-wqrq1-comparison-new
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-04-15-1216-wqrq1-comparison-new

Collaborator

iany0 commented Apr 16, 2026

/gcbrun exp -n wqrq2 -m vertex_ai_gemini-2-5-pro-chat -ag -b comparison-new -- --use-error-memory --memory-project-filter exclude-current

github-actions bot commented Apr 16, 2026

Requested GKE Job: ofg-pr-1216-wqrq2

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wqrq2
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-04-16-1216-wqrq2-comparison-new/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-04-16-1216-wqrq2-comparison-new
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-04-16-1216-wqrq2-comparison-new

Collaborator

iany0 commented Apr 16, 2026

/gcbrun exp -n wqrq44 -m vertex_ai_gemini-3-1-pro-chat -ll global -ag -b hard -- --use-error-memory

github-actions bot commented Apr 16, 2026

Requested GKE Job: ofg-pr-1216-wqrq44

JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-1216-wqrq44
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2026-04-16-1216-wqrq44-hard/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-04-16-1216-wqrq44-hard
BUCKET (GS): gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2026-04-16-1216-wqrq44-hard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet