Merge epic/file-schema-v6 into main#785
Conversation
… migration (#618) * feat(proto): add proto definitions and build setup for file schema v7 migration - Add Protobuf definitions for KHI file schema v7 (split into shared.proto, style.proto, timeline.proto, etc.) to migrate from the previous JSON-based file format. - Set up Buf configuration (buf.yaml, buf.gen.yaml) to generate Go and TypeScript code. - Add `proto-gen` target to `scripts/make/codegen.mk` using `npx @bufbuild/buf` to support the new build process. - Generate Go code in `pkg/generated/khifile/` and TypeScript code in `web/src/app/generated/`. * fixed typos on proto fields * Specified proto rule and build target * Rebuild the generated files from the updated proto * Added linguist-generated for ignoreing generated files by proto * fix reviewed points and lint error * fix version number * Skip proto breaking change check until it's merged * Fixed shared type names * fix typos on comments
* feat: update InternedValue proto to have int64 value * Adding KHI file schema v6 namespaced ID generator
* feat: update InternedValue proto to have int64 value * Adding KHI file schema v6 namespaced ID generator * Adding generated files I forgot to push
* feat: update InternedValue proto to have int64 value * Adding KHI file schema v6 namespaced ID generator * Adding generated files I forgot to push * Fix the lint thing on the generated codes * Adding interning utility used in khi file format v6
* feat(web): add v6 parser binary reader and streamer * address review comments
* Removing unused CORS code (#619) * Removing unused CORS code * Update pkg/core/init/default/defaultextension.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Replaced unsafe innerHTML with textContent (#620) * feat(startup): implement smart component and integrate data loading (#599) * feat(startup): add inspection list components * feat(startup): implement smart component and integrate data loading #558 * Fix name of the function pointed out by the review * chore: fix make setup error in jj and update dev guide (#629) * chore: fix make setup error in jj and update dev guide * fix commented issues on review * fixed minor issues on Makefile (#644) * Merge epic/csm-parser into main (#645) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name (#635) * Add fleet PJ form task and query task to gather CSM TD audit logs (#636) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for TD resource logs (#637) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for traffic director resource logs * 🐛 fix set-input suggestion being wrapped for every lines (#638) * fix set-input suggestion being wrapped for every lines * Update web/src/app/timeline-toolbar/components/set-input-popup.component.scss Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Removing unused CORS code (#619) * Removing unused CORS code * Update pkg/core/init/default/defaultextension.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Replaced unsafe innerHTML with textContent (#620) * feat(startup): implement smart component and integrate data loading (#599) * feat(startup): add inspection list components * feat(startup): implement smart component and integrate data loading #558 * Fix name of the function pointed out by the review * chore: fix make setup error in jj and update dev guide (#629) * chore: fix make setup error in jj and update dev guide * fix commented issues on review * fixed minor issues on Makefile (#644) * Merge epic/csm-parser into main (#645) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name (#635) * Add fleet PJ form task and query task to gather CSM TD audit logs (#636) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for TD resource logs (#637) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for traffic director resource logs * 🐛 fix set-input suggestion being wrapped for every lines (#638) * fix set-input suggestion being wrapped for every lines * Update web/src/app/timeline-toolbar/components/set-input-popup.component.scss Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix bugs to register the CSM TD parser tasks (#647) * Fix the same issue of #638 (#659) * Adding make clean command (#651) * Addming make clean command * fixed commented points * Removing unused old headers on frontend (#660) * removed prefix 'v2' in shader files (#661) * rename commonlogk8sauditv2 to commonlogk8saudit (#662) --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Removing unused CORS code (#619) * Removing unused CORS code * Update pkg/core/init/default/defaultextension.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Replaced unsafe innerHTML with textContent (#620) * feat(startup): implement smart component and integrate data loading (#599) * feat(startup): add inspection list components * feat(startup): implement smart component and integrate data loading #558 * Fix name of the function pointed out by the review * chore: fix make setup error in jj and update dev guide (#629) * chore: fix make setup error in jj and update dev guide * fix commented issues on review * fixed minor issues on Makefile (#644) * Merge epic/csm-parser into main (#645) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name (#635) * Add fleet PJ form task and query task to gather CSM TD audit logs (#636) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for TD resource logs (#637) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for traffic director resource logs * 🐛 fix set-input suggestion being wrapped for every lines (#638) * fix set-input suggestion being wrapped for every lines * Update web/src/app/timeline-toolbar/components/set-input-popup.component.scss Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix bugs to register the CSM TD parser tasks (#647) * Fix the same issue of #638 (#659) * Adding make clean command (#651) * Addming make clean command * fixed commented points * Removing unused old headers on frontend (#660) * removed prefix 'v2' in shader files (#661) * rename commonlogk8sauditv2 to commonlogk8saudit (#662) * fixed the flaky backend test due to race condition (#665) * improved sticky header to show resource layer (#663) --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This type is duplication of the pb.InternedStruct. It's internal type of the domain store but its a shared proto type which is expected not to change by file version update.
* Removing unused CORS code (#619) * Removing unused CORS code * Update pkg/core/init/default/defaultextension.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Replaced unsafe innerHTML with textContent (#620) * feat(startup): implement smart component and integrate data loading (#599) * feat(startup): add inspection list components * feat(startup): implement smart component and integrate data loading #558 * Fix name of the function pointed out by the review * chore: fix make setup error in jj and update dev guide (#629) * chore: fix make setup error in jj and update dev guide * fix commented issues on review * fixed minor issues on Makefile (#644) * Merge epic/csm-parser into main (#645) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name (#635) * Add fleet PJ form task and query task to gather CSM TD audit logs (#636) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for TD resource logs (#637) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for traffic director resource logs * 🐛 fix set-input suggestion being wrapped for every lines (#638) * fix set-input suggestion being wrapped for every lines * Update web/src/app/timeline-toolbar/components/set-input-popup.component.scss Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix bugs to register the CSM TD parser tasks (#647) * Fix the same issue of #638 (#659) * Adding make clean command (#651) * Addming make clean command * fixed commented points * Removing unused old headers on frontend (#660) * removed prefix 'v2' in shader files (#661) * rename commonlogk8sauditv2 to commonlogk8saudit (#662) * fixed the flaky backend test due to race condition (#665) * improved sticky header to show resource layer (#663) * Removed redundant v from the version string (#671) * bugfix: automatic log scroll is not matching to the actual element (#675) * bugfix: ignore changing selected timeline when user clicked a log from log list (#674) * bugfix: automatic scroll to the beginning time of log query when user selected a timeline without point to a log (#676) * bugfix: scaling around right edge of query duration may be clipped to jump to other timing (#677) --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Added style override dialog to preview timeline styles easier * fixed GCA's commented part
* Migrate log searching to WebWorker * fixed GCA's commented part
* Implemented cancellation * fixed GCA's commented part
… folders (#765) * adjusting colors and styles * refactor: reuse common contract methods directly in autoscaler parser
* Optimized memory usage for searching * fixed GCA's reviewed part
* bug: fix log scrolling by arrow keys * fixed GCA's reviewed part
* update descriptions on feature tasks * fixed GCA's reviewed part
* Improved search feature on diffs/logs * Address review comments for PR 773
… are associated with the same log (#772) * bug: fix unintentionally jump to another timeline when 2 or more logs are associated with the same log * Addressed review comments for PR 772
* Removing unused CORS code (#619) * Removing unused CORS code * Update pkg/core/init/default/defaultextension.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Replaced unsafe innerHTML with textContent (#620) * feat(startup): implement smart component and integrate data loading (#599) * feat(startup): add inspection list components * feat(startup): implement smart component and integrate data loading #558 * Fix name of the function pointed out by the review * chore: fix make setup error in jj and update dev guide (#629) * chore: fix make setup error in jj and update dev guide * fix commented issues on review * fixed minor issues on Makefile (#644) * Merge epic/csm-parser into main (#645) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name (#635) * Add fleet PJ form task and query task to gather CSM TD audit logs (#636) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for TD resource logs (#637) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for traffic director resource logs * 🐛 fix set-input suggestion being wrapped for every lines (#638) * fix set-input suggestion being wrapped for every lines * Update web/src/app/timeline-toolbar/components/set-input-popup.component.scss Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix bugs to register the CSM TD parser tasks (#647) * Fix the same issue of #638 (#659) * Adding make clean command (#651) * Addming make clean command * fixed commented points * Removing unused old headers on frontend (#660) * removed prefix 'v2' in shader files (#661) * rename commonlogk8sauditv2 to commonlogk8saudit (#662) * fixed the flaky backend test due to race condition (#665) * improved sticky header to show resource layer (#663) * Removed redundant v from the version string (#671) * bugfix: automatic log scroll is not matching to the actual element (#675) * bugfix: ignore changing selected timeline when user clicked a log from log list (#674) * bugfix: automatic scroll to the beginning time of log query when user selected a timeline without point to a log (#676) * bugfix: scaling around right edge of query duration may be clipped to jump to other timing (#677) * Updated CODEOWNER file (#678) * Updated the release.yml for GitHub (#679) * fix: confirm refresh when backend disconnects (#692) * fix: confirm refresh when backend disconnects Signed-off-by: Haihan Jiang <haihanj99@gmail.com> * chore: use absolute imports in root tests Signed-off-by: Haihan Jiang <haihanj99@gmail.com> * Align unload confirmation with disconnect review --------- Signed-off-by: Haihan Jiang <haihanj99@gmail.com> * Handle slog attributes in KHI log formatter (#708) * Handle slog attributes in KHI log formatter * Optimize KHI log attribute formatting * Handle KHI log formatter review feedback * Change default listen host to 127.0.0.1 from localhost (#759) * Improve retry logic to consider temporal failure due to ECP issue (#760) * Improve retry logic to consider temporal failure due to ECP issue * fixed GCA's reviewed points --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Signed-off-by: Haihan Jiang <haihanj99@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Haihan Jiang <haihanj99@gmail.com>
* Optimize log lookup and resolve sticky header overlay placement - Resolve target timeline fallback order (selectedTimeline -> descendants -> all timelines) - Compensate hover placement offset when hovering directly over the sticky header - Optimize preceding and succeeding log extraction using bisectLeft - Standardize timestamps to bigint (nanoseconds) across hover resolution * Visualize current cursor time with a dedicated separator line - Propagate cursor time as bigint (nanoseconds) to the hover overlay - Update lastClickedTime dynamically on mouse movement across empty timeline periods - Conditionally insert a dedicated cursor separator row (isCursor: true) only when hovering over empty timeline periods - Render cursor row with flex container and CSS dashed lines flanking centered text - Enforce dashed lines expansion with flex: 1 1 20px and height: 1px - Update Storybook stories to include mock cursorTime property * addressed comments on PR777
#776) * feat: show RevisionState description popup on hover in timeline legend * addressed comments on PR776
* feat: embed graph view as a GoldenLayout tab * Remove diagram button from the toolbar * addressed PR comments on #780
* make storybook not freezed by serialization * addressed PR781 review comments
* Make graph draw feature async * addressed PR782 review comments
There was a problem hiding this comment.
Code Review
This pull request introduces the new KHI file format v6, migrating the serialization and parsing architecture to Protobuf. It adds extensive coding standards, guidelines, and architecture documentation, while removing legacy HTTP client, history, and parser models. It also updates the Google Cloud API call options to retry transient unauthenticated errors and refactors log ingesters and timeline mappers to use the new v6 Builder and accumulators. The review feedback suggests minor optimizations to avoid unnecessary allocations in WithAttrs and NewFieldFilterNode when attributes or keys are empty, as well as removing a redundant context error check in the log ingester task.
| func NewFieldFilterNode(original Node, keys []string) Node { | ||
| if original == nil { | ||
| return nil | ||
| } | ||
| if original.Type() != MapNodeType { | ||
| return original | ||
| } |
There was a problem hiding this comment.
If keys is empty, we can return original directly. This avoids wrapping the node in a fieldFilterNode and allocating the ignored map unnecessarily.
| func NewFieldFilterNode(original Node, keys []string) Node { | |
| if original == nil { | |
| return nil | |
| } | |
| if original.Type() != MapNodeType { | |
| return original | |
| } | |
| func NewFieldFilterNode(original Node, keys []string) Node { | |
| if original == nil { | |
| return nil | |
| } | |
| if len(keys) == 0 || original.Type() != MapNodeType { | |
| return original | |
| } |
References
- When implementing a node wrapper or filter that only applies to a specific node type (such as MapNodeType), perform the type check in the constructor or factory function and return the original node directly if it is not of the target type, rather than wrapping it and checking the type inside individual methods.
| if hasErr() { | ||
| return | ||
| } | ||
| if err := ctx.Err(); err != nil { | ||
| setErr(err) | ||
| return | ||
| } |
There was a problem hiding this comment.
* Removing unused CORS code (#619) * Removing unused CORS code * Update pkg/core/init/default/defaultextension.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Replaced unsafe innerHTML with textContent (#620) * feat(startup): implement smart component and integrate data loading (#599) * feat(startup): add inspection list components * feat(startup): implement smart component and integrate data loading #558 * Fix name of the function pointed out by the review * chore: fix make setup error in jj and update dev guide (#629) * chore: fix make setup error in jj and update dev guide * fix commented issues on review * fixed minor issues on Makefile (#644) * Merge epic/csm-parser into main (#645) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name (#635) * Add fleet PJ form task and query task to gather CSM TD audit logs (#636) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for TD resource logs (#637) * Add Inventory/Discovery tasks for parsing the map from NEG name to BS name * Adding form task to receive the fleet project and query task to gather logs from Cloud Logging * Add timeline mapper for traffic director resource logs * 🐛 fix set-input suggestion being wrapped for every lines (#638) * fix set-input suggestion being wrapped for every lines * Update web/src/app/timeline-toolbar/components/set-input-popup.component.scss Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: kyasbal <kyasbal1994@gmail.com> --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix bugs to register the CSM TD parser tasks (#647) * Fix the same issue of #638 (#659) * Adding make clean command (#651) * Addming make clean command * fixed commented points * Removing unused old headers on frontend (#660) * removed prefix 'v2' in shader files (#661) * rename commonlogk8sauditv2 to commonlogk8saudit (#662) * fixed the flaky backend test due to race condition (#665) * improved sticky header to show resource layer (#663) * Removed redundant v from the version string (#671) * bugfix: automatic log scroll is not matching to the actual element (#675) * bugfix: ignore changing selected timeline when user clicked a log from log list (#674) * bugfix: automatic scroll to the beginning time of log query when user selected a timeline without point to a log (#676) * bugfix: scaling around right edge of query duration may be clipped to jump to other timing (#677) * Updated CODEOWNER file (#678) * Updated the release.yml for GitHub (#679) * fix: confirm refresh when backend disconnects (#692) * fix: confirm refresh when backend disconnects Signed-off-by: Haihan Jiang <haihanj99@gmail.com> * chore: use absolute imports in root tests Signed-off-by: Haihan Jiang <haihanj99@gmail.com> * Align unload confirmation with disconnect review --------- Signed-off-by: Haihan Jiang <haihanj99@gmail.com> * Handle slog attributes in KHI log formatter (#708) * Handle slog attributes in KHI log formatter * Optimize KHI log attribute formatting * Handle KHI log formatter review feedback * Change default listen host to 127.0.0.1 from localhost (#759) * Improve retry logic to consider temporal failure due to ECP issue (#760) * Improve retry logic to consider temporal failure due to ECP issue * fixed GCA's reviewed points * bug: fix CSM's access log query was wrong for GDC clusters (#779) * bug: fix CSM's access log query was wrong for GDC clusters * addressed comment in #779 --------- Signed-off-by: kyasbal <kyasbal1994@gmail.com> Signed-off-by: Haihan Jiang <haihanj99@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Haihan Jiang <haihanj99@gmail.com>
TODO: Description will be added soon