build(evaluation): add eval framework as a private submodule by lenafiedor · Pull Request #5190 · uzh-bf/klicker-uzh

lenafiedor · 2026-07-21T09:08:36Z

Description

Added the evaluation module from ai-infrastructure as submodule for Klicker for the future chatbot evaluation and quality assessment.

ClickUp Task: Chatbot: Adapt evaluation module for Klicker

Proposed Changes

Add evaluation as submodule in Klicker
Add ground truth QA pairs at data/ground_truth/klicker_fineco

Automated Verification

Run pnpm run check:all (linting, typechecking, formatting, syncpack)
Run target unit tests (pnpm --filter <package> test)

Summary by CodeRabbit

New Features
- Added evaluation materials covering finance concepts, portfolio theory, alpha, and covariance matrices.
- Added synthetic ground-truth datasets with explanations, formulas, examples, and source references.
- Added an optional evaluation framework that can be initialized explicitly when needed.
Documentation
- Documented the evaluation assets, dataset contents, and framework setup instructions.

greptile-apps · 2026-07-21T09:11:36Z

Confidence Score: 5/5

Safe to merge — the submodule is entirely opt-in (update = none, HTTPS URL), CI and regular clones are unaffected, and the committed data files contain only synthetic educational content with no personal data.

The change is purely additive: a new top-level directory with a passively registered submodule and committed Markdown ground-truth files. Nothing in the build, dev stack, or CI pipeline depends on the submodule being present.

Files Needing Attention: No files require special attention. The evaluation/README.md could benefit from a companion wiki entry under docs/, but the README itself is accurate and complete.

Important Files Changed

Filename	Overview
.gitmodules	Registers evaluation/framework as a submodule over HTTPS with update = none; correctly configured so CI and recursive clones skip it
evaluation/README.md	Documents the submodule opt-in pattern and data directory contents clearly; the initialization command is correct
evaluation/framework	Submodule entry pinned to commit 8ddc718; update = none means it is not fetched during regular workflow
evaluation/data/ground_truth/klicker_fineco/gt_001_cml_sharpe_ratio.md	Synthetic CAPM/CML ground-truth QA; technically correct financial content, no personal data
evaluation/data/ground_truth/klicker_fineco/gt_008_covariance_matrix_conditions.md	Synthetic covariance matrix QA; determinant check for matrix D is correct (det = 5x1 minus 9 = -4 < 0)

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[git clone uzh-bf/klicker-uzh] --> B{--recurse-submodules?}
    B -- yes --> C[evaluation/framework skipped\nupdate = none]
    B -- no --> C
    C --> D[evaluation/data/ committed normally\nground-truth .md files available]
    E[UZH-internal user] --> F[git submodule update\n--init --checkout\nevaluation/framework]
    F --> G[evaluation/framework\nmaterialized from\ngitlab.uzh.ch/ai-infrastructure/evaluation]
    G --> H[DeepEval harness\nreads data/ground_truth/]
    D --> H
    H --> I[Chatbot quality\nassessment results]

_{Reviews (4): Last reviewed commit: "fix(evaluation): restructure eval submod..." | Re-trigger Greptile}

coderabbitai · 2026-07-21T09:11:58Z

📝 Walkthrough

Walkthrough

The evaluation submodule is registered and updated, its usage is documented, and eight financial economics ground-truth Markdown entries are added covering portfolio theory, alpha, and covariance-matrix topics.

Changes

Evaluation module integration

Layer / File(s)	Summary
Submodule registration and documentation `.gitmodules`, `evaluation/README.md`, `evaluation/framework`	Registers the private evaluation framework submodule with updates disabled, records its new commit, and documents the framework and synthetic datasets.

Portfolio and CAPM ground truth

Layer / File(s)	Summary
Portfolio and CAPM entries `evaluation/data/ground_truth/klicker_fineco/gt_001_.md`, `gt_002_.md`, `gt_006_*.md`	Adds ground-truth explanations for the CML Sharpe-ratio slope, tangency-portfolio risk-aversion allocation, and portfolio-weight interpretation.

Alpha ground truth

Layer / File(s)	Summary
Alpha entries `evaluation/data/ground_truth/klicker_fineco/gt_003_.md`, `gt_004_.md`, `gt_005_.md`, `gt_007_.md`	Adds explanations for weighted zero-sum alpha, heterogeneous ex-ante alpha, ex-post alpha measurement, and deriving a third investor’s alpha.

Covariance ground truth

Layer / File(s)	Summary
Covariance-matrix entry `evaluation/data/ground_truth/klicker_fineco/gt_008_covariance_matrix_conditions.md`	Defines covariance-matrix validity criteria and evaluates four candidate matrices, identifying matrix C as valid.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: rschlaefli

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: adding the evaluation framework as a private submodule.
Description check	✅ Passed	The description covers the summary, ClickUp task, proposed changes, and automated verification, with only some template sections left incomplete.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…ones - move submodule modules/eval -> evaluation/framework - use HTTPS URL and update=none so clones (incl. --recursive) never fail on the UZH-private framework repo; internal opt-in init documented - move ground-truth data under evaluation/data/ - add evaluation/README.md describing the layout

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@evaluation/data/ground_truth/klicker_fineco/gt_007_third_investor_alpha.md`:
- Around line 9-21: Define r^i, r^j, and r^3 consistently as wealth weights,
clarify whether they are raw wealth or normalized shares, and use that notation
throughout the derivation. Explicitly state that investor 3 has positive (and
therefore nonzero) wealth before dividing by r^3.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 82e88366-cc3e-4c76-837c-7b4ebae75ae6

📥 Commits

Reviewing files that changed from the base of the PR and between c8de9c8 and 8a27d23.

📒 Files selected for processing (11)

.gitmodules
evaluation/README.md
evaluation/data/ground_truth/klicker_fineco/gt_001_cml_sharpe_ratio.md
evaluation/data/ground_truth/klicker_fineco/gt_002_tangency_portfolio_risk_aversion.md
evaluation/data/ground_truth/klicker_fineco/gt_003_weighted_alpha_zero_sum.md
evaluation/data/ground_truth/klicker_fineco/gt_004_ex_ante_alpha_heterogeneous_expectations.md
evaluation/data/ground_truth/klicker_fineco/gt_005_ex_post_alpha_same_measure.md
evaluation/data/ground_truth/klicker_fineco/gt_006_lambda_portfolio_weights.md
evaluation/data/ground_truth/klicker_fineco/gt_007_third_investor_alpha.md
evaluation/data/ground_truth/klicker_fineco/gt_008_covariance_matrix_conditions.md
evaluation/framework

coderabbitai · 2026-07-27T08:22:15Z

+Use the zero-sum property of alpha with wealth weights:
+
+$$
+\alpha^i r^i + \alpha^j r^j + \alpha^3 r^3 = 0
+$$
+
+Solve for the unknown alpha of investor 3:
+
+$$
+\alpha^3 = -\frac{\alpha^i r^i + \alpha^j r^j}{r^3}
+$$
+
+So you multiply each known investor's alpha by their wealth, add those terms, change the sign, and divide by the third investor's wealth.


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Define the wealth weights and denominator assumption.

This entry switches from a^i in gt_003_weighted_alpha_zero_sum.md to r^i without defining whether r^i is raw wealth or a normalized wealth share. It also divides by r^3 without stating that the third investor has nonzero wealth. Use one notation consistently and state the positive-wealth assumption.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evaluation/data/ground_truth/klicker_fineco/gt_007_third_investor_alpha.md` around lines 9 - 21, Define r^i, r^j, and r^3 consistently as wealth weights, clarify whether they are raw wealth or normalized shares, and use that notation throughout the derivation. Explicitly state that investor 3 has positive (and therefore nonzero) wealth before dividing by r^3.

sonarqubecloud · 2026-07-27T08:22:34Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

lenafiedor requested a review from rschlaefli as a code owner July 21, 2026 09:08

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. dependencies feature labels Jul 21, 2026

greptile-apps Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread .gitmodules Outdated

lenafiedor force-pushed the eval-submodule branch from a639d8a to 7a7e431 Compare July 21, 2026 09:12

rschlaefli changed the title ~~feat(modules/eval): Eval as a submodule~~ build(evaluation): add eval framework as a private submodule Jul 27, 2026

lenafiedor and others added 3 commits July 27, 2026 10:13

add eval submodule

1c52622

add klicker GT to main repo & update eval module

1ca7ce1

lenafiedor force-pushed the eval-submodule branch from 36381b7 to 8a27d23 Compare July 27, 2026 08:19

coderabbitai Bot reviewed Jul 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

build(evaluation): add eval framework as a private submodule - #5190

build(evaluation): add eval framework as a private submodule#5190
lenafiedor wants to merge 3 commits into
v3from
eval-submodule

lenafiedor commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

greptile-apps Bot commented Jul 21, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 27, 2026

Uh oh!

sonarqubecloud Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lenafiedor commented Jul 21, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Proposed Changes

Automated Verification

Summary by CodeRabbit

Uh oh!

greptile-apps Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Jul 27, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

lenafiedor commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading

greptile-apps Bot commented Jul 21, 2026 •

edited

Loading

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading