-
Notifications
You must be signed in to change notification settings - Fork 14
build(evaluation): add eval framework as a private submodule #5190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lenafiedor
wants to merge
3
commits into
v3
Choose a base branch
from
eval-submodule
base: v3
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| [submodule "evaluation/framework"] | ||
| path = evaluation/framework | ||
| url = https://gitlab.uzh.ch/ai-infrastructure/evaluation.git | ||
| update = none |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Evaluation | ||
|
|
||
| AI-quality evaluation assets for KlickerUZH. | ||
|
|
||
| - `framework/` — git submodule pointing at the UZH-internal | ||
| [`ai-infrastructure/evaluation`](https://gitlab.uzh.ch/ai-infrastructure/evaluation) | ||
| harness (transport-neutral DeepEval runner). The repository is **private to | ||
| UZH**: the submodule is declared with `update = none`, so regular clones — | ||
| including `git clone --recursive` — skip it and nothing in the build, dev | ||
| stack, or CI depends on it being present. | ||
| - `data/` — committed evaluation datasets (e.g. ground-truth QA cases under | ||
| `data/ground_truth/`). Synthetic examples only; no production or personal | ||
| data. | ||
|
|
||
| UZH-internal users with GitLab access can materialize the framework explicitly: | ||
|
|
||
| ```bash | ||
| git submodule update --init --checkout evaluation/framework | ||
| ``` | ||
|
|
||
| How the framework consumes the datasets in `data/` (ground-truth roots, tool | ||
| profiles, metric suites) is still being wired up and documented. |
22 changes: 22 additions & 0 deletions
22
evaluation/data/ground_truth/klicker_fineco/gt_001_cml_sharpe_ratio.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| --- | ||
| question: Does the CML in the CAPM have the slope of the Sharpe ratio? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## CML and Sharpe Ratio | ||
|
|
||
| Yes. In the CAPM, the Capital Market Line has the slope of the market portfolio's Sharpe ratio: | ||
|
|
||
| $$ | ||
| \frac{\mu^M - R_f}{\sigma_M} | ||
| $$ | ||
|
|
||
| where $\mu^M$ is the expected return of the market portfolio, $R_f$ is the risk-free rate, and $\sigma_M$ is the standard deviation of the market portfolio. | ||
|
|
||
| So the CML shows the expected excess return per unit of total risk. This is different from the SML: the SML uses beta on the x-axis and has slope $\mu^M - R_f$, the market risk premium. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, portfolio theory and CAPM | ||
| - FinEco FS26 Guide, Sharpe ratio and tangency portfolio |
30 changes: 30 additions & 0 deletions
30
...ion/data/ground_truth/klicker_fineco/gt_002_tangency_portfolio_risk_aversion.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| question: How can I calculate the shares in the risk-free asset and risky portfolio if risk aversion is 5.6 and I already calculated the optimal tangency portfolio without risk aversion? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Risk-Free Asset and Tangency Portfolio Shares | ||
|
|
||
| If the composition of the tangency portfolio is already known, risk aversion determines only how much total wealth is invested in that tangency portfolio. | ||
|
|
||
| Use: | ||
|
|
||
| $$ | ||
| \lambda^* = \frac{\mu(R_T) - R_f}{\psi \sigma_T^2} | ||
| $$ | ||
|
|
||
| With $\psi = 5.6$. Here $\lambda^*$ is the total share invested in the tangency portfolio, $\mu(R_T)$ is the expected return of the tangency portfolio, $R_f$ is the risk-free rate, and $\sigma_T^2$ is the variance of the tangency portfolio. | ||
|
|
||
| The share in the risk-free asset is: | ||
|
|
||
| $$ | ||
| 1 - \lambda^* | ||
| $$ | ||
|
|
||
| The relative weights inside the tangency portfolio stay the same as before. Risk aversion only scales the total risky position up or down. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, utility-maximizing portfolios | ||
| - FinEco FS26 Guide, portfolio optimization |
30 changes: 30 additions & 0 deletions
30
evaluation/data/ground_truth/klicker_fineco/gt_003_weighted_alpha_zero_sum.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| question: What is the formula where the weighted sum of alpha and a equals zero, and how should I interpret it? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Weighted Zero-Sum Condition for Alpha | ||
|
|
||
| The condition is: | ||
|
|
||
| $$ | ||
| \sum_{i=1}^{I} a^i \alpha^i = 0 | ||
| $$ | ||
|
|
||
| The $a^i$ are weights for the investors, for example based on wealth and risk aversion. The $\alpha^i$ are the investors' expected alphas. | ||
|
|
||
| The interpretation is: in market equilibrium, the weighted average alpha across investors must be zero. If some influential investors expect a positive alpha, other investors must have negative alphas with enough weight to offset it. | ||
|
|
||
| For two investors, the condition becomes: | ||
|
|
||
| $$ | ||
| a^1\alpha^1 + a^2\alpha^2 = 0 | ||
| $$ | ||
|
|
||
| So the two weighted alpha contributions must cancel each other out. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, CAPM with heterogeneous expectations | ||
| - FinEco FS26 Guide, alpha and investor weights |
24 changes: 24 additions & 0 deletions
24
.../ground_truth/klicker_fineco/gt_004_ex_ante_alpha_heterogeneous_expectations.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| --- | ||
| question: Are the formulas for ex-ante and ex-post alpha different under homogeneous and heterogeneous expectations? If yes, what is the formula under heterogeneous expectations? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Ex-Ante Alpha Under Heterogeneous Expectations | ||
|
|
||
| Yes, ex-ante alpha differs because under heterogeneous expectations each investor $i$ can have their own expected return $\mu_k^i$. | ||
|
|
||
| A useful form is: | ||
|
|
||
| $$ | ||
| \alpha_k^i = (\mu_k^i - R_f) - \beta_k(\bar{\mu}^M - R_f) | ||
| $$ | ||
|
|
||
| Here $\mu_k^i$ is investor $i$'s expected return for asset $k$, and $\bar{\mu}^M$ is the market expectation aggregated across investors. | ||
|
|
||
| Under homogeneous expectations, everyone uses the same expected returns, so there is no investor-specific $\mu_k^i$. Under heterogeneous expectations, the alpha is investor-specific. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, homogeneous CAPM and alpha | ||
| - Financial Economics script FS26, CAPM with heterogeneous expectations |
20 changes: 20 additions & 0 deletions
20
evaluation/data/ground_truth/klicker_fineco/gt_005_ex_post_alpha_same_measure.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| --- | ||
| question: Is ex-post alpha measured the same way under homogeneous and heterogeneous expectations? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Ex-Post Alpha | ||
|
|
||
| Yes. Ex-post alpha is measured the same way under homogeneous and heterogeneous expectations because it uses realized returns: | ||
|
|
||
| $$ | ||
| \hat{\alpha}_k = R_k - \left(R_f + \beta_k(R^M - R_f)\right) | ||
| $$ | ||
|
|
||
| It is the realized return of asset $k$ minus the return explained by the CAPM for its beta. The expectations differ ex ante, but once returns are realized, the ex-post measurement is the same. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, alpha definition and interpretation | ||
| - FinEco FS26 Guide, ex-ante and ex-post alpha |
28 changes: 28 additions & 0 deletions
28
evaluation/data/ground_truth/klicker_fineco/gt_006_lambda_portfolio_weights.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| --- | ||
| question: If lambda equals (0.2, 0.5, 0.3), what do the individual entries mean, and how much is invested in the risk-free asset and portfolios A and B? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Interpreting Portfolio Weights | ||
|
|
||
| The vector $\lambda = (0.2, 0.5, 0.3)$ describes portfolio weights. | ||
|
|
||
| In the usual notation: | ||
|
|
||
| - $\lambda_0 = 0.2$: 20% in the risk-free asset | ||
| - $\lambda_1 = 0.5$: 50% in risky portfolio or asset A | ||
| - $\lambda_2 = 0.3$: 30% in risky portfolio or asset B | ||
|
|
||
| The weights sum to $0.2 + 0.5 + 0.3 = 1$, so they represent 100% of invested wealth. | ||
|
|
||
| The portfolio return can be written as: | ||
|
|
||
| $$ | ||
| R_P = \lambda_0 R_f + \lambda_1 R_A + \lambda_2 R_B | ||
| $$ | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, portfolio weights | ||
| - FinEco FS26 Guide, portfolio theory |
26 changes: 26 additions & 0 deletions
26
evaluation/data/ground_truth/klicker_fineco/gt_007_third_investor_alpha.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| --- | ||
| question: How can I derive the alpha of a third portfolio if I know investor i's wealth and alpha, investor j's wealth and alpha, and only investor 3's wealth? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Alpha of the Third Investor | ||
|
|
||
| Use the zero-sum property of alpha with wealth weights: | ||
|
|
||
| $$ | ||
| \alpha^i r^i + \alpha^j r^j + \alpha^3 r^3 = 0 | ||
| $$ | ||
|
|
||
| Solve for the unknown alpha of investor 3: | ||
|
|
||
| $$ | ||
| \alpha^3 = -\frac{\alpha^i r^i + \alpha^j r^j}{r^3} | ||
| $$ | ||
|
|
||
| So you multiply each known investor's alpha by their wealth, add those terms, change the sign, and divide by the third investor's wealth. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, alpha and market equilibrium | ||
| - FinEco FS26 Guide, CAPM alpha properties | ||
24 changes: 24 additions & 0 deletions
24
evaluation/data/ground_truth/klicker_fineco/gt_008_covariance_matrix_conditions.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| --- | ||
| question: >- | ||
| Given the four matrices A=(1 2; -2 4), B=(1 2; 2 -1), C=(4 -2; -2 3), and D=(5 3; 3 1), which of them can be covariance matrices? | ||
| expected_tools_by_profile: | ||
| catalog_expert_v1: [EXPERT_df_fineco_expert] | ||
| --- | ||
|
|
||
| ## Covariance Matrix Conditions | ||
|
|
||
| A covariance matrix must be symmetric, have non-negative variances on the diagonal, and be positive semidefinite. | ||
|
|
||
| For the given matrices: | ||
|
|
||
| - $A = \begin{pmatrix}1 & 2 \\ -2 & 4\end{pmatrix}$ is not symmetric, so it cannot be a covariance matrix. | ||
| - $B = \begin{pmatrix}1 & 2 \\ 2 & -1\end{pmatrix}$ has a negative diagonal element, so it cannot be a covariance matrix. | ||
| - $C = \begin{pmatrix}4 & -2 \\ -2 & 3\end{pmatrix}$ is symmetric, has positive diagonal entries, and $\det(C) = 4 \cdot 3 - (-2)^2 = 8 > 0$, so it can be a covariance matrix. | ||
| - $D = \begin{pmatrix}5 & 3 \\ 3 & 1\end{pmatrix}$ is symmetric, but $\det(D) = 5 \cdot 1 - 3^2 = -4 < 0$, so it is not positive semidefinite. | ||
|
|
||
| Therefore, only matrix C can be a covariance matrix. | ||
|
|
||
| ## Sources | ||
|
|
||
| - Financial Economics script FS26, covariance matrices | ||
| - FinEco FS26 Guide, variance-covariance matrices |
Submodule framework
added at
8ddc71
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Define the wealth weights and denominator assumption.
This entry switches from
a^iingt_003_weighted_alpha_zero_sum.mdtor^iwithout defining whetherr^iis raw wealth or a normalized wealth share. It also divides byr^3without stating that the third investor has nonzero wealth. Use one notation consistently and state the positive-wealth assumption.🤖 Prompt for AI Agents