Skip to content

Add ML Flow#14

Merged
SamuelHLewis merged 3 commits intodevelopfrom
add-ml-flow
Mar 20, 2026
Merged

Add ML Flow#14
SamuelHLewis merged 3 commits intodevelopfrom
add-ml-flow

Conversation

@ccs-gs
Copy link
Copy Markdown
Contributor

@ccs-gs ccs-gs commented Mar 19, 2026

Summary

  • Added optional MLflow tracking to evaluation/run_evaluation.py (modeled after the WIS evaluation flow) so evaluations can log:
    • run params (mapper, truth set path, prompt name/path, sample count)
    • run metrics (accuracy_percent, accuracy_fraction, correct_predictions, evaluation_duration_seconds)
    • artifacts (selected prompt file and output results CSV)
  • Added new CLI flags for MLflow configuration:
    • --mlflow
    • --mlflow-tracking-uri
    • --mlflow-experiment-name
    • --mlflow-run-name
  • Added mlflow dependency to requirements.txt.
  • Updated README.md with MLflow usage options.
  • Added/extended tests in tests/test_run_evaluation.py:
    • existing output CSV behavior remains covered
    • new test verifies MLflow params, metrics, and artifacts are logged when enabled
  • Added .env.example with required Azure OpenAI variables and optional MLflow environment variables.

Why

This enables reproducible, trackable evaluation runs with experiment history and artifacts, making prompt/model comparison easier over time and aligning this repo with the evaluation observability pattern used in the referenced WIS script.

@ccs-gs ccs-gs marked this pull request as ready for review March 19, 2026 14:29
@SamuelHLewis
Copy link
Copy Markdown
Collaborator

@copilot review this PR, focusing on how results are logged in Azure MLFlow

Copy link
Copy Markdown

Copilot AI commented Mar 20, 2026

@SamuelHLewis I've opened a new pull request, #15, to work on those changes. Once the pull request is ready, I'll request review from you.

@SamuelHLewis
Copy link
Copy Markdown
Collaborator

Copilot review only flagged the possibility of making MLFlow optional, which we're deliberately not doing, so no changes are needed after this

@SamuelHLewis SamuelHLewis merged commit e92cbea into develop Mar 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants