feat(adaptive-learning): implement adaptive learning test#5113
feat(adaptive-learning): implement adaptive learning test#5113jabbadizzleCode wants to merge 7 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Confidence Score: 5/5This looks safe to merge.
|
| Filename | Overview |
|---|---|
| packages/adaptive-learning/src/index.ts | Adds the core adaptive-learning math and selection utilities. |
| packages/shared-components/src/adaptive/utils.ts | Adds shared display helpers for adaptive levels and theta formatting. |
| packages/graphql/package.json | Adds the adaptive-learning workspace package dependency. |
| packages/prisma/src/prisma/schema/adaptive.prisma | Adds the adaptive assessment schema models. |
Reviews (6): Last reviewed commit: "Merge branch 'v3' of github.com:uzh-bf/k..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const assessment = await ctx.prisma.$transaction(async (prisma) => { | ||
| if (input.id) { | ||
| const existing = await prisma.adaptiveAssessment.findUnique({ | ||
| where: { id: input.id }, | ||
| }) | ||
|
|
||
| if (!existing || existing.courseId !== input.courseId) { | ||
| throw new Error('Adaptive assessment not found.') | ||
| } | ||
|
|
||
| await prisma.adaptiveAssessmentResponse.deleteMany({ | ||
| where: { attempt: { assessmentId: input.id } }, | ||
| }) | ||
| await prisma.adaptiveAssessmentAttempt.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
| await prisma.adaptiveAssessmentResultMessage.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
| await prisma.adaptiveAssessmentElement.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
| await prisma.adaptiveAssessmentSubCompetence.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
| await prisma.adaptiveAssessmentCompetence.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
| await prisma.adaptiveAssessmentLevel.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
|
|
||
| return prisma.adaptiveAssessment.update({ | ||
| where: { id: input.id }, | ||
| data: assessmentData(input, course.ownerId), | ||
| }) | ||
| } | ||
|
|
||
| return prisma.adaptiveAssessment.create({ | ||
| data: { | ||
| ...assessmentData(input, course.ownerId), | ||
| courseId: input.courseId, | ||
| }, | ||
| }) | ||
| }) | ||
|
|
||
| await createAdaptiveAssessmentConfig(assessment.id, input, ctx) | ||
|
|
||
| return ctx.prisma.adaptiveAssessment.findUnique({ | ||
| where: { id: assessment.id }, | ||
| include: assessmentInclude, | ||
| }) |
There was a problem hiding this comment.
The first $transaction deletes all levels, competences, sub-competences, elements, and result messages for an existing assessment. createAdaptiveAssessmentConfig (which recreates them) is called after that transaction completes. If the config creation throws—invalid element mapping, DB constraint, or any transient error—the assessment is left with its config fully deleted but not recreated. At that point the assessment has no levels, no question pool, and no result messages, and there is no automatic recovery path. The delete-then-recreate should be wrapped in a single atomic $transaction so both steps either succeed together or roll back together.
| await prisma.adaptiveAssessmentResponse.deleteMany({ | ||
| where: { attempt: { assessmentId: input.id } }, | ||
| }) | ||
| await prisma.adaptiveAssessmentAttempt.deleteMany({ | ||
| where: { assessmentId: input.id }, | ||
| }) | ||
| await prisma.adaptiveAssessmentResultMessage.deleteMany({ |
There was a problem hiding this comment.
upsertAdaptiveAssessment wipes every AdaptiveAssessmentAttempt and AdaptiveAssessmentResponse for an assessment every time it is saved, regardless of publication status. If a lecturer edits any field on a published assessment while students have already taken it (or are currently taking it), all historical and in-progress attempt data is silently destroyed. There is no status check (DRAFT vs PUBLISHED) here. At minimum, the deletion of attempts and responses should be skipped—or blocked entirely with an error—unless the assessment is still in DRAFT status.
| return ordered.map((level, index) => { | ||
| const start = min + (span * index) / ordered.length | ||
| const end = min + (span * (index + 1)) / ordered.length |
There was a problem hiding this comment.
mapLevelsToBands divides the theta range into N equal slices (width = span/N). mapLevelsToTheta in packages/adaptive-learning/src/index.ts places level thetas at N-1-spaced intervals and uses midpoints as boundaries (width ≈ span/(N-1)). For a typical 4-level setup with range [-3, 3] the discrepancy is visible: the frontend shows the A2/B1 boundary at -1.5 while the backend assigns B1 only for θ ≥ -2. A user at θ = 1.8 would see themselves visually inside the "C1 band" but receive the B2 label.
| return ordered.map((level, index) => { | |
| const start = min + (span * index) / ordered.length | |
| const end = min + (span * (index + 1)) / ordered.length | |
| return ordered.map((level, index) => { | |
| const denominator = Math.max(ordered.length - 1, 1) | |
| const theta = min + (span * index) / denominator | |
| const prevTheta = index === 0 ? min : min + (span * (index - 1)) / denominator | |
| const nextTheta = | |
| index === ordered.length - 1 | |
| ? max | |
| : min + (span * (index + 1)) / denominator | |
| const start = index === 0 ? min : (theta + prevTheta) / 2 | |
| const end = index === ordered.length - 1 ? max : (theta + nextTheta) / 2 |
| await ctx.prisma.$transaction([ | ||
| ctx.prisma.adaptiveAssessmentResponse.create({ | ||
| data: { | ||
| attemptId, | ||
| adaptiveElementId, | ||
| elementId: adaptiveElement.elementId, | ||
| order: attempt.responses.length, | ||
| response: response as DB.Prisma.InputJsonValue, | ||
| correct: correctness, | ||
| thetaBefore: attempt.currentTheta, | ||
| thetaAfter: nextState.theta, | ||
| standardErrorAfter: nextState.standardError, | ||
| elapsedSeconds, | ||
| }, | ||
| }), | ||
| ctx.prisma.adaptiveAssessmentElement.update({ | ||
| where: { id: adaptiveElementId }, | ||
| data: { exposure: { increment: 1 } }, | ||
| }), | ||
| ]) | ||
|
|
||
| const updatedAttempt = await ctx.prisma.adaptiveAssessmentAttempt.findUnique({ | ||
| where: { id: attemptId }, | ||
| include: attemptInclude, | ||
| }) | ||
| if (!updatedAttempt) throw new Error('Adaptive attempt not found.') | ||
|
|
||
| const shouldFinalize = | ||
| allCompetencesStopped(updatedAttempt) || | ||
| !selectNextAdaptiveElement(updatedAttempt) | ||
|
|
||
| if (shouldFinalize) { | ||
| const level = mapThetaToLevel( | ||
| nextState.theta, | ||
| updatedAttempt.assessment.levels, | ||
| { | ||
| min: updatedAttempt.assessment.thetaMin, | ||
| max: updatedAttempt.assessment.thetaMax, | ||
| } | ||
| ) | ||
|
|
||
| await ctx.prisma.adaptiveAssessmentAttempt.update({ | ||
| where: { id: attemptId }, | ||
| data: { | ||
| status: DB.AdaptiveAssessmentAttemptStatus.COMPLETED, | ||
| currentTheta: nextState.theta, | ||
| currentStandardError: nextState.standardError, | ||
| finalTheta: nextState.theta, | ||
| finalStandardError: nextState.standardError, | ||
| finalLevelLabel: level?.label ?? null, | ||
| elapsedSeconds, | ||
| completedAt: new Date(), | ||
| thetaHistory: thetaHistoryForAttempt(updatedAttempt), | ||
| standardErrorHistory: standardErrorHistoryForAttempt(updatedAttempt), | ||
| }, | ||
| }) | ||
| } else { | ||
| await ctx.prisma.adaptiveAssessmentAttempt.update({ | ||
| where: { id: attemptId }, | ||
| data: { | ||
| currentTheta: nextState.theta, | ||
| currentStandardError: nextState.standardError, | ||
| thetaHistory: thetaHistoryForAttempt(updatedAttempt), | ||
| standardErrorHistory: standardErrorHistoryForAttempt(updatedAttempt), | ||
| }, | ||
| }) | ||
| } | ||
|
|
||
| return getAdaptiveAttemptState({ attemptId }, ctx) |
There was a problem hiding this comment.
The response order is set to attempt.responses.length at the time the attempt is read (step 1). Two requests that both read the same attempt state before either commits will both compute the same order value, and the second DB write will fail with a unique-constraint violation on (attemptId, order) rather than a clean error message. In practice, adaptive tests are driven one question at a time so this is unlikely, but a retry from the client or a slow network could trigger it. Using a DB-level nextval sequence or inserting with an ON CONFLICT guard would make the order assignment concurrency-safe.
|
Full review pushed to this branch: Headline items before any student-facing use:
|
| await ctx.prisma.adaptiveAssessmentAttempt.update({ | ||
| where: { id: attemptId }, | ||
| data: { | ||
| status: DB.AdaptiveAssessmentAttemptStatus.COMPLETED, | ||
| currentTheta: nextState.theta, | ||
| currentStandardError: nextState.standardError, | ||
| finalTheta: nextState.theta, | ||
| finalStandardError: nextState.standardError, | ||
| finalLevelLabel: level?.label ?? null, | ||
| elapsedSeconds, | ||
| completedAt: new Date(), | ||
| thetaHistory: thetaHistoryForAttempt(updatedAttempt), | ||
| standardErrorHistory: standardErrorHistoryForAttempt(updatedAttempt), | ||
| }, | ||
| }) | ||
| } else { | ||
| await ctx.prisma.adaptiveAssessmentAttempt.update({ | ||
| where: { id: attemptId }, | ||
| data: { | ||
| currentTheta: nextState.theta, | ||
| currentStandardError: nextState.standardError, | ||
| thetaHistory: thetaHistoryForAttempt(updatedAttempt), | ||
| standardErrorHistory: standardErrorHistoryForAttempt(updatedAttempt), | ||
| }, | ||
| }) |
There was a problem hiding this comment.
Concurrent answer submissions can still write an incorrect attempt state. Each request creates its response, then updates currentTheta and thetaHistory from the response list it loaded before either request committed. If two different answers arrive at the same time, the last update can overwrite the attempt estimate with a history that includes only its own new response, even though both response rows now exist. The attempt state update needs to be serialized with the response insert or retried from the latest response set.
…e-learning review Part 5 compares the implemented item-level 3PL CAT against classification testing (SPRT/CCT), estimator alternatives (MAP/WLE), DIALANG/Linguaskill designs, multistage testing, comparative-judgment item leveling, and exposure control. Verdict: keep the architecture; five concrete roadmap adjustments in section 5.7. All citations retrieved via literature search.
|
Added Part 5 — literature check on approach optimality to TL;DR: the item-level CAT architecture is the right call — keep it. But the literature reframes a few things:
Concrete roadmap deltas in §5.7, full references with DOIs at the end of the file. |
| return ordered.map((level, index) => { | ||
| const start = min + (span * index) / ordered.length | ||
| const end = min + (span * (index + 1)) / ordered.length |
There was a problem hiding this comment.
This still renders level bands with span / ordered.length, while the backend assigns default NEAREST levels with theta points spread over ordered.length - 1 intervals and midpoint boundaries. With four levels over [-3, 3], this UI draws boundaries at -1.5, 0, and 1.5, but mapThetaToLevel assigns levels at -2, 0, and 2. A learner at θ=1.75 is therefore shown inside the top visual band while the backend result label still maps to the previous level, so the displayed band and assigned level can disagree.
Context Used: AGENTS.md (source)
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| levelLabel: string | ||
| } | ||
|
|
||
| async function seedTestkursAdaptiveElements(prisma: Prisma.PrismaClient) { |
| return elements | ||
| } | ||
|
|
||
| async function seedTestkursAdaptiveAssessment( |
| level: Prisma.AdaptiveAssessmentLevel | ||
| } | ||
|
|
||
| async function seedTestkursAdaptiveAssessmentAttempts( |
2 participants
ClickUp Links