Add cohort-specific study plans, prerequisite extraction and two new programmes - #61
Merged
Conversation
…programmes
A student admitted in HT2023 can now pick their cohort and see years 1-3 as they
will study them, with prerequisites resolved to the kursplan that applied to
their round. CMAST and CMATD are added from scratch using the same pipeline.
Previously the extractor assembled one calendar läsår from three different
cohorts, which is the wrong thing for a student reading their own plan.
WHY A COHORT'S PLAN HAS TO BE STITCHED TOGETHER
KTH publishes the läsår currently being taught and the next one, deletes the
years a cohort has already passed, and lists its future years with all-zero
creditsPerPeriod:
cohort year 1 year 2 year 3
HT2023 gone gone own
HT2024 gone own own
HT2025 own own future
HT2026 own future future
No cohort has all three of its own years, and the gap runs in both directions.
Each missing year is borrowed from the nearest cohort that publishes it,
preferring the earlier one on a tie. Availability is probed rather than computed,
because the publishing window moves every year.
Each cohort file opens with a cohortMeta entry recording, per year, the
sourceCohort, whether it was approximated, and a confidence derived by comparing
a year both cohorts publish. Agreement is evidence, not proof: CTFYS year 1 is
identical between 2025 and 2026 while its year 3 changed between 2023 and 2024.
The UI shows a one-line summary under the selectors with the detail behind an
info affordance, which links to KTH's published plan as the authority.
KOPPS IS RETIRED - THE COURSE PAGE IS THE LIVE SOURCE
The KOPPS API still answers but no longer receives updates. DD1328's course page
carries two kursplan versions (VT2026 and VT2024); the API returns only VT2024.
The newer text lists DD1333, CTMAT's own first-year programming course, so
reading KOPPS left DD1328/DD1380/DD1385 with no prerequisites at all. Measured
over all 217 course codes: 166 identical, 51 different, and the page is the
fuller text every time. The page's render state is now primary for
prerequisites, grading scale, cycle level and examination modules. Only the
English title still comes from KOPPS, because the English course page returns
HTTP 500.
KURSPLAN VERSIONS ARE PER-COHORT
Prerequisites change, so a cohort must get the version that applied to its round.
The course page's syllabusList carries every version tagged with
course_valid_from, and that value is the KTH term code - verified against the
kursutveckling archive, where EI1320's five PDFs match the five versions exactly.
The PDFs therefore never need parsing. CTFYS HT2023 sits EI1320 in 20252 and gets
the 20212 kursplan (SI1200 alone); HT2024 sits it in 20262 and gets 20261
(SI1200 or SF1693). 23 CTFYS courses have more than one version.
PREREQUISITES COME FROM FREE TEXT
They live in the syllabus eligibility field as prose, while the schema wants
prerequisitesCompleted and prerequisitesParticipation. Three things make the
interpretation tractable: intersecting with the programme's own courses collapses
alternative lists and keeps cross-programme codes out of the arrows; type is
decided per clause; and a clause with no type marker inherits one signalled
elsewhere in the same text. Ranges are expanded - KOPPS writes alternatives as
DD1310-DD1319 as well as slash lists, and expanding CINEK's DD1418 recovers
DD1317, DD1324 and SF1918. Requirements are split on capitalised openers, since
EI1320's syllabus runs two of them together with no punctuation between.
Measured against the 19 hand-curated CTFYS prerequisites: 19/19 exact, none
missed, none spurious. Two rules were derived from that set, and CTFYS is the
only programme with curated prerequisites, so treat it as a fit to the available
labels rather than proof it generalises. Uncertain items go to
prerequisite-review/<PROG>.md; two of its sections are aimed at the program
director rather than at data entry - a course whose own list of qualifying
courses omits ours (report it to that course's owner), and a prerequisite that
does not precede the course.
FULL-TIME LOAD AS A DATA-QUALITY CHECK
Full-time study is 15 hp per period, so a year should add up to 15 in every
period. validate-data now warns on any cell that does not, merging inriktning
lanes that say the same thing. Short is almost always the missing space for
valfria kurser; over means double-counted option groups or a "minst N hp ur
grupp" pool the schema cannot express. CTFYS produces no warnings at all, which
is the calibration check.
Placeholders are filled from the shortfall and corroborated by the prose: CTMAT
says "Utrymmet för valfria kurser är 7,5 hp per period hela läsåret" and CTFYS
"På våren i årskurs 3 finns ett utrymme på 15,0 hp valfria kurser" - both agree
with the computed gap, and the CTFYS case reproduces the hand-authored
XY123Z/XY456Z placeholders exactly. Filling is refused where a year also has a
period over full-time; CFATE year 3 gets its arithmetic spelled out instead.
TWO RENDERER FIXES
Chart height was a feedback loop: it was seeded from the height its own previous
render had written onto the SVG node, then only ever grown. Switching from a tall
programme to a short one stranded the legend far below the chart - measured CTFYS
659 -> TIEMM 2994 -> CTFYS 3882 against 659 on a fresh load. It now derives from
the data and is set unconditionally.
Course order controls vertical alignment, because each period's bars stack in
file order. alignEntries() minimises total drift with a multi-start local search:
every simple comparator tested was worse than the hand-curated order, so it seeds
from the file's own order plus two alternatives and keeps the best, which means it
can never degrade a file. CTFYS 15->6, CTMAT 26->5, CFATE 50->15, CINEK 118->9,
CMAST 316->5, CMATD 44->3.
TWO NEW PROGRAMMES, AND WHAT THEY REVEALED
CMAST and CMATD were added with no hand-curated data, pointing dataFile at their
HT2025 cohort file. Both needed the specializations registry that no SCI
programme does: structure follows the owning school (ITM programmes split by
inriktning from year 2, SCI ones do not), while prerequisite wording does not,
because the text belongs to the course rather than the programme.
CMAST also introduced electiveCondition 'R' (rekommenderad), used by no other
programme - 144 participations, one per course recommended for the master track
an inriktning leads to. The KNOWN_CONDITIONS assertion caught them rather than
dropping them silently. They are a pool, not a prescribed set (adding them takes
AEE year 3 from 15/15/0/0 to 15/33/30/6 hp), so they get the same treatment as
'V': reported, written only with --electives.
ALSO
- TIEMM is withdrawn from the UI via a new `disabled` flag. `verified: false`
only hides a plan behind a checkbox, which is right for "extracted but not
signed off"; a master's programme whose years 4-5 structure the renderer
does not yet handle needs to be off entirely. Its data is still validated.
- Palette convention: Matematik is blue and elective space yellow in every
programme; the rest take green, brick and turquoise. Yellow previously meant
Datateknik in CTMAT and CFATE and Ingenjörsämnen in COPEN, colliding with the
elective boxes. Placeholders are coloured by category rather than by a
cosmetics entry, since their codes are generated.
- CFATE recategorisation: SI1146 to Matematik, SD1002/MJ1401/ME1003 to
Ingenjörsämnen. All four were previously in no group and rendering default.
- --align refuses files that are not course data. programs.json, the cosmetics
files and academic-periods.json are all JSON arrays too, so a shell glob
could rewrite them - harmlessly in content, but expanding their compact
arrays into a 400-line diff.
- STUDY-PLAN-CONVENTIONS.md records how study plans differ between schools,
for discussion with the people who own them.
- The size-limit budget goes 250 -> 330 kB. The committed cohort archive counts
toward it and production chunk names are hashed, so app and data cannot be
split by path; package.json and ci.yml record the composition.
Verified: all eight programmes extract four cohorts each and validate with 0
errors; lint, tsc --noEmit and build clean; size 270.63 kB of 330 kB. The UI was
driven in real Chrome across every programme - selector, cohort selector, URL
state, chart, height behaviour on programme switch, provenance tooltip and the
withdrawal of TIEMM all check out with no console errors.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL flagged one double-unescaping bug and three time-of-check/time-of-use file reads on PR #61. All four are in scripts/extract-from-kopps.mjs. The extractor's output is unchanged: regenerating CTFYS, CTMAT and CMAST from the live source gives files byte-identical to the ones already committed. DOUBLE UNESCAPING (js/double-escaping) decodeHtmlText chained six .replace() calls and then ran the whole chain a second time, so one replacement's output could be re-read by the next. Unescaping "&" to "&" ahead of the """ step turned "&quot;" into a double quote rather than the literal """ it encodes, and the second pass did the same thing to anything the first pass produced. The comment claimed KTH input is "sometimes double-encoded", so the second pass was measured rather than assumed. Over 87 pages - all eight programmes, three terms, fifteen course pages, 146819 string values - the entities that occur are (788), ä (25), & (25), å (23), " (18), ö (13), ' (12) and   (5). None is double-encoded. The second pass never fired; it only created the hazard. Now one regex over one pass, so `replace` cannot rescan what it has written. Checked against the old implementation on all 146819 measured strings: zero differences. The two behave differently only on input the source does not produce, and there the new one is right - "&#39;" decodes to the literal "'" instead of an apostrophe, and "😀" to the intended character rather than a truncated one, because String.fromCodePoint replaces String.fromCharCode. A code point outside Unicode's range is left as written instead of crashing. < and > are still deliberately left encoded. The tag stripper runs afterwards, so decoding them would hand it angle brackets that were never markup and delete the text between. Neither entity occurs in the measured data. TIME-OF-CHECK/TIME-OF-USE (js/file-system-race, three instances) alignFiles, fillPrereqs and fillElectivesInCuratedFile each called existsSync and then read the file. Replaced with a readTextOrNull helper that reads and handles ENOENT, so there is no gap between the check and the use. An existence check also passes for a directory, a broken symlink and a file the process cannot read, all of which fail at the read a moment later, so this is more accurate as well as shorter. lasarFromPeriods and writeCohortIndex had the same shape without being flagged - the first because its read sits inside a try, the second because it is a readdir - and are converted too, which retires the existsSync import. Error messages and exit behaviour are unchanged: a missing --align target still warns and continues, a missing curated file still throws the same text. No app code is touched. The script is not part of the Next build graph; it is reachable only through `npm run extract-plan`. Verified: lint, tsc --noEmit and validate-data --cohorts all clean (191 warnings, unchanged). Both --align and --fill-electives exercised, including their missing-file paths. CTFYS, CTMAT and CMAST re-extracted for all four cohorts against the live source: output byte-identical to the committed files, and identical to what the pre-patch script produces from the same input.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A student picks their admission year and sees the plan as they will study it — years 1–3, with
prerequisites resolved to the kursplan version that applied to their round. CMAST and CMATD are added
from scratch on the same pipeline, taking the app from 6 programmes to 8 (7 offered; TIEMM is
withdrawn, see below).
Previously the extractor assembled one calendar läsår from three different cohorts — right for a
snapshot of a teaching year, wrong for a student reading their own plan.
Why a cohort's plan has to be stitched together
KTH publishes the läsår currently being taught and the next one, deletes the years a cohort has
already passed, and lists its future years with all-zero
creditsPerPeriod:No cohort has all three of its own years, and the gap runs in both directions. Worth flagging
because it's the opposite of the obvious assumption: the problem is not only "next year isn't
scheduled yet", it's also "your first year has been deleted".
Each missing year is borrowed from the nearest cohort that publishes it, preferring the earlier one on
a tie. Availability is probed, not computed — the publishing window moves every year, and a probe
is self-correcting where a formula goes stale.
Provenance is recorded, and shown
Each cohort file opens with a
cohortMetaentry giving per year:sourceCohort,approximated, anda
confidenceofexact/high/low/unknown. Confidence comes from comparing a differentyear both cohorts publish — agreement is evidence they follow the same layout, not proof: CTFYS year 1
is identical between 2025 and 2026 while its year 3 changed between 2023 and 2024. The UI keeps that
distinction:
Naming KTH's published plan as the authority is deliberate — everything borrowed is a best guess, and
the student should be one click from the real thing.
KOPPS is retired; the course page is the live source
This started as "why do the DD courses in CTMAT get no prerequisites?" and ended somewhere more
useful. The KOPPS API still answers but no longer receives updates:
DD1333is CTMAT's own first-year programming course, so reading KOPPS left DD1328, DD1380 andDD1385 with no prerequisites at all. Measured over all 217 course codes: 166 identical, 51
different, and the page is the fuller text every time — EI1320's API text is
"Slutförd kurs motsvarande SI1200"while the page has"…SI1200 eller SF1693", dropping analternative entirely.
The course page's render state is now primary for prerequisites, grading scale, cycle level and
examination modules. The one field only KOPPS has is the English title — the English course page
returns HTTP 500, exactly like the English study-plan route — so
nameEninherits KOPPS's stalenessand its typos (KD1000 is "Chemical Principles for Sustainabillty" there).
Kursplan versions are per-cohort, and the PDFs turned out unnecessary
Prerequisites change, so a cohort must get the version that applied to its round. EI1320:
Read off the five PDFs in the kursutveckling archive — but the page's
syllabusListholds exactlythose five versions, each tagged with
course_valid_from, and that value is the KTH term code.So the PDFs never need parsing; the structured history was already in the state we were fetching.
23 CTFYS courses have more than one version, 15 in CTMAT, 100 in TIEMM. Any cohort resolving to a
non-newest version says so in the review file.
Prerequisites from free text
The eligibility field is prose; the schema wants two typed lists. Three things make it tractable:
intersecting with the programme's own courses collapses the alternative lists (of
DD1310/…/DD1331only DD1331 is in CTFYS) and keeps cross-programme codes out of the arrows; type isdecided per clause, so one text can yield both kinds; and a clause with no type marker inherits one
signalled elsewhere in the same text.
Two parsing bugs worth calling out, both found by review rather than by the tests:
DD1310-DD1319as well as slash lists,and a plain code scan sees only the endpoints. 43 range expressions across the programmes;
expanding CINEK's
DD1418recoversDD1317,DD1324andSF1918.with no period between. Split on sentence boundaries alone, SI1200 and SK1104 — both required —
were reported as alternatives, telling a coordinator to "decide which applies". Splitting also on
capitalised requirement openers fixes it, and surfaced a category that had been dropped silently:
requirements on a single examination module ("slutfört moment LAB1 i SH1017").
Measured against the 19 hand-curated CTFYS prerequisites: 19/19 exact, 0 missed, 0 spurious. Two
of the rules came from that set, and CTFYS is the only programme with curated prerequisites, so read
that as a fit to the only labels available rather than proof it generalises.
Uncertain items go to
prerequisite-review/<PROGRAM>.md. Two sections are aimed at the programdirector, not at data entry:
knowledge area this programme teaches while its list of qualifying courses omits ours. The action is
to report it to that course's owner, not patch it locally. DD1385 and DD1380 are the live cases,
neither revised since HT2021; DD1328 needed the same and was fixed in its 2026 revision.
slutfördmust finish before the course starts,deltagandemay overlap (CTFYS's SK1104 spansP2–P3 and legitimately participates in SF1674 in P3, so it isn't flagged).
Full-time load as a data-quality check
Full-time study is 15 hp per period, so a year should add up to 15 in every period.
validate-datanow warns on any cell that doesn't, merging inriktning lanes that say the same thing (CMAST has 17
lanes and produced 34 identical warnings before).
CTFYS produces no warnings at all — the calibration check: quiet on the programme that was already
right. Only excesses ≥3 hp are reported, because CTFYS year 1 P1 is legitimately 16.5 (DD1301 is an
optional 1.5 hp course).
Filling the gap, corroborated by the prose
The CTFYS case reproduces the hand-authored
XY123Z/XY456Zplaceholders exactly, which is thestrongest evidence the rule is right. Filling is refused where a year also has a period over
full-time — an excess means the model is incomplete, and placeholders on top would pile invention on a
wrong base. CFATE year 3 gets its arithmetic spelled out instead:
Two renderer fixes
Chart height was a feedback loop. It was seeded from
svgRef.current.clientHeight— the heightits own previous render had written onto that node — then only ever grown:
Not merely "failed to shrink" — it compounded. Now derived from the data and set unconditionally.
Course order controls vertical alignment, because each period's bars stack in file order, so a
course spanning periods sits at whatever cumulative height precedes it in each period. I measured
before choosing an approach, since a drag-to-reorder edit mode was the alternative:
Every simple sort was worse than the hand-curated order — the hand ordering encodes alignment no
comparator can see, so sorting would have destroyed information. The search seeds from the file's own
order plus two alternatives and keeps the best, so it can never degrade a file, and it's deterministic.
CTFYS 15→6, CTMAT 26→5, CFATE 50→15, CINEK 118→9, CMAST 316→5, CMATD 44→3.
A drag UI was rejected because the cohort archive regenerates: hand-placed order would be erased on
every re-extraction unless stored separately, which is 30+ files to maintain by hand against an
automatic ~90 % reduction that survives regeneration.
Two new programmes, and what they revealed
CMAST and CMATD were added with no hand-curated data, pointing
dataFileat their HT2025 cohort filerather than duplicating it into a curated one.
Structure follows the owning school; wording does not. The SCI programmes (CTFYS, CTMAT, CFATE,
COPEN) return a single curriculum with no inriktningar in years 1–3. The ITM ones split from year 2:
CINEK 4, CMATD 6, CMAST 3 in year 2 and 14 in year 3. So both new programmes needed the
specializationsregistry that no SCI programme does.Prerequisite wording doesn't split that way, because the text belongs to the course — 23 courses
appear in both an SCI and an ITM programme with identical text. The variation tracks department and
cycle level: TIEMM is the outlier on every measure (80 % "motsvarande" against 15–29 % for the
bachelor programmes) because it's a master's programme drawing 28 DD courses.
CMAST introduced
electiveCondition: R(rekommenderad) — 144 participations, used by no otherprogramme, one per course recommended for the master track an inriktning leads to. The
KNOWN_CONDITIONSassertion caught them rather than dropping them silently, which is what that checkexists for. They're a pool, not a prescribed set — adding them takes AEE year 3 from
15/15/0/0to15/33/30/6hp — so they get the same treatment asV.This also fixed a bug of mine:
--fill-electiveswas asking KOPPS whether a programme hasinriktningar, and KOPPS returns the master choices for years 4–5 as specialisations (46 for CMAST, 24
for CMATD, dozens for CTFYS). It would have refused to fill elective space in every civilingenjör
programme; CTMAT only escaped because it happens to have zero. Now keyed off the data.
TIEMM withdrawn
A new
disabled: trueflag takes a programme out of the UI entirely — absent from the dropdown andnot reachable via
?program=.verified: falseonly hides a plan behind the "show unverified"checkbox, which is the right level for "extracted but not signed off"; TIEMM needs more than that,
because it's a master's programme whose years 4–5 structure the bachelor-oriented renderer doesn't yet
handle. Its data is still validated — the point is to stop showing it, not to stop checking it.
Master's programmes want a dedicated pass.
Also in here
green, brick and turquoise. Yellow previously meant Datateknik in CTMAT and CFATE and Ingenjörsämnen
in COPEN, colliding with the elective boxes. Placeholders are coloured by category rather than by a
cosmetics entry, because their codes are generated (
XY{year}{period}0Z) and a cosmetics list wouldneed re-syncing on every regeneration. CINEK and TIEMM still have Matematik = green with blue held by
Industriell ekonomi — both at the five-family cap, so aligning them is a swap, left out of scope.
were in no group and rendering default blue.
--alignrefuses non-course files.programs.json, the cosmetics files andacademic-periods.jsonare all JSON arrays too, so a shell glob could rewrite them — harmlessly incontent, but expanding their compact arrays into a 400-line diff. It now requires a positive signal.
STUDY-PLAN-CONVENTIONS.md— a new document recording how study plans differ between schools,written for discussion with the people who own them rather than as a spec.
README.mdcovers the cohort view, the archive, the two visibility flags, the extractorand the new scripts.
CLAUDE.mdcarries the durable detail, including a step-by-step for adding aprogramme written from doing these two.
The size budget goes 250 → 330 kB
The committed cohort archive counts toward
size-limit, and production chunk names are hashed, so appand data can't be separated by path — I checked for a structural split and there isn't one. Raised
deliberately, with the composition recorded in
package.jsonandci.yml(~201 kB app, ~50 kBarchive, ~2 kB per cohort file) so a future failure can be attributed rather than guessed at. Current:
270.63 kB of 330 kB.
npm run validate-datanow passes--cohorts, so CI validates the archive too.Test plan
pre-existing cosmetics ones, now counted per cohort file, plus the new full-time load reports)
npm run lint,npx tsc --noEmit,npm run buildcleannpm run size270.63 kB of 330 kBchart, provenance line and tooltip all render with no console errors
?program=TIEMMfalls back to CTFYS--alignverified lossless on the curated files: identical entry multiset, identical fieldvalues, trailing-newline convention preserved, so the diff is purely moved lines
--prereqsverified additive: CTFYS's 19 curated prerequisites survive every regenerationKnown gaps
across the programmes are the honest measure of what still needs a human. TIEMM has the most and is
where the interpretation is thinnest.
examsfor multi-period courses is a convention, not a fact. One exam per exam-bearing module inthe highest-credit periods, 10/17 against curated values; every placement is flagged. Closing it
needs real timetable data — Ladok's aktivitetstillfällen, which the sibling
academic-performance-portalalready imports.shortfall across every lane, so almost certainly common-part elective space. Not filled, because
fillElectiveSpacerefuses whenever a programme has inriktningar and sums all lanes together ratherthan per-lane. Making it lane-aware would handle uniform cases like this while still refusing
genuinely mixed years.
teacheranddescriptionare never populated, and cosmetics files still need a human.