UCT ← Back to the explorer's documentation page  ·  source: docs/REPLICATION.md

Replication Log — 2025 Baseline

This document records, in full detail, how the 2025 dataset was produced, so the process can be repeated for future handbook years and audited later.

Baseline statement. The 2025 handbooks represent the initial state of UCT's academic offerings before further credit-load re-think editions are introduced. The git tag baseline-2025 marks the commit that produced this state. All later handbook years are loaded alongside (every table has a year column) and compared against this baseline.

1. Environment

• Windows 11, Python 3.12 (any 3.10+ should work)
• pip install -r requirements.txt (pdfplumber, pypdf, pandas, openpyxl)
• All commands run from the repository root.

2. Inputs

Raw PDFs live in faculty-handbooks-undergraduate/ and are immutable:

Convention for future years: YYYY-<fac>-ug.pdf, YYYY-_fees.pdf. Drop the files in, never rename or edit existing ones.

3. Pipeline (exact commands, in order)

python -m extractors.fees.extract --year 2025
python -m extractors.com.extract --year 2025
python build_main_dataset.py --year 2025
python validation/validate.py --year 2025
python build_final_dataset.py --year 2025      # step 5: final-clean layer
python validation/validate_final.py --year 2025  # step 6: final-layer assertions

Steps 1–4 build and check the as-printed layer (exactly what the handbooks print, defects preserved). Steps 5–6 build and check the final-clean layer (main_dataset_final.csv, ideal_student_summary_final.csv): discrepancies resolved by the ordered rule set R0/R3/R1/R2/R4 with the curated per-faculty registers (resolutions/<fac>.csv) — full method and justification in FINAL-DATASET-METHOD.md. The batch runner executes steps 5–6 for every loaded year after the per-year loop (cross-edition rules make the final layer a whole-dataset computation).

Each step is deterministic and re-runnable; interim page dumps go to data/interim/ (gitignored), outputs to data/processed/ (committed), and exception reports to validation/ (committed). --skip-dump reuses an existing page dump during development.

Step 1 — fees extractor (extractors/fees/extract.py)

Section boundaries are found by full-line heading matches ("11. UCT ACADEMIC FEES", "12. UCT ACADEMIC COURSES", "13. RESIDENCES") — full-line, because the table of contents contains the same text with dotted leaders.

• §12 course fees → course_fees.csv. Row grammar CODE TITLE 10,440. 2025 yield: 4,496 rows, 4,495 distinct codes, zero conflicting fees; 483 exact reprint duplicates dropped; 1 line unparseable (see Hazards H13) and recovered via extractors/fees/overrides.py.
• §11 published programme fees → programme_fees_published.csv. The printed layout interleaves margin notes with fee rows, so a state machine applies these rules: subsection 11.1 (fee ranges) is skipped; a text buffer becomes a programme label only when a "1st Year" row arrives; buffered text before later-year rows is margin noise (kept as margin_note); stream sub-labels without a degree keyword inherit the last degree-bearing label; dotted single-line fees keep study_year empty; year rows arriving with an empty buffer directly after a dotted fee belong to that dotted programme (part-time PG blocks). 2025 yield: 525 rows.

Step 2 — Commerce extractor (extractors/com/extract.py)

Pages are classified by their running-header line ("RULES FOR ADVANCED DIPLOMAS" / "PROGRAMMES OF STUDY" → programme parsing; "DEPARTMENTS IN THE FACULTY..." / "FACULTIES AND DEPARTMENTS..." → course catalogue).

Programme blocks. A specialisation starts at a [PLANCODE] line (or an inline Advanced-Diploma heading). The 1–2 title lines above it are reconstructed by walking back to a line starting "Bachelor of…"/"Advanced Diploma…". Umbrella lines ("Bachelor of Commerce Augmented", "… Extended Academic Development") set the variant for subsequent blocks. Plan codes are normalised (O→0, zero-padding) — see Hazards H1.

Curriculum tables. Inside a block, "(First…Fifth) Year Core Modules" headings open year tables; repeated headings for the same year get table_index 2+ (kept, never ideal). Row grammar and structures handled:

2025 yield: 73 specialisations (33 BBusSc, 34 BCom, 3 AdvDip; 27 regular, 23 augmented, 23 extended), 2,323 curriculum rows, 268 stated totals.

Course catalogue. Entries recognised as CODE TITLE (≥60% uppercase title) confirmed by an NN NQF credits at NQF level N line within 4 lines; labelled fields (Convener / Course entry requirements / Course outline / Lecture times / DP requirements / Assessment) accumulate until the next entry. 2025 yield: 307 courses.

Step 3 — assembly (build_main_dataset.py)

Produces main_dataset.csv — the single source of truth: one row per specialisation × study-year × course-slot carrying degree, credit, course, fee and provenance columns plus the ideal_student boolean.

Ideal-student rules (deterministic):

1. core rows → taken.
2. option rows → taken iff choice_member <= choice_pick_n.
3. elective slots → taken; blank credits inferred as stated_year_total − sum(other ideal rows) when exactly one blank slot exists (credits_inferred=True).
4. alternative rows and table_index > 1 rows → never taken.
5. Fees: exact course-code match first; then suffix-variant fallback (MAM1010→MAM1010F, BUS4050W→BUS4050H), recorded in fee_source; elective slots get median same-level fee × slot_credits/18, flagged estimated_median.

Published-fee matching: labels are parsed to (degree, AD?, normalised specialisation) with an alias table for renames ("Analytics" → "Statistics and Data Sciences", "Finance (Non CA Option)" → "Finance, Investment and Banking", …); postgraduate labels are excluded. Academic-Development fees are published once per specialisation — the block's year-count decides which AD variant it prices (5 year-rows → the 5-year extended BBusSc), method recorded as ad_duration/ad_shared.

Also produces ideal_student_summary.csv: per specialisation-year credits and cost vs the handbook's stated total and the published fee.

Step 4 — validation (validation/validate.py)

Writes credit_check_2025.csv, fee_check_2025.csv, missing_fees_2025.csv.

2025 baseline results:

The MISMATCH rows are findings, not failures: most are the handbook's own arithmetic (totals that count both OR branches, exclude a listed row, or a typo like the 382-for-182 stated total on CB025BUS09 year 1) or final-year tables whose choice rule is printed nowhere (the CSC 4th-year menu). They are preserved verbatim in the dataset and listed in the exception reports for the analysis team; decisions to pin them are made in per-extractor overrides files with written rationale, never by editing outputs.

4. Hazard catalogue (all confirmed in 2025 sources)

5. Adding a new handbook year — batch processing

Drop the year's PDFs (YYYY-<fac>-ug.pdf for each faculty, plus YYYY-_fees.pdf) into faculty-handbooks-undergraduate/, then:

python run_pipeline.py --years all        # or --years 2027, or --years 2021-2027

The runner executes fees → every faculty extractor whose PDF is present → assemble → validate per year, then the final-clean layer for all loaded years, and prints a yield summary. Writers are merge-by-year: re-running a year replaces exactly that year's rows in the processed CSVs and leaves every other year byte-identical (verified for the 2025 baseline on every change). Then:

1. Compare the new year's yields to the table in §6 — a large drop means the layout changed; check the header families first (Hazards H18-H20).
2. New specialisation names/renames → extend SPEC_ALIASES; print defects → the relevant overrides file with reason and source page.
3. Commit the updated CSVs and validation reports; existing years' rows must show no diff.

6. Commerce yield reference (per-edition, as-printed layer)

Per-edition COM yields for regression comparison when parsers change: 73/74/75/75/73/71 specialisations and 2,434/2,468/2,491/2,318/2,323/2,248 curriculum rows for 2021→2026. Current cross-faculty status lives in §8. The credit re-think is visible directly, e.g. CB019BUS01 (BCom Actuarial Science) year 1: 185 credits (2021-2023) → 180 credits (2024-2026), with computed ideal fees equal to the published fee to the rand in 2021-2025.

Layout drift encountered and handled across editions:

Failing to recognise a heading is the costliest hazard class: the previous block silently swallows the next programme's tables (doubling its row sums) — exactly what detector R2b in the final layer flags as check_extraction. H22–H24, H26 and the EB015CON04 case were all found this way; an exactly-2× row-sum is the diagnostic signature.

Variant assignment precedence (robust across all layouts): page-header hint (2026) → known programme-code family map (VARIANT_BY_PROGCODE) → umbrella- line tracking (2021-2025).

7. The shared engine and the EBE faculty (added 2026-08-06)

The COM parser was promoted to a faculty-configurable engine, common/handbook_parser.py, once EBE proved to use the same publisher template. Each faculty supplies a FacultyConfig (extractors/<fac>/extract.py): plan-code grammar, degree parser, page classification, heading grammar, variant resolution, department map, and optional extras. The promotion was verified value-identical for all six COM editions (curriculum and courses byte-equal; one deliberate improvement — range-total support now captures a 2026 AdvDip total that previously sat in notes).

EBE config deltas (stable across 2021-2026 — EBE shows no cross-edition layout drift, unlike COM):

• Plan codes EB###XXX##; the 800-series are the 5-year Extended Curriculum Programmes → variant extended.
• Year headings say "Core Courses" (with H26/H27 tolerances).
• Stated totals are often ranges ("108-156") because elective loads are ranges ("Approved elective courses … 0-48"): the minimum anchors the ideal student (is_minimum=True), the maximum is kept in stated_total_max.
• Every programme is followed by an "ELECTIVE COURSES" pool section → requirement=alternative.
• Specialisation sub-streams (Geomatics Geoinformatics: Computer Science vs EGS) share one plan code — the second block is DUPLICATE-flagged and its rows suppressed (first-listed stream is the ideal), as is the transferee access programme that reuses EB001CHE01/EB002CIV01 (see DEV-TODO).
• The Electrical/Mechatronics/Mechanical families print in-year elective menus under "(Ordinal) Year (Further) Elective Core Courses (EE/EC/ME)" headings, plus "Optional Courses" sections (H38). Menu rows are alternative; the printed instruction decides the ideal load — "Select courses amounting to at least 48 credits from the following:" becomes a 48-credit minimum elective slot, "Select two out of the following three courses." a pick-2 menu. "(minimum)"-qualified totals ("Total credits per year (minimum) … 138") are captured with is_minimum. 2025/2026 add footnote markers to year headings ("First Year Core Courses (EE)*†", H37) — unrecognised, they silently cost whole year-tables. Slot lines come in five shapes (H39): "Approved elective courses … 0-48", bare "Electives … 18" (2021 Property Studies), "Elective … 18 5" (Geomatics), "F/S/P/L *Approved Complementary Studies Elective F/S/P/L … 18 7", and the Mechanical year-4 pair with level-range tails ("… 18 5-8", "**Approved F and S Open electives …Totalling at least … 24 5-8"). These fixes moved EBE from ~62% to 94% consistent spec-years.
• Published-fee labels ("BSc Eng (Chemical)") are matched by an EBE-specific label parser; ECP variants are not published separately — duration matching assigns each block to the right variant.

Tables shared by several faculties are written merge-by-faculty-within- year (write_year_rows(..., keep=...)): an extractor re-run replaces only its own faculty's rows.

LAW config deltas (stable across 2021-2026 except H31 in 2026):

• Undergraduate content is the "RULES FOR LLB DEGREE STREAMS" section: three streams with 5-character programme codes and no department suffix — [LP001] graduate LLB, [LB002] four-year undergraduate LLB, [LB003] legacy five-year stream (no new intake after 2019 → variant extended; absent from 2026).
• Year headings: "First Year YEAR 1 (PRELIMINARY LEVEL)" (LB003 uses COM-style "Core Modules"); totals are per level with THREE wordings — H29's two plus LB003's bare "Total credits for first year … 90" (H40; the earlier belief that LB003 "prints no credit totals" was wrong — the wording was simply unmatched). LB003 also wraps discontinued-course rows before their credits ("PVL1006W … (No longer on" / "offer after 2019) … 36 5") — recovered by course_row_nocred + course_cred_cont (H40); with both fixes LB003 reconciles (its printed years sum to the printed 660 stream total; the 2024/2025 editions drop discontinued rows and are flagged, correctly, as findings).
• Cross-faculty requirement lines ("AND two semester courses in another faculty … 36 5") are captured as elective slots via the LAW extra_elective grammar; wrapped descriptions merge with their continuation line (H30).
• The many bracketed postgraduate codes (LM…/LG002…) sit in postgraduate sections excluded by page classification.
• Published fees (fees book §11.6) are flat annual amounts per stream ("Undergraduate LLB … R 76 810") — applied to every study year with match method flat_annual; per-year divergence from the flat figure is expected and is not a data defect.

FHS (bespoke parser, extractors/fhs/extract.py). FHS does not fit the block engine: programme blocks are bracket lines carrying one or more MB codes with shared curricula (H32), totals trail their tables with several wordings and slashed variant values (H33), the MBChB runs SIX years with rule-prefixed headings ("FBA3.8 Sixth Year"), and 2022-2023 use per-degree running headers while 2021-2023 print multi-line brackets (H34). The parser reuses the shared grammar (course rows, OR handling) and the engine's catalogue parser/classifier. Conventions: primary code carries the curriculum; second code of a pair is the intervention/extended variant; known residual — the combined Audiology/Speech-Language block interleaves both degrees' sub-tables before shared totals and needs a dedicated splitter (its spec-years are flagged unresolved; see DEV-TODO.md).

8. Multi-year status (all six faculties, final layer 2026-08-06,

after the H37-H40 engine fixes)

Consistent / resolved / unresolved per faculty (SCI and HUM spec-years are majors and carry no_anchor — no per-year totals exist for them, §9):

Main dataset: 23,592 rows (COM 14,282 + EBE 4,729 + SCI 1,571 + HUM 1,372 + FHS 1,187 + LAW 451); 3,287 specialisation-years (1,949 consistent, 80 resolved, 337 unresolved pending adjudication, 921 no_anchor major-years). The H37-H40 fixes resolved what DEV-TODO had queued as adjudication clusters: EBE went from 344/185 consistent/unresolved to 509/31, LAW from 42/25 to 60/7 (the remaining LAW 7 are LB003 2024/2025 years whose printed tables drop discontinued courses — genuine print drift, listed in the pending reports). Median computed-vs-published fee delta is 0.0% for COM, EBE and FHS (the MBChB's years 1-3 match to the rand); LAW's flat-annual and SCI/HUM's degree-flat published fees make per-year/per-major deltas structurally divergent (LP001 year 1 matches to the rand).

9. SCI and HUM — majors as the curriculum unit (added 2026-08-06)

Science and Humanities complete the six-faculty load, on bespoke parsers in the FHS mould (extractors/sci/, extractors/hum/). Their unit of curriculum is the major, not the specialisation:

• SCI: ~22 "Major in X" blocks with [MAM01]-style stream brackets in the "DEGREES OFFERED IN THE FACULTY" section; plan codes synthesised as SB001 + stream (SB001MAM01). "Either / Or both" alternatives and "And two of …" menus; credits printed inline, wrapped rows completed from the book's own catalogue. The postgraduate rules and curricula print under the SAME running header, so the majors parser stops at the BSc Honours rules heading (H41).
• HUM: ~40 "Requirements for a major in X" blocks with [AFS01]-style brackets printed INSIDE the department sections; plan codes HB001 + stream. Rows are bare "CODE Title" lines — credits and NQF levels are joined from the catalogue. Majors serve both the BA and the BSocSc.

Majors print no per-year credit totals — by design. The credit anchors for these faculties are degree-level (the rules layer, §10): SCI FB7.1 (≥ 360 NQF credits from 2025; "nine full-year courses" before), HUM's award minima (20 semester courses / 10 senior / 2 majors / 3 years, extended programme 4 years). The finaliser therefore gives SCI/HUM spec-years the status no_anchor instead of unresolved — there is nothing to adjudicate against per year — and keeps them out of the pending reports. Published fees are per degree, not per major: the fees book's "Bachelor of Science" block prices every SCI major and "Bachelor of Arts and Bachelor of Social Science" every HUM major (match method degree_flat, like LAW's flat_annual); the specialised Humanities programmes' own blocks (Fine Art, BMus, BSW, PPE, Film & Media) have no major rows to price and surface in the unmatched report by design.

10. The rules layer (degree_rules.csv, added 2026-08-06)

The faculty-rules sections — never previously parsed — print degree-level facts the curriculum tables cannot supply: minimum total credits for the degree, level-specific credit requirements, durations, and (LAW) whole-stream grand totals. common/degree_rules.py extracts them into data/processed/degree_rules.csv (one row per printed rule statement, with source_page and a verbatim quote; merged by year and faculty like every shared table). Each faculty extractor runs it after its main parse.

Why it matters — the credit re-think is written into the rules:

EBE's blanket rule FB3.2 (4-year degrees ≥ 576 / 3-year ≥ 432) drops to 560 in 2026 — after the departments moved first (Electrical prints cohort-split minima from 2025, "registers in 2025 … 560 / registered before 2025 … 576"; Mechanical follows in 2026; Civil prints "576 (or 560 if admitted … from 2025)"; Geomatics phases in 519/511 totals in 2026). LAW prints stream grand totals every edition: graduate LLB 504 (flat), 4-year undergraduate LLB 660 → 637 in 2026, legacy LB003 660. FHS professional degrees are duration-ruled, not credit-ruled (the one credit rule is FBC3.1, intercalated BSc(Med) ≥ 360).

SCI re-based its composition rules at the 2025 edition — from full-year-course counts to credits: FB7.1 "at least nine full-year courses" (2021-2024) becomes "at least 360 NQF credits of which at least 180 must be Science credits" (2025-2026); FB7.2 "four full-year senior courses" becomes "120 credits at level 7". The majors rule renumbered FB7.6 → FB7.5 at the same moment (H35 again). HUM's award minima are stable across all six editions: BA/BSocSc = 3 years, 20 semester courses, 10 senior, 2 majors (≥ 1 from Humanities departments, FB3); the extended programme (HB061/HB062) the same over 4 years; specialised programmes (Fine Art) print "a minimum of 27 semester courses". Because SCI (pre-2025) and HUM count in courses, degree_rules.csv carries min_total_courses / min_senior_courses / min_majors with a course_unit column ("full-year courses" vs "semester courses") — the numbers are recorded as printed, never converted.

Notable negative finding: the rules sections do not print the missing selection rules for the COM unlabelled choice menus (CSC 4th year, PPE year 2, senior Economics) — but sibling editions' Programmes of Study do ("required to take two options", 2022/2024/2026), which is now cited as evidence in the affected resolutions/com.csv entries.

Validation. validate_final.py step 5 reconciles each specialisation's whole-degree credit sum against the applicable rule (validation/degree_check_<year>.csv; statuses OK / BELOW_MIN / ELECTIVE_GAP / NO_RULE — see the docstring). It also checks the EBE 5-year-ECP invariant (EB8xx totals must equal their EB0xx twin's). Results are findings, never failures. Highlights of the first run: LP001 sums to its printed 504 exactly in all six editions; the 2026 BAS curriculum (376) sits 56 credits below FB3.2's 3-year minimum, and the 2026 Property Studies stated minimum (452) exceeds its own table sums (411) — both genuine handbook inconsistencies now surfaced mechanically.

Parsing contract (hazards H35/H36): key on the degree-heading text, never the rule code; whitespace-collapse before matching; expect HEQSF wording in 2021 and cohort-split sentences from 2025.