Pick a question family first, then a variant within it

[sidprasad]

Why

English-to-LTL questions were structurally rare, and each of their three framing arms rarer still.

Selection treated tracesatisfaction_mc, tracesatisfaction_yn and englishtoltl as three peers and split the probability mass between them, so the trace-reading skill got two shares to english-to-LTL's one. Worse in the tail: because each type carried its own QUESTION_TYPE_FLOOR = 0.15, the trace family had an effective floor of 0.30 and a ceiling of 0.85, while english-to-LTL was floored at 0.15.

Measured weights before this change:

student history	trace family	english-to-LTL	per framing arm
cold start	66.7%	33.3%	11.1%
equally good at all three (60%)	66.7%	33.3%	11.1%
good at english-to-LTL, weak on traces	85.0%	15.0%	5.0%

That last row is the complaint: one deontic question in twenty.

What changed

There are really two skills, each with presentation variants — reading a trace against a formula (mc, yn), and formalizing English into LTL (abstract, lights, abac). Selection is now hierarchical over the two.

• QUESTION_FAMILIES declares the grouping; family_of() maps a type to its family.
• calculate_question_family_weights() scores a family on its pooled record, with the same Laplace-smoothed error rate as before and a QUESTION_FAMILY_FLOOR = 0.3 (adaptation swings between 30/70 either way).
• calculate_question_type_weights() splits each family's weight evenly across its subtypes. It still returns a distribution over the three types summing to 1, so the profile page, the JSON export and experiments/policy_comparison.py are unchanged in shape.
• build_exercise reserves its first slots for the highest-scoring question of each family rather than of each type. num_questions is drawn from [3, 8], so at the low end one slot per type was the entire exercise: a 3-question exercise was always exactly one of each and the weights chose nothing at all. Both skills still always appear.

Subtypes are deliberately not drilled against each other. A yes/no trace question is guessable at 50% and a six-option multiple-choice one at ~17%, so their raw error rates were never comparable, and both read a trace against a formula either way.

Result

Cold start goes from 33/33/33 to 50% english-to-LTL / 25% each trace type. A student who has mastered english-to-LTL still sees it 30% of the time rather than 15%, so each framing arm is floored at 10% rather than 5%.

Measured end to end over 40 generated exercises against real SPOT (same seed, same literal/complexity draws as generate_exercise):

	before	after
english-to-LTL share of served questions	39.9%	52.1%
exercises containing no english-to-LTL	3/40	0/40

(Served shares exceed the nominal weights because the english-to-LTL builder yields ~98% of drawn candidates against ~80% for the trace builders — the trace builders return None when no candidate misconception is available.)

Reviewer notes — experiment impact

The three english-to-LTL framings stay uniformly assigned per question; they are randomized experiment arms and nothing about their assignment changed. They simply accrue faster (~17% of questions each at cold start, floored at 10%).

I checked whether the themed-arm fallback was skewing the mix — themed arms fall back to abstract when a formula cannot be themed — and measured 0/360 fallbacks for both lights and abac, so assignment was already uniform in practice. The per-arm ratio is untouched; only per-student exposure counts shift, so analyses spanning this release should segment on it.

Testing

Full suite passes (344 tests). test/test_question_type_weights.py is reworked to the new semantics: a TestQuestionFamilyWeights class covering the pooled scoring, the family floor, and the specific regression (equally wrong on all three types is now a ~50/50 family split, not 2/3 trace), plus type-level tests asserting the marginals agree with the family weights and that the trace subtypes stay even however they are answered.

Version bumped to 2.1.9 with CHANGELOG entries.

---

Follow-ups in this PR

Exercise length is now 4–6 questions, down from 3–8. An exercise is a sitting, and a range that wide made its length unknowable — two in a row could differ by more than a factor of two.

The profile page's "Question Mix" card now shows the mix by skill. The family split left the old per-type card in an odd state: the two trace-satisfaction bars are equal at every point in every student's history, so three bars where two never differ read as a rendering bug. Its caption ("Types you miss more are drilled more") had also become false — missing yes/no questions raises both trace bars equally rather than drilling yes/no. It now reads "Skills you miss more are drilled more" over two bars. The per-type weights are still what a question is drawn with, so they stay in the JSON export, which gains question_family_weights alongside them and goes to schema_version 3.

The "Estimated Misconceptions Over Time" chart is collapsed by default — it is the most detailed thing on the page and the least glanceable, so the page opens on the summary cards. Two things worth a look:

• The heading is a real <button> inside its <h3>, so it is focusable and operable from the keyboard. The <p role="button"> disclosures on the login page carry no tabindex and are not; this deliberately does not copy that. It reuses their rotating .toggle-arrow triangle, an affordance that does not depend on color. Bootstrap's .btn:focus glow is suppressed on it — being :focus rather than :focus-visible, it stayed painted around the heading after a mouse click, in Bootstrap's own default blue rather than the theme's — and the :focus-visible ring is restated at (0,3,0) to outrank .btn:focus { outline: 0 }.
• Chart.js measures its canvas as it constructs the chart, and a canvas inside a collapsed container measures zero, so the chart is built on first open (.one('shown.bs.collapse', …)) rather than on page load.

Verified in the running app

A generated exercise came back with 5 questions, opening on a deontic English-to-LTL item. The card reads "Trace satisfaction 50% / English to LTL 50%". The chart section starts collapsed with aria-expanded="false" and the canvas at 0×0; on opening it builds a 690×345 chart with both misconception series and no console errors. The keyboard focus ring resolves to the theme's var(--focus) with Bootstrap's blue glow gone.

One caveat on that last point: I could not confirm Enter-to-activate through the browser harness, because its synthetic key events do not trigger native button activation — the pre-existing navbar toggler, untouched by this PR, behaves the same way under it. Keyboard operability here rests on the control being a real <button type="button"> (tabIndex 0, verified) rather than on a simulated keypress.