Skip to content

Say what themed questions' letters mean; quiet the report-unclear control - #176

Merged
sidprasad merged 2 commits into
mainfrom
themed-question-key-and-report-affordance
Jul 24, 2026
Merged

Say what themed questions' letters mean; quiet the report-unclear control#176
sidprasad merged 2 commits into
mainfrom
themed-question-key-and-report-affordance

Conversation

@sidprasad

@sidprasad sidprasad commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Three student-facing fixes to english-to-LTL questions, plus the footer badge. Version bumped to 2.1.5 with CHANGELOG entries.

1. Themed questions now say what their letters mean

The sentence was in words ("the document is open") while every answer option was in letters (d, c), and nothing connected the two. So the question also tested whether a student could guess that d names the document being open, and not the document, which has more than one state. That is a guessing game about naming rather than a test of LTL.

Each themed question now carries a key listing only the literals its formula uses, so it never hints at attributes the question does not involve:

d: the document is open
c: the user's clearance is active

New ltltoeng_contextualized.legend(). Entries are the plain state phrases even in the deontic arm: a letter denotes a state of the world, and the obligation belongs to the policy being asked about. Options are mutations of the answer and never introduce literals of their own, so the answer's literals cover them.

Reviewers, note the experiment implication: both themed arms are affected. Themed responses logged before this measured LTL reading confounded with guessing the naming, so per-arm analyses spanning this change should be segmented. The abstract control arm is deliberately untouched, since it quotes its literals in the prose already and changing the control would be worse.

2. The deontic scenario now asks itself in the question prompt

Previously the scenario and a "Company policy:" lead-in were glued onto the front of question with newlines and rendered at heading size, giving the framing the same visual weight as the property to formalize. The card header now reads:

You are auditing access to a confidential document. Which of the following LTL formulae best represents this policy?

built from the theme's preamble and a new rule_noun field. Other arms keep the generic "…best represents this English sentence?". The card body is left with the key plus the one sentence to formalize.

question is now the sentence alone, and the logged question_text is the key plus that sentence. The scenario is recoverable from translation_mode, which already identifies the arm.

3. The "unclear sentence" report control no longer reads as a heading

It was a bold rust question ("Is this English sentence confusing or unclear?") sitting an equal distance from the stem above and the radio list below, so it grouped with neither; its .row.ml-2 wrapper also pulled it left of the stem's text edge, since Bootstrap's .row sets margin-left: -15px and ml-2 only partly cancels it.

It is now a quiet caption-style link reading "Report unclear wording" (imperative, so it cannot be misread as the question to answer), tucked under the sentence with asymmetric spacing that groups it with the stem.

Two accessibility points worth a look:

  • Going quiet means losing the color signal, so it carries a permanent underline rather than color alone (WCAG 1.4.1), and uses --ink-2 at 8:1 rather than the muted --ink-3, which is 4.17:1 on the card and under AA for small text. Color returns on hover/focus. The key uses --ink-2 for the same reason.
  • The focus ring had to be restated. The theme's global ring is a zero-specificity :where(...):focus-visible rule, which Bootstrap's .btn:focus { outline: 0 } outranks on any button, so keyboard users would have gotten only a color shift. .btn.unclear-flag:focus-visible restates it at (0,3,0).

The modal, its route (/logenglishltlrating), and its payload are unchanged. The control stays a sibling of .actualQuestion, never inside it, so its label cannot leak into the logged question_text.

4. Footer

The green "Distinguished Paper at CAV 2025" button, shown on every page of the tutor, is now a plain "CAV 2025 Paper" text link beside Blog Post / Brown PLT / GitHub, pointing at the same paper. The README, which addresses researchers rather than students, still names the award.

Verification

Full suite green (331 tests), with 4 new/updated in test/test_ab_contextualized.py covering the key's contents, its omission of unused letters, and its absence from the abstract arm.

Manually verified against a running tutor with a throwaway DB, since no automated coverage exists for this element:

  • Alignment: control, key, and stem all at left=36px.
  • Spacing: 7.8px above the control, 18.4px below, so proximity groups it with the sentence.
  • Contrast: 8:1 for both the control and the key, measured against the card surface; text-muted would have given 4.17:1.
  • Keyboard: Tab reaches the control, the teal ring appears, Enter opens the modal.
  • Round trip: submit returns 200, the modal closes, the ack appears directly under the link.
  • Log integrity: question_text holds the key and sentence, with no "Report unclear wording".
  • All three arms render correctly, and trace question types are unchanged.

Known, not addressed here

Some deontic renderings still double the modal, e.g. "It must eventually not be the case that the document must always be open." Same family as the G(X x) / G(x U y) cases fixed in 2.1.1, but a different formula shape and out of scope for this PR.

The rest of the exercise page still pairs text-muted with small at the sub-AA 4.17:1 (card-header meta, question description). Pre-existing and page-wide; it wants its own pass.

sidprasad and others added 2 commits July 24, 2026 18:38
Three student-facing fixes to english-to-LTL questions, plus the footer.

Themed questions never said what their letters stood for. The sentence was
in words ("the document is open") while every answer option was in letters
(d, c), so a student also had to guess that d names the document being
*open*, not the document, which has more than one state. That is a guessing
game about naming, not a test of LTL. Each themed question now carries a key
listing only the literals its formula uses. Both themed arms are affected;
the abstract control arm is untouched, since it quotes its literals already.
Themed responses logged before this measured LTL reading confounded with
guessing the naming, so per-arm analyses spanning it should be segmented.

The deontic arm's scenario now asks itself in the question prompt ("You are
auditing access to a confidential document. Which of the following LTL
formulae best represents company policy?") instead of being glued onto the
front of the sentence with newlines and rendered at heading size, which gave
the framing the same weight as the property to formalize. Theme gains
rule_noun; the card body is left with the key and the one sentence.

The "unclear sentence" report control read as a heading for the answer
options: a bold rust question, equidistant from the stem above and the radio
list below, pulled left of the stem's text edge by a .row wrapper. It is now
a quiet caption-style link, "Report unclear wording", tucked under the
sentence. It carries a permanent underline rather than color alone (WCAG
1.4.1) and uses --ink-2 at 8:1 rather than the muted --ink-3 at 4.17:1,
which is under AA at that size. Its focus ring had to be restated: the
theme's global ring is a zero-specificity :where() rule that Bootstrap's
.btn:focus { outline: 0 } outranks on any button.

Finally, the footer's "Distinguished Paper at CAV 2025" button is now a
plain "CAV 2025 Paper" link beside the others, pointing at the same paper.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"company policy" named an employer the scenario never introduces. Uses
"this policy" rather than the literal "the following policy", which would
repeat "following" against the prompt's own "Which of the following LTL
formulae"; "this policy" also parallels the generic arm's "this English
sentence".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sidprasad
sidprasad merged commit 75fe12f into main Jul 24, 2026
2 checks passed
@sidprasad
sidprasad deleted the themed-question-key-and-report-affordance branch July 24, 2026 22:43
sidprasad added a commit that referenced this pull request Jul 24, 2026
Main's #176 shipped as 2.1.4/2.1.5, so this PR's changelog entries and
version.html move to 2.1.6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant