Say what themed questions' letters mean; quiet the report-unclear control - #176
Merged
Merged
Conversation
Three student-facing fixes to english-to-LTL questions, plus the footer.
Themed questions never said what their letters stood for. The sentence was
in words ("the document is open") while every answer option was in letters
(d, c), so a student also had to guess that d names the document being
*open*, not the document, which has more than one state. That is a guessing
game about naming, not a test of LTL. Each themed question now carries a key
listing only the literals its formula uses. Both themed arms are affected;
the abstract control arm is untouched, since it quotes its literals already.
Themed responses logged before this measured LTL reading confounded with
guessing the naming, so per-arm analyses spanning it should be segmented.
The deontic arm's scenario now asks itself in the question prompt ("You are
auditing access to a confidential document. Which of the following LTL
formulae best represents company policy?") instead of being glued onto the
front of the sentence with newlines and rendered at heading size, which gave
the framing the same weight as the property to formalize. Theme gains
rule_noun; the card body is left with the key and the one sentence.
The "unclear sentence" report control read as a heading for the answer
options: a bold rust question, equidistant from the stem above and the radio
list below, pulled left of the stem's text edge by a .row wrapper. It is now
a quiet caption-style link, "Report unclear wording", tucked under the
sentence. It carries a permanent underline rather than color alone (WCAG
1.4.1) and uses --ink-2 at 8:1 rather than the muted --ink-3 at 4.17:1,
which is under AA at that size. Its focus ring had to be restated: the
theme's global ring is a zero-specificity :where() rule that Bootstrap's
.btn:focus { outline: 0 } outranks on any button.
Finally, the footer's "Distinguished Paper at CAV 2025" button is now a
plain "CAV 2025 Paper" link beside the others, pointing at the same paper.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"company policy" named an employer the scenario never introduces. Uses "this policy" rather than the literal "the following policy", which would repeat "following" against the prompt's own "Which of the following LTL formulae"; "this policy" also parallels the generic arm's "this English sentence". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sidprasad
added a commit
that referenced
this pull request
Jul 24, 2026
Main's #176 shipped as 2.1.4/2.1.5, so this PR's changelog entries and version.html move to 2.1.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three student-facing fixes to english-to-LTL questions, plus the footer badge. Version bumped to 2.1.5 with CHANGELOG entries.
1. Themed questions now say what their letters mean
The sentence was in words ("the document is open") while every answer option was in letters (
d,c), and nothing connected the two. So the question also tested whether a student could guess thatdnames the document being open, and not the document, which has more than one state. That is a guessing game about naming rather than a test of LTL.Each themed question now carries a key listing only the literals its formula uses, so it never hints at attributes the question does not involve:
New
ltltoeng_contextualized.legend(). Entries are the plain state phrases even in the deontic arm: a letter denotes a state of the world, and the obligation belongs to the policy being asked about. Options are mutations of the answer and never introduce literals of their own, so the answer's literals cover them.Reviewers, note the experiment implication: both themed arms are affected. Themed responses logged before this measured LTL reading confounded with guessing the naming, so per-arm analyses spanning this change should be segmented. The abstract control arm is deliberately untouched, since it quotes its literals in the prose already and changing the control would be worse.
2. The deontic scenario now asks itself in the question prompt
Previously the scenario and a "Company policy:" lead-in were glued onto the front of
questionwith newlines and rendered at heading size, giving the framing the same visual weight as the property to formalize. The card header now reads:built from the theme's
preambleand a newrule_nounfield. Other arms keep the generic "…best represents this English sentence?". The card body is left with the key plus the one sentence to formalize.questionis now the sentence alone, and the loggedquestion_textis the key plus that sentence. The scenario is recoverable fromtranslation_mode, which already identifies the arm.3. The "unclear sentence" report control no longer reads as a heading
It was a bold rust question ("Is this English sentence confusing or unclear?") sitting an equal distance from the stem above and the radio list below, so it grouped with neither; its
.row.ml-2wrapper also pulled it left of the stem's text edge, since Bootstrap's.rowsetsmargin-left: -15pxandml-2only partly cancels it.It is now a quiet caption-style link reading "Report unclear wording" (imperative, so it cannot be misread as the question to answer), tucked under the sentence with asymmetric spacing that groups it with the stem.
Two accessibility points worth a look:
--ink-2at 8:1 rather than the muted--ink-3, which is 4.17:1 on the card and under AA for small text. Color returns on hover/focus. The key uses--ink-2for the same reason.:where(...):focus-visiblerule, which Bootstrap's.btn:focus { outline: 0 }outranks on any button, so keyboard users would have gotten only a color shift..btn.unclear-flag:focus-visiblerestates it at (0,3,0).The modal, its route (
/logenglishltlrating), and its payload are unchanged. The control stays a sibling of.actualQuestion, never inside it, so its label cannot leak into the loggedquestion_text.4. Footer
The green "Distinguished Paper at CAV 2025" button, shown on every page of the tutor, is now a plain "CAV 2025 Paper" text link beside Blog Post / Brown PLT / GitHub, pointing at the same paper. The README, which addresses researchers rather than students, still names the award.
Verification
Full suite green (331 tests), with 4 new/updated in
test/test_ab_contextualized.pycovering the key's contents, its omission of unused letters, and its absence from the abstract arm.Manually verified against a running tutor with a throwaway DB, since no automated coverage exists for this element:
text-mutedwould have given 4.17:1.question_textholds the key and sentence, with no "Report unclear wording".Known, not addressed here
Some deontic renderings still double the modal, e.g. "It must eventually not be the case that the document must always be open." Same family as the
G(X x)/G(x U y)cases fixed in 2.1.1, but a different formula shape and out of scope for this PR.The rest of the exercise page still pairs
text-mutedwithsmallat the sub-AA 4.17:1 (card-header meta, question description). Pre-existing and page-wide; it wants its own pass.