[Detail Bug] Chat: Bidi-only Unicode strings are treated as “visible”, allowing empty chat bubbles and bypassing backend empty-output guards

[detail-app]

Detail Bug Report

https://app.detail.dev/org_befd6425-a158-4e24-9d4d-1e5c08769515/bugs/bug_db6db108-a8b0-439b-a39e-1a878ca7a639

Introduced in #109 by @WilliamAGH on Jul 17, 2026

Summary

• Context: The backend has three safeguards intended to reject/timeout chat input/output that has no visible content: request validation in ChatStreamRequest, streamed chunk visibility tracking in OpenAIStreamingService, and a watchdog that enforces visible output deadlines. All three rely on UnicodeVisibleContent.hasVisibleContent().
• Bug: UnicodeVisibleContent.hasVisibleContent() uses a manual allowlist of invisible characters instead of systematically treating all Unicode FORMAT characters (category Cf) as invisible. As a result, bidi-only strings (e.g., \u200E\u200F\u202A) are incorrectly considered “visible” (true).
• Actual vs. expected: Actual: hasVisibleContent("\u200E\u200F\u202A") == true, so ChatStreamRequest accepts bidi-only input and streaming safeguards treat bidi-only chunks as visible, preventing timeouts; UI can render empty message bubbles. Expected: bidi-only (FORMAT-only) strings should be considered not visible, rejected as input and not counted as visible streamed output.
• Impact: Users can send bidi-only messages (even though ZWSP-only is rejected), and LLMs can emit bidi-only streams without triggering backend errors/timeouts, resulting in empty message bubbles and incorrect streaming/error UI behavior.

Code with Bug

private static boolean isVisible(int codePoint) {
    // ... (manual allowlist of “invisible” code points)
    return !Character.isWhitespace(codePoint)
            && !Character.isSpaceChar(codePoint)
            && !INVISIBLE_CODEPOINTS.contains(codePoint); // <-- BUG 🔴 incomplete allowlist misses many Cf chars (e.g., bidi markers)
}

Explanation

Bidi markers like LRM/RLM and bidi embeddings (e.g., U+200E, U+200F, U+202A–U+202E) are Unicode FORMAT (Cf) characters. The current implementation only marks a small set of Cf characters as invisible via an allowlist (e.g., ZWSP/ZWJ/ZWNJ/FEFF/WORD JOINER), so other Cf characters are treated as visible. Since ChatStreamRequest validation, OpenAIStreamingService’s emittedVisibleText tracking, and the watchdog all depend on UnicodeVisibleContent.hasVisibleContent(), bidi-only content bypasses all three safeguards.

Codebase Inconsistency

StructuredLogValue is reported to handle FORMAT characters systematically (by Unicode category), while UnicodeVisibleContent uses an incomplete allowlist. The repo also already had to “patch” the allowlist reactively (adding ZWNJ/ZWJ) after discovering empty-bubble behavior, indicating the allowlist approach is error-prone.

Failing Test

// src/test/java/com/williamcallahan/javachat/domain/text/UnicodeVisibleContentBidiTest.java

Test output (from ./gradlew test --tests UnicodeVisibleContentBidiTest):

FAIL: ChatStreamRequest accepted bidi-only input: U+200E U+200F U+202A
FAIL: Bidi-only string -> hasVisibleContent=true

Recommended Fix

Replace the allowlist approach with systematic Unicode category handling (treat Character.getType(codePoint) == Character.FORMAT as not visible), e.g.:

private static boolean isVisible(int codePoint) {
    return !Character.isWhitespace(codePoint)
            && !Character.isSpaceChar(codePoint)
            && Character.getType(codePoint) != Character.FORMAT;
}

(Frontend should similarly avoid a manual allowlist and treat FORMAT characters as invisible.)

History

This bug was introduced in commit 72207bf3, which added UnicodeVisibleContent with a hardcoded invisible-character allowlist. A follow-up commit a2da3c32 continued the reactive allowlist approach by adding more zero-width characters after discovery, but bidi FORMAT characters remain unhandled.