Fix markdown links being destroyed by machine translation (#2299)#5033
Draft
aseckin wants to merge 1 commit into
Draft
Fix markdown links being destroyed by machine translation (#2299)#5033aseckin wants to merge 1 commit into
aseckin wants to merge 1 commit into
Conversation
agoogle_translate_text passed mimeType as an HTTP header rather than a JSON body field. Google Cloud Translation v3 reads it from the body and defaults to "text/html", so all content has been translated as HTML. Our content is markdown. In HTML mode an inline tag splits it into separate text nodes, and the service drops the ones without prose. A link written as [<u>label</u>](url) loses the "[" and "](url)" around the tag entirely (es -> en), or gains spaces that break the syntax (en -> es), leaving underlined text that is no longer clickable. This affects every language pair where source != target, and every translated markdown field, not just question descriptions. Send mimeType in the payload so markdown is treated as opaque text. The \n -> <br> hack goes with it: instead of round-tripping newlines through a tag, split the text on newlines and send the lines via the contents array (one request, as the API declares contents as a list). Line structure is then preserved by construction rather than relying on how the service handles whitespace. html.unescape stays. It is a no-op when the response is already unescaped, and removing it is an unverified risk. Existing rows stay broken until re-translated, so add a disposable retranslate_html_links command that nulls content_last_md5 for the affected originals so queryset_filter_outdated_translations picks them up again. It reports counts by default and only runs with --apply. It skips manually-translated content and content with no detected language, which was never translated and so has nothing to repair. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Preview EnvironmentYour preview environment is ready!
Details
ℹ️ Preview Environment InfoIsolation:
Limitations:
Cleanup:
|
Contributor
Author
|
will wait until we decide how much value we get from auto-translations |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix markdown links being destroyed by machine translation
Fixes #2299
Problem
Links in translated content render as underlined text that isn't clickable.
agoogle_translate_textpassedmimeTypeas an HTTP header instead of a JSON body field. Google Cloud Translation v3 reads it from the body and defaults totext/html— so all of our content has always been translated as HTML, even though it's markdown.In HTML mode an inline tag splits the markdown into separate text nodes, and the service drops the ones without prose in them. Confirmed against question 35557:
un [<u>candidato ultranacionalista…</u>](https://ro.wikipedia.org/…)an <u>ultranationalist candidate…</u> , wonThe
[and](url)are gone; the orphaned<u>is why the text still looks like a link. The existing comment in that function ("it translates text as if it was HTML") had spotted the symptom — the\n→<br>hack andhtml.unescape()were built to cope with it. The same function was the cause.Scope (the issue's two open questions)
Every language pair, not just Spanish-first.
agoogle_translate_textis the single chokepoint; the source language is irrelevant. English-origin questions are broken too, just differently —[<u>significant amount of controversy</u>](url)becomes[ <u>gran cantidad de controversia</u> ] (url)in Spanish, where the injected spaces break the syntax just as fatally. English→Chinese happens to survive. es→en is simply the most visible pair to a mostly-English audience.Every translated markdown field, not just Question Info — Question, GroupOfQuestions, Post, Notebook, Comment, KeyFactorDriver, Project.
Worth knowing: plain
[text](url)links are fine and survive translation. Breakage requires an HTML tag inside the link label, which is what makes a targeted backfill viable instead of re-translating everything.Changes
utils/translation.py— sendmimeType: "text/plain"in the payload so markdown is treated as opaque text. The<br>hack goes with it: instead of round-tripping newlines through a tag, split the text on newlines and send the lines via thecontentsarray (still one request — the API declarescontentsas a list). Line structure is preserved by construction rather than depending on how the service handles whitespace.html.unescape()stays — it's a no-op when the response is already unescaped, and removing it is an unverified risk.misc/management/commands/retranslate_html_links.py— disposable backfill. Existing rows stay broken otherwise, sincecontent_last_md5still matches their unchanged original. Nulls the hash soqueryset_filter_outdated_translationspicks them up. Reports counts by default, only writes with--apply. Skips manually-translated content and content with no detected language (never translated, so nothing to repair).tests/unit/test_utils/test_translation.py— first tests for this module.No frontend changes, no migration. The renderer is fine — it was handed text that no longer contained a link.
Testing
786 unit tests pass, including 16 new ones. Also ran the real pipeline against the actual question 35557 row with a stubbed HTTP layer: the link now reaches the API in one piece and survives reassembly,
mimeTypeis in the body, no<br>injection, paragraphs intact.Backfill dry run reports 850 affected objects (261 notebooks, 309 questions, 58 groups, 222 comments). Spot-checked the regex — every sample was a genuine
[<u>…</u>](url), no false positives.Before merging
Plain-text mode still needs one check against the live API on staging — no
GOOGLE_TRANSLATE_SERVICE_ACCOUNT_KEYavailable locally. The segmentedcontentsdesign removes the newline-preservation risk, but staging is where it first meets the real service. Re-translate 35557 and confirmdescription_encomes back with the link intact.