Add 臺 -> 台 to TWVariants.txt. Avoid converting "台湾" to "臺灣" in s2tw/s2twp to match the behaviors of s2hk - #8
Open
frankslin wants to merge 13 commits into
Open
Add 臺 -> 台 to TWVariants.txt. Avoid converting "台湾" to "臺灣" in s2tw/s2twp to match the behaviors of s2hk#8frankslin wants to merge 13 commits into
s2tw/s2twp to match the behaviors of s2hk#8frankslin wants to merge 13 commits into
Conversation
frankslin
force-pushed
the
fix/taiwan-variants
branch
from
January 3, 2026 07:23
a5be725 to
954cdb9
Compare
* Add WASM demo scaffold and project notes * Add OpenCC WASM demo with converter UI and test runner - 补充 WASM 编译结果在前端 JS 中的用法 * Polish WASM demo UI and paths, run tests, and streamline converter export * Add wasm-based OpenCC package and update demo to consume it * Add wasm-based OpenCC package, static demo bundle, and benchmarking page * Add copyright notice and LICENSE
…eparation This commit enhances the opencc-wasm library with TypeScript support and implements a cleaner build architecture with semantic separation between intermediate build artifacts and publishable distribution. TypeScript Support: - Add comprehensive type definitions (index.d.ts) with full JSDoc documentation - Define interfaces: ConverterOptions, ConverterFunction, OpenCCNamespace, etc. - Provide complete type safety for better IDE support and developer experience Build Architecture Redesign (semantic separation): - build/ - Intermediate WASM artifacts (gitignored, for tests/development) * build/opencc-wasm.esm.js - ESM WASM glue * build/opencc-wasm.cjs - CJS WASM glue * build/opencc-wasm.wasm - WASM binary - dist/ - Publishable distribution (committed, for npm) * dist/esm/ - ESM package entry * dist/cjs/ - CJS package entry * dist/data/ - OpenCC config and dictionary files Invariants and Semantics: - Tests import source (index.js) → loads from build/ - Published package exports dist/ only - build/ = internal intermediate artifacts - dist/ = publishable artifacts - Clear separation ensures tests validate actual build output Enhanced .gitignore: - Add build/ to gitignore (intermediate artifacts) - Add node_modules/, logs, OS-specific files (.DS_Store, Thumbs.db) - Exclude editor configurations (.vscode/, .idea/) - Add cache and temporary file exclusions Two-Stage Build Process: Stage 1 (build.sh): - Compiles C++ to WASM using Emscripten - Outputs to build/ directory Stage 2 (build-api.js): - Copies WASM artifacts from build/ to dist/ - Transforms source paths for production - Generates API wrappers for ESM and CJS - Copies data files Package Configuration (package.json): - Add "types" field pointing to index.d.ts - Update "main" and "module" to point to API wrappers in dist/ - Add comprehensive "exports" map: * "." - Main API (ESM/CJS wrappers) * "./wasm" - Direct access to WASM glue for advanced users * "./dist/*" - Wildcard for flexible file access - Include LICENSE and NOTICE in published files Documentation: - Add comprehensive README section explaining build architecture - Document project structure with invariants - Explain semantic separation between build/ and dist/ Benefits: - Better TypeScript integration and IDE autocomplete - Cleaner, more maintainable directory structure - Tests validate actual build output, not stale dist files - Clear semantic separation between internal and publishable artifacts - Professional project setup following modern npm best practices - Long-term maintainability through clear invariants
## Summary
- add a `//data/config:config_dict_validation_test` to test dictionaries and configs against a `testcases.json` file
- switch all CLI/Python/Node tests to consume `testcases.json` as the single source of truth; drop `.in/.ans` dependencies and adjust Bazel/CMake wiring
- streamline dictionary build outputs (no standalone `TWPhrases{IT,Name,Other}.ocd2`) and align DictionaryTest with the actual generated dict set
- add maintenance helpers (refresh_assets.sh cleanup and fix, rapidjson dep/path for CLI test) and keep wasm assets in sync via `testcases.json`
## Testing
- bazel test //data/dictionary:dictionary_test
- bazel test //test:command_line_converter_test
- bazel test //python/tests:test_opencc
- node/test.js (sync/async/promise) using updated testcases.json
----
* feature: add a new ConfigDictValidationTest.cpp to be executed in bazel
* Changeover to JSON-based testcases and clean dictionary outputs
- Switch all tests (C++ CLI, Python, Node) to consume `testcases.json` and drop `.in`/`.ans` dependencies; keep filegroup for the JSON.
- Prune TWPhrases sub-dictionary artifacts and align DictionaryTest to current generated dict set.
- Add rapidjson dep/path for CLI test, refresh_assets script fixes, and keep Bazel Python toolchain note.
* Normalize CommandLineConvertTest for CRLF comparisons on Windows
* Address review feedback for tests and Bazel-only validation
- Rename and guard streams in CommandLineConvertTest; ensure input file opens and normalize CRLF.
- Fix node test promise handling to propagate errors correctly.
- Mark ConfigDictValidationTest as Bazel-only to skip CMake builds.
…cases.json (#10) - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures ---- * wasm-lib: refresh assets script and switch tests to consolidated testcases.json - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures * Rebuild the wasm-lib and update the documentations
新增完整的貢獻指南文檔，包含： - 如何新增詞典條目（強調使用 Tab 字元分隔） - 如何使用排序工具確保詞典正確排序 - 如何安裝 Bazel 並執行測試 - 如何撰寫測試案例（測試驅動開發流程） - 簡轉繁轉換的特殊注意事項（需測試多個配置） 使用台灣繁體中文撰寫。
1. 新增演算法與理論局限性分析文件 - 詳細說明最大正向匹配分詞演算法 - 分析轉換鏈機制與詞典系統 - 探討理論局限性（一對多歧義、缺乏上下文理解、維護負擔） - 與現代方法（統計模型、神經網路）的比較 2. 更新 AGENTS.md - 新增「延伸閱讀」章節 - 連結到技術文件和貢獻指南 3. 新增 Claude Code 配置 - .claude/hooks/session_start.sh - 會話啟動時顯示專案資訊 - .claude/skills/opencc-dict-edit.md - 詞典編輯技能 - .claude/skills/opencc-algorithm-explain.md - 演算法解釋技能 這些配置幫助 AI 代理更好地理解 OpenCC 專案架構與開發流程。
Avoid converting "台湾" to "臺灣" in `s2tw`/`s2twp` to match the behaviors of `s2hk`.
frankslin
force-pushed
the
fix/taiwan-variants
branch
from
January 3, 2026 23:06
954cdb9 to
8e3bf3b
Compare
frankslin
changed the title
s2tw/s2twp to match the behaviors of s2hk
frankslin
force-pushed
the
master
branch
12 times, most recently
from
January 14, 2026 20:23
c07c90b to
bf7636e
Compare
frankslin
force-pushed
the
master
branch
6 times, most recently
from
January 21, 2026 18:30
9f5c667 to
5c555c7
Compare
frankslin
force-pushed
the
master
branch
3 times, most recently
from
January 28, 2026 15:16
58b2124 to
683fb4e
Compare
frankslin
force-pushed
the
master
branch
6 times, most recently
from
March 15, 2026 18:31
6384603 to
a6c6bb5
Compare
frankslin
force-pushed
the
master
branch
3 times, most recently
from
March 18, 2026 19:15
ddc006b to
c75b5e6
Compare
frankslin
force-pushed
the
master
branch
2 times, most recently
from
March 31, 2026 05:41
e192243 to
0cfb89d
Compare
frankslin
pushed a commit
that referenced
this pull request
Jul 24, 2026
Correctness hardening, decoder unification, and API cleanup from review; also extends character-level filtering to the single-dictionary fast path, which the review identified as the largest remaining win. Detailed Changes: - **Shared UTF-8 decoding (review #1-#3)**: - The scanner's 3-byte branch is now explicitly guarded (charLength == 2 || charLength == 3) instead of relying on the implicit invariant that length 1 cannot reach it. - internal::DecodeCodePoint23() is the single decoder used by both the skip-table builder and the scanner, so a key's first character always maps to the bit the scanner tests; the incorrect comment about invalid continuation bytes is replaced with the byte-consumption equivalence argument. - New invariant test: for every dictionary key, SkipUnmatchable() must return 0 on both the fast path and the table path. - **Utf8SkipTable invariants (review #4, #5)**: - Character-level mode is now derived from bmpCandidates being non-empty; the separate charLevel flag is gone. - IDS operator marking iterates UTF8Util::kFirst/kLast IdeographicDescriptionOperator and consults IdeographicDescriptionOperatorArity(), replacing the hard-coded 0xE2 and 0x2FF0..0x2FFF literals; the constants live next to the arity switch with a sync note. - **API and ABI (review #6-#8)**: - PrefixMatch.hpp no longer includes Utf8SkipScan.hpp (and thus no SIMD intrinsic headers); the skip table lives in the opaque Tables pimpl for both paths and sizeof(PrefixMatch) is back to its previous value. Note PrefixMatch.hpp and Utf8SkipScan.hpp are in LIBOPENCC_PRIVATE_HEADERS and are not installed. - dynamic_cast<MarisaDict> is replaced by a virtual Dict::EnumerateKeys(cb) with a GetLexicon()-walking default; MarisaDict overrides it with a trie walk (using the materialized lexicon when one already exists) and DictGroup recurses into children, restoring group handling on the fast path. prefix_match_lib no longer depends on marisa_dict_lib. OPENCC_ABI_VERSION bumped 1.4 -> 1.5 for the new virtual. - **Fast-path character-level filtering and caching (review #9, #10)**: - Both paths now build the same character-level skip table via EnumerateKeys, and fast-path tables go through the existing Tables cache (with a distinct key prefix). Measured convert-phase speedups vs the byte-level baseline on the 1.9MB zuozhuan corpus: t2tw 5.3x, tw2sp 2.9x, t2s 1.7x, s2twp 1.3x, s2tw 1.2x, s2t unchanged; load times unchanged within a few ms. - **Scan details (review #11, #12)**: - SWAR fallback resolves the mismatch byte with ctz on little-endian instead of rescanning; 32-bit ARM NEON now takes the vector path. - **Tests (review section 4)**: - Differential fuzz test: 300 random inputs mixing dictionary keys, CJK, IDS operators, and raw invalid/truncated bytes, comparing Conversion::Convert against a per-character reference loop (same output or same exception). - 4-byte character coverage (lead-byte filtering with and without 4-byte keys), enumeration-failure fallback that clears a partially built bitmap mid-walk, and unsigned literals in skip-scan comparisons. Benchmarks are reproducible with src/tools/SpeedBenchmark.cpp or the CLI's --measured_result flag; corpora were test/benchmark/zuozhuan.txt and its s2t-converted Traditional variant, plus test/golden/input/ us_constitution_zhs.txt repeated 100x.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BYVoid#1001