Skip to content

feat(query): Inverted Index support Japanese language tokenizer - #20412

Draft
b41sh wants to merge 1 commit into
databendlabs:mainfrom
b41sh:feat-inverted-index-japanese
Draft

feat(query): Inverted Index support Japanese language tokenizer#20412
b41sh wants to merge 1 commit into
databendlabs:mainfrom
b41sh:feat-inverted-index-japanese

Conversation

@b41sh

@b41sh b41sh commented Aug 30, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR adds Japanese language support to inverted indexes using lindera-tantivy with the embedded IPADIC dictionary.

Changes

  • Add japanese as a supported inverted-index tokenizer.
  • Add two Japanese analyzer filters:
    • japanese_stop, uses JapaneseStopTagsTokenFilter to removes common Japanese functional parts of speech, including particles, auxiliary verbs, conjunctions, symbols, fillers, and non-verbal.
    • japanese_stemmer, uses JapaneseBaseFormTokenFilter to normalizes inflected verbs and adjectives to their dictionary forms. For example, 食べました and 食べた are normalized to 食べる.

fixes: #20367

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: An AI coding agent assisted with using the lindera-tantivy to support Japanese tokenizer
  • Responsible human: @b41sh
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Aug 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 CI Job Analysis

Workflow: 33316447908

📊 Summary

  • Total Jobs: 32
  • Failed Jobs: 1
  • Retryable: 0
  • Code Issues: 1

NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

  • linux / build: Not retryable (Code/Test)

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Support Japanese tokenizer for inverted index / full-text search

1 participant