space/none mode potentiel issue with case_markup

[Zenglinxiao]

When using case_markup in space/none mode, unexpected behavior happens:

>>> pyonmttok.Tokenizer("none", case_markup=True).tokenize("你好世界，这是一个Test。") 
... (['｟mrk_case_modifier_C｠', '你好世界，这是一个test。'], None)
>>> pyonmttok.Tokenizer("none", case_markup=True).detokenize(['｟mrk_case_modifier_C｠', '你好世界，这是一个test。'])
... '你好世界，这是一个test。'

As you can see, .detokenize can not rebuild the original text. Same behavior exists for space.

While mode conservative or aggressive does not suffer this issue. But the result compare to no case_markup is not consistent, as they split the text to insert markup placeholder.

>>> pyonmttok.Tokenizer("conservative").tokenize("你好世界，这是一个Test。")
... (['你好世界', '，', '这是一个Test', '。'], None)
>>> pyonmttok.Tokenizer("conservative", case_markup=True).tokenize("你好世界，这是一个Test。") 
... (['你好世界', '，', '这是一个', '｟mrk_case_modifier_C｠', 'test', '。'], None)

[guillaumekln]

Case markups are not really supported for the "none" and "space" tokenization modes.

case_markup enables segment_case to avoid tokens with mixed casing, but "space" and "none" modes are not allowed to split in the middle of tokens. "space" mode can only split on spaces and "none" mode do not split at all.

Should we just raise an error in this case? Or maybe can you describe what was your expectation when using case_markup and space/none modes?

[Zenglinxiao]

I want to use the case_markup feature with sentencepiece in order to decrease the vocab duplication caused by the case, not sure what's the best practice to do it.
As in the sp_model_path section, you mentioned using none for sentencepiece. Therefore I came into this issue.
Just wondering if the none is a must for the use of sentencepiece though.

Correct me if I'm wrong:
With the original sentencepiece, each sentence is handle by: replace whitespace with spacer, then doing the segmentation.
With Tokenizer(sentencepiece model) under none, each sentence is handled as: split placeholder off from the sentence, the rest of the sentence feed to sentencepiece to do segmentation. But if the placeholder is in the middle of the sentence, then the original sentence will be split into two-part and feed to sentencepiece separately. In the end, even with none, the input to sentencepiece is still a list of tokens(I personally prefer "phrase" in this case) rather than a single sentence, therefore not same behavior with spm_encode.

I then tried the following experiments:

EX1: use `pyonmttok.Tokenizer(...)` -> `.tokenize_file(corpus_file)` -> train sentencepiece model on this pretokenized corpus
EX2: Initialize `pyonmttok.Tokenizer(...)` as pretokenizer -> `learner(tokenizer, **other_opts)` -> ingest corpus_file -> learn model

These two appoarch gives different model & vocab, which I think caused by the way Tokenizer ingest file. Models learned with Tokenizer use "tokens"("phrase") rather than "sentence".
This idea of ingest_tokens won't cause issue apparently with BPE, but for sentencepiece which expect a sentence and don't assume language-depend logic (space as natural delimiter to words is language-depend I think) may not guarantee the same result w.r.t. original sentencepiece.
So, Do you have any idea or recommandation on how to correctly use Tokenizer when working with sentencepiece?

[guillaumekln]

Your understanding is correct.

The behavior is the same as spm_encode as long as you don't use placeholders. When you use placeholders, you are using a feature that is specific to the Tokenizer and that SentencePiece has no concept of. At this point we expect users to only use the Tokenizer, for training and applying SentencePiece models.

So the recommendation would simply be: do not use SentencePiece scripts directly.

The use case sounds reasonable. A similar issue came up in https://forum.opennmt.net/t/problem-in-tokenize-and-detokenize-during-translation/3954. The difficulty is that we need to lowercase the phrase before SentencePiece so that different casing result in the same segmentation. We would need to add some code to find the original casing after applying SentencePiece. I will look into it.

Alternatively, you can try using mode "conservative" or "aggressive". SentencePiece will be used as a subtokenizer like BPE.

[panosk]

I'm also trying to make sentencepiece work with case_markup. I got it working somehow by adding the Tokenizer's case placeholders as user_defined_symbols in sentencepiece. I still get a few <unk>s that I don't get when using sentencepiece in none mode, but with this hack they are reduced a lot. Now the question is: should I lowercase the corpus in order to train the sentencepiece model and get a vocabulary with all subwords lowercased?

[panosk]

OK, this actually seems to work. I lowercased, created a sentencepiece model and vocab with onmt-build-vocab with the case placeholders as user_defined_symbols and trained with the raw training files a test model for 5k steps. There are only very few unks that mostly occur in case splitting (which makes sense) and in uppercase tokens (that I can't figure out why).

Maybe this could be adapted in code, so when sentencepiece is used with a mode other than none and with case_markup, the case_markup placeholders should be predefined as user_defined_symbols.

@guillaumekln , could you please clarify what happens and in what order when creating a sentencepiece model and vocab with onmt-build-vocab with mode aggressive and case_markup?

[guillaumekln]

could you please clarify what happens and in what order when creating a sentencepiece model and vocab with onmt-build-vocab with mode aggressive and case_markup?

Actually this is not possible with onmt-build-vocab from OpenNMT-tf. It always applies a none tokenization before training the SentencePiece model. Looks like we need to add some errors when trying to configure a custom tokenization (or add support for it).

To set a different tokenization, you could use the SentencePieceLearner directly. Here's what happen when you use a tokenizer with aggressive and case_markup:

1. we apply the aggressive tokenization
2. we lowercase the tokens
3. we feed each lowercase tokens as a "sentence" to SentencePiece

The insertion of case markup tokens does not happen in this learning phase. They are added during tokenization after applying the SentencePiece model.

[panosk]

Actually this is not possible with onmt-build-vocab from OpenNMT-tf. It always applies a none tokenization before training the SentencePiece model. Looks like we need to add some errors when trying to configure a custom tokenization (or add support for it).

I see, so essentially it's like running the spm_train directly and onmt-build-vocab just takes care of converting the vocabulary.

To set a different tokenization, you could use the SentencePieceLearner directly.

My attempt was to avoid a separate preprocessing step and have everything ready with onmt-build-vocab -> train, but this seems necessary.

The insertion of case markup tokens does not happen in this learning phase. They are added during tokenization after applying the SentencePiece model.

Yes, and if the SentencePiece model already contains the case markup tokens as user defined symbols, then sentencepiece ignores them when it decodes so the case can be restored correctly and the translated text seems (mostly) fine. But some inconsistencies remain, due to case splitting that creates unseen tokens/subwords to sentencepiece.

[guillaumekln]

My attempt was to avoid a separate preprocessing step and have everything ready with onmt-build-vocab -> train, but this seems necessary.

Yes. I added support for pre-tokenization in the PR linked above.

But some inconsistencies remain, due to case splitting that creates unseen tokens/subwords to sentencepiece.

When using aggressive and case_markup, case splitting is applied as part of the aggressive tokenization and before SentencePiece. So there should not be unseen tokens in this case.

[panosk]

That's fantastic, thanks!

[panosk]

I thought I should leave some feedback on this:

• I get lots of unks, all of them after punctuation marks (parenthesis, quotes, etc). I inspected a bit and I noticed that OpenNMTTokenizer does not add the space marker in front of such symbols. How could we eliminate this inconsistency with the way SentencePiece handles punctuation marks?
• When SentencePiece is used after pre-tokenization, there is a catch: the number of lines fed to SentencePiece do not correspond to the actual corpus lines but to single tokens, and this makes the sentencepiece_trainer explode and crash with bad malloc before creating the suffix array, even though there's still RAM available. After I put a limit of 100M sentences (which should be single tokens, actually) I was able to train the model without issues. I suspect I can push this limit to 150-200M.

[guillaumekln]

I get lots of unks, all of them after punctuation marks (parenthesis, quotes, etc). I inspected a bit and I noticed that OpenNMTTokenizer does not add the space marker in front of such symbols. How could we eliminate this inconsistency with the way SentencePiece handles punctuation marks?

You generated the vocabulary with onmt-build-vocab from OpenNMT-tf, right? When using SentencePiece with pre-tokenization, the output tokens are actually not meant to be compatible with the vocabulary generated by SentencePiece. We should fix the script to rebuild the vocabulary in this case.

[panosk]

Yes, the vocab is built with onmt-build-vocab. I just noticed the related PR in OpenNMT-tf repo, thanks!

[panosk]

Some more feedback: I updated pyonmttok and OpenNMT-tf and tried to build a new vocab with sentencepiece and case_markup. The sp model and the vocab are build, but the user-defined symbols are not included in the vocabulary, even though they are recognized and mentioned from sentencepiece when training starts.
Also, now the only option that is accepted from onmt-build-vocab for building a sentencepiece model is none. This means we loose some of the goodies aggressive offers but at least we should be able to use case_markup, right?