Skip to content

Add StopWordsRemover feature transformer - #99

Merged
irfanghat merged 4 commits into
irfanghat:branch-4.1-stablefrom
devsimiyu:feature/stop-words-remover
Jul 10, 2026
Merged

Add StopWordsRemover feature transformer#99
irfanghat merged 4 commits into
irfanghat:branch-4.1-stablefrom
devsimiyu:feature/stop-words-remover

Conversation

@devclarenjoki

Copy link
Copy Markdown
Contributor

Summary

Adds a StopWordsRemover feature transformer, modeled on the existing Tokenizer, that filters stop words out of a token-array column.

  • StopWordsRemover supports inputCol, outputCol, caseSensitive, and a custom stopWords list; when stopWords is unset the Spark server falls back to its default list.
  • Extends ParamValue / ParamMap to carry std::vector<std::string>, emitted as a string Array literal in MlParams (toString updated to match).
  • Builds the MlRelation transform (OPERATOR_TYPE_TRANSFORMER), mirroring Tokenizer.

Files

  • src/ml/feature/stop_words_remover.h / .cpp — the transformer
  • src/ml/param/param_map.h / .cpp — string-array param support
  • tests/spark/ml/feature/stop_words_remover.cpp — integration tests (default list, custom list, chained after Tokenizer)

Testing

Integration tests run against a live Spark Connect server (docker compose up spark). Please verify locally before merge.

@irfanghat
irfanghat merged commit f7a4d24 into irfanghat:branch-4.1-stable Jul 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants