Skip to content

Add transformers to requirements - #13

Open
ericschreiber wants to merge 1 commit into
tugot17:mainfrom
ericschreiber:patch-1
Open

Add transformers to requirements#13
ericschreiber wants to merge 1 commit into
tugot17:mainfrom
ericschreiber:patch-1

Conversation

@ericschreiber

Copy link
Copy Markdown
Contributor

No description provided.

@tugot17

tugot17 commented Sep 9, 2025

Copy link
Copy Markdown
Owner

where do we use this?

@iPieter

iPieter commented Jan 15, 2026

Copy link
Copy Markdown

I am having the same issue, just replicating the readme's instruction.

In TextSampler, you load a tokenizer from the transformer library:

class TextSampler:
    ... 
    def __init__(self, 
                 tokenizer_name: str,
                 dataset_loader: DatasetLoader):
        ...
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants