test(tokenizer): skip tiktoken tests when encoding blob access is forbidden

[BaSui01]

Motivation

• Avoid CI/runtime flakes when tiktoken tries to download encoding blobs from openaipublic.blob.core.windows.net and receives HTTP 403/Forbidden.
• Preserve unit coverage of tokenizer mapping logic while making runtime, network-dependent assertions environment-aware.

Description

• Add requireTiktokenAvailable helper in llm/tokenizer/tiktoken_test.go that calls tok.CountTokens("health-check") and t.Skip when the error contains Forbidden or 403.
• Import strings and wire the helper into runtime-dependent tests by calling requireTiktokenAvailable at the start of TestTiktokenTokenizer_CountTokens, TestTiktokenTokenizer_Encode_Decode, TestTiktokenTokenizer_CountMessages, TestTiktokenTokenizer_Name, and TestTiktokenTokenizer_MaxTokens.
• Keep pure unit assertions (model->encoding mapping and registration) unchanged so structural behavior is still validated without network access.

Testing

• Reproduced original failure with go test ./... where llm/tokenizer failed due to init tiktoken encoding cl100k_base: ... Forbidden (failing tests before the change).
• Ran go test ./llm/tokenizer -v after the change and the package tests passed; the network-dependent tests were SKIPped in the restricted environment (skipped: TestTiktokenTokenizer_CountTokens, TestTiktokenTokenizer_Encode_Decode, TestTiktokenTokenizer_CountMessages, TestTiktokenTokenizer_Name, TestTiktokenTokenizer_MaxTokens).
• Structural tokenizer tests (e.g., TestNewTiktokenTokenizer, TestRegisterOpenAITokenizers) passed as normal.

---

Codex Task

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.