Skip to content

docs(tiktok): document Creator Search Insights (CSI) scraping#390

Open
eyalgaon wants to merge 2 commits into
browser-use:mainfrom
eyalgaon:tiktok-csi-skill
Open

docs(tiktok): document Creator Search Insights (CSI) scraping#390
eyalgaon wants to merge 2 commits into
browser-use:mainfrom
eyalgaon:tiktok-csi-skill

Conversation

@eyalgaon
Copy link
Copy Markdown

@eyalgaon eyalgaon commented May 25, 2026

Summary

Adds a new domain skill at domain-skills/tiktok/csi-scraping.md for TikTok Creator Search Insights (tiktok.com/csi), an auth-walled dashboard exposing trending search queries by category. Captures everything the next agent needs to skip the discovery work I had to do:

  • URL + login wall detection (/csi redirects to /login for anonymous users)
  • The private API endpoint /api/search/inspired_query/recommended_queries, request body shape, and category filter values (Food, Fashion, Gaming, Tourism, Science, Sports)
  • Response fields worth keeping (query_text, popularity_v2, trending_seq_v2 time series, query_labels incl. content_gap, textnet hierarchy)
  • Pagination strategy (has_more + cursor, 50/page accepted)
  • A gotcha that cost me a few iterations: js() wraps in a non-async IIFE, so top-level await inside the captured fetch loop won't compile — use .then() chains or paginate from the Python side, one page per js() call

No coordinates, no cookies, no user-specific state — only the durable shape of the site, per the SKILL.md guidance on what a domain skill should capture.

Test plan

  • Used the documented flow end-to-end against a real logged-in session — pulled 174 food trends, 28 flagged content_gap
  • Verified the captured endpoint shape is reproducible across runs
  • Confirmed the js()-non-async IIFE behavior matches the documented workaround

🤖 Generated with Claude Code


Summary by cubic

Adds a new domain skill documenting how to scrape TikTok Creator Search Insights (tiktok.com/csi) behind login. Covers private API capture, pagination, and regional targeting via tt-target-idc to get country-appropriate trends.

  • New Features
    • Added domain-skills/tiktok/csi-scraping.md with login-wall detection and session reuse.
    • Captures /api/search/inspired_query/recommended_queries via a window.fetch interceptor; documents body (category/language/tab), limit 50, offset/cursor, and session_refresh_index.
    • Lists key response fields to keep: query_text, popularity_v2, trending_seq_v2, query_labels, textnet, video_num.
    • Regional targeting: swap tt-target-idc/store-idc and drop tt-target-idc-sign to reroute; recommend language_filters: ["en"] for best coverage.
    • Gotchas: js() wraps a non-async IIFE (use .then()), and Filters dropdown toggle behavior.

Written for commit 5b8c980. Summary will update on new commits. Review in cubic

CSI (tiktok.com/csi) exposes trending search queries by category and is
auth-walled (redirects to /login for anonymous users). Captures the
private API endpoint (/api/search/inspired_query/recommended_queries),
request body shape, category filter values including Food, response
fields worth keeping (popularity_v2, trending_seq_v2, query_labels,
textnet), pagination, and the js()-non-async-IIFE gotcha that breaks
top-level await inside the captured fetch loop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@browser-harness-review
Copy link
Copy Markdown

browser-harness-review Bot commented May 25, 2026

✅ Skill review passed

Reviewed 1 file(s) — no findings.

Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Re-trigger cubic

CSI is geo-personalized to the logged-in account region, not the IP and
not via any UI control. Swapping tt-target-idc (and dropping the bound
tt-target-idc-sign) reroutes the API call to a different IDC and
returns region-appropriate data — confirmed alisg→useast1a moves an
Israeli account from "brunch tel aviv" results to "hot cheetos" results.

Also notes the language_filters sweet spot — ["en"] balances coverage
and quality, ["en-US"] sometimes returns 0 items, empty/missing
surfaces ~40% non-target-region noise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant