An agent system that judges a short-form video script (Reels) against real Instagram references, instead of judging it in a vacuum. Built with LangGraph and Claude.
The point of the project is not to generate a script — it is to fail a script with criteria, and to show the arithmetic of how it got there.
research (ReAct) -> extract_patterns -> [hook | retention | cta] -> consolidate
^ |
+-------- rewrite <--------+
A ReAct agent in the research node, because the path research takes cannot
be drawn in advance. It sweeps the niche and decides where to dig deeper:
- discovery — a single paid call covering several hashtags at once, with the results de-duplicated and sorted by real engagement;
- deep dive (optional) — opens the profile of a creator that stood out, to check whether the pattern repeats or was a one-off hit.
Instagram does not expose performance ordering in hashtag search, and the actor
has no parameter for it — the schema only accepts posts | reels | stories. But
resultsType decides whether there is any metric at all:
resultsType |
engagement returned |
|---|---|
posts (default) |
likesCount: 0 on every item |
reels |
real — 57 of 60 items in one measurement, top at 8,638 likes |
Since the actor also accepts several hashtags in a single paid call, you can
sweep volume under reels and sort by engagement in your own code. "Top posts"
is rebuilt client-side. This was measured against the API, not assumed — and the
opposite assumption cost an entire version of this project.
A deterministic graph for everything else, because the path is fixed and the rubric must not improvise. Three reviewers run in parallel with independent rubrics (hook, retention, CTA), consolidation takes the average, and a conditional loop sends the script back for a rewrite while the score sits below the bar, up to a revision cap.
python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m ipykernel install --user --name script-evaluator --display-name "Python (script-evaluator)"cp .env.example .envFill in ANTHROPIC_API_KEY (required) and APIFY_TOKEN (optional).
The notebook looks for keys in this order: environment variables already
exported, then ./.env, then ~/.config/secrets/personal.env.
Without APIFY_TOKEN the notebook runs in simulated mode and stamps the
warning on every output. That is enough to watch the architecture work, not to
draw any conclusion about any niche.
Open script_evaluator.ipynb, select the Python (script-evaluator) kernel and
run everything. Adjust TEST_SCRIPT and niche in the run cell.
Parameters at the top of the notebook:
| constant | effect |
|---|---|
MODEL |
the Anthropic model used in every node |
PASSING_SCORE |
minimum average to approve |
REVISION_CAP |
how many rewrites before giving up |
SEARCH_CAP |
hard stop on paid Apify calls |
.venv/bin/uvicorn api:app --port 8000 # terminal 1
cd web && npm install && npm run dev # terminal 2Opens at http://localhost:5173. The graph is the main canvas: each node is a
card that lights up as the run reaches it, and its gray footer is where the run
writes — the hashtags being swept, the cost climbing, each axis's score, the
pass number. The form and the generated material sit in the left column.
The cards can be dragged and stay connected, because the edges are a function of position rather than fixed lines. Zoom sits at the bottom left. Everything is reachable by keyboard: tab to a card, arrow keys move it.
Research depth is a slider: it sets how many paid Apify calls the run may
make, and shows the estimated ceiling in dollars beside it. The slider's range
comes from /health rather than being hardcoded in the client, so the two sides
cannot drift apart.
A cap chosen by the client is a request, not a stop. SEARCH_CAP_MAX in
pipeline.py is the number that actually decides, and the server answers 422 to
anything above it — during validation, before a graph is built, so a refusal
costs nothing.
Every run is live — there is no replay mode. Each run spends Apify credit and takes a few minutes. The cost shows up while it runs, not only at the end.
The Saira font lives in web/public/fonts/ under SIL OFL 1.1, its own license,
independent of the code's license.
The Apify actors are PAY_PER_EVENT. Real measurements from this project:
| call | items | cost |
|---|---|---|
instagram-hashtag-scraper |
3 hashtags, 40 reels | US$ 0.0345 |
instagram-scraper (profile) |
15 posts | US$ 0.0322 |
| full graph run | 2 searches | US$ 0.0667 |
With the current limits (PER_HASHTAG_LIMIT=15, PER_PROFILE_LIMIT=15) and the
default SEARCH_CAP=3, a full run costs somewhere under ten cents. At the
SEARCH_CAP_MAX=6 ceiling it is roughly twice that. The Starter plan's monthly
credit covers a few hundred runs.
The cap is enforced in the tool's code, not in the prompt. A ReAct agent with a paid tool and a merely textual limit is an invoice risk: a prompt instruction is a request, not a stop. The same reasoning applies one layer out, which is why the ceiling on what a client may ask for lives on the server.
The judge and the writer are the same model. It agrees with itself more than it should. The scores are directional, not a metric.
The sample is recent reels, not the historical archive. Sorting by engagement
happens within what was swept, so the "best" is the best among the recent ones
under those hashtags — not the best in the niche of all time. Raising
PER_HASHTAG_LIMIT widens the base and improves the sample, at proportional
cost.
Video views do not come through. videoViewCount arrives null on both
routes. Only likes and comments are trustworthy.
Choosing the hashtags is your job and it determines everything. Hashtags that are too broad drag in noise from outside the niche; too narrow and they bring tiny accounts. It is the parameter that changes result quality the most, and no agent guesses it better than someone who knows the market.
The notebook's cost accounting understates. usage_total_usd is not always
consolidated at the instant the call returns. The authoritative number is the one
in the Apify console.
Closing the tab stops the spending, but not instantly. The server writes a heartbeat every two seconds, so a dead connection surfaces within one interval and the tools refuse to spend from there on. What it cannot do is interrupt a paid call already in flight — LangGraph has no hook to abort a node midway — so at most one more search completes after you leave.
The system reads public Instagram content — usernames and captions belonging to
real people. .gitignore blocks the scraped samples, but not the notebook's
outputs: if you run it and save, that data goes into the commit. Clear the
outputs before committing, or accept the publication knowingly. Respect the
platform's terms of use and applicable data law.
The app's header shows the LangGraph logo, a mark of LangChain, Inc. It lives
in web/public/brands/ and is used only to identify the technology the project
uses — it implies no sponsorship, affiliation or endorsement. The original colors
were kept on purpose: recoloring someone else's mark to match your palette is
tampering with their identity. If you republish this project under another name,
replace or remove the file.
The same goes for the other names in the header — Anthropic, Apify, FastAPI, React — which appear as text only and belong to their respective owners.
The license below covers the code. The font and the marks carry their own: Saira is under SIL OFL 1.1, and the logos belong to whoever holds them.
PolyForm Noncommercial 1.0.0 — the code is open for reading, study, modification and redistribution, but commercial use is not permitted.
That covers individuals on personal projects, research, teaching and non-profit organizations. It does not cover a product, a paid service, or any use inside a company with a commercial purpose. For that, talk to me.
It is not an OSI-approved license: it is source-available, not open source in the strict sense. The distinction is deliberate.