Keywords: open source idea generator, GitHub project scaffolder, Hacker News GitHub gap finder, AI ML web scraping project ideas, RAG open source gaps, Python portfolio factory, pytest fleet runner, OSS opportunity scoring, cookiecutter alternative for discovery, discover open source opportunities
AutoOSS Factory is a local Python CLI that collects Hacker News and GitHub signals, scores curated AI / ML / scraping gaps, writes a JSON report, scaffolds a starter repo, and runs pytest across your portfolio clones. It does not auto-post to social networks.
AutoOSS Factory is the automation layer around a public AI / scraping portfolio. You run it on your machine. It finds gaps, ranks them, drops a Python skeleton under ~/autooss-workspace/ (or WORKSPACE_ROOT), and can optionally gh repo create --push. Promotion stays manual: the content command writes Medium and X drafts you copy yourself.
| Piece | What it does today |
|---|---|
| Discovery | Offline seed catalog (6 gaps) plus HN Firebase top stories and GitHub repo search |
| Scoring | 7 gap templates ranked with 0.35 * novelty + 0.25 * demand + 0.20 * feasibility + 0.20 * portfolio |
| Scaffold | pyproject.toml, MIT LICENSE, CLI, placeholder modules, tests/test_smoke.py |
| Fleet | pytest on local clones in the catalog; optional load checks on localhost health URLs only |
| Content | One Medium draft + one X post (and thread) per portfolio project, no em dashes |
Python 3.11+. MIT licensed. Scoring is keyword heuristics, not an LLM ranker. OPENAI_* env vars exist in config and are unused.
How do I find open-source project ideas in AI, RAG, and web scraping?
Run python -m src.autooss.cli discover. The factory always loads 6 seed gaps, then tries HN (default 30 filtered stories) and GitHub search (up to 5 queries, 5 repos each, stars:>50). It matches those signals to 7 templates (SchemaWeaver, ScrapeGuard, CrawlSync, RAGAutopsy, ExtractionNet, FreshnessAwareRAG, SentinelScrape) and writes data/opportunities/<run>.json.
Does this automatically publish GitHub repos and tweet for me?
No. Push is opt-in (--push or AUTO_PUSH=true) and needs gh auth plus GITHUB_TOKEN for higher API limits. Social posting is intentionally not wired up. autooss content only writes copy-paste files under data/content/YYYY-MM-DD/.
Is the scaffold a finished product?
No. scaffold_opportunity() writes a runnable skeleton: package layout, a JSON CLI, and a smoke test. You still implement the real modules. Already shipped products (sentinelscrape, freshness-aware-rag, schemaweaver, scrapeguard, crawlsync, autooss-factory) are skipped so the loop does not recreate them.
| Approach | What you get | Tradeoff |
|---|---|---|
| AutoOSS Factory | Repeatable discover, score, scaffold, fleet-test loop focused on AI / scraping | Templates are curated (7 gaps). Scaffolds are placeholders. Mac must be awake for LaunchAgent/cron. |
| Browse HN / GitHub by hand | Full human judgment, no extra tooling | No scored queue, no JSON run history, no fleet pytest |
| Cookiecutter / copier alone | Stronger templates if you already know the project | No trend collection, no scoring, no portfolio test runner |
| "Give me a startup idea" chat | Fast brainstorming | No HN/GitHub evidence, no skip list, no tests, easy to duplicate shipped work |
Use this if you want a queue of gaps and a green test fleet, not a bot that invents production reverse-engineering for you.
seeds (always) + HN Firebase + GitHub Search API
|
v
keyword match against GAP_TEMPLATES
|
v
score = 0.35*novelty + 0.25*demand + 0.20*feasibility + 0.20*portfolio
|
+--> data/opportunities/<run>.json
|
v
scaffold top N missing names under WORKSPACE_ROOT (default ~/autooss-workspace)
|
+-- optional --> git init + gh repo create --push
|
v
fleet: venv + pip install + pytest on each catalog clone
|
v
content (optional): Medium + X copy-paste packs
Discovery (src/autooss/discovery/collect.py): HN titles must contain keywords such as scrape, rag, llm, crawl, browser, vector, embedding. GitHub queries come from GITHUB_SEARCH_QUERIES (default includes web scraping, RAG pipeline, LLM extraction, anti-bot). Delays: 0.05s per HN item, 0.8s per GitHub query without a token (0.2s with GITHUB_TOKEN). Network failures are logged; seeds still score.
Scoring (src/autooss/scoring/opportunities.py): each live signal bumps demand when its title/tags hit a template keyword list. Seed titles that name a template floor demand at 0.75. Shipped products have lower novelty / portfolio so they rank below new gaps.
Daily pipeline (src/autooss/pipeline/daily.py): collect, score, persist, scaffold, optional push. Full cycle (cycle) runs fleet tests, then daily, then re-tests the fleet if something new was scaffolded.
Fleet (src/autooss/fleet/): catalog paths default to Path.home() / <repo>. Load tests hit only http://127.0.0.1:8000/health and :8001/health when listed. If the service is down, the load check is skipped and is not a fleet failure.
Push (src/autooss/github_ops/push.py): git init -b main, commit, gh repo create {owner}/{name} --public --push. Default owner: pandeyvishwas51-oss.
git clone https://github.com/pandeyvishwas51-oss/autooss-factory.git
cd autooss-factory
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env
# optional: GITHUB_TOKEN for higher GitHub Search limits and for --pushRequirements: Python 3.11+, httpx, pydantic, pydantic-settings, python-dateutil, rich. Dev extra: pytest. Optional extra .[llm] installs openai but nothing in the scoring path calls it yet.
After install, the console script is autooss. python -m src.autooss.cli works from a clone with PYTHONPATH=. as well.
Offline scoring (no network required beyond what you already have):
from src.autooss.discovery.collect import SEED_OPPORTUNITIES
from src.autooss.scoring.opportunities import signals_to_opportunities
opps = signals_to_opportunities(SEED_OPPORTUNITIES, max_n=10)
for o in opps:
print(f"{o.total_score:.3f} {o.name}: {o.one_liner}")CLI:
# Discover + score only (HN/GitHub best-effort; seeds always apply)
python -m src.autooss.cli discover
# Discover, score, scaffold 1 new project under ~/autooss-workspace
python -m src.autooss.cli daily --top 1
# Same, and push with gh (needs gh auth)
python -m src.autooss.cli daily --top 1 --push
# pytest every catalog clone; load-test local health URLs if they are up
python -m src.autooss.cli fleet
python -m src.autooss.cli fleet --no-load
# Fleet + discover + scaffold
python -m src.autooss.cli cycle --top 1
# Copy-paste Medium + X drafts (not posted)
python -m src.autooss.cli contentBackground runner (macOS LaunchAgent, every 6 hours while logged in):
bash scripts/install_automation.shCron backups: scripts/daily_loop.sh (09:00) and scripts/nightly_content.sh (21:30). Details: docs/AUTOMATION.md. Agent session checklist: docs/AGENT_RUNBOOK.md.
| Repo | Focus |
|---|---|
| sentinelscrape | Anti-bot probe + ML + A/B |
| freshness-aware-rag | Freshness re-rank for RAG |
| schemaweaver | Schema discovery |
| scrapeguard | Scraper health |
| crawlsync | Crawl dataset diff |
- Public APIs only (HN Firebase, GitHub Search) with polite delays
- Set
GITHUB_TOKENto stay under GitHub's unauthenticated search cap - Does not post to HN, Reddit, X, or Medium
- Load tests never leave localhost
- Scaffolds are starting points. Expand real code before calling a repo done.
Copy .env.example. Settings live in src/autooss/config.py:
| Variable | Default | Role |
|---|---|---|
GITHUB_TOKEN |
empty | Auth for GitHub Search and push |
GITHUB_OWNER |
pandeyvishwas51-oss |
gh repo create owner |
HN_TOP_N |
30 |
Max filtered HN stories |
GITHUB_SEARCH_QUERIES |
scraping / RAG / LLM / anti-bot / fingerprint | Comma-separated search strings (first 5 used) |
MAX_OPPORTUNITIES |
10 |
Cap on scored templates returned |
SCAFFOLD_TOP_N |
1 |
How many new skeletons per daily run |
AUTO_PUSH |
false |
Push without passing --push |
WORKSPACE_ROOT |
~/autooss-workspace |
Where scaffolds land |
OPENAI_API_KEY |
empty | Reserved. Heuristic scoring does not call an LLM |
pip install -e ".[dev]"
pytest -vtests/test_scoring.py covers seed ranking, scaffold file layout, and offline seeds. tests/test_fleet.py checks the catalog names and that a dead localhost health URL is skipped. tests/test_content.py asserts one pack per project and no em/en dashes. collect_all_signals() may touch the network; failures fall back to seeds.
autooss-factory/
├── src/autooss/
│ ├── cli.py # autooss / python -m src.autooss.cli
│ ├── config.py # env-backed Settings
│ ├── models.py # TrendSignal, Opportunity, DailyRunReport
│ ├── discovery/collect.py # seeds, HN, GitHub search
│ ├── scoring/opportunities.py
│ ├── scaffold/generator.py
│ ├── pipeline/daily.py # discover -> score -> scaffold
│ ├── pipeline/full_cycle.py # fleet + daily
│ ├── fleet/catalog.py # local clone paths
│ ├── fleet/operator.py # pytest + localhost load
│ ├── github_ops/push.py # gh repo create
│ └── content/blogger.py # Medium + X drafts
├── tests/
├── scripts/ # daily_loop, fleet_operator, nightly_content
├── docs/
└── data/ # opportunities, runs, content (gitignored payloads)
A personal OSS factory: discover gaps in AI, ML, RAG, and web scraping, score them, scaffold the next Python repo, and keep existing clones tested.
No. Demand is keyword overlap with live signals plus a seed floor. Novelty, feasibility, and portfolio weights are constants on each template. OPENAI_API_KEY is unused.
Only if you pass --push or set AUTO_PUSH=true, and only for names not in the shipped skip list and not already present in the workspace. Default is scaffold locally, no push.
The clone path is missing, pip install -e ".[dev]" failed, or pytest tests/ returned non-zero. A down localhost health endpoint is not a failure (service_down_skipped).
Yes. Edit GAP_TEMPLATES in src/autooss/scoring/opportunities.py and SEED_OPPORTUNITIES in src/autooss/discovery/collect.py. Re-run discover and read the new JSON.
Python 3.11+ (requires-python = ">=3.11").
On macOS, bash scripts/install_automation.sh installs LaunchAgent com.pandeyvishwas51.autooss (every 21600 seconds + at login). Sleeping Mac means paused jobs. This is not a hosted 24/7 service.
See CONTRIBUTING.md. Short version: pip install -e ".[dev]", pytest -v, pull request with a concrete before/after.
MIT. Free for personal and commercial use. Keep the copyright notice.
AutoOSS Factory finds the next AI / scraping gap, scores it, and scaffolds the repo. You write the real product. You decide when to publish.