Open-source workflow for aggregating AI news, translating global coverage into Chinese, and publishing daily digests across channels.
Repository · Releases · Launch Kit · Demo
- Release status: read from release notes index (source of truth for latest patch level)
- Current planning: Roadmap
- Open maintenance milestones: open milestones
- Deferred engineering: Deferred: PyPI
- Before opening follow-up work, confirm the latest release notes entry and align updates to the active milestone.
- Before each milestone closeout, verify release assets and release docs using release checklist.
AI News Open is an open-source AI news stack built for maintainers, content teams, and operators who need more than a toy feed reader. It turns scattered domestic and international AI sources into one workflow: ingest, clean up, deduplicate, extract article bodies, translate global stories into Chinese, generate daily digests, archive them, and publish them across channels.
- Aggregate domestic and international AI news without paying for a commercial news API.
- Translate global AI coverage into Chinese titles, summaries, and "why it matters" notes.
- Ship one stack that includes CLI, FastAPI, a zero-build admin console, and publishing targets.
- Meet open-source engineering expectations with tests, CI, lint, Docker, issue templates, and security policy.
| Layer | Included |
|---|---|
| Sources | Domestic and international RSS/Atom source registry |
| Processing | Cleanup, deduplication, extraction, enrichment, digest generation |
| Interfaces | CLI, FastAPI API, zero-build admin dashboard |
| Publishing | Telegram, Feishu, static site, WeChat draft/publish |
| Engineering | Tests, lint, pre-commit, CI, Docker, changelog, security policy |
- Run your own AI news digest as an individual maintainer.
- Build a Chinese editorial workflow for international AI news.
- Support a lightweight AI media product or internal intelligence feed.
- Push daily digests to Telegram, Feishu, a static site, or a WeChat official account.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install .
cp .env.example .env
python -m ainews run-pipeline --since-hours 48 --limit 30 --max-items 30 --use-llm --persist --export
python -m ainews serve --port 8000After startup you can:
- Open the admin console at
http://127.0.0.1:8000/ - Use the Operations panel to inspect
/health, recent pipeline runs, source cooldowns, source alerts, and publication failures in one screen - Browse the article pool, digest archive, publication history, and WeChat publish status
- Trigger ingest, extraction, translation, digest generation, and publishing from the dashboard
- Freeze a preview into a stored digest snapshot, edit ranking/section/title/summary overrides, and publish the confirmed snapshot instead of a fresh recompute
- Sample demo page: docs/demo/index.html
- Sample digest markdown (ZH): docs/demo/sample-digest.md
- Sample digest markdown (EN): docs/demo/sample-digest.en.md
- Sample digest JSON: docs/demo/sample-digest.json
- Sample health payload: docs/demo/sample-health.json
- Sample operations payload: docs/demo/sample-operations.json
- Sample publication history: docs/demo/sample-publications.json
python -m pip install -e ".[dev]"
pre-commit install
make checkmake check is the local maintainer gate. It runs lint, coverage, package build validation, and the /health smoke check in one command. Use make coverage or make smoke separately only when you are iterating on one layer.
Use this path when the web console is the main operating surface:
- Start with the First Deploy Guide to get the API and console running on
http://127.0.0.1:8000/. - Follow the Operator Console Walkthrough for the safe ingest, extraction, digest review, snapshot editing, and publication preview loop.
- Use Troubleshooting when the console renders as plain text, assets do not load, or the preview-mode strip reports a static or fallback view.
- Before shipping a patch release, use the Release Checklist and its post-publish evidence snapshot so console, release notes, assets, and smoke results stay aligned.
- Architecture Overview
- Compatibility Contract
- Configuration Matrix
- First Deploy Guide
- Operator Console Walkthrough
- Deployment Guide
- Database Migrations
- Troubleshooting
- Monitoring
- Maintainer Bootstrap
- Roadmap
- Support Lifecycle
- PR Review Policy
- Release Notes
- Release Artifacts
- Release Recovery Notes
- Use Cases
- Community Triage
- Contributor Playbook
- Release Checklist
- Support Policy
The current version includes:
- A domestic and international AI source registry with Chinese sites, global media, and official blogs
- RSS/Atom ingestion, basic cleanup, deduplicated persistence
- Cross-source duplicate clustering using canonical URLs, resolved targets, normalized titles, and content fingerprints
- Article body extraction, source-specific cleanup, and local storage
- LLM-powered translation and summary enrichment for international stories
- Chinese daily digest generation and digest history
- Editorial controls for
pin,must_include,suppress, duplicate-primary selection, digest selection preview with explicit inclusion/exclusion reasons, and a frozen digest editor with publish-time overrides - A publication layer for Telegram, Feishu, a static site, and WeChat draft publishing
- Feishu card messages and automatic WeChat cover upload
- Publication history management and WeChat publish-status refresh
- A
FastAPIHTTP API - A zero-build admin console
- CLI commands for ingest, extraction, enrichment, digest generation, publication, and full pipeline execution
- SQLite storage
- Unit tests, API smoke tests, Dockerfile, CI workflows
ruff, coverage,pre-commit, issue/PR templates, Security, and Code of Conduct
The default sources were verified as reachable on 2026-04-07, including:
- Chinese:
36Kr,TMTPost,IT之家,Google News CN AI - Global:
OpenAI News,Google AI Blog,Google DeepMind Blog,Hugging Face Blog,TechCrunch AI,The Verge AI,VentureBeat AI,Google News Global AI
This version optimizes for four practical constraints:
- No paid news API dependency. Everything starts from public RSS or Atom feeds.
- Easy extensibility. All default sources live in
src/ainews/sources.default.json. - International stories can be extracted first and then translated into Chinese titles, summaries, and "why it matters" notes through a configurable LLM.
- The same project can run as a service, a CLI task, a dashboard-backed tool, or a scheduled workflow on a server, in Docker, or in GitHub Actions.
src/ainews/
api.py FastAPI entrypoint
cli.py CLI entrypoint
config.py Environment variables and settings
content_extractor.py Article body extraction
feed_parser.py RSS / Atom parsing
http.py HTTP fetching helpers
llm.py OpenAI-compatible LLM client
models.py Data models
publisher.py Digest publishing layer
repository.py SQLite storage
service.py Ingest and aggregation service
web/ Admin console
sources.default.json Default source registry
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install .
cp .env.example .env
python -m ainews ingest
python -m ainews extract --since-hours 48 --limit 20
python -m ainews stats
python -m ainews print-digest --region all --limit 20
python -m ainews publish --use-llm --persist --export --target static_site
python -m ainews serve --port 8000If you are maintaining or publicly shipping the repository, also run:
python -m pip install -e ".[dev]"
pre-commit install
make checkOpen the console at:
http://127.0.0.1:8000/
For local visual checks, you can also open src/ainews/web/index.html directly as a file. In file:/// mode the page is a static preview (backend actions are read-only).
If the page appears as raw text in an embedded/in-app browser, open it through HTTP instead:
cd "$PWD" && python -m http.server 8000 then visit http://127.0.0.1:8000/src/ainews/web/index.html.
If you prefer to start the API directly:
uvicorn ainews.api:create_app --factory --host 0.0.0.0 --port 8000If you prefer to run it in containers:
docker compose up --buildIf you want Prometheus and Grafana alongside the API:
docker compose --profile monitoring up --buildIf your local pip is recent enough and you want editable installs:
python -m pip install -e ".[dev]"This repository already includes the expected open-source project baseline:
- Community docs:
CONTRIBUTING.md,SUPPORT.md,CODE_OF_CONDUCT.md,SECURITY.md,CHANGELOG.md,GOVERNANCE.md,MAINTAINERS.md,CITATION.cff,docs/community-triage.md,docs/support-lifecycle.md - Collaboration templates: GitHub issue templates, pull request template,
CODEOWNERS, and review policy - Quality gates:
ruff, unit tests, coverage, package build validation,pre-commit - Automation: CI, tag-based release workflow, CodeQL, Dependabot
- Release verification: published artifact checksum and install smoke workflow
- Packaging and runtime: non-root Docker runtime,
HEALTHCHECK,compose.yaml,.dockerignore,.editorconfig - Supply chain: release checksums, CycloneDX SBOM, build provenance, PyPI trusted publishing workflow
- Observability: Prometheus-compatible
/metrics, source runtime history, housekeeping workflow, and ready-to-run monitoring profile - Demo assets: sample site content, GitHub Pages workflow, sample digest markdown and JSON output
Before publishing the repository, still confirm two things:
- Your private security reporting channel is configured in GitHub Security Advisories.
- The organization name, repository URLs, and maintainer metadata in
README.mdandpyproject.tomlmatch your real public values.
Security reports, maintainer triage, and disclosure expectations are documented in SECURITY.md. Support boundaries are documented in SUPPORT.md and Support Lifecycle.
If you are preparing a GitHub launch, you can reuse:
docs/github-launch-kit.mddocs/project-intro.md
Recommended community scaffolding:
ROADMAP.mdSUPPORT.mddocs/community-triage.mddocs/support-lifecycle.mdGOVERNANCE.mdMAINTAINERS.mddocs/architecture.md.github/labels.yml
If you want international stories translated into Chinese and daily digests generated by an LLM, configure an OpenAI-compatible endpoint in .env:
AINEWS_LLM_PROVIDER=openai_compatible
AINEWS_LLM_BASE_URL=your-compatible-endpoint
AINEWS_LLM_API_KEY=your-key
AINEWS_LLM_MODEL=your-modelThen run:
python -m ainews extract --since-hours 48 --limit 20
python -m ainews enrich --since-hours 48 --limit 20
python -m ainews print-digest --region all --limit 20 --use-llm --persistNotes:
extractfetches article bodies for downstream translation and summarization.enrichonly targets international stories and fills Chinese title, summary, and importance fields.print-digest --use-llmprefers translated content to generate a Chinese daily digest.- If no LLM is configured, the system falls back to a rule-based digest template so the workflow remains available.
If you want the full pipeline in one command:
python -m ainews run-pipeline --since-hours 48 --limit 30 --max-items 30 --use-llm --persist --exportThat pipeline executes:
- Ingest the latest stories
- Extract article bodies
- Translate international stories
- Generate a digest
- Export
output/*.mdandoutput/*.json
To publish as part of the same pipeline:
python -m ainews run-pipeline \
--since-hours 48 \
--limit 30 \
--max-items 30 \
--use-llm \
--persist \
--export \
--publish \
--target static_siteNotes:
publishandrun-pipeline --publishautomatically persist the digest so publication status can be refreshed later and idempotency can work.- Publishing the same stored
digestto the sametargetreturnsskippedby default and does not create duplicate publication records. - If you intentionally want to publish again, add
--force-republish.
The content extractor ships with two layers:
- Generic article detection that prefers containers such as
article,main,entry-content, andpost-content - Source-specific cleanup rules that currently prioritize the real article body for
36KrandIT之家, while dropping recommendation panels, share widgets, breadcrumbs, comments, and other site noise
Even if beautifulsoup4 is not installed, the project falls back to a standard-library parser and still applies source-specific extraction rules for 36Kr and IT之家.
The current release supports four publication targets:
telegram: send text digests through the Bot APIfeishu: send digests through a custom webhook, withtextandinteractivecard modeswechat: create drafts in a WeChat official account and optionally submit them for publication; supports automatic cover upload to producethumb_media_idstatic_site: generate a zero-dependency static page andlatest.json
Examples:
python -m ainews publish --use-llm --target telegram
python -m ainews publish --use-llm --target feishu --target static_site
python -m ainews publish --use-llm --target wechat --wechat-submit
python -m ainews publish --digest-id 1 --target static_site --force-republishIf --target is omitted, the system reads AINEWS_PUBLISH_TARGETS.
If you want cards instead of plain text by default:
AINEWS_FEISHU_MESSAGE_TYPE=cardThe implementation tries interactive cards first and falls back to text automatically if the card send fails.
If you do not want to prepare a thumb_media_id manually, provide a cover image source instead:
AINEWS_WECHAT_APP_ID=your-app-id
AINEWS_WECHAT_APP_SECRET=your-app-secret
AINEWS_WECHAT_THUMB_IMAGE_PATH=assets/wechat-cover.jpg
# or
AINEWS_WECHAT_THUMB_IMAGE_URL=https://example.com/wechat-cover.jpg
AINEWS_WECHAT_THUMB_UPLOAD_TYPE=thumbNotes:
thumbmode uses the permanent material upload API withtype=thumb, which is better suited for cover images. Per the official constraint, the image must beJPGand under64KB.- If you already have a media asset, you can still set
AINEWS_WECHAT_THUMB_MEDIA_IDdirectly. - The current implementation only uploads the cover asset automatically. It does not yet rewrite external image links inside the article body.
If you use --wechat-submit or AINEWS_WECHAT_PUBLISH_AFTER_DRAFT=true, the system stores submitted publication records and lets you refresh the final publication state later.
Use:
python -m ainews list-publications --target wechat --limit 20
python -m ainews refresh-publications --target wechat --limit 20The current implementation maps the official freepublish/get status into:
pending: still being publishedok: publication succeedederror: originality validation failure, generic failure, review rejection, deletion after publication, or account restriction
The root path / provides an out-of-the-box admin page that supports:
- News ingest
- Batch translation for international stories
- Batch article body extraction
- Digest generation and review
- Target selection and one-click publishing
- Publication history with manual WeChat refresh
- Digest history
- Manual curation such as pinning, hiding, and editorial notes
If you want simple protection for admin routes, set:
AINEWS_ADMIN_TOKEN=your-secret-tokenThe frontend will automatically send X-Admin-Token to the admin API.
python -m ainews ingest
python -m ainews extract --limit 20
python -m ainews enrich --limit 20
python -m ainews print-digest --use-llm --persist
python -m ainews run-pipeline --use-llm --persist --export
python -m ainews publish --use-llm --persist --target static_site
python -m ainews publish --digest-id 1 --target static_site --force-republish
python -m ainews list-digests --limit 10
python -m ainews list-publications --limit 20
python -m ainews refresh-publications --target wechat --limit 20
python -m ainews stats
python -m ainews serve --port 8000Health check. Returns status, current service version, database checks, and schema_version.
List enabled sources.
Trigger one ingest run.
Example:
curl -X POST "http://127.0.0.1:8000/ingest?source_id=36kr-ai&source_id=openai-news"List stored articles.
Example:
curl "http://127.0.0.1:8000/articles?region=domestic&since_hours=24&limit=20"Return the aggregated digest view. This route is read-only by default. If use_llm=true is added, it will try to generate a digest through the currently configured LLM.
Example:
curl "http://127.0.0.1:8000/digest/daily?region=all&since_hours=24&limit=30"Return article, enrichment, digest archive, and LLM configuration statistics.
Batch-translate international stories.
curl -X POST "http://127.0.0.1:8000/admin/enrich" \
-H "Content-Type: application/json" \
-H "X-Admin-Token: your-secret-token" \
-d '{"since_hours":48,"limit":20}'Batch-extract article bodies.
curl -X POST "http://127.0.0.1:8000/admin/extract" \
-H "Content-Type: application/json" \
-H "X-Admin-Token: your-secret-token" \
-d '{"since_hours":48,"limit":20}'Generate and optionally persist a Chinese daily digest.
curl -X POST "http://127.0.0.1:8000/admin/digests/generate" \
-H "Content-Type: application/json" \
-H "X-Admin-Token: your-secret-token" \
-d '{"region":"all","since_hours":48,"limit":20,"use_llm":true,"persist":true}'Build a ranked selection preview with explicit inclusion, suppression, duplicate-secondary, and ranked-out decisions.
Freeze the current preview into a stored editable digest draft. Optional editor_items can override selection, manual rank, section title, publish title, and publish summary.
Update a frozen digest draft in place. Publishing with digest_id uses this stored snapshot so the outbound digest matches the reviewed editor state instead of a live recompute.
Apply manual curation such as hide, pin, or editorial note.
Run ingest, extraction, enrichment, digest generation, export, and optionally publication in one call.
Build or load a digest and publish it to configured targets.
curl -X POST "http://127.0.0.1:8000/admin/publish" \
-H "Content-Type: application/json" \
-H "X-Admin-Token: your-secret-token" \
-d '{"targets":["static_site","telegram"],"use_llm":true,"persist":true,"export":true,"force_republish":false}'View recent publication records, including target platform, status, external ID, and response summary.
Optional query parameters:
digest_idtargetstatus
Refresh publication state for platforms that support polling. This is currently used mainly for WeChat freepublish/get.
curl -X POST "http://127.0.0.1:8000/admin/publications/refresh" \
-H "Content-Type: application/json" \
-H "X-Admin-Token: your-secret-token" \
-d '{"target":"wechat","limit":20,"only_pending":true}'See .env.example for a concrete sample.
AINEWS_DATABASE_URL: SQLite database locationAINEWS_SOURCES_FILE: source registry fileAINEWS_HOME: working directory root, defaults to the current command directoryAINEWS_OUTPUT_DIR: exported digest directoryAINEWS_STATIC_SITE_DIR: static site output directoryAINEWS_STATIC_SITE_BASE_URL: optional external base URL for the static siteAINEWS_REQUEST_TIMEOUT: fetch timeout in secondsAINEWS_DEFAULT_LOOKBACK_HOURS: default lookback windowAINEWS_MAX_ARTICLES_PER_SOURCE: default per-source ingest capAINEWS_ALLOWED_ORIGINS: API CORS allowlistAINEWS_ADMIN_TOKEN: optional admin API tokenAINEWS_LOG_LEVEL: log level, typicallyINFOorDEBUGAINEWS_LOG_FORMAT:textorjsonAINEWS_EXTRACTION_TEXT_LIMIT: maximum number of locally stored characters per extracted articleAINEWS_LLM_ARTICLE_CONTEXT_CHARS: maximum number of article-body characters sent to the LLMAINEWS_LLM_PROVIDER: defaults toopenai_compatibleAINEWS_LLM_BASE_URL: LLM base URLAINEWS_LLM_API_KEY: LLM API keyAINEWS_LLM_MODEL: LLM model nameAINEWS_LLM_TIMEOUT: LLM timeoutAINEWS_LLM_TEMPERATURE: temperature for translation and digest generationAINEWS_LLM_DIGEST_MAX_ARTICLES: maximum number of articles used for digest generationAINEWS_PUBLISH_TARGETS: default publish targets, comma-separated, for exampletelegram,static_siteAINEWS_TELEGRAM_BOT_TOKEN: Telegram bot tokenAINEWS_TELEGRAM_CHAT_ID: Telegram chat ID or channel nameAINEWS_TELEGRAM_DISABLE_NOTIFICATION: Telegram silent delivery toggleAINEWS_FEISHU_WEBHOOK: Feishu custom bot webhookAINEWS_FEISHU_SECRET: optional Feishu signing secretAINEWS_FEISHU_MESSAGE_TYPE:textorcardAINEWS_WECHAT_ACCESS_TOKEN: optional fixed WeChat access tokenAINEWS_WECHAT_APP_ID: AppID used for access token retrievalAINEWS_WECHAT_APP_SECRET: AppSecret used for access token retrievalAINEWS_WECHAT_THUMB_MEDIA_ID: WeChat cover material ID, required for draft creation unless you upload one automaticallyAINEWS_WECHAT_THUMB_IMAGE_PATH: local cover image path, can replaceAINEWS_WECHAT_THUMB_MEDIA_IDAINEWS_WECHAT_THUMB_IMAGE_URL: remote cover image URL, can replaceAINEWS_WECHAT_THUMB_MEDIA_IDAINEWS_WECHAT_THUMB_UPLOAD_TYPE: upload type for the cover, defaultthumbAINEWS_WECHAT_AUTHOR: WeChat article author nameAINEWS_WECHAT_CONTENT_SOURCE_URL: optional "Read more" URL in the WeChat articleAINEWS_WECHAT_NEED_OPEN_COMMENT: whether comments are enabledAINEWS_WECHAT_ONLY_FANS_CAN_COMMENT: whether only followers can commentAINEWS_WECHAT_PUBLISH_AFTER_DRAFT: whether to submit the draft for publication automatically
- Exported JSON includes top-level
schema_version publishandrun-pipeline --publishare idempotent by default on the tuple(stored digest, target)- Database upgrades follow Database Migrations; the current schema version is
3 - Public compatibility guarantees are documented in Compatibility Contract
The repository already has a solid open-source skeleton, but if you want to push further toward production use, these are the next four upgrades:
- Add source health checks, retries, and ingest monitoring.
- Add permissions and multi-user editing logs to the admin console.
- Add more source-specific extraction rules beyond the current generic DOM strategy.
- Add more publishing targets or deeper platform support, such as richer Telegram formatting, automatic inline image upload for WeChat, and more publication polling.
The repository already includes one scheduled workflow example:
It runs run-pipeline every day and uploads the generated digest files as workflow artifacts.
python -m unittest discover -s tests -vMIT. See LICENSE.