Keep Reader sitemap HTML-only

PROJECT_STATUS.md

@@ -19,14 +19,10 @@ maintenance-first support.
 
 ## Timeline
 
-- **2026-08-02:** Deleted the retired `reader` Turso database after D1 cutover
-  acceptance and verified the public Worker remained healthy.
-- **2026-08-01:** Migrated production relational persistence from Turso to a
-  project-owned Cloudflare D1 database after schema, row-count, aggregate,
-  ownership, auth/session, and document-path parity checks. R2 remains the PDF
-  object store.
-- **2026-07-31:** Kept the public sitemap HTML-only and added source-level
-  sitemap/catalog/Markdown parity coverage.
+- **2026-07-31:** Kept the public sitemap limited to the four canonical HTML
+  pages while retaining six machine resources through robots, llms, and the
+  agent catalog. Explicit tests now enforce sitemap, catalog, and Markdown
+  parity; production deployment remains separate.
 - **2026-07-31:** Added locally verified Open Graph/Twitter image metadata and
   SoftwareApplication structured data to the public landing layout; production
   deployment remains separate.

public/api-ai.json

@@ -18,6 +18,20 @@
       "kind": "spa",
       "description": "Product home"
     },
+    {
+      "id": "faq",
+      "url": "https://read.significanthobbies.com/faq",
+      "md": "https://read.significanthobbies.com/faq.md",
+      "kind": "static",
+      "description": "Frequently asked questions"
+    },
+    {
+      "id": "changelog",
+      "url": "https://read.significanthobbies.com/changelog",
+      "md": "https://read.significanthobbies.com/changelog.md",
+      "kind": "static",
+      "description": "Verified product history"
+    },
     {
       "id": "login",
       "url": "https://read.significanthobbies.com/login",

public/changelog.md

@@ -0,0 +1,32 @@
+# Reader changelog
+
+Meaningful improvements to saving, reading, annotating, and revisiting research.
+
+## 2026-07-13 — RSS and Atom reading joined the library
+
+- Authenticated readers can import OPML, manage feeds, refresh them, and track
+  unread items.
+- Feed items can be saved into the article and PDF library for annotation and
+  later research.
+
+## 2026-07-03 — Memory capture became durable
+
+- Saved research memories gained authenticated persistence, search, and a
+  dedicated reading surface.
+- Existing browser memories can be imported into the account-backed
+  collection.
+
+## 2026-07-02 — Safer failure recovery
+
+- Unhandled API and Worker failures now return controlled responses.
+- A React error boundary provides a recovery path when a page cannot render.
+
+## 2026-06-20 — Reader moved to a faster app shell
+
+- Reader moved to Vite and React with a focused Hono Worker while preserving
+  its production identity.
+- Articles, PDFs, highlights, notes, boards, lists, search, and AI-assisted
+  reading stayed together in one library.
+
+[Roadmap](https://github.com/Significant-Hobbies/reader/issues) ·
+[Source](https://github.com/Significant-Hobbies/reader)

public/faq.md

@@ -0,0 +1,41 @@
+# Reader FAQ
+
+Reader is a personal research library for capturing web articles and PDFs,
+reading without distractions, preserving highlights and notes, and using AI
+over sources you choose.
+
+## Capture and reading
+
+Paste an article URL or use the Chrome extension to save the current page.
+Reader cleans web pages with Mozilla Readability. Uploaded PDFs join the same
+library and have their text extracted so they can be searched and discussed.
+
+## Organisation
+
+Tags provide cross-cutting labels, lists group related sources, and boards give
+projects a Kanban-style workflow such as “to read,” “reading,” and “done.”
+Full-text search covers saved article bodies, PDF text, titles, and notes.
+
+## Highlights, notes, and progress
+
+Highlights anchor to the selected text so they persist across visits. Notes
+live beside the source, and boards can track reading progress across a research
+project.
+
+## AI assistance
+
+Reader can summarize an open article or PDF and answer questions using its
+extracted text. You can bring an OpenAI, Anthropic, or Gemini key stored in the
+browser, use the Free AI gateway, or use the local development bridge. Reader
+does not store browser-held provider keys on its server.
+
+## Accounts and privacy
+
+The browser-local library works without an account. Google sign-in enables the
+account-backed library and sync. Saved articles, PDFs, annotations, lists,
+boards, and AI conversations remain private and are excluded from the public
+sitemap and agent catalog.
+
+[Open Reader](https://read.significanthobbies.com/) ·
+[Sign in](https://read.significanthobbies.com/login) ·
+[View the changelog](https://read.significanthobbies.com/changelog)

public/llms-full.txt

@@ -21,6 +21,8 @@ Personal libraries require auth and are not agent-indexed. Public marketing surf
 ## Product links
 
 - Home: https://read.significanthobbies.com/ — App (auth for library)
+- FAQ: https://read.significanthobbies.com/faq — Frequently asked questions
+- Changelog: https://read.significanthobbies.com/changelog — Verified product history
 - Login: https://read.significanthobbies.com/login — Sign in
 
 ## Machine surfaces

public/llms.txt

@@ -6,12 +6,15 @@
 
 - [Home](https://read.significanthobbies.com/): App (auth for library)
 - [FAQ](https://read.significanthobbies.com/faq): Frequently asked questions
+- [Changelog](https://read.significanthobbies.com/changelog): Verified product history
 - [Login](https://read.significanthobbies.com/login): Sign in
 
 ## Machine surfaces
 
 - [Agent catalog](https://read.significanthobbies.com/api/ai): JSON inventory of public surfaces
 - [Homepage markdown](https://read.significanthobbies.com/index.md): Product brief without JS
+- [Full agent brief](https://read.significanthobbies.com/llms-full.txt): Complete product and privacy context
+- [HTML sitemap](https://read.significanthobbies.com/sitemap.xml): Canonical public pages
 - [This index](https://read.significanthobbies.com/llms.txt)
 
 ## Optional

public/sitemap.xml

@@ -1,5 +1,7 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url><loc>https://read.significanthobbies.com/</loc></url>
+  <url><loc>https://read.significanthobbies.com/faq</loc></url>
+  <url><loc>https://read.significanthobbies.com/changelog</loc></url>
   <url><loc>https://read.significanthobbies.com/login</loc></url>
 </urlset>

src/agent-edge.mjs

@@ -13,8 +13,8 @@
 export const AGENT_SURFACE = {
   "name": "Reader",
   "url": "https://read.significanthobbies.com",
-  "llmsFullTxt": "# Reader — full agent brief\n\nResearch library: capture, annotate, and AI-chat over your reading — private by default.\n\n## Index\n\n# Reader\n\nResearch library for capture, annotation, and AI chat over your reading.\n\n## Privacy\n\nPersonal libraries require auth and are not agent-indexed. Public marketing surfaces only.\n\n## Agent entrypoints\n\n- https://read.significanthobbies.com/llms.txt\n- https://read.significanthobbies.com/api/ai\n- https://read.significanthobbies.com/index.md\n\n## Product links\n\n- Home: https://read.significanthobbies.com/ — App (auth for library)\n- Login: https://read.significanthobbies.com/login — Sign in\n\n## Machine surfaces\n\n- https://read.significanthobbies.com/llms.txt\n- https://read.significanthobbies.com/llms-full.txt\n- https://read.significanthobbies.com/api/ai\n- https://read.significanthobbies.com/index.md\n- https://read.significanthobbies.com/sitemap.xml\n- https://read.significanthobbies.com/robots.txt\n\n## Contact / fleet\n\n- Fleet: https://sassmaker.com\n- Agent email for directory verification: sarthakagrawal@agentmail.to\n",
-  "llmsTxt": "# Reader\n\n> Research library: capture, annotate, and AI-chat over your reading — private by default.\n\n## Product\n\n- [Home](https://read.significanthobbies.com/): App (auth for library)\n- [Login](https://read.significanthobbies.com/login): Sign in\n\n## Machine surfaces\n\n- [Agent catalog](https://read.significanthobbies.com/api/ai): JSON inventory of public surfaces\n- [Homepage markdown](https://read.significanthobbies.com/index.md): Product brief without JS\n- [This index](https://read.significanthobbies.com/llms.txt)\n\n## Optional\n\n- [Foundry](https://sassmaker.com): Parent fleet showcase\n",
+  "llmsFullTxt": "# Reader — full agent brief\n\nResearch library: capture, annotate, and AI-chat over your reading — private by default.\n\n## Index\n\n# Reader\n\nResearch library for capture, annotation, and AI chat over your reading.\n\n## Privacy\n\nPersonal libraries require auth and are not agent-indexed. Public marketing surfaces only.\n\n## Agent entrypoints\n\n- https://read.significanthobbies.com/llms.txt\n- https://read.significanthobbies.com/api/ai\n- https://read.significanthobbies.com/index.md\n\n## Product links\n\n- Home: https://read.significanthobbies.com/ — App (auth for library)\n- FAQ: https://read.significanthobbies.com/faq — Frequently asked questions\n- Changelog: https://read.significanthobbies.com/changelog — Verified product history\n- Login: https://read.significanthobbies.com/login — Sign in\n\n## Machine surfaces\n\n- https://read.significanthobbies.com/llms.txt\n- https://read.significanthobbies.com/llms-full.txt\n- https://read.significanthobbies.com/api/ai\n- https://read.significanthobbies.com/index.md\n- https://read.significanthobbies.com/sitemap.xml\n- https://read.significanthobbies.com/robots.txt\n\n## Contact / fleet\n\n- Fleet: https://sassmaker.com\n- Agent email for directory verification: sarthakagrawal@agentmail.to\n",
+  "llmsTxt": "# Reader\n\n> Research library: capture, annotate, and AI-chat over your reading — private by default.\n\n## Product\n\n- [Home](https://read.significanthobbies.com/): App (auth for library)\n- [FAQ](https://read.significanthobbies.com/faq): Frequently asked questions\n- [Changelog](https://read.significanthobbies.com/changelog): Verified product history\n- [Login](https://read.significanthobbies.com/login): Sign in\n\n## Machine surfaces\n\n- [Agent catalog](https://read.significanthobbies.com/api/ai): JSON inventory of public surfaces\n- [Homepage markdown](https://read.significanthobbies.com/index.md): Product brief without JS\n- [Full agent brief](https://read.significanthobbies.com/llms-full.txt): Complete product and privacy context\n- [HTML sitemap](https://read.significanthobbies.com/sitemap.xml): Canonical public pages\n- [This index](https://read.significanthobbies.com/llms.txt)\n\n## Optional\n\n- [Foundry](https://sassmaker.com): Parent fleet showcase\n",
   "indexMd": "# Reader\n\nResearch library for capture, annotation, and AI chat over your reading.\n\n## Privacy\n\nPersonal libraries require auth and are not agent-indexed. Public marketing surfaces only.\n\n## Agent entrypoints\n\n- https://read.significanthobbies.com/llms.txt\n- https://read.significanthobbies.com/api/ai\n- https://read.significanthobbies.com/index.md\n",
   "catalog": {
     "name": "Reader",
@@ -36,6 +36,20 @@ export const AGENT_SURFACE = {
         "kind": "spa",
         "description": "Product home"
       },
+      {
+        "id": "faq",
+        "url": "https://read.significanthobbies.com/faq",
+        "md": "https://read.significanthobbies.com/faq.md",
+        "kind": "static",
+        "description": "Frequently asked questions"
+      },
+      {
+        "id": "changelog",
+        "url": "https://read.significanthobbies.com/changelog",
+        "md": "https://read.significanthobbies.com/changelog.md",
+        "kind": "static",
+        "description": "Verified product history"
+      },
       {
         "id": "login",
         "url": "https://read.significanthobbies.com/login",

src/agent-surfaces.test.ts

@@ -1,58 +1,70 @@
-import { existsSync, readFileSync } from 'node:fs';
+import { readFileSync } from 'node:fs';
 import { resolve } from 'node:path';
 import { describe, expect, it } from 'vitest';
+import { AGENT_SURFACE } from './agent-edge.mjs';
 
-const publicPath = (name: string) => resolve(process.cwd(), 'public', name);
-const readPublicFile = (name: string) => readFileSync(publicPath(name), 'utf8');
+const readPublic = (path: string) => readFileSync(resolve(process.cwd(), 'public', path), 'utf8');
 
-const sitemap = readPublicFile('sitemap.xml');
-const catalog = JSON.parse(readPublicFile('api-ai.json')) as {
-  url: string;
-  llms: string;
-  llmsFull: string;
-  sitemap: string;
-  robots: string;
-  surfaces: Array<{ id: string; url: string; md: string }>;
-};
-
-const sitemapUrls = [...sitemap.matchAll(/<loc>([^<]+)<\/loc>/g)].map((match) => match[1]);
+function sitemapUrls(): string[] {
+  return [...readPublic('sitemap.xml').matchAll(/<loc>([^<]+)<\/loc>/g)].map(
+    (match) => match[1] ?? ''
+  );
+}
 
 describe('public agent surface parity', () => {
-  it('keeps the sitemap limited to cataloged HTML pages', () => {
-    expect(sitemapUrls.sort()).toEqual(catalog.surfaces.map((surface) => surface.url).sort());
+  it('keeps the sitemap limited to canonical HTML surfaces', () => {
+    const catalogUrls = AGENT_SURFACE.catalog.surfaces.map((surface) => surface.url);
+    const urls = sitemapUrls();
 
-    for (const url of sitemapUrls) {
-      const pathname = new URL(url).pathname;
-      expect(pathname).not.toMatch(/\.(?:md|txt|json|xml)$/);
-      expect(pathname).not.toBe('/api/ai');
+    expect(urls).toEqual(catalogUrls);
+    expect(urls).toHaveLength(4);
+    for (const url of urls) {
+      const path = new URL(url).pathname;
+      expect(path).not.toMatch(/\.(?:json|md|txt|xml)$/);
+      expect(path).not.toMatch(/^\/api(?:\/|$)/);
     }
   });
 
-  it('keeps one checked-in Markdown mirror for every cataloged HTML page', () => {
-    for (const surface of catalog.surfaces) {
-      const markdownUrl = new URL(surface.md);
-      expect(markdownUrl.origin).toBe(catalog.url);
-      expect(markdownUrl.pathname).toMatch(/\.md$/);
-      expect(existsSync(publicPath(markdownUrl.pathname.slice(1)))).toBe(true);
+  it('provides a substantive Markdown counterpart for every HTML surface', () => {
+    const publicOrigin = new URL(AGENT_SURFACE.url).origin;
+
+    for (const surface of AGENT_SURFACE.catalog.surfaces) {
+      expect(surface.md).toBeTruthy();
+      const markdownUrl = new URL(surface.md ?? '');
+      expect(markdownUrl.origin).toBe(publicOrigin);
+      const markdown = readPublic(markdownUrl.pathname.replace(/^\//, ''));
+      expect(markdown).toMatch(/^#\s+\S/m);
+      expect(markdown.trim().length).toBeGreaterThan(120);
     }
   });
 
-  it('advertises machine resources outside the HTML sitemap', () => {
-    const discoverySources = [
-      readPublicFile('robots.txt'),
-      readPublicFile('llms.txt'),
-      readPublicFile('api-ai.json'),
-    ].join('\n');
-
-    for (const resource of [
-      '/llms.txt',
-      '/llms-full.txt',
-      '/index.md',
-      '/api/ai',
-      '/robots.txt',
-      '/sitemap.xml',
-    ]) {
-      expect(discoverySources).toContain(resource);
+  it('keeps machine resources discoverable but outside the HTML sitemap', () => {
+    const urls = new Set(sitemapUrls());
+    const robots = readPublic('robots.txt');
+    const llms = readPublic('llms.txt');
+    const catalog = AGENT_SURFACE.catalog;
+    const machineResources = [
+      catalog.llms,
+      catalog.llmsFull,
+      catalog.sitemap,
+      catalog.robots,
+      `${AGENT_SURFACE.url}/api/ai`,
+      `${AGENT_SURFACE.url}/index.md`,
+    ];
+
+    for (const resource of machineResources) {
+      expect(urls).not.toContain(resource);
     }
+    expect(robots).toContain(`Sitemap: ${catalog.sitemap}`);
+    expect(robots).toContain('Allow: /llms.txt');
+    expect(robots).toContain('Allow: /llms-full.txt');
+    expect(robots).toContain('Allow: /index.md');
+    expect(robots).toContain('Allow: /api/ai');
+    expect(llms).toContain(catalog.sitemap);
+    expect(llms).toContain(catalog.llmsFull);
+    expect(llms).toContain('/api/ai');
+    expect(AGENT_SURFACE.llmsTxt).toBe(llms);
+    expect(AGENT_SURFACE.llmsFullTxt).toBe(readPublic('llms-full.txt'));
+    expect(AGENT_SURFACE.catalog.surfaces).toEqual(JSON.parse(readPublic('api-ai.json')).surfaces);
   });
 });