Skip to content

docs: refresh README.md and CLAUDE.md against current source - #183

Merged
marevol merged 1 commit into
masterfrom
docs/refresh-readme-and-claude-md
Jul 28, 2026
Merged

docs: refresh README.md and CLAUDE.md against current source#183
marevol merged 1 commit into
masterfrom
docs/refresh-readme-and-claude-md

Conversation

@marevol

@marevol marevol commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Brings README.md and CLAUDE.md back in line with the current source. The immediate trigger was the broken Maven Central badge, but the docs had drifted far enough that most of the README's code examples no longer compiled.

Badges

maven-badges.herokuapp.com now returns 404 (Heroku retired the free tier). Replaced with a shields.io Maven Central badge pointing at fess-crawler — the artifact users actually depend on — and added a javadoc.io badge. All badge and link targets verified to return 200.

Corrections

Every item below was checked against the source, not inferred:

Documented Actual
15.2.0-SNAPSHOT current release
JUnit 4 JUnit 5 (Jupiter) — there is no junit:junit dependency
crawler.crawlerContext / crawler.urlFilter both protected; external code needs the delegating setters
container.singleton("httpClient", HcHttpClient.class, ...) HcHttpClient is abstract — use Hc5HttpClient
setSocketTimeout setSoTimeout
setMaxConnections setMaxTotalConnections (init-param key is the singular maxTotalConnection)
setTrustAllCertificates setIgnoreSslCertificate
crawlerContext.setDefaultIntervalTime(...) does not exist — four delayMillis* knobs on DefaultIntervalController
FileTransformer.setMaxContentSize(...) does not exist — limits live on ContentLengthHelper
accessResult.getContent() getAccessResultData().getDataAsString()
OpenSearchDataService + setIndexName/setHostname/setPort no no-arg constructor and no such setters; configured via OpenSearchCrawlerConfig
ExecutionTimeoutException under CrawlerSystemException under ExtractException
MultipleCrawlingAccessException under CrawlerSystemException under CrawlingAccessException
max line length 120 140 (per the shared formatter config in fess-parent)

The old "Basic Usage" example registered three components as singletons. StandardCrawlerContainer instantiates eagerly, so @Resource injection resolved to null and addUrl() NPE'd; a crawl needs roughly 25 components. The in-repo tests never caught this because they live in the same package as Crawler and so can reach its protected fields.

What replaced it

The Quick Start is now the Lasta DI bootstrap — SingletonLaContainerFactory.setConfigPath("crawler.xml") + init() — which gives a fully wired crawler in a few lines. It was compiled and run against the built jars: it crawls a seed page, follows child links, and reports results through DataService.

Every Java snippet in the README was extracted and compiled against fess-crawler and fess-crawler-lasta. The DI override examples were booted in a real container and their effects asserted.

Newly documented

  • s3:// and gcs:// clients, and that S3Client falls back to the AWS default credentials provider chain when accessKey/secretKey are blank (IAM roles, IRSA, instance profiles)
  • HtmlExtractor extracts default metadata by default; JSON-LD is opt-in
  • Weight-based extractor selection and ExtractorBuilder for extracting from a stream without crawling
  • Content-length caps are enforced during download, not after buffering
  • fess-crawler-lasta declares jakarta.transaction-api as provided, so consumers must add it or startup fails with ClassNotFoundException: jakarta.transaction.Transactional$TxType
  • ZipExtractor, TarExtractor, ApiExtractor and CommandExtractor ship in the jar but are not in the default DI
  • Redefining extractorFactory in a DI file that includes crawler.xml replaces it wholesale and silently drops every default MIME mapping — verified: getExtractor("text/html") returns null. The README now shows appending instead, with a warning
  • CI runs mvn -B package on a single JDK 21, so javadoc:jar and license:check gate the build even when mvn test is green
  • JaCoCo reports coverage but enforces no threshold — the ">80%" line was aspirational
  • Surefire runs test classes in parallel; JodExtractorTest is excluded; several tests need Docker

Notes

Documentation only — no source or build changes.

Replace the dead maven-badges.herokuapp.com badge (now 404) with a
shields.io Maven Central badge, and add a javadoc.io badge.

Correct the documented API, which had drifted from the code:

- Dependency version 15.2.0-SNAPSHOT -> current release
- JUnit 4 -> JUnit 5 (Jupiter); base classes PlainTestCase/LastaDiTestCase
- Crawler.crawlerContext / urlFilter are protected, not public; use the
  delegating setters and addIncludeFilter/addExcludeFilter instead
- HcHttpClient is abstract; register Hc5HttpClient (or Hc4HttpClient)
- setSocketTimeout -> setSoTimeout, setMaxConnections ->
  setMaxTotalConnections, setTrustAllCertificates -> setIgnoreSslCertificate
- DefaultIntervalController has no setDefaultIntervalTime; document the
  four delayMillis* knobs
- FileTransformer has no setMaxContentSize; size limits live on
  ContentLengthHelper
- AccessResult has no getContent(); use
  getAccessResultData().getDataAsString()
- OpenSearchDataService takes an OpenSearchCrawlerConfig arg and has no
  host/port setters; show the real XML wiring
- Exception tree: ExecutionTimeoutException is under ExtractException,
  MultipleCrawlingAccessException under CrawlingAccessException, plus
  OpenSearchAccessException
- Formatter line length is 140, not 120

Replace the Quick Start with the Lasta DI bootstrap, which is verified to
run end to end. Every Java snippet in the README now compiles against the
built jars.

Document previously missing behavior: S3/GCS clients and the AWS default
credentials provider chain, HtmlExtractor default metadata and opt-in
JSON-LD, extractor weight selection and ExtractorBuilder, in-download
content-length enforcement, the jakarta.transaction-api provided-scope
requirement, that ZipExtractor/TarExtractor/ApiExtractor/CommandExtractor
are not in the default DI, and that redefining extractorFactory in an
including DI file silently drops all default MIME mappings.

Note in CLAUDE.md that CI runs mvn package, so javadoc:jar and
license:check gate the build even when mvn test is green, and that JaCoCo
enforces no coverage threshold.
@marevol
marevol merged commit 6cc4491 into master Jul 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant