docs: refresh README.md and CLAUDE.md against current source - #183
Merged
Conversation
Replace the dead maven-badges.herokuapp.com badge (now 404) with a shields.io Maven Central badge, and add a javadoc.io badge. Correct the documented API, which had drifted from the code: - Dependency version 15.2.0-SNAPSHOT -> current release - JUnit 4 -> JUnit 5 (Jupiter); base classes PlainTestCase/LastaDiTestCase - Crawler.crawlerContext / urlFilter are protected, not public; use the delegating setters and addIncludeFilter/addExcludeFilter instead - HcHttpClient is abstract; register Hc5HttpClient (or Hc4HttpClient) - setSocketTimeout -> setSoTimeout, setMaxConnections -> setMaxTotalConnections, setTrustAllCertificates -> setIgnoreSslCertificate - DefaultIntervalController has no setDefaultIntervalTime; document the four delayMillis* knobs - FileTransformer has no setMaxContentSize; size limits live on ContentLengthHelper - AccessResult has no getContent(); use getAccessResultData().getDataAsString() - OpenSearchDataService takes an OpenSearchCrawlerConfig arg and has no host/port setters; show the real XML wiring - Exception tree: ExecutionTimeoutException is under ExtractException, MultipleCrawlingAccessException under CrawlingAccessException, plus OpenSearchAccessException - Formatter line length is 140, not 120 Replace the Quick Start with the Lasta DI bootstrap, which is verified to run end to end. Every Java snippet in the README now compiles against the built jars. Document previously missing behavior: S3/GCS clients and the AWS default credentials provider chain, HtmlExtractor default metadata and opt-in JSON-LD, extractor weight selection and ExtractorBuilder, in-download content-length enforcement, the jakarta.transaction-api provided-scope requirement, that ZipExtractor/TarExtractor/ApiExtractor/CommandExtractor are not in the default DI, and that redefining extractorFactory in an including DI file silently drops all default MIME mappings. Note in CLAUDE.md that CI runs mvn package, so javadoc:jar and license:check gate the build even when mvn test is green, and that JaCoCo enforces no coverage threshold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings
README.mdandCLAUDE.mdback in line with the current source. The immediate trigger was the broken Maven Central badge, but the docs had drifted far enough that most of the README's code examples no longer compiled.Badges
maven-badges.herokuapp.comnow returns 404 (Heroku retired the free tier). Replaced with a shields.io Maven Central badge pointing atfess-crawler— the artifact users actually depend on — and added a javadoc.io badge. All badge and link targets verified to return 200.Corrections
Every item below was checked against the source, not inferred:
15.2.0-SNAPSHOTjunit:junitdependencycrawler.crawlerContext/crawler.urlFilterprotected; external code needs the delegating setterscontainer.singleton("httpClient", HcHttpClient.class, ...)HcHttpClientis abstract — useHc5HttpClientsetSocketTimeoutsetSoTimeoutsetMaxConnectionssetMaxTotalConnections(init-param key is the singularmaxTotalConnection)setTrustAllCertificatessetIgnoreSslCertificatecrawlerContext.setDefaultIntervalTime(...)delayMillis*knobs onDefaultIntervalControllerFileTransformer.setMaxContentSize(...)ContentLengthHelperaccessResult.getContent()getAccessResultData().getDataAsString()OpenSearchDataService+setIndexName/setHostname/setPortOpenSearchCrawlerConfigExecutionTimeoutExceptionunderCrawlerSystemExceptionExtractExceptionMultipleCrawlingAccessExceptionunderCrawlerSystemExceptionCrawlingAccessExceptionfess-parent)The old "Basic Usage" example registered three components as singletons.
StandardCrawlerContainerinstantiates eagerly, so@Resourceinjection resolved tonullandaddUrl()NPE'd; a crawl needs roughly 25 components. The in-repo tests never caught this because they live in the same package asCrawlerand so can reach its protected fields.What replaced it
The Quick Start is now the Lasta DI bootstrap —
SingletonLaContainerFactory.setConfigPath("crawler.xml")+init()— which gives a fully wired crawler in a few lines. It was compiled and run against the built jars: it crawls a seed page, follows child links, and reports results throughDataService.Every Java snippet in the README was extracted and compiled against
fess-crawlerandfess-crawler-lasta. The DI override examples were booted in a real container and their effects asserted.Newly documented
s3://andgcs://clients, and thatS3Clientfalls back to the AWS default credentials provider chain whenaccessKey/secretKeyare blank (IAM roles, IRSA, instance profiles)HtmlExtractorextracts default metadata by default; JSON-LD is opt-inExtractorBuilderfor extracting from a stream without crawlingfess-crawler-lastadeclaresjakarta.transaction-apiasprovided, so consumers must add it or startup fails withClassNotFoundException: jakarta.transaction.Transactional$TxTypeZipExtractor,TarExtractor,ApiExtractorandCommandExtractorship in the jar but are not in the default DIextractorFactoryin a DI file that includescrawler.xmlreplaces it wholesale and silently drops every default MIME mapping — verified:getExtractor("text/html")returnsnull. The README now shows appending instead, with a warningmvn -B packageon a single JDK 21, sojavadoc:jarandlicense:checkgate the build even whenmvn testis greenJodExtractorTestis excluded; several tests need DockerNotes
Documentation only — no source or build changes.