feat: add elasticsearch backend provider

[houyuwushang]

Description

Adds a first-class Elasticsearch backend provider.

Delta Proxy Cache is used only for aggregation requests whose date-histogram buckets can be merged independently: size: 0, one mandatory range on the configured timestamp field, one top-level date_histogram, complete aligned buckets, ascending key order, and no pipeline or scripted aggregations. Supported intervals are fixed intervals and UTC calendar minute/hour/day intervals. Partial edge buckets, response-shaping URL options, and unsupported query shapes fall back to Object Proxy Cache using the exact request as the cache key. Incomplete Elasticsearch responses are rejected by the DPC modeler and are not cached.

The provider preserves each bucket's JSON and aggregation metadata, including nested metrics, and reconstructs hits.total from the returned buckets. The timestamp field defaults to @timestamp and can be changed with backends.<name>.elasticsearch.timestamp_field. Elasticsearch targets are also registered with the Basic authenticator observer for user_router configurations.

The developer environment adds Elasticsearch, generated log data, Kibana routed through Trickster, and seeded data-view, saved-search, log-volume visualization, and dashboard objects.

Fixes #7.

Type of Change

• • Bug fix
• • New feature
• • Optimization
• • Test coverage
• • Documentation
• • Infrastructure

AI Disclosure

• • This contribution DOES NOT include AI-generated changes
• • This contribution DOES include AI-generated changes, and I have reviewed the relevant contributing guidelines.

Validation

• go test ./pkg/backends/elasticsearch/... ./pkg/backends/options ./pkg/backends/providers/... ./pkg/backends/alb ./pkg/proxy/authenticator/registry ./pkg/proxy/handlers/elasticsearch ./pkg/config/validate -count=1
• go test ./pkg/backends/elasticsearch -run '^TestQueryHandlerCachesOnlyCompleteHistogramBuckets$' -count=20
• go test -race ./pkg/backends/elasticsearch/... ./pkg/proxy/authenticator/registry -count=1
• go vet ./pkg/backends/elasticsearch/... ./pkg/proxy/authenticator/registry
• go tool golangci-lint run --timeout 5m ./pkg/backends/elasticsearch/... ./pkg/proxy/authenticator/registry
• go test -run '^$' ./...
• validated both the full example and developer-environment configurations with Trickster
• checked the Compose YAML, POSIX seed-script syntax, and embedded Kibana saved-object JSON
• git diff --check origin/main...HEAD

An earlier revision was exercised through a clean Linux Docker Compose startup, including Elasticsearch/Kibana, Trickster-backed count and date-histogram requests, and the seeded Kibana objects. The log-volume visualization added in this audit has been structurally validated but has not been rerun in Docker.

[jranson]

@houyuwushang Thanks for this! One thing we'd like to see is an updated developer environment docker-compose.yml that includes:

• a new Elasticsearch backend added to the existing Trickster config yaml
• Elasticsearch containers included in the compose, with the Elasticsearch UX/Frontend configured with Trickster as the Elasticsearch API (instead of pointing the frontend directly at the underlying API)
• Seeded log data to enable out-of-box testing/verification (e.g., logs dynamically generated on environment startup that include recent timestamps to ensure they are properly cached by Trickster - and possibly some older timestamps to exercise time-based cacheable boundaries; refer to the ClickHouse data seeding in the developer env for ideas)

[houyuwushang]

Updated this PR to include the developer Elasticsearch environment requested here:

• added an es1 Elasticsearch backend to the developer Trickster config, using the memory cache and @timestamp field
• added elasticsearch, elasticsearch_seed, and kibana services to the developer docker-compose file
• configured Kibana's ELASTICSEARCH_HOSTS to point at Trickster's es1 API (http://host.docker.internal:8480/es1) rather than the underlying Elasticsearch service
• added a startup seed script that recreates trickster-dev-logs with generated recent and older @timestamp log records for date histogram testing
• documented the Kibana URL and seeded index in the developer environment README

Validated with:

• go run cmd/trickster/main.go -validate-config -config docs/developer/environment/trickster-config/trickster.yaml
• go test ./pkg/backends/elasticsearch/... ./pkg/backends/options ./pkg/backends/providers ./pkg/config/validate -count=1
• go test -run '^$' ./...
• go tool golangci-lint run --timeout 5m ./pkg/backends/elasticsearch/... ./pkg/backends/options ./pkg/backends/providers ./pkg/proxy/handlers/elasticsearch ./pkg/config/validate
• YAML structural check of docs/developer/environment/docker-compose.yml via Python/PyYAML
• git diff --check

I could not run docker compose config or start the compose stack locally because this environment does not have the Docker CLI available.

[jranson]

I could not run docker compose config or start the compose stack locally because this environment does not have the Docker CLI available.

@houyuwushang this comment appears to be from your AI agent. While it's OK to use agentic AI, you will need to install docker locally, run the docker compose and visually verify the new functionality for the developer environment. As reviewers, we are using and verifying changes with the developer environment; and other contributors will expect it to work out of the box without any additional manual steps. My initial tests of the updated docker compose indicate that the new Elasticsearch containers are running, but the Kibana instance prints a lot of errors and does not fully integrate with the trickster backend. For example, the Kibana UX does not show the trickster-dev-logs index in the index list, and also lacks any pre-configured dashboards to demonstrate and exercise the new functionality using the generated logs.

[houyuwushang]

@jranson Thanks, that feedback is fair. To clarify the earlier validation note: the machine I used for the first update did not have a Docker-capable developer environment available, so I should have moved the runtime validation to a suitable environment before posting that update. I have now done that and pushed 4dc4df7 based on the runtime testing.

That update adds:

• host.docker.internal:host-gateway for Kibana so Linux Docker environments can resolve the host Trickster process
• a kibana_seed service that waits for Kibana, then creates the trickster-dev-logs data view, saved search, and Trickster Dev Logs dashboard
• README notes for the seeded Kibana dashboard URL

Additional validation performed on a Docker-capable Linux environment:

• docker compose config
• sh -n docs/developer/environment/docker-compose-data/kibana-config/seeding/seed.sh
• clean compose rebuild/start with fresh volumes for the ES/Kibana developer services
• go run cmd/trickster/main.go -config docs/developer/environment/trickster-config/trickster.yaml --log-level debug
• verified http://127.0.0.1:8480/es1/_cat/indices?v includes trickster-dev-logs
• verified direct Elasticsearch and Trickster-backed _count both return 252 documents for trickster-dev-logs
• verified a date histogram query through http://127.0.0.1:8480/es1/trickster-dev-logs/_search
• verified Kibana status is available
• verified kibana_seed exits 0 and creates the data view, saved search, and dashboard

GitHub checks are green on the updated head as well.

[houyuwushang]

I took another pass at the DPC boundary before review and found a correctness issue: a partial final date-histogram bucket could be treated as complete and then reused from cache.

The update admits only complete, aligned buckets from a single required timestamp range. Unsupported histogram shapes use the exact-request OPC path; incomplete Elasticsearch responses are rejected and not cached. I also preserved aggregation metadata and reconstructed hits.total.

I rebased the branch onto current main. The seeded Kibana dashboard now includes a log-volume histogram that exercises this path, and the PR description reflects the supported query shape.