feat(outputs.zerobus): Add plugin

[zlata-stefanovic-db]

Summary

Add an output plugin that writes metrics to a Unity Catalog Delta table using the
Databricks Zerobus Ingest service, which accepts records over gRPC and
commits them into Delta directly.

Metrics can already reach Zerobus through the generic http output by posting JSON
records to its REST endpoint. A dedicated plugin uses the streaming gRPC API through
a Databricks Go SDK instead, so records are acknowledged individually, retries resume
rather than restart, and the destination table layout is handled by the plugin rather
than assembled by hand in the configuration, and at a much higher throughput.

The plugin has two schema modes:

• static (the default) writes one row per metric into a fixed four-column table
  (measurement, timestamp_ns, tags, fields) with the metric fields in a
  VARIANT column, so field names and types can change without altering the
  destination table.
• table_schema reads the destination table's columns from Unity Catalog and maps
  tags and fields onto same-named columns, producing a flat, typed row. The schema is
  picked up again when the table is altered, without restarting Telegraf.

Other behaviour worth calling out:

• A batch is only reported as written once Databricks has acknowledged every record.
• A retry resumes where the previous attempt stopped instead of re-sending
  acknowledged record.
• concurrent_streams (up to 100) spreads one batch across several streams for
  throughput on large batches. The README explains when this is worth enabling and
  that ordering is only guaranteed per stream.

Dependencies

Adds github.com/databricks/zerobus-sdk/purego v0.1.0 (Apache 2.0), a Databricks
Zerobus Go SDK. docs/LICENSE_OF_DEPENDENCIES.md is updated accordingly.

Tests

Unit tests cover default and required options, tuning validation, both schema modes,
the protobuf descriptor and VARIANT encoding, value conversion and its rejection
rules, connection setup and teardown, batch splitting, retry resumption after partial
failure, and distribution across concurrent streams, all against a fake ingest
stream. An integration test writes to a real Databricks workspace when credentials
are provided.

Checklist

• No AI generated code was used in this PR
• AI generated code used in this PR follows the InfluxData Policy on AI-Generated Code Contributions

I have signed the InfluxData CLA.

Related issues

No prior issue exists for this plugin; this pull request introduces it directly.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 5.28 % for linux amd64 (new size: 324.5 MB, nightly size 308.2 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

. DEB	. RPM	. TAR . GZ	. ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[srebhan]

Thank you very much for this awesome contribution @zlata-stefanovic-db! Before we go into a more detailed review, I do have a few questions and suggestions on how we can merge this...

1. Please split this PR into smaller ones. Start with a bare minimum plugin and add features and operation modes in follow up PRs. This allows us to review code more efficiently and get things in early.
2. Do we really need the static operation mode? This code requires a lot of additional code (e.g. the "metric" implementation) and doesn't add much, I think, as you can already cover this with the JSON mode you mentioned.
3. Please get rid of the indirection when constructing the client. I know, this is for testing but in my experience it doesn't buy you much if you need to fake the client as it only tests your test code. Go with the integration tests instead. This will get rid of quite some indirection and code.
4. Do we really need concurrent writes? This adds a lot of complexity.
5. Why do we need this complex retry behavior? Can't we utilize Telegraf's ability to buffer the metrics instead?

I think that's enough for a first round... ;-)

Thanks again and I'm looking forward to getting this ready and merged.

[zlata-stefanovic-db]

Hi @srebhan thank you a lot for the fast reply!

1. I'll opened a PR (feat(outputs.zerobus): Add plugin #19444) with the bare minimum plugin and will stack new features on top.

2. The idea behind the static mode was "any Telegraf metric can go into a single Delta table without knowing columns" as a default experience so customers don't need a custom table for every measurement & when measurement fields are evolvable (Zerobus doesn't support ingestion into evolvable schemes completely now).
   As for why we decided to make this a separate mode and not do the JSON-over-http, it is for user experience, throughput advantages of streaming over unary calls (request/response), out of the box OAuth support, recovery handling within the SDK, etc. Do you think this is sufficient for this 'static' mode to stay as a part of this output plugin?

3. Noted, will do that.

4. I'll also remove it from the initial PR. The concurrent writes aimed to increase the throughput. One "stream" is capped at 100MB/s on the server side, so if we want to enable more than this, concurrent writes are a nice way to do this.

5. The idea is to not replay the already sent records. Zerobus sends acknowledgements per record and those records are guaranteed to be durable in the target table. If something happens mid batch and we retry the whole batch, we'd duplicate all of the successfully sent records from the initial request. The resume logic still uses Telegraf’s buffer, it only skips already-acked records.

For the first contribution I’m happy simplify, but I'd like to include some of the throughput enhancing behaviors in follow-ups, since Zerobus is a high-throughput ingestion service where that duplicate/cost issue shows up quickly.

Thanks a lot again!

[srebhan]

Thanks for the quick response @zlata-stefanovic-db! Let me go through your items...

1. I'll opened a PR [...]

Will give it a review ASAP.

2. The idea behind the static mode was "any Telegraf metric can go into a single Delta table without knowing columns" [...]. Do you think this is sufficient for this 'static' mode to stay as a part of this output plugin?

While I do understand the idea behind this mode I'm worried about the "metric" protobuf definition and the code that comes with it. For me this is not something Telegraf specific but rather a generic time-series representation required for that particular SDK. My concern is that people will want to modify this and we end up maintaining a thing that should not be there in the first place...

Edit: Having read PR #19444 wouldn't it be sufficient to define the varadic table in the service and only add an option saying "add all fields to columns X" instead of defining the whole proto here?

4. I'll also remove it from the initial PR. The concurrent writes aimed to increase the throughput. One "stream" is capped at 100MB/s on the server side, so if we want to enable more than this, concurrent writes are a nice way to do this.

To be honest I wish this were implemented in the SDK as it is also useful for other users.

5. The idea is to not replay the already sent records. [...]

Well Telegraf has the notion of partial writes which allow exactly this, you specify the metrics accepted and the ones which are permanently rejected and all the rest is rescheduled for the next write cycle. Isn't this what you want?

[zlata-stefanovic-db]

Edit: Having read PR #19444 wouldn't it be sufficient to define the varadic table in the service and only add an option saying "add all fields to columns X" instead of defining the whole proto here?

Yes thank you, I just wanted to suggest this. We can keep the table_schema mode only in the initial PR and treat this as a follow up if you think this approach is acceptable.

To be honest I wish this were implemented in the SDK as it is also useful for other users.

Agreed. I'm going to figure out how this can be done within the SDK in the future.

Well Telegraf has the notion of partial writes which allow exactly this, you specify the metrics accepted and the ones which are permanently rejected and all the rest is rescheduled for the next write cycle. Isn't this what you want?

Agreed as well. I’ll drop the plugin-local resume logic and rely on Telegraf’s partial-write handling.

[srebhan]

I'll convert this PR to a draft for now so I don't get confused with the other one. :-)