java-syntheticalert

Drive a synthetic alert metric from Java, so a Triple Pat check-in timer can verify your alerting pipeline end to end. Works with Micrometer, the Prometheus Java client, and OpenTelemetry.

Why

A broken alerting pipeline looks exactly like a healthy system. No alerts might mean nothing is wrong, or it might mean your alerting is down, and your alerting system is the one thing that cannot alert you about itself.

This library provides a time-based callback to drive a synthetic alert metric. You register the callback as a gauge in your existing metrics setup, alert on the gauge like any other metric, and route the alert to a Triple Pat check-in timer. Every delivered alert then becomes a check-in, and every firing is another fire drill for the whole path from metric to notification. If the check-ins ever stop, your alerting pipeline is broken, and the Triple Pat app raises an alarm through a separate channel to tell you so. An example alert rule and Alertmanager route are below.

Usage

Gradle:

implementation("com.triplepat:syntheticalert:VERSION")

Maven:

<dependency>
  <groupId>com.triplepat</groupId>
  <artifactId>syntheticalert</artifactId>
  <version>VERSION</version>
</dependency>

The library needs Java 17 or later, depends on nothing but the JSpecify nullness annotations, and starts no threads. It is a single object that answers the question "should the synthetic alert be firing right now?", and you hand its value method to your metrics client as a gauge callback. The same calls work unchanged from Kotlin, where the @NullMarked API shows up as non-null types.

Micrometer

Alongside your existing Micrometer setup, against the registry you already have (in Spring Boot, the autoconfigured MeterRegistry). Micrometer's Prometheus registry renders the dotted name as triplepat_synthetic_alert:

import com.triplepat.syntheticalert.SyntheticAlert;
import io.micrometer.core.instrument.Gauge;

SyntheticAlert alert = SyntheticAlert.create();
Gauge.builder("triplepat.synthetic.alert", alert, SyntheticAlert::value)
    .description(
        "Set to 1 when the synthetic alert should fire and 0 otherwise. "
            + "Alert on this metric and route the alert to a Triple Pat check-in "
            + "timer to continuously test your alerting pipeline.")
    .strongReference(true)
    .register(registry);

strongReference(true) matters. Micrometer holds a gauge's state object weakly by default, so if nothing else keeps the alert alive it is garbage collected and the gauge reports NaN from then on. With the strong reference the gauge owns the alert for as long as the registry lives.

Prometheus Java client

The 1.x client (io.prometheus:prometheus-metrics-core) has a gauge that calls back at scrape time. register() with no argument uses the default registry:

import com.triplepat.syntheticalert.SyntheticAlert;
import io.prometheus.metrics.core.metrics.GaugeWithCallback;

SyntheticAlert alert = SyntheticAlert.create();
GaugeWithCallback.builder()
    .name("triplepat_synthetic_alert")
    .help("Set to 1 when the synthetic alert should fire and 0 otherwise.")
    .callback(cb -> cb.call(alert.value()))
    .register();

OpenTelemetry

The same alert serves an OpenTelemetry observable gauge. The OTel-to-Prometheus exporter turns the dotted metric name into triplepat_synthetic_alert:

meter
    .gaugeBuilder("triplepat.synthetic.alert")
    .setDescription("Set to 1 when the synthetic alert should fire and 0 otherwise.")
    .buildWithCallback(m -> m.record(alert.value()));

One JVM, one alert

A synthetic alert is a schedule, and a schedule has to have one owner. Inside a JVM that is easy: create one SyntheticAlert and register it once, however many threads scrape it. Replicas of a service are separate scrape targets, each raising its own alert on its own schedule, and every delivered alert is a real fire drill, so more replicas only mean more check-ins.

The schedule

Each firing holds the gauge at 1 for exactly 10 minutes. The silent gap between firings, from the end of one to the start of the next, is memoryless: exponentially distributed with a mean of one hour.

Memoryless gaps make the firings an attempt at a Poisson process, which cannot synchronize with cron jobs or scrape cycles, and which by the PASTA theorem sees your pipeline as it typically is rather than at some special moment.

As a nod to practicality the gap is truncated. It is never less than 10 minutes, so the alert visibly resolves between firings, and never more than two hours, so the check-in timer can be sized. The truncation pulls the realized mean gap down to about 49 minutes and makes the process only roughly Poisson. If you need the PASTA property and can tolerate wider variation in start times, set a lower min and a higher max, then size the timer for the larger max. That recovers most of the Poisson behavior; for the last few percent, use a mean much longer than the firing duration, since the interval between firing starts is the firing plus the gap.

The schedule advances lazily, at scrape time, from System.nanoTime(). If nobody scrapes for a while, the next scrape replays every transition it missed, so the process stays honest whatever your scrape interval.

There is no magic here: one line is a serviceable substitute, firing for the first ten minutes of every hour:

import io.micrometer.core.instrument.Gauge;
import java.time.LocalTime;
import java.time.ZoneId;

Gauge.builder(
        "triplepat.synthetic.alert",
        () -> LocalTime.now(ZoneId.systemDefault()).getMinute() < 10 ? 1 : 0)
    .register(registry);

But that version fires at the top of every hour, exactly when your cron jobs are doing something interesting. The memoryless schedule cannot synchronize with anything, and that is the point of the library. If you want a deterministic schedule anyway, the line above is all you need.

Options

Every duration is a java.time.Duration, so the unit is in the code rather than in the documentation:

SyntheticAlert alert =
    SyntheticAlert.builder()
        .meanInterval(Duration.ofMinutes(30))
        .maxInterval(Duration.ofHours(1))
        .build();

Builder method	Effect	Default
meanInterval(d)	Mean silent gap between firings	Duration.ofHours(1)
minInterval(d)	Lower bound on the silent gap	Duration.ofMinutes(10)
maxInterval(d)	Upper bound on the silent gap	Duration.ofHours(2)
firingDuration(d)	How long each firing holds the gauge at 1	Duration.ofMinutes(10)

The firing duration must be shorter than the mean interval, and the min and max intervals must bracket the mean. Bad options throw IllegalArgumentException from build(), never at scrape time. The API is @NullMarked and there is no runtime null check; a null fails with the JVM's own NullPointerException. Setting all three intervals equal is allowed: every gap is then exactly that long and the schedule is periodic, which is pointless in production but handy for deterministic debugging. The defaults are the constants DEFAULT_MEAN_INTERVAL, DEFAULT_MIN_INTERVAL, DEFAULT_MAX_INTERVAL, and DEFAULT_FIRING_DURATION, and SyntheticAlert.create() uses all four.

Alert on the metric

groups:
  - name: synthetic
    rules:
      - alert: SyntheticAlert
        expr: triplepat_synthetic_alert == 1
        labels:
          severity: synthetic
        annotations:
          summary: Synthetic alert exercising the alerting pipeline.

Route the alert to a check-in timer

Create a check-in timer at Triple Pat, then point the alert at it. Prefer email delivery: mail transfer agents queue, retry, and try every backend listed in DNS, so a check-in email is more likely to arrive than a single webhook request to a single destination. Send to the same timer at both the .com and .net addresses. The two domains are served by independent DNS providers, so if one zone cannot be resolved the other address still delivers, and extra simultaneous check-ins are harmless. Merge this into your existing Alertmanager config (the fragment assumes you already have a default receiver and working smtp_* defaults):

route:
  routes:
    - matchers:
        - alertname="SyntheticAlert"
      receiver: triplepat
      group_wait: 0s
receivers:
  - name: triplepat
    email_configs:
      - to: YOUR-TIMER-UUID@checkin.triplepat.com
        send_resolved: false
      - to: YOUR-TIMER-UUID@checkin.triplepat.net
        send_resolved: false

send_resolved: false keeps the resolve notification from counting as an extra check-in, so each firing checks in when it starts and not again when it resolves.

If you cannot send email, replace the triplepat receiver above with this webhook receiver instead. Alertmanager rejects a configuration that defines the same receiver name twice:

receivers:
  - name: triplepat
    webhook_configs:
      - url: https://triplepat.com/api/v1/checkin/YOUR-TIMER-UUID
        send_resolved: false

Sizing the timer

Set the check-in timer's interval to at least max interval + firing duration + your alerting pipeline's latency. With the defaults (silent gaps of at most two hours, plus 10 minutes of firing), a three-hour timer is comfortable.

License

Apache-2.0. See LICENSE.