Observability of arroyopy

[dylanmcreynolds]

We would love to have a UI that lets us observe what's the arroyopy blocks are doing. I'm thinking of metrics similar to what prometheus and Jaeger provide.

First step requirements:

• An indicator that the operator is running
• A graph of messages per second over time
• A graph of time per message over time
• A toggle to pause/restart operation
• A remotely viewable log

There are a lot of technology choices on the backend for accomplishing this. The two graphs are very similar to the metrics that prometheus stores and graphana displays in dashboards.

The viewable log is a finch component, or at least something that @SeijDeLeon has built into the Queue Server UI.

So, let's try this! I propose creating a PR that both @alisonscheng and I work on.

On the backend:

• Add OpenTelemetry APIs to listener, operator, publisher classes that store metrics for each.
• Add configuration in the block yaml files for prometheus
• Add a new block level end point for toggling pause/unpause-
• Add endpoint with hearbeat information and MAYBE simple metrics in case we don't want to go through prometheus...make this simple to install and use

One the front end:

• Well, create a UI for this repo that talks to those backends.
• We will have to think real hard about the design with this requirement: it would be nice to have an observability UI for arroyo blocks, and the basic components should be in this repo, and testable here. But in the wild, a beamline might be using N blocks of arroyo things, so the a ui that shows multiple at the same time will be nice. But that's not MVP.