Properly centralize the rust CI workflow

[tlater-famedly]

Problem

We’re currently copying around our main workflow and making small changes to it in every repository. This is causing drift and makes it hard to make larger changes without creating 20+ PRs, and generally results in inconsistent approaches to project testing, which makes maintenance harder.

While switching to a monorepo approach could help with this, and appears to be how GitHub’s workflows/actions are designed anyway, it’s not a full solution; we will continue to have public projects which need some kind of clear, standardized CI flow that offers everything the projects need.

This issue is starting to cause real problems in production, as it has made us reluctant to perform proper e2e tests in repositories that haven't yet copied the main rust workflow and re-implemented it; current example is a change to tim-aggregator which did not end up working as expected in real deployments, but we have had similar issues with tim-proxy and victoriametrics in the past. Usually these issues crop up when we are under time pressure to deliver something, which isn't ideal.

We should put a centralized workflow in place which actually covers our needs in practice, and isn't only capable of running integration/unit tests. We have the following design scope:

---

Limitations of GitHub CI

GitHub CI puts some interesting limitations on our design scope here:

• The test hosts are not very configurable; it’s a fairly vanilla ubuntu or leave it
  • Theoretically we could self-host runners as an alternative
• Workflows are permitted to run their jobs in containers, which is the intended way to configure the build system
  • The limitation here is that we are then confined to a container; starting other containers (for e.g. databases) is only possible through GitHub’s built-in service provisioning, which can then no longer be shared with other contexts in which we run our tests, and therefore results in lots of code duplication at best
  • Container images will also still be downloaded onto the host every time the CI starts. This means that we should try to minimize our image sizes as much as possible, and publish them into the GitHub registry to maximize download speeds.
• Depending on other workflows or actions in the same repository is impractical
• Public/private secrets are problematic, as our workflow repository must be public so that it can be used in public repositories.
  • This mostly means that testing our workflows in CI is quite hard.

Requirements

We want our workflows to perform the following:

• Various general purpose lints
  • editorconfig compliance
  • Typos
  • General markup language lints
  • Shellcheck
  • Whitespace checks
  • …
• Various rust-specific lints
  • rustfmt
  • cargo udeps
  • cargo clippy
• Tests with varying host dependency requirements
  • We use cargo nextest to define these; it supports creating test groups
  • Notably, we have:
    • Unit tests
      • Tests which perform simple sanity checks on individual functions and modules
    • Integration tests
      • Tests which assert wider functionality across fully composed modules, but use mocks, and perhaps create test files as their only host dependencies
    • e2e tests
      • Tests which run the full project, and link it up against real dependent services to assert functionality against real API requests
      • We have a separate repository for tests like this, but it has proven useful to have e2e tests scoped specifically to individual components, which don't need to set up the entire backend, as well

We also have some less functional requirements:

• CI should be reactive, giving any results it can as quickly as possible
  • CI ultimately interfaces with humans - even if we are developers, interfacing with humans means that you want to limit the latency of your responses as much as possible. Not having to think about when CI is going to finish, let alone babysitting a pipeline with repeated small mistakes, frees up mental cycles.
• CI workflows should be reasonably split
  • It’s really stupid to wait for a CI job for 20 minutes only for rustfmt to tell you you accidentally added a newline somewhere right before the actual test starts.
  • This can be alleviated somewhat by more conscientiously running checks locally, but often local tooling doesn't match tooling in repositories perfectly (notably, rustfmt often mismatches, since we use nightly), and spending time running checks locally means you're not doing actual work - trivial tasks can and should be farmed out to CI, as long as developers can still trivially execute them locally when they need to figure out a problem.
• Workflows should not be split so much that individual workflows typically take less than one minute (not taking into account the image download)
  • GitHub CI charges with minute resolution, spinning up very small jobs can be wasteful.
• We should avoid duplicating builds
  • This is mostly for performance reasons, but I point it out specifically because in a lot of repositories currently we build our project once to run tests on the host, and then once again to build it into a container. This is just wasteful.
• We should reuse our workflows instead of redefining them everywhere
• All of this should tie into our CD story; CI should be able to deploy our code into production
  • We should still be able to run tests and deployments locally
  • Not all testing should be performed in a production environment; unit and integration tests provide a lot of useful feedback with much less latency

[tlater-famedly]

I've written up my idea for a solution to this here: https://www.notion.so/famedly/Explicitly-marking-E2E-tests-so-that-they-can-be-exported-and-executed-in-a-non-host-environment-1b64c3a9792080f09037c33d63f670c8?pvs=4

See this experimental commit/PR for how it would look in practice: 9d2800b

[sirewix]

We should use pre-commit/action: a GitHub action to run pre-commit

[tlater-famedly]

We should use pre-commit/action: a GitHub action to run pre-commit

Yes, but we should do more than just that, there are quite a few checks we should perform as part of the linting job that we just don't right now.

It's all a lil' pointless if all other repositories that do anything meaningful write their own workflows, though.