Skip to content

ServerlessBackend creates a new W&B run per training step, causing "Run finished" alerts to fire on every step #638

@go5paopao

Description

@go5paopao

Hi, I ran into an issue while training with ServerlessBackend.

Problem

When using ServerlessBackend, each call to backend.train() creates a separate W&B run. This means that if "Run
finished" Slack notifications are enabled in W&B User Settings, a notification is sent on every training step — 50
steps means 50 Slack notifications, which is extremely noisy in practice.

Reproduction

for step in range(NUM_STEPS):
    finished = await art.gather_trajectory_groups(groups)
    result = await backend.train(model, finished, learning_rate=1e-5)
    # ↑ Each call creates and finishes a separate W&B run

Expected behavior

One of the following would address this:

  • Consolidate the entire training session (model.register() through backend.close()) into a single W&B run
  • Handle per-step runs in a way that does not trigger W&B alerts (e.g. via wandb.Settings)
  • Provide a user-configurable option to control this behavior

Workaround

Currently the only option is to disable "Run finished" notifications globally in W&B User Settings, but this silences
notifications for all projects — not ideal. W&B does not currently support per-project alert configuration.

Thanks for the great library!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions