Avoid detached ORM objects in queued train-predict forecasting jobs

[BelhsanHmida]

Problem

Queued train-predict forecasting jobs can cross database session boundaries with ORM-backed state attached to the TrainPredictPipeline instance.

This was discussed in #2034, where the immediate symptom was that the forecaster DataSource was not committed before the worker tried to load it. A related issue appeared after committing: ORM-backed sensors and regressors could become expired or detached before RQ serialized/executed the job.

Related history

• Fix detached DataSource and expired sensors when enqueuing forecast jobs #2034 investigated the detached DataSource / expired sensor problem.

  • It proposed reattaching ORM objects with db.session.merge() / refresh-style handling.
  • It was closed after the immediate fix was implemented elsewhere.

• Fix/small forecasting pipeline fixes #2011 merged the first/simple fix.

  • It committed the data source before forecasting jobs were created.
  • It also fixed wrap-up job handling and returned job metadata.
  • This fixed the immediate data_source not in database failure, but did not fully address the deeper issue of queueing ORM-backed pipeline state.

• Fix/detached forecasting sensors #2035 proposes the deeper fix.

  • It changes queued train-predict cycle jobs to pass serialized config/parameters plus data_source_id.
  • It reconstructs the pipeline inside the worker and loads sensors/data source through the worker's active DB session.
  • It adds regression coverage that queued job kwargs do not contain ORM instances.

Desired outcome

Queued forecasting jobs should not pickle or transport session-bound SQLAlchemy objects between the web/API process and worker process.

RQ payloads should remain plain, serializable data, and workers should load ORM objects within their own active DB session.

Acceptance criteria

• Forecast jobs triggered through the API can be enqueued and executed without detached DataSource, target sensor, save sensor, or regressor sensor failures.
• RQ cycle job kwargs contain no ORM model instances.
• Worker-side pipeline execution loads all required ORM objects through the worker DB session.
• Regression tests cover the queued-job payload shape and async forecasting path.