FR: Variance-adjusted dynamic weighting for days_previous load history

[chriswright2705]

The current days_previous weighting scheme assigns weights purely as a function
of day position — day 1 carries the highest weight (e.g. 1.00) decaying to a
lower value at day N. This creates two related problems:

1. A single corrupt or anomalous day at day 1 (caused by a sensor dropout,
   inverter counter reset after a power interruption, or GivTCP data gap)
   carries maximum weight in the forecast regardless of how implausible its
   value is. A day with near-zero recorded load — when true load was 25+ kWh —
   pulls the weighted average sharply downward and suppresses overnight load
   forecasts for the same time-of-day slots in subsequent planning cycles.
   load_scaling cannot correct this because it is a forward multiplier
   (1.5 x 0 = 0).

2. The existing load_filter_modal switch discards only the single lowest
   historical day, with no equivalent protection against anomalously high
   days (e.g. from sensor overcounting). The weighting is asymmetric.

The consequence on affected installations is that Predbat underestimates
overnight load, plans insufficient battery reserve, and may discharge to
minimum SoC before dawn — incurring grid import at peak rates.

---

Describe the solution you'd like.

A variance-adjusted dynamic weighting mode where each day's weight is a
function of both its recency (current behaviour) and its consistency with
the central tendency of the full historical window.

Proposed mechanism:

1. At each planning cycle, compute the median of all days_previous daily
   load totals as the central tendency reference.

2. For each day, compute its relative deviation from the median:
   deviation = abs(day_total - median) / median

3. Map the deviation to a damping factor using a Huber-style function:

   • deviation <= threshold_low (e.g. 0.30): damping factor = 1.0 (full weight)
   • threshold_low < deviation <= threshold_high (e.g. 0.80): linear taper
     from 1.0 to 0.0
   • deviation > threshold_high: damping factor = 0.0 (excluded)

4. Final weight for each day = recency_weight x damping_factor

5. Thresholds configurable in apps.yaml with sensible defaults.

This approach:

• Is self-correcting: a corrupt zero day reduces its own weight automatically
• Is symmetric: handles both low outliers (dropouts) and high outliers
  (overcounting) without separate mechanisms
• Handles partial corruptions (suppressed but non-zero values) gracefully —
  a day at 30% of normal load gets proportionally reduced weight rather than
  binary include/exclude treatment
• Requires no external scripts or manual intervention
• Subsumes the load_filter_modal asymmetry limitation

---

Describe alternatives you've considered.

1. load_scaling: a forward multiplier. Cannot correct stored zero values.
   Applies uniformly across all slots including daytime where the forecast
   may already be accurate, causing overestimation.

2. days_previous exclusion via apps.yaml: effective but requires external
   detection logic, scheduled revert, and careful Python scripting to avoid
   file corruption. A maintenance burden on the user.

3. load_filter_modal (current): discards only the single lowest day.
   Asymmetric, binary, and limited to one outlier regardless of window size.

4. External pre-processing script: implement the variance-adjusted weight
   computation externally and write the resulting weight list to
   days_previous_weight in apps.yaml every planning cycle. Functional but
   inelegant — the logic belongs inside Predbat where it has direct access
   to the historical data.

---

Additional context.

The failure mode this addresses is reproducible and occurs on any installation
subject to brief mains supply interruptions (3-10 seconds) that cause the
Raspberry Pi running HA to restart uncleanly, leaving the SQLite recorder
database in an unfinished state. The GivTCP load sensor then shows zero or
near-zero accumulation for the affected day. With a days_previous window of
10 days and day 1 weight 1.00, a single such event suppresses overnight load
forecasts for up to 10 days.

The proposed Huber damping function is well-established in robust statistics
for exactly this use case — preserving the influence of consistent data while
reducing the influence of outliers without hard exclusion. It is more
statistically principled than a trimmed mean (which discards fixed counts
regardless of deviation magnitude) and more resilient than pure recency
weighting.

Suggested new apps.yaml parameters:

Enable variance-adjusted dynamic weighting (default: false)

load_variance_weighting: true

Huber damping thresholds (relative deviation from median)

load_variance_threshold_low: 0.30 # Below this: full weight
load_variance_threshold_high: 0.80 # Above this: zero weight

These parameters would be optional, defaulting to current behaviour when
not set, ensuring full backward compatibility.