Support discovery packets (00 00 control bytes) in frame parsing

.vscode/tasks.json

@@ -65,6 +65,32 @@
                 "close": false
             },
             "problemMatcher": []
+        },
+        {
+            "label": "Framing: run",
+            "type": "shell",
+            "command": "gcc -I. -I.. -o run_framing test_framing.c log_capture.c && ./run_framing; ec=$?; rm -f ./run_framing; exit $ec",
+            "options": { "cwd": "${workspaceFolder}/test" },
+            "group": "test",
+            "presentation": {
+                "reveal": "always",
+                "panel": "shared",
+                "close": false
+            },
+            "problemMatcher": []
+        },
+        {
+            "label": "Framing: bless goldens",
+            "type": "shell",
+            "command": "gcc -I. -I.. -o run_framing test_framing.c log_capture.c && ./run_framing --bless; ec=$?; rm -f ./run_framing; exit $ec",
+            "options": { "cwd": "${workspaceFolder}/test" },
+            "group": "test",
+            "presentation": {
+                "reveal": "always",
+                "panel": "shared",
+                "close": false
+            },
+            "problemMatcher": []
         }
     ],
     "inputs": [

CHANGELOG.md

@@ -24,6 +24,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Note about CMD 0x12 observed with payload 0x01 0x00. Added to PROTOCOL.md and new sample trace.
 - Parsing for CMD 0x12 updated so `0x01 0x00` no longer reported as unexpected.
 - Full decoding of CMD 0x12 status byte for Gas Heaters (HiNRG `0x0072` and ICI (`0x0074`))
+- Rewrote frame level parser to use a sliding window for better resync on errors
+- Track frame resync events per failure type (no-start, header/data checksum, control, length, end-byte, overflow) and show the breakdown on the status page, separate from message-level decode errors
+- Unknown Messages page now tags each captured frame with its specific reason (no start, overflow, header/data checksum, length, bad framing) instead of a generic "error" chip
+- **Breaking (HTTP):** `/status` JSON's `message_counts.errors` (per-type protocol error counters) and `message_counts.error_detail` (introduced in 1.6.0-rc1) are removed, replaced by the new per-type `resyncs` object (`total`, `no_start`, `header_checksum`, `bad_control`, `bad_length`, `bad_end`, `data_checksum`, `buffer_overflow`) emitted by the sliding-window frame parser. Any external consumer reading the old fields needs to switch to `resyncs`.
+- Frame parser and decoder now also accept "discovery" packets (control bytes `0x00 0x00`): a shorter 11-byte frame shape with no payload and no data-checksum byte, alongside the existing `0x80 0x00` data packets
 ### Removed
 ### Fixed
 ### Deprecated

PROTOCOL.md

@@ -59,7 +59,7 @@ All messages follow this basic structure:
 | 0      | START           | Always `0x02`                                                     |
 | 1-2    | SOURCE          | Source device address (big endian)                                |
 | 3-4    | DEST            | Destination device address (big endian)                           |
-| 5-6    | CONTROL         | Control bytes (typically `0x80 0x00`)                             |
+| 5-6    | CONTROL         | Control bytes (`0x80 0x00` or `0x00 0x00`)                        |
 | 7      | COMMAND         | Command byte (message type)                                       |
 | 8      | LENGTH          | Total message length in bytes (including START and END bytes)     |
 | 9      | HEADER_CHECKSUM | Sum of bytes 0–8, masked to 8 bits (`sum(bytes[0..8]) & 0xFF`)    |

README.md

@@ -244,6 +244,8 @@ From VS Code inside the devcontainer: **Cmd+Shift+P → "Tasks: Run Task"** list
 - **Replay: all samples** — just the replay step.
 - **Replay: bless all samples** — rewrite expected output in every sample using the current decoder's output. Use after intentional decoder/logging changes, then review `git diff test/samples/` before committing.
 - **Replay: single file** / **Replay: bless single file** — same but prompted for a single path.
+- **Framing: run** — exercise the sliding-window frame parser against the goldens in `test/frames/`.
+- **Framing: bless goldens** — regenerate `test/frames/*_output.txt` from the current parser. Use after intentional framing changes, then review `git diff test/frames/` before committing.
 
 ### Adding a regression sample
 
@@ -252,6 +254,19 @@ From VS Code inside the devcontainer: **Cmd+Shift+P → "Tasks: Run Task"** list
 3. Run **Replay: bless single file** against it — this normalises the expected output to the current decoder.
 4. Review with `git diff`, then commit. The file is now a regression test.
 
+### Frame parser (sliding window) tests
+
+The frame parser is tested separately from the decoder: each line of `test/frames/observed_error_frames.txt` (real bus captures) and `test/frames/synthetic_errors.txt` (synthetic, one per failure mode) is fed to the parser, and the ordered stream of resync events (`Resync:  <type>`) and decoded frames is diffed against the matching `--- Case N ---` block in the `*_output.txt` golden. On a mismatch the failure prints the source location as `path:line`.
+
+The goldens are blessable from the CLI, mirroring replay:
+
+```bash
+cd test
+gcc -I. -I.. -o run_framing test_framing.c log_capture.c && ./run_framing --bless; rm -f run_framing
+```
+
+After blessing, review `git diff test/frames/` — the diff is the exact record of how parser behaviour changed.
+
 ### Improving "Unhandled" message logging
 
 When the decoder gains support for a previously-unknown command, replaying old samples surfaces it as a mismatch (e.g. `Unhandled CMD=0xXX` → real decode). Bless the affected samples and the diff in git is the exact documentation of what improved.

main/CMakeLists.txt

@@ -11,6 +11,7 @@ idf_component_register(
         "mqtt_commands.c"
         "web_handlers.c"
         "tcp_bridge.c"
+        "framing.c"
         "message_decoder.c"
         "register_requester.c"
         "unknown_buffer.c"

main/framing.c

@@ -0,0 +1,225 @@
+#include "framing.h"
+#include "esp_log.h"
+#include <string.h>
+#include <stdio.h>
+
+static const char *TAG = "BUS_FRAMING";
+
+// Header validation only ever inspects bytes 0-9 (see layout in framing.h);
+// cap early-failure captures to that window rather than the whole remaining
+// buffer. Otherwise the captured bytes (and length) vary with how much
+// trailing valid data happens to be queued, which both pollutes the
+// unknown-message diagnostics with unrelated trailing frames and defeats
+// dedup in unknown_buffer_record (which keys on reason + len + raw bytes).
+#define FRAMING_HEADER_CAPTURE_LEN 10
+
+// Format up to `cap` bytes of `data` as a space-separated hex string into
+// `hex_str` (size `hex_str_size`). Does not append a truncation marker —
+// callers that care whether `len > cap` add their own suffix to the log line.
+static void framing_format_hex_dump(const uint8_t *data, int len, int cap, char *hex_str, size_t hex_str_size)
+{
+    int hex_pos = 0;
+    int dump_len = MIN(len, cap);
+    for (int i = 0; i < dump_len && hex_pos < (int)hex_str_size - 3; i++) {
+        hex_pos += snprintf(&hex_str[hex_pos], hex_str_size - hex_pos, "%02X ", data[i]);
+    }
+    hex_str[hex_pos] = '\0';
+}
+
+// Capture `capture_len` bytes from the front of `fb` into out_frame/out_len
+// for diagnostics, then resync by discarding a single byte and shifting the
+// rest of the buffer down. The per-byte memmove is O(n), so draining a full
+// garbage buffer one byte at a time is O(n^2) - acceptable only because
+// BUS_MESSAGE_MAX_SIZE is small (256B).
+static void framing_resync_one_byte(framing_buffer_t *fb, uint8_t *out_frame, int *out_len, int capture_len)
+{
+    if (out_frame) {
+        memcpy(out_frame, fb->buffer, capture_len);
+    }
+    if (out_len) {
+        *out_len = capture_len;
+    }
+    memmove(fb->buffer, &fb->buffer[1], fb->len - 1);
+    fb->len--;
+}
+
+framing_packet_type_t framing_classify_packet(uint8_t ctrl_hi, uint8_t ctrl_lo)
+{
+    if (ctrl_hi == 0x80 && ctrl_lo == 0x00) {
+        return FRAMING_PACKET_DATA;
+    }
+    if (ctrl_hi == 0x00 && ctrl_lo == 0x00) {
+        return FRAMING_PACKET_DISCOVERY;
+    }
+    return FRAMING_PACKET_INVALID;
+}
+
+void framing_init(framing_buffer_t *fb)
+{
+    if (fb) {
+        fb->len = 0;
+    }
+}
+
+bool framing_add_bytes(framing_buffer_t *fb, const uint8_t *data, int len)
+{
+    if (!fb || !data || len <= 0) {
+        return false;
+    }
+    if (fb->len + len > BUS_MESSAGE_MAX_SIZE) {
+        return false;
+    }
+    memcpy(&fb->buffer[fb->len], data, len);
+    fb->len += len;
+    return true;
+}
+
+framing_result_t framing_process_next(framing_buffer_t *fb, uint8_t *out_frame, int *out_len)
+{
+    if (!fb) {
+        return FRAMING_NEED_MORE_DATA;
+    }
+
+    // Need at least minimum message: START + SRC + DST + CTRL + CMD + LEN + HCHK + END = 11 bytes
+    // (the smallest valid frame has zero payload bytes and no separate data
+    // checksum byte - see the msg_len == 11 special case below)
+    if (fb->len < 11) {
+        return FRAMING_NEED_MORE_DATA;
+    }
+
+    // Find start byte (0x02)
+    int start_idx = -1;
+    for (int i = 0; i < fb->len; i++) {
+        if (fb->buffer[i] == 0x02) {
+            start_idx = i;
+            break;
+        }
+    }
+
+    // No start byte found - discard everything
+    if (start_idx == -1) {
+        char hex_str[100];
+        framing_format_hex_dump(fb->buffer, fb->len, 32, hex_str, sizeof(hex_str));
+        ESP_LOGW(TAG, "No start byte in buffer, discarding %d bytes: %s%s",
+                 fb->len, hex_str, fb->len > 32 ? "..." : "");
+        // Hand the discarded bytes back so the caller can capture them.
+        if (out_frame) {
+            memcpy(out_frame, fb->buffer, fb->len);
+        }
+        if (out_len) {
+            *out_len = fb->len;
+        }
+        fb->len = 0;
+        return FRAMING_NO_START_BYTE;
+    }
+
+    // Discard bytes before start. This happens silently with respect to the
+    // resync counters/unknown buffer (those only track the byte-stepped BAD_*
+    // resyncs and the no-start-at-all case) - log it so a multi-byte jump like
+    // this is still visible somewhere.
+    if (start_idx > 0) {
+        char hex_str[100];
+        framing_format_hex_dump(fb->buffer, start_idx, 32, hex_str, sizeof(hex_str));
+        ESP_LOGW(TAG, "Discarding %d byte(s) before next start byte: %s%s",
+                 start_idx, hex_str, start_idx > 32 ? "..." : "");
+        memmove(fb->buffer, &fb->buffer[start_idx], fb->len - start_idx);
+        fb->len -= start_idx;
+    }
+
+    // Check we have enough for header validation (10 bytes: 0..9)
+    if (fb->len < 10) {
+        return FRAMING_NEED_MORE_DATA;
+    }
+
+    // Validate header checksum: byte 9 must be sum(bytes 0-8) & 0xFF
+    // We test checksum first, and only then the other header bytes - this way,
+    // a bad length will only survive if the checksum is coincidentally correct.
+    uint8_t expected_hchk = 0;
+    for (int i = 0; i < 9; i++) {
+        expected_hchk += fb->buffer[i];
+    }
+    if (fb->buffer[9] != expected_hchk) {
+        char hex_str[100];
+        framing_format_hex_dump(fb->buffer, fb->len, 32, hex_str, sizeof(hex_str));
+        ESP_LOGW(TAG, "Invalid header checksum: calculated %02X, got %02X, data: %s%s, resyncing by 1 byte",
+                 expected_hchk, fb->buffer[9], hex_str, fb->len > 32 ? "..." : "");
+        framing_resync_one_byte(fb, out_frame, out_len, MIN(fb->len, FRAMING_HEADER_CAPTURE_LEN));
+        return FRAMING_BAD_HEADER_CHECKSUM;
+    }
+
+    // Validate control bytes (positions 5-6) and classify the packet type
+    framing_packet_type_t packet_type = framing_classify_packet(fb->buffer[5], fb->buffer[6]);
+    if (packet_type == FRAMING_PACKET_INVALID) {
+        ESP_LOGW(TAG, "Invalid control bytes: %02X %02X (expected 80 00 or 00 00), resyncing by 1 byte",
+                 fb->buffer[5], fb->buffer[6]);
+        framing_resync_one_byte(fb, out_frame, out_len, MIN(fb->len, FRAMING_HEADER_CAPTURE_LEN));
+        return FRAMING_BAD_CONTROL_BYTES;
+    }
+
+    // Read length from byte 8. Discovery packets are always exactly 11 bytes
+    // (header + END, no payload, no data checksum); data packets are always
+    // at least 12 (header + data-checksum byte + END, plus any payload).
+    int msg_len = fb->buffer[8];
+    if (packet_type == FRAMING_PACKET_DISCOVERY) {
+        if (msg_len != 11) {
+            ESP_LOGW(TAG, "Invalid length field for discovery packet: 0x%02X (%d), expected 0x0B (11), resyncing by 1 byte",
+                     msg_len, msg_len);
+            framing_resync_one_byte(fb, out_frame, out_len, MIN(fb->len, FRAMING_HEADER_CAPTURE_LEN));
+            return FRAMING_BAD_LENGTH;
+        }
+    } else {
+        if (msg_len < 12 || msg_len > BUS_MESSAGE_MAX_SIZE) {
+            ESP_LOGW(TAG, "Invalid length field for data packet: 0x%02X (%d), expected >= 0x0C (12), resyncing by 1 byte",
+                     msg_len, msg_len);
+            framing_resync_one_byte(fb, out_frame, out_len, MIN(fb->len, FRAMING_HEADER_CAPTURE_LEN));
+            return FRAMING_BAD_LENGTH;
+        }
+    }
+
+    // Check if we have received the full message yet
+    if (fb->len < msg_len) {
+        return FRAMING_NEED_MORE_DATA;
+    }
+
+    // Verify end byte (0x03)
+    if (fb->buffer[msg_len - 1] != 0x03) {
+        ESP_LOGW(TAG, "Invalid end byte (expected 03, got %02X) at length %d, resyncing by 1 byte",
+                 fb->buffer[msg_len - 1], msg_len);
+        framing_resync_one_byte(fb, out_frame, out_len, msg_len);
+        return FRAMING_BAD_END_BYTE;
+    }
+
+    // Verify data checksum: sum(bytes 10..msg_len-3) & 0xFF.
+    // Discovery packets have no data-checksum byte at all (see above), so
+    // there's nothing to verify for them.
+    if (packet_type == FRAMING_PACKET_DATA) {
+        uint32_t data_sum = 0;
+        for (int i = 10; i < msg_len - 2; i++) {
+            data_sum += fb->buffer[i];
+        }
+        uint8_t calculated_dchk = data_sum & 0xFF;
+        if (fb->buffer[msg_len - 2] != calculated_dchk) {
+            ESP_LOGW(TAG, "Invalid data checksum: calculated %02X, got %02X, resyncing by 1 byte",
+                     calculated_dchk, fb->buffer[msg_len - 2]);
+            framing_resync_one_byte(fb, out_frame, out_len, msg_len);
+            return FRAMING_BAD_DATA_CHECKSUM;
+        }
+    }
+
+    // Valid frame — copy out before consuming from buffer
+    if (out_frame) {
+        memcpy(out_frame, fb->buffer, msg_len);
+    }
+    if (out_len) {
+        *out_len = msg_len;
+    }
+
+    // Remove processed message from buffer
+    int remaining = fb->len - msg_len;
+    if (remaining > 0) {
+        memmove(fb->buffer, &fb->buffer[msg_len], remaining);
+    }
+    fb->len = remaining;
+
+    return FRAMING_FRAME_READY;
+}

main/framing.h

@@ -0,0 +1,66 @@
+#ifndef FRAMING_H
+#define FRAMING_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include "config.h"
+
+typedef enum {
+    FRAMING_NEED_MORE_DATA,        // incomplete, wait for more bytes
+    FRAMING_NO_START_BYTE,         // no 0x02 found, buffer cleared
+    FRAMING_BAD_HEADER_CHECKSUM,   // header checksum mismatch, resynced by 1
+    FRAMING_BAD_CONTROL_BYTES,     // bytes 5-6 not 0x80 0x00 or 0x00 0x00, resynced by 1
+    FRAMING_BAD_LENGTH,            // length field out of range, resynced by 1
+    FRAMING_BAD_END_BYTE,          // end byte not 0x03, resynced by 1
+    FRAMING_BAD_DATA_CHECKSUM,     // data checksum mismatch, resynced by 1
+    FRAMING_FRAME_READY,           // valid frame written to out_frame
+} framing_result_t;
+
+// The control bytes (offsets 5-6) determine the shape of the rest of the
+// frame. A data packet (0x80 0x00) always carries a data-checksum byte
+// (offset N-2), even with zero payload bytes, so its minimum length is 12.
+// A discovery packet (0x00 0x00) carries no payload and no separate
+// data-checksum byte at all - it's exactly 11 bytes (header + END), and
+// byte 9 is purely the header checksum. Shared between the reassembly layer
+// (framing.c) and the decoder (message_decoder.c), which both need to branch
+// on this classification rather than re-deriving it from raw control bytes.
+typedef enum {
+    FRAMING_PACKET_DATA,       // control bytes 0x80 0x00
+    FRAMING_PACKET_DISCOVERY,  // control bytes 0x00 0x00
+    FRAMING_PACKET_INVALID,    // anything else
+} framing_packet_type_t;
+
+/**
+ * Classify a frame's control bytes (offsets 5-6) into a packet type.
+ */
+framing_packet_type_t framing_classify_packet(uint8_t ctrl_hi, uint8_t ctrl_lo);
+
+// Context structure holding the reassembly buffer
+typedef struct {
+    uint8_t buffer[BUS_MESSAGE_MAX_SIZE];
+    int len;
+} framing_buffer_t;
+
+/**
+ * Initialize a framing buffer context.
+ */
+void framing_init(framing_buffer_t *fb);
+
+/**
+ * Add bytes to the framing buffer.
+ * Returns true if bytes were successfully added, false if buffer is full.
+ */
+bool framing_add_bytes(framing_buffer_t *fb, const uint8_t *data, int len);
+
+/**
+ * Extract one complete message from the reassembly buffer.
+ * On FRAMING_FRAME_READY the frame is written to out_frame and out_len; on
+ * every other resync result (NO_START / BAD_*) the bytes that triggered the
+ * resync are written there instead, so the caller can capture them.
+ * FRAMING_NEED_MORE_DATA means stop and wait; all other results mean call
+ * again immediately.
+ */
+framing_result_t framing_process_next(framing_buffer_t *fb, uint8_t *out_frame, int *out_len);
+
+#endif // FRAMING_H