ESP32‑P4 port: crash in add_idct8x8 when decoding High Profile streams

[youkorr]

Hello edge264 team,
I’m porting edge264 to ESP32‑P4 (RISC‑V, 400 MHz dual‑core) to decode H.264 High Profile streams from IP cameras in an embedded project.I’ve compiled edge264 myself for ESP32‑P4 and integrated it as a C++ wrapper component in an ESPHome‑based project.
Project context

My project: https://github.com/youkorr/ip-camera-viewer (branch claude/test-coverage-analysis-2viujv)
I compiled edge264 for ESP32‑P4 using [brief description of your toolchain/flags, e.g., ESP‑IDF 5.x, RISC‑V toolchain, -Os, etc.].
The decoder is wrapped in a C++ class (H264HpDecoder) that calls edge264_alloc, edge264_decode_NAL, edge264_get_frame, etc.
Memory is allocated in PSRAM via heap_caps_aligned_alloc with 64‑byte alignment (see psram_alloc_cb below).
I’m using single‑thread mode (n_threads=0) to avoid FreeRTOS stack issues.

Relevant code snippets:

// h264_hp_decoder.cpp
void *psram_alloc_cb(size_t size, size_t align, void *user) {
size_t effective_align = (align < 64) ? 64 : align;
return heap_caps_aligned_alloc(effective_align, size, MALLOC_CAP_SPIRAM);
}

bool H264HpDecoder::begin(int n_threads) {
// Force single-thread for stability
n_threads = 0;
dec_ = edge264_alloc(n_threads, psram_alloc_cb, psram_free_cb, nullptr, nullptr);
// ...
}

Problem
When decoding a High Profile stream (profile_idc=100, level_idc=51), the ESP32‑P4 crashes inside add_idct8x8:

PC: 0x400D9A82 → add_idct8x8 at edge264.c
Reason: Fault - Unknown
Crashed core: 1

Baseline profiles work fine. The crash happens after the first IDR frame is decoded.
What I’ve tried

Forcing n_threads=0 in edge264_alloc.
Using PSRAM with 64‑byte alignment for sample and MB buffers.
Increasing FreeRTOS task stack sizes.
Adding argument checks before calling edge264_decode_NAL.
Testing with a simpler High Profile stream (low resolution, low bitrate) — same crash.

Questions

Are there any known issues or recommendations for running edge264 on RISC‑V targets, especially regarding IDCT 8×8 and High Profile decoding?
Are there specific alignment requirements for buffers passed to edge264_decode_NAL / add_idct8x8 that I might be missing?
Could the crash be related to assumptions about SIMD instructions or cache behavior that don’t hold on ESP32‑P4?
Do you have any suggestions for debugging add_idct8x8 on a non‑x86/ARM64 target?

I’m happy to share my build configuration, memory layout, and more detailed logs if that helps.
Thanks for your work on edge264 — it’s a great decoder, and I’d love to make it work reliably on ESP32‑P4!

[traffaillac]

Hi @youkorr ,

You are actually the first known user on a RISC-V CPU, I am super happy it could work at least in baseline!!! I'll be very curious to see how well it performs vs other decoders once we fix the defects.

RISC-V normally uses the default backend using Clang vector extensions. I guess you are using Clang, otherwise this is a use case I am not aware of. The two major constraints to use edge264 are:

• to align video memory to 16 bytes (anything higher makes no difference), with normally absolutely no constraint on buffers passed to decode_NAL
• to be little-endian (big endian is actually too much hassle for little actual use)

This crash is weird, because IDCT code is generally the simplest, with no unaligned accesses or fancy vector shuffling. It could be related to legacy assumptions indeed, typically alignment and possibly cache too. However something that happens quite often in my experience is a silent bug happening earlier, that makes the decoding collapse later.

to help you debug this:

• strip your input video to a small sequence (1~5 s) that exhibits the crash (ffmpeg can do that)
• if you have access to a x86/ARM device, decode it using edge264_test -V video.264 to get a full trace log
• run the same command on your ESP32-P4 device, then diff the two traces to find the actual moment they differ
• you may also try compiling with CFLAGS=-g then using debugging with lldb to find the exact point in code where the crash happens

Feel free to send me the trace logs, and line of crash obtained with lldb, my mail is on https://traffaillac.github.io/

Cheers,
Thibault

[youkorr]

Hi @thibault, thanks a lot for the detailed answer — it matched our findings exactly,
and the good news is we're now very close to finalizing a working RISC-V / ESP32-P4
port. Baseline already runs; High Profile is almost there.

Quick confirmations first: yes, Clang only; the ESP32-P4 is little-endian; and our video
buffers are 16-byte aligned (we allocate the decoder's samples+mbs as a single 64-byte
aligned PSRAM block to honor internal_alloc's *mbs = *samples + samples_size contract).

We found a genuine bug in the CLANG (generic) backend, plus a debugging gotcha worth
documenting:

1. pow2x4 is incorrect on the CLANG backend (affects all RISC-V users)

In the SIMD == CLANG path:

static u32x4 pow2x4(i32x4 a) { return (u32x4)set32(1) << a; }

The doc says pow2x4 must yield “2^element, or zero if the element is negative”, and the
SSE/NEON paths get that for free (_mm_sllv_epi32 / vshlq_u32 saturate out-of-range
shifts to 0). But a plain C/LLVM vector shift by >= 32 or by a negative count is UB, so on
RISC-V pow2x4(255) returned 0x80000000 instead of 0. That corrupted refs_to_mask,
giving pic_width_in_mbs = 0 and a divide-by-zero / collapse later — exactly the “silent
bug earlier, crash later” you described.
Fix that works (and stays bit-exact vs SSE on our test vector):

static u32x4 pow2x4(i32x4 a) {
    return ((u32x4)set32(1) << (a & set32(31))) & (u32x4)((u32x4)a < (u32x4)set32(32));
}


Reproducible on x86 by forcing SIMD=CLANG.
2. qemu-user hides misalignment — real RISC-V hardware traps
We initially “validated” on qemu-riscv32, which silently emulates unaligned accesses, so
our tests passed while the real chip faulted. Building the actual rv32 target with
-fsanitize=alignment -fsanitize-trap=alignment and running under qemu reproduces what
the hardware does. With that, our 8 s clip decodes through with zero alignment traps,
bit-exact vs the x86/SSE build (same checksum).
3. Current status — the “silent earlier bug” on a live stream
Our 8 s capture (High Profile, CABAC, profile_idc=100, 640x480) decodes bit-exact on
x86-SSE and on the rv32/CLANG build under qemu. The live RTSP stream from the same
camera still collapses on-device: mid-decode, state in ctx->t (pic_width_in_mbs /
next_deblock_addr) gets clobbered and the per-MB / deblock loops run away (we caught a
25 s loopTask hang with MEPC inside deblock_mb, PC in flash → a real runaway, not
slowness). We’ve bounded those loops defensively so the watchdog can’t reboot, but that’s
treating the symptom — the root is the earlier silent corruption you mentioned, on
content our 8 s clip doesn’t contain. We’ll apply your trace-diff method next on a
crashing clip.
So we’re really not far. Once it’s solid, we’d be glad to contribute the RISC-V /
ESP32-P4 port back (the pow2x4 fix, the alignment-trap validation flow, and the build
recipe) so future RISC-V users get it out of the box. Happy to open a PR for the pow2x4
fix right away, and to email you the 8 s capture + patches — whatever is most useful.
Thanks again, this is the most helpful pointer we’ve had.

[youkorr]

Je viens de voire votre profile j'aurais pus écrire en français

[youkorr]

Update — we found the root cause, and it's exactly the "silent bug earlier, crash
later" you predicted. Thank you for that pointer, it set us on the right track.

Root cause: worker_loop does __builtin_ctz(ready_tasks) with ready_tasks == 0

The on-device crashes (watchdog reboots with the PC inside add_idct8x8 /
deblock_mb) were never really in those kernels — they were a corrupted task
context decoding garbage. Here is the chain:

• parse_slice_layer_without_partitioning sets a task's ready bit only when its
  reference frames are already decoded (edge264_headers.c, around):

  dec->ready_tasks |= ((dec->task_dependencies[task_id] & ~ready_frames(dec)) == 0) << task_id;

• A slice that references a frame missing from the DPB leaves ready_tasks == 0.
In single-thread mode (n_threads == 0) worker_loop is then still called
unconditionally:

else
dec->worker_loop(dec);

•	In worker_loop, with ready_tasks == 0:

assert((unsigned)c.d->ready_tasks - 1 < 65535); // compiled out with -DNDEBUG
int task_id = __builtin_ctz(c.d->ready_tasks); // ctz(0) is UNDEFINED
...
c.t = c.d->tasks[task_id]; // OOB read of a whole Edge264Task

•	ctz(0) returns a wild task_id, so c.t = c.d->tasks[task_id] copies ~1.6 KB

out of bounds and fills c.t (pic_width_in_mbs, next_deblock_addr, the
bitstream pointers…) with garbage. The decode then collapses a few macroblocks
later. With -DNDEBUG (our device build) the assert doesn’t catch it, so it’s
silent — matching your description perfectly.
Why it only happened on the live stream
Our 8 s file capture decodes bit-exact everywhere (x86-SSE and an rv32/CLANG
build under qemu, same checksum). The live RTSP stream loses packets, so FU-A
reassembly occasionally yields a slice that references a now-missing frame → the
path above. We reproduced it deterministically on x86 by truncating a single
slice NAL and decoding under AddressSanitizer with -DNDEBUG:

before: heap-buffer-overflow in worker_loop (the c.t = tasks[task_id] copy)
after : 400 truncated-NAL trials decode with no error

Fix we’re using
Guard worker_loop entry so it never does ctz(0):

if (__builtin_expect(!c.d->ready_tasks, 0)) {
if (c.thread_id < 0)
return (void *)0; // single-thread: nothing runnable (missing ref) — skip slice
continue; // multi-thread: go back and wait
}

This stops the corruption; clean streams are byte-for-byte unchanged
(FR=30 CK=... identical). Happy to send a PR.
Second finding: pow2x4 is wrong on the CLANG backend (all RISC-V users)

static u32x4 pow2x4(i32x4 a) { return (u32x4)set32(1) << a; }

The doc says “2^element, or zero if the element is negative”. SSE/NEON get that for
free (_mm_sllv_epi32 / vshlq_u32 saturate out-of-range shifts to 0), but a plain
C/LLVM vector shift by >= 32 or negative is UB, so on RISC-V pow2x4(255) returned
0x80000000 instead of 0 — corrupting refs_to_mask and giving pic_width_in_mbs = 0 (another early silent bug). Fix that stays bit-exact vs SSE:

static u32x4 pow2x4(i32x4 a) {
return ((u32x4)set32(1) << (a & set32(31))) & (u32x4)((u32x4)a < (u32x4)set32(32));
}

Reproducible on x86 with SIMD=CLANG.
Status
Baseline and High Profile now decode correctly on ESP32-P4, and the decoder no
longer collapses on lossy live input. We also validated alignment faithfully by
building the actual rv32 target with -fsanitize=alignment -fsanitize-trap=alignment
under qemu (the default qemu-user silently emulates unaligned accesses, which had
hidden real-hardware faults from us) — it now runs trap-free and bit-exact.
We’d be glad to open PRs for the two fixes above and contribute the RISC-V /
ESP32-P4 build recipe so future RISC-V users get it out of the box. Thanks again —
this is the most useful guidance we’ve had.

[traffaillac]

It's fine to write in English, as it makes it easier for other people to follow up in this thread later if needed :)

Damn you're right about the bug, my bad, I uploaded a fix that is a bit simpler (no need to mask out a, since to ChatGPT's knowledge :D there is no ISA that faults on invalid shift amount).

static u32x4 pow2x4(i32x4 a) {
  return (a >= 0) & (u32x4)set32(1) << a;
}

This weakness on bitfields has been recurring over the years, I am planning to work on formal verification in the following 6~12 months, so I am not putting more defensive programming at the moment. Please tell me if this works now or needs more attention.

Cheers,
Thibault

[youkorr]

Thanks for the quick fix! One thing though — I think the simpler version still
misses the case that actually bit us, and I have an rv32 test to show it.

The failing input isn't a negative a, it's a = 255: unused RefPicList slots
hold the 0xFF sentinel, and refs_to_mask feeds them through cvtlo8u16 /
cvtlo16u32 (unsigned), so pow2x4 receives 255 (positive). The problem isn't
faulting — you're right that no ISA faults on an out-of-range shift — it's that the
result differs:

• SSE _mm_sllv_epi32 and NEON vshlq saturate an out-of-range count to 0.
• RISC-V sll masks the count to its low 5 bits, so 1 << 255 == 1 << 31
  == 0x80000000, not 0.

So (a >= 0) & (set32(1) << a) leaves pow2x4(255) = 0x80000000 on RISC-V, which
sets a bogus bit in refs_to_mask → bogus task_dependencies → the task never
becomes ready.

rv32 result (runtime values, under qemu):

a=255 : (a>=0)&(1<<a) -> 0x80000000 // current fix
(1<<(a&31)) & (a<32) -> 0x00000000 // matches SSE/NEON
a=31 : both -> 0x80000000
a=-1 : both -> 0x00000000

Heads-up for anyone reproducing: with compile-time-constant inputs Clang folds
the out-of-range shift to 0, so it looks fine — you only see 0x80000000 with
runtime values.

End to end: with (a >= 0) & (set32(1) << a) our test clip's decode breaks; with
the a < 32 mask it's bit-exact (FR=30, identical checksum to x86-SSE). So I think
you still need to bound a to < 32 (or mask the result), e.g.:

static u32x4 pow2x4(i32x4 a) {
    return ((u32x4)set32(1) << (a & set32(31))) & (u32x4)((u32x4)a < (u32x4)set32(32));
}

The bigger fix, in case it got lost above: the actual on-device crashes
(add_idct8x8 / deblock_mb watchdog reboots) were a __builtin_ctz(ready_tasks)
with ready_tasks == 0 in worker_loop. A slice that references a frame missing
from the DPB (reference lost to RTP packet loss on a live stream, or a truncated
NAL) leaves ready_tasks == 0, but single-thread mode still calls worker_loop,
so ctz(0) returns a wild task_id and c.t = c.d->tasks[task_id] reads ~1.6 KB
out of bounds — filling c.t with garbage and collapsing the decode a few MBs
later (silent under -DNDEBUG). Reproduced on host with ASan by truncating one
slice NAL. We guard the entry:

if (__builtin_expect(!c.d->ready_tasks, 0)) {
    if (c.thread_id < 0) return (void *)0; // single-thread: nothing runnable
    continue;                              // multi-thread: go back and wait
}

With both fixes in, High Profile now decodes correctly on ESP32-P4 and survives
lossy live input. Happy to PR both (pow2x4 + the worker_loop guard). I’ll
confirm on-device shortly.

[traffaillac]

Good point, I updated the fix, hopefully doing things "right" this time: the bit extensions in function refs_to_mask now preserve signedness.

For context, historically I've always favored unsigned extensions because they are generally slightly faster on Intel CPUs. Now with wider CPUs support, readability becomes more important, so I tend to reduce the use of these tricks :D

Cheers,
Thibault

[youkorr]

Hi, I'm getting back to you to let you know that the porting of Edge 264 to high profile on RISCV works. If you'd like, I could send you a video or an example.

[traffaillac]

Awesome, thanks!!

If you have a short clip to send me, that would definitely help avoid regressions.

Now considering the second fix, feel free to send a PR indeed, it seems like a sound fix. If you have a test file to prevent future regressions on this fix, that would be even better. I should mention that I plan the reimplementation of multithreading sometime around Q4-2026, so things may break if not backed by tests ^^

Cheers,
Thibault

[youkorr]

If you want, I can create a PR that you can use as an example. (RSTP high profile)?