Test: Run ABI checks as Zephyr applications - #1318
Conversation
Build the ABI checker and assembly sources directly with Zephyr's target toolchain. Select the Armv8.1-M checker for M55 and preserve OPT/AUTO through the run stage so the checker is actually executed. Signed-off-by: Brendan Moran <brendan.moran@arm.com>
CBMC Results (ML-DSA-87, REDUCE-RAM)
Full Results (210 proofs)
|
CBMC Results (ML-DSA-44)
Full Results (210 proofs)
|
CBMC Results (ML-DSA-44, REDUCE-RAM)
Full Results (210 proofs)
|
CBMC Results (ML-DSA-65)
Full Results (210 proofs)
|
CBMC Results (ML-DSA-65, REDUCE-RAM)
Full Results (210 proofs)
|
CBMC Results (ML-DSA-87)
Full Results (210 proofs)
|
Include OPT, the selected FIPS202 backend, and configurable test counts in the active build marker so changes to those inputs rebuild stale Zephyr binaries. Track the native assembly amalgamation and its direct development-source include as explicit dependencies, and allow QEMU execution timeouts to be overridden. Signed-off-by: Brendan Moran <brendan.moran@arm.com>
There was a problem hiding this comment.
Mac Mini (M1, 2020) benchmarks (opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
46508 cycles |
46513 cycles |
1.00 |
ML-DSA-44 sign |
131107 cycles |
131135 cycles |
1.00 |
ML-DSA-44 verify |
47332 cycles |
47337 cycles |
1.00 |
ML-DSA-65 keypair |
81725 cycles |
81706 cycles |
1.00 |
ML-DSA-65 sign |
215525 cycles |
215412 cycles |
1.00 |
ML-DSA-65 verify |
79336 cycles |
79328 cycles |
1.00 |
ML-DSA-87 keypair |
132437 cycles |
132458 cycles |
1.00 |
ML-DSA-87 sign |
277496 cycles |
277522 cycles |
1.00 |
ML-DSA-87 verify |
133590 cycles |
133603 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Mac Mini (M1, 2020) benchmarks (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
112854 cycles |
113111 cycles |
1.00 |
ML-DSA-44 sign |
401397 cycles |
402435 cycles |
1.00 |
ML-DSA-44 verify |
119457 cycles |
119734 cycles |
1.00 |
ML-DSA-65 keypair |
192977 cycles |
192965 cycles |
1.00 |
ML-DSA-65 sign |
650054 cycles |
650025 cycles |
1.00 |
ML-DSA-65 verify |
193034 cycles |
193034 cycles |
1 |
ML-DSA-87 keypair |
318991 cycles |
318942 cycles |
1.00 |
ML-DSA-87 sign |
828972 cycles |
828891 cycles |
1.00 |
ML-DSA-87 verify |
321846 cycles |
321789 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A76 (Raspberry Pi 5) benchmarks (opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
112171 cycles |
112330 cycles |
1.00 |
ML-DSA-44 sign |
353453 cycles |
353551 cycles |
1.00 |
ML-DSA-44 verify |
117256 cycles |
117062 cycles |
1.00 |
ML-DSA-65 keypair |
194268 cycles |
194722 cycles |
1.00 |
ML-DSA-65 sign |
583812 cycles |
583744 cycles |
1.00 |
ML-DSA-65 verify |
193154 cycles |
192815 cycles |
1.00 |
ML-DSA-87 keypair |
319972 cycles |
320714 cycles |
1.00 |
ML-DSA-87 sign |
747761 cycles |
747066 cycles |
1.00 |
ML-DSA-87 verify |
317817 cycles |
318491 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A76 (Raspberry Pi 5) benchmarks (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
211647 cycles |
211728 cycles |
1.00 |
ML-DSA-44 sign |
758952 cycles |
760109 cycles |
1.00 |
ML-DSA-44 verify |
228909 cycles |
229502 cycles |
1.00 |
ML-DSA-65 keypair |
375121 cycles |
376008 cycles |
1.00 |
ML-DSA-65 sign |
1247459 cycles |
1247901 cycles |
1.00 |
ML-DSA-65 verify |
371351 cycles |
372060 cycles |
1.00 |
ML-DSA-87 keypair |
600321 cycles |
601421 cycles |
1.00 |
ML-DSA-87 sign |
1584179 cycles |
1584160 cycles |
1.00 |
ML-DSA-87 verify |
616037 cycles |
617191 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Intel Xeon 4th gen (c7i)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
43392 cycles |
43263 cycles |
1.00 |
ML-DSA-44 sign |
130045 cycles |
130366 cycles |
1.00 |
ML-DSA-44 verify |
45214 cycles |
45182 cycles |
1.00 |
ML-DSA-65 keypair |
75858 cycles |
75598 cycles |
1.00 |
ML-DSA-65 sign |
213603 cycles |
213718 cycles |
1.00 |
ML-DSA-65 verify |
74435 cycles |
74455 cycles |
1.00 |
ML-DSA-87 keypair |
122992 cycles |
123023 cycles |
1.00 |
ML-DSA-87 sign |
271057 cycles |
271302 cycles |
1.00 |
ML-DSA-87 verify |
120694 cycles |
120667 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Intel Xeon 4th gen (c7i) (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
91638 cycles |
91783 cycles |
1.00 |
ML-DSA-44 sign |
350875 cycles |
351152 cycles |
1.00 |
ML-DSA-44 verify |
99310 cycles |
99373 cycles |
1.00 |
ML-DSA-65 keypair |
154153 cycles |
154245 cycles |
1.00 |
ML-DSA-65 sign |
570979 cycles |
571358 cycles |
1.00 |
ML-DSA-65 verify |
160106 cycles |
159962 cycles |
1.00 |
ML-DSA-87 keypair |
255595 cycles |
255684 cycles |
1.00 |
ML-DSA-87 sign |
721247 cycles |
721581 cycles |
1.00 |
ML-DSA-87 verify |
264639 cycles |
264500 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
AMD EPYC 4th gen (c7a)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
46912 cycles |
46815 cycles |
1.00 |
ML-DSA-44 sign |
140220 cycles |
139083 cycles |
1.01 |
ML-DSA-44 verify |
50177 cycles |
49716 cycles |
1.01 |
ML-DSA-65 keypair |
82954 cycles |
82146 cycles |
1.01 |
ML-DSA-65 sign |
229193 cycles |
227761 cycles |
1.01 |
ML-DSA-65 verify |
84942 cycles |
81848 cycles |
1.04 |
ML-DSA-87 keypair |
131317 cycles |
130894 cycles |
1.00 |
ML-DSA-87 sign |
285080 cycles |
280929 cycles |
1.01 |
ML-DSA-87 verify |
130131 cycles |
129910 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'AMD EPYC 4th gen (c7a)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-65 verify |
84942 cycles |
81848 cycles |
1.04 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton5
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
51561 cycles |
51554 cycles |
1.00 |
ML-DSA-44 sign |
162082 cycles |
162116 cycles |
1.00 |
ML-DSA-44 verify |
54689 cycles |
54683 cycles |
1.00 |
ML-DSA-65 keypair |
90123 cycles |
90128 cycles |
1.00 |
ML-DSA-65 sign |
267346 cycles |
267370 cycles |
1.00 |
ML-DSA-65 verify |
89804 cycles |
89787 cycles |
1.00 |
ML-DSA-87 keypair |
145833 cycles |
145945 cycles |
1.00 |
ML-DSA-87 sign |
336157 cycles |
336126 cycles |
1.00 |
ML-DSA-87 verify |
145434 cycles |
145433 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton3
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
71577 cycles |
71611 cycles |
1.00 |
ML-DSA-44 sign |
209180 cycles |
209213 cycles |
1.00 |
ML-DSA-44 verify |
74891 cycles |
74885 cycles |
1.00 |
ML-DSA-65 keypair |
125973 cycles |
125974 cycles |
1.00 |
ML-DSA-65 sign |
345386 cycles |
345443 cycles |
1.00 |
ML-DSA-65 verify |
124117 cycles |
124066 cycles |
1.00 |
ML-DSA-87 keypair |
206131 cycles |
206188 cycles |
1.00 |
ML-DSA-87 sign |
439573 cycles |
439586 cycles |
1.00 |
ML-DSA-87 verify |
204619 cycles |
204608 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton3 (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
138507 cycles |
138213 cycles |
1.00 |
ML-DSA-44 sign |
485764 cycles |
485757 cycles |
1.00 |
ML-DSA-44 verify |
149249 cycles |
149268 cycles |
1.00 |
ML-DSA-65 keypair |
241614 cycles |
241525 cycles |
1.00 |
ML-DSA-65 sign |
791757 cycles |
791668 cycles |
1.00 |
ML-DSA-65 verify |
241535 cycles |
241531 cycles |
1.00 |
ML-DSA-87 keypair |
395319 cycles |
395213 cycles |
1.00 |
ML-DSA-87 sign |
1014300 cycles |
1014034 cycles |
1.00 |
ML-DSA-87 verify |
404135 cycles |
404022 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton2
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
112425 cycles |
112420 cycles |
1.00 |
ML-DSA-44 sign |
353839 cycles |
353921 cycles |
1.00 |
ML-DSA-44 verify |
117190 cycles |
117209 cycles |
1.00 |
ML-DSA-65 keypair |
194756 cycles |
194771 cycles |
1.00 |
ML-DSA-65 sign |
584015 cycles |
584013 cycles |
1.00 |
ML-DSA-65 verify |
192993 cycles |
193002 cycles |
1.00 |
ML-DSA-87 keypair |
320858 cycles |
322044 cycles |
1.00 |
ML-DSA-87 sign |
747201 cycles |
750253 cycles |
1.00 |
ML-DSA-87 verify |
318607 cycles |
319948 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton2 (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
212324 cycles |
212379 cycles |
1.00 |
ML-DSA-44 sign |
760952 cycles |
761588 cycles |
1.00 |
ML-DSA-44 verify |
229894 cycles |
230058 cycles |
1.00 |
ML-DSA-65 keypair |
376865 cycles |
376620 cycles |
1.00 |
ML-DSA-65 sign |
1251424 cycles |
1248734 cycles |
1.00 |
ML-DSA-65 verify |
373154 cycles |
372591 cycles |
1.00 |
ML-DSA-87 keypair |
601328 cycles |
601584 cycles |
1.00 |
ML-DSA-87 sign |
1584529 cycles |
1608263 cycles |
0.99 |
ML-DSA-87 verify |
617458 cycles |
617795 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A72 (Raspberry Pi 4) benchmarks (opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: f4cf612 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
218141 cycles |
215701 cycles |
1.01 |
ML-DSA-44 sign |
599969 cycles |
594374 cycles |
1.01 |
ML-DSA-44 verify |
216942 cycles |
214981 cycles |
1.01 |
ML-DSA-65 keypair |
380296 cycles |
378444 cycles |
1.00 |
ML-DSA-65 sign |
991970 cycles |
978381 cycles |
1.01 |
ML-DSA-65 verify |
364915 cycles |
360940 cycles |
1.01 |
ML-DSA-87 keypair |
640436 cycles |
636245 cycles |
1.01 |
ML-DSA-87 sign |
1326613 cycles |
1316684 cycles |
1.01 |
ML-DSA-87 verify |
620612 cycles |
618556 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Arm Cortex-A72 (Raspberry Pi 4) benchmarks (opt)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.
| Benchmark suite | Current: e979f85 | Previous: f4cf612 | Ratio |
|---|---|---|---|
ML-DSA-65 verify |
373100 cycles |
360940 cycles |
1.03 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A72 (Raspberry Pi 4) benchmarks (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: f4cf612 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
296972 cycles |
301643 cycles |
0.98 |
ML-DSA-44 sign |
1123840 cycles |
1141890 cycles |
0.98 |
ML-DSA-44 verify |
326177 cycles |
331704 cycles |
0.98 |
ML-DSA-65 keypair |
547501 cycles |
547829 cycles |
1.00 |
ML-DSA-65 sign |
1860180 cycles |
1866763 cycles |
1.00 |
ML-DSA-65 verify |
526610 cycles |
529253 cycles |
1.00 |
ML-DSA-87 keypair |
842689 cycles |
850050 cycles |
0.99 |
ML-DSA-87 sign |
2343362 cycles |
2368120 cycles |
0.99 |
ML-DSA-87 verify |
871540 cycles |
877810 cycles |
0.99 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
SpacemiT K1 8 (Banana Pi F3) benchmarks (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
760363 cycles |
760547 cycles |
1.00 |
ML-DSA-44 sign |
3140312 cycles |
3141975 cycles |
1.00 |
ML-DSA-44 verify |
859475 cycles |
859659 cycles |
1.00 |
ML-DSA-65 keypair |
1288909 cycles |
1289672 cycles |
1.00 |
ML-DSA-65 sign |
5082499 cycles |
5094204 cycles |
1.00 |
ML-DSA-65 verify |
1367852 cycles |
1368660 cycles |
1.00 |
ML-DSA-87 keypair |
2110619 cycles |
2109927 cycles |
1.00 |
ML-DSA-87 sign |
6371152 cycles |
6359962 cycles |
1.00 |
ML-DSA-87 verify |
2226580 cycles |
2226328 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-M55 (NUCLEO-N657X0-Q) benchmarks (opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
2129870 cycles |
2328440 cycles |
0.91 |
ML-DSA-44 sign |
17851737 cycles |
17851737 cycles |
1 |
ML-DSA-44 verify |
2398861 cycles |
2398876 cycles |
1.00 |
ML-DSA-65 keypair |
3731128 cycles |
4025122 cycles |
0.93 |
ML-DSA-65 sign |
14793522 cycles |
14793522 cycles |
1 |
ML-DSA-65 verify |
4012814 cycles |
4012792 cycles |
1.00 |
ML-DSA-87 keypair |
6482838 cycles |
6864887 cycles |
0.94 |
ML-DSA-87 sign |
24785143 cycles |
24785157 cycles |
1.00 |
ML-DSA-87 verify |
6864412 cycles |
6864412 cycles |
1 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-M55 (NUCLEO-N657X0-Q) benchmarks (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: 054cbd9 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
2328440 cycles |
2328440 cycles |
1 |
ML-DSA-44 sign |
17851737 cycles |
17851737 cycles |
1 |
ML-DSA-44 verify |
2398876 cycles |
2398876 cycles |
1 |
ML-DSA-65 keypair |
4025122 cycles |
4025122 cycles |
1 |
ML-DSA-65 sign |
14793522 cycles |
14793522 cycles |
1 |
ML-DSA-65 verify |
4012792 cycles |
4012792 cycles |
1 |
ML-DSA-87 keypair |
6864887 cycles |
6864887 cycles |
1 |
ML-DSA-87 sign |
24785157 cycles |
24785157 cycles |
1 |
ML-DSA-87 verify |
6864412 cycles |
6864412 cycles |
1 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A55 (Snapdragon 888) benchmarks (opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: f4cf612 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
271840 cycles |
271896 cycles |
1.00 |
ML-DSA-44 sign |
812569 cycles |
812577 cycles |
1.00 |
ML-DSA-44 verify |
272459 cycles |
273810 cycles |
1.00 |
ML-DSA-65 keypair |
473077 cycles |
465436 cycles |
1.02 |
ML-DSA-65 sign |
1377280 cycles |
1340764 cycles |
1.03 |
ML-DSA-65 verify |
456763 cycles |
453631 cycles |
1.01 |
ML-DSA-87 keypair |
796635 cycles |
798294 cycles |
1.00 |
ML-DSA-87 sign |
1834256 cycles |
1827503 cycles |
1.00 |
ML-DSA-87 verify |
776010 cycles |
773667 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
mkannwischer
left a comment
There was a problem hiding this comment.
The benchmarks in CI are the same for OPT=1 and OPT=0 so something does not seem to be working correctly.
Could you look into that @bremoran as you are already touching the OPT=0/1 parts?
There was a problem hiding this comment.
Arm Cortex-A55 (Snapdragon 888) benchmarks (no-opt)
Details
| Benchmark suite | Current: 5ee4dbb | Previous: f4cf612 | Ratio |
|---|---|---|---|
ML-DSA-44 keypair |
465352 cycles |
466025 cycles |
1.00 |
ML-DSA-44 sign |
2141420 cycles |
2142080 cycles |
1.00 |
ML-DSA-44 verify |
558987 cycles |
558559 cycles |
1.00 |
ML-DSA-65 keypair |
784185 cycles |
784222 cycles |
1.00 |
ML-DSA-65 sign |
3486212 cycles |
3496946 cycles |
1.00 |
ML-DSA-65 verify |
867563 cycles |
867603 cycles |
1.00 |
ML-DSA-87 keypair |
1274348 cycles |
1269447 cycles |
1.00 |
ML-DSA-87 sign |
4360898 cycles |
4335969 cycles |
1.01 |
ML-DSA-87 verify |
1398209 cycles |
1393383 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
Signed-off-by: Brendan Moran <brendan.moran@arm.com>
d730a9b to
5ee4dbb
Compare
This is demonstrated resolved in PR #1277, which is rebased on top of #1318:
Here is the benchmark with OPT=0: #1277 (review)
|
mkannwischer
left a comment
There was a problem hiding this comment.
Thanks @bremoran. The core of the PR looks good.
Could you slightly clean up the abicheck.c for consistency with the other tests?
Signed-off-by: Brendan Moran <brendan.moran@arm.com>
Hi @mkannwischer thank you for your review. I have pushed an update to address your comments. |
mkannwischer
left a comment
There was a problem hiding this comment.
Thanks. LGTM.
I prefer to wait with the merge until after v2 has been released (which should be very shortly).
Summary
Enable ABI checking for the
mps3-an547Cortex-M55 Zephyr target.Previously, the ABI-check target bypassed the custom platform build path and linked an executable directly with
$(LD). Under Zephyr CI this produced a host executable, which was then passed to Arm QEMU and faulted at startup.This change:
$(LINK)/CUSTOM_BUILDmechanism;armv81mexplicitly formps3-an547, rather than inferring it from the genericarm-none-eabicompiler prefix;OPTandAUTOwhenscripts/testsinvokesrun_abicheck, ensuring the optimized ABI check is actually executed; andNormal non-Zephyr ABI-check builds continue to use the existing linker path.
This PR is an alternative to #1312.
Scope
This initially enables Zephyr ABI checking only for
mps3-an547. Other Zephyr targets are unchanged unless ABI checking is explicitly requested, in which case they report that the target is unsupported.Validation
keccak_f1600_x4_mve_asmABI check under M55 QEMU: passedgit diff --check: passedCloses #1312