ci: run the full smoke test once, and split the GNUmakefile - #5616
Conversation
The GNUmakefile had 1295 lines and mixed build configuration, LLVM bootstrapping, device generation, four test suites, the smoke tests, release packaging, and lint tools. The smoke tests alone were 500 lines. Move each part into its own file in make/ and include them from GNUmakefile. config.mk must be included first because the other files use its variables in immediate assignments and conditionals. There is no change in behavior. The parsed make database is identical for the default build and for ASSERT=1, STATIC=1, XTENSA=0, STM32=0, WASM=0, and CROSS=aarch64-linux-gnu, except for MAKEFILE_LIST and the .PHONY list, which now also includes targets that were phony but not declared. The Dockerfile copies GNUmakefile alone before it builds LLVM, to keep that layer independent of the source tree. It must copy make/ too.
The smoke test was one recipe of about 500 lines with 236 builds that always ran in sequence. Split it at the group boundaries that were already there, so that: - `make -j smoketest` builds the groups in parallel. A full run goes from 325 to 91 seconds on a 32 core machine. - CI can shard the groups across runners. Each group writes to its own name in build/smoke/, because all builds wrote to test.hex before and would overwrite each other in a parallel build. The output extension selects the format, so it stays per line. Add smoketest-quick, which builds one board for each processor architecture. The full smoke test answers "can TinyGo build for every board", which does not depend on the host OS, so it only needs to run on one OS. The other jobs use smoketest-quick. The comment above the esp32c3 group had 4 spaces of indentation instead of a tab. That was harmless in the middle of a recipe, but it is now the first line of a group, where it would stop the recipe from starting. The set of build commands is unchanged for the default flags and for XTENSA=0, STM32=0, and WASM=0. All 226 checksums are the same as before, for a sequential build and for `make -j16`.
The smoke test ran six times for each push: twice on Linux, twice on macOS, once on Windows, and once in the compatibility test. Together that was about 97 minutes of the CI time. The smoke test checks that TinyGo can build a binary for each board. That does not depend on the host OS. The only part that does is one build behind a Windows check. Add a smoketest-linux job that runs the full set, split across four runners that use the tarball from the build-linux job. The groups are balanced with the measured build time of each group. Remove the full smoke test from test-linux-build, which the new job replaces, and use smoketest-quick for the other four jobs. Expected result: about 55 to 39 minutes for the slowest workflow, and about 97 to 35 minutes of total smoke test time.
It is the only target that uses the aarch64 LLVM triple. With it, smoketest-quick covers all 13 triples that the full smoke test uses.
|
Size difference with the dev branch: Binary size differenceflash ram before after diff before after diff 19684 19684 0 0.00% 7052 7052 0 0.00% tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/adafruit4650 63228 63228 0 0.00% 6788 6788 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/adt7410/main.go 10024 10024 0 0.00% 5348 5348 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/adxl345/main.go 14628 14628 0 0.00% 7396 7396 0 0.00% tinygo build -size short -o ./build/test.hex -target=pybadge ./examples/amg88xx 10184 10184 0 0.00% 5348 5348 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/apa102/main.go 12592 12592 0 0.00% 7172 7172 0 0.00% tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/apds9960/proximity/main.go 11136 11136 0 0.00% 5360 5360 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/apa102/itsybitsy-m0/main.go 7588 7588 0 0.00% 2312 2312 0 0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/at24cx/main.go 9272 9272 0 0.00% 5340 5340 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bh1750/main.go 8660 8660 0 0.00% 5340 5340 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/blinkm/main.go 71408 71408 0 0.00% 3656 3656 0 0.00% tinygo build -size short -o ./build/test.hex -target=pinetime ./examples/bma42x/main.go 67120 67120 0 0.00% 6804 6804 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmi160/main.go 28952 28952 0 0.00% 5380 5380 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmp180/main.go 65720 65720 0 0.00% 6828 6828 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmp280/main.go 13080 13080 0 0.00% 5412 5412 0 0.00% tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/bmp388/main.go 24024 24024 0 0.00% 6220 6220 0 0.00% tinygo build -size short -o ./build/test.hex -target=metro-rp2350 ./examples/bno08x/i2c/main.go 8008 8008 0 0.00% 3344 3344 0 0.00% tinygo build -size short -o ./build/test.hex -target=bluepill ./examples/ds1307/sram/main.go 22116 22116 0 0.00% 3548 3548 0 0.00% tinygo build -size short -o ./build/test.hex -target=bluepill ./examples/ds1307/time/main.go 30556 30556 0 0.00% 5576 5576 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/ds3231/alarms/main.go 44492 44492 0 0.00% 5576 5576 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/ds3231/basic/main.go 4592 4592 0 0.00% 2272 2272 0 0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/easystepper/main.go 71884 71884 0 0.00% 7588 7588 0 0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/flash/console/spi 67996 67996 0 0.00% 9628 9628 0 0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/flash/console/qspi 7128 7128 0 0.00% 2276 2276 0 0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/gc9a01/main.go 694072 694072 0 0.00% 131564 131564 0 0.00% |
CI resultsAll 20 checks pass. These are the times from this branch against the baseline run of Smoke test jobs
The four shards took 7:00, 7:14, 7:33, and 7:53. They agree to less than one minute, so the groups do not need a new balance. Workflow times
Two results that need an explanationThe macOS and Windows workflow times are noisy for this run. The job times above are correct, but these two totals hide the change. macos-15-intel gives 49 min, not the 36 min that the smoke test change predicts. The smoke step did drop by 18.6 min. However, each other step on that runner became slower by about 12.6 min in total. This PR does not change any of these steps.
macos-14 is the control. It behaved as expected and went from 31 min to 22 min, with all steps other than the smoke test unchanged. The Windows workflow gives 33 min because A second run on this branch will show if these two numbers are runner variance. Next stepThe slowest job is now |
jakebailey
left a comment
jakebailey
left a comment
There was a problem hiding this comment.
This seems fine, though perhaps one more step when trying to find something and I don't remember where it is
soypat
left a comment
soypat
left a comment
There was a problem hiding this comment.
I'm not a fan of leaning more into make. Since this seems like a net addition and not a replacement/modification of existing tests I urge us to reconsider adding more make
| @@ -0,0 +1,41 @@ | |||
| # Build the TinyGo compiler, plus housekeeping and code generation helpers. | |||
There was a problem hiding this comment.
Just throwing it out there: are we really happy with Makefiles? Just a reminder there's precedent in TinyGo for moving off of the makefile build system towards Go based build systems
There was a problem hiding this comment.
are we really happy with Makefiles
I have no problem with Makefiles.
|
Thanks for reviews now squash/merging. |
3 participants
Why
The smoke test ran six times for each push: twice on Linux, twice on macOS, once on Windows, and once in the compatibility test. Measured on a recent run, that is about 97 minutes of the CI time.
test-linux-buildassert-test-linuxbuild-macos(macos-15-intel)build-macos(macos-14)smoke-test-windowstest-compatThe smoke test checks that TinyGo can build a binary for each of 144 boards. That does not depend on the host OS. The only part that does is one build behind a Windows check. So five of the six runs add very little.
The GNUmakefile also had 1295 lines and mixed build configuration, LLVM bootstrapping, device generation, four test suites, the smoke test, release packaging, and lint tools. The smoke test alone was 500 lines.
What changed
Split the GNUmakefile into topic files in
make/.GNUmakefilenow holds the default goal and the include lines.config.mkmust come first, because the other files use its variables in immediate assignments and conditionals.Split the smoke test into 14 groups. The split follows the group boundaries that were already in the file, with no lines moved. Each group writes to its own name in
build/smoke/, because all builds wrote totest.hexbefore and would overwrite each other in a parallel build.make -j smoketestnow works.Added
smoketest-quick, which builds one board for each processor architecture. It covers all 13 LLVM triples that the full smoke test uses.Changed the workflows. A new
smoketest-linuxjob runs the full set, split across four runners, using the tarball frombuild-linux. The groups are balanced with the measured build time of each group. The other four jobs usesmoketest-quick.Effect
Expected: about 55 to 39 minutes for the slowest workflow, and about 97 to 35 minutes of total smoke test time.
The slowest job then becomes
assert-test-linuxat about 39 minutes. Splitting it into a build step and parallel test jobs, aswindows.ymldoes, is the next step and is not in this PR.Tests
make -j16.XTENSA=0,STM32=0,WASM=0, and all three together.ASSERT=1,STATIC=1,XTENSA=0,STM32=0,WASM=0, andCROSS=aarch64-linux-gnu. OnlyMAKEFILE_LISTand the.PHONYlist differ. The.PHONYlist now also declares targets that were phony but not declared.make helpprints the same output.smoketest-quickwere built from a cold cache.make lint,make spell, andmake fmt-checkpass.-j16on a 32 core machine.smoketest-quicktakes 85 seconds from a cold cache.Notes
The Dockerfile copies
GNUmakefileon its own before it builds LLVM, to keep that layer independent of the source tree. It must copymake/too, or thetinygo-llvmstage fails.The comment above the esp32c3 group had 4 spaces of indentation instead of a tab. That was harmless in the middle of a recipe, but it is now the first line of a group, where it would stop the recipe from starting.
The ARM builds in
build-linux-crossare unchanged. They take 3 to 4 minutes, are not on the critical path, and are a pure cross compile.qemu-useris only used asLLVM_CONFIG_PREFIXwhile LLVM builds, which is cached and skipped on almost every run. There is also no 32 bit armhf runner.