Skip to content

feat: add musl libc runtime and native C/Python workflows#281

Merged
TRSWNCA merged 11 commits into
masterfrom
feat/musl-libc
Jul 6, 2026
Merged

feat: add musl libc runtime and native C/Python workflows#281
TRSWNCA merged 11 commits into
masterfrom
feat/musl-libc

Conversation

@TRSWNCA

@TRSWNCA TRSWNCA commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a patched musl libc runtime that forwards Linux syscalls into AlloyStack LibOS hostcalls.
  • Add a C AsBuffer API for reference-based workflow data exchange, including non-destructive buffer borrowing.
  • Port wordcount, parallel sort, and long-chain C workloads from Wasmtime to native musl DSOs.
  • Build CPython 3.11.9 as a static musl library from the pinned third_party/cpython submodule.
  • Add a native built-in pyas module and pass workflow arguments through Python sys.argv.
  • Add native musl CPython workflows and Wasmtime versus musl end-to-end comparison recipes.
  • Add directory handles, getdents64, path stat, clock, random, futex, mmap, and related libc compatibility support.
  • Serialize FATFS metadata access so concurrent native CPython instances do not violate fatfs internal borrowing rules.

Native Python workflows

The following native musl configurations were added:

  • wordcount C1 and C3
  • parallel sort C1 and C3
  • function chain N5

Concurrent CPython 3.11 instances currently use distinct DSO copies so each workload receives an independent CPython global runtime.

Validation

  • Built the musl runtime and CPython with just musl_cpython.
  • Ran native musl CPython wordcount C1 and compared reducer output with an independent recount.
  • Ran native musl CPython parallel sort C1 and its ordering checker.
  • Ran native musl CPython function chain N5.
  • Ran native musl CPython parallel sort C3 with three concurrent CPython DSOs.
  • Re-ran the existing Wasmtime CPython wordcount, parallel sort, and function-chain workflows after restoring the legacy readdir hostcall name.
  • Confirmed the native CPython DSO has no glibc DT_NEEDED dependency.

Experiment recipes

  • just musl_c_end_to_end_latency
  • just musl_py_end_to_end_latency
  • just py_end_to_end_latency_compare

The existing Python latency recipe now preserves and prints the complete asvisor error instead of hiding it behind grep.

Current limitations

  • Py_FinalizeEx and post-main allocator teardown are not yet safe. CPython is treated as an isolation-lifetime runtime and cleanup is deferred to isolation unloading.
  • The compatibility pyas.access_buffer API copies into an existing Python buffer. pyas.borrow_buffer provides a zero-copy read-only memoryview for workloads migrated to the new API.
  • CPython C3 concurrency currently uses separate DSO copies rather than subinterpreters.
  • wordcount C3 requires an input dataset compatible with the fixed 25,677-byte script buffer. The current local image contains approximately 35 MB fake_data_1 and fake_data_2 files, whose reduced output exceeds that capacity.
  • Full performance recipes were not run as part of this implementation.

Comment thread as_hostcall/src/lib.rs

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不需要borrow_buffer的语义

@TRSWNCA TRSWNCA left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不需要borrow_buffer的一套语义,同时musl python的接口还是在采用copy的方式,没有零拷贝

@TRSWNCA

TRSWNCA commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Benchmark

Tested on:

  • CPU: Intel(R) Core(TM) i7-10700 (16) @ 4.80 GHz
  • Disk: ST2000DM001-1ER164 (SSD) 1.8TB

Word Count

Unit: ms

runtime parallelism 10MB 100MB 300MB
rust C1 79.104 757.638 2268.314
rust C3 36.503 314.195 922.197
rust C5 31.699 227.090 663.979
wasm-c C1 299.279 2387.993 7095.795
wasm-c C3 200.901 890.402 2467.586
wasm-c C5 274.950 751.190 1632.348
wasm-py C1 1092.912 8955.991 26754.322
wasm-py C3 869.074 3572.548 9410.658
wasm-py C5 1132.036 3016.441 6964.269
musl-c C1 147.171 2248.459 13052.668
musl-c C3 75.075 781.759 3053.358
musl-c C5 51.720 523.537 1759.068
musl-py C1 493.236 4339.102 14037.764
musl-py C3 352.915 1583.907 4461.630
musl-py C5 419.338 1283.076 3081.894

Parallel Sort

Unit: ms

runtime parallelism 1MB 25MB 50MB
rust C1 13.350 328.367 700.638
rust C3 10.838 199.952 453.829
rust C5 10.402 188.218 364.902
wasm-c C1 591.099 16281.043 N/A
wasm-c C3 220.988 5980.738 N/A
wasm-c C5 151.141 4114.035 N/A
wasm-py C1 584.279 10230.233 20992.677
wasm-py C3 793.674 5649.050 10980.069
wasm-py C5 1149.108 5292.692 9571.251
musl-c C1 175.673 4945.973 10471.834
musl-c C3 62.404 1662.802 3473.104
musl-c C5 41.252 1072.429 2241.401
musl-py C1 312.996 6346.976 12864.340
musl-py C3 352.578 3702.550 7453.610
musl-py C5 458.735 3283.261 6202.339

Long Chain

Unit: ms

runtime chain_len 1MB 64MB 256MB
rust n5 1.360 26.047 114.898
rust n10 1.984 48.788 216.746
rust n15 2.797 70.335 324.416
wasm-c n5 5.202 34.414 116.353
wasm-c n10 8.408 76.533 449.604
wasm-c n15 12.116 117.584 823.776
wasm-py n5 205.243 288.193 533.810
wasm-py n10 207.786 391.917 978.220
wasm-py n15 207.361 496.095 1388.698
musl-c n5 1.537 1.326 1.328
musl-c n10 1.862 1.678 1.583
musl-c n15 2.306 1.852 2.368
musl-py n5 70.525 74.743 87.599
musl-py n10 70.747 74.415 88.712
musl-py n15 70.564 75.283 88.309

@TRSWNCA TRSWNCA merged commit 2f1eff2 into master Jul 6, 2026
1 check passed
@TRSWNCA TRSWNCA deleted the feat/musl-libc branch July 6, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant