diff --git a/.gitignore b/.gitignore index 5c48ec53..1cd19a2a 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,10 @@ tinywasm venv wasmtime-as-lib image_content/* +**/__pycache__/ +*.py[cod] +reports/latency_report.md +reports/latency_report_results.json .VSCodeCounter .gdbinit @@ -13,4 +17,4 @@ _members.txt **/*.img **/fake*txt -**/*.bin \ No newline at end of file +**/*.bin diff --git a/.gitmodules b/.gitmodules index 4a039db0..a15df7e5 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,9 @@ [submodule "nom-uri"] path = nom-uri url = https://github.com/Skasselbard/nom-uri.git +[submodule "third_party/musl"] + path = third_party/musl + url = https://git.musl-libc.org/git/musl +[submodule "third_party/cpython"] + path = third_party/cpython + url = https://github.com/python/cpython.git diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..fdf17bf8 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,116 @@ +# AlloyStack 协作指南 + +## 项目定位 + +AlloyStack 是面向 Serverless workflow 的 Rust LibOS 运行时。它在单一地址空间中： + +- 按需加载用户函数及系统服务，以降低冷启动开销。 +- 通过引用传递 workflow 的中间数据，减少序列化与复制。 +- 让 Rust 用户函数以 `#![no_std]` 方式依赖 `as_std`，并直接链接到 LibOS，而不是 libc。 +- 通过定制的 `wasmtime-as-lib` 执行 WebAssembly，并由 `wasmtime_wasi_api` 将 WASI 调用接入 LibOS，从而承载 C/CPython workload。 + +注意：仓库实现使用 **Wasmtime**，不是 Wasmer。讨论或修改 WebAssembly 路径时，以代码中的 `wasmtime_wasi_api` 和 `wasmtime-as-lib` 为准。 + +## 目录职责 + +- `bins/asvisor/`：主要运行时命令行入口。 +- `libasvisor/`：隔离实例、模块加载、workflow 调度、指标和 hostcall 接入。 +- `as_hostcall/`：LibOS hostcall 及文件、内存、网络等后端。 +- `as_std/`：Rust 用户函数使用的 `no_std` 接口、分配器、panic 和数据传递支持。 +- `as_std_proc_macro/`：用户函数相关过程宏。 +- `common_service/`：可独立构建并按需加载的 LibOS 服务。 +- `wasmtime_wasi_api/`：Wasmtime/WASI 到 LibOS 的适配层。 +- `user/`：用户函数及 Wasmtime/CPython workload；不属于根 Cargo workspace。 +- `isol_config/`：workflow JSON 定义，是运行时装配 workflow 的事实来源。 +- `libasvisor/tests/`：运行时集成测试。 +- `baseline/`：对比实现和实验脚本，不属于根 Cargo workspace。 +- `fs_images/`、`image_content/`：文件系统镜像及挂载内容。 + +根 workspace 刻意排除了 `user/`、`common_service/` 和 `baseline/`。不要仅凭根目录 `cargo test` 判断这些组件已经构建或验证。 + +## 工具链与构建 + +项目基准环境是 Ubuntu 22.04、GCC 11.4 和 `nightly-2023-12-01`。部分实验依赖 Intel MPK、sudo、TAP 设备或已挂载的 FAT 镜像。 + +常用命令： + +```bash +just init +just asvisor +just libos +just rust_func +just all_libos +target/release/asvisor --files isol_config/.json +``` + +`justfile` 默认以 release 模式构建。可通过变量覆盖功能开关，例如： + +```bash +just enable_mpk=1 rust_func +just enable_file_buffer=1 rust_func +just enable_release=0 asvisor +``` + +相关 feature 约束： + +- `mpk`/`enable_mpk` 启用内存保护键；需要兼容硬件和运行环境。 +- `pkey_per_func` 建立在 MPK 之上。 +- `file-based` 用文件中转代替引用传递，主要用于消融实验。 +- `as_std` 的 `unwinding` 与 `panic_def` 互斥，不得同时启用。 + +不要擅自运行需要 sudo、修改网络设备、挂载镜像或执行完整性能评测的 recipe。此类命令包括但不限于 `cold_start_latency`、`end_to_end_latency`、`breakdown`、`p99` 和 `resource_consume`。 + +## Workflow 配置 + +workflow 在 `isol_config/*.json` 中定义： + +- `services`：需加载的 LibOS 服务及其 `.so` 文件。 +- `apps`：用户函数及其 `.so` 文件。 +- `groups`：按顺序执行的阶段；同一 group 中的函数可并行执行。 +- group 级 `args` 会作为默认参数，单个 app 的 `args` 会覆盖同名值。 +- 运行时会为 group 中每个 app 注入字符串形式的 `id`。 +- 可选字段包括 `fs_image` 和 `with_libos`。 + +新增或修改 workflow 时，应同步确认： + +1. 所有 service 和 app 名称与构建产物名称一致。 +2. 所需 service 已列入 `services`，并可通过 `just libos ` 构建。 +3. app 可通过 `just rust_func ` 或对应的 `wasm_func` recipe 构建。 +4. group 参数均为字符串，并与函数实际读取的键一致。 +5. workflow 所需数据确实存在于对应文件系统镜像中。 + +## 修改原则 + +- 保持单地址空间、按需加载和引用传递这三条核心路径的语义；不要无意中引入 libc 依赖或跨边界复制。 +- 修改用户函数接口时，同时检查 `as_std`、`as_hostcall`、运行时装载逻辑和相关 workflow 配置。 +- 修改 WASI 实现时，检查 `wasmtime_wasi_api/src/lib.rs` 的 linker 注册及 `wasi.rs` 中对应实现。 +- 新增用户函数时遵循相邻 `user/*` crate 的布局、crate type、build script 和 feature 传递方式。 +- 不要把 `user/` 或 `common_service/` 草率加入根 workspace；现有排除与 `as_std` 的冲突 feature 组合有关。 +- 不要提交生成物、挂载内容、实验日志或大型数据文件，除非任务明确要求。 +- 仓库含来源于 RuxOS 的代码；修改相关实现时保留既有来源和许可证信息。 + +## 验证要求 + +验证范围应与修改范围匹配，优先使用最小且有针对性的命令： + +```bash +cargo fmt --all -- --check +cargo test -p libasvisor +cargo build -p asvisor +just libos +just rust_func +target/release/asvisor --files isol_config/.json +``` + +注意： + +- 根 workspace 构建可能需要获取定制 git 依赖；网络不可用时应明确报告，不能把依赖获取失败描述成代码失败。 +- 集成测试和 workflow 执行可能依赖 MPK、镜像、网络权限或已构建的动态库。报告验证结果时区分代码错误与环境前置条件不足。 +- 性能相关修改不能只证明“能运行”；应使用相同构建模式、feature、数据集和 workflow 配置进行前后对比。 +- 未实际执行的测试不得声称通过。 + +## 沟通约定 + +- 回答使用中文，保持结构化、直接和可核验。 +- 对项目描述、代码行为和实验结论都保持审慎；发现文档与实现冲突时，以代码和可复现实验为依据，并明确指出冲突。 +- 不做过度夸赞。信息不足且会影响技术结论时，主动索要配置、日志、复现步骤或实验数据。 diff --git a/Cargo.lock b/Cargo.lock index 1d60b4c4..76516da7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -141,6 +141,7 @@ dependencies = [ "derive_more", "env_logger", "libasvisor", + "libc", "log", "thiserror-no-std", "tokio", diff --git a/Cargo.toml b/Cargo.toml index 667083fb..3f93560c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -17,7 +17,10 @@ members = [ # In current implementation, as_std has conflict features (e.g. `panic_def` and # `unwinding`). Of course, if add "user/" to members, will also have problem of # feature gate. -exclude = ["user/", "common_service/", "baseline/"] +# +# as_musl enables as_std's application-side allocator and panic definitions, so +# it must also be built separately through the musl user crates. +exclude = ["user/", "common_service/", "baseline/", "as_musl/"] default-members = ["bins/asvisor"] resolver = "2" diff --git a/README.md b/README.md index 43c72bb0..4b858e4c 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,8 @@ AlloyStack/ To run a new test application on AlloyStack, user need to develop functions in the `user/` directory. Then, edit the workflow specification files in the `isol_config/` directory to declare how functions compose the workflow, specify dependencies on LibOS modules, and define input parameters for functions. If the workflow involves reading datasets from files, the datasets must also be added to the file system image. Please use the following command to extract the provided image archive, which contains the source code for the Python benchmarks. For detailed documentation, please refer to [AlloyStack User Guide](./doc/). +For source-compatible C functions backed by musl, see +[Running C functions with musl](./doc/musl_functions.md). ## Evaluation ### Cold start latency diff --git a/as_hostcall/Cargo.toml b/as_hostcall/Cargo.toml index d6671d5e..ef291391 100644 --- a/as_hostcall/Cargo.toml +++ b/as_hostcall/Cargo.toml @@ -20,6 +20,7 @@ thiserror-no-std = "2.0.2" fatfs = [] socket = [] signal = [] +sys = [] fdtab = ["fatfs", "socket"] mmap_file_backend = ["fdtab"] diff --git a/as_hostcall/src/fatfs.rs b/as_hostcall/src/fatfs.rs index 7c588054..836de414 100644 --- a/as_hostcall/src/fatfs.rs +++ b/as_hostcall/src/fatfs.rs @@ -8,7 +8,10 @@ pub type FatfsWriteFunc = fn(Fd, &[u8]) -> FatfsResult; pub type FatfsReadFunc = fn(Fd, &mut [u8]) -> FatfsResult; pub type FatfsCloseFunc = fn(Fd) -> FatfsResult<()>; pub type FatfsSeekFunc = fn(Fd, u32) -> FatfsResult<()>; +pub type FatfsSeek64Func = fn(Fd, i64, i32) -> FatfsResult; pub type FatfsStatFunc = fn(Fd) -> FatfsResult; +pub type FatfsPathStatFunc = fn(&str) -> FatfsResult; +pub type FatfsReadDirFunc = fn(&str, &mut [u8]) -> FatfsResult; pub type FatfsResult = Result; diff --git a/as_hostcall/src/fdtab.rs b/as_hostcall/src/fdtab.rs index a3e72d3b..b98f30ac 100644 --- a/as_hostcall/src/fdtab.rs +++ b/as_hostcall/src/fdtab.rs @@ -6,15 +6,19 @@ use thiserror_no_std::Error; use crate::{ fatfs::FatfsError, socket::SmoltcpError, - types::{Fd, OpenFlags, OpenMode, Size, SockFd, Stat, DirEntry}, + types::{DirEntry, Fd, OpenFlags, OpenMode, Size, SockFd, Stat}, }; pub type OpenFunc = fn(&str, OpenFlags, OpenMode) -> FdtabResult; +pub type OpenDirFunc = fn(&str) -> FdtabResult; +pub type GetDentsFunc = fn(Fd, &mut [u8]) -> FdtabResult; pub type WriteFunc = fn(Fd, &[u8]) -> FdtabResult; pub type ReadFunc = fn(Fd, &mut [u8]) -> FdtabResult; pub type CloseFunc = fn(Fd) -> FdtabResult<()>; pub type LseekFunc = fn(Fd, u32) -> FdtabResult<()>; +pub type Lseek64Func = fn(Fd, i64, i32) -> FdtabResult; pub type StatFunc = fn(Fd) -> FdtabResult; +pub type PathStatFunc = fn(&str) -> FdtabResult; pub type ReadDirFunc = fn(&str) -> FdtabResult>; pub type ConnectFunc = fn(SocketAddrV4) -> FdtabResult; pub type BindFunc = fn(SocketAddrV4) -> FdtabResult; diff --git a/as_hostcall/src/lib.rs b/as_hostcall/src/lib.rs index 6a4c624e..3fdc3552 100644 --- a/as_hostcall/src/lib.rs +++ b/as_hostcall/src/lib.rs @@ -20,6 +20,8 @@ pub mod mpk; pub mod signal; #[cfg(feature = "socket")] pub mod socket; +#[cfg(feature = "sys")] +pub mod sys; pub mod types; use alloc::{borrow::ToOwned, string::String}; @@ -43,12 +45,20 @@ pub enum CommonHostCall { Read, #[display(fmt = "open")] Open, + #[display(fmt = "open_dir")] + OpenDir, + #[display(fmt = "getdents")] + GetDents, #[display(fmt = "close")] Close, #[display(fmt = "lseek")] Lseek, + #[display(fmt = "lseek64")] + Lseek64, #[display(fmt = "stat")] Stat, + #[display(fmt = "path_stat")] + PathStat, #[display(fmt = "readdir")] ReadDir, #[display(fmt = "connect")] @@ -73,8 +83,14 @@ pub enum CommonHostCall { FatfsClose, #[display(fmt = "fatfs_seek")] FatfsSeek, + #[display(fmt = "fatfs_seek64")] + FatfsSeek64, #[display(fmt = "fatfs_stat")] FatfsStat, + #[display(fmt = "fatfs_path_stat")] + FatfsPathStat, + #[display(fmt = "fatfs_readdir")] + FatfsReadDir, #[display(fmt = "addrinfo")] SmoltcpAddrInfo, @@ -93,6 +109,10 @@ pub enum CommonHostCall { #[display(fmt = "buffer_alloc")] BufferAlloc, + #[display(fmt = "buffer_alloc_raw")] + BufferAllocRaw, + #[display(fmt = "buffer_register")] + BufferRegister, #[display(fmt = "access_buffer")] AccessBuffer, #[display(fmt = "buffer_dealloc")] @@ -118,6 +138,20 @@ pub enum CommonHostCall { #[display(fmt = "libos_sigaction")] SigAction, + + #[display(fmt = "host_futex")] + Futex, + #[display(fmt = "host_gettid")] + GetTid, + #[display(fmt = "host_getrandom")] + GetRandom, + + // Keep new hostcalls at the end to preserve existing discriminants used by + // prebuilt user modules. + #[display(fmt = "buffer_set_len")] + BufferSetLen, + #[display(fmt = "buffer_len")] + BufferLen, } #[derive(Debug, Display)] @@ -136,10 +170,14 @@ impl HostCallID { CommonHostCall::Write | CommonHostCall::Open + | CommonHostCall::OpenDir + | CommonHostCall::GetDents | CommonHostCall::Read | CommonHostCall::Close | CommonHostCall::Lseek + | CommonHostCall::Lseek64 | CommonHostCall::Stat + | CommonHostCall::PathStat | CommonHostCall::ReadDir | CommonHostCall::Connect | CommonHostCall::Socket @@ -153,7 +191,10 @@ impl HostCallID { | CommonHostCall::FatfsRead | CommonHostCall::FatfsClose | CommonHostCall::FatfsSeek - | CommonHostCall::FatfsStat => "fatfs".to_owned(), + | CommonHostCall::FatfsSeek64 + | CommonHostCall::FatfsStat + | CommonHostCall::FatfsPathStat + | CommonHostCall::FatfsReadDir => "fatfs".to_owned(), CommonHostCall::SmoltcpAddrInfo | CommonHostCall::SmoltcpConnect @@ -164,6 +205,10 @@ impl HostCallID { | CommonHostCall::SmoltcpClose => "socket".to_owned(), CommonHostCall::BufferAlloc + | CommonHostCall::BufferAllocRaw + | CommonHostCall::BufferRegister + | CommonHostCall::BufferSetLen + | CommonHostCall::BufferLen | CommonHostCall::AccessBuffer | CommonHostCall::BufferDealloc | CommonHostCall::Mmap @@ -177,6 +222,10 @@ impl HostCallID { CommonHostCall::SigAction => "signal".to_owned(), CommonHostCall::GetTime | CommonHostCall::NanoSleep => "time".to_owned(), + + CommonHostCall::Futex | CommonHostCall::GetTid | CommonHostCall::GetRandom => { + "sys".to_owned() + } }, HostCallID::Custom(_) => todo!(), } diff --git a/as_hostcall/src/mm.rs b/as_hostcall/src/mm.rs index 8f8f1530..49f30870 100644 --- a/as_hostcall/src/mm.rs +++ b/as_hostcall/src/mm.rs @@ -16,7 +16,11 @@ bitflags! { } pub type BufferAllocFunc = fn(&str, Layout, u64) -> MMResult; -pub type AccessBufferFunc = fn(&str) -> Option<(usize, u64)>; +pub type BufferAllocRawFunc = fn(Layout) -> MMResult; +pub type BufferRegisterFunc = fn(&str, usize, u64, usize) -> MMResult<()>; +pub type BufferSetLenFunc = fn(&str, usize) -> MMResult<()>; +pub type BufferLenFunc = fn(&str) -> Option; +pub type AccessBufferFunc = fn(&str) -> Option<(usize, u64, usize)>; pub type BufferDeallocFunc = fn(usize, Layout); pub type MemmapFunc = fn(usize, usize, ProtFlags, Fd) -> MMResult; pub type MemunmapFunc = fn(&mut [u8], bool) -> MMResult<()>; @@ -34,4 +38,8 @@ pub enum MMError { FileBackendErr(#[from] MmapFileErr), #[error("libc api error: {0}")] LibcErr(String), + #[error("buffer allocation failed")] + NoMemory, + #[error("buffer slot already exists")] + BufferSlotExists, } diff --git a/as_hostcall/src/sys.rs b/as_hostcall/src/sys.rs new file mode 100644 index 00000000..0882e09a --- /dev/null +++ b/as_hostcall/src/sys.rs @@ -0,0 +1,3 @@ +pub type FutexFunc = unsafe extern "C" fn(*mut i32, i32, i32, *const u8, *mut i32, i32) -> isize; +pub type GetTidFunc = extern "C" fn() -> isize; +pub type GetRandomFunc = unsafe extern "C" fn(*mut u8, usize, u32) -> isize; diff --git a/as_hostcall/src/types.rs b/as_hostcall/src/types.rs index d676c295..ee461200 100644 --- a/as_hostcall/src/types.rs +++ b/as_hostcall/src/types.rs @@ -41,6 +41,7 @@ bitflags! { pub struct OpenFlags: u32 { const O_APPEND = 1; const O_CREAT = 2; + const O_TRUNC = 4; } #[derive(PartialEq, Clone)] @@ -55,6 +56,7 @@ pub type Size = usize; // time for stat #[derive(Debug)] +#[repr(C)] pub struct TimeSpec { /// Whole seconds part of the timespec. pub tv_sec: core::ffi::c_longlong, @@ -64,6 +66,7 @@ pub struct TimeSpec { // file stat #[derive(Debug)] +#[repr(C)] pub struct Stat { // pub st_size: Size, /// Device identifier. @@ -113,7 +116,11 @@ pub type HostStdioFunc = fn(&[u8]) -> Size; pub type SockFd = u32; // time -pub type GetTimeFunc = fn() -> Result; +/// Nanoseconds since the Unix epoch. `u64::MAX` reports a clock error. +/// +/// Keep this hostcall's return value scalar: complex Rust return values use an +/// indirect ABI that is unsafe across independently loaded LibOS modules. +pub type GetTimeFunc = extern "C" fn() -> u64; pub type NanoSleepFunc = fn(u64, u64); // isol_info diff --git a/as_musl/Cargo.toml b/as_musl/Cargo.toml new file mode 100644 index 00000000..81efb8a2 --- /dev/null +++ b/as_musl/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "as_musl" +version = "0.1.0" +edition = "2021" + +[lib] +test = false +doctest = false + +[dependencies] +as_std = { path = "../as_std", features = ["alloc_def", "panic_def"] } +as_hostcall = { path = "../as_hostcall", features = ["fdtab", "mm", "sys"] } + +[features] +default = [] +mpk = ["as_std/mpk", "as_hostcall/enable_mpk"] diff --git a/as_musl/include/alloy/args.h b/as_musl/include/alloy/args.h new file mode 100644 index 00000000..f983b2aa --- /dev/null +++ b/as_musl/include/alloy/args.h @@ -0,0 +1,35 @@ +#ifndef ALLOY_ARGS_H +#define ALLOY_ARGS_H + +#include +#include +#include + +static inline const char *as_arg_value(int argc, char **argv, const char *key) +{ + size_t key_len = strlen(key); + for (int i = 1; i < argc; ++i) { + if (argv[i][0] == '-' && argv[i][1] == '-' && + strncmp(argv[i] + 2, key, key_len) == 0 && + argv[i][key_len + 2] == '=') + return argv[i] + key_len + 3; + } + return NULL; +} + +static inline int as_arg_int(int argc, char **argv, const char *key, int *out) +{ + const char *value = as_arg_value(argc, argv, key); + char *end; + long parsed; + + if (!value || !out) + return -1; + parsed = strtol(value, &end, 10); + if (*value == '\0' || *end != '\0' || parsed < INT_MIN || parsed > INT_MAX) + return -1; + *out = (int)parsed; + return 0; +} + +#endif diff --git a/as_musl/include/alloy/asbuffer.h b/as_musl/include/alloy/asbuffer.h new file mode 100644 index 00000000..c85241e0 --- /dev/null +++ b/as_musl/include/alloy/asbuffer.h @@ -0,0 +1,36 @@ +#ifndef ALLOY_ASBUFFER_H +#define ALLOY_ASBUFFER_H + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +typedef struct as_buffer { + /* Public byte range. Reserved fields must not be modified by callers. */ + void *data; + size_t len; + size_t capacity; + uintptr_t _allocation; +} as_buffer_t; + +#define AS_BUFFER_INIT { NULL, 0, 0, 0 } + +/* + * All functions return zero on success and a negative errno value on failure. + * publish transfers ownership to the named slot and clears the producer handle. + * take transfers ownership to the consumer, which must call release exactly once. + */ +int as_buffer_alloc(size_t capacity, as_buffer_t *out); +int as_buffer_publish(const char *slot, as_buffer_t *buffer, size_t len); +int as_buffer_len(const char *slot, size_t *out); +int as_buffer_take(const char *slot, as_buffer_t *out); +int as_buffer_release(as_buffer_t *buffer); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/as_musl/src/lib.rs b/as_musl/src/lib.rs new file mode 100644 index 00000000..e840401b --- /dev/null +++ b/as_musl/src/lib.rs @@ -0,0 +1,752 @@ +#![no_std] +#![feature(decl_macro)] + +extern crate alloc; + +use alloc::{format, string::String, vec::Vec}; +use as_hostcall::{ + fdtab::FdtabError, + mm::{MMError, ProtFlags}, + types::{OpenFlags, OpenMode, Stat}, +}; +use as_std::{libos::libos, println}; +use core::{ + alloc::Layout, + ffi::{c_char, c_void, CStr}, + mem, ptr, + sync::atomic::{AtomicU64, Ordering}, +}; + +const ENOENT: isize = 2; +const EBADF: isize = 9; +const ENOMEM: isize = 12; +const EFAULT: isize = 14; +const EINVAL: isize = 22; +const ENOTTY: isize = 25; +const ENOSYS: isize = 38; +const EEXIST: i32 = 17; +const EOVERFLOW: i32 = 75; +const EPROTO: i32 = 71; +const EMSGSIZE: i32 = 90; +const AT_FDCWD: isize = -100; +const AT_EMPTY_PATH: isize = 0x1000; + +const SYS_READ: isize = 0; +const SYS_WRITE: isize = 1; +const SYS_OPEN: isize = 2; +const SYS_CLOSE: isize = 3; +const SYS_STAT: isize = 4; +const SYS_FSTAT: isize = 5; +const SYS_LSEEK: isize = 8; +const SYS_MMAP: isize = 9; +const SYS_MPROTECT: isize = 10; +const SYS_MUNMAP: isize = 11; +const SYS_BRK: isize = 12; +const SYS_RT_SIGPROCMASK: isize = 14; +const SYS_IOCTL: isize = 16; +const SYS_READV: isize = 19; +const SYS_WRITEV: isize = 20; +const SYS_MREMAP: isize = 25; +const SYS_MADVISE: isize = 28; +const SYS_GETCWD: isize = 79; +const SYS_READLINK: isize = 89; +const SYS_FCNTL: isize = 72; +const SYS_GETTID: isize = 186; +const SYS_FUTEX: isize = 202; +const SYS_GETDENTS64: isize = 217; +const SYS_CLOCK_GETTIME: isize = 228; +const SYS_OPENAT: isize = 257; +const SYS_NEWFSTATAT: isize = 262; +const SYS_GETRANDOM: isize = 318; + +const O_ACCMODE: isize = 0o3; +const O_WRONLY: isize = 0o1; +const O_RDWR: isize = 0o2; +const O_CREAT: isize = 0o100; +const O_TRUNC: isize = 0o1000; +const O_APPEND: isize = 0o2000; +const O_DIRECTORY: isize = 0o200000; + +const F_GETFD: isize = 1; +const F_SETFD: isize = 2; +const F_GETFL: isize = 3; +const F_SETFL: isize = 4; + +const PROT_READ: isize = 1; +const PROT_WRITE: isize = 2; +const PROT_EXEC: isize = 4; +const MREMAP_MAYMOVE: isize = 1; + +static REPORTED_SYSCALLS: [AtomicU64; 8] = [ + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), +]; + +const AS_BUFFER_MAGIC: u64 = 0x4153_4255_4646_4552; +const AS_BUFFER_FINGERPRINT: u64 = 0xd62e_ea91_45c8_8f31; + +#[repr(C)] +struct AsBufferHeader { + magic: u64, + len: usize, + capacity: usize, +} + +#[repr(C)] +pub struct AsBuffer { + pub data: *mut c_void, + pub len: usize, + pub capacity: usize, + pub allocation: usize, +} + +impl AsBuffer { + const fn empty() -> Self { + Self { + data: ptr::null_mut(), + len: 0, + capacity: 0, + allocation: 0, + } + } +} + +fn as_buffer_layout(capacity: usize) -> Result { + let size = mem::size_of::() + .checked_add(capacity) + .ok_or(-EOVERFLOW)?; + Layout::from_size_align(size, mem::align_of::()).map_err(|_| -EINVAL as i32) +} + +fn mm_error_code(error: &MMError) -> i32 { + match error { + MMError::NoMemory => -ENOMEM as i32, + MMError::BufferSlotExists => -EEXIST, + _ => -EINVAL as i32, + } +} + +unsafe fn buffer_slot<'a>(slot: *const c_char) -> Result<&'a str, i32> { + if slot.is_null() { + return Err(-EINVAL as i32); + } + let slot = CStr::from_ptr(slot).to_str().map_err(|_| -EINVAL as i32)?; + if slot.is_empty() { + return Err(-EINVAL as i32); + } + Ok(slot) +} + +unsafe fn checked_header(buffer: *mut AsBuffer) -> Result<*mut AsBufferHeader, i32> { + if buffer.is_null() { + return Err(-EINVAL as i32); + } + let buffer_ref = &mut *buffer; + if buffer_ref.allocation == 0 || buffer_ref.data.is_null() { + return Err(-EINVAL as i32); + } + let header = buffer_ref.allocation as *mut AsBufferHeader; + if (*header).magic != AS_BUFFER_MAGIC + || (*header).capacity != buffer_ref.capacity + || (header.add(1) as *mut c_void) != buffer_ref.data + { + return Err(-EINVAL as i32); + } + Ok(header) +} + +#[no_mangle] +pub unsafe extern "C" fn as_buffer_alloc(capacity: usize, out: *mut AsBuffer) -> i32 { + if out.is_null() { + return -EINVAL as i32; + } + ptr::write(out, AsBuffer::empty()); + let layout = match as_buffer_layout(capacity) { + Ok(layout) => layout, + Err(error) => return error, + }; + let allocation = match libos!(buffer_alloc_raw(layout)) { + Ok(allocation) => allocation, + Err(error) => return mm_error_code(&error), + }; + let header = allocation as *mut AsBufferHeader; + ptr::write( + header, + AsBufferHeader { + magic: AS_BUFFER_MAGIC, + len: 0, + capacity, + }, + ); + ptr::write( + out, + AsBuffer { + data: header.add(1) as *mut c_void, + len: 0, + capacity, + allocation, + }, + ); + 0 +} + +#[no_mangle] +pub unsafe extern "C" fn as_buffer_publish( + slot: *const c_char, + buffer: *mut AsBuffer, + len: usize, +) -> i32 { + let slot = match buffer_slot(slot) { + Ok(slot) => slot, + Err(error) => return error, + }; + let header = match checked_header(buffer) { + Ok(header) => header, + Err(error) => return error, + }; + if len > (*header).capacity { + return -EMSGSIZE; + } + (*header).len = len; + match libos!(buffer_register( + slot, + header as usize, + AS_BUFFER_FINGERPRINT, + len + )) { + Ok(()) => { + ptr::write(buffer, AsBuffer::empty()); + 0 + } + Err(error) => mm_error_code(&error), + } +} + +#[no_mangle] +pub unsafe extern "C" fn as_buffer_len(slot: *const c_char, out: *mut usize) -> i32 { + if out.is_null() { + return -EINVAL as i32; + } + let slot = match buffer_slot(slot) { + Ok(slot) => slot, + Err(error) => return error, + }; + match libos!(buffer_len(slot)) { + Some(len) => { + ptr::write(out, len); + 0 + } + None => -ENOENT as i32, + } +} + +#[no_mangle] +pub unsafe extern "C" fn as_buffer_take(slot: *const c_char, out: *mut AsBuffer) -> i32 { + if out.is_null() { + return -EINVAL as i32; + } + ptr::write(out, AsBuffer::empty()); + let slot = match buffer_slot(slot) { + Ok(slot) => slot, + Err(error) => return error, + }; + let (allocation, fingerprint, data_len) = match libos!(access_buffer(slot)) { + Some(metadata) => metadata, + None => return -ENOENT as i32, + }; + if fingerprint != AS_BUFFER_FINGERPRINT { + let _ = libos!(buffer_register(slot, allocation, fingerprint, data_len)); + return -EPROTO; + } + let header = allocation as *mut AsBufferHeader; + if (*header).magic != AS_BUFFER_MAGIC || data_len > (*header).capacity { + let _ = libos!(buffer_register(slot, allocation, fingerprint, data_len)); + return -EPROTO; + } + (*header).len = data_len; + ptr::write( + out, + AsBuffer { + data: header.add(1) as *mut c_void, + len: (*header).len, + capacity: (*header).capacity, + allocation, + }, + ); + 0 +} + +#[no_mangle] +pub unsafe extern "C" fn as_buffer_release(buffer: *mut AsBuffer) -> i32 { + let header = match checked_header(buffer) { + Ok(header) => header, + Err(error) => return error, + }; + let layout = match as_buffer_layout((*header).capacity) { + Ok(layout) => layout, + Err(error) => return error, + }; + (*header).magic = 0; + libos!(buffer_dealloc(header as usize, layout)); + ptr::write(buffer, AsBuffer::empty()); + 0 +} + +#[repr(C)] +struct IoVec { + base: *mut u8, + len: usize, +} + +#[repr(C)] +struct LinuxTimeSpec { + sec: i64, + nsec: i64, +} + +fn character_device_stat() -> Stat { + Stat { + st_dev: 0, + st_ino: 0, + st_nlink: 1, + st_mode: 0o020666, + st_uid: 0, + st_gid: 0, + __pad0: 0, + st_rdev: 0, + st_size: 0, + st_blksize: 0, + st_blocks: 0, + st_atime: as_hostcall::types::TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + st_mtime: as_hostcall::types::TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + st_ctime: as_hostcall::types::TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + __unused: [0, 0, 0], + } +} + +fn neg_errno(errno: isize) -> isize { + -errno +} + +fn fd_error(error: FdtabError, opening: bool) -> isize { + match error { + FdtabError::BadInputFd(_, _) | FdtabError::NoExistFd(_) => { + if opening { + neg_errno(ENOENT) + } else { + neg_errno(EBADF) + } + } + _ => neg_errno(EINVAL), + } +} + +fn open_file(path: *const c_char, flags: isize) -> isize { + if path.is_null() { + return neg_errno(EFAULT); + } + let path = match unsafe { CStr::from_ptr(path) }.to_str() { + Ok(path) => path, + Err(_) => return neg_errno(EINVAL), + }; + let path = path.trim_start_matches('/'); + + if flags & O_DIRECTORY != 0 { + return match libos!(open_dir(path)) { + Ok(fd) => fd as isize, + Err(error) => fd_error(error, true), + }; + } + + let mode = match flags & O_ACCMODE { + O_WRONLY => OpenMode::WR, + O_RDWR => OpenMode::RDWR, + _ => OpenMode::RD, + }; + let mut open_flags = OpenFlags::empty(); + if flags & O_CREAT != 0 { + open_flags |= OpenFlags::O_CREAT; + } + if flags & O_TRUNC != 0 { + open_flags |= OpenFlags::O_TRUNC; + } + if flags & O_APPEND != 0 { + open_flags |= OpenFlags::O_APPEND; + } + + match libos!(open(path, open_flags, mode)) { + Ok(fd) => fd as isize, + Err(error) => fd_error(error, true), + } +} + +unsafe fn stat_path(path: *const c_char, stat: *mut Stat) -> isize { + if path.is_null() || stat.is_null() { + return neg_errno(EFAULT); + } + let path = match CStr::from_ptr(path).to_str() { + Ok(path) => path, + Err(_) => return neg_errno(EINVAL), + }; + match libos!(path_stat(path)) { + Ok(metadata) => { + ptr::write(stat, metadata); + 0 + } + Err(_) => neg_errno(ENOENT), + } +} + +unsafe fn read_one(fd: u32, ptr: *mut u8, len: usize) -> isize { + if ptr.is_null() && len != 0 { + return neg_errno(EFAULT); + } + let buffer = core::slice::from_raw_parts_mut(ptr, len); + match libos!(read(fd, buffer)) { + Ok(size) => size as isize, + Err(error) => fd_error(error, false), + } +} + +unsafe fn write_one(fd: u32, ptr: *const u8, len: usize) -> isize { + if ptr.is_null() && len != 0 { + return neg_errno(EFAULT); + } + let buffer = core::slice::from_raw_parts(ptr, len); + match libos!(write(fd, buffer)) { + Ok(size) => size as isize, + Err(error) => fd_error(error, false), + } +} + +unsafe fn read_vectored(fd: u32, iov: *mut IoVec, count: usize) -> isize { + if iov.is_null() && count != 0 { + return neg_errno(EFAULT); + } + let mut total = 0isize; + for item in core::slice::from_raw_parts_mut(iov, count) { + let read = read_one(fd, item.base, item.len); + if read < 0 { + return if total == 0 { read } else { total }; + } + total += read; + if read as usize != item.len { + break; + } + } + total +} + +unsafe fn write_vectored(fd: u32, iov: *const IoVec, count: usize) -> isize { + if iov.is_null() && count != 0 { + return neg_errno(EFAULT); + } + let mut total = 0isize; + for item in core::slice::from_raw_parts(iov, count) { + let written = write_one(fd, item.base, item.len); + if written < 0 { + return if total == 0 { written } else { total }; + } + total += written; + if written as usize != item.len { + break; + } + } + total +} + +fn report_unsupported(number: isize) -> isize { + let should_report = if (0..512).contains(&number) { + let bit = 1u64 << (number as usize & 63); + REPORTED_SYSCALLS[number as usize / 64].fetch_or(bit, Ordering::Relaxed) & bit == 0 + } else { + true + }; + if should_report { + println!("as_musl: unsupported syscall {}", number); + } + neg_errno(ENOSYS) +} + +#[no_mangle] +pub unsafe extern "C" fn alloy_syscall( + number: isize, + a1: isize, + a2: isize, + a3: isize, + a4: isize, + a5: isize, + a6: isize, +) -> isize { + match number { + SYS_READ => read_one(a1 as u32, a2 as *mut u8, a3 as usize), + SYS_WRITE => write_one(a1 as u32, a2 as *const u8, a3 as usize), + SYS_READV => read_vectored(a1 as u32, a2 as *mut IoVec, a3 as usize), + SYS_WRITEV => write_vectored(a1 as u32, a2 as *const IoVec, a3 as usize), + SYS_OPEN => open_file(a1 as *const c_char, a2), + SYS_OPENAT if a1 == AT_FDCWD => open_file(a2 as *const c_char, a3), + SYS_OPENAT => neg_errno(EINVAL), + SYS_CLOSE => match libos!(close(a1 as u32)) { + Ok(()) => 0, + Err(error) => fd_error(error, false), + }, + SYS_LSEEK => match libos!(lseek64(a1 as u32, a2 as i64, a3 as i32)) { + Ok(offset) => offset as isize, + Err(error) => fd_error(error, false), + }, + SYS_GETDENTS64 => { + if (a2 as *mut u8).is_null() && a3 != 0 { + return neg_errno(EFAULT); + } + let buffer = core::slice::from_raw_parts_mut(a2 as *mut u8, a3 as usize); + match libos!(getdents(a1 as u32, buffer)) { + Ok(size) => size as isize, + Err(error) => fd_error(error, false), + } + } + SYS_STAT => stat_path(a1 as *const c_char, a2 as *mut Stat), + SYS_FSTAT => { + if (a2 as *mut Stat).is_null() { + return neg_errno(EFAULT); + } + if (0..=2).contains(&a1) { + core::ptr::write(a2 as *mut Stat, character_device_stat()); + return 0; + } + match libos!(stat(a1 as u32)) { + Ok(stat) => { + core::ptr::write(a2 as *mut Stat, stat); + 0 + } + Err(error) => fd_error(error, false), + } + } + SYS_NEWFSTATAT if a1 >= 0 && a4 & AT_EMPTY_PATH != 0 => { + alloy_syscall(SYS_FSTAT, a1, a3, 0, 0, 0, 0) + } + SYS_NEWFSTATAT if a1 == AT_FDCWD => stat_path(a2 as *const c_char, a3 as *mut Stat), + SYS_MMAP => { + let mut prot = ProtFlags::empty(); + if a3 & PROT_READ != 0 { + prot |= ProtFlags::READ; + } + if a3 & PROT_WRITE != 0 { + prot |= ProtFlags::WRITE; + } + if a3 & PROT_EXEC != 0 { + prot |= ProtFlags::EXEC; + } + let fd = if a5 < 0 { u32::MAX } else { a5 as u32 }; + match libos!(mmap(a1 as usize, a2 as usize, prot, fd)) { + Ok(addr) => addr as isize, + Err(_) => neg_errno(ENOMEM), + } + } + SYS_MUNMAP => { + if a1 == 0 { + return neg_errno(EINVAL); + } + let region = core::slice::from_raw_parts_mut(a1 as *mut u8, a2 as usize); + match libos!(munmap(region, false)) { + Ok(()) => 0, + Err(_) => neg_errno(EINVAL), + } + } + SYS_MPROTECT => { + let mut prot = ProtFlags::empty(); + if a3 & PROT_READ != 0 { + prot |= ProtFlags::READ; + } + if a3 & PROT_WRITE != 0 { + prot |= ProtFlags::WRITE; + } + if a3 & PROT_EXEC != 0 { + prot |= ProtFlags::EXEC; + } + match libos!(mprotect(a1 as usize, a2 as usize, prot)) { + Ok(()) => 0, + Err(_) => neg_errno(EINVAL), + } + } + SYS_MREMAP => { + if a1 == 0 || a2 == 0 || a3 == 0 || a4 != MREMAP_MAYMOVE || a5 != 0 { + return neg_errno(EINVAL); + } + let prot = ProtFlags::READ | ProtFlags::WRITE; + let new_addr = match libos!(mmap(0, a3 as usize, prot, u32::MAX)) { + Ok(addr) => addr, + Err(_) => return neg_errno(ENOMEM), + }; + core::ptr::copy_nonoverlapping( + a1 as *const u8, + new_addr as *mut u8, + core::cmp::min(a2 as usize, a3 as usize), + ); + let old_region = core::slice::from_raw_parts_mut(a1 as *mut u8, a2 as usize); + if libos!(munmap(old_region, false)).is_err() { + let new_region = core::slice::from_raw_parts_mut(new_addr as *mut u8, a3 as usize); + let _ = libos!(munmap(new_region, false)); + return neg_errno(EINVAL); + } + new_addr as isize + } + SYS_MADVISE => 0, + SYS_BRK => neg_errno(ENOMEM), + // CPython is configured not to install signal handlers, but musl still + // masks application signals while updating its pthread key table. + SYS_RT_SIGPROCMASK => 0, + SYS_IOCTL => neg_errno(ENOTTY), + SYS_GETCWD => { + if a1 == 0 { + return neg_errno(EFAULT); + } + if a2 < 2 { + return neg_errno(ENOMEM); + } + let buffer = core::slice::from_raw_parts_mut(a1 as *mut u8, a2 as usize); + buffer[0] = b'/'; + buffer[1] = 0; + 2 + } + SYS_READLINK => neg_errno(EINVAL), + SYS_FCNTL => match a2 { + F_GETFD | F_GETFL => 0, + F_SETFD | F_SETFL => 0, + _ => neg_errno(EINVAL), + }, + SYS_FUTEX => libos!(futex( + a1 as *mut i32, + a2 as i32, + a3 as i32, + a4 as *const u8, + a5 as *mut i32, + a6 as i32 + )), + SYS_GETTID => libos!(gettid()), + SYS_CLOCK_GETTIME => { + if (a2 as *mut LinuxTimeSpec).is_null() { + return neg_errno(EFAULT); + } + let nanos = libos!(get_time()); + if nanos == u64::MAX { + neg_errno(EINVAL) + } else { + ptr::write( + a2 as *mut LinuxTimeSpec, + LinuxTimeSpec { + sec: (nanos / 1_000_000_000) as i64, + nsec: (nanos % 1_000_000_000) as i64, + }, + ); + 0 + } + } + SYS_GETRANDOM => libos!(getrandom(a1 as *mut u8, a2 as usize, a3 as u32)), + _ => report_unsupported(number), + } +} + +extern "C" { + fn alloy_musl_thread_init(); + fn alloy_musl_flush() -> i32; +} + +unsafe fn run_c_main_impl( + function_name: &str, + c_main: unsafe extern "C" fn(i32, *mut *mut c_char) -> i32, + flush: bool, +) -> FaaSFuncResult<()> { + alloy_musl_thread_init(); + + // Process-style runtimes may not leave the app allocator usable for + // teardown. The no-flush entry preallocates its result before C runs. + let preallocated_success = if flush { None } else { Some(().into()) }; + let mut storage: Vec> = Vec::new(); + let mut program = function_name.as_bytes().to_vec(); + program.push(0); + storage.push(program); + + for (key, value) in as_std::args::all() { + if key.as_bytes().contains(&0) || value.as_bytes().contains(&0) { + return Err(String::from("C argv contains NUL").into()); + } + let mut arg = format!("--{}={}", key, value).into_bytes(); + arg.push(0); + storage.push(arg); + } + + let mut argv: Vec<*mut c_char> = storage + .iter_mut() + .map(|value| value.as_mut_ptr() as *mut c_char) + .collect(); + argv.push(core::ptr::null_mut()); + + let code = c_main((argv.len() - 1) as i32, argv.as_mut_ptr()); + let flush_code = if flush { alloy_musl_flush() } else { 0 }; + if !flush { + // The isolation owns this heap and reclaims it when unloaded. Avoid + // touching allocator metadata after a process-style libc runtime ran. + core::mem::forget(argv); + core::mem::forget(storage); + } + if code != 0 { + Err(format!("C main returned {}", code).into()) + } else if flush_code != 0 { + Err(String::from("musl fflush failed").into()) + } else { + Ok(preallocated_success.unwrap_or_else(|| ().into())) + } +} + +pub unsafe fn run_c_main( + function_name: &str, + c_main: unsafe extern "C" fn(i32, *mut *mut c_char) -> i32, +) -> FaaSFuncResult<()> { + run_c_main_impl(function_name, c_main, true) +} + +pub unsafe fn run_c_main_no_flush( + function_name: &str, + c_main: unsafe extern "C" fn(i32, *mut *mut c_char) -> i32, +) -> FaaSFuncResult<()> { + run_c_main_impl(function_name, c_main, false) +} + +pub macro entry($c_main:ident) { + extern "C" { + fn $c_main(argc: i32, argv: *mut *mut core::ffi::c_char) -> i32; + } + + #[no_mangle] + pub fn main() -> $crate::FaaSFuncResult<()> { + unsafe { $crate::run_c_main(env!("CARGO_PKG_NAME"), $c_main) } + } +} + +pub macro entry_no_flush($c_main:ident) { + extern "C" { + fn $c_main(argc: i32, argv: *mut *mut core::ffi::c_char) -> i32; + } + + #[no_mangle] + pub fn main() -> $crate::FaaSFuncResult<()> { + unsafe { $crate::run_c_main_no_flush(env!("CARGO_PKG_NAME"), $c_main) } + } +} + +pub use as_std::agent::FaaSFuncResult; diff --git a/as_std/Cargo.toml b/as_std/Cargo.toml index 0926c47b..3ee48610 100644 --- a/as_std/Cargo.toml +++ b/as_std/Cargo.toml @@ -14,6 +14,7 @@ as_hostcall = { workspace = true, features = [ "fdtab", "fatfs", "mm", + "sys", ] } as_std_proc_macro = { path = "../as_std_proc_macro" } diff --git a/as_std/src/agent.rs b/as_std/src/agent.rs index c86edca1..6b9760f8 100644 --- a/as_std/src/agent.rs +++ b/as_std/src/agent.rs @@ -30,6 +30,16 @@ impl FaaSFuncError { } } +#[cfg(not(feature = "file-based"))] +pub fn buffer_len(slot: &str) -> Option { + crate::libos::libos!(buffer_len(slot)) +} + +#[cfg(not(feature = "file-based"))] +pub fn buffer_set_len(slot: &str, data_len: usize) -> as_hostcall::mm::MMResult<()> { + crate::libos::libos!(buffer_set_len(slot, data_len)) +} + #[cfg(not(feature = "file-based"))] mod refer_based_impl { use core::{alloc::Layout, borrow::Borrow, mem::ManuallyDrop}; @@ -70,7 +80,10 @@ mod refer_based_impl { } else { let fingerprint = T::__fingerprint(); - libos!(buffer_alloc(&slot, l, fingerprint)).expect("alloc failed.") as *mut T + libos!(buffer_alloc(&slot, l, fingerprint)) + .unwrap_or_else(|error| { + panic!("alloc failed for buffer slot {:?}: {:?}", slot, error) + }) as *mut T // let val = T::default(); // println!("will write addr=0x{:x}", addr as usize); @@ -90,9 +103,9 @@ mod refer_based_impl { } pub fn from_buffer_slot(slot: String) -> Option { - let buffer_meta: Option<(usize, u64)> = libos!(access_buffer(&slot)); + let buffer_meta: Option<(usize, u64, usize)> = libos!(access_buffer(&slot)); - buffer_meta.map(|(raw_ptr, fingerprint)| { + buffer_meta.map(|(raw_ptr, fingerprint, _data_len)| { if fingerprint != T::__fingerprint() { println!("wrong data type, {}, {}", fingerprint, T::__fingerprint()); panic!(""); @@ -106,6 +119,21 @@ mod refer_based_impl { } }) } + + /// Takes a buffer produced by the same loaded module and reclaims it + /// when the returned value is dropped. + /// + /// # Safety + /// + /// Any heap allocations owned by `T` must have been allocated by the + /// same global allocator instance as the caller. In particular, this + /// must not be used to reclaim a buffer created by another `.so`. + pub unsafe fn from_buffer_slot_owned(slot: String) -> Option { + Self::from_buffer_slot(slot).map(|mut buffer| { + buffer.used = true; + buffer + }) + } } impl Default for DataBuffer @@ -152,7 +180,11 @@ mod refer_based_impl { fn drop(&mut self) { if self.used { let ptr = Box::into_raw(unsafe { ManuallyDrop::take(&mut self.inner) }); - // println!("drop DataBuffer val: 0x{:x}", ptr as usize); + // The Box storage comes from the LibOS buffer allocator, while + // fields owned by T (for example String) use the user heap. + // Run T's destructor first, then return only the outer storage + // to the buffer allocator. + unsafe { core::ptr::drop_in_place(ptr) }; libos!(buffer_dealloc(ptr as usize, Layout::new::())); } } diff --git a/as_std/src/args.rs b/as_std/src/args.rs index 2c1cfd08..26bf9130 100644 --- a/as_std/src/args.rs +++ b/as_std/src/args.rs @@ -5,6 +5,25 @@ pub(crate) struct ArgsItem { } pub fn get(name: &str) -> Option<&'static str> { + args_list() + .iter() + .find(|item| item.key == name) + .map(|item| item.val.as_str()) +} + +pub fn all() -> Vec<(String, String)> { + args_list() + .iter() + .map(|item| { + ( + String::from(item.key.as_str()), + String::from(item.val.as_str()), + ) + }) + .collect() +} + +fn args_list() -> &'static heapless::Vec { let mut args_base_addr: usize; unsafe { core::arch::asm!( @@ -13,13 +32,6 @@ pub fn get(name: &str) -> Option<&'static str> { }; let page_size = 0x1000; let args_base_addr = (args_base_addr + page_size - 1) & (!page_size + 1); - let args_list = unsafe { &mut *(args_base_addr as *mut heapless::Vec) }; - - for item in args_list { - if item.key == name { - return Some(item.val.as_str()); - } - } - - None + unsafe { &*(args_base_addr as *const heapless::Vec) } } +use alloc::{string::String, vec::Vec}; diff --git a/as_std/src/libos/mod.rs b/as_std/src/libos/mod.rs index 864a8f74..7381ae49 100644 --- a/as_std/src/libos/mod.rs +++ b/as_std/src/libos/mod.rs @@ -25,10 +25,14 @@ pub struct UserHostCall { write_addr: Option, open_addr: Option, + open_dir_addr: Option, + getdents_addr: Option, read_addr: Option, close_addr: Option, lseek_addr: Option, + lseek64_addr: Option, stat_addr: Option, + path_stat_addr: Option, readdir_addr: Option, connect_addr: Option, socket_addr: Option, @@ -42,7 +46,10 @@ pub struct UserHostCall { fatfs_read_addr: Option, fatfs_close_addr: Option, fatfs_seek_addr: Option, + fatfs_seek64_addr: Option, fatfs_stat_addr: Option, + fatfs_path_stat_addr: Option, + fatfs_readdir_addr: Option, smoltcp_addrinfo_addr: Option, smoltcp_connect_addr: Option, @@ -53,6 +60,10 @@ pub struct UserHostCall { smoltcp_close_addr: Option, alloc_buffer_addr: Option, + alloc_buffer_raw_addr: Option, + register_buffer_addr: Option, + set_buffer_len_addr: Option, + buffer_len_addr: Option, access_buffer_addr: Option, dealloc_buffer_addr: Option, mmap_addr: Option, @@ -67,6 +78,9 @@ pub struct UserHostCall { nanosleep_addr: Option, sigaction_addr: Option, + futex_addr: Option, + gettid_addr: Option, + getrandom_addr: Option, } impl UserHostCall { @@ -85,10 +99,14 @@ impl UserHostCall { CommonHostCall::Write => &mut self.write_addr, CommonHostCall::Open => &mut self.open_addr, + CommonHostCall::OpenDir => &mut self.open_dir_addr, + CommonHostCall::GetDents => &mut self.getdents_addr, CommonHostCall::Read => &mut self.read_addr, CommonHostCall::Close => &mut self.close_addr, CommonHostCall::Lseek => &mut self.lseek_addr, + CommonHostCall::Lseek64 => &mut self.lseek64_addr, CommonHostCall::Stat => &mut self.stat_addr, + CommonHostCall::PathStat => &mut self.path_stat_addr, CommonHostCall::ReadDir => &mut self.readdir_addr, CommonHostCall::Connect => &mut self.connect_addr, CommonHostCall::Socket => &mut self.socket_addr, @@ -102,7 +120,10 @@ impl UserHostCall { CommonHostCall::FatfsRead => &mut self.fatfs_read_addr, CommonHostCall::FatfsClose => &mut self.fatfs_close_addr, CommonHostCall::FatfsSeek => &mut self.fatfs_seek_addr, + CommonHostCall::FatfsSeek64 => &mut self.fatfs_seek64_addr, CommonHostCall::FatfsStat => &mut self.fatfs_stat_addr, + CommonHostCall::FatfsPathStat => &mut self.fatfs_path_stat_addr, + CommonHostCall::FatfsReadDir => &mut self.fatfs_readdir_addr, CommonHostCall::SmoltcpAddrInfo => &mut self.smoltcp_addrinfo_addr, CommonHostCall::SmoltcpConnect => &mut self.smoltcp_connect_addr, @@ -113,6 +134,10 @@ impl UserHostCall { CommonHostCall::SmoltcpClose => &mut self.smoltcp_close_addr, CommonHostCall::BufferAlloc => &mut self.alloc_buffer_addr, + CommonHostCall::BufferAllocRaw => &mut self.alloc_buffer_raw_addr, + CommonHostCall::BufferRegister => &mut self.register_buffer_addr, + CommonHostCall::BufferSetLen => &mut self.set_buffer_len_addr, + CommonHostCall::BufferLen => &mut self.buffer_len_addr, CommonHostCall::AccessBuffer => &mut self.access_buffer_addr, CommonHostCall::BufferDealloc => &mut self.dealloc_buffer_addr, CommonHostCall::Mmap => &mut self.mmap_addr, @@ -127,6 +152,9 @@ impl UserHostCall { CommonHostCall::NanoSleep => &mut self.nanosleep_addr, CommonHostCall::SigAction => &mut self.sigaction_addr, + CommonHostCall::Futex => &mut self.futex_addr, + CommonHostCall::GetTid => &mut self.gettid_addr, + CommonHostCall::GetRandom => &mut self.getrandom_addr, }; if entry_addr.is_none() { diff --git a/as_std/src/libos/utils.rs b/as_std/src/libos/utils.rs index fbf0d083..1fa447ab 100644 --- a/as_std/src/libos/utils.rs +++ b/as_std/src/libos/utils.rs @@ -6,10 +6,14 @@ pub macro func_type { (spawn_fault_handler) => (as_hostcall::types::SpawnFaultThreadFunc), (write) => (as_hostcall::fdtab::WriteFunc), (open) => (as_hostcall::fdtab::OpenFunc), + (open_dir) => (as_hostcall::fdtab::OpenDirFunc), + (getdents) => (as_hostcall::fdtab::GetDentsFunc), (read) => (as_hostcall::fdtab::ReadFunc), (close) => (as_hostcall::fdtab::CloseFunc), (lseek) => (as_hostcall::fdtab::LseekFunc), + (lseek64) => (as_hostcall::fdtab::Lseek64Func), (stat) => (as_hostcall::fdtab::StatFunc), + (path_stat) => (as_hostcall::fdtab::PathStatFunc), (readdir) => (as_hostcall::fdtab::ReadDirFunc), (connect) => (as_hostcall::fdtab::ConnectFunc), (bind) => (as_hostcall::fdtab::BindFunc), @@ -19,7 +23,10 @@ pub macro func_type { (fatfs_read) => (as_hostcall::fatfs::FatfsReadFunc), (fatfs_close) => (as_hostcall::fatfs::FatfsCloseFunc), (fatfs_seek) => (as_hostcall::fatfs::FatfsSeekFunc), + (fatfs_seek64) => (as_hostcall::fatfs::FatfsSeek64Func), (fatfs_stat) => (as_hostcall::fatfs::FatfsStatFunc), + (fatfs_path_stat) => (as_hostcall::fatfs::FatfsPathStatFunc), + (fatfs_readdir) => (as_hostcall::fatfs::FatfsReadDirFunc), (stdout) => (as_hostcall::types::HostStdioFunc), (addrinfo) => (as_hostcall::socket::SmoltcpAddrInfoFunc), (smol_connect) => (as_hostcall::socket::SmoltcpConnectFunc), @@ -29,6 +36,10 @@ pub macro func_type { (smol_accept) => (as_hostcall::socket::SmoltcpAcceptFunc), (smol_close) => (as_hostcall::socket::SmoltcpCloseFunc), (buffer_alloc) => (as_hostcall::mm::BufferAllocFunc), + (buffer_alloc_raw) => (as_hostcall::mm::BufferAllocRawFunc), + (buffer_register) => (as_hostcall::mm::BufferRegisterFunc), + (buffer_set_len) => (as_hostcall::mm::BufferSetLenFunc), + (buffer_len) => (as_hostcall::mm::BufferLenFunc), (access_buffer) => (as_hostcall::mm::AccessBufferFunc), (buffer_dealloc) => (as_hostcall::mm::BufferDeallocFunc), (mmap) => (as_hostcall::mm::MemmapFunc), @@ -39,6 +50,9 @@ pub macro func_type { (get_time) => (as_hostcall::types::GetTimeFunc), (nanosleep) => (as_hostcall::types::NanoSleepFunc), (sigaction) => (as_hostcall::signal::SigActionFunc), + (futex) => (as_hostcall::sys::FutexFunc), + (gettid) => (as_hostcall::sys::GetTidFunc), + (getrandom) => (as_hostcall::sys::GetRandomFunc), } pub macro hostcall_id { @@ -47,10 +61,14 @@ pub macro hostcall_id { (spawn_fault_handler) => (as_hostcall::CommonHostCall::SpawnFaultThread), (write) => (as_hostcall::CommonHostCall::Write), (open) => (as_hostcall::CommonHostCall::Open), + (open_dir) => (as_hostcall::CommonHostCall::OpenDir), + (getdents) => (as_hostcall::CommonHostCall::GetDents), (read) => (as_hostcall::CommonHostCall::Read), (close) => (as_hostcall::CommonHostCall::Close), (lseek) => (as_hostcall::CommonHostCall::Lseek), + (lseek64) => (as_hostcall::CommonHostCall::Lseek64), (stat) => (as_hostcall::CommonHostCall::Stat), + (path_stat) => (as_hostcall::CommonHostCall::PathStat), (readdir) => (as_hostcall::CommonHostCall::ReadDir), (connect) => (as_hostcall::CommonHostCall::Connect), (bind) => (as_hostcall::CommonHostCall::Bind), @@ -60,7 +78,10 @@ pub macro hostcall_id { (fatfs_read) => (as_hostcall::CommonHostCall::FatfsRead), (fatfs_close) => (as_hostcall::CommonHostCall::FatfsClose), (fatfs_seek) => (as_hostcall::CommonHostCall::FatfsSeek), + (fatfs_seek64) => (as_hostcall::CommonHostCall::FatfsSeek64), (fatfs_stat) => (as_hostcall::CommonHostCall::FatfsStat), + (fatfs_path_stat) => (as_hostcall::CommonHostCall::FatfsPathStat), + (fatfs_readdir) => (as_hostcall::CommonHostCall::FatfsReadDir), (stdout) => (as_hostcall::CommonHostCall::Stdout), (addrinfo) => (as_hostcall::CommonHostCall::SmoltcpAddrInfo), (smol_connect) => (as_hostcall::CommonHostCall::SmoltcpConnect), @@ -70,6 +91,10 @@ pub macro hostcall_id { (smol_accept) => (as_hostcall::CommonHostCall::SmoltcpAccept), (smol_close) => (as_hostcall::CommonHostCall::SmoltcpClose), (buffer_alloc) => (as_hostcall::CommonHostCall::BufferAlloc), + (buffer_alloc_raw) => (as_hostcall::CommonHostCall::BufferAllocRaw), + (buffer_register) => (as_hostcall::CommonHostCall::BufferRegister), + (buffer_set_len) => (as_hostcall::CommonHostCall::BufferSetLen), + (buffer_len) => (as_hostcall::CommonHostCall::BufferLen), (access_buffer) => (as_hostcall::CommonHostCall::AccessBuffer), (buffer_dealloc) => (as_hostcall::CommonHostCall::BufferDealloc), (mmap) => (as_hostcall::CommonHostCall::Mmap), @@ -80,6 +105,9 @@ pub macro hostcall_id { (get_time) => (as_hostcall::CommonHostCall::GetTime), (nanosleep) => (as_hostcall::CommonHostCall::NanoSleep), (sigaction) => (as_hostcall::CommonHostCall::SigAction), + (futex) => (as_hostcall::CommonHostCall::Futex), + (gettid) => (as_hostcall::CommonHostCall::GetTid), + (getrandom) => (as_hostcall::CommonHostCall::GetRandom), } pub macro libos { diff --git a/as_std/src/time.rs b/as_std/src/time.rs index b297cf6b..c64fd290 100644 --- a/as_std/src/time.rs +++ b/as_std/src/time.rs @@ -4,21 +4,21 @@ use crate::libos::libos; #[derive(Clone, Copy)] pub struct SystemTime { - nanos: u128, + nanos: u64, } pub const UNIX_EPOCH: SystemTime = SystemTime { nanos: 0 }; impl SystemTime { pub fn now() -> Self { - let nanos = libos!(get_time()).unwrap(); - // println!("get_time -> {}", nanos); + let nanos = libos!(get_time()); + assert_ne!(nanos, u64::MAX, "get_time failed"); Self { nanos } } pub fn duration_since(&self, earlier: SystemTime) -> Duration { let sub = self.nanos - earlier.nanos; - Duration::new((sub / 1_000_000_000) as u64, (sub % 1_000_000_000) as u32) + Duration::new(sub / 1_000_000_000, (sub % 1_000_000_000) as u32) } pub fn elapsed(&self) -> Duration { diff --git a/bins/asvisor/Cargo.toml b/bins/asvisor/Cargo.toml index e2998c28..5d85ce98 100644 --- a/bins/asvisor/Cargo.toml +++ b/bins/asvisor/Cargo.toml @@ -14,6 +14,7 @@ clap = { version = "4.3.21", features = ["derive"] } derive_more = "0.99.17" thiserror-no-std = "2.0.2" anyhow = { version = "1.0.82" } +libc = "0.2" tokio = { version = "1.32.0", features = [ "macros", "rt-multi-thread", diff --git a/bins/asvisor/src/main.rs b/bins/asvisor/src/main.rs index 4a1fc96b..176d0f2e 100644 --- a/bins/asvisor/src/main.rs +++ b/bins/asvisor/src/main.rs @@ -1,4 +1,4 @@ -use std::{sync::Arc, thread::sleep, time::Duration}; +use std::{io, sync::Arc, thread::sleep, time::Duration}; use clap::{arg, Parser}; use derive_more::Display; @@ -50,6 +50,29 @@ struct Args { /// block after workflow execution. #[arg(short, long, default_value_t = false)] non_exit: bool, + + /// Allow fatal signals to generate a core dump. + #[arg(long, default_value_t = false)] + enable_core_dump: bool, +} + +fn disable_core_dumps() -> io::Result<()> { + let limit = libc::rlimit { + rlim_cur: 0, + rlim_max: 0, + }; + + // A piped core_pattern (for example systemd-coredump) may ignore + // RLIMIT_CORE. PR_SET_DUMPABLE prevents the dump from being generated at + // all, while the limit also covers regular file-based core dumps. + if unsafe { libc::prctl(libc::PR_SET_DUMPABLE, 0, 0, 0, 0) } != 0 { + return Err(io::Error::last_os_error()); + } + if unsafe { libc::setrlimit(libc::RLIMIT_CORE, &limit) } != 0 { + return Err(io::Error::last_os_error()); + } + + Ok(()) } fn build_all_isol(args: &Args) -> Vec> { @@ -117,6 +140,9 @@ async fn async_msvisor_start(isols: &[Arc]) { fn main() { logger::init(); let args = Args::parse(); + if !args.enable_core_dump { + disable_core_dumps().expect("failed to disable core dumps"); + } let isols = build_all_isol(&args); #[cfg(feature = "multi_workflow")] diff --git a/build_musl_cpython.rs b/build_musl_cpython.rs new file mode 100644 index 00000000..07ce181b --- /dev/null +++ b/build_musl_cpython.rs @@ -0,0 +1,122 @@ +use std::{env, fs, path::PathBuf, process::Command}; + +fn run(command: &mut Command) { + assert!( + command + .status() + .expect("failed to execute build command") + .success(), + "build command failed: {:?}", + command + ); +} + +fn main() { + let crate_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let repo_root = crate_dir.parent().unwrap().parent().unwrap(); + let out_dir = PathBuf::from(env::var_os("OUT_DIR").unwrap()); + let musl_dir = repo_root.join("target/musl-alloy"); + let musl_lib = musl_dir.join("lib/libmusl_alloy.a"); + let cpython_dir = repo_root.join("target/cpython-musl"); + let python_include = PathBuf::from(env::var_os("PYTHON_INCLUDE").unwrap_or_else(|| { + repo_root.join("third_party/cpython/Include").into_os_string() + })); + let python_config_include = PathBuf::from( + env::var_os("PYTHON_CONFIG_INCLUDE") + .unwrap_or_else(|| cpython_dir.join("build-submodule").into_os_string()), + ); + let python_lib_dir = PathBuf::from( + env::var_os("PYTHON_LIB_DIR") + .unwrap_or_else(|| cpython_dir.join("build-submodule").into_os_string()), + ); + let python_lib = python_lib_dir.join("libpython3.11.a"); + let source = crate_dir.join("function.c"); + + assert!( + musl_lib.is_file(), + "missing {}; run `just musl` first", + musl_lib.display() + ); + assert!( + python_include.join("Python.h").is_file() && python_lib.is_file(), + "missing musl CPython; run `just musl_cpython`" + ); + + let cc = env::var_os("CC").unwrap_or_else(|| "cc".into()); + let gcc_include = Command::new(&cc) + .arg("-print-file-name=include") + .output() + .expect("failed to query compiler include directory"); + assert!(gcc_include.status.success()); + let gcc_include = String::from_utf8(gcc_include.stdout) + .unwrap() + .trim() + .to_owned(); + + let object = out_dir.join("function.o"); + run(Command::new(&cc) + .arg("-std=c11") + .arg("-O2") + .arg("-fPIC") + .arg("-ffreestanding") + .arg("-fno-stack-protector") + .arg("-fno-builtin") + .arg("-nostdinc") + .arg("-D_GNU_SOURCE") + .arg("-D_POSIX_C_SOURCE=200809L") + .arg("-isystem") + .arg(musl_dir.join("include")) + .arg("-I") + .arg(repo_root.join("as_musl/include")) + .arg("-isystem") + .arg(&python_config_include) + .arg("-isystem") + .arg(&python_include) + .arg("-isystem") + .arg(gcc_include) + .arg("-Dmain=alloy_c_main") + .arg("-c") + .arg(&source) + .arg("-o") + .arg(&object)); + + let app_archive = out_dir.join("liballoy_cpython_app.a"); + let ar = env::var_os("AR").unwrap_or_else(|| "ar".into()); + run(Command::new(ar).arg("rcs").arg(&app_archive).arg(&object)); + + println!("cargo:rerun-if-changed={}", source.display()); + println!("cargo:rerun-if-changed={}", musl_lib.display()); + println!("cargo:rerun-if-env-changed=PYTHON_INCLUDE"); + println!("cargo:rerun-if-env-changed=PYTHON_CONFIG_INCLUDE"); + println!("cargo:rerun-if-env-changed=PYTHON_LIB_DIR"); + println!("cargo:rustc-link-search=native={}", out_dir.display()); + println!("cargo:rustc-link-search=native={}", python_lib_dir.display()); + println!( + "cargo:rustc-link-search=native={}", + musl_dir.join("lib").display() + ); + println!("cargo:rustc-link-lib=static=alloy_cpython_app"); + println!("cargo:rustc-link-lib=static=python3.11"); + println!("cargo:rustc-link-lib=static=musl_alloy"); + println!("cargo:rustc-link-arg=-Wl,-Bsymbolic-functions"); + println!("cargo:rustc-link-arg=-Wl,--gc-sections"); + + let profile = env::var("PROFILE").unwrap(); + let target_dir = repo_root.join("target").join(&profile); + fs::create_dir_all(&target_dir).unwrap(); + let source_library = crate_dir + .join("target") + .join(&profile) + .join("libmusl_cpython.so"); + let target_library = target_dir.join("libmusl_cpython.so"); + + match fs::symlink_metadata(&target_library) { + Ok(metadata) if metadata.file_type().is_symlink() => {} + Ok(_) => panic!("{} exists and is not a symlink", target_library.display()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + #[cfg(unix)] + std::os::unix::fs::symlink(&source_library, &target_library).unwrap(); + } + Err(error) => panic!("inspect {} failed: {}", target_library.display(), error), + } +} diff --git a/build_musl_user.rs b/build_musl_user.rs new file mode 100644 index 00000000..42f0c80f --- /dev/null +++ b/build_musl_user.rs @@ -0,0 +1,104 @@ +use std::{env, fs, path::PathBuf, process::Command}; + +fn run(command: &mut Command) { + assert!( + command + .status() + .expect("failed to execute build command") + .success(), + "build command failed: {:?}", + command + ); +} + +fn main() { + let crate_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let repo_root = crate_dir.parent().unwrap().parent().unwrap(); + let out_dir = PathBuf::from(env::var_os("OUT_DIR").unwrap()); + let musl_dir = repo_root.join("target/musl-alloy"); + let alloy_include = repo_root.join("as_musl/include"); + let workload_include = repo_root.join("user/musl_parallel_common"); + let musl_lib = musl_dir.join("lib/libmusl_alloy.a"); + let source = crate_dir.join("function.c"); + + assert!( + musl_lib.is_file(), + "missing {}; run `just musl` first", + musl_lib.display() + ); + assert!(source.is_file(), "missing {}", source.display()); + + let cc = env::var_os("CC").unwrap_or_else(|| "cc".into()); + let gcc_include = Command::new(&cc) + .arg("-print-file-name=include") + .output() + .expect("failed to query compiler include directory"); + assert!(gcc_include.status.success()); + let gcc_include = String::from_utf8(gcc_include.stdout) + .unwrap() + .trim() + .to_owned(); + + let object = out_dir.join("function.o"); + run(Command::new(&cc) + .arg("-std=c11") + .arg("-O2") + .arg("-fPIC") + .arg("-ffreestanding") + .arg("-fno-stack-protector") + .arg("-fno-builtin") + .arg("-nostdinc") + .arg("-D_GNU_SOURCE") + .arg("-D_POSIX_C_SOURCE=200809L") + .arg("-isystem") + .arg(musl_dir.join("include")) + .arg("-isystem") + .arg(gcc_include) + .arg("-I") + .arg(&alloy_include) + .arg("-I") + .arg(&workload_include) + .arg("-Dmain=alloy_c_main") + .arg("-c") + .arg(&source) + .arg("-o") + .arg(&object)); + + let app_archive = out_dir.join("liballoy_c_app.a"); + let ar = env::var_os("AR").unwrap_or_else(|| "ar".into()); + run(Command::new(ar).arg("rcs").arg(&app_archive).arg(&object)); + + println!("cargo:rerun-if-changed={}", source.display()); + println!("cargo:rerun-if-changed={}", alloy_include.display()); + println!("cargo:rerun-if-changed={}", workload_include.display()); + println!("cargo:rerun-if-changed={}", musl_lib.display()); + println!("cargo:rustc-link-search=native={}", out_dir.display()); + println!( + "cargo:rustc-link-search=native={}", + musl_dir.join("lib").display() + ); + println!("cargo:rustc-link-lib=static=alloy_c_app"); + println!("cargo:rustc-link-lib=static=musl_alloy"); + println!("cargo:rustc-link-arg=-Wl,-Bsymbolic-functions"); + println!("cargo:rustc-link-arg=-Wl,--gc-sections"); + + let profile = env::var("PROFILE").unwrap(); + let function_name = crate_dir.file_name().unwrap().to_string_lossy(); + let target_dir = repo_root.join("target").join(&profile); + fs::create_dir_all(&target_dir).unwrap(); + let source_library = crate_dir + .join("target") + .join(&profile) + .join(format!("lib{}.so", function_name)); + let target_library = target_dir.join(format!("lib{}.so", function_name)); + + match fs::symlink_metadata(&target_library) { + Ok(metadata) if metadata.file_type().is_symlink() => {} + Ok(_) => panic!("{} exists and is not a symlink", target_library.display()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + #[cfg(unix)] + std::os::unix::fs::symlink(&source_library, &target_library).unwrap(); + } + Err(error) => panic!("inspect {} failed: {}", target_library.display(), error), + } +} diff --git a/common_service/fatfs/src/apis.rs b/common_service/fatfs/src/apis.rs index 07b3ee93..12ede9be 100644 --- a/common_service/fatfs/src/apis.rs +++ b/common_service/fatfs/src/apis.rs @@ -48,6 +48,24 @@ pub fn fatfs_seek(fd: Fd, pos: u32) -> FatfsResult<()> { Ok(()) } +#[no_mangle] +pub fn fatfs_seek64(fd: Fd, offset: i64, whence: i32) -> FatfsResult { + let mut table = FTABLE + .lock() + .map_err(|e| FatfsError::AcquireLockErr(e.to_string()))?; + + let file = table.get_file_mut(fd).ok_or(FatfsError::BadInputFd(fd))?; + let from = match whence { + 0 if offset >= 0 => std::io::SeekFrom::Start(offset as u64), + 1 => std::io::SeekFrom::Current(offset), + 2 => std::io::SeekFrom::End(offset), + _ => return Err(FatfsError::Unknown), + }; + file.seek(from) + .map(|pos| pos as i64) + .map_err(|e| FatfsError::HostIOErr(e.to_string())) +} + #[no_mangle] pub fn fatfs_stat(fd: Fd) -> FatfsResult { let mut table = FTABLE @@ -87,6 +105,82 @@ pub fn fatfs_stat(fd: Fd) -> FatfsResult { }) } +fn make_stat(size: Size, mode: u32) -> Stat { + Stat { + st_dev: 0, + st_ino: 0, + st_nlink: 1, + st_mode: mode, + st_uid: 0, + st_gid: 0, + __pad0: 0, + st_rdev: 0, + st_size: size, + st_blksize: 0, + st_blocks: 0, + st_atime: TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + st_mtime: TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + st_ctime: TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + __unused: [0, 0, 0], + } +} + +#[no_mangle] +pub fn fatfs_path_stat(path: &str) -> FatfsResult { + // fatfs::FileSystem uses RefCell for the underlying disk. Serialize + // metadata operations with open/read/write operations in FTABLE. + let _table = FTABLE + .lock() + .map_err(|e| FatfsError::AcquireLockErr(e.to_string()))?; + let root = get_fs_ref().root_dir(); + let path = path.trim_start_matches('/'); + + if path.is_empty() || root.open_dir(path).is_ok() { + return Ok(make_stat(0, 0o040755)); + } + + let mut file = root.open_file(path).map_err(|_| FatfsError::Unknown)?; + let size = file + .stream_len() + .map_err(|error| FatfsError::HostIOErr(error.to_string()))?; + Ok(make_stat(size as Size, 0o100644)) +} + +#[no_mangle] +pub fn fatfs_readdir(path: &str, buffer: &mut [u8]) -> FatfsResult { + let _table = FTABLE + .lock() + .map_err(|e| FatfsError::AcquireLockErr(e.to_string()))?; + let path = path.trim_start_matches('/'); + let dir = get_fs_ref() + .root_dir() + .open_dir(path) + .map_err(|_| FatfsError::Unknown)?; + let mut required = 0; + for entry in dir.iter() { + let entry = entry.map_err(|error| FatfsError::HostIOErr(error.to_string()))?; + let name = entry.file_name(); + let record_len = 3 + name.len(); + if required + record_len <= buffer.len() { + buffer[required] = if entry.is_dir() { 4 } else { 8 }; + buffer[required + 1..required + 3] + .copy_from_slice(&(name.len() as u16).to_ne_bytes()); + buffer[required + 3..required + record_len].copy_from_slice(name.as_bytes()); + } + required += record_len; + } + Ok(required) +} + #[no_mangle] pub fn fatfs_open(p: &str, flags: OpenFlags) -> FatfsResult { let mut table = FTABLE @@ -95,13 +189,17 @@ pub fn fatfs_open(p: &str, flags: OpenFlags) -> FatfsResult { let root_dir = get_fs_ref().root_dir(); - let file = if flags.contains(OpenFlags::O_CREAT) { + let mut file = if flags.contains(OpenFlags::O_CREAT) { root_dir.create_file(p) } else { root_dir.open_file(p) } .map_err(|_e| FatfsError::Unknown)?; + if flags.contains(OpenFlags::O_TRUNC) { + file.truncate().map_err(|_| FatfsError::Unknown)?; + } + let fd = { let file = ManuallyDrop::new(Box::new(file)); table.add_file(file.as_ref()) diff --git a/common_service/fdtab/src/apis.rs b/common_service/fdtab/src/apis.rs index ca3ee0b1..0565ec4b 100644 --- a/common_service/fdtab/src/apis.rs +++ b/common_service/fdtab/src/apis.rs @@ -1,6 +1,6 @@ use core::net::SocketAddrV4; -use alloc::borrow::ToOwned; +use alloc::{borrow::ToOwned, string::String, vec, vec::Vec}; use as_hostcall::{ fdtab::{FdtabError, FdtabResult}, types::{Fd, OpenFlags, OpenMode, Size, SockFd, Stat}, @@ -25,6 +25,11 @@ pub fn read(fd: Fd, buf: &mut [u8]) -> FdtabResult { Ok(match file.src { DataSource::FatFS(raw_fd) => libos!(fatfs_read(raw_fd, buf))?, DataSource::Net(socket) => libos!(recv(socket, buf))?, + DataSource::Directory { .. } => Err(FdtabError::UndefinedOperation { + op: "read".to_owned(), + fd, + fd_type: "Directory".to_owned(), + })?, }) }) } @@ -47,6 +52,11 @@ pub fn write(fd: Fd, buf: &[u8]) -> FdtabResult { Ok(match file.src { DataSource::FatFS(raw_fd) => libos!(fatfs_write(raw_fd, buf))?, DataSource::Net(sockfd) => libos!(send(sockfd, buf)).map(|_| buf.len())?, + DataSource::Directory { .. } => Err(FdtabError::UndefinedOperation { + op: "write".to_owned(), + fd, + fd_type: "Directory".to_owned(), + })?, }) }) } @@ -67,11 +77,41 @@ pub fn lseek(fd: Fd, pos: u32) -> FdtabResult<()> { fd, fd_type: "Net".to_owned(), })?, + DataSource::Directory { .. } => Err(FdtabError::UndefinedOperation { + op: "lseek".to_owned(), + fd, + fd_type: "Directory".to_owned(), + })?, }; Ok(()) }) } +#[no_mangle] +pub fn lseek64(fd: Fd, offset: i64, whence: i32) -> FdtabResult { + if let 0..=2 = fd { + Err(FdtabError::BadInputFd(">2".to_owned(), fd))? + } + + FD_TABLE.with_file(fd, |file| -> FdtabResult { + let file = file.ok_or(FdtabError::NoExistFd(fd))?; + + match file.src { + DataSource::FatFS(raw_fd) => Ok(libos!(fatfs_seek64(raw_fd, offset, whence))?), + DataSource::Net(_) => Err(FdtabError::UndefinedOperation { + op: "lseek64".to_owned(), + fd, + fd_type: "Net".to_owned(), + }), + DataSource::Directory { .. } => Err(FdtabError::UndefinedOperation { + op: "lseek64".to_owned(), + fd, + fd_type: "Directory".to_owned(), + }), + } + }) +} + #[no_mangle] pub fn stat(fd: Fd) -> FdtabResult { if let 0..=2 = fd { @@ -88,10 +128,134 @@ pub fn stat(fd: Fd) -> FdtabResult { fd, fd_type: "Net".to_owned(), })?, + DataSource::Directory { .. } => Stat { + st_dev: 0, + st_ino: 0, + st_nlink: 1, + st_mode: 0o040755, + st_uid: 0, + st_gid: 0, + __pad0: 0, + st_rdev: 0, + st_size: 0, + st_blksize: 0, + st_blocks: 0, + st_atime: as_hostcall::types::TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + st_mtime: as_hostcall::types::TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + st_ctime: as_hostcall::types::TimeSpec { + tv_sec: 0, + tv_nsec: 0, + }, + __unused: [0, 0, 0], + }, }) }) } +#[no_mangle] +pub fn path_stat(path: &str) -> FdtabResult { + Ok(libos!(fatfs_path_stat(path))?) +} + +#[no_mangle] +pub fn read_dir(path: &str) -> FdtabResult> { + let required = libos!(fatfs_readdir(path, &mut []))?; + let mut packed = vec![0; required]; + let written = libos!(fatfs_readdir(path, &mut packed))?; + packed.truncate(written); + + let mut entries = Vec::new(); + let mut cursor = 0; + while cursor < packed.len() { + if packed.len() - cursor < 3 { + return Err(FdtabError::RuxfsError("invalid directory record".to_owned())); + } + let entry_type = packed[cursor] as u32; + let name_len = u16::from_ne_bytes([packed[cursor + 1], packed[cursor + 2]]) as usize; + cursor += 3; + if name_len > packed.len() - cursor { + return Err(FdtabError::RuxfsError("invalid directory name".to_owned())); + } + let entry_name = core::str::from_utf8(&packed[cursor..cursor + name_len]) + .map_err(|_| FdtabError::RuxfsError("non-UTF-8 directory name".to_owned()))?; + entries.push(as_hostcall::types::DirEntry { + dir_path: String::from(path), + entry_name: String::from(entry_name), + entry_type, + }); + cursor += name_len; + } + Ok(entries) +} + +#[no_mangle] +pub fn open_dir(path: &str) -> FdtabResult { + const HEADER_LEN: usize = 19; + + let portable = read_dir(path)?; + let required = portable + .iter() + .map(|entry| (HEADER_LEN + entry.entry_name.len() + 1 + 7) & !7) + .sum(); + let mut entries = vec![0; required]; + let mut cursor = 0; + for (index, entry) in portable.iter().enumerate() { + let name = entry.entry_name.as_bytes(); + let record_len = (HEADER_LEN + name.len() + 1 + 7) & !7; + let record = &mut entries[cursor..cursor + record_len]; + record[0..8].copy_from_slice(&((index + 1) as u64).to_ne_bytes()); + record[8..16].copy_from_slice(&((index + 1) as i64).to_ne_bytes()); + record[16..18].copy_from_slice(&(record_len as u16).to_ne_bytes()); + record[18] = entry.entry_type as u8; + record[HEADER_LEN..HEADER_LEN + name.len()].copy_from_slice(name); + cursor += record_len; + } + Ok(FD_TABLE.add_file(File { + mode: OpenMode::RD, + src: DataSource::Directory { entries, cursor: 0 }, + })) +} + +#[no_mangle] +pub fn getdents(fd: Fd, buffer: &mut [u8]) -> FdtabResult { + FD_TABLE.with_file_mut(fd, |file| -> FdtabResult { + let file = file.ok_or(FdtabError::NoExistFd(fd))?; + let DataSource::Directory { entries, cursor } = &mut file.src else { + return Err(FdtabError::UndefinedOperation { + op: "getdents".to_owned(), + fd, + fd_type: "non-directory".to_owned(), + }); + }; + + let mut written = 0usize; + while *cursor < entries.len() { + if entries.len() - *cursor < 18 { + return Err(FdtabError::RuxfsError("invalid dirent record".to_owned())); + } + let record_len = + u16::from_ne_bytes([entries[*cursor + 16], entries[*cursor + 17]]) as usize; + if record_len == 0 || record_len > entries.len() - *cursor { + return Err(FdtabError::RuxfsError("invalid dirent length".to_owned())); + } + if record_len > buffer.len() - written { + break; + } + buffer[written..written + record_len] + .copy_from_slice(&entries[*cursor..*cursor + record_len]); + *cursor += record_len; + written += record_len; + } + Ok(written) + }) +} + #[no_mangle] pub fn open(path: &str, flags: OpenFlags, mode: OpenMode) -> FdtabResult { let raw_fd = libos!(fatfs_open(path, flags))?; @@ -114,6 +278,7 @@ pub fn close(fd: Fd) -> FdtabResult<()> { match file.src { DataSource::FatFS(raw_fd) => libos!(fatfs_close(raw_fd))?, DataSource::Net(socket) => libos!(smol_close(socket))?, + DataSource::Directory { .. } => {} }; Ok(()) } diff --git a/common_service/fdtab/src/lib.rs b/common_service/fdtab/src/lib.rs index 43153abe..41eb59da 100644 --- a/common_service/fdtab/src/lib.rs +++ b/common_service/fdtab/src/lib.rs @@ -16,6 +16,10 @@ pub mod apis; enum DataSource { FatFS(Fd), Net(SockFd), + Directory { + entries: Vec, + cursor: usize, + }, } #[derive(Clone)] diff --git a/common_service/mm/src/faas_buffer/mod.rs b/common_service/mm/src/faas_buffer/mod.rs index d1473ef6..70714ecf 100644 --- a/common_service/mm/src/faas_buffer/mod.rs +++ b/common_service/mm/src/faas_buffer/mod.rs @@ -4,15 +4,19 @@ use core::{alloc::Layout, ptr::NonNull}; use alloc::{borrow::ToOwned, string::String}; +use as_hostcall::{ + mm::{MMError, MMResult}, + SERVICE_HEAP_SIZE, +}; use linked_list_allocator::LockedHeap; -use as_hostcall::{mm::MMResult, SERVICE_HEAP_SIZE}; -use hashbrown::HashMap; +use hashbrown::{hash_map::Entry, HashMap}; use lazy_static::lazy_static; use spin::Mutex; lazy_static! { - static ref BUFFER_REGISTER: Mutex> = Mutex::new(HashMap::new()); + static ref BUFFER_REGISTER: Mutex> = + Mutex::new(HashMap::new()); static ref BUFFER_ALLOCATOR: LockedHeap = unsafe { LockedHeap::new( as_std::init_context::ISOLATION_CTX.lock().heap_range.1 as *mut u8, @@ -22,25 +26,58 @@ lazy_static! { } #[no_mangle] -pub fn buffer_alloc(slot: &str, l: Layout, fingerprint: u64) -> MMResult { +pub fn buffer_alloc_raw(l: Layout) -> MMResult { let addr = BUFFER_ALLOCATOR .lock() .allocate_first_fit(l) - .expect("alloc mem failed"); - - let addr = addr.as_ptr() as usize; + .map_err(|_| MMError::NoMemory)?; - BUFFER_REGISTER - .lock() - .insert(slot.to_owned(), (addr, fingerprint)); + Ok(addr.as_ptr() as usize) +} - // as_std::println!("buffer_alloc layout={:?}, addr=0x{:x}", l, addr); +#[no_mangle] +pub fn buffer_register( + slot: &str, + addr: usize, + fingerprint: u64, + data_len: usize, +) -> MMResult<()> { + let mut register = BUFFER_REGISTER.lock(); + match register.entry(slot.to_owned()) { + Entry::Vacant(entry) => { + entry.insert((addr, fingerprint, data_len)); + Ok(()) + } + Entry::Occupied(_) => Err(MMError::BufferSlotExists), + } +} +#[no_mangle] +pub fn buffer_alloc(slot: &str, l: Layout, fingerprint: u64) -> MMResult { + let addr = buffer_alloc_raw(l)?; + if let Err(error) = buffer_register(slot, addr, fingerprint, l.size()) { + buffer_dealloc(addr, l); + return Err(error); + } Ok(addr) } #[no_mangle] -pub fn access_buffer(slot: &str) -> Option<(usize, u64)> { +pub fn buffer_set_len(slot: &str, data_len: usize) -> MMResult<()> { + BUFFER_REGISTER + .lock() + .get_mut(slot) + .map(|metadata| metadata.2 = data_len) + .ok_or_else(|| MMError::InvaildArg("existing buffer slot".into(), 0)) +} + +#[no_mangle] +pub fn buffer_len(slot: &str) -> Option { + BUFFER_REGISTER.lock().get(slot).map(|metadata| metadata.2) +} + +#[no_mangle] +pub fn access_buffer(slot: &str) -> Option<(usize, u64, usize)> { let mut register = BUFFER_REGISTER.lock(); // as_std::println!("buffer register: "); // for (k, v) in register.iter() { diff --git a/common_service/mm/src/mmap.rs b/common_service/mm/src/mmap.rs index b0fbac3e..9531f054 100644 --- a/common_service/mm/src/mmap.rs +++ b/common_service/mm/src/mmap.rs @@ -3,25 +3,27 @@ extern crate alloc; use core::alloc::Layout; use alloc::borrow::ToOwned; -use libc::c_void; use as_hostcall::{ mm::{MMError, MMResult, ProtFlags}, types::Fd, }; use as_std::libos::libos; +use libc::c_void; const PAGE_SIZE: usize = 0x1000; +fn page_aligned_length(length: usize) -> MMResult { + length + .checked_add(PAGE_SIZE - 1) + .map(|length| length & !(PAGE_SIZE - 1)) + .filter(|length| *length != 0) + .ok_or(MMError::NoMemory) +} + #[no_mangle] pub fn libos_mmap(addr: usize, length: usize, prot: ProtFlags, fd: Fd) -> MMResult { - if length % PAGE_SIZE > 0 { - Err(MMError::InvaildArg( - "length % PAGE_SIZE == 0".to_owned(), - length, - ))? - } - - let layout = Layout::from_size_align(length, PAGE_SIZE)?; + let aligned_length = page_aligned_length(length)?; + let layout = Layout::from_size_align(aligned_length, PAGE_SIZE)?; let mmap_addr = unsafe { let addr = { @@ -31,21 +33,23 @@ pub fn libos_mmap(addr: usize, length: usize, prot: ProtFlags, fd: Fd) -> MMResu addr as *mut libc::c_void } }; - if libc::munmap(addr, length) != 0 { + if addr.is_null() { + Err(MMError::NoMemory)? + } + if libc::munmap(addr, aligned_length) != 0 { Err(MMError::LibcErr("munmap failed".to_owned()))? } if libc::mmap( addr, - length, + aligned_length, trans_protflag(prot), - libc::MAP_PRIVATE | libc::MAP_ANONYMOUS, + libc::MAP_PRIVATE | libc::MAP_ANONYMOUS | libc::MAP_FIXED, 0, 0, ) != addr { Err(MMError::LibcErr("mmap failed".to_owned()))? } - addr as usize }; @@ -53,7 +57,8 @@ pub fn libos_mmap(addr: usize, length: usize, prot: ProtFlags, fd: Fd) -> MMResu return Ok(mmap_addr); } - let mm_region = unsafe { core::slice::from_raw_parts_mut(mmap_addr as *mut c_void, length) }; + let mm_region = + unsafe { core::slice::from_raw_parts_mut(mmap_addr as *mut c_void, aligned_length) }; libos!(register_file_backend(mm_region, fd))?; // println!("finish mmap, mmap_addr={}", mmap_addr); @@ -66,7 +71,7 @@ pub fn libos_munmap(mem_region: &mut [u8], file_based: bool) -> MMResult<()> { libos!(unregister_file_backend(mem_region.as_ptr() as usize))?; } - let aligned_length = (mem_region.len() + PAGE_SIZE - 1) & (!PAGE_SIZE + 1); + let aligned_length = page_aligned_length(mem_region.len())?; unsafe { if libc::mprotect( mem_region.as_mut_ptr() as usize as *mut libc::c_void, @@ -87,7 +92,7 @@ pub fn libos_munmap(mem_region: &mut [u8], file_based: bool) -> MMResult<()> { #[no_mangle] pub fn libos_mprotect(addr: usize, length: usize, prot: ProtFlags) -> MMResult<()> { - let aligned_length = (length + PAGE_SIZE - 1) & (!PAGE_SIZE + 1); + let aligned_length = page_aligned_length(length)?; unsafe { if libc::mprotect( addr as *mut libc::c_void, diff --git a/common_service/sys/Cargo.lock b/common_service/sys/Cargo.lock new file mode 100644 index 00000000..f59be7a2 --- /dev/null +++ b/common_service/sys/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sys" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", + "libc", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/common_service/sys/Cargo.toml b/common_service/sys/Cargo.toml new file mode 100644 index 00000000..4bd35ab1 --- /dev/null +++ b/common_service/sys/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "sys" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_hostcall = { path = "../../as_hostcall", features = ["sys"] } +as_std = { path = "../../as_std", features = ["alloc_def", "panic_def"] } +libc = "0.2.150" + +[features] +default = [] +mpk = ["as_hostcall/enable_mpk", "as_std/mpk"] diff --git a/common_service/sys/src/lib.rs b/common_service/sys/src/lib.rs new file mode 100644 index 00000000..d5558c77 --- /dev/null +++ b/common_service/sys/src/lib.rs @@ -0,0 +1,42 @@ +#![no_std] + +pub use as_std; +use core::ffi::c_void; + +unsafe fn syscall_result(ret: libc::c_long) -> isize { + if ret == -1 { + -(*libc::__errno_location() as isize) + } else { + ret as isize + } +} + +#[no_mangle] +pub unsafe extern "C" fn host_futex( + uaddr: *mut i32, + op: i32, + val: i32, + timeout: *const u8, + uaddr2: *mut i32, + val3: i32, +) -> isize { + syscall_result(libc::syscall( + libc::SYS_futex, + uaddr, + op, + val, + timeout as *const c_void, + uaddr2, + val3, + )) +} + +#[no_mangle] +pub extern "C" fn host_gettid() -> isize { + unsafe { libc::syscall(libc::SYS_gettid) as isize } +} + +#[no_mangle] +pub unsafe extern "C" fn host_getrandom(buf: *mut u8, len: usize, flags: u32) -> isize { + syscall_result(libc::syscall(libc::SYS_getrandom, buf, len, flags)) +} diff --git a/common_service/time/src/lib.rs b/common_service/time/src/lib.rs index 4342266c..9d25d622 100644 --- a/common_service/time/src/lib.rs +++ b/common_service/time/src/lib.rs @@ -1,18 +1,22 @@ #![no_std] -extern crate alloc; -use alloc::string::{String, ToString}; use nix::{ libc::{timespec, CLOCK_REALTIME}, time::clock_gettime, }; #[no_mangle] -pub fn get_time() -> Result { - let r = clock_gettime(CLOCK_REALTIME.into()).map_err(|e| e.to_string())?; - let a = r.tv_sec() as u128 * 1_000_000_000; - let b = r.tv_nsec() as u128; - Ok(a + b) +pub extern "C" fn get_time() -> u64 { + let Ok(r) = clock_gettime(CLOCK_REALTIME.into()) else { + return u64::MAX; + }; + let Some(nanos) = (r.tv_sec() as u64) + .checked_mul(1_000_000_000) + .and_then(|seconds| seconds.checked_add(r.tv_nsec() as u64)) + else { + return u64::MAX; + }; + nanos } #[no_mangle] @@ -28,9 +32,9 @@ pub fn host_nanosleep(sec: u64, nsec: u64) { #[test] fn get_time_test() { - let t = get_time().unwrap(); - assert!(t > 1_697_111_969 * 1_000_000_000, "error time: {}", t); - assert!(t < 1_735_660_800 * 1_000_000_000, "error time: {}", t); + let t = get_time(); + assert_ne!(t, u64::MAX, "clock_gettime failed"); + assert!(t > 1_577_836_800 * 1_000_000_000, "error time: {}", t); } #[test] diff --git a/doc/musl_functions.md b/doc/musl_functions.md new file mode 100644 index 00000000..475a6947 --- /dev/null +++ b/doc/musl_functions.md @@ -0,0 +1,80 @@ +# Running C functions with musl + +AlloyStack can build C functions against a patched musl libc. The C source uses +normal libc APIs, while musl redirects operating-system-facing calls to +AlloyStack's LibOS services. + +## Prerequisites + +Initialize the pinned musl submodule once: + +```bash +git submodule update --init third_party/musl +``` + +Build the patched, position-independent musl archive: + +```bash +just musl +``` + +## Function layout + +A musl C function is a regular user crate with: + +```text +user// +├── Cargo.toml +├── function.c +└── src/lib.rs +``` + +The Rust library is a no-std shim: + +```rust +#![no_std] + +as_musl::entry!(alloy_c_main); +``` + +The C source keeps a standard entry point: + +```c +int main(int argc, char **argv) +{ + return 0; +} +``` + +The build script renames it to `alloy_c_main` without requiring a source edit. +Workflow arguments are exposed as `--key=value`; `argv[0]` is the function +crate name. + +Build a function with: + +```bash +just musl_func +``` + +The example workflow is: + +```bash +just musl_example +``` + +It launches four concurrent instances of the same C function and exercises +stdio, malloc, and FATFS file operations. + +## Initial compatibility boundary + +The first implementation supports: + +- `read`, `write`, `readv`, and `writev` +- `open`, `openat(AT_FDCWD)`, `close`, `lseek`, and `fstat` +- anonymous `mmap`, `munmap`, `mprotect`, and `madvise` +- musl-internal `futex`, `gettid`, and `getrandom` + +Unsupported syscalls return `ENOSYS` and are logged on their first occurrence. +The initial implementation does not support pthread creation, signals, +fork/exec, musl dynamic linking, explicit `exit`/`abort`, or prebuilt ELF +binaries. C `main` must return normally. diff --git a/isol_config/long_chain_n5.json b/isol_config/long_chain_n5.json index 94bb1b2f..4bce84c1 100644 --- a/isol_config/long_chain_n5.json +++ b/isol_config/long_chain_n5.json @@ -23,56 +23,40 @@ ], "apps": [ [ - "array_sum0", - "libarray_sum.so" - ], - [ - "array_sum1", - "libarray_sum.so" - ], - [ - "array_sum2", - "libarray_sum.so" - ], - [ - "array_sum3", - "libarray_sum.so" - ], - [ - "array_sum4", + "array_sum", "libarray_sum.so" ] ], "groups": [ { "list": [ - "array_sum0" + "array_sum" ], - "args": {"array_num": "0"} + "args": {"n": "0"} }, { "list": [ - "array_sum1" + "array_sum" ], - "args": {"array_num": "1"} + "args": {"n": "1"} }, { "list": [ - "array_sum2" + "array_sum" ], - "args": {"array_num": "2"} + "args": {"n": "2"} }, { "list": [ - "array_sum3" + "array_sum" ], - "args": {"array_num": "3"} + "args": {"n": "3"} }, { "list": [ - "array_sum4" + "array_sum" ], - "args": {"array_num": "4"} + "args": {"n": "4"} } ] -} \ No newline at end of file +} diff --git a/isol_config/musl_cpython.json b/isol_config/musl_cpython.json new file mode 100644 index 00000000..4c59312e --- /dev/null +++ b/isol_config/musl_cpython.json @@ -0,0 +1,22 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["cpython", "libmusl_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["cpython"], + "args": { + "pyfile_path": "/wasm_bench/time.py" + } + } + ] +} diff --git a/isol_config/musl_cpython_functionchain_n10.json b/isol_config/musl_cpython_functionchain_n10.json new file mode 100644 index 00000000..384aced4 --- /dev/null +++ b/isol_config/musl_cpython_functionchain_n10.json @@ -0,0 +1,23 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["functionchain", "libmusl_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["functionchain"], + "args": { + "pyfile_path": "/wasm_bench/functionchain.py", + "func_num": "10" + } + } + ] +} diff --git a/isol_config/musl_cpython_functionchain_n5.json b/isol_config/musl_cpython_functionchain_n5.json new file mode 100644 index 00000000..1a4e3d4e --- /dev/null +++ b/isol_config/musl_cpython_functionchain_n5.json @@ -0,0 +1,23 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["functionchain", "libmusl_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["functionchain"], + "args": { + "pyfile_path": "/wasm_bench/functionchain.py", + "func_num": "5" + } + } + ] +} diff --git a/isol_config/musl_cpython_parallel_sort_c1.json b/isol_config/musl_cpython_parallel_sort_c1.json new file mode 100644 index 00000000..a085401f --- /dev/null +++ b/isol_config/musl_cpython_parallel_sort_c1.json @@ -0,0 +1,24 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["parallel_sort", "libmusl_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["parallel_sort"], + "args": { + "pyfile_path": "/wasm_bench/parallel_sort.py", + "sorter_num": "1", + "merger_num": "1" + } + } + ] +} diff --git a/isol_config/musl_cpython_parallel_sort_c3.json b/isol_config/musl_cpython_parallel_sort_c3.json new file mode 100644 index 00000000..ce958258 --- /dev/null +++ b/isol_config/musl_cpython_parallel_sort_c3.json @@ -0,0 +1,26 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["parallel_sort_0", "libmusl_cpython_0.so"], + ["parallel_sort_1", "libmusl_cpython_1.so"], + ["parallel_sort_2", "libmusl_cpython_2.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["parallel_sort_0", "parallel_sort_1", "parallel_sort_2"], + "args": { + "pyfile_path": "/wasm_bench/parallel_sort.py", + "sorter_num": "3", + "merger_num": "3" + } + } + ] +} diff --git a/isol_config/musl_cpython_transfer.json b/isol_config/musl_cpython_transfer.json new file mode 100644 index 00000000..9205e82a --- /dev/null +++ b/isol_config/musl_cpython_transfer.json @@ -0,0 +1,23 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["transfer", "libmusl_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["transfer"], + "args": { + "pyfile_path": "/wasm_bench/transfer.py", + "data_size": "4096" + } + } + ] +} diff --git a/isol_config/musl_cpython_wordcount_c1.json b/isol_config/musl_cpython_wordcount_c1.json new file mode 100644 index 00000000..e8b721a1 --- /dev/null +++ b/isol_config/musl_cpython_wordcount_c1.json @@ -0,0 +1,24 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["wordcount", "libmusl_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["wordcount"], + "args": { + "pyfile_path": "/wasm_bench/wordcount.py", + "mapper_num": "1", + "reducer_num": "1" + } + } + ] +} diff --git a/isol_config/musl_cpython_wordcount_c3.json b/isol_config/musl_cpython_wordcount_c3.json new file mode 100644 index 00000000..9298d5f2 --- /dev/null +++ b/isol_config/musl_cpython_wordcount_c3.json @@ -0,0 +1,26 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["wordcount_0", "libmusl_cpython_0.so"], + ["wordcount_1", "libmusl_cpython_1.so"], + ["wordcount_2", "libmusl_cpython_2.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["wordcount_0", "wordcount_1", "wordcount_2"], + "args": { + "pyfile_path": "/wasm_bench/wordcount.py", + "mapper_num": "3", + "reducer_num": "3" + } + } + ] +} diff --git a/isol_config/musl_hello.json b/isol_config/musl_hello.json new file mode 100644 index 00000000..7583d97a --- /dev/null +++ b/isol_config/musl_hello.json @@ -0,0 +1,41 @@ +{ + "services": [ + [ + "sys", + "libsys.so" + ], + [ + "stdio", + "libstdio.so" + ], + [ + "mm", + "libmm.so" + ], + [ + "fdtab", + "libfdtab.so" + ], + [ + "fatfs", + "libfatfs.so" + ] + ], + "apps": [ + [ + "musl_hello", + "libmusl_hello.so" + ] + ], + "groups": [ + { + "list": [ + "musl_hello", + "musl_hello", + "musl_hello", + "musl_hello" + ], + "args": {} + } + ] +} diff --git a/isol_config/musl_longchain.json b/isol_config/musl_longchain.json new file mode 100644 index 00000000..75cfec85 --- /dev/null +++ b/isol_config/musl_longchain.json @@ -0,0 +1,53 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"] + ], + "apps": [ + ["func", "libmusl_longchain.so"] + ], + "groups": [ + { + "list": ["func"], + "args": {"func_num": "0", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "1", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "2", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "3", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "4", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "5", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "6", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "7", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "8", "chain_len": "10"} + }, + { + "list": ["func"], + "args": {"func_num": "9", "chain_len": "10"} + } + ] +} diff --git a/isol_config/musl_parallel_sort_c3.json b/isol_config/musl_parallel_sort_c3.json new file mode 100644 index 00000000..9c7c6617 --- /dev/null +++ b/isol_config/musl_parallel_sort_c3.json @@ -0,0 +1,45 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["sorter", "libmusl_sorter.so"], + ["splitter", "libmusl_splitter.so"], + ["merger", "libmusl_merger.so"], + ["checker", "libmusl_checker.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["sorter", "sorter", "sorter"], + "args": { + "sorter_num": "3", + "merger_num": "3" + } + }, + { + "list": ["splitter", "splitter", "splitter"], + "args": { + "sorter_num": "3", + "merger_num": "3" + } + }, + { + "list": ["merger", "merger", "merger"], + "args": { + "sorter_num": "3", + "merger_num": "3" + } + }, + { + "list": ["checker"], + "args": { + "merger_num": "3" + } + } + ] +} diff --git a/isol_config/musl_trans_data.json b/isol_config/musl_trans_data.json new file mode 100644 index 00000000..b53fd3d0 --- /dev/null +++ b/isol_config/musl_trans_data.json @@ -0,0 +1,20 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["time", "libtime.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"] + ], + "apps": [ + ["trans_data", "libmusl_trans_data.so"] + ], + "groups": [ + { + "list": ["trans_data"], + "args": { + "data_size": "4096" + } + } + ] +} diff --git a/isol_config/musl_wordcount_c3.json b/isol_config/musl_wordcount_c3.json new file mode 100644 index 00000000..5179d2df --- /dev/null +++ b/isol_config/musl_wordcount_c3.json @@ -0,0 +1,28 @@ +{ + "services": [ + ["sys", "libsys.so"], + ["stdio", "libstdio.so"], + ["mm", "libmm.so"], + ["fdtab", "libfdtab.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["mapper", "libmusl_mapper.so"], + ["reducer", "libmusl_reducer.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["mapper", "mapper", "mapper"], + "args": { + "reducer_num": "3" + } + }, + { + "list": ["reducer", "reducer", "reducer"], + "args": { + "mapper_num": "3" + } + } + ] +} diff --git a/isol_config/wasmtime_cpython_transfer.json b/isol_config/wasmtime_cpython_transfer.json new file mode 100644 index 00000000..a7c30fcd --- /dev/null +++ b/isol_config/wasmtime_cpython_transfer.json @@ -0,0 +1,22 @@ +{ + "services": [ + ["fdtab", "libruxfdtab.so"], + ["time", "libtime.so"], + ["mm", "libmm.so"], + ["stdio", "libstdio.so"], + ["fatfs", "libfatfs.so"] + ], + "apps": [ + ["cpython", "libwasmtime_cpython.so"] + ], + "fs_image": "fs_images/fatfs.img", + "groups": [ + { + "list": ["cpython"], + "args": { + "pyfile_path": "/wasm_bench/transfer.py", + "data_size": "4096" + } + } + ] +} diff --git a/isol_config/wasmtime_trans_data.json b/isol_config/wasmtime_trans_data.json index 33a1af2c..a8f1fbcf 100644 --- a/isol_config/wasmtime_trans_data.json +++ b/isol_config/wasmtime_trans_data.json @@ -26,5 +26,15 @@ "trans_data", "libwasmtime_trans_data.so" ] + ], + "groups": [ + { + "list": [ + "trans_data" + ], + "args": { + "data_size": "4096" + } + } ] - } \ No newline at end of file + } diff --git a/justfile b/justfile index c1af6a44..3086af31 100644 --- a/justfile +++ b/justfile @@ -30,6 +30,8 @@ rust_func func_name: libos lib_name: cargo build {{ release_flag }} {{ if enable_mpk == "1" { "--features mpk" } else { "" } }} \ --manifest-path common_service/{{ lib_name }}/Cargo.toml + cp common_service/{{ lib_name }}/target/{{profile}}/lib{{ lib_name }}.so \ + target/{{profile}}/ pass_args: just rust_func func_a @@ -99,6 +101,122 @@ wasm_func func_name: @-rm target/{{profile}}/lib{{ func_name }}.so just symbol_link {{ func_name }} +musl: + bash scripts/build_musl.sh + +musl_func func_name: musl + cargo build {{ release_flag }} {{ mpk_feature_flag }} \ + --manifest-path user/{{ func_name }}/Cargo.toml + +musl_example: asvisor musl + just enable_mpk={{enable_mpk}} enable_release={{enable_release}} libos sys + just enable_mpk={{enable_mpk}} enable_release={{enable_release}} libos stdio + just enable_mpk={{enable_mpk}} enable_release={{enable_release}} libos mm + just enable_mpk={{enable_mpk}} enable_release={{enable_release}} libos fdtab + just enable_mpk={{enable_mpk}} enable_release={{enable_release}} libos fatfs + cargo build {{ release_flag }} {{ mpk_feature_flag }} \ + --manifest-path user/musl_hello/Cargo.toml + target/{{profile}}/asvisor --files isol_config/musl_hello.json + +musl_wordcount: musl + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_mapper/Cargo.toml + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_reducer/Cargo.toml + +musl_parallel_sort: musl + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_sorter/Cargo.toml + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_splitter/Cargo.toml + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_merger/Cargo.toml + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_checker/Cargo.toml + +musl_long_chain: musl + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_longchain/Cargo.toml + +musl_trans_data: musl + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_trans_data/Cargo.toml + +musl_cpython: musl + bash scripts/build_cpython_musl.sh + cargo build {{ release_flag }} {{ mpk_feature_flag }} --manifest-path user/musl_cpython/Cargo.toml + for id in 0 1 2 3 4; do cp -L target/{{profile}}/libmusl_cpython.so target/{{profile}}/libmusl_cpython_${id}.so; done + +all_musl_c: musl_wordcount musl_parallel_sort musl_long_chain musl_trans_data +all_py_musl: musl_cpython + +musl_c_end_to_end_latency: asvisor all_libos all_c_wasm all_musl_c + #!/usr/bin/env bash + set -euo pipefail + + run_case() { + local workload="$1" + local backend="$2" + local config="$3" + local output + local metric + + if ! output=$(target/{{profile}}/asvisor --files "$config" --metrics total-dur 2>&1); then + echo "$output" >&2 + return 1 + fi + if ! metric=$(printf '%s\n' "$output" | grep -m1 '"total_dur(ms)"'); then + echo "$output" >&2 + echo "missing total_dur metric for $workload ($backend)" >&2 + return 1 + fi + printf '%-16s %-8s %s\n' "$workload" "$backend" "$metric" + } + + printf '%-16s %-8s %s\n' 'workload' 'backend' 'metric' + run_case wordcount wasmtime isol_config/wasmtime_wordcount_c3.json + run_case wordcount musl isol_config/musl_wordcount_c3.json + run_case parallel-sort wasmtime isol_config/wasmtime_parallel_sort_c3.json + run_case parallel-sort musl isol_config/musl_parallel_sort_c3.json + run_case longchain wasmtime isol_config/wasmtime_longchain.json + run_case longchain musl isol_config/musl_longchain.json + +musl_py_end_to_end_latency: asvisor all_libos all_py_musl + #!/usr/bin/env bash + set -euo pipefail + + sudo -E ./scripts/sync_python_workloads.sh + + for case in \ + "wordcount:isol_config/musl_cpython_wordcount_c1.json" \ + "parallel-sort:isol_config/musl_cpython_parallel_sort_c1.json" \ + "function-chain:isol_config/musl_cpython_functionchain_n5.json" + do + workload="${case%%:*}" + config="${case#*:}" + output=$(target/{{profile}}/asvisor --files "$config" --metrics total-dur 2>&1) + metric=$(printf '%s\n' "$output" | grep -m1 '"total_dur(ms)"') + printf '%-16s %s\n' "$workload" "$metric" + done + +py_end_to_end_latency_compare: asvisor all_libos all_py_wasm all_py_musl + #!/usr/bin/env bash + set -euo pipefail + + sudo -E ./scripts/sync_python_workloads.sh + + run_case() { + local workload="$1" + local backend="$2" + local config="$3" + local output + local metric + + output=$(target/{{profile}}/asvisor --files "$config" --metrics total-dur 2>&1) + metric=$(printf '%s\n' "$output" | grep -m1 '"total_dur(ms)"') + printf '%-16s %-8s %s\n' "$workload" "$backend" "$metric" + } + + printf '%-16s %-8s %s\n' 'workload' 'backend' 'metric' + run_case wordcount wasmtime isol_config/wasmtime_cpython_wordcount_c1.json + run_case wordcount musl isol_config/musl_cpython_wordcount_c1.json + run_case parallel-sort wasmtime isol_config/wasmtime_cpython_parallel_sort_c1.json + run_case parallel-sort musl isol_config/musl_cpython_parallel_sort_c1.json + run_case function-chain wasmtime isol_config/wasmtime_cpython_functionchain_n5.json + run_case function-chain musl isol_config/musl_cpython_functionchain_n5.json + c_wordcount: just wasm_func wasmtime_mapper just wasm_func wasmtime_reducer @@ -112,6 +230,9 @@ c_parallel_sort: c_long_chain: just wasm_func wasmtime_longchain +c_trans_data: + bash user/wasmtime_trans_data/build.sh + all_c_wasm: c_wordcount c_parallel_sort c_long_chain python_wordcount: @@ -142,9 +263,115 @@ gen_data: init: rustup override set 'nightly-2023-12-01' rustup target add x86_64-unknown-linux-musl + rustup target add x86_64-unknown-none [ -f fs_images/fatfs.img ] || unzip fs_images/fatfs.zip -d fs_images [ -d image_content ] || mkdir image_content +sync_python_workloads: + ./scripts/sync_python_workloads.sh + +latency_report *args: + #!/usr/bin/env bash + set -euo pipefail + + output="reports/latency_report.md" + results="reports/latency_report_results.json" + config="" + collect="0" + clean="0" + + for arg in {{ args }}; do + case "$arg" in + output=*) + output="${arg#output=}" + ;; + results=*) + results="${arg#results=}" + ;; + config=*) + config="${arg#config=}" + ;; + collect=*) + collect="${arg#collect=}" + ;; + clean=*) + clean="${arg#clean=}" + ;; + *) + echo "unknown latency_report argument: $arg" >&2 + exit 2 + ;; + esac + done + + cmd=(python3 scripts/report_latency.py --output "$output" --results "$results") + if [[ "$collect" == "1" ]]; then + cmd+=(--collect) + fi + if [[ "$clean" == "1" ]]; then + cmd+=(--clean) + fi + if [[ -n "$config" ]]; then + cmd+=(--config "$config") + fi + + "${cmd[@]}" + +rust_latency_report *args: + #!/usr/bin/env bash + set -euo pipefail + + output="reports/rust_latency_report.md" + results="reports/rust_latency_report_results.json" + collect="0" + clean="0" + repeat="1" + skip_prepare="0" + timeout="600" + + for arg in {{ args }}; do + case "$arg" in + output=*) + output="${arg#output=}" + ;; + results=*) + results="${arg#results=}" + ;; + collect=*) + collect="${arg#collect=}" + ;; + clean=*) + clean="${arg#clean=}" + ;; + repeat=*) + repeat="${arg#repeat=}" + ;; + skip_prepare=*) + skip_prepare="${arg#skip_prepare=}" + ;; + timeout=*) + timeout="${arg#timeout=}" + ;; + *) + echo "unknown rust_latency_report argument: $arg" >&2 + exit 2 + ;; + esac + done + + cmd=(python3 scripts/rust_latency_report.py --output "$output" --results "$results" --repeat "$repeat" --timeout "$timeout") + if [[ "$collect" == "1" ]]; then + cmd+=(--collect) + fi + if [[ "$clean" == "1" ]]; then + cmd+=(--clean) + fi + if [[ "$skip_prepare" == "1" ]]; then + cmd+=(--skip-prepare) + fi + + "${cmd[@]}" + asvisor: cargo build {{ release_flag }} @@ -200,19 +427,36 @@ c_end_to_end_latency: asvisor all_libos all_c_wasm target/{{profile}}/asvisor --files isol_config/wasmtime_longchain.json --metrics total-dur 2>&1 | grep 'total_dur' py_end_to_end_latency: asvisor all_libos all_py_wasm - # Python applications. - -sudo mount fs_images/fatfs.img image_content 2>/dev/null - sudo -E ./scripts/gen_data.py 1 '1 * 1024 * 1024' 1 '1 * 1024 * 1024' + #!/usr/bin/env bash + set -euo pipefail + sudo mount fs_images/fatfs.img image_content 2>/dev/null || true + sudo -E ./scripts/sync_python_workloads.sh + sudo -E ./scripts/gen_data.py 1 '1 * 1024 * 1024' 1 '1 * 1024 * 1024' sleep 5 - @echo 'Python word count cost: ' - target/{{profile}}/asvisor --files isol_config/wasmtime_cpython_wordcount_c1.json --metrics total-dur 2>&1 | grep 'total_dur' - - @echo 'Python parallel sorting cost: ' - target/{{profile}}/asvisor --files isol_config/wasmtime_cpython_parallel_sort_c1.json --metrics total-dur 2>&1 | grep 'total_dur' - - @echo 'Python long chain cost: ' - target/{{profile}}/asvisor --files isol_config/wasmtime_cpython_functionchain_n5.json --metrics total-dur 2>&1 | grep 'total_dur' + + run_case() { + local label="$1" + local config="$2" + local output + local metric + + echo "$label" + if ! output=$(target/{{profile}}/asvisor --files "$config" --metrics total-dur 2>&1); then + echo "$output" >&2 + return 1 + fi + if ! metric=$(printf '%s\n' "$output" | grep -m1 '"total_dur(ms)"'); then + echo "$output" >&2 + echo "missing total_dur metric for $config" >&2 + return 1 + fi + echo "$metric" + } + + run_case 'Python word count cost:' isol_config/wasmtime_cpython_wordcount_c1.json + run_case 'Python parallel sorting cost:' isol_config/wasmtime_cpython_parallel_sort_c1.json + run_case 'Python long chain cost:' isol_config/wasmtime_cpython_functionchain_n5.json breakdown: asvisor all_libos -sudo mount fs_images/fatfs.img image_content 2>/dev/null @@ -285,4 +529,4 @@ resource_consume: asvisor all_libos parallel_sort ./resourcetester 80 | grep 'total consume mem:' mv monitor.log as_parallel_sort_resouce_c5_25_80.txt - ./scripts/comp_resource.py \ No newline at end of file + ./scripts/comp_resource.py diff --git a/libasvisor/src/service/elf_service.rs b/libasvisor/src/service/elf_service.rs index 0b7dd307..54cd67cc 100644 --- a/libasvisor/src/service/elf_service.rs +++ b/libasvisor/src/service/elf_service.rs @@ -14,11 +14,11 @@ use anyhow::anyhow; use lazy_static::lazy_static; use libloading::{Library, Symbol}; -use log::info; use as_hostcall::{ types::{DropHandlerFunc, IsolationID, MetricEvent, ServiceName}, IsolationContext, SERVICE_HEAP_SIZE, SERVICE_STACK_SIZE, }; +use log::info; use nix::libc::{PF_KEY, RTLD_DI_LMID}; use thiserror::Error; diff --git a/musl/patches/0001-alloystack-runtime.patch b/musl/patches/0001-alloystack-runtime.patch new file mode 100644 index 00000000..b6d33f2a --- /dev/null +++ b/musl/patches/0001-alloystack-runtime.patch @@ -0,0 +1,188 @@ +diff --git a/arch/x86_64/pthread_arch.h b/arch/x86_64/pthread_arch.h +index 65e880c..01da0ab 100644 +--- a/arch/x86_64/pthread_arch.h ++++ b/arch/x86_64/pthread_arch.h +@@ -1,8 +1,8 @@ ++uintptr_t __alloy_get_tp(void); ++ + static inline uintptr_t __get_tp() + { +- uintptr_t tp; +- __asm__ ("mov %%fs:0,%0" : "=r" (tp) ); +- return tp; ++ return __alloy_get_tp(); + } + + #define MC_PC gregs[REG_RIP] +diff --git a/arch/x86_64/syscall_arch.h b/arch/x86_64/syscall_arch.h +index 92d5c17..51bda17 100644 +--- a/arch/x86_64/syscall_arch.h ++++ b/arch/x86_64/syscall_arch.h +@@ -1,70 +1,41 @@ + #define __SYSCALL_LL_E(x) (x) + #define __SYSCALL_LL_O(x) (x) + ++long alloy_syscall(long, long, long, long, long, long, long); ++ + static __inline long __syscall0(long n) + { +- unsigned long ret; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n) : "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, 0, 0, 0, 0, 0, 0); + } + + static __inline long __syscall1(long n, long a1) + { +- unsigned long ret; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1) : "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, a1, 0, 0, 0, 0, 0); + } + + static __inline long __syscall2(long n, long a1, long a2) + { +- unsigned long ret; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1), "S"(a2) +- : "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, a1, a2, 0, 0, 0, 0); + } + + static __inline long __syscall3(long n, long a1, long a2, long a3) + { +- unsigned long ret; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1), "S"(a2), +- "d"(a3) : "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, a1, a2, a3, 0, 0, 0); + } + + static __inline long __syscall4(long n, long a1, long a2, long a3, long a4) + { +- unsigned long ret; +- register long r10 __asm__("r10") = a4; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1), "S"(a2), +- "d"(a3), "r"(r10): "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, a1, a2, a3, a4, 0, 0); + } + + static __inline long __syscall5(long n, long a1, long a2, long a3, long a4, long a5) + { +- unsigned long ret; +- register long r10 __asm__("r10") = a4; +- register long r8 __asm__("r8") = a5; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1), "S"(a2), +- "d"(a3), "r"(r10), "r"(r8) : "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, a1, a2, a3, a4, a5, 0); + } + + static __inline long __syscall6(long n, long a1, long a2, long a3, long a4, long a5, long a6) + { +- unsigned long ret; +- register long r10 __asm__("r10") = a4; +- register long r8 __asm__("r8") = a5; +- register long r9 __asm__("r9") = a6; +- __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1), "S"(a2), +- "d"(a3), "r"(r10), "r"(r8), "r"(r9) : "rcx", "r11", "memory"); +- return ret; ++ return alloy_syscall(n, a1, a2, a3, a4, a5, a6); + } + +-#define VDSO_USEFUL +-#define VDSO_CGT_SYM "__vdso_clock_gettime" +-#define VDSO_CGT_VER "LINUX_2.6" +-#define VDSO_GETCPU_SYM "__vdso_getcpu" +-#define VDSO_GETCPU_VER "LINUX_2.6" +- + #define IPC_64 0 +diff --git a/Makefile b/Makefile +index 2bc0b8b..937b92f 100644 +--- a/Makefile ++++ b/Makefile +@@ -167,6 +167,13 @@ lib/libc.a: $(AOBJS) + $(AR) rc $@ $(AOBJS) + $(RANLIB) $@ + ++ALLOY_LOBJS = $(filter-out obj/src/thread/__tls_get_addr.lo,$(LOBJS)) ++ ++lib/libc-alloy.a: $(ALLOY_LOBJS) ++ rm -f $@ ++ $(AR) rc $@ $(ALLOY_LOBJS) ++ $(RANLIB) $@ ++ + $(EMPTY_LIBS): + rm -f $@ + $(AR) rc $@ +diff --git a/src/alloy/runtime.c b/src/alloy/runtime.c +new file mode 100644 +index 0000000..5af38ac +--- /dev/null ++++ b/src/alloy/runtime.c +@@ -0,0 +1,64 @@ ++#include ++#include ++#include "atomic.h" ++#include "libc.h" ++#include "pthread_impl.h" ++#include "stdio_impl.h" ++#include "syscall.h" ++ ++extern char **__environ; ++extern hidden FILE __stdin_FILE; ++extern hidden FILE __stdout_FILE; ++extern hidden FILE __stderr_FILE; ++ ++static __thread struct pthread alloy_thread; ++static size_t alloy_auxv[] = { 0 }; ++static char *alloy_environ[] = { 0 }; ++static volatile int alloy_runtime_state; ++ ++uintptr_t __alloy_get_tp(void) ++{ ++ return (uintptr_t)&alloy_thread; ++} ++ ++static void init_process_state(void) ++{ ++ int state = a_cas(&alloy_runtime_state, 0, 1); ++ if (!state) { ++ libc.page_size = 4096; ++ libc.auxv = alloy_auxv; ++ libc.can_do_threads = 1; ++ libc.threaded = 1; ++ libc.need_locks = 1; ++ __environ = alloy_environ; ++ __stdin_FILE.lock = 0; ++ __stdout_FILE.lock = 0; ++ __stderr_FILE.lock = 0; ++ a_store(&alloy_runtime_state, 2); ++ return; ++ } ++ while (alloy_runtime_state != 2) a_spin(); ++} ++ ++void alloy_musl_thread_init(void) ++{ ++ init_process_state(); ++ ++ if (!alloy_thread.self) { ++ long tid = alloy_syscall(SYS_gettid, 0, 0, 0, 0, 0, 0); ++ if (tid <= 0) tid = 1; ++ alloy_thread.self = &alloy_thread; ++ alloy_thread.prev = &alloy_thread; ++ alloy_thread.next = &alloy_thread; ++ alloy_thread.tid = tid; ++ alloy_thread.locale = &libc.global_locale; ++ alloy_thread.canceldisable = PTHREAD_CANCEL_DISABLE; ++ alloy_thread.detach_state = DT_JOINABLE; ++ } ++ alloy_thread.errno_val = 0; ++} ++ ++int alloy_musl_flush(void) ++{ ++ return fflush(0); ++} diff --git a/scripts/build_cpython_musl.sh b/scripts/build_cpython_musl.sh new file mode 100644 index 00000000..592e9628 --- /dev/null +++ b/scripts/build_cpython_musl.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +CACHE="$ROOT/target/cpython-musl" +TOOLCHAIN="$CACHE/toolchain" +SOURCE="$ROOT/third_party/cpython" +BUILD="$CACHE/build-submodule" +CONFIG_SITE_FILE="$CACHE/config.site" + +resolve_build_python() { + local candidate="" + local version="" + + if [[ -n "${BUILD_PYTHON:-}" ]]; then + candidate="$BUILD_PYTHON" + elif command -v python3.11 >/dev/null 2>&1; then + candidate="$(command -v python3.11)" + elif command -v python3 >/dev/null 2>&1; then + candidate="$(command -v python3)" + else + echo "missing build python; set BUILD_PYTHON to a Python 3.11 executable" >&2 + return 1 + fi + + version="$("$candidate" -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')" + if [[ "$version" != "3.11" ]]; then + echo "build python must be Python 3.11, got $version from $candidate" >&2 + echo "set BUILD_PYTHON=/path/to/python3.11 and rerun" >&2 + return 1 + fi + + printf '%s\n' "$candidate" +} + +compiler_supports() { + local option="$1" + + "$TOOLCHAIN/bin/musl-gcc" "$option" \ + -x c -c /dev/null -o /dev/null >/dev/null 2>&1 +} + +if [[ ! -f "$SOURCE/Include/Python.h" ]]; then + echo "missing CPython submodule; run: git submodule update --init third_party/cpython" >&2 + exit 1 +fi + +if [[ ! -x "$TOOLCHAIN/bin/musl-gcc" ]]; then + mkdir -p "$CACHE/musl-build" "$TOOLCHAIN" + ( + cd "$CACHE/musl-build" + "$ROOT/third_party/musl/configure" --prefix="$TOOLCHAIN" + make -j"${JOBS:-$(nproc)}" + make install + ) +fi + +ln -sf "$(command -v ar)" "$TOOLCHAIN/bin/x86_64-linux-musl-ar" +ln -sf "$(command -v readelf)" "$TOOLCHAIN/bin/x86_64-linux-musl-readelf" + +MUSL_CFLAGS="-O2 -fPIC" +if compiler_supports -fno-link-libatomic; then + # GCC 16 may inject the internal placeholder -latomic_asneeded. The + # musl-gcc specs do not expand it, so suppress automatic libatomic linking. + MUSL_CFLAGS+=" -fno-link-libatomic" +fi + +mkdir -p "$CACHE" +cat >"$CONFIG_SITE_FILE" <&2 + exit 1 +fi + +rm -rf "$source_dir" "$build_dir" "$install_dir" +mkdir -p "$source_dir" "$build_dir" "$install_dir/lib" +cp -a "$upstream/." "$source_dir/" + +for patch_file in "$patch_dir"/*.patch; do + patch -d "$source_dir" -p1 < "$patch_file" +done + +cd "$build_dir" +"$source_dir/configure" \ + --prefix="$install_dir" \ + --syslibdir="$install_dir/lib" \ + CC="${CC:-cc}" \ + CFLAGS="-O2 -fPIC -ftls-model=global-dynamic" + +make -s -j"${JOBS:-$(nproc)}" lib/libc-alloy.a install-headers +cp lib/libc-alloy.a "$install_dir/lib/libmusl_alloy.a" + +echo "built $install_dir/lib/libmusl_alloy.a" diff --git a/scripts/build_user.sh b/scripts/build_user.sh index 04fe6007..04b92879 100755 --- a/scripts/build_user.sh +++ b/scripts/build_user.sh @@ -24,10 +24,11 @@ for file in $(find user -name 'Cargo.toml' \ -not -path 'user/should_panic/Cargo.toml' \ -not -path 'user/never_stop/Cargo.toml' \ -not -path 'user/tinywasm*/Cargo.toml' \ + -not -path 'user/musl*/Cargo.toml' \ -not -path 'user/wasmtime*/Cargo.toml'); do echo "Build $file". if ! bash -c "cargo build $feature_arg --manifest-path $file $release_flag"; then echo "Build $file failed!" exit 1 fi -done \ No newline at end of file +done diff --git a/scripts/latency_report_config.json b/scripts/latency_report_config.json new file mode 100644 index 00000000..21ee895b --- /dev/null +++ b/scripts/latency_report_config.json @@ -0,0 +1,39 @@ +{ + "unset_cc": true, + "prepare": [ + "./scripts/sync_python_workloads.sh" + ], + "data_transfer_latency": { + "cells": { + "4*1024": { + "wasm-py": null, + "musl-c": null, + "musl-py": null + }, + "64*1024": { + "wasm-c": null, + "wasm-py": null, + "musl-c": null, + "musl-py": null + }, + "1024*1024": { + "wasm-c": null, + "wasm-py": null, + "musl-c": null, + "musl-py": null + }, + "16*1024*1024": { + "wasm-c": null, + "wasm-py": null, + "musl-c": null, + "musl-py": null + }, + "256*1024*1024": { + "wasm-c": null, + "wasm-py": null, + "musl-c": null, + "musl-py": null + } + } + } +} diff --git a/scripts/report_latency.py b/scripts/report_latency.py new file mode 100644 index 00000000..88bede86 --- /dev/null +++ b/scripts/report_latency.py @@ -0,0 +1,643 @@ +#!/usr/bin/env python3 + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import time +from pathlib import Path + + +BACKENDS = ["rust", "wasm-c", "wasm-py", "musl-c", "musl-py"] +DATA_SIZES = ["4*1024", "64*1024", "1024*1024", "16*1024*1024", "256*1024*1024"] +WORKLOADS = ["longchain", "ps", "wc"] + +TOTAL_DUR_RE = re.compile(r'"total_dur\(ms\)":\s*([0-9]+(?:\.[0-9]+)?)') +TRANS_DATA_RE = re.compile(r"trans data time:\s*([0-9]+(?:\.[0-9]+)?)(µs|ms)") +TRANSFER_COST_NS_RE = re.compile(r"transfer cost:\s*([0-9]+(?:\.[0-9]+)?)\s*ns") +RUST_COST_NS_RE = re.compile(r"data size:\s*[0-9]+\s*bytes,\s*cost\s*([0-9]+(?:\.[0-9]+)?)\s*ns") +CURRENTLY_CLEAN_AFTER_CELL = False +PROGRESS_INTERVAL_SECONDS = 5 + + +def repo_root() -> Path: + return Path(__file__).resolve().parent.parent + + +def default_config() -> dict: + profile = "release" + asvisor = f"target/{profile}/asvisor" + + data_transfer_cells = { + size: {backend: None for backend in BACKENDS} + for size in DATA_SIZES + } + + for size in DATA_SIZES: + data_transfer_cells[size]["rust"] = { + "command": f"printf '%s\\n' '{size}' > user/data_size.config && just pass_args >/dev/null && {asvisor} --files isol_config/pass_complex_args.json --metrics all", + "parser": "rust_cost_ns_ms", + "cleanup": [ + "cargo clean --manifest-path user/func_a/Cargo.toml", + "cargo clean --manifest-path user/func_b/Cargo.toml", + ], + } + data_transfer_cells[size]["wasm-c"] = { + "config": "isol_config/wasmtime_trans_data.json", + "config_args": {"data_size": "{data_size_bytes}"}, + "command": f"just wasm_func wasmtime_trans_data >/dev/null && {asvisor} --files {{config_path}} --metrics all", + "parser": "transfer_cost_ns_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_trans_data/Cargo.toml", + ], + } + data_transfer_cells[size]["wasm-py"] = { + "config": "isol_config/wasmtime_cpython_transfer.json", + "config_args": {"data_size": "{data_size_bytes}"}, + "command": f"just wasm_func wasmtime_cpython >/dev/null && {asvisor} --files {{config_path}} --metrics all", + "parser": "transfer_cost_ns_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_cpython/Cargo.toml", + ], + } + data_transfer_cells[size]["musl-c"] = { + "config": "isol_config/musl_trans_data.json", + "config_args": {"data_size": "{data_size_bytes}"}, + "command": f"just musl_func musl_trans_data >/dev/null && {asvisor} --files {{config_path}} --metrics all", + "parser": "transfer_cost_ns_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_trans_data/Cargo.toml", + ], + } + data_transfer_cells[size]["musl-py"] = { + "config": "isol_config/musl_cpython_transfer.json", + "config_args": {"data_size": "{data_size_bytes}"}, + "command": f"env -u CC cargo build --release --manifest-path user/musl_cpython/Cargo.toml >/dev/null && {asvisor} --files {{config_path}} --metrics all", + "parser": "transfer_cost_ns_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_cpython/Cargo.toml", + ], + } + + end_to_end_cells = { + "longchain": { + "rust": { + "command": f"just long_chain >/dev/null && {asvisor} --files isol_config/long_chain_n10.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/array_sum/Cargo.toml", + ], + }, + "wasm-c": { + "command": f"just c_long_chain >/dev/null && {asvisor} --files isol_config/wasmtime_longchain.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_longchain/Cargo.toml", + ], + }, + "wasm-py": { + "command": f"just python_long_chain >/dev/null && {asvisor} --files isol_config/wasmtime_cpython_functionchain_n10.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_cpython_func/Cargo.toml", + ], + }, + "musl-c": { + "command": f"just musl_long_chain >/dev/null && {asvisor} --files isol_config/musl_longchain.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_longchain/Cargo.toml", + ], + }, + "musl-py": { + "command": f"just musl_cpython >/dev/null && {asvisor} --files isol_config/musl_cpython_functionchain_n10.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_cpython/Cargo.toml", + "rm -f target/release/libmusl_cpython.so target/release/libmusl_cpython_0.so target/release/libmusl_cpython_1.so target/release/libmusl_cpython_2.so target/release/libmusl_cpython_3.so target/release/libmusl_cpython_4.so", + ], + }, + }, + "ps": { + "rust": { + "command": f"just parallel_sort >/dev/null && {asvisor} --files isol_config/parallel_sort_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/file_reader/Cargo.toml", + "cargo clean --manifest-path user/sorter/Cargo.toml", + "cargo clean --manifest-path user/splitter/Cargo.toml", + "cargo clean --manifest-path user/merger/Cargo.toml", + ], + }, + "wasm-c": { + "command": f"just c_parallel_sort >/dev/null && {asvisor} --files isol_config/wasmtime_parallel_sort_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_sorter/Cargo.toml", + "cargo clean --manifest-path user/wasmtime_spliter/Cargo.toml", + "cargo clean --manifest-path user/wasmtime_merger/Cargo.toml", + "cargo clean --manifest-path user/wasmtime_checker/Cargo.toml", + ], + }, + "wasm-py": { + "command": f"just python_parallel_sort >/dev/null && {asvisor} --files isol_config/wasmtime_cpython_parallel_sort_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_cpython_parallel_sort/Cargo.toml", + ], + }, + "musl-c": { + "command": f"just musl_parallel_sort >/dev/null && {asvisor} --files isol_config/musl_parallel_sort_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_sorter/Cargo.toml", + "cargo clean --manifest-path user/musl_splitter/Cargo.toml", + "cargo clean --manifest-path user/musl_merger/Cargo.toml", + "cargo clean --manifest-path user/musl_checker/Cargo.toml", + ], + }, + "musl-py": { + "command": f"just musl_cpython >/dev/null && {asvisor} --files isol_config/musl_cpython_parallel_sort_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_cpython/Cargo.toml", + "rm -f target/release/libmusl_cpython.so target/release/libmusl_cpython_0.so target/release/libmusl_cpython_1.so target/release/libmusl_cpython_2.so target/release/libmusl_cpython_3.so target/release/libmusl_cpython_4.so", + ], + }, + }, + "wc": { + "rust": { + "command": f"just map_reduce >/dev/null && {asvisor} --files isol_config/map_reduce_large_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/mapper/Cargo.toml", + "cargo clean --manifest-path user/reducer/Cargo.toml", + ], + }, + "wasm-c": { + "command": f"just c_wordcount >/dev/null && {asvisor} --files isol_config/wasmtime_wordcount_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_mapper/Cargo.toml", + "cargo clean --manifest-path user/wasmtime_reducer/Cargo.toml", + ], + }, + "wasm-py": { + "command": f"just python_wordcount >/dev/null && {asvisor} --files isol_config/wasmtime_cpython_wordcount_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/wasmtime_cpython_wordcount/Cargo.toml", + ], + }, + "musl-c": { + "command": f"just musl_wordcount >/dev/null && {asvisor} --files isol_config/musl_wordcount_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_mapper/Cargo.toml", + "cargo clean --manifest-path user/musl_reducer/Cargo.toml", + ], + }, + "musl-py": { + "command": f"just musl_cpython >/dev/null && {asvisor} --files isol_config/musl_cpython_wordcount_c3.json --metrics total-dur", + "parser": "total_dur_ms", + "cleanup": [ + "cargo clean --manifest-path user/musl_cpython/Cargo.toml", + "rm -f target/release/libmusl_cpython.so target/release/libmusl_cpython_0.so target/release/libmusl_cpython_1.so target/release/libmusl_cpython_2.so target/release/libmusl_cpython_3.so target/release/libmusl_cpython_4.so", + ], + }, + }, + } + + return { + "profile": profile, + "unset_cc": True, + "prepare": [ + "./scripts/sync_python_workloads.sh", + ], + "data_transfer_latency": { + "columns": BACKENDS, + "rows": DATA_SIZES, + "cells": data_transfer_cells, + }, + "end_to_end_latency": { + "columns": BACKENDS, + "rows": WORKLOADS, + "cells": end_to_end_cells, + }, + } + + +def empty_results(config: dict) -> dict: + data_transfer = { + row: {column: "N/A" for column in config["data_transfer_latency"]["columns"]} + for row in config["data_transfer_latency"]["rows"] + } + end_to_end = { + row: {column: "N/A" for column in config["end_to_end_latency"]["columns"]} + for row in config["end_to_end_latency"]["rows"] + } + return { + "data_transfer_latency": data_transfer, + "end_to_end_latency": end_to_end, + } + + +def merge_dict(base: dict, override: dict) -> dict: + result = dict(base) + for key, value in override.items(): + if isinstance(value, dict) and isinstance(result.get(key), dict): + result[key] = merge_dict(result[key], value) + else: + result[key] = value + return result + + +def load_config(path: str | None) -> dict: + config = default_config() + if not path: + return config + with open(path, "r", encoding="utf-8") as fh: + override = json.load(fh) + return merge_dict(config, override) + + +def run_command( + command: str, + cwd: Path, + unset_cc: bool, + interactive: bool = False, + progress_label: str | None = None, +) -> str: + env = os.environ.copy() + if unset_cc: + env.pop("CC", None) + if interactive: + completed = subprocess.run( + ["bash", "-lc", command], + cwd=cwd, + env=env, + text=True, + ) + output = "" + else: + process = subprocess.Popen( + ["bash", "-lc", command], + cwd=cwd, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + ) + started_at = time.monotonic() + while True: + try: + output, _ = process.communicate(timeout=PROGRESS_INTERVAL_SECONDS) + break + except subprocess.TimeoutExpired: + if progress_label: + elapsed = int(time.monotonic() - started_at) + print( + f"[progress] {progress_label}: running for {elapsed}s", + file=sys.stderr, + flush=True, + ) + completed = process + if completed.returncode != 0: + raise RuntimeError(output.strip() or f"command failed: {command}") + return output + + +def run_prepare_command(command: str, cwd: Path, unset_cc: bool) -> None: + try: + run_command(command, cwd, unset_cc) + return + except RuntimeError as exc: + message = str(exc) + if "rerun this script with sudo" not in message: + raise + + sudo_command = f"sudo -E {command}" + run_command(sudo_command, cwd, unset_cc, interactive=True) + + +def run_cleanup_commands( + commands: list[str], + cwd: Path, + unset_cc: bool, + context: dict[str, str], +) -> None: + for command in commands: + run_command(command.format(**context), cwd, unset_cc) + + +def build_temp_config( + cwd: Path, + config_path: str, + config_args: dict[str, str], + context: dict[str, str], +) -> str: + source_path = cwd / config_path + with open(source_path, "r", encoding="utf-8") as fh: + config = json.load(fh) + for group in config.get("groups", []): + args = group.get("args", {}) + for key, value in config_args.items(): + args[key] = value.format(**context) + with tempfile.NamedTemporaryFile( + "w", + suffix=".json", + prefix="latency-report-", + delete=False, + dir="/tmp", + encoding="utf-8", + ) as fh: + json.dump(config, fh) + fh.flush() + return fh.name + + +def eval_data_size(expr: str) -> int: + parts = [part.strip() for part in expr.split("*")] + value = 1 + for part in parts: + if not part: + raise ValueError(f"invalid data size expression: {expr}") + value *= int(part) + return value + + +def parse_metric(output: str, parser_name: str) -> float: + if parser_name == "total_dur_ms": + match = TOTAL_DUR_RE.search(output) + if not match: + raise ValueError("missing total_dur(ms)") + return float(match.group(1)) + + if parser_name == "trans_data_ms": + match = TRANS_DATA_RE.search(output) + if not match: + raise ValueError("missing trans data time") + value = float(match.group(1)) + unit = match.group(2) + return value / 1000.0 if unit == "µs" else value + + if parser_name == "transfer_cost_ns_ms": + match = TRANSFER_COST_NS_RE.search(output) + if not match: + raise ValueError("missing transfer cost") + return float(match.group(1)) / 1_000_000.0 + + if parser_name == "rust_cost_ns_ms": + match = RUST_COST_NS_RE.search(output) + if not match: + raise ValueError("missing rust transfer cost") + return float(match.group(1)) / 1_000_000.0 + + raise ValueError(f"unsupported parser: {parser_name}") + + +def collect_table( + section_name: str, + section: dict, + cwd: Path, + unset_cc: bool, +) -> dict: + rows = section["rows"] + columns = section["columns"] + cells = section["cells"] + results: dict[str, dict[str, str]] = {} + total_cells = sum( + cells.get(row, {}).get(column) is not None + for row in rows + for column in columns + ) + cell_index = 0 + + for row in rows: + results[row] = {} + for column in columns: + cell = cells.get(row, {}).get(column) + if cell is None: + results[row][column] = "N/A" + continue + cell_index += 1 + progress_label = ( + f"{section_name} [{cell_index}/{total_cells}] " + f"row={row}, backend={column}" + ) + if "value_ms" in cell: + results[row][column] = f"{float(cell['value_ms']):.3f}" + print( + f"[done] {progress_label}: {results[row][column]} ms (fixed)", + file=sys.stderr, + flush=True, + ) + continue + temp_config_path = None + context = { + "data_size": row, + "data_size_bytes": row, + "workload": row, + "backend": column, + } + started_at = time.monotonic() + print(f"[start] {progress_label}", file=sys.stderr, flush=True) + try: + data_size_bytes = str(eval_data_size(row)) if "*" in row else row + context["data_size_bytes"] = data_size_bytes + command = cell["command"] + if "config" in cell: + temp_config_path = build_temp_config( + cwd, + cell["config"], + cell.get("config_args", {}), + context, + ) + context["config_path"] = temp_config_path + output = run_command( + command.format(**context), + cwd, + unset_cc, + progress_label=progress_label, + ) + value = parse_metric(output, cell["parser"]) + results[row][column] = f"{value:.3f}" + elapsed = time.monotonic() - started_at + print( + f"[done] {progress_label}: {results[row][column]} ms " + f"in {elapsed:.1f}s", + file=sys.stderr, + flush=True, + ) + except Exception as exc: + error = str(exc).splitlines()[-1] + results[row][column] = f"N/A ({error})" + elapsed = time.monotonic() - started_at + print( + f"[failed] {progress_label}: {error} after {elapsed:.1f}s", + file=sys.stderr, + flush=True, + ) + finally: + if CURRENTLY_CLEAN_AFTER_CELL and cell is not None: + cleanup_commands = cell.get("cleanup", []) + if cleanup_commands: + try: + run_cleanup_commands(cleanup_commands, cwd, unset_cc, context) + except Exception as exc: + results[row][column] = ( + f"{results[row][column]} | cleanup failed: " + f"{str(exc).splitlines()[-1]}" + ) + if temp_config_path: + try: + os.unlink(temp_config_path) + except OSError: + pass + return results + + +def render_table(title: str, first_col: str, rows: list[str], columns: list[str], values: dict) -> str: + lines = [f"## {title}", ""] + header = [first_col, *columns] + align = ["---"] * len(header) + lines.append("| " + " | ".join(header) + " |") + lines.append("| " + " | ".join(align) + " |") + for row in rows: + row_values = [row] + [values[row][column] for column in columns] + lines.append("| " + " | ".join(row_values) + " |") + lines.append("") + return "\n".join(lines) + + +def write_output(path: Path, content: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + + +def load_results(path: Path, config: dict) -> dict: + if not path.exists(): + raise FileNotFoundError( + f"missing results file: {path}; run with --collect first" + ) + with open(path, "r", encoding="utf-8") as fh: + loaded = json.load(fh) + results = empty_results(config) + for section in ("data_transfer_latency", "end_to_end_latency"): + if section not in loaded: + continue + for row, columns in loaded[section].items(): + if row not in results[section]: + continue + for column, value in columns.items(): + if column in results[section][row]: + results[section][row][column] = value + return results + + +def write_results(path: Path, results: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "w", encoding="utf-8") as fh: + json.dump(results, fh, indent=2) + fh.write("\n") + + +def main() -> int: + parser = argparse.ArgumentParser(description="Generate latency report in Markdown.") + parser.add_argument( + "--output", + default="reports/latency_report.md", + help="output markdown path", + ) + parser.add_argument( + "--results", + default="reports/latency_report_results.json", + help="raw results json path", + ) + parser.add_argument( + "--config", + help="optional JSON config overriding commands or fixed values", + ) + parser.add_argument( + "--collect", + action="store_true", + help="run experiments and refresh the raw results json", + ) + parser.add_argument( + "--clean", + action="store_true", + help="run configured cleanup commands after each collected cell", + ) + parser.add_argument( + "--dump-config", + action="store_true", + help="print the default config JSON and exit", + ) + args = parser.parse_args() + + if args.dump_config: + json.dump(default_config(), sys.stdout, indent=2) + sys.stdout.write("\n") + return 0 + + cwd = repo_root() + config = load_config(args.config) + global CURRENTLY_CLEAN_AFTER_CELL + CURRENTLY_CLEAN_AFTER_CELL = args.clean + output_path = Path(args.output) + if not output_path.is_absolute(): + output_path = cwd / output_path + results_path = Path(args.results) + if not results_path.is_absolute(): + results_path = cwd / results_path + + if args.collect: + unset_cc = bool(config.get("unset_cc", True)) + for command in config.get("prepare", []): + run_prepare_command(command, cwd, unset_cc) + results = { + "data_transfer_latency": collect_table( + "data_transfer_latency", + config["data_transfer_latency"], + cwd, + unset_cc, + ), + "end_to_end_latency": collect_table( + "end_to_end_latency", + config["end_to_end_latency"], + cwd, + unset_cc, + ), + } + write_results(results_path, results) + else: + results = load_results(results_path, config) + + content = "\n\n".join( + [ + render_table( + "data_transfer_latency", + "data_size", + config["data_transfer_latency"]["rows"], + config["data_transfer_latency"]["columns"], + results["data_transfer_latency"], + ).rstrip(), + render_table( + "end_to_end_latency", + "workload", + config["end_to_end_latency"]["rows"], + config["end_to_end_latency"]["columns"], + results["end_to_end_latency"], + ).rstrip(), + ] + ) + "\n" + write_output(output_path, content) + print(output_path) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/rust_latency_report.py b/scripts/rust_latency_report.py new file mode 100644 index 00000000..0e7e7aca --- /dev/null +++ b/scripts/rust_latency_report.py @@ -0,0 +1,774 @@ +#!/usr/bin/env python3 + +import argparse +import json +import os +import re +import signal +import subprocess +import sys +import tempfile +import time +from pathlib import Path + + +RESULTS_SCHEMA_VERSION = 3 +TOTAL_DUR_RE = re.compile(r'"total_dur\(ms\)":\s*([0-9]+(?:\.[0-9]+)?)') +PROGRESS_INTERVAL_SECONDS = 5 +DEFAULT_TEST_TIMEOUT_SECONDS = 10 * 60 +BACKENDS = ["rust", "wasm-c", "wasm-py", "musl-c", "musl-py"] + +WORD_COUNT_ROWS = [ + ("10MB", "10 * 1024 * 1024"), + ("100MB", "100 * 1024 * 1024"), + ("300MB", "300 * 1024 * 1024"), +] +PARALLEL_SORT_ROWS = [ + ("1MB", "1 * 1024 * 1024"), + ("25MB", "25 * 1024 * 1024"), + ("50MB", "50 * 1024 * 1024"), +] +LONG_CHAIN_ROWS = [ + ("1MB", "1 * 1024 * 1024"), + ("64MB", "64 * 1024 * 1024"), + ("256MB", "256 * 1024 * 1024"), +] + +PARALLEL_COLUMNS = ["C1", "C3", "C5"] +LONG_CHAIN_COLUMNS = ["n5", "n10", "n15"] + +WORD_COUNT_CONFIGS = { + "C1": "isol_config/map_reduce_large_c1.json", + "C3": "isol_config/map_reduce_large_c3.json", + "C5": "isol_config/map_reduce_large_c5.json", +} +PARALLEL_SORT_CONFIGS = { + "C1": "isol_config/parallel_sort_c1.json", + "C3": "isol_config/parallel_sort_c3.json", + "C5": "isol_config/parallel_sort_c5.json", +} +LONG_CHAIN_CONFIGS = { + "n5": "isol_config/long_chain_n5.json", + "n10": "isol_config/long_chain_n10.json", + "n15": "isol_config/long_chain_n15.json", +} + +CONFIG_TEMPLATES = { + "Word Count": { + "wasm-c": "isol_config/wasmtime_wordcount_c3.json", + "wasm-py": "isol_config/wasmtime_cpython_wordcount_c3.json", + "musl-c": "isol_config/musl_wordcount_c3.json", + "musl-py": "isol_config/musl_cpython_wordcount_c3.json", + }, + "Parallel Sort": { + "wasm-c": "isol_config/wasmtime_parallel_sort_c3.json", + "wasm-py": "isol_config/wasmtime_cpython_parallel_sort_c3.json", + "musl-c": "isol_config/musl_parallel_sort_c3.json", + "musl-py": "isol_config/musl_cpython_parallel_sort_c3.json", + }, + "Long Chain": { + "wasm-c": "isol_config/wasmtime_longchain.json", + "wasm-py": "isol_config/wasmtime_cpython_functionchain_n10.json", + "musl-c": "isol_config/musl_longchain.json", + "musl-py": "isol_config/musl_cpython_functionchain_n10.json", + }, +} + + +def repo_root() -> Path: + return Path(__file__).resolve().parent.parent + + +def eval_size_expr(expr: str) -> int: + parts = [part.strip() for part in expr.split("*")] + value = 1 + for part in parts: + if not part: + raise ValueError(f"invalid data size expression: {expr}") + value *= int(part) + return value + + +def empty_results() -> dict: + return { + "Word Count": { + backend: { + row: {column: "N/A" for column in PARALLEL_COLUMNS} + for row, _ in WORD_COUNT_ROWS + } + for backend in BACKENDS + }, + "Parallel Sort": { + backend: { + row: {column: "N/A" for column in PARALLEL_COLUMNS} + for row, _ in PARALLEL_SORT_ROWS + } + for backend in BACKENDS + }, + "Long Chain": { + backend: { + row: {column: "N/A" for column in LONG_CHAIN_COLUMNS} + for row, _ in LONG_CHAIN_ROWS + } + for backend in BACKENDS + }, + } + + +def run_command( + command: str, + cwd: Path, + unset_cc: bool, + *, + progress_label: str | None = None, + interactive: bool = False, + timeout_seconds: int | None = None, +) -> str: + env = os.environ.copy() + if unset_cc: + env.pop("CC", None) + + if interactive: + completed = subprocess.run(["bash", "-lc", command], cwd=cwd, env=env, text=True) + if completed.returncode != 0: + raise RuntimeError(f"command failed: {command}") + return "" + + process = subprocess.Popen( + ["bash", "-lc", command], + cwd=cwd, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + start_new_session=True, + ) + started_at = time.monotonic() + while True: + elapsed = time.monotonic() - started_at + if timeout_seconds is not None and elapsed >= timeout_seconds: + os.killpg(process.pid, signal.SIGTERM) + try: + output, _ = process.communicate(timeout=5) + except subprocess.TimeoutExpired: + os.killpg(process.pid, signal.SIGKILL) + output, _ = process.communicate() + detail = output.strip() + if detail: + detail += "\n" + raise RuntimeError( + f"{detail}command timed out after {timeout_seconds}s" + ) + wait_seconds = PROGRESS_INTERVAL_SECONDS + if timeout_seconds is not None: + wait_seconds = min(wait_seconds, timeout_seconds - elapsed) + try: + output, _ = process.communicate(timeout=wait_seconds) + break + except subprocess.TimeoutExpired: + if progress_label: + elapsed = int(time.monotonic() - started_at) + print( + f"[progress] {progress_label}: running for {elapsed}s", + file=sys.stderr, + flush=True, + ) + + if process.returncode != 0: + raise RuntimeError(output.strip() or f"command failed: {command}") + return output + + +def run_prepare(cwd: Path, unset_cc: bool) -> None: + commands = [ + "sudo mount fs_images/fatfs.img image_content 2>/dev/null || true", + "just asvisor >/dev/null", + "just all_libos >/dev/null", + "just map_reduce >/dev/null", + "just parallel_sort >/dev/null", + "just all_c_wasm >/dev/null", + "just all_py_wasm >/dev/null", + "just all_musl_c >/dev/null", + "just all_py_musl >/dev/null", + "sudo -E ./scripts/sync_python_workloads.sh", + ] + for command in commands: + print(f"[prepare] {command}", file=sys.stderr, flush=True) + run_command( + command, + cwd, + unset_cc, + progress_label=f"prepare: {command}", + interactive=command.startswith("sudo "), + ) + ensure_image_mounted(cwd, unset_cc) + run_command("sync", cwd, unset_cc, progress_label="sync filesystem image") + unmount_image(cwd, unset_cc) + + +def ensure_image_mounted(cwd: Path, unset_cc: bool) -> None: + try: + run_command("mountpoint -q image_content", cwd, unset_cc) + except RuntimeError as exc: + raise RuntimeError( + "image_content is not mounted; run `sudo mount fs_images/fatfs.img image_content` " + "or run without skip_prepare=1" + ) from exc + + +def mount_image(cwd: Path, unset_cc: bool) -> None: + try: + run_command("mountpoint -q image_content", cwd, unset_cc) + return + except RuntimeError: + pass + run_command( + "sudo mount fs_images/fatfs.img image_content", + cwd, + unset_cc, + progress_label="mount filesystem image", + interactive=True, + ) + ensure_image_mounted(cwd, unset_cc) + + +def unmount_image(cwd: Path, unset_cc: bool) -> None: + try: + run_command("mountpoint -q image_content", cwd, unset_cc) + except RuntimeError: + return + run_command( + "sudo umount image_content", + cwd, + unset_cc, + progress_label="unmount filesystem image", + interactive=True, + ) + + +def parse_total_dur(output: str) -> float: + match = TOTAL_DUR_RE.search(output) + if not match: + raise ValueError("missing total_dur(ms)") + return float(match.group(1)) + + +def gen_data_command(workload: str, workers: int, size_expr: str) -> str: + if workload == "Word Count": + return f"sudo -E ./scripts/gen_data.py {workers} '{size_expr}' 0 0" + if workload == "Parallel Sort": + return f"sudo -E ./scripts/gen_data.py 0 0 {workers} '{size_expr}'" + raise ValueError(f"unsupported generated-data workload: {workload}") + + +def clean_workload_data(cwd: Path, unset_cc: bool, workload: str) -> None: + if workload == "Word Count": + pattern = "image_content/fake_data_*.txt" + elif workload == "Parallel Sort": + pattern = "image_content/sort_data_*.txt" + else: + return + command = f"sudo -E bash -lc 'rm -f {pattern}'" + run_command(command, cwd, unset_cc, progress_label=f"clean {workload} input data", interactive=True) + + +def clean_generated_data(cwd: Path, unset_cc: bool) -> None: + mount_image(cwd, unset_cc) + command = ( + "sudo -E bash -lc " + "'rm -f image_content/*.imd " + "image_content/fake_data_*.txt " + "image_content/sort_data_*.txt'" + ) + try: + run_command( + command, + cwd, + unset_cc, + progress_label="clean generated data", + interactive=True, + ) + run_command("sync", cwd, unset_cc, progress_label="sync cleaned data") + finally: + unmount_image(cwd, unset_cc) + + +def verify_generated_data(cwd: Path, workload: str, workers: int, size_expr: str) -> None: + if workload == "Word Count": + prefix = "fake_data_" + elif workload == "Parallel Sort": + prefix = "sort_data_" + else: + return + + files = sorted((cwd / "image_content").glob(f"{prefix}*.txt")) + if len(files) != workers: + names = ", ".join(path.name for path in files[:10]) + raise RuntimeError( + f"{workload} generated {len(files)} files, expected {workers}; files=[{names}]" + ) + + expected_total = eval_size_expr(size_expr) + sizes = [path.stat().st_size for path in files] + actual_total = sum(sizes) + if any(size <= 0 for size in sizes): + raise RuntimeError(f"{workload} generated empty input file") + if actual_total < expected_total: + raise RuntimeError( + f"{workload} generated {actual_total} bytes, expected at least {expected_total}" + ) + + +def rust_config(workload: str, column: str) -> str: + if workload == "Word Count": + return WORD_COUNT_CONFIGS[column] + if workload == "Parallel Sort": + return PARALLEL_SORT_CONFIGS[column] + if workload == "Long Chain": + return LONG_CHAIN_CONFIGS[column] + raise ValueError(f"unsupported workload: {workload}") + + +def generated_config( + cwd: Path, + temp_dir: Path, + workload: str, + backend: str, + column: str, + size_expr: str, +) -> str: + with open(cwd / CONFIG_TEMPLATES[workload][backend], "r", encoding="utf-8") as fh: + config = json.load(fh) + + size_bytes = str(eval_size_expr(size_expr)) + amount = int(column[1:]) + + if workload == "Word Count": + group = config["groups"][0] + if backend == "musl-py": + app_names = [f"wordcount_{idx}" for idx in range(amount)] + config["apps"] = [ + [name, f"libmusl_cpython_{idx}.so"] + for idx, name in enumerate(app_names) + ] + group["list"] = app_names + else: + group["list"] = [group["list"][0]] * amount + if len(config["groups"]) > 1: + reducer_group = config["groups"][1] + reducer_group["list"] = [reducer_group["list"][0]] * amount + for item in config["groups"]: + item["args"]["mapper_num"] = str(amount) + item["args"]["reducer_num"] = str(amount) + elif workload == "Parallel Sort": + group = config["groups"][0] + if backend == "musl-py": + app_names = [f"parallel_sort_{idx}" for idx in range(amount)] + config["apps"] = [ + [name, f"libmusl_cpython_{idx}.so"] + for idx, name in enumerate(app_names) + ] + group["list"] = app_names + else: + for item in config["groups"]: + first = item["list"][0] + if "checker" not in first: + item["list"] = [first] * amount + for item in config["groups"]: + item["args"]["sorter_num"] = str(amount) + item["args"]["merger_num"] = str(amount) + elif workload == "Long Chain": + if backend.endswith("-py"): + config["groups"][0]["args"]["func_num"] = str(amount) + config["groups"][0]["args"]["data_size"] = size_bytes + else: + app_name = config["groups"][0]["list"][0] + config["groups"] = [ + { + "list": [app_name], + "args": { + "func_num": str(stage), + "chain_len": str(amount), + "data_size": size_bytes, + }, + } + for stage in range(amount) + ] + else: + raise ValueError(f"unsupported workload: {workload}") + + path = temp_dir / ( + f"{workload.lower().replace(' ', '_')}-{backend}-{column}-{size_bytes}.json" + ) + with open(path, "w", encoding="utf-8") as fh: + json.dump(config, fh, indent=2) + fh.write("\n") + return str(path) + + +def collect_cell( + cwd: Path, + unset_cc: bool, + workload: str, + backend: str, + row_label: str, + column: str, + config_path: str, + repeat: int, + timeout_seconds: int, +) -> str: + progress_label = ( + f"{workload} runtime={backend}, row={row_label}, column={column}" + ) + print(f"[start] {progress_label}", file=sys.stderr, flush=True) + started_at = time.monotonic() + + try: + # The LibOS opens fs_images/fatfs.img directly. Never let it race with + # the kernel FAT driver operating on the mounted image. + unmount_image(cwd, unset_cc) + values = [] + for run_id in range(1, repeat + 1): + run_label = f"{progress_label}, run={run_id}/{repeat}" + output = run_command( + f"target/release/asvisor --files {config_path} --metrics total-dur", + cwd, + unset_cc, + progress_label=run_label, + timeout_seconds=timeout_seconds, + ) + values.append(parse_total_dur(output)) + + value = sum(values) / len(values) + elapsed = time.monotonic() - started_at + print( + f"[done] {progress_label}: {value:.3f} ms in {elapsed:.1f}s", + file=sys.stderr, + flush=True, + ) + return f"{value:.3f}" + except Exception as exc: + error = str(exc).splitlines()[-1] + elapsed = time.monotonic() - started_at + print( + f"[failed] {progress_label}: {error} after {elapsed:.1f}s", + file=sys.stderr, + flush=True, + ) + return f"N/A ({error})" + + +def prepare_generated_data( + cwd: Path, + unset_cc: bool, + workload: str, + row_label: str, + size_expr: str, + column: str, +) -> None: + workers = int(column[1:]) + label = f"{workload} row={row_label}, column={column}" + mount_image(cwd, unset_cc) + try: + clean_workload_data(cwd, unset_cc, workload) + run_command( + gen_data_command(workload, workers, size_expr), + cwd, + unset_cc, + progress_label=f"generate data: {label}", + interactive=True, + ) + run_command("sync", cwd, unset_cc, progress_label=f"sync data: {label}") + verify_generated_data(cwd, workload, workers, size_expr) + finally: + unmount_image(cwd, unset_cc) + + +def build_rust_long_chain( + cwd: Path, + unset_cc: bool, + row_label: str, + size_expr: str, +) -> None: + config_file = cwd / "user" / "function_chain_data_size.config" + original = config_file.read_text(encoding="utf-8") + config_file.write_text(f"{size_expr}\n", encoding="utf-8") + try: + run_command( + "just rust_func array_sum >/dev/null", + cwd, + unset_cc, + progress_label=f"build Rust Long Chain row={row_label}", + ) + finally: + config_file.write_text(original, encoding="utf-8") + + +def collect_results( + cwd: Path, + unset_cc: bool, + repeat: int, + clean: bool, + checkpoint_path: Path, + timeout_seconds: int, +) -> dict: + results = empty_results() + matrix = [ + ("Word Count", WORD_COUNT_ROWS, PARALLEL_COLUMNS), + ("Parallel Sort", PARALLEL_SORT_ROWS, PARALLEL_COLUMNS), + ("Long Chain", LONG_CHAIN_ROWS, LONG_CHAIN_COLUMNS), + ] + + with tempfile.TemporaryDirectory(prefix="alloy-latency-") as temp: + temp_dir = Path(temp) + for workload, size_rows, columns in matrix: + for row_label, size_expr in size_rows: + rust_long_chain_ready = True + if workload == "Long Chain": + try: + build_rust_long_chain(cwd, unset_cc, row_label, size_expr) + except Exception as exc: + rust_long_chain_ready = False + error = str(exc).splitlines()[-1] + + for column in columns: + data_ready = True + if workload != "Long Chain": + try: + prepare_generated_data( + cwd, + unset_cc, + workload, + row_label, + size_expr, + column, + ) + except Exception as exc: + data_ready = False + error = str(exc).splitlines()[-1] + print( + f"[failed] prepare {workload} row={row_label}, " + f"column={column}: {error}", + file=sys.stderr, + flush=True, + ) + + for backend in BACKENDS: + if not data_ready: + value = f"N/A (data preparation failed: {error})" + elif ( + workload == "Long Chain" + and backend == "rust" + and not rust_long_chain_ready + ): + value = f"N/A (Rust build failed: {error})" + else: + try: + config_path = ( + rust_config(workload, column) + if backend == "rust" + else generated_config( + cwd, + temp_dir, + workload, + backend, + column, + size_expr, + ) + ) + value = collect_cell( + cwd, + unset_cc, + workload, + backend, + row_label, + column, + config_path, + repeat, + timeout_seconds, + ) + except Exception as exc: + value = f"N/A ({str(exc).splitlines()[-1]})" + results[workload][backend][row_label][column] = value + write_results(checkpoint_path, results) + + if clean and workload != "Long Chain": + try: + clean_generated_data(cwd, unset_cc) + except Exception as exc: + print( + f"[warn] cleanup failed after {workload} " + f"row={row_label}, column={column}: " + f"{str(exc).splitlines()[-1]}", + file=sys.stderr, + flush=True, + ) + + return results + + +def load_results(path: Path) -> dict: + if not path.exists(): + raise FileNotFoundError(f"missing results file: {path}; run with collect=1 first") + with open(path, "r", encoding="utf-8") as fh: + loaded = json.load(fh) + metadata = loaded.get("_metadata") if isinstance(loaded, dict) else None + if not isinstance(metadata, dict) or metadata.get("schema_version") != RESULTS_SCHEMA_VERSION: + raise RuntimeError( + f"incompatible or legacy results file: {path}; rerun with collect=1" + ) + loaded = loaded.get("results", {}) + results = empty_results() + for section, backends in loaded.items(): + if section not in results: + continue + for backend, rows in backends.items(): + if backend not in results[section]: + continue + for row, columns in rows.items(): + if row not in results[section][backend]: + continue + for column, value in columns.items(): + if column in results[section][backend][row]: + results[section][backend][row][column] = value + return results + + +def write_results(path: Path, results: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + payload = { + "_metadata": { + "schema_version": RESULTS_SCHEMA_VERSION, + "generated_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), + "unit": "ms", + }, + "results": results, + } + temporary_path = path.with_name(f".{path.name}.tmp") + with open(temporary_path, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=2) + fh.write("\n") + fh.flush() + os.fsync(fh.fileno()) + os.replace(temporary_path, path) + + +def render_metric_cell(value: str) -> str: + try: + return f"{float(value):8.3f}" + except (TypeError, ValueError): + return value + + +def render_runtime_table( + title: str, + dimension_name: str, + data_sizes: list[str], + dimensions: list[str], + values: dict, +) -> str: + lines = [f"## {title}", "", "Unit: ms", ""] + header = ["runtime", dimension_name, *data_sizes] + lines.append("| " + " | ".join(header) + " |") + lines.append( + "| " + + " | ".join(["---", "---", *(["---:"] * len(data_sizes))]) + + " |" + ) + for backend in BACKENDS: + for dimension in dimensions: + row_values = [backend, dimension] + row_values.extend( + render_metric_cell(values[backend][data_size][dimension]) + for data_size in data_sizes + ) + lines.append("| " + " | ".join(row_values) + " |") + lines.append("") + return "\n".join(lines) + + +def render_report(results: dict) -> str: + return "\n\n".join( + [ + render_runtime_table( + "Word Count", + "parallelism", + [row for row, _ in WORD_COUNT_ROWS], + PARALLEL_COLUMNS, + results["Word Count"], + ).rstrip(), + render_runtime_table( + "Parallel Sort", + "parallelism", + [row for row, _ in PARALLEL_SORT_ROWS], + PARALLEL_COLUMNS, + results["Parallel Sort"], + ).rstrip(), + render_runtime_table( + "Long Chain", + "chain_len", + [row for row, _ in LONG_CHAIN_ROWS], + LONG_CHAIN_COLUMNS, + results["Long Chain"], + ).rstrip(), + ] + ) + "\n" + + +def resolve_path(cwd: Path, value: str) -> Path: + path = Path(value) + return path if path.is_absolute() else cwd / path + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Generate full-size end-to-end latency report for all runtimes." + ) + parser.add_argument("--output", default="reports/rust_latency_report.md") + parser.add_argument("--results", default="reports/rust_latency_report_results.json") + parser.add_argument("--collect", action="store_true") + parser.add_argument("--clean", action="store_true") + parser.add_argument("--repeat", type=int, default=1) + parser.add_argument( + "--timeout", + type=int, + default=DEFAULT_TEST_TIMEOUT_SECONDS, + help="maximum seconds for each workflow run (default: 600)", + ) + parser.add_argument("--skip-prepare", action="store_true") + parser.add_argument("--keep-cc", action="store_true") + args = parser.parse_args() + + if args.repeat < 1: + raise ValueError("--repeat must be >= 1") + if args.timeout < 1: + raise ValueError("--timeout must be >= 1") + + cwd = repo_root() + output_path = resolve_path(cwd, args.output) + results_path = resolve_path(cwd, args.results) + unset_cc = not args.keep_cc + + if args.collect: + if not args.skip_prepare: + run_prepare(cwd, unset_cc) + results = collect_results( + cwd, + unset_cc, + args.repeat, + args.clean, + results_path, + args.timeout, + ) + write_results(results_path, results) + else: + results = load_results(results_path) + + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(render_report(results), encoding="utf-8") + print(output_path) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/sync_python_workloads.sh b/scripts/sync_python_workloads.sh new file mode 100755 index 00000000..b979ccb7 --- /dev/null +++ b/scripts/sync_python_workloads.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)" +source_dir="$repo_root/user/python_workloads/wasm_bench" +image="${FS_IMAGE:-$repo_root/fs_images/fatfs.img}" +mount_dir="${IMAGE_MOUNT:-$repo_root/image_content}" +files=(wordcount.py parallel_sort.py functionchain.py transfer.py) + +if mountpoint -q "$mount_dir"; then + target_dir="$mount_dir/wasm_bench" + if [[ ! -w "$target_dir" ]]; then + echo "$target_dir is not writable; rerun this script with sudo" >&2 + exit 1 + fi + for file in "${files[@]}"; do + install -m 0755 "$source_dir/$file" "$target_dir/$file" + done +else + command -v mcopy >/dev/null || { + echo "mcopy is required to update an unmounted FAT image" >&2 + exit 1 + } + for file in "${files[@]}"; do + mcopy -o -i "$image" "$source_dir/$file" "::/wasm_bench/$file" + done +fi + +echo "synchronized Python workloads to $image" diff --git a/third_party/cpython b/third_party/cpython new file mode 160000 index 00000000..de54cf5b --- /dev/null +++ b/third_party/cpython @@ -0,0 +1 @@ +Subproject commit de54cf5be371a6f5e2e9f208c38def5f81d3ef02 diff --git a/third_party/musl b/third_party/musl new file mode 160000 index 00000000..9fa28ece --- /dev/null +++ b/third_party/musl @@ -0,0 +1 @@ +Subproject commit 9fa28ece75d8a2191de7c5bb53bed224c5947417 diff --git a/user/array_sum/src/lib.rs b/user/array_sum/src/lib.rs index 74ed5278..ef42081a 100644 --- a/user/array_sum/src/lib.rs +++ b/user/array_sum/src/lib.rs @@ -34,14 +34,22 @@ pub fn main() -> Result<()> { let previous_cnt: i32 = if i == 0 { 0 } else { - DataBuffer::::from_buffer_slot(format!("slot_{}", i - 1)) - .expect("missing data buffer?") - .count + unsafe { + DataBuffer::::from_buffer_slot_owned(format!("slot_{}", i - 1)) + .expect("missing data buffer?") + .count + } }; - let mut next_buffer: DataBuffer = DataBuffer::with_slot(format!("slot_{}", n)); - next_buffer.raw_data = n.repeat(DATA_SIZE); + let output_slot = format!("slot_{}", n); + let mut next_buffer: DataBuffer = DataBuffer::with_slot(output_slot.clone()); + let fill_byte = [b'0' + (i % 10) as u8]; + let fill = core::str::from_utf8(&fill_byte).expect("invalid fill byte"); + next_buffer.raw_data = fill.repeat(DATA_SIZE); next_buffer.count = previous_cnt + 1; + #[cfg(not(feature = "file-based"))] + as_std::agent::buffer_set_len(&output_slot, next_buffer.raw_data.len()) + .expect("failed to set longchain buffer length"); println!("count is {}", next_buffer.count); Ok(().into()) diff --git a/user/checker/src/lib.rs b/user/checker/src/lib.rs index 9e2f2324..9da2b16f 100644 --- a/user/checker/src/lib.rs +++ b/user/checker/src/lib.rs @@ -21,10 +21,7 @@ struct VecArg { #[no_mangle] pub fn main() -> Result<()> { - println!( - "com_start4: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_start4"); let merger_num: u32 = { let m = args::get("merger_num").unwrap(); m.parse().unwrap() @@ -49,10 +46,12 @@ pub fn main() -> Result<()> { last_max = *first } } - println!( - "com_end4: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_end4"); println!("sort result is ok, total sort {} numbers", counter); Ok(().into()) } + +fn print_timestamp(label: &str) { + let micros = SystemTime::now().duration_since(UNIX_EPOCH).as_micros(); + println!("{}: {}.{:06}", label, micros / 1_000_000, micros % 1_000_000); +} diff --git a/user/file_reader/src/lib.rs b/user/file_reader/src/lib.rs index fc05e424..f0e2588b 100644 --- a/user/file_reader/src/lib.rs +++ b/user/file_reader/src/lib.rs @@ -32,10 +32,7 @@ pub fn main() -> Result<()> { let mut buffer: DataBuffer = DataBuffer::with_slot(slot_name.to_owned()); // let mut f = File::open(input_file)?; // let mut buf = String::new(); - println!( - "read_start: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("read_start"); let mut f = File::open(input_file)?; #[cfg(not(feature = "pkey_per_func"))] { @@ -44,6 +41,9 @@ pub fn main() -> Result<()> { .expect("read file failed."); println!("file_reader: read_size={}", size); } + #[cfg(not(feature = "file-based"))] + as_std::agent::buffer_set_len(slot_name, buffer.content.len()) + .expect("failed to set input buffer length"); #[cfg(feature = "pkey_per_func")] { let mut data = String::new(); @@ -53,10 +53,12 @@ pub fn main() -> Result<()> { } // println!("buffer_long={}", buffer.content.len()); // f.read_to_string(&mut buf).expect("read file failed."); - println!( - "read_end: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("read_end"); Ok(().into()) } + +fn print_timestamp(label: &str) { + let micros = SystemTime::now().duration_since(UNIX_EPOCH).as_micros(); + println!("{}: {}.{:06}", label, micros / 1_000_000, micros % 1_000_000); +} diff --git a/user/mapper/src/lib.rs b/user/mapper/src/lib.rs index 9175c05e..2c2a0ecd 100644 --- a/user/mapper/src/lib.rs +++ b/user/mapper/src/lib.rs @@ -139,6 +139,20 @@ fn mapper_func(my_id: &str, reducer_num: u64) -> Result<()> { } } + #[cfg(not(feature = "file-based"))] + for (reducer, buffer) in data_buffers.iter().enumerate() { + let data_len = buffer + .shuffle + .iter() + .map(|(word, _)| word.len() + core::mem::size_of::()) + .sum(); + as_std::agent::buffer_set_len( + &format!("{}-{}", my_id, reducer), + data_len, + ) + .expect("failed to set mapper buffer length"); + } + println!( "shuffle end, cost {}ms", SystemTime::elapsed(&start).as_millis() diff --git a/user/merger/src/lib.rs b/user/merger/src/lib.rs index c81ca151..59bd8cb7 100644 --- a/user/merger/src/lib.rs +++ b/user/merger/src/lib.rs @@ -27,10 +27,7 @@ struct VecArg { #[no_mangle] pub fn main() -> Result<()> { - println!( - "com_start3: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_start3"); let my_id = args::get("id").unwrap(); let sorter_num: u32 = { let m = args::get("sorter_num").unwrap(); @@ -43,21 +40,29 @@ pub fn main() -> Result<()> { }) .collect(); - let mut merged_result: DataBuffer = - DataBuffer::with_slot(format!("merge_result_{}", my_id)); + let output_slot = format!("merge_result_{}", my_id); + let mut merged_result: DataBuffer = DataBuffer::with_slot(output_slot.clone()); merge_partitions( partitions.iter().map(|buffer| &buffer.array).collect(), &mut merged_result.array, ); + #[cfg(not(feature = "file-based"))] + as_std::agent::buffer_set_len( + &output_slot, + merged_result.array.len() * core::mem::size_of::(), + ) + .expect("failed to set merger buffer length"); // println!("merged_result: {:?}", merged_result); - println!( - "com_end3: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_end3"); Ok(().into()) } +fn print_timestamp(label: &str) { + let micros = SystemTime::now().duration_since(UNIX_EPOCH).as_micros(); + println!("{}: {}.{:06}", label, micros / 1_000_000, micros % 1_000_000); +} + fn merge_partitions(partitions: Vec<&NumberArray>, dst: &mut NumberArray) { let mut indices: Vec = partitions.iter().map(|_| 0).collect(); diff --git a/user/musl_checker/Cargo.lock b/user/musl_checker/Cargo.lock new file mode 100644 index 00000000..579468d1 --- /dev/null +++ b/user/musl_checker/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_checker" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_checker/Cargo.toml b/user/musl_checker/Cargo.toml new file mode 100644 index 00000000..836faf09 --- /dev/null +++ b/user/musl_checker/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_checker" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_checker/function.c b/user/musl_checker/function.c new file mode 100644 index 00000000..e2481a38 --- /dev/null +++ b/user/musl_checker/function.c @@ -0,0 +1,39 @@ +#include +#include +#include + +int main(int argc, char **argv) +{ + int merger_num, previous = 0, have_previous = 0; + size_t total = 0; + + if (as_arg_int(argc, argv, "merger_num", &merger_num) || merger_num <= 0) + return 2; + for (int i = 0; i < merger_num; ++i) { + char slot[32]; + as_buffer_t buffer = AS_BUFFER_INIT; + struct int_vec values = {0}; + snprintf(slot, sizeof(slot), "checker_%d", i); + if (as_buffer_take(slot, &buffer) || + int_vec_parse_buffer(&buffer, &values)) { + if (buffer._allocation) + as_buffer_release(&buffer); + int_vec_free(&values); + return 3; + } + for (size_t j = 0; j < values.len; ++j) { + if (have_previous && values.data[j] < previous) { + as_buffer_release(&buffer); + int_vec_free(&values); + return 4; + } + previous = values.data[j]; + have_previous = 1; + total++; + } + as_buffer_release(&buffer); + int_vec_free(&values); + } + printf("musl parallel sort checked %zu values\n", total); + return 0; +} diff --git a/user/musl_checker/src/lib.rs b/user/musl_checker/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_checker/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_cpython/Cargo.lock b/user/musl_cpython/Cargo.lock new file mode 100644 index 00000000..632d01e7 --- /dev/null +++ b/user/musl_cpython/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_cpython" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_cpython/Cargo.toml b/user/musl_cpython/Cargo.toml new file mode 100644 index 00000000..42b90ee2 --- /dev/null +++ b/user/musl_cpython/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_cpython" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_cpython.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_cpython/function.c b/user/musl_cpython/function.c new file mode 100644 index 00000000..cd1bf6a5 --- /dev/null +++ b/user/musl_cpython/function.c @@ -0,0 +1,318 @@ +#include +#include +#include +#include +#include +#include + +int bcmp(const void *left, const void *right, size_t size) +{ + return memcmp(left, right, size); +} + +static PyObject *pyas_error(const char *operation, int error) +{ + PyErr_Format(PyExc_RuntimeError, "%s failed: errno=%d", operation, -error); + return NULL; +} + +static PyObject *pyas_buffer_register(PyObject *self, PyObject *args) +{ + const char *slot; + Py_ssize_t capacity; + as_buffer_t buffer = AS_BUFFER_INIT; + PyObject *view; + int error; + + (void)self; + if (!PyArg_ParseTuple(args, "sn:buffer_register", &slot, &capacity)) + return NULL; + if (capacity < 0) { + PyErr_SetString(PyExc_ValueError, "buffer capacity must be non-negative"); + return NULL; + } + + error = as_buffer_alloc((size_t)capacity, &buffer); + if (error) + return pyas_error("as_buffer_alloc", error); + memset(buffer.data, 0, (size_t)capacity); + + view = PyMemoryView_FromMemory((char *)buffer.data, capacity, PyBUF_WRITE); + if (!view) { + as_buffer_release(&buffer); + return NULL; + } + + /* + * Publish before returning, matching the original WASM pyas semantics. + * The returned memoryview is a non-owning producer view; synchronization + * through a flag must happen after the producer finishes writing. + */ + error = as_buffer_publish(slot, &buffer, (size_t)capacity); + if (error) { + Py_DECREF(view); + if (buffer.data) + as_buffer_release(&buffer); + return pyas_error("as_buffer_publish", error); + } + return view; +} + +static PyObject *pyas_access_buffer(PyObject *self, PyObject *args) +{ + const char *slot; + PyObject *target; + Py_buffer target_view; + as_buffer_t source = AS_BUFFER_INIT; + int error; + + (void)self; + if (!PyArg_ParseTuple(args, "sO:access_buffer", &slot, &target)) + return NULL; + if (PyObject_GetBuffer(target, &target_view, PyBUF_WRITABLE) < 0) + return NULL; + + error = as_buffer_take(slot, &source); + if (error == -ENOENT) { + memset(target_view.buf, 0, (size_t)target_view.len); + PyBuffer_Release(&target_view); + Py_RETURN_NONE; + } + if (error) { + PyBuffer_Release(&target_view); + return pyas_error("as_buffer_take", error); + } + if ((size_t)target_view.len < source.len) { + size_t required = source.len; + Py_ssize_t available = target_view.len; + error = as_buffer_publish(slot, &source, required); + PyBuffer_Release(&target_view); + if (error) { + if (source.data) + as_buffer_release(&source); + return pyas_error("as_buffer_publish", error); + } + PyErr_Format(PyExc_ValueError, + "target buffer is too small: need %zu, got %zd", + required, available); + return NULL; + } + + memcpy(target_view.buf, source.data, source.len); + if ((size_t)target_view.len > source.len) + memset((char *)target_view.buf + source.len, 0, + (size_t)target_view.len - source.len); + PyBuffer_Release(&target_view); + error = as_buffer_publish(slot, &source, source.len); + if (error) { + if (source.data) + as_buffer_release(&source); + return pyas_error("as_buffer_publish", error); + } + Py_RETURN_NONE; +} + +static PyObject *pyas_buffer_len(PyObject *self, PyObject *args) +{ + const char *slot; + size_t len; + int error; + + (void)self; + if (!PyArg_ParseTuple(args, "s:buffer_len", &slot)) + return NULL; + error = as_buffer_len(slot, &len); + if (error) + return pyas_error("as_buffer_len", error); + return PyLong_FromSize_t(len); +} + +static PyObject *pyas_move_buffer(PyObject *self, PyObject *args) +{ + const char *source_slot; + const char *target_slot; + as_buffer_t buffer = AS_BUFFER_INIT; + int error; + + (void)self; + if (!PyArg_ParseTuple(args, "ss:move_buffer", &source_slot, &target_slot)) + return NULL; + error = as_buffer_take(source_slot, &buffer); + if (error) + return pyas_error("as_buffer_take", error); + error = as_buffer_publish(target_slot, &buffer, buffer.len); + if (error) { + if (buffer.data) + as_buffer_release(&buffer); + return pyas_error("as_buffer_publish", error); + } + Py_RETURN_NONE; +} + +static PyObject *pyas_take_buffer(PyObject *self, PyObject *args) +{ + const char *slot; + as_buffer_t source = AS_BUFFER_INIT; + PyObject *view; + int error; + + (void)self; + if (!PyArg_ParseTuple(args, "s:take_buffer", &slot)) + return NULL; + error = as_buffer_take(slot, &source); + if (error == -ENOENT) + Py_RETURN_NONE; + if (error) + return pyas_error("as_buffer_take", error); + view = PyMemoryView_FromMemory((char *)source.data, + (Py_ssize_t)source.len, PyBUF_READ); + if (!view) { + error = as_buffer_publish(slot, &source, source.len); + if (error) { + if (source.data) + as_buffer_release(&source); + return pyas_error("as_buffer_publish", error); + } + } + return view; +} + +static PyMethodDef pyas_methods[] = { + {"buffer_register", pyas_buffer_register, METH_VARARGS, + "Allocate and publish a writable shared buffer."}, + {"access_buffer", pyas_access_buffer, METH_VARARGS, + "Copy a shared buffer into a writable Python buffer without consuming it."}, + {"buffer_len", pyas_buffer_len, METH_VARARGS, + "Return the valid byte length stored in a shared-buffer slot."}, + {"move_buffer", pyas_move_buffer, METH_VARARGS, + "Move a shared buffer to another slot without copying."}, + {"take_buffer", pyas_take_buffer, METH_VARARGS, + "Consume a shared buffer and return a zero-copy, read-only view."}, + {NULL, NULL, 0, NULL}, +}; + +static struct PyModuleDef pyas_module = { + PyModuleDef_HEAD_INIT, + "pyas", + "AlloyStack shared-buffer bindings.", + -1, + pyas_methods, +}; + +PyMODINIT_FUNC PyInit_pyas(void) +{ + return PyModule_Create(&pyas_module); +} + +static const char *find_arg(int argc, char **argv, const char *key) +{ + size_t key_len = strlen(key); + for (int i = 1; i < argc; ++i) { + if (argv[i][0] == '-' && argv[i][1] == '-' && + strncmp(argv[i] + 2, key, key_len) == 0 && + argv[i][key_len + 2] == '=') + return argv[i] + key_len + 3; + } + return NULL; +} + +static int report_status(PyStatus status) +{ + if (!PyStatus_Exception(status)) + return 0; + fprintf(stderr, "CPython init failed: %s\n", + status.err_msg ? status.err_msg : "unknown error"); + return 1; +} + +static int configure_argv(PyConfig *config, const char *script, + int argc, char **argv) +{ + const char *id = find_arg(argc, argv, "id"); + const char *data_size = find_arg(argc, argv, "data_size"); + const char *mapper_num = find_arg(argc, argv, "mapper_num"); + const char *reducer_num = find_arg(argc, argv, "reducer_num"); + const char *sorter_num = find_arg(argc, argv, "sorter_num"); + const char *merger_num = find_arg(argc, argv, "merger_num"); + const char *func_num = find_arg(argc, argv, "func_num"); + char *python_argv[5]; + Py_ssize_t python_argc = 0; + PyStatus status; + + python_argv[python_argc++] = (char *)(script ? script : "python"); + if (data_size) + python_argv[python_argc++] = (char *)data_size; + if (func_num) { + python_argv[python_argc++] = (char *)func_num; + } else if (sorter_num && merger_num) { + python_argv[python_argc++] = (char *)(id ? id : "0"); + python_argv[python_argc++] = (char *)sorter_num; + python_argv[python_argc++] = (char *)merger_num; + } else if (mapper_num && reducer_num) { + python_argv[python_argc++] = (char *)(id ? id : "0"); + python_argv[python_argc++] = (char *)mapper_num; + python_argv[python_argc++] = (char *)reducer_num; + } + + config->parse_argv = 0; + status = PyConfig_SetBytesArgv(config, python_argc, python_argv); + return report_status(status); +} + +int main(int argc, char **argv) +{ + const char *script = find_arg(argc, argv, "pyfile_path"); + PyConfig config; + PyStatus status; + int rc = 0; + + PyConfig_InitIsolatedConfig(&config); + config.install_signal_handlers = 0; + config.site_import = 0; + config.write_bytecode = 0; + config.module_search_paths_set = 1; + + if (PyImport_AppendInittab("pyas", PyInit_pyas) < 0) { + fprintf(stderr, "failed to register built-in pyas module\n"); + goto init_failed; + } + status = PyConfig_SetString(&config, &config.program_name, L"python"); + if (report_status(status)) + goto init_failed; + status = PyConfig_SetString(&config, &config.home, L"/"); + if (report_status(status)) + goto init_failed; + status = PyWideStringList_Append(&config.module_search_paths, L"Lib"); + if (report_status(status)) + goto init_failed; + if (configure_argv(&config, script, argc, argv)) + goto init_failed; + status = Py_InitializeFromConfig(&config); + if (report_status(status)) + goto init_failed; + PyConfig_Clear(&config); + + if (script) { + FILE *file = fopen(script, "r"); + if (!file) { + fprintf(stderr, "cannot open Python script: %s\n", script); + rc = 2; + } else { + rc = PyRun_SimpleFileExFlags(file, script, 1, NULL); + } + } else { + rc = PyRun_SimpleString("print('native musl CPython is running')"); + } + /* + * The LibOS currently exposes only one musl thread descriptor per host + * thread. Py_FinalizeEx() deletes CPython's pthread keys and can block in + * musl's process-wide key-table teardown. The isolation is unloaded after + * this entry point returns, so defer finalization until that pthread + * lifecycle is implemented by as_musl. + */ + return rc; + +init_failed: + PyConfig_Clear(&config); + return 1; +} diff --git a/user/musl_cpython/src/lib.rs b/user/musl_cpython/src/lib.rs new file mode 100644 index 00000000..bf8fb0f8 --- /dev/null +++ b/user/musl_cpython/src/lib.rs @@ -0,0 +1,5 @@ +#![no_std] + +// CPython behaves as a process-lifetime runtime. Until its allocator teardown +// is isolated from the Rust app heap, avoid post-main allocation and flushing. +as_musl::entry_no_flush!(alloy_c_main); diff --git a/user/musl_hello/Cargo.lock b/user/musl_hello/Cargo.lock new file mode 100644 index 00000000..b28664c4 --- /dev/null +++ b/user/musl_hello/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_hello" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_hello/Cargo.toml b/user/musl_hello/Cargo.toml new file mode 100644 index 00000000..6cb2af38 --- /dev/null +++ b/user/musl_hello/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_hello" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_hello/function.c b/user/musl_hello/function.c new file mode 100644 index 00000000..89b0f1a7 --- /dev/null +++ b/user/musl_hello/function.c @@ -0,0 +1,81 @@ +#include +#include +#include +#include +#include +#include +#include + +static const char *find_id(int argc, char **argv) +{ + const char prefix[] = "--id="; + for (int i = 1; i < argc; ++i) { + if (strncmp(argv[i], prefix, sizeof(prefix) - 1) == 0) + return argv[i] + sizeof(prefix) - 1; + } + return "unknown"; +} + +int main(int argc, char **argv) +{ + const char *id = find_id(argc, argv); + char path[64]; + char *payload = malloc(256 * 1024); + char readback[128] = {0}; + struct stat st; + + if (!payload) + return 10; + memset(payload, 'A', 256 * 1024 - 1); + payload[256 * 1024 - 1] = '\0'; + char *grown = realloc(payload, 512 * 1024); + if (!grown) { + free(payload); + return 18; + } + payload = grown; + memset(payload + 256 * 1024, 'B', 256 * 1024); + + if (snprintf(path, sizeof(path), "musl-%s.txt", id) < 0) { + free(payload); + return 11; + } + + FILE *file = fopen(path, "w+"); + if (!file) { + free(payload); + return 12; + } + if (fwrite(payload, 1, 127, file) != 127 || fflush(file) != 0) { + fclose(file); + free(payload); + return 13; + } + if (fseek(file, 0, SEEK_SET) != 0 || + fread(readback, 1, 127, file) != 127 || + memcmp(payload, readback, 127) != 0) { + fclose(file); + free(payload); + return 14; + } + if (fstat(fileno(file), &st) != 0 || st.st_size != 127) { + fclose(file); + free(payload); + return 15; + } + if (fclose(file) != 0) { + free(payload); + return 16; + } + + printf("musl function id=%s file=%s size=%ld\n", id, path, (long)st.st_size); + + errno = 0; + if (syscall(SYS_getpid) != -1 || errno != ENOSYS) { + free(payload); + return 17; + } + + free(payload); + return 0; +} diff --git a/user/musl_hello/src/lib.rs b/user/musl_hello/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_hello/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_longchain/Cargo.lock b/user/musl_longchain/Cargo.lock new file mode 100644 index 00000000..45928dca --- /dev/null +++ b/user/musl_longchain/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_longchain" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_longchain/Cargo.toml b/user/musl_longchain/Cargo.toml new file mode 100644 index 00000000..9111fd19 --- /dev/null +++ b/user/musl_longchain/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_longchain" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_longchain/function.c b/user/musl_longchain/function.c new file mode 100644 index 00000000..e706c305 --- /dev/null +++ b/user/musl_longchain/function.c @@ -0,0 +1,48 @@ +#include +#include +#include + +int main(int argc, char **argv) +{ + int id, stage, chain_len, data_size; + char input_slot[32], output_slot[32]; + as_buffer_t buffer = AS_BUFFER_INIT; + int rc; + + if (as_arg_int(argc, argv, "id", &id) || + as_arg_int(argc, argv, "func_num", &stage) || + as_arg_int(argc, argv, "chain_len", &chain_len) || + as_arg_int(argc, argv, "data_size", &data_size) || + stage < 0 || stage >= chain_len || data_size < 2) + return 2; + + if (stage == 0) { + rc = as_buffer_alloc((size_t)data_size, &buffer); + if (rc) + return 3; + ((char *)buffer.data)[0] = '0'; + ((char *)buffer.data)[1] = '\0'; + } else { + snprintf(input_slot, sizeof(input_slot), "buffer_%d_%d", stage - 1, id); + rc = as_buffer_take(input_slot, &buffer); + if (rc || buffer.len != (size_t)data_size) + return 4; + ((char *)buffer.data)[0]++; + } + + if (stage + 1 == chain_len) { + rc = (((char *)buffer.data)[0] == '0' + stage) ? 0 : 5; + if (as_buffer_release(&buffer)) + return 6; + printf("musl longchain result=%c\n", '0' + stage); + return rc; + } + + snprintf(output_slot, sizeof(output_slot), "buffer_%d_%d", stage, id); + rc = as_buffer_publish(output_slot, &buffer, (size_t)data_size); + if (rc) { + as_buffer_release(&buffer); + return 7; + } + return 0; +} diff --git a/user/musl_longchain/src/lib.rs b/user/musl_longchain/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_longchain/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_mapper/Cargo.lock b/user/musl_mapper/Cargo.lock new file mode 100644 index 00000000..3618b8bc --- /dev/null +++ b/user/musl_mapper/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_mapper" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_mapper/Cargo.toml b/user/musl_mapper/Cargo.toml new file mode 100644 index 00000000..4c818762 --- /dev/null +++ b/user/musl_mapper/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_mapper" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_mapper/function.c b/user/musl_mapper/function.c new file mode 100644 index 00000000..b689cec0 --- /dev/null +++ b/user/musl_mapper/function.c @@ -0,0 +1,143 @@ +#include +#include +#include +#include +#include +#include +#include + +#define BUCKETS 1024 +#define WORD_SIZE 256 + +struct word_entry { + struct word_entry *next; + char *word; + unsigned count; + unsigned partition; +}; + +static uint64_t word_hash(const char *word) +{ + uint64_t hash = UINT64_C(1469598103934665603); + while (*word) { + hash ^= (unsigned char)*word++; + hash *= UINT64_C(1099511628211); + } + return hash; +} + +static void free_table(struct word_entry **table) +{ + for (size_t i = 0; i < BUCKETS; ++i) { + struct word_entry *entry = table[i]; + while (entry) { + struct word_entry *next = entry->next; + free(entry->word); + free(entry); + entry = next; + } + } +} + +int main(int argc, char **argv) +{ + struct word_entry **table = calloc(BUCKETS, sizeof(*table)); + int id, reducer_num, rc = 0; + char path[64], word[WORD_SIZE]; + FILE *file = NULL; + size_t *sizes = NULL, *offsets = NULL; + as_buffer_t *buffers = NULL; + + if (!table || as_arg_int(argc, argv, "id", &id) || + as_arg_int(argc, argv, "reducer_num", &reducer_num) || + reducer_num <= 0) + return 2; + snprintf(path, sizeof(path), "fake_data_%d.txt", id); + file = fopen(path, "r"); + if (!file) { + free(table); + return 3; + } + while (fscanf(file, "%255s", word) == 1) { + uint64_t hash; + size_t bucket; + struct word_entry *entry; + for (char *p = word; *p; ++p) + *p = (char)tolower((unsigned char)*p); + hash = word_hash(word); + bucket = hash % BUCKETS; + for (entry = table[bucket]; entry; entry = entry->next) { + if (strcmp(entry->word, word) == 0) { + entry->count++; + break; + } + } + if (!entry) { + entry = malloc(sizeof(*entry)); + if (!entry || !(entry->word = strdup(word))) { + free(entry); + rc = 4; + goto cleanup; + } + entry->count = 1; + entry->partition = hash % (unsigned)reducer_num; + entry->next = table[bucket]; + table[bucket] = entry; + } + } + fclose(file); + file = NULL; + + sizes = calloc((size_t)reducer_num, sizeof(*sizes)); + offsets = calloc((size_t)reducer_num, sizeof(*offsets)); + buffers = calloc((size_t)reducer_num, sizeof(*buffers)); + if (!sizes || !offsets || !buffers) { + rc = 5; + goto cleanup; + } + for (size_t i = 0; i < BUCKETS; ++i) + for (struct word_entry *entry = table[i]; entry; entry = entry->next) + sizes[entry->partition] += + (size_t)snprintf(NULL, 0, "%s %u\n", entry->word, entry->count); + + for (int i = 0; i < reducer_num; ++i) { + if (as_buffer_alloc(sizes[i] + 1, &buffers[i])) { + rc = 6; + goto cleanup; + } + } + for (size_t i = 0; i < BUCKETS; ++i) { + for (struct word_entry *entry = table[i]; entry; entry = entry->next) { + unsigned p = entry->partition; + offsets[p] += (size_t)snprintf( + (char *)buffers[p].data + offsets[p], + buffers[p].capacity - offsets[p], + "%s %u\n", + entry->word, + entry->count); + } + } + for (int i = 0; i < reducer_num; ++i) { + char slot[32]; + ((char *)buffers[i].data)[offsets[i]] = '\0'; + snprintf(slot, sizeof(slot), "buffer_%d_%d", i, id); + if (as_buffer_publish(slot, &buffers[i], offsets[i] + 1)) { + rc = 7; + goto cleanup; + } + } +cleanup: + if (file) + fclose(file); + if (buffers) { + for (int i = 0; i < reducer_num; ++i) + if (buffers[i]._allocation) + as_buffer_release(&buffers[i]); + } + free(buffers); + free(offsets); + free(sizes); + free_table(table); + free(table); + return rc; +} diff --git a/user/musl_mapper/src/lib.rs b/user/musl_mapper/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_mapper/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_merger/Cargo.lock b/user/musl_merger/Cargo.lock new file mode 100644 index 00000000..b8f5df20 --- /dev/null +++ b/user/musl_merger/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_merger" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_merger/Cargo.toml b/user/musl_merger/Cargo.toml new file mode 100644 index 00000000..26d15ad0 --- /dev/null +++ b/user/musl_merger/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_merger" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_merger/function.c b/user/musl_merger/function.c new file mode 100644 index 00000000..fa2ca9dd --- /dev/null +++ b/user/musl_merger/function.c @@ -0,0 +1,94 @@ +#include +#include +#include +#include + +struct heap_node { + int value; + int source; + size_t index; +}; + +static void heap_down(struct heap_node *heap, int size, int index) +{ + for (;;) { + int smallest = index, left = index * 2 + 1, right = left + 1; + if (left < size && heap[left].value < heap[smallest].value) + smallest = left; + if (right < size && heap[right].value < heap[smallest].value) + smallest = right; + if (smallest == index) + return; + struct heap_node tmp = heap[index]; + heap[index] = heap[smallest]; + heap[smallest] = tmp; + index = smallest; + } +} + +int main(int argc, char **argv) +{ + int id, sorter_num, merger_num, rc = 0, heap_size = 0; + char slot[32]; + struct int_vec *sources = NULL, result = {0}; + struct heap_node *heap = NULL; + + if (as_arg_int(argc, argv, "id", &id) || + as_arg_int(argc, argv, "sorter_num", &sorter_num) || + as_arg_int(argc, argv, "merger_num", &merger_num) || + sorter_num <= 0 || id < 0 || id >= merger_num) + return 2; + sources = calloc((size_t)sorter_num, sizeof(*sources)); + heap = malloc((size_t)sorter_num * sizeof(*heap)); + if (!sources || !heap) { + rc = 3; + goto cleanup; + } + for (int i = 0; i < sorter_num; ++i) { + as_buffer_t buffer = AS_BUFFER_INIT; + snprintf(slot, sizeof(slot), "merger_%d_%d", i, id); + if (as_buffer_take(slot, &buffer) || + int_vec_parse_buffer(&buffer, &sources[i])) { + if (buffer._allocation) + as_buffer_release(&buffer); + rc = 4; + goto cleanup; + } + as_buffer_release(&buffer); + if (sources[i].len) { + heap[heap_size++] = (struct heap_node){ + .value = sources[i].data[0], .source = i, .index = 0}; + } + } + for (int i = heap_size / 2; i-- > 0;) + heap_down(heap, heap_size, i); + while (heap_size) { + struct heap_node node = heap[0]; + if (int_vec_push(&result, node.value)) { + rc = 5; + goto cleanup; + } + node.index++; + if (node.index < sources[node.source].len) { + node.value = sources[node.source].data[node.index]; + heap[0] = node; + } else { + heap[0] = heap[--heap_size]; + } + if (heap_size) + heap_down(heap, heap_size, 0); + } + snprintf(slot, sizeof(slot), "checker_%d", id); + if (publish_int_vec(slot, result.data, result.len)) + rc = 6; + +cleanup: + if (sources) { + for (int i = 0; i < sorter_num; ++i) + int_vec_free(&sources[i]); + } + free(sources); + free(heap); + int_vec_free(&result); + return rc; +} diff --git a/user/musl_merger/src/lib.rs b/user/musl_merger/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_merger/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_parallel_common/int_utils.h b/user/musl_parallel_common/int_utils.h new file mode 100644 index 00000000..05071bb0 --- /dev/null +++ b/user/musl_parallel_common/int_utils.h @@ -0,0 +1,121 @@ +#ifndef ALLOY_INT_UTILS_H +#define ALLOY_INT_UTILS_H + +#include +#include +#include +#include + +struct int_vec { + int *data; + size_t len; + size_t capacity; +}; + +static int int_vec_push(struct int_vec *vec, int value) +{ + if (vec->len == vec->capacity) { + size_t capacity = vec->capacity ? vec->capacity * 2 : 256; + int *data; + if (capacity < vec->capacity || capacity > SIZE_MAX / sizeof(*data)) + return -1; + data = realloc(vec->data, capacity * sizeof(*data)); + if (!data) + return -1; + vec->data = data; + vec->capacity = capacity; + } + vec->data[vec->len++] = value; + return 0; +} + +static void int_vec_free(struct int_vec *vec) +{ + free(vec->data); + vec->data = NULL; + vec->len = 0; + vec->capacity = 0; +} + +static int int_vec_parse_file(FILE *file, struct int_vec *out) +{ + int ch, value = 0, in_number = 0; + while ((ch = fgetc(file)) != EOF) { + if (ch >= '0' && ch <= '9') { + if (value > (INT_MAX - (ch - '0')) / 10) + return -1; + value = value * 10 + ch - '0'; + in_number = 1; + } else if (in_number) { + if (int_vec_push(out, value)) + return -1; + value = 0; + in_number = 0; + } + } + if (in_number && int_vec_push(out, value)) + return -1; + return 0; +} + +static int int_vec_parse_buffer(const as_buffer_t *buffer, struct int_vec *out) +{ + const unsigned char *bytes = buffer->data; + int value = 0, in_number = 0; + if (!buffer->len || bytes[buffer->len - 1] != '\0') + return -1; + for (size_t i = 0; i + 1 < buffer->len; ++i) { + int ch = bytes[i]; + if (ch >= '0' && ch <= '9') { + if (value > (INT_MAX - (ch - '0')) / 10) + return -1; + value = value * 10 + ch - '0'; + in_number = 1; + } else if (in_number) { + if (int_vec_push(out, value)) + return -1; + value = 0; + in_number = 0; + } + } + if (in_number && int_vec_push(out, value)) + return -1; + return 0; +} + +static int publish_int_vec(const char *slot, const int *values, size_t len) +{ + as_buffer_t buffer = AS_BUFFER_INIT; + size_t capacity, used = 0; + int rc; + + if (len > (SIZE_MAX - 1) / 12) + return -1; + capacity = len * 12 + 1; + rc = as_buffer_alloc(capacity, &buffer); + if (rc) + return rc; + for (size_t i = 0; i < len; ++i) { + int written = snprintf( + (char *)buffer.data + used, capacity - used, "%d ", values[i]); + if (written < 0 || (size_t)written >= capacity - used) { + as_buffer_release(&buffer); + return -1; + } + used += (size_t)written; + } + ((char *)buffer.data)[used++] = '\0'; + rc = as_buffer_publish(slot, &buffer, used); + if (rc) + as_buffer_release(&buffer); + return rc; +} + +static int int_compare(const void *left, const void *right) +{ + int a = *(const int *)left; + int b = *(const int *)right; + return (a > b) - (a < b); +} + +#endif diff --git a/user/musl_reducer/Cargo.lock b/user/musl_reducer/Cargo.lock new file mode 100644 index 00000000..f2a80b48 --- /dev/null +++ b/user/musl_reducer/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_reducer" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_reducer/Cargo.toml b/user/musl_reducer/Cargo.toml new file mode 100644 index 00000000..8d236fc1 --- /dev/null +++ b/user/musl_reducer/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_reducer" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_reducer/function.c b/user/musl_reducer/function.c new file mode 100644 index 00000000..511b3831 --- /dev/null +++ b/user/musl_reducer/function.c @@ -0,0 +1,105 @@ +#include +#include +#include +#include +#include +#include + +#define BUCKETS 1024 + +struct word_entry { + struct word_entry *next; + char *word; + unsigned long count; +}; + +static uint64_t word_hash(const char *word) +{ + uint64_t hash = UINT64_C(1469598103934665603); + while (*word) { + hash ^= (unsigned char)*word++; + hash *= UINT64_C(1099511628211); + } + return hash; +} + +static void free_table(struct word_entry **table) +{ + for (size_t i = 0; i < BUCKETS; ++i) { + struct word_entry *entry = table[i]; + while (entry) { + struct word_entry *next = entry->next; + free(entry->word); + free(entry); + entry = next; + } + } +} + +int main(int argc, char **argv) +{ + struct word_entry **table = calloc(BUCKETS, sizeof(*table)); + int id, mapper_num, rc = 0; + unsigned long total = 0, unique = 0; + + if (!table || as_arg_int(argc, argv, "id", &id) || + as_arg_int(argc, argv, "mapper_num", &mapper_num) || + mapper_num <= 0) + return 2; + + for (int mapper = 0; mapper < mapper_num; ++mapper) { + char slot[32], *save = NULL, *line; + as_buffer_t buffer = AS_BUFFER_INIT; + snprintf(slot, sizeof(slot), "buffer_%d_%d", id, mapper); + if (as_buffer_take(slot, &buffer)) { + rc = 3; + goto cleanup; + } + if (!buffer.len || ((char *)buffer.data)[buffer.len - 1] != '\0') { + as_buffer_release(&buffer); + rc = 4; + goto cleanup; + } + for (line = strtok_r(buffer.data, "\n", &save); line; + line = strtok_r(NULL, "\n", &save)) { + char word[256]; + unsigned count; + size_t bucket; + struct word_entry *entry; + if (sscanf(line, "%255s %u", word, &count) != 2) { + as_buffer_release(&buffer); + rc = 5; + goto cleanup; + } + bucket = word_hash(word) % BUCKETS; + for (entry = table[bucket]; entry; entry = entry->next) + if (strcmp(entry->word, word) == 0) + break; + if (!entry) { + entry = malloc(sizeof(*entry)); + if (!entry || !(entry->word = strdup(word))) { + free(entry); + as_buffer_release(&buffer); + rc = 6; + goto cleanup; + } + entry->count = 0; + entry->next = table[bucket]; + table[bucket] = entry; + unique++; + } + entry->count += count; + total += count; + } + if (as_buffer_release(&buffer)) { + rc = 7; + goto cleanup; + } + } + printf("musl reducer_%d unique=%lu total=%lu\n", id, unique, total); + +cleanup: + free_table(table); + free(table); + return rc; +} diff --git a/user/musl_reducer/src/lib.rs b/user/musl_reducer/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_reducer/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_sorter/Cargo.lock b/user/musl_sorter/Cargo.lock new file mode 100644 index 00000000..855821b6 --- /dev/null +++ b/user/musl_sorter/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_sorter" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_sorter/Cargo.toml b/user/musl_sorter/Cargo.toml new file mode 100644 index 00000000..71184add --- /dev/null +++ b/user/musl_sorter/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_sorter" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_sorter/function.c b/user/musl_sorter/function.c new file mode 100644 index 00000000..e912963e --- /dev/null +++ b/user/musl_sorter/function.c @@ -0,0 +1,59 @@ +#include +#include +#include + +int main(int argc, char **argv) +{ + int id, sorter_num, merger_num, rc = 0; + char path[64], slot[32]; + FILE *file = NULL; + struct int_vec values = {0}; + + if (as_arg_int(argc, argv, "id", &id) || + as_arg_int(argc, argv, "sorter_num", &sorter_num) || + as_arg_int(argc, argv, "merger_num", &merger_num) || + sorter_num <= 0 || merger_num <= 0) + return 2; + snprintf(path, sizeof(path), "sort_data_%d.txt", id); + file = fopen(path, "r"); + if (!file) + return 3; + if (int_vec_parse_file(file, &values)) { + rc = 4; + goto cleanup; + } + fclose(file); + file = NULL; + qsort(values.data, values.len, sizeof(*values.data), int_compare); + + if (id == 0 && merger_num > 1) { + struct int_vec pivots = {0}; + for (int i = 1; i < merger_num; ++i) { + size_t index = (size_t)i * values.len / (size_t)merger_num; + if (index >= values.len || int_vec_push(&pivots, values.data[index])) { + int_vec_free(&pivots); + rc = 5; + goto cleanup; + } + } + for (int i = 0; i < sorter_num; ++i) { + snprintf(slot, sizeof(slot), "pivot_%d", i); + if (publish_int_vec(slot, pivots.data, pivots.len)) { + int_vec_free(&pivots); + rc = 6; + goto cleanup; + } + } + int_vec_free(&pivots); + } + + snprintf(slot, sizeof(slot), "sorter_%d", id); + if (publish_int_vec(slot, values.data, values.len)) + rc = 7; + +cleanup: + if (file) + fclose(file); + int_vec_free(&values); + return rc; +} diff --git a/user/musl_sorter/src/lib.rs b/user/musl_sorter/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_sorter/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_splitter/Cargo.lock b/user/musl_splitter/Cargo.lock new file mode 100644 index 00000000..c58d92ef --- /dev/null +++ b/user/musl_splitter/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_splitter" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_splitter/Cargo.toml b/user/musl_splitter/Cargo.toml new file mode 100644 index 00000000..4c0a4d9c --- /dev/null +++ b/user/musl_splitter/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_splitter" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_splitter/function.c b/user/musl_splitter/function.c new file mode 100644 index 00000000..3a4dbb15 --- /dev/null +++ b/user/musl_splitter/function.c @@ -0,0 +1,71 @@ +#include +#include +#include +#include + +int main(int argc, char **argv) +{ + int id, sorter_num, merger_num, rc = 0; + char slot[32]; + as_buffer_t input = AS_BUFFER_INIT; + struct int_vec pivots = {0}, values = {0}; + struct int_vec *parts = NULL; + + if (as_arg_int(argc, argv, "id", &id) || + as_arg_int(argc, argv, "sorter_num", &sorter_num) || + as_arg_int(argc, argv, "merger_num", &merger_num) || + id < 0 || id >= sorter_num || merger_num <= 0) + return 2; + + if (merger_num > 1) { + snprintf(slot, sizeof(slot), "pivot_%d", id); + if (as_buffer_take(slot, &input) || + int_vec_parse_buffer(&input, &pivots)) { + rc = 3; + goto cleanup; + } + as_buffer_release(&input); + } + snprintf(slot, sizeof(slot), "sorter_%d", id); + if (as_buffer_take(slot, &input) || + int_vec_parse_buffer(&input, &values)) { + rc = 4; + goto cleanup; + } + as_buffer_release(&input); + + parts = calloc((size_t)merger_num, sizeof(*parts)); + if (!parts) { + rc = 5; + goto cleanup; + } + for (size_t i = 0; i < values.len; ++i) { + int partition = 0; + while ((size_t)partition < pivots.len && + values.data[i] >= pivots.data[partition]) + partition++; + if (int_vec_push(&parts[partition], values.data[i])) { + rc = 6; + goto cleanup; + } + } + for (int i = 0; i < merger_num; ++i) { + snprintf(slot, sizeof(slot), "merger_%d_%d", id, i); + if (publish_int_vec(slot, parts[i].data, parts[i].len)) { + rc = 7; + goto cleanup; + } + } + +cleanup: + if (input._allocation) + as_buffer_release(&input); + if (parts) { + for (int i = 0; i < merger_num; ++i) + int_vec_free(&parts[i]); + } + free(parts); + int_vec_free(&values); + int_vec_free(&pivots); + return rc; +} diff --git a/user/musl_splitter/src/lib.rs b/user/musl_splitter/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_splitter/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/musl_trans_data/Cargo.lock b/user/musl_trans_data/Cargo.lock new file mode 100644 index 00000000..057c6578 --- /dev/null +++ b/user/musl_trans_data/Cargo.lock @@ -0,0 +1,296 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_musl" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" + +[[package]] +name = "derive_more" +version = "0.99.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.118", +] + +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32", + "stable_deref_trait", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "linked_list_allocator" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b23ac50abb8261cb38c6e2a7192d3302e0836dac1628f6a93b82b4fad185897" +dependencies = [ + "spinning_top", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "musl_trans_data" +version = "0.1.0" +dependencies = [ + "as_musl", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", +] + +[[package]] +name = "spinning_top" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b9eb1a2f4c41445a3a0ff9abc5221c5fcd28e1f13cd7c0397706f9ac938ddb0" +dependencies = [ + "lock_api", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror-impl-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58e6318948b519ba6dc2b442a6d0b904ebfb8d411a3ad3e07843615a72249758" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "thiserror-no-std" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3ad459d94dd517257cc96add8a43190ee620011bb6e6cdc82dafd97dfafafea" +dependencies = [ + "thiserror-impl-no-std", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/musl_trans_data/Cargo.toml b/user/musl_trans_data/Cargo.toml new file mode 100644 index 00000000..3dae792d --- /dev/null +++ b/user/musl_trans_data/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "musl_trans_data" +version = "0.1.0" +edition = "2021" +build = "../../build_musl_user.rs" + +[lib] +crate-type = ["dylib"] + +[dependencies] +as_musl = { path = "../../as_musl" } + +[features] +default = [] +mpk = ["as_musl/mpk"] diff --git a/user/musl_trans_data/function.c b/user/musl_trans_data/function.c new file mode 100644 index 00000000..c33a34f5 --- /dev/null +++ b/user/musl_trans_data/function.c @@ -0,0 +1,69 @@ +#include +#include +#include +#include +#include +#include +#include + +static uint64_t now_ns(void) +{ + struct timeval tv; + gettimeofday(&tv, NULL); + return (uint64_t)tv.tv_sec * 1000000000ull + + (uint64_t)tv.tv_usec * 1000ull; +} + +static size_t find_data_size(int argc, char **argv) +{ + const char prefix[] = "--data_size="; + for (int i = 1; i < argc; ++i) { + if (strncmp(argv[i], prefix, sizeof(prefix) - 1) == 0) { + char *end = NULL; + unsigned long long value = strtoull( + argv[i] + sizeof(prefix) - 1, &end, 10); + if (end && *end == '\0') + return (size_t)value; + } + } + return 4 * 1024; +} + +int main(int argc, char **argv) +{ + const char *slot = "slot_1"; + as_buffer_t producer = AS_BUFFER_INIT; + as_buffer_t consumer = AS_BUFFER_INIT; + size_t data_size = find_data_size(argc, argv); + uint64_t start_ns; + uint64_t end_ns; + int rc; + + rc = as_buffer_alloc(data_size, &producer); + if (rc) + return -rc; + if (data_size) + memset(producer.data, 'a', data_size); + + start_ns = now_ns(); + rc = as_buffer_publish(slot, &producer, data_size); + if (rc) { + as_buffer_release(&producer); + return -rc; + } + rc = as_buffer_take(slot, &consumer); + end_ns = now_ns(); + if (rc) + return -rc; + + if (data_size && consumer.data) + (void)((volatile unsigned char *)consumer.data)[0]; + + printf("transfer cost: %llu ns size: %zu\n", + (unsigned long long)(end_ns - start_ns), data_size); + + rc = as_buffer_release(&consumer); + if (rc) + return -rc; + return 0; +} diff --git a/user/musl_trans_data/src/lib.rs b/user/musl_trans_data/src/lib.rs new file mode 100644 index 00000000..ffe409e7 --- /dev/null +++ b/user/musl_trans_data/src/lib.rs @@ -0,0 +1,3 @@ +#![no_std] + +as_musl::entry!(alloy_c_main); diff --git a/user/python_workloads/README.md b/user/python_workloads/README.md new file mode 100644 index 00000000..ac5b9840 --- /dev/null +++ b/user/python_workloads/README.md @@ -0,0 +1,14 @@ +# Python workflow sources + +`wasm_bench/` is the canonical source for the CPython word-count, +parallel-sort, and function-chain workloads. Both Wasmtime CPython and native +musl CPython execute these files from `/wasm_bench` in the FAT image. + +Synchronize source changes before running a workflow: + +```bash +./scripts/sync_python_workloads.sh +``` + +If `fs_images/fatfs.img` is mounted at `image_content` and the mount is not +writable by the current user, run the command with `sudo -E`. diff --git a/user/python_workloads/wasm_bench/functionchain.py b/user/python_workloads/wasm_bench/functionchain.py new file mode 100755 index 00000000..2ba0f0e6 --- /dev/null +++ b/user/python_workloads/wasm_bench/functionchain.py @@ -0,0 +1,77 @@ +import pyas +import sys +import time + +def get_now(): + return time.time() + +buffer_size = 256 * 1024 * 1024 + +def take_data_buffer(slot_name, size): + get_buffer_len = getattr(pyas, "buffer_len", None) + if get_buffer_len is not None: + size = get_buffer_len(slot_name) + take_buffer = getattr(pyas, "take_buffer", None) + if take_buffer is not None: + return take_buffer(slot_name) + buffer = bytearray(size) + pyas.access_buffer(slot_name, buffer) + return buffer + +def func_inner(func_num, func_n): + # print("python: func {} start".format(func_num), flush=True) + + if func_num == 0: + to_name = "func_{}".format(func_num) + setattr(__import__("__main__"), to_name, pyas.buffer_register(to_name, buffer_size)) + buffer = getattr(__import__("__main__"), to_name) + if buffer_size: + buffer[0] = 1 + buffer[-1] = 1 + # print(getattr(__import__("__main__"), to_name), flush=True) + elif func_num == func_n - 1: + from_name = "func_{}".format(func_num - 1) + buffer = take_data_buffer(from_name, buffer_size) + if hasattr(__import__("__main__"), from_name): + delattr(__import__("__main__"), from_name) + # print(buffer, flush=True) + else: + from_name = "func_{}".format(func_num - 1) + to_name = "func_{}".format(func_num) + move_buffer = getattr(pyas, "move_buffer", None) + if move_buffer is not None: + move_buffer(from_name, to_name) + if hasattr(__import__("__main__"), from_name): + delattr(__import__("__main__"), from_name) + return + source = take_data_buffer(from_name, buffer_size) + capacity = ( + len(source) + if getattr(pyas, "take_buffer", None) is not None + else buffer_size + ) + buffer = pyas.buffer_register(to_name, capacity) + if buffer == None: + print("find None! id: {}".format(func_num), flush=True) + else: + print("id: {} ok!".format(func_num), flush=True) + setattr(__import__("__main__"), to_name, buffer) + buffer[:len(source)] = source + if hasattr(__import__("__main__"), from_name): + delattr(__import__("__main__"), from_name) + + # print("python: func {} finished!".format(func_num), flush=True) + + +def main(): + global buffer_size + if len(sys.argv) >= 3: + buffer_size = int(sys.argv[1]) + func_num = int(sys.argv[2]) + else: + func_num = int(sys.argv[1]) + for i in range(func_num): + func_inner(i, func_num) + +if __name__ == '__main__': + main() diff --git a/user/python_workloads/wasm_bench/parallel_sort.py b/user/python_workloads/wasm_bench/parallel_sort.py new file mode 100755 index 00000000..8c49bcf3 --- /dev/null +++ b/user/python_workloads/wasm_bench/parallel_sort.py @@ -0,0 +1,300 @@ +import pyas +import sys +import time +import heapq + +buffer_size = 60000000 +flag_size = 2 +length_size = 20 + +def get_now(): + return time.time() + +def take_data_buffer(slot_name, size): + take_buffer = getattr(pyas, "take_buffer", None) + if take_buffer is not None: + return take_buffer(slot_name) + get_buffer_len = getattr(pyas, "buffer_len", None) + if get_buffer_len is not None: + size = get_buffer_len(slot_name) + else: + length_buffer = bytearray(length_size) + pyas.access_buffer(slot_name + ".len", length_buffer) + size = int(bytes(length_buffer).rstrip(b"\x00")) + buffer = bytearray(size) + pyas.access_buffer(slot_name, buffer) + return buffer + +def publish_data_buffer(slot_name, encoded_data): + buffer = pyas.buffer_register(slot_name, len(encoded_data)) + buffer[:] = encoded_data + setattr(__import__("__main__"), slot_name, buffer) + if ( + getattr(pyas, "buffer_len", None) is None + and getattr(pyas, "take_buffer", None) is None + ): + length_slot = slot_name + ".len" + length_buffer = pyas.buffer_register(length_slot, length_size) + encoded_len = str(len(encoded_data)).encode() + length_buffer[:len(encoded_len)] = encoded_len + setattr(__import__("__main__"), length_slot, length_buffer) + +setattr(__import__("__main__"), "trans_start_times", 0) +def get_trans_start_time(): + setattr(__import__("__main__"), "trans_start_times", getattr(__import__("__main__"), "trans_start_times") + 1) + return getattr(__import__("__main__"), "trans_start_times") + +setattr(__import__("__main__"), "compute_start_times", 0) +def get_compute_start_time(): + setattr(__import__("__main__"), "compute_start_times", getattr(__import__("__main__"), "compute_start_times") + 1) + return getattr(__import__("__main__"), "compute_start_times") + +setattr(__import__("__main__"), "trans_end_times", 0) +def get_trans_end_time(): + setattr(__import__("__main__"), "trans_end_times", getattr(__import__("__main__"), "trans_end_times") + 1) + return getattr(__import__("__main__"), "trans_end_times") + +setattr(__import__("__main__"), "compute_end_times", 0) +def get_compute_end_time(): + setattr(__import__("__main__"), "compute_end_times", getattr(__import__("__main__"), "compute_end_times") + 1) + return getattr(__import__("__main__"), "compute_end_times") + +setattr(__import__("__main__"), "trans_start_times_checker", 0) +def get_trans_start_time_checker(): + setattr(__import__("__main__"), "trans_start_times_checker", getattr(__import__("__main__"), "trans_start_times_checker") + 1) + return getattr(__import__("__main__"), "trans_start_times_checker") + +setattr(__import__("__main__"), "compute_start_times_checker", 0) +def get_compute_start_time_checker(): + setattr(__import__("__main__"), "compute_start_times_checker", getattr(__import__("__main__"), "compute_start_times_checker") + 1) + return getattr(__import__("__main__"), "compute_start_times_checker") + +setattr(__import__("__main__"), "trans_end_times_checker", 0) +def get_trans_end_time_checker(): + setattr(__import__("__main__"), "trans_end_times_checker", getattr(__import__("__main__"), "trans_end_times_checker") + 1) + return getattr(__import__("__main__"), "trans_end_times_checker") + +setattr(__import__("__main__"), "compute_end_times_checker", 0) +def get_compute_end_time_checker(): + setattr(__import__("__main__"), "compute_end_times_checker", getattr(__import__("__main__"), "compute_end_times_checker") + 1) + return getattr(__import__("__main__"), "compute_end_times_checker") + +def sorter(my_id, sorter_num, merger_num): + # print("python: sorter {} start".format(my_id), flush=True) + + # flag_register + for i in range(sorter_num): + slot_name = "sorter_flag_{}_{}".format(my_id, i) + setattr(__import__("__main__"), slot_name, pyas.buffer_register(slot_name, flag_size)) + encoded_data = "0".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + + print("compute{}start{}: {}".format(my_id, get_compute_start_time(), get_now())) + nums = [] + with open("sort_data_{}.txt".format(my_id), "r", errors='ignore') as f: + print("read{}start: {}".format(my_id, get_now())) + data = f.readlines() + print("read{}end: {}".format(my_id, get_now())) + for line in data: + nums.extend(map(int, line.split(','))) + nums.sort() + + if my_id == 0 and merger_num > 1: + pivot = [nums[(i + 1) * len(nums) // merger_num] for i in range(merger_num - 1)] + for i in range(sorter_num): + slot_name = "pivot_{}".format(i) + encoded_data = " ".join(map(str, pivot)).encode() + publish_data_buffer(slot_name, encoded_data) + # print("python: sorter {} pass {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) # important + + slot_name = "sorter_{}".format(my_id) + encoded_data = " ".join(map(str, nums)).encode() + publish_data_buffer(slot_name, encoded_data) + # print("python: sorter {} pass {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) # important + print("compute{}end{}: {}".format(my_id, get_compute_end_time(), get_now())) + + # print("python: sorter {} finished!".format(my_id), flush=True) + + # flag_finish + for i in range(sorter_num): + slot_name = "sorter_flag_{}_{}".format(my_id, i) + encoded_data = "1".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + +def spilter(my_id, sorter_num, merger_num): + # flag_access + while True: + flag = 0 + for i in range(sorter_num): + slot_name = "sorter_flag_{}_{}".format(i, my_id) + buffer = bytearray(flag_size) + pyas.access_buffer(slot_name, buffer) + flag += int(buffer[0]) - 48 + # print("python: spilter {} wait for flag, flag is {}".format(my_id, flag), flush=True) + if flag == sorter_num: + break + # print("python: spliter {} start".format(my_id), flush=True) + + # flag_register + for i in range(sorter_num): + slot_name = "spilter_flag_{}_{}".format(my_id, i) + setattr(__import__("__main__"), slot_name, pyas.buffer_register(slot_name, flag_size)) + encoded_data = "0".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + + print("compute{}start{}: {}".format(my_id, get_compute_start_time(), get_now())) + pivot = [] + if merger_num > 1: + slot_name = "pivot_{}".format(my_id) + print("trans{}start{}: {}".format(my_id, get_trans_start_time(), get_now())) + buffer = take_data_buffer(slot_name, buffer_size) + data = str(buffer, "utf-8").rstrip("\x00") + # print("python: spilter {} recv {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) + print("trans{}end{}: {}".format(my_id, get_trans_end_time(), get_now())) + pivot = data.split() + pivot = list(map(int, pivot)) + + slot_name = "sorter_{}".format(my_id) + print("trans{}start{}: {}".format(my_id, get_trans_start_time(), get_now())) + buffer = take_data_buffer(slot_name, buffer_size) + data = str(buffer, "utf-8").rstrip("\x00") + # print("python: spilter {} recv {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) + print("trans{}end{}: {}".format(my_id, get_trans_end_time(), get_now())) + nums = data.split() + nums = list(map(int, nums)) + + slot_data = [[] for _ in range(merger_num)] + for num in nums: + row = 0 + while row < len(pivot) and num >= pivot[row]: + row += 1 + slot_data[row].append(num) + + for i in range(merger_num): + slot_name = "merger_{}_{}".format(my_id, i) + encoded_data = " ".join(map(str, slot_data[i])).encode() + publish_data_buffer(slot_name, encoded_data) + # print("python: spilter {} pass {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) # important + print("compute{}end{}: {}".format(my_id, get_compute_end_time(), get_now())) + # print("python: spliter {} finished!".format(my_id), flush=True) + + # flag_finish + for i in range(sorter_num): + slot_name = "spilter_flag_{}_{}".format(my_id, i) + encoded_data = "1".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + +def merger(my_id, sorter_num, merger_num): + # flag_access + while True: + flag = 0 + for i in range(sorter_num): + slot_name = "spilter_flag_{}_{}".format(i, my_id) + buffer = bytearray(flag_size) + pyas.access_buffer(slot_name, buffer) + flag += int(buffer[0]) - 48 + # print("python: merger {} wait for flag, flag is {}".format(my_id, flag), flush=True) + if flag == sorter_num: + break + # print("python: merger {} start!".format(my_id), flush=True) + + # flag_register + for i in range(merger_num): + slot_name = "merger_flag_{}_{}".format(my_id, i) + setattr(__import__("__main__"), slot_name, pyas.buffer_register(slot_name, flag_size)) + encoded_data = "0".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + # print("python: merger {} pass {} buffer: {}".format(my_id, slot_name, getattr(__import__("__main__"), slot_name))) + + print("compute{}start{}: {}".format(my_id, get_compute_start_time(), get_now())) + sorted_partitions = [] + for i in range(sorter_num): + slot_name = "merger_{}_{}".format(i, my_id) + print("trans{}start{}: {}".format(my_id, get_trans_start_time(), get_now())) + buffer = take_data_buffer(slot_name, buffer_size) + _data = str(buffer, "utf-8").rstrip("\x00") + # print("python: merger {} recv {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) + print("trans{}end{}: {}".format(my_id, get_trans_end_time(), get_now())) + sorted_partitions.append(map(int, _data.split())) + + slot_name = "checker_{}".format(my_id) + final_data = list(heapq.merge(*sorted_partitions)) + + encoded_data = " ".join(map(str, final_data)).encode() + publish_data_buffer(slot_name, encoded_data) + # print("python: merger {} pass {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) # important + print("compute{}end{}: {}".format(my_id, get_compute_end_time(), get_now())) + # print("python: merger {} finished!".format(my_id), flush=True) + + # flag_finish + for i in range(merger_num): + slot_name = "merger_flag_{}_{}".format(my_id, i) + encoded_data = "1".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + # print("python: merger {} pass {} buffer: {}".format(my_id, slot_name, getattr(__import__("__main__"), slot_name))) + +def checker(my_id, sorter_num, merger_num): + # print("python: checker {} start".format(my_id), flush=True) + + if my_id != 0: + while True: + flag = 0 + slot_name = "checker_flag_{}".format(my_id) + buffer = bytearray(flag_size) + pyas.access_buffer(slot_name, buffer) + # print("python: checker {} recv {} buffer: {}".format(my_id, slot_name, buffer)) + flag += int(buffer[0]) - 48 + if flag == 1: + return + # flag_access + while True: + flag = 0 + for i in range(merger_num): + slot_name = "merger_flag_{}_{}".format(i, my_id) + buffer = bytearray(flag_size) + pyas.access_buffer(slot_name, buffer) + # print("python: checker {} recv {} buffer: {}".format(my_id, slot_name, buffer)) + flag += int(buffer[0]) - 48 + # print("python: checker {} wait for flag, flag is {}".format(my_id, flag), flush=True) + if flag == merger_num: + break + print("compute{}start{}:checker {}".format(my_id, get_compute_start_time_checker(), get_now())) + result = [] + for i in range(sorter_num): + slot_name = "checker_{}".format(i) + print("trans{}start{}:checker {}".format(my_id, get_trans_start_time_checker(), get_now())) + buffer = take_data_buffer(slot_name, buffer_size) + _data = str(buffer, "utf-8").rstrip("\x00") + # print("python: checker {} recv {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) + print("trans{}end{}:checker {}".format(my_id, get_trans_end_time_checker(), get_now())) + data = map(int, _data.split()) + result.extend(data) + + print("compute{}end{}:checker {}".format(my_id, get_compute_end_time_checker(), get_now())) + + # flag_finish + for i in range(1, merger_num): + slot_name = "checker_flag_{}".format(i) + setattr(__import__("__main__"), slot_name, pyas.buffer_register(slot_name, flag_size)) + encoded_data = "1".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + print("compute{}start{}:checker {}".format(my_id, get_compute_start_time_checker(), get_now())) + for i in range(1, len(result)): + if result[i] < result[i-1]: + print("python: checker Error: {} < {}".format(result[i], result[i-1])) + exit(1) + print("compute{}end{}:checker {}".format(my_id, get_compute_end_time_checker(), get_now())) + # print("python: checker {} finished!".format(my_id), flush=True) + +def main(): + my_id = int(sys.argv[1]) + sorter_num = int(sys.argv[2]) + merger_num = int(sys.argv[3]) + + sorter(my_id, sorter_num, merger_num) + spilter(my_id, sorter_num, merger_num) + merger(my_id, sorter_num, merger_num) + checker(my_id, sorter_num, merger_num) + +if __name__ == '__main__': + main() diff --git a/user/python_workloads/wasm_bench/transfer.py b/user/python_workloads/wasm_bench/transfer.py new file mode 100644 index 00000000..b89e4461 --- /dev/null +++ b/user/python_workloads/wasm_bench/transfer.py @@ -0,0 +1,35 @@ +import sys +import time + +import pyas + + +def parse_size() -> int: + if len(sys.argv) >= 2: + return int(sys.argv[1]) + return 4 * 1024 + + +def main() -> None: + buffer_size = parse_size() + slot_name = "slot_1" + producer = pyas.buffer_register(slot_name, buffer_size) + + start = time.time_ns() + take_buffer = getattr(pyas, "take_buffer", None) + if take_buffer is not None: + consumer = take_buffer(slot_name) + else: + consumer = bytearray(buffer_size) + pyas.access_buffer(slot_name, consumer) + end = time.time_ns() + + if consumer is not None and buffer_size > 0: + _ = consumer[0] + if producer is not None and buffer_size > 0: + producer[0] = 1 + + print(f"transfer cost: {end - start} ns size: {buffer_size}") + + +main() diff --git a/user/python_workloads/wasm_bench/wordcount.py b/user/python_workloads/wasm_bench/wordcount.py new file mode 100755 index 00000000..428c73de --- /dev/null +++ b/user/python_workloads/wasm_bench/wordcount.py @@ -0,0 +1,136 @@ +import pyas +import sys +import time + +buffer_size = 500000 +flag_size = 2 +length_size = 20 + +def get_now(): + return time.time() + +def take_data_buffer(slot_name, size): + take_buffer = getattr(pyas, "take_buffer", None) + if take_buffer is not None: + return take_buffer(slot_name) + get_buffer_len = getattr(pyas, "buffer_len", None) + if get_buffer_len is not None: + size = get_buffer_len(slot_name) + else: + length_buffer = bytearray(length_size) + pyas.access_buffer(slot_name + ".len", length_buffer) + size = int(bytes(length_buffer).rstrip(b"\x00")) + buffer = bytearray(size) + pyas.access_buffer(slot_name, buffer) + return buffer + +def publish_data_buffer(slot_name, encoded_data): + buffer = pyas.buffer_register(slot_name, len(encoded_data)) + buffer[:] = encoded_data + setattr(__import__("__main__"), slot_name, buffer) + if ( + getattr(pyas, "buffer_len", None) is None + and getattr(pyas, "take_buffer", None) is None + ): + length_slot = slot_name + ".len" + length_buffer = pyas.buffer_register(length_slot, length_size) + encoded_len = str(len(encoded_data)).encode() + length_buffer[:len(encoded_len)] = encoded_len + setattr(__import__("__main__"), length_slot, length_buffer) + +def mapper(my_id, reducer_num): + # print("python: mapper {} start, reducer_num is {}".format(my_id, reducer_num), flush=True) + for i in range(reducer_num): + slot_name = "flag_{}_{}".format(my_id, i) + setattr(__import__("__main__"), slot_name, pyas.buffer_register(slot_name, flag_size)) + encoded_data = "0".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + + print("compute{}start1: {}".format(my_id, get_now())) + input_file = "fake_data_{}.txt".format(my_id) + # print("python: Reading from {}".format(input_file), flush=True) + words_count = {} + with open(input_file, "r") as f: + print("read{}start1: {}".format(my_id, get_now())) + data = f.readlines() + print("read{}end1: {}".format(my_id, get_now())) + for line in data: + words = line.split() + words = [word.lower() for word in words] + for word in words: + if word in words_count: + words_count[word] += 1 + else: + words_count[word] = 1 + # print("python: mapper {} words counted: {}".format(my_id, len(words_count)), flush=True) + + slot_data = ["" for _ in range(reducer_num)] + for word, count in words_count.items(): + slot = hash(word) % reducer_num + slot_data[slot] += "{} {}\n".format(word, count) + + for i, data in enumerate(slot_data): + slot_name = "buffer_{}_{}".format(my_id, i) + encoded_data = data.encode() + publish_data_buffer(slot_name, encoded_data) + # print("python: mapper {} pass {} size: {}".format(my_id, slot_name, len(encoded_data)), flush=True) # important + print("compute{}end1: {}".format(my_id, get_now())) + for i in range(reducer_num): + slot_name = "flag_{}_{}".format(my_id, i) + encoded_data = "1".encode() + getattr(__import__("__main__"), slot_name)[:1] = encoded_data + + # print("python: mapper {} finished!".format(my_id), flush=True) + +def reducer(my_id, mapper_num): + while True: + flag = 0 + for i in range(mapper_num): + slot_name = "flag_{}_{}".format(i, my_id) + buffer = bytearray(flag_size) + pyas.access_buffer(slot_name, buffer) + flag += int(buffer[0]) - 48 + # print("python: reducer {} wait for flag, flag is {}".format(my_id, flag), flush=True) + if flag == mapper_num: + break + + # print("python: reducer {} start, mapper_num is {}".format(my_id, mapper_num), flush=True) + print("compute{}start2: {}".format(my_id, get_now())) + data = "" + print("trans{}start: {}".format(my_id, get_now())) + for i in range(mapper_num): + slot_name = "buffer_{}_{}".format(i, my_id) + buffer = take_data_buffer(slot_name, buffer_size) + slot_data = str(buffer, "utf-8").rstrip("\x00") + print("python: reducer {} recv {} size: {}".format(my_id, slot_name, len(slot_data)), flush=True) # important + data += slot_data + print("trans{}end: {}".format(my_id, get_now())) + counter = {} + for line in data.split("\n"): + if not line: + continue + word, count = line.split() + count = int(count) + if word in counter: + counter[word] += count + else: + counter[word] = count + output_file = "reducer_{}.txt".format(my_id) + print("read{}start2: {}".format(my_id, get_now())) + with open(output_file, "w") as f: + for word, count in counter.items(): + f.write("{} {}\n".format(word, count)) + print("read{}end2: {}".format(my_id, get_now())) + print("compute{}end2: {}".format(my_id, get_now())) + # print("python: reducer {} finished!".format(my_id), flush=True) + +def main(): + my_id = int(sys.argv[1]) + mapper_num = int(sys.argv[2]) + reducer_num = int(sys.argv[3]) + + mapper(my_id, reducer_num) + reducer(my_id, mapper_num) + +if __name__ == '__main__': + main() diff --git a/user/sorter/src/lib.rs b/user/sorter/src/lib.rs index 1cddc6cf..1d223e6b 100644 --- a/user/sorter/src/lib.rs +++ b/user/sorter/src/lib.rs @@ -42,10 +42,7 @@ struct Pivots { #[no_mangle] pub fn main() -> Result<()> { - println!( - "com_start1: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_start1"); let my_id = args::get("id").unwrap(); let sorter_num: usize = { let n = args::get("sorter_num").unwrap(); @@ -72,8 +69,8 @@ pub fn main() -> Result<()> { DataBuffer::from_buffer_slot(format!("input-part-{}", my_id)).unwrap(); let content = input.content.as_str(); - let mut buffer: DataBuffer = - DataBuffer::with_slot(format!("sorter-resp-part-{}", my_id)); + let output_slot = format!("sorter-resp-part-{}", my_id); + let mut buffer: DataBuffer = DataBuffer::with_slot(output_slot.clone()); let start = SystemTime::now(); for num in content.split(',') { @@ -96,6 +93,9 @@ pub fn main() -> Result<()> { let start = SystemTime::now(); buffer.array.sort(); + #[cfg(not(feature = "file-based"))] + as_std::agent::buffer_set_len(&output_slot, buffer.array.len() * core::mem::size_of::()) + .expect("failed to set sorter buffer length"); println!( "numbers array length is {}, sort cost {}ms", buffer.array.len(), @@ -120,6 +120,12 @@ pub fn main() -> Result<()> { let mut pivots_buffer: DataBuffer = DataBuffer::with_slot(format!("pivots-{}", i)); pivots_buffer.array = pivots.clone(); + #[cfg(not(feature = "file-based"))] + as_std::agent::buffer_set_len( + &format!("pivots-{}", i), + pivots_buffer.array.len() * core::mem::size_of::(), + ) + .expect("failed to set pivot buffer length"); } #[cfg(feature = "pkey_per_func")] { @@ -129,9 +135,11 @@ pub fn main() -> Result<()> { } } } - println!( - "com_end1: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_end1"); Ok(().into()) } + +fn print_timestamp(label: &str) { + let micros = SystemTime::now().duration_since(UNIX_EPOCH).as_micros(); + println!("{}: {}.{:06}", label, micros / 1_000_000, micros % 1_000_000); +} diff --git a/user/splitter/src/lib.rs b/user/splitter/src/lib.rs index ddd226ba..18cc0c06 100644 --- a/user/splitter/src/lib.rs +++ b/user/splitter/src/lib.rs @@ -30,10 +30,12 @@ struct Pivots { #[no_mangle] pub fn main() -> Result<()> { let my_id = args::get("id").unwrap(); + let start_micros = timestamp_micros(); println!( - "splitter id: {}, com_start2: {}", + "splitter id: {}, com_start2: {}.{:06}", my_id, - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 + start_micros / 1_000_000, + start_micros % 1_000_000 ); let numbers: DataBuffer = @@ -48,8 +50,8 @@ pub fn main() -> Result<()> { let partitions = split_numbers(&numbers.array, &pivots.array); for (idx, partition) in partitions.iter().enumerate() { - let mut part: DataBuffer = - DataBuffer::with_slot(format!("splitter-{}-resp-part-{}", my_id, idx)); + let output_slot = format!("splitter-{}-resp-part-{}", my_id, idx); + let mut part: DataBuffer = DataBuffer::with_slot(output_slot.clone()); #[cfg(feature = "pkey_per_func")] { part.array.resize(partition.len(), 0).unwrap(); @@ -61,6 +63,12 @@ pub fn main() -> Result<()> { { part.array = partition.clone(); } + #[cfg(not(feature = "file-based"))] + as_std::agent::buffer_set_len( + &output_slot, + part.array.len() * core::mem::size_of::(), + ) + .expect("failed to set splitter buffer length"); } println!( @@ -72,14 +80,20 @@ pub fn main() -> Result<()> { .map(|part| part.len()) .collect::>() ); - println!( - "com_end2: {}", - SystemTime::now().duration_since(UNIX_EPOCH).as_micros() as f64 / 1000000f64 - ); + print_timestamp("com_end2"); Ok(().into()) } +fn timestamp_micros() -> u128 { + SystemTime::now().duration_since(UNIX_EPOCH).as_micros() +} + +fn print_timestamp(label: &str) { + let micros = timestamp_micros(); + println!("{}: {}.{:06}", label, micros / 1_000_000, micros % 1_000_000); +} + fn split_numbers(numbers: &[u32], pivots: &[u32]) -> Vec> { let mut result = Vec::new(); let mut current_start = 0; diff --git a/user/wasmtime_checker/build.sh b/user/wasmtime_checker/build.sh index 03f89ec6..d7dbe8d0 100755 --- a/user/wasmtime_checker/build.sh +++ b/user/wasmtime_checker/build.sh @@ -1,5 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + # $CPP checker.cpp -o checker.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast -$CPP checker_ori.cpp -o checker.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast +"$WASI_CXX" checker_ori.cpp -o checker.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast # $CC checker.c -o checker.wasm wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n checker.wasm @@ -11,5 +19,5 @@ cargo build --target x86_64-unknown-none --release && cc \ -shared \ -o target/x86_64-unknown-none/release/libwasmtime_checker.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_checker/target/x86_64-unknown-none/release/libwasmtime_checker.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_checker.so - +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_checker.so" \ + "$REPO_ROOT/target/release/libwasmtime_checker.so" diff --git a/user/wasmtime_checker/checker.cwasm b/user/wasmtime_checker/checker.cwasm index 8b9daccf..c2f89e06 100644 Binary files a/user/wasmtime_checker/checker.cwasm and b/user/wasmtime_checker/checker.cwasm differ diff --git a/user/wasmtime_checker/checker.wasm b/user/wasmtime_checker/checker.wasm index cd9ea684..6e66d336 100755 Binary files a/user/wasmtime_checker/checker.wasm and b/user/wasmtime_checker/checker.wasm differ diff --git a/user/wasmtime_checker/checker_ori.cpp b/user/wasmtime_checker/checker_ori.cpp index 5babc84d..69eab515 100644 --- a/user/wasmtime_checker/checker_ori.cpp +++ b/user/wasmtime_checker/checker_ori.cpp @@ -3,7 +3,8 @@ #include #include -__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_len"))) long long buffer_len(void *slot_name, int name_size); // #define MAX_ARRAY_LENGTH 152221 // #define MAX_BUFFER_SIZE 1024*1024+152221 @@ -35,10 +36,13 @@ int main(int argc, char* argv[]) { sprintf(slot_name, "checker_%d", i); // printf("pivotname: %s\n", slot_name); char *buffer; - buffer = (char *)malloc(bufferSize * sizeof(char)); - memset(buffer, 0, bufferSize * sizeof(char)); - buffer[0] = '\0'; // 初始化为空字符串 - access_buffer(slot_name, strlen(slot_name), buffer, bufferSize); + long long length = buffer_len(slot_name, strlen(slot_name)); + if (length < 0) + return 3; + buffer = (char *)malloc((size_t)length); + memset(buffer, 0, (size_t)length); + buffer[0] = '\0'; // 初始化为空字符串 + access_buffer(slot_name, strlen(slot_name), buffer, (int)length); char *ptr = buffer; int num; while (sscanf(ptr, "%d", &num) == 1) { @@ -54,16 +58,14 @@ int main(int argc, char* argv[]) { } free(buffer); } - // printf("result_array: "); - // for (int i = 0; i < index-1; i++) { - // if (result[i] > result[i+1]) { - // printf("sort error!\n"); - // return 0; - // } - // printf("%d ", result[i]); - // } - // printf("%d\n", result[index-1]); - // printf("checker_%d all finished!\n", id); + for (int i = 0; i + 1 < index; i++) { + if (result[i] > result[i + 1]) { + fprintf(stderr, "sort error at index %d: %d > %d\n", + i, result[i], result[i + 1]); + return 4; + } + } + printf("wasmtime parallel sort checked %d values\n", index); get_time(); return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_cpython/Cargo.lock b/user/wasmtime_cpython/Cargo.lock index 87fcbc8f..2ddb7452 100644 --- a/user/wasmtime_cpython/Cargo.lock +++ b/user/wasmtime_cpython/Cargo.lock @@ -26,6 +26,39 @@ version = "1.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d5a26814d8dcb93b0e5a0ff3c6d80a8843bafb21b39e8e18a6f05471870e110" +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + [[package]] name = "autocfg" version = "1.4.0" @@ -206,7 +239,7 @@ dependencies = [ "proc-macro2", "quote", "rustc_version", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -457,37 +490,6 @@ dependencies = [ "autocfg", ] -[[package]] -name = "ms_hostcall" -version = "0.1.0" -dependencies = [ - "bitflags", - "derive_more", - "thiserror-no-std", -] - -[[package]] -name = "ms_std" -version = "0.1.0" -dependencies = [ - "cfg-if", - "heapless", - "linked_list_allocator", - "ms_hostcall", - "ms_std_proc_macro", - "spin", - "thiserror-no-std", -] - -[[package]] -name = "ms_std_proc_macro" -version = "0.1.0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.79", -] - [[package]] name = "naked-function" version = "0.1.5" @@ -506,7 +508,7 @@ checksum = "5b4123e70df5fe0bb370cff166ae453b9c5324a2cfc932c0f7e55498147a0475" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -613,12 +615,6 @@ dependencies = [ "windows-sys", ] -[[package]] -name = "ryu" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f" - [[package]] name = "sc" version = "0.2.7" @@ -639,34 +635,45 @@ checksum = "61697e0a1c7e512e84a621326239844a24d8207b4669b41bc18b32ea5cbf988b" [[package]] name = "serde" -version = "1.0.210" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8e3592472072e6e22e0a54d5904d9febf8508f65fb8552499a1abc7d1078c3a" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.210" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "243902eda00fad750862fc144cea25caca5e20d615af0a81bee94ca738f1df1f" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] name = "serde_json" -version = "1.0.128" +version = "1.0.150" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ff5456707a1de34e7e37f2a6fd3d3f808c318259cbd01ab6377795054b483d8" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" dependencies = [ "itoa", "memchr", - "ryu", "serde", + "serde_core", + "zmij", ] [[package]] @@ -744,9 +751,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.79" +version = "2.0.87" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89132cd0bf050864e1d38dc3bbc07a0eb8e7530af26344d3d2bbbef83499f590" +checksum = "25aa4ce346d03a6dcd68dd8b4010bcb74e54e62c90c573f394c46eae99aba32d" dependencies = [ "proc-macro2", "quote", @@ -776,7 +783,7 @@ checksum = "08904e7672f5eb876eaaf87e0ce17857500934f4981c4a0ab2b4aa98baac7fc3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -911,7 +918,7 @@ dependencies = [ "anyhow", "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", "wasmtime-component-util", "wasmtime-wit-bindgen", "wit-parser", @@ -1009,7 +1016,7 @@ checksum = "de5a9bc4f44ceeb168e9e8e3be4e0b4beb9095b468479663a9e24c667e36826f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -1045,9 +1052,9 @@ dependencies = [ name = "wasmtime_cpython" version = "0.1.0" dependencies = [ + "as_hostcall", + "as_std", "lazy_static", - "ms_hostcall", - "ms_std", "sjlj", "spin", "wasmtime_wasi_api", @@ -1057,11 +1064,11 @@ dependencies = [ name = "wasmtime_wasi_api" version = "0.1.0" dependencies = [ + "as_hostcall", + "as_std", + "as_std_proc_macro", "hashbrown 0.14.5", "lazy_static", - "ms_hostcall", - "ms_std", - "ms_std_proc_macro", "sjlj", "spin", "wasmtime", @@ -1192,5 +1199,11 @@ checksum = "fa4f8080344d4671fb4e831a13ad1e68092748387dfc4f55e356242fae12ce3e" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/wasmtime_cpython/src/lib.rs b/user/wasmtime_cpython/src/lib.rs index 97123d69..394e44d4 100755 --- a/user/wasmtime_cpython/src/lib.rs +++ b/user/wasmtime_cpython/src/lib.rs @@ -8,7 +8,7 @@ use spin::Mutex; use core::mem::forget; use as_hostcall::types::{OpenFlags, OpenMode}; -use as_std::{agent::FaaSFuncResult as Result, args, libos::libos, println, time::{SystemTime, UNIX_EPOCH},}; +use as_std::{agent::FaaSFuncResult as Result, args, libos::libos}; use wasmtime_wasi_api::{wasmtime, LibosCtx}; use wasmtime::Store; @@ -38,10 +38,13 @@ fn func_body(my_id: &str, pyfile_path: &str) -> Result<()> { wasmtime_wasi_api::JMP_BUF_MAP.lock().insert(my_id.to_string(), Arc::new(jmpbuf)); } - let wasi_args: Vec = Vec::from([ + let mut wasi_args: Vec = Vec::from([ "python.wasm".to_string(), pyfile_path.to_string(), ]); + if let Some(data_size) = args::get("data_size") { + wasi_args.push(data_size.to_string()); + } wasmtime_wasi_api::set_wasi_args(my_id, wasi_args); let _open_root = *MUST_OPEN_ROOT; diff --git a/user/wasmtime_cpython_func/src/lib.rs b/user/wasmtime_cpython_func/src/lib.rs index 768ff75d..31d5a372 100755 --- a/user/wasmtime_cpython_func/src/lib.rs +++ b/user/wasmtime_cpython_func/src/lib.rs @@ -24,7 +24,7 @@ lazy_static::lazy_static! { }; } -fn func_body(pyfile_path: &str, func_num: u64) -> Result<()> { +fn func_body(pyfile_path: &str, func_num: u64, data_size: usize) -> Result<()> { let my_map_id = &format!("func_{}", func_num); let mut jmpbuf = sjlj::JumpBuf::new(); @@ -43,6 +43,7 @@ fn func_body(pyfile_path: &str, func_num: u64) -> Result<()> { let wasi_args: Vec = Vec::from([ "python.wasm".to_string(), pyfile_path.to_string(), + data_size.to_string(), func_num.to_string(), ]); wasmtime_wasi_api::set_wasi_args(my_map_id, wasi_args); @@ -77,6 +78,10 @@ pub fn main() -> Result<()> { .expect("missing arg func_num") .parse() .unwrap_or_else(|_| panic!("bad arg, func_num={}", args::get("func_num").unwrap())); + let data_size: usize = args::get("data_size") + .unwrap_or("268435456") + .parse() + .expect("bad data_size"); - func_body(pyfile_path, func_num) + func_body(pyfile_path, func_num, data_size) } diff --git a/user/wasmtime_longchain/build.sh b/user/wasmtime_longchain/build.sh new file mode 100755 index 00000000..4c1eca32 --- /dev/null +++ b/user/wasmtime_longchain/build.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail + +WASI_CC="${WASI_CC:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" + +cd "$SCRIPT_DIR" +"$WASI_CC" func.c -o func.wasm -O2 +wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n func.wasm diff --git a/user/wasmtime_longchain/func.c b/user/wasmtime_longchain/func.c index 43bd228f..dc454b36 100644 --- a/user/wasmtime_longchain/func.c +++ b/user/wasmtime_longchain/func.c @@ -3,31 +3,49 @@ #include #include -__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); -__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_len"))) long long buffer_len(void *slot_name, int name_size); int main(int argc, char* argv[]) { - int id = atoi(argv[1]); - int func_num = atoi(argv[2]); - printf("func.c recieve: id: %d, func_num: %d\n", id, func_num); - - char slot_name[20]; - int bufferSize = 2; - char *buffer = (char *)malloc(bufferSize * sizeof(char)); - - if (func_num == 0) { - sprintf(slot_name, "buffer_%d_%d", func_num, id); - buffer[0] = '0'; - buffer[1] = '\0'; + int id = atoi(argv[1]); + int func_num = atoi(argv[2]); + int chain_len = argc > 3 ? atoi(argv[3]) : 10; + int bufferSize = argc > 4 ? atoi(argv[4]) : 2; + if (bufferSize < 2 || func_num < 0 || func_num >= chain_len) + return 2; + printf("func.c recieve: id: %d, func_num: %d\n", id, func_num); + + char slot_name[20]; + char *buffer; + + if (func_num == 0) { + buffer = (char *)malloc(bufferSize); + if (buffer == NULL) + return 3; + sprintf(slot_name, "buffer_%d_%d", func_num, id); + buffer[0] = '0'; + buffer[bufferSize - 1] = '\0'; buffer_register(slot_name, strlen(slot_name), buffer, bufferSize); - } else { - sprintf(slot_name, "buffer_%d_%d", func_num-1, id); - access_buffer(slot_name, strlen(slot_name), buffer, bufferSize); - sprintf(slot_name, "buffer_%d_%d", func_num, id); + } else { + sprintf(slot_name, "buffer_%d_%d", func_num-1, id); + long long length = buffer_len(slot_name, strlen(slot_name)); + if (length < 0) + return 3; + bufferSize = (int)length; + buffer = (char *)malloc(bufferSize); + if (buffer == NULL) + return 4; + access_buffer(slot_name, strlen(slot_name), buffer, bufferSize); + if (func_num + 1 == chain_len) { + free(buffer); + return 0; + } + sprintf(slot_name, "buffer_%d_%d", func_num, id); buffer[0] += 1; buffer_register(slot_name, strlen(slot_name), buffer, bufferSize); } free(buffer); printf("func_%d finished!\n", func_num); return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_longchain/func.cwasm b/user/wasmtime_longchain/func.cwasm index a53d7c9c..81ded3e4 100644 Binary files a/user/wasmtime_longchain/func.cwasm and b/user/wasmtime_longchain/func.cwasm differ diff --git a/user/wasmtime_longchain/func.wasm b/user/wasmtime_longchain/func.wasm index 5dde85eb..5277ba9b 100755 Binary files a/user/wasmtime_longchain/func.wasm and b/user/wasmtime_longchain/func.wasm differ diff --git a/user/wasmtime_longchain/src/lib.rs b/user/wasmtime_longchain/src/lib.rs index 786f34d8..33ed11de 100644 --- a/user/wasmtime_longchain/src/lib.rs +++ b/user/wasmtime_longchain/src/lib.rs @@ -37,15 +37,17 @@ lazy_static::lazy_static! { }; } -fn func_body(my_id: &str, func_num: u64) -> Result<()> { +fn func_body(my_id: &str, func_num: u64, chain_len: u64, data_size: usize) -> Result<()> { // #[cfg(feature = "log")] println!("rust: my_id: {:?}, func_num: {:?}", my_id, func_num); - let wasi_args: Vec = Vec::from([ - "fake system path!".to_string(), - my_id.to_string(), - func_num.to_string(), - ]); + let wasi_args: Vec = Vec::from([ + "fake system path!".to_string(), + my_id.to_string(), + func_num.to_string(), + chain_len.to_string(), + data_size.to_string(), + ]); wasmtime_wasi_api::set_wasi_args(my_id, wasi_args); let _open_root = *MUST_OPEN_ROOT; @@ -61,10 +63,12 @@ fn func_body(my_id: &str, func_num: u64) -> Result<()> { .get_typed_func::<(), ()>(&mut store, "_start") .map_err(|e| e.to_string())?; - main.call(store, ()).map_err(|e| e.to_string())?; + main.call(&mut store, ()).map_err(|e| e.to_string())?; + // Published buffers point into this instance's linear memory. + forget(store); - if func_num == 9 { + if func_num + 1 == chain_len { let data = DataBuffer::::from_buffer_slot("Conference".to_owned()); if let Some(buffer) = data { let dur = buffer.current_time.elapsed(); @@ -79,10 +83,18 @@ fn func_body(my_id: &str, func_num: u64) -> Result<()> { #[no_mangle] pub fn main() -> Result<()> { let my_id = args::get("id").unwrap(); - let func_num: u64 = args::get("func_num") + let func_num: u64 = args::get("func_num") .expect("missing arg func_num") - .parse() - .unwrap_or_else(|_| panic!("bad arg, func_num={}", args::get("func_num").unwrap())); + .parse() + .unwrap_or_else(|_| panic!("bad arg, func_num={}", args::get("func_num").unwrap())); + let chain_len: u64 = args::get("chain_len") + .unwrap_or("10") + .parse() + .expect("bad chain_len"); + let data_size: usize = args::get("data_size") + .unwrap_or("2") + .parse() + .expect("bad data_size"); if func_num == 0 { let mut d = DataBuffer::::with_slot("Conference".to_owned()); d.current_time = SystemTime::now(); @@ -90,5 +102,5 @@ pub fn main() -> Result<()> { // println!("start_time: {:?}", start_time); } - func_body(my_id, func_num) -} + func_body(my_id, func_num, chain_len, data_size) +} diff --git a/user/wasmtime_mapper/build.sh b/user/wasmtime_mapper/build.sh index 3e4d4025..371b7bf1 100755 --- a/user/wasmtime_mapper/build.sh +++ b/user/wasmtime_mapper/build.sh @@ -1,4 +1,12 @@ -$CPP mapper_new.cpp -o mapper.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + +"$WASI_CXX" mapper_new.cpp -o mapper.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast # $CC mapper.c -o mapper.wasm -O3 wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n mapper.wasm @@ -11,7 +19,8 @@ cargo build --target x86_64-unknown-none --release && cc \ -shared \ -o target/x86_64-unknown-none/release/libwasmtime_mapper.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_mapper/target/x86_64-unknown-none/release/libwasmtime_mapper.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_mapper.so +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_mapper.so" \ + "$REPO_ROOT/target/release/libwasmtime_mapper.so" # cargo build --target x86_64-unknown-none && cc \ # -Wl,--gc-sections -nostdlib \ diff --git a/user/wasmtime_mapper/mapper.cwasm b/user/wasmtime_mapper/mapper.cwasm index aac27a41..f1b7ede1 100644 Binary files a/user/wasmtime_mapper/mapper.cwasm and b/user/wasmtime_mapper/mapper.cwasm differ diff --git a/user/wasmtime_mapper/mapper.wasm b/user/wasmtime_mapper/mapper.wasm index 502d4cb7..d219631a 100755 Binary files a/user/wasmtime_mapper/mapper.wasm and b/user/wasmtime_mapper/mapper.wasm differ diff --git a/user/wasmtime_mapper/mapper_new.cpp b/user/wasmtime_mapper/mapper_new.cpp index bdd827df..15238d9e 100644 --- a/user/wasmtime_mapper/mapper_new.cpp +++ b/user/wasmtime_mapper/mapper_new.cpp @@ -13,9 +13,7 @@ using namespace std; #define MAX_SLOT_NUM 10 #define MAX_WORDS 18000000 -#define MAX_BUFFER_SIZE 500000 - -string words[MAX_WORDS]; +string words[MAX_WORDS]; int counts[MAX_WORDS]; __attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); @@ -51,20 +49,17 @@ int main(int argc, char* argv[]) { cout << "mapper_" << id << " read success!" << endl; - string buffer[MAX_SLOT_NUM]; - string slot_name[MAX_SLOT_NUM]; - - for (auto& pair : word_map) { - int partition_index = hash{}(pair.first) % reducer_num; - slot_name[partition_index] = "buffer_" + to_string(partition_index) + "_" + to_string(id); - buffer[partition_index].append(pair.first + " " + to_string(pair.second) + "\n"); - } - - for (int i = 0; i < reducer_num; i++) { - buffer[i].resize(MAX_BUFFER_SIZE, '\0'); - } - - int word_index = word_map.size(); + string buffer[MAX_SLOT_NUM]; + string slot_name[MAX_SLOT_NUM]; + for (int i = 0; i < reducer_num; ++i) + slot_name[i] = "buffer_" + to_string(i) + "_" + to_string(id); + + for (auto& pair : word_map) { + int partition_index = hash{}(pair.first) % reducer_num; + buffer[partition_index].append(pair.first + " " + to_string(pair.second) + "\n"); + } + + int word_index = word_map.size(); cout << "mapper_" << id << " solved " << word_index << " words!" << endl; for (int i = 0; i < reducer_num; i++) { @@ -74,4 +69,4 @@ int main(int argc, char* argv[]) { cout << "mapper_" << id << " write finished!" << endl; return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_merger/build.sh b/user/wasmtime_merger/build.sh index 496e87ec..9db717ff 100755 --- a/user/wasmtime_merger/build.sh +++ b/user/wasmtime_merger/build.sh @@ -1,5 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + # $CPP merger.cpp -o merger.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast -$CPP merger_ori.cpp -o merger.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast +"$WASI_CXX" merger_ori.cpp -o merger.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast # $CC merger.c -o merger.wasm wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n merger.wasm @@ -11,5 +19,5 @@ cargo build --target x86_64-unknown-none --release && cc \ -shared \ -o target/x86_64-unknown-none/release/libwasmtime_merger.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_merger/target/x86_64-unknown-none/release/libwasmtime_merger.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_merger.so - +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_merger.so" \ + "$REPO_ROOT/target/release/libwasmtime_merger.so" diff --git a/user/wasmtime_merger/merger.cwasm b/user/wasmtime_merger/merger.cwasm index 34a956b9..45a6be43 100644 Binary files a/user/wasmtime_merger/merger.cwasm and b/user/wasmtime_merger/merger.cwasm differ diff --git a/user/wasmtime_merger/merger.wasm b/user/wasmtime_merger/merger.wasm index aadad419..b587f2a1 100755 Binary files a/user/wasmtime_merger/merger.wasm and b/user/wasmtime_merger/merger.wasm differ diff --git a/user/wasmtime_merger/merger_ori.cpp b/user/wasmtime_merger/merger_ori.cpp index 575b4666..1abbeb56 100644 --- a/user/wasmtime_merger/merger_ori.cpp +++ b/user/wasmtime_merger/merger_ori.cpp @@ -3,8 +3,9 @@ #include #include -__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); -__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_len"))) long long buffer_len(void *slot_name, int name_size); // #define MAX_ARRAY_LENGTH 152221 // #define MAX_BUFFER_SIZE 1024*1024+152221 @@ -81,10 +82,13 @@ int main(int argc, char* argv[]) { char slot_name[20]; sprintf(slot_name, "merger_%d_%d", i, id); char *buffer; - buffer = (char *)malloc(MAX_BUFFER_SIZE * sizeof(char)); - memset(buffer, 0, MAX_BUFFER_SIZE * sizeof(char)); - buffer[0] = '\0'; // 初始化为空字符串 - access_buffer(slot_name, strlen(slot_name), buffer, MAX_BUFFER_SIZE); + long long length = buffer_len(slot_name, strlen(slot_name)); + if (length < 0) + return 3; + buffer = (char *)malloc((size_t)length); + memset(buffer, 0, (size_t)length); + buffer[0] = '\0'; // 初始化为空字符串 + access_buffer(slot_name, strlen(slot_name), buffer, (int)length); char *ptr = buffer; int num; while (sscanf(ptr, "%d", &num) == 1) { @@ -144,11 +148,12 @@ int main(int argc, char* argv[]) { // rigister mergered array - char slot_name[20]; - sprintf(slot_name, "checker_%d", id); - char *buffer; - buffer = (char *)malloc(MAX_BUFFER_SIZE * sizeof(char)); - memset(buffer, 0, MAX_BUFFER_SIZE * sizeof(char)); + char slot_name[20]; + sprintf(slot_name, "checker_%d", id); + size_t output_size = 1; + for (int i = 0; i < resultIndex; i++) + output_size += (size_t)snprintf(NULL, 0, "%d ", result[i]); + char *buffer = (char *)malloc(output_size); buffer[0] = '\0'; // 初始化为空字符串 char *ptr = buffer; for (int i = 0; i < resultIndex; i++) { @@ -162,11 +167,11 @@ int main(int argc, char* argv[]) { // buffer[strlen(buffer) - 1] = '\0'; *ptr++ = '\0'; printf("result_index: %d\n", resultIndex); - buffer_register(slot_name, strlen(slot_name), buffer, MAX_BUFFER_SIZE); + buffer_register(slot_name, strlen(slot_name), buffer, ptr - buffer); free(buffer); // printf("merger_%d all finished!\n", id); get_time(); return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_reducer/build.sh b/user/wasmtime_reducer/build.sh index aed52545..4a8aff85 100755 --- a/user/wasmtime_reducer/build.sh +++ b/user/wasmtime_reducer/build.sh @@ -1,5 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + # $CC reducer.c -o reducer.wasm -$CPP reducer_new.cpp -o reducer.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast +"$WASI_CXX" reducer_new.cpp -o reducer.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n reducer.wasm @@ -11,7 +19,8 @@ cargo build --target x86_64-unknown-none --release && cc \ -shared \ -o target/x86_64-unknown-none/release/libwasmtime_reducer.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_reducer/target/x86_64-unknown-none/release/libwasmtime_reducer.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_reducer.so +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_reducer.so" \ + "$REPO_ROOT/target/release/libwasmtime_reducer.so" # cargo build --target x86_64-unknown-none && cc \ diff --git a/user/wasmtime_reducer/reducer.cwasm b/user/wasmtime_reducer/reducer.cwasm index da2d9a9d..e700e2ef 100644 Binary files a/user/wasmtime_reducer/reducer.cwasm and b/user/wasmtime_reducer/reducer.cwasm differ diff --git a/user/wasmtime_reducer/reducer.wasm b/user/wasmtime_reducer/reducer.wasm index c36bcb3b..7032f592 100755 Binary files a/user/wasmtime_reducer/reducer.wasm and b/user/wasmtime_reducer/reducer.wasm differ diff --git a/user/wasmtime_reducer/reducer_new.cpp b/user/wasmtime_reducer/reducer_new.cpp index 11e0e851..d66f48fb 100644 --- a/user/wasmtime_reducer/reducer_new.cpp +++ b/user/wasmtime_reducer/reducer_new.cpp @@ -10,9 +10,8 @@ using namespace std; #define MAX_SLOT_NUM 10 #define MAX_WORDS 18000000 -#define MAX_BUFFER_SIZE 500000 - __attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_len"))) long long buffer_len(void *slot_name, int name_size); void get_time() { timeval tv{}; @@ -33,8 +32,11 @@ int main(int argc, char* argv[]) { string buffer[MAX_SLOT_NUM]; for (int i = 0; i < slot_num; ++i) { - slot_name[i] = "buffer_" + to_string(i) + "_" + to_string(id); - buffer[i] = string(MAX_BUFFER_SIZE, 0); + slot_name[i] = "buffer_" + to_string(id) + "_" + to_string(i); + long long size = buffer_len((void*)slot_name[i].c_str(), slot_name[i].length()); + if (size < 0) + return 3; + buffer[i].resize((size_t)size); access_buffer((void*)slot_name[i].c_str(), slot_name[i].length(), (void*)buffer[i].c_str(), buffer[i].size()); } @@ -61,4 +63,4 @@ int main(int argc, char* argv[]) { cout << "reducer_" << id << " finished!" << endl; get_time(); return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_sorter/build.sh b/user/wasmtime_sorter/build.sh index 5dc84ed5..496139f6 100755 --- a/user/wasmtime_sorter/build.sh +++ b/user/wasmtime_sorter/build.sh @@ -1,5 +1,13 @@ -# $CPP sorter.cpp -o sorter.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast -$CPP sorter_ori.cpp -o sorter.wasm -fno-exceptions -fno-rtti -ffast-math -fomit-frame-pointer -Ofast #-funroll-loops +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + +# Keep the sorter optimization level aligned with the native musl C build. +"$WASI_CXX" sorter_ori.cpp -o sorter.wasm -fno-exceptions -fno-rtti -fomit-frame-pointer -O2 # $CC sorter.c -o sorter.wasm wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n sorter.wasm @@ -12,7 +20,8 @@ cargo build --target x86_64-unknown-none --release && cc \ -o target/x86_64-unknown-none/release/libwasmtime_sorter.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_sorter/target/x86_64-unknown-none/release/libwasmtime_sorter.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_sorter.so +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_sorter.so" \ + "$REPO_ROOT/target/release/libwasmtime_sorter.so" diff --git a/user/wasmtime_sorter/sorter.cwasm b/user/wasmtime_sorter/sorter.cwasm index 2c4b1374..1c501dc4 100644 Binary files a/user/wasmtime_sorter/sorter.cwasm and b/user/wasmtime_sorter/sorter.cwasm differ diff --git a/user/wasmtime_sorter/sorter.wasm b/user/wasmtime_sorter/sorter.wasm index 23deaa7e..89821988 100755 Binary files a/user/wasmtime_sorter/sorter.wasm and b/user/wasmtime_sorter/sorter.wasm differ diff --git a/user/wasmtime_sorter/sorter_ori.cpp b/user/wasmtime_sorter/sorter_ori.cpp index 330e7698..005d3110 100644 --- a/user/wasmtime_sorter/sorter_ori.cpp +++ b/user/wasmtime_sorter/sorter_ori.cpp @@ -1,11 +1,11 @@ #include #include #include +#include #include #include #include #include -#include using namespace std; @@ -20,22 +20,22 @@ __attribute__((import_module("env"), import_name("buffer_register"))) void buffe #define MAX_ARRAY_LENGTH 3805441 #define MAX_BUFFER_SIZE 25*1024*1024+3805441 -// 比较函数，用于 qsort int compare(const void *a, const void *b) { - return (*(int *)a - *(int *)b); // 升序排序 + int left = *(const int *)a; + int right = *(const int *)b; + return (left > right) - (left < right); } -char nc(FILE *stream) { - static char buf[1<<25], *p1 = buf, *p2 = buf; - return p1 == p2 && (p2 = (p1 = buf) + fread(buf, 1, 1 << 25, stream), p1 == p2) ? EOF : *p1 ++; -} - -int readfile(FILE *stream) { - int x = 0, ch = nc(stream); - for (; ch < '0' || ch > '9'; ch = nc(stream)); - for (; ch >= '0' && ch <= '9'; ch = nc(stream)) - x = (x << 1) + (x << 3) + (ch ^ 48); - return x; +int nc(FILE *stream) { + static unsigned char buf[1 << 25], *current = buf, *end = buf; + if (current == end) { + size_t size = fread(buf, 1, sizeof(buf), stream); + current = buf; + end = buf + size; + if (current == end) + return EOF; + } + return *current++; } void get_time() { @@ -72,15 +72,26 @@ int main(int argc, char* argv[]) { // time(&now); // printf("%ld read start\n", now); // write(1, "read start\n", sizeof("read start\n")); - char number[10]; get_time(); - // while (array[index++] = readfile(file)); - int ch = nc(file), x = 0; - for (; ch != EOF; ch = nc(file)) { - if (ch == ' ' || ch == '\n') arrays.push_back(x), x = 0; - else x = x * 10 + ch - '0'; + int ch, value = 0; + bool in_number = false; + while ((ch = nc(file)) != EOF) { + if (ch >= '0' && ch <= '9') { + if (value > (INT_MAX - (ch - '0')) / 10) { + fprintf(stderr, "integer overflow in %s\n", input_file); + fclose(file); + return 4; + } + value = value * 10 + ch - '0'; + in_number = true; + } else if (in_number) { + arrays.push_back(value); + value = 0; + in_number = false; + } } - arrays.push_back(x); + if (in_number) + arrays.push_back(value); index = arrays.size(); printf("sorter_index: %d\n", index); get_time(); @@ -96,8 +107,7 @@ int main(int argc, char* argv[]) { printf("index: %d\n", index); fclose(file); get_time(); - // qsort(arrays.data(), index, sizeof(int), compare); - sort(arrays.data(), arrays.data()+index); + qsort(arrays.data(), index, sizeof(*arrays.data()), compare); get_time(); // printf("sorter_%d sort finished!\n", id); @@ -107,10 +117,11 @@ int main(int argc, char* argv[]) { int idx = (i+1) * index / merger_num; pivot[i] = arrays[idx]; } - char *buffer; - buffer = (char *)malloc(MAX_BUFFER_SIZE * sizeof(char)); - memset(buffer, 0, MAX_BUFFER_SIZE * sizeof(char)); - buffer[0] = '\0'; + size_t pivot_size = 1; + for (int i = 0; i < merger_num - 1; i++) + pivot_size += (size_t)snprintf(NULL, 0, "%d ", pivot[i]); + char *buffer = (char *)malloc(pivot_size); + buffer[0] = '\0'; for (int i = 0; i < merger_num-1; i++) { char temp[12]; snprintf(temp, sizeof(temp), "%d ", pivot[i]); @@ -121,7 +132,7 @@ int main(int argc, char* argv[]) { char slot_name[20]; sprintf(slot_name, "pivot_%d", k); // printf("pivotname: %s\n", slot_name); - buffer_register(slot_name, strlen(slot_name), buffer, MAX_BUFFER_SIZE); + buffer_register(slot_name, strlen(slot_name), buffer, strlen(buffer) + 1); } // free(buffer); } @@ -131,8 +142,10 @@ int main(int argc, char* argv[]) { // write(1, "alloc start\n", sizeof("alloc start\n")); char slot_name[20]; sprintf(slot_name, "sorter_%d", id); - char *buffer; - buffer = (char *)malloc(MAX_BUFFER_SIZE * sizeof(char)); + size_t output_size = 1; + for (int value : arrays) + output_size += (size_t)snprintf(NULL, 0, "%d ", value); + char *buffer = (char *)malloc(output_size); // time(&now); // printf("%ld alloc finished\n", now); // write(1, "alloc finished\n", sizeof("alloc finished\n")); @@ -155,7 +168,7 @@ int main(int argc, char* argv[]) { // buffer[strlen(buffer) - 1] = '\0'; // write(1, "buffer make finished\n", sizeof("buffer make finished\n")); get_time(); - buffer_register(slot_name, strlen(slot_name), buffer, MAX_BUFFER_SIZE); + buffer_register(slot_name, strlen(slot_name), buffer, ptr - buffer); get_time(); // write(1, "buffer register finished\n", sizeof("buffer register finished\n")); // free(buffer); @@ -163,4 +176,4 @@ int main(int argc, char* argv[]) { // write(1, "all finished\n", sizeof("all finished\n")); // get_time(); return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_spliter/build.sh b/user/wasmtime_spliter/build.sh index 2dca26a3..b56b0aa0 100755 --- a/user/wasmtime_spliter/build.sh +++ b/user/wasmtime_spliter/build.sh @@ -1,5 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + # $CPP spliter.cpp -o spliter.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast -$CPP spliter_ori.cpp -o spliter.wasm # -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast +"$WASI_CXX" spliter_ori.cpp -o spliter.wasm # -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast # $CC spliter.c -o spliter.wasm wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n spliter.wasm @@ -12,5 +20,5 @@ cargo build --target x86_64-unknown-none --release && cc \ -o target/x86_64-unknown-none/release/libwasmtime_spliter.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_spliter/target/x86_64-unknown-none/release/libwasmtime_spliter.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_spliter.so - +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_spliter.so" \ + "$REPO_ROOT/target/release/libwasmtime_spliter.so" diff --git a/user/wasmtime_spliter/spliter.cwasm b/user/wasmtime_spliter/spliter.cwasm index 882b840f..3aed0eb6 100644 Binary files a/user/wasmtime_spliter/spliter.cwasm and b/user/wasmtime_spliter/spliter.cwasm differ diff --git a/user/wasmtime_spliter/spliter.wasm b/user/wasmtime_spliter/spliter.wasm index 35fa9e4c..a967dd4e 100755 Binary files a/user/wasmtime_spliter/spliter.wasm and b/user/wasmtime_spliter/spliter.wasm differ diff --git a/user/wasmtime_spliter/spliter_ori.cpp b/user/wasmtime_spliter/spliter_ori.cpp index c8e5ecd9..f2c3c4ed 100644 --- a/user/wasmtime_spliter/spliter_ori.cpp +++ b/user/wasmtime_spliter/spliter_ori.cpp @@ -3,8 +3,9 @@ #include #include -__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); -__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); +__attribute__((import_module("env"), import_name("buffer_len"))) long long buffer_len(void *slot_name, int name_size); // #define MAX_ARRAY_LENGTH 152221 // #define MAX_BUFFER_SIZE 1024*1024+152221 @@ -41,10 +42,13 @@ int main(int argc, char* argv[]) { if (merger_num > 1) { sprintf(slot_name, "pivot_%d", id); char *pivot_buffer; - pivot_buffer = (char *)malloc(bufferSize * sizeof(char)); - memset(pivot_buffer, 0, bufferSize * sizeof(char)); - pivot_buffer[0] = '\0'; // 初始化为空字符串 - access_buffer(slot_name, strlen(slot_name), pivot_buffer, bufferSize); + long long length = buffer_len(slot_name, strlen(slot_name)); + if (length < 0) + return 3; + pivot_buffer = (char *)malloc((size_t)length); + memset(pivot_buffer, 0, (size_t)length); + pivot_buffer[0] = '\0'; // 初始化为空字符串 + access_buffer(slot_name, strlen(slot_name), pivot_buffer, (int)length); ptr = pivot_buffer; // printf("pivot_buffer: %s", pivot_buffer); while (sscanf(ptr, "%d", &num) == 1) { @@ -65,11 +69,14 @@ int main(int argc, char* argv[]) { // access sorter buffer sprintf(slot_name, "sorter_%d", id); char *sorter_buffer; - sorter_buffer = (char *)malloc(bufferSize * sizeof(char)); - memset(sorter_buffer, 0, bufferSize * sizeof(char)); + long long sorter_length = buffer_len(slot_name, strlen(slot_name)); + if (sorter_length < 0) + return 4; + sorter_buffer = (char *)malloc((size_t)sorter_length); + memset(sorter_buffer, 0, (size_t)sorter_length); sorter_buffer[0] = '\0'; // 初始化为空字符串 get_time(); - access_buffer(slot_name, strlen(slot_name), sorter_buffer, bufferSize); + access_buffer(slot_name, strlen(slot_name), sorter_buffer, (int)sorter_length); get_time(); int sorter_index = 0; ptr = sorter_buffer; @@ -115,19 +122,18 @@ int main(int argc, char* argv[]) { printf("merger_index_before_register: %d\n", array_index[0]); get_time(); get_time(); - printf("array0: %d :%d\n", (void*)(array_index), *array_index); - for (int i = 0; i < merger_num; i++) { - printf("i(%d) = %d array0: %d :%d\n", (void*)&i, i, (void*)(array_index), *array_index); - char slot_name[20]; - sprintf(slot_name, "merger_%d_%d", id, i); - char *merger_buffer = (char *)malloc(bufferSize * sizeof(char)); + for (int i = 0; i < merger_num; i++) { + char slot_name[20]; + sprintf(slot_name, "merger_%d_%d", id, i); + size_t output_size = 1; + for (int j = 0; j < array_index[i]; j++) + output_size += (size_t)snprintf(NULL, 0, "%d ", array[i][j]); + char *merger_buffer = (char *)malloc(output_size); // memset(merger_buffer, 0, bufferSize * sizeof(char)); merger_buffer[0] = '\0'; // 初始化为空字符串 char *merger_ptr = merger_buffer; - printf("array0: %d :%d\n", (void*)(array_index), *array_index); - printf("i: %d; array_index_before_for: %d\n", i, array_index[i]); - printf("%d: %d\n", (void*)(array_index+1), *(array_index+1)); - printf("array_index[0]: %d\n", array_index[0]); + printf("i: %d; array_index_before_for: %d\n", i, array_index[i]); + printf("array_index[0]: %d\n", array_index[0]); for (int j = 0; j < array_index[i]; j++) { char temp[12]; // 临时缓冲区，注意要足够大以容纳最大整数和一个空格 snprintf(temp, sizeof(temp), "%d ", array[i][j]); // 将整数转换为字符串，并加上空格 @@ -139,7 +145,7 @@ int main(int argc, char* argv[]) { // 去掉最后一个多余的空格 // buffer[strlen(buffer) - 1] = '\0'; *merger_ptr++ = '\0'; - buffer_register(slot_name, strlen(slot_name), merger_buffer, MAX_BUFFER_SIZE); + buffer_register(slot_name, strlen(slot_name), merger_buffer, merger_ptr - merger_buffer); // buffer_register(slot_name, strlen(slot_name), buffer, bufferSize); // free(buffer); // free(array[i]); @@ -149,4 +155,4 @@ int main(int argc, char* argv[]) { // printf("spliter_%d all finished!\n", id); // get_time(); return 0; -} \ No newline at end of file +} diff --git a/user/wasmtime_trans_data/Cargo.lock b/user/wasmtime_trans_data/Cargo.lock index b1b79911..3c4a2bc7 100644 --- a/user/wasmtime_trans_data/Cargo.lock +++ b/user/wasmtime_trans_data/Cargo.lock @@ -26,6 +26,39 @@ version = "1.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d5a26814d8dcb93b0e5a0ff3c6d80a8843bafb21b39e8e18a6f05471870e110" +[[package]] +name = "as_hostcall" +version = "0.1.0" +dependencies = [ + "bitflags", + "derive_more", + "thiserror-no-std", +] + +[[package]] +name = "as_std" +version = "0.1.0" +dependencies = [ + "as_hostcall", + "as_std_proc_macro", + "cfg-if", + "heapless", + "linked_list_allocator", + "serde", + "serde_json", + "spin", + "thiserror-no-std", +] + +[[package]] +name = "as_std_proc_macro" +version = "0.1.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + [[package]] name = "autocfg" version = "1.4.0" @@ -206,7 +239,7 @@ dependencies = [ "proc-macro2", "quote", "rustc_version", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -457,37 +490,6 @@ dependencies = [ "autocfg", ] -[[package]] -name = "ms_hostcall" -version = "0.1.0" -dependencies = [ - "bitflags", - "derive_more", - "thiserror-no-std", -] - -[[package]] -name = "ms_std" -version = "0.1.0" -dependencies = [ - "cfg-if", - "heapless", - "linked_list_allocator", - "ms_hostcall", - "ms_std_proc_macro", - "spin", - "thiserror-no-std", -] - -[[package]] -name = "ms_std_proc_macro" -version = "0.1.0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.79", -] - [[package]] name = "naked-function" version = "0.1.5" @@ -506,7 +508,7 @@ checksum = "5b4123e70df5fe0bb370cff166ae453b9c5324a2cfc932c0f7e55498147a0475" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -613,12 +615,6 @@ dependencies = [ "windows-sys", ] -[[package]] -name = "ryu" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f" - [[package]] name = "sc" version = "0.2.7" @@ -639,34 +635,45 @@ checksum = "61697e0a1c7e512e84a621326239844a24d8207b4669b41bc18b32ea5cbf988b" [[package]] name = "serde" -version = "1.0.210" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8e3592472072e6e22e0a54d5904d9febf8508f65fb8552499a1abc7d1078c3a" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.210" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "243902eda00fad750862fc144cea25caca5e20d615af0a81bee94ca738f1df1f" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] name = "serde_json" -version = "1.0.128" +version = "1.0.150" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ff5456707a1de34e7e37f2a6fd3d3f808c318259cbd01ab6377795054b483d8" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" dependencies = [ "itoa", "memchr", - "ryu", "serde", + "serde_core", + "zmij", ] [[package]] @@ -744,9 +751,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.79" +version = "2.0.87" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89132cd0bf050864e1d38dc3bbc07a0eb8e7530af26344d3d2bbbef83499f590" +checksum = "25aa4ce346d03a6dcd68dd8b4010bcb74e54e62c90c573f394c46eae99aba32d" dependencies = [ "proc-macro2", "quote", @@ -776,7 +783,7 @@ checksum = "08904e7672f5eb876eaaf87e0ce17857500934f4981c4a0ab2b4aa98baac7fc3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -911,7 +918,7 @@ dependencies = [ "anyhow", "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", "wasmtime-component-util", "wasmtime-wit-bindgen", "wit-parser", @@ -1009,7 +1016,7 @@ checksum = "de5a9bc4f44ceeb168e9e8e3be4e0b4beb9095b468479663a9e24c667e36826f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] [[package]] @@ -1045,9 +1052,9 @@ dependencies = [ name = "wasmtime_trans_data" version = "0.1.0" dependencies = [ + "as_hostcall", + "as_std", "lazy_static", - "ms_hostcall", - "ms_std", "spin", "wasmtime_wasi_api", ] @@ -1056,11 +1063,11 @@ dependencies = [ name = "wasmtime_wasi_api" version = "0.1.0" dependencies = [ + "as_hostcall", + "as_std", + "as_std_proc_macro", "hashbrown 0.14.5", "lazy_static", - "ms_hostcall", - "ms_std", - "ms_std_proc_macro", "sjlj", "spin", "wasmtime", @@ -1191,5 +1198,11 @@ checksum = "fa4f8080344d4671fb4e831a13ad1e68092748387dfc4f55e356242fae12ce3e" dependencies = [ "proc-macro2", "quote", - "syn 2.0.79", + "syn 2.0.87", ] + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/user/wasmtime_trans_data/build.sh b/user/wasmtime_trans_data/build.sh index f64b608a..45d8ee4b 100755 --- a/user/wasmtime_trans_data/build.sh +++ b/user/wasmtime_trans_data/build.sh @@ -1,4 +1,12 @@ -$CPP trans_data.cpp -o trans_data.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast +#!/usr/bin/env bash +set -euo pipefail + +WASI_CXX="${WASI_CXX:-${WASI_SDK_PATH:-/opt/wasi-sdk}/bin/clang++}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +cd "$SCRIPT_DIR" + +"$WASI_CXX" trans_data.cpp -o trans_data.wasm -fno-exceptions -fno-rtti -ffast-math -funroll-loops -fomit-frame-pointer -Ofast wasmtime compile --target x86_64-unknown-none -W threads=n,tail-call=n trans_data.wasm @@ -10,4 +18,5 @@ cargo build --target x86_64-unknown-none --release && cc \ -shared \ -o target/x86_64-unknown-none/release/libwasmtime_trans_data.so -ln -s /home/wyj/dyx_workplace/mslibos/user/wasmtime_trans_data/target/x86_64-unknown-none/release/libwasmtime_trans_data.so /home/wyj/dyx_workplace/mslibos/target/release/libwasmtime_trans_data.so +ln -sfn "$SCRIPT_DIR/target/x86_64-unknown-none/release/libwasmtime_trans_data.so" \ + "$REPO_ROOT/target/release/libwasmtime_trans_data.so" diff --git a/user/wasmtime_trans_data/src/lib.rs b/user/wasmtime_trans_data/src/lib.rs index 97e91d24..4e9876eb 100644 --- a/user/wasmtime_trans_data/src/lib.rs +++ b/user/wasmtime_trans_data/src/lib.rs @@ -1,13 +1,12 @@ -#![no_std] - -extern crate alloc; -use core::mem::forget; - -use alloc::{string::{String, ToString}, vec::Vec}; -use spin::Mutex; - -use as_hostcall::types::{OpenFlags, OpenMode}; -use as_std::{agent::FaaSFuncResult as Result, args, println, libos::libos, time::{SystemTime, UNIX_EPOCH}}; +#![no_std] + +extern crate alloc; + +use alloc::{string::ToString, vec::Vec}; +use spin::Mutex; + +use as_hostcall::types::{OpenFlags, OpenMode}; +use as_std::{agent::FaaSFuncResult as Result, args, println, libos::libos}; use wasmtime_wasi_api::{wasmtime, LibosCtx}; @@ -24,15 +23,21 @@ lazy_static::lazy_static! { }; } -fn func_body() -> Result<()> { - let _open_root = *MUST_OPEN_ROOT; - - let lock = INIT_LOCK.lock(); - let (engine, module, linker) = wasmtime_wasi_api::build_wasm(CWASM); - drop(lock); - - let mut store = Store::new(&engine, LibosCtx{id: "0".to_string()}); - let instance = linker.instantiate(&mut store, &module)?; +fn func_body() -> Result<()> { + let data_size = args::get("data_size").unwrap_or("4096"); + let _open_root = *MUST_OPEN_ROOT; + + let lock = INIT_LOCK.lock(); + let (engine, module, linker) = wasmtime_wasi_api::build_wasm(CWASM); + drop(lock); + + wasmtime_wasi_api::set_wasi_args("0", Vec::from([ + "trans_data.wasm".to_string(), + data_size.to_string(), + ])); + + let mut store = Store::new(&engine, LibosCtx{id: "0".to_string()}); + let instance = linker.instantiate(&mut store, &module)?; let memory = instance.get_memory(&mut store, "memory").unwrap(); let pages = memory.grow(&mut store, 20000).unwrap(); @@ -42,10 +47,10 @@ fn func_body() -> Result<()> { .get_typed_func::<(), ()>(&mut store, "_start") .map_err(|e| e.to_string())?; - main.call(store, ()).map_err(|e| e.to_string())?; + main.call(store, ()).map_err(|e| e.to_string())?; Ok(().into()) -} +} #[no_mangle] pub fn main() -> Result<()> { diff --git a/user/wasmtime_trans_data/trans_data.cpp b/user/wasmtime_trans_data/trans_data.cpp index 3f1f8549..c572748a 100644 --- a/user/wasmtime_trans_data/trans_data.cpp +++ b/user/wasmtime_trans_data/trans_data.cpp @@ -6,36 +6,51 @@ #include #include #include -#include -#include +#include +#include +#include using namespace std; -#define MAX_BUFFER_SIZE 4 * 1024 - - -__attribute__((import_module("env"), import_name("buffer_register"))) void buffer_register(void *slot_name, int name_size, void *buffer, int buffer_size); -__attribute__((import_module("env"), import_name("access_buffer"))) void access_buffer(void *slot_name, int name_size, void *buffer, int buffer_size); - - -void get_time() { - timeval tv{}; - gettimeofday(&tv, nullptr); - printf("%lld.%06lld\n", tv.tv_sec, tv.tv_usec); -} - -int main() { - get_time(); - string slot_name = "tmp"; - // string buffer(MAX_BUFFER_SIZE, '0'); - vector buffer(MAX_BUFFER_SIZE, 0); - for(int i=0;i(tv.tv_sec) * 1000000000ull + + static_cast(tv.tv_usec) * 1000ull; +} + +static size_t parse_size(int argc, char **argv) { + if (argc < 2) { + return 4 * 1024; + } + char *end = nullptr; + unsigned long long value = strtoull(argv[1], &end, 10); + if (end != nullptr && *end == '\0') { + return static_cast(value); + } + return 4 * 1024; +} + +int main(int argc, char **argv) { + size_t buffer_size = parse_size(argc, argv); + string slot_name = "tmp"; + vector buffer(buffer_size, 'a'); + uint64_t start_ns; + uint64_t end_ns; + + start_ns = now_ns(); + buffer_register((void*)slot_name.c_str(), slot_name.length(), (void*)buffer.data(), buffer_size); + access_buffer((void*)slot_name.c_str(), slot_name.length(), (void*)buffer.data(), buffer_size); + end_ns = now_ns(); + + if (buffer_size > 0) { + volatile char touched = buffer[0]; + (void)touched; + } + printf("transfer cost: %llu ns size: %zu\n", + (unsigned long long)(end_ns - start_ns), buffer_size); + return 0; +} diff --git a/user/wasmtime_trans_data/trans_data.cwasm b/user/wasmtime_trans_data/trans_data.cwasm index 80636f08..a81bb58f 100644 Binary files a/user/wasmtime_trans_data/trans_data.cwasm and b/user/wasmtime_trans_data/trans_data.cwasm differ diff --git a/user/wasmtime_trans_data/trans_data.wasm b/user/wasmtime_trans_data/trans_data.wasm index 6b79b136..50e2baef 100755 Binary files a/user/wasmtime_trans_data/trans_data.wasm and b/user/wasmtime_trans_data/trans_data.wasm differ diff --git a/wasmtime_wasi_api/src/data_buffer.rs b/wasmtime_wasi_api/src/data_buffer.rs index 4ba6bc9a..c65ea95a 100644 --- a/wasmtime_wasi_api/src/data_buffer.rs +++ b/wasmtime_wasi_api/src/data_buffer.rs @@ -1,8 +1,10 @@ extern crate alloc; -use alloc::{string::String, vec::Vec}; +use alloc::{string::String, vec, vec::Vec}; +use core::{alloc::Layout, ptr}; -use as_std::agent::DataBuffer; +use as_hostcall::Verify; +use as_std::libos::libos; #[cfg(feature = "log")] use as_std::{ println, @@ -67,9 +69,40 @@ pub fn buffer_register( // #[cfg(feature = "log")] // println!("content={:?}", content); - let mut wasm_buffer: DataBuffer = DataBuffer::with_slot(slot_name); - wasm_buffer.0 = buffer_base; - wasm_buffer.1 = buffer_size as usize; + let layout = Layout::new::(); + let buffer_addr = libos!(buffer_alloc_raw(layout)) + .expect("failed to allocate wasm buffer metadata"); + unsafe { + ptr::write( + buffer_addr as *mut WasmDataBuffer, + WasmDataBuffer(buffer_base, buffer_size as usize), + ); + } + if let Err(error) = libos!(buffer_register( + &slot_name, + buffer_addr, + WasmDataBuffer::__fingerprint(), + buffer_size as usize + )) { + libos!(buffer_dealloc(buffer_addr, layout)); + panic!("failed to register wasm buffer: {:?}", error); + } +} + +pub fn buffer_len( + mut caller: Caller<'_, LibosCtx>, + slot_name_base: i32, + slot_name_size: i32, +) -> i64 { + let memory = caller.get_export("memory").unwrap().into_memory().unwrap(); + let mut slot_name = vec![0; slot_name_size as usize]; + memory + .read(&caller, slot_name_base as usize, &mut slot_name) + .unwrap(); + let slot_name = String::from_utf8(slot_name).expect("[Err] Not a valid UTF-8 sequence"); + libos!(buffer_len(&slot_name)) + .and_then(|len| i64::try_from(len).ok()) + .unwrap_or(-1) } pub fn access_buffer( @@ -98,17 +131,29 @@ pub fn access_buffer( #[cfg(feature = "log")] println!("slot_name={}", slot_name); - let wasm_buffer: DataBuffer = DataBuffer::from_buffer_slot(slot_name) - .unwrap_or_else(|| { - #[cfg(feature = "log")] - println!("[Err] access_buffer didn't find the slot_name, return a empty buffer!"); - let mut content: Vec = Vec::with_capacity(buffer_size as usize); - content.resize(buffer_size as usize, 0); - let mut buffer: DataBuffer = DataBuffer::new(); - buffer.0 = content.as_mut_ptr(); - buffer.1 = buffer_size as usize; - buffer - }); + let Some((buffer_addr, fingerprint, data_len)) = libos!(access_buffer(&slot_name)) + else { + #[cfg(feature = "log")] + println!("[Debug] access_buffer didn't find slot_name={}", slot_name); + + let data = memory.data_mut(&mut caller); + data.get_mut(buffer_offset as usize..) + .and_then(|buffer| buffer.get_mut(..buffer_size as usize)) + .expect("access_buffer target is outside wasm memory") + .fill(0); + return; + }; + if fingerprint != WasmDataBuffer::__fingerprint() { + libos!(buffer_register( + &slot_name, + buffer_addr, + fingerprint, + data_len + )) + .expect("failed to restore mismatched wasm buffer"); + panic!("wrong wasm buffer fingerprint"); + } + let wasm_buffer = unsafe { &*(buffer_addr as *const WasmDataBuffer) }; #[cfg(feature = "log")] println!( @@ -116,10 +161,26 @@ pub fn access_buffer( wasm_buffer.0, wasm_buffer.1 ); - if buffer_size as usize != wasm_buffer.1 { - panic!("buffer_size={}, wasm_buffer.1={}, access_buffer's size is different from buffer_register's size", buffer_size, wasm_buffer.1) + if (buffer_size as usize) < data_len { + panic!( + "buffer_size={}, data_len={}, access_buffer target is too small", + buffer_size, data_len + ) } - let buffer = unsafe { core::slice::from_raw_parts(wasm_buffer.0, wasm_buffer.1) }; + + // access_buffer is a non-consuming read. access_buffer() in the mm service + // removes the slot while handing out its allocation, so publish the same + // allocation again before returning. This also lets flag polling observe a + // value repeatedly until every consumer has seen it. + libos!(buffer_register( + &slot_name, + buffer_addr, + WasmDataBuffer::__fingerprint(), + data_len + )) + .expect("failed to restore buffer slot after access"); + + let buffer = unsafe { core::slice::from_raw_parts(wasm_buffer.0, data_len) }; // #[cfg(feature = "log")] // println!("buffer: {:?}", buffer); memory diff --git a/wasmtime_wasi_api/src/lib.rs b/wasmtime_wasi_api/src/lib.rs index 20b1a85f..3d4b06d0 100644 --- a/wasmtime_wasi_api/src/lib.rs +++ b/wasmtime_wasi_api/src/lib.rs @@ -36,6 +36,9 @@ fn import_all(linker: &mut Linker) { data_buffer::buffer_register, ) .unwrap(); + linker + .func_wrap("env", "buffer_len", data_buffer::buffer_len) + .unwrap(); linker .func_wrap( "env", @@ -338,4 +341,4 @@ fn import_all(linker: &mut Linker) { ) .unwrap(); -} \ No newline at end of file +}