elfuse real-workload syscall frequency ranking (Top 50)

[Max042004]

Environment: origin/main (435a546) + PR #191 (OCI tooling, 9 commits) + instrumentation
commit 3312482 + unpack setuid-EPERM fallback fix (not separately committed).

Methodology

• Added an ELFUSE_SYSCALL_STATS_DIR whole-lifetime stats mode (built on the
  existing src/debug/syscall-hist.c): recording does not freeze at the first
  execve, fork children (fresh posix_spawn images) record independently, and
  each guest process writes <host-pid>.stats (name count total_ns) at exit
  (atexit / elfuse_launch). Summing the per-process files yields exact
  process-tree totals.
• The watchdog was raised to --timeout 3600 via an ELFUSE_BIN wrapper so
  long compile bursts are not killed (the 10 s default per-iteration alarm;
  note --timeout 0 is not "no timeout" — 0 selects worker-thread semantics).

Workloads (five categories, 116 guest processes)

workload	image	content	syscalls	processes
jvm	eclipse-temurin:21	javac compile of a multi-class program + java run (collections, file I/O, SHA-256, 8 threads, subprocess)	3,028,653	8
python	python:3.12-slim	JSON/regex churn, sqlite3 (20k inserts + queries), 400 small-file I/O, os.walk over the stdlib, 10 subprocesses, 4 threads	1,364,400	11
go	golang:1.23-alpine	go version + gofmt over stdlib trees (net/runtime/crypto) + gofmt -d fmt	46,037	6
node	node:22-alpine	module loading, fs/crypto/zlib/JSON, then an HTTP server taking 302 host-side curl requests before /quit	18,740	1
c-compile	gcc:14	30-file C project via make -j1 + sqlite amalgamation (8 MB TU) gcc -O1 compile+link + a sqlite3 query	82,606	90

Total: 4,540,436 syscalls, 89 distinct.

Top 50 ranking (five workloads combined, descending)

#	syscall	count	share%	cum%	jvm	python	go	node	c-compile
1	futex	2,559,392	56.369	56.37	1,517,216	1,040,892	166	1,118	0
2	clock_gettime	1,849,178	40.727	97.10	1,507,075	308,525	30,326	3,251	1
3	readlinkat	48,107	1.060	98.16	109	3	1	15	47,979
4	openat	12,457	0.274	98.43	240	1,213	1,572	633	8,799
5	read	9,449	0.208	98.64	409	972	2,903	1,230	3,935
6	close	6,978	0.154	98.79	209	1,255	1,573	936	3,005
7	mmap	6,152	0.135	98.93	662	181	87	2,477	2,745
8	fcntl	5,737	0.126	99.05	31	2,035	2,926	628	117
9	newfstatat	5,347	0.118	99.17	106	1,582	1,488	0	2,171
10	fstat	5,330	0.117	99.29	171	2,131	2	14	3,012
11	lseek	3,785	0.083	99.37	104	1,712	0	11	1,958
12	munmap	3,532	0.078	99.45	195	99	2	2,406	830
13	epoll_ctl	3,515	0.077	99.53	0	0	2,903	612	0
14	write	3,011	0.066	99.59	260	454	4	305	1,988
15	mremap	2,050	0.045	99.64	0	4	0	2,046	0
16	brk	2,004	0.044	99.68	27	81	6	9	1,881
17	rt_sigaction	1,928	0.042	99.73	170	767	353	35	603
18	epoll_pwait	1,769	0.039	99.76	0	0	260	1,509	0
19	mprotect	1,234	0.027	99.79	219	98	6	79	832
20	rt_sigprocmask	1,155	0.025	99.82	353	69	64	63	606
21	ioctl	980	0.022	99.84	20	869	2	5	84
22	pread64	912	0.020	99.86	405	507	0	0	0
23	getdents64	724	0.016	99.87	10	426	234	0	54
24	faccessat	709	0.016	99.89	36	11	0	0	662
25	madvise	624	0.014	99.90	70	4	485	64	1
26	nanosleep	573	0.013	99.92	0	0	573	0	0
27	prlimit64	367	0.008	99.92	26	11	3	6	321
28	setsockopt	305	0.007	99.93	2	0	0	303	0
29	writev	304	0.007	99.94	0	0	2	302	0
30	accept4	302	0.007	99.94	0	0	0	302	0
31	shutdown	302	0.007	99.95	0	0	0	302	0
32	execve	196	0.004	99.96	8	10	5	0	173
33	pwrite64	192	0.004	99.96	0	192	0	0	0
34	getcwd	163	0.004	99.96	6	57	1	1	98
35	set_robust_list	163	0.004	99.97	55	15	0	0	93
36	rseq	154	0.003	99.97	49	15	0	0	90
37	clone3	131	0.003	99.97	41	4	0	0	86
38	sched_getaffinity	124	0.003	99.98	121	0	3	0	0
39	set_tid_address	120	0.003	99.98	9	11	8	2	90
40	exit_group	116	0.003	99.98	8	11	6	1	90
41	wait4	116	0.003	99.98	7	10	10	0	89
42	sysinfo	90	0.002	99.99	3	0	0	0	87
43	prctl	43	0.001	99.99	42	0	0	1	0
44	renameat	42	0.001	99.99	0	42	0	0	0
45	pipe2	41	0.001	99.99	7	30	0	4	0
46	unlinkat	41	0.001	99.99	3	4	0	0	34
47	clone	39	0.001	99.99	6	10	14	6	3
48	dup3	37	0.001	99.99	9	20	8	0	0
49	gettimeofday	34	0.001	99.99	4	0	0	0	30
50	statx	34	0.001	99.99	33	0	0	1	0

Beyond the top 50: 39 syscalls, 348 calls (0.008%).

Key takeaways

1. futex (56.4%) + clock_gettime (40.7%) = 97.1%. Thread-heavy runtimes
   (JVM, CPython) are completely dominated by these two.
2. A futex fast-path is the second target: mostly uncontended wake/wait;
   1.51 M calls from the JVM, 1.04 M from CPython.
3. readlinkat at Tighten madvise Linux parity for hole/anon/shared #3 comes almost entirely from gcc (47,979 of 48,107):
   the gcc driver re-resolves its own and its tools' paths for every TU — a
   signature of compile workloads.
4. The file-I/O family (openat/read/close/fstat/newfstatat/lseek/write)
   sums to about 1%, but it is the part that hits the host FS on every call,
   with a much higher per-call cost than futex.
5. Shares correlate strongly with workload mix (jvm+python are 97% of total
   volume); the per-workload columns are more meaningful than the combined
   share. Node's signature is epoll/accept4/writev/mremap, go (gofmt) is
   fcntl/epoll_ctl/read, compiles are readlinkat/openat/brk.

Ranks 51–89 (the full tail)

#	syscall	count	jvm	python	go	node	c-compile
51	sigaltstack	33	9	0	24	0	0
52	getrusage	29	0	0	0	0	29
53	ppoll	23	3	20	0	0	0
54	exit	22	13	4	0	5	0
55	readv	22	0	0	0	22	0
56	socket	21	14	2	0	1	4
57	close_range	20	0	20	0	0	0
58	capget	18	0	0	0	18	0
59	connect	14	8	2	0	0	4
60	dup	13	1	0	0	0	12
61	fchdir	12	12	0	0	0	0
62	fdatasync	12	0	12	0	0	0
63	sched_yield	10	6	0	4	0	0
64	uname	10	4	0	2	3	1
65	statfs	9	9	0	0	0	0
66	epoll_create1	7	0	1	3	3	0
67	rt_sigreturn	7	5	0	2	0	0
68	eventfd2	6	0	0	3	3	0
69	ftruncate	6	6	0	0	0	0
70	mkdirat	6	4	1	0	1	0
71	umask	6	4	0	0	0	2
72	getrandom	5	0	3	0	0	2
73	chdir	4	1	0	0	0	3
74	clock_getres	4	3	0	0	1	0
75	faccessat2	4	3	0	1	0	0
76	io_uring_setup (425)	3	0	0	0	3	0
77	clock_nanosleep	3	3	0	0	0	0
78	fchown	3	0	3	0	0	0
79	flock	3	3	0	0	0	0
80	getpeername	2	1	0	0	0	1
81	splice	2	2	0	0	0	0
82	tgkill	2	0	0	2	0	0
83	bind	1	0	0	0	1	0
84	fchmodat	1	0	0	0	0	1
85	getresgid	1	1	0	0	0	0
86	getresuid	1	1	0	0	0	0
87	listen	1	0	0	0	1	0
88	llistxattr	1	1	0	0	0	0
89	membarrier	1	0	0	0	1	0

Ranks 51+ total 348 calls (0.008%) — almost entirely one-shot startup calls
(capget, getresuid, membarrier) or workload signatures (node's socket family,
python's fdatasync/close_range).

[henrybear327]

Hi @Max042004 can you update the description to point to your instrumentation commit? Thank you!