Skip to content

Enable use of kernel copies of files in more cases - #20056

Closed
stuhood wants to merge 3 commits into
pantsbuild:mainfrom
stuhood:stuhood/kernel-copy
Closed

Enable use of kernel copies of files in more cases#20056
stuhood wants to merge 3 commits into
pantsbuild:mainfrom
stuhood:stuhood/kernel-copy

Conversation

@stuhood

@stuhood stuhood commented Oct 18, 2023

Copy link
Copy Markdown
Member

std::io::copy will use syscalls to execute copies in-kernel on a best-effort basis, but the syscalls which provide copies for file ranges are limited, and the wrapping Write implementation must be identified as one backed by file handles without intermediate layers.

std::fs::copy, on the other hand, can more reliably use syscalls to copy in kernel, including using fclonefileat/fcopyfile on macOS to effectively create a decoupled hardlink.

This change moves to using tokio::fs::copy for capturing of files into the filesystem-based store.

@stuhood

stuhood commented Oct 18, 2023

Copy link
Copy Markdown
Member Author

The first two commits are #20055: this was separated out to avoid cherry-picking it.

@thejcannon

Copy link
Copy Markdown
Member

Can you share any information on how you found this? And maybe some numbers associated, since it's marked performance?

@stuhood

stuhood commented Oct 18, 2023

Copy link
Copy Markdown
Member Author

Can you share any information on how you found this? And maybe some numbers associated, since it's marked performance?

I just noticed it while working #20055. I recalled the difference between std::io::copy and std::fs::copy from our previous looks at how to fix hardlink'ing+Docker on macOS.

I'll see if I can add a benchmark.

@stuhood

stuhood commented Oct 18, 2023

Copy link
Copy Markdown
Member Author

Can you share any information on how you found this? And maybe some numbers associated, since it's marked performance?

I can't see a meaningful difference in our existing benchmark of snapshot capture. Will close. Thanks for encouraging rigor here!

@stuhood stuhood closed this Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants