`Failed to link` - `file exists` on cache pull

[kotborealis]

While using buildstream I found the following issue:

[--:--:--] STATUS [59065303] self_junction_a.bst:example.bst: Pulling data for artifact 59065303 <- https://...
[--:--:--] INFO [59065303] self_junction_a.bst:example.bst: Pulled artifact 59065303 <- https://...
[--:--:--] INFO [59065303] self_junction_b.bst:example.bst: Pulled artifact 59065303 <- https://...
[00:00:13] SUCCESS [59065303] self_junction_a.bst:example.bst: Pull
[00:00:14] FAILURE [59065303] self_junction_b.bst:example.bst: Failed to link '.../bst-cache/artifacts/refs/project/example/590653033f888793cebe4368a90d0cce91bdacb468fc29ba369cc16449888189 -> .../bst-cache/artifacts/refs/project/example/f4f620fd79b8ad9b0d271c4bd18bb00483b6d92614dc3b2aec8eac049ae4df7f': [Errno 17] File exists: '.../bst-cache/artifacts/refs/project/example/590653033f888793cebe4368a90d0cce91bdacb468fc29ba369cc16449888189' -> '.../bst-cache/artifacts/refs/project/example/f4f620fd79b8ad9b0d271c4bd18bb00483b6d92614dc3b2aec8eac049ae4df7f'

In my project I use multiple self-junctions - junctions that load the project itself to build it differently or to overwrite some elements.
While pulling artifacts in parallel for different self-junctions from cache there's this issue Failed to link ... File exists.

I cannot reproduce this error locally, only on specific CI.

I found mentions of the same error in existing issues/PRs:

• Intermittent test failure in test_shell_pull_cached_buildtree #1740
• Add support for Python 3.12 #1881 (comment)
• Stop passing URL opener across process boundaries #1820 (comment)

My guess is that its some race caused by safe_link function.

I'm trying to produce some fix for this race, what should I consider while tinkering with safe_link?
My best bet is to make link operation more atomic by first making hard-link in some temp directory and them moving it to original destination.