d3d11: refcount per-HWND Metal view to fix ANGLE white screen (#183)

[seanellul]

Fixes #183.

Problem

Games that render through ANGLE (GLES over D3D11; they ship libEGL.dll + libGLESv2.dll) present a pure white window under DXMT. Audio and input work and there are no errors from the game, ANGLE, or DXMT. The same binaries render correctly under D3DMetal, so the game and ANGLE are fine; the gap is DXMT-side.

Root cause

winemac.drv's newMetalViewWithDevice: returns a per-window singleton (if (_metalView) return _metalView;, no extra retain), and macdrv_view_release_metal_view does removeFromSuperview + release (dealloc). The D3D11 swapchain creates/releases that view per swapchain with no refcount.

The repro game (My Singing Monsters, appid 1419170, free, 32-bit) destroys and recreates its EGL window surface during startup. EGL requires deferred destruction there: destroying a surface that is still current only schedules it, and the surface is really destroyed once it stops being current. ANGLE implements exactly that, so at the D3D level the order is always: swapchain #2 gets created on the same HWND first, then the old surface dies on the next eglMakeCurrent and swapchain #1's destructor runs. Both chains hold the same singleton view, so the old destructor detaches the view the live swapchain is presenting into. Every later present succeeds into an orphaned view and the window shows its blank background. Apps that create one swapchain per window never hit this.

Trace from an instrumented build (logging only, stock view lifetime):

CreateSwapChain hwnd=0x110194 size=1024x768  bufferCount=1 swapEffect=1
SwapChain ctor  hwnd=0x110194 view=140676770627888
CreateSwapChain hwnd=0x110194 size=1920x1080 bufferCount=1 swapEffect=1
SwapChain ctor  hwnd=0x110194 view=140676770627888   <- same view (winemac singleton)
SwapChain dtor  hwnd=0x110194 view=140676770627888 -> ReleaseMetalView (detach)
Present1 #120 ... #1080   <- presents keep succeeding into the detached view

Window capture during this run is 100% white. A diagnostic build that cleared every drawable to magenta still produced a pure white on-screen window, which is how the detached-layer theory was confirmed in the first place. ANGLE source links backing the surface-recreation mechanism are in the comments below.

Fix

A process-global refcounted HWND -> {view, layer, refcount} map in d3d11_swapchain.cpp. The view is created once per window and released only when the last swapchain on that HWND is destroyed. The tricky parts:

• No WMT:: calls under the lock. CreateMetalViewFromHWND runs before the lock is taken (it returns the singleton, so a race just yields the same handle); the view to release is extracted under the lock and ReleaseMetalView is called after the lock is dropped.
• Recycled HWNDs. Windows reuses handle values, so acquire compares the current singleton handle against the stored entry. On mismatch it replaces the entry and releases the stale view; otherwise a new window would inherit a detached view, which is the original bug again, silently.
• Release only the canonical stored handle, exactly once. Never the caller's parameter, and never on a map miss, so no double-release or use-after-free.

With the fix, the same startup sequence logs CREATE refcount=1, REUSE refcount=2, KEPT ALIVE refcount=1, and the game renders (window capture 0.2% white, ~4,700 distinct colors).

Why d3d11, not winemetal.so

The lifetime bug is a property of how the swapchain owns the view, so the refcount belongs with the swapchain's create/destroy. Keeping it PE-side also means it ships without an ABI-coupled winemetal.so rebuild. If you'd prefer the refcount to live in winemetal as a single source of truth for any future caller, happy to move it.

Testing

• My Singing Monsters now displays correctly (an easy repro: free, 32-bit).
• Non-regression on single-swapchain apps, which must be behavior-identical: Among Us (Unity, 32-bit), Pool Blitz (UE4, 64-bit), and a UE5 title all render as before. They hit create -> refcount 1, destroy -> release, the same as stock.
• Applies cleanly to master.

Known minor edge

g_metal_views.emplace could throw std::bad_alloc and leak the freshly created view on OOM. It's an allocation-failure-only path, no worse than the surrounding code, so I left it unguarded to keep the diff minimal. Happy to wrap it if you want it hardened.

[3Shain]

Hmm this is another proof of how fragile the integration between DXMT and Wine is. Is it legal to create multiple swapchain from the same HWND or are there some missing lockings? It would be still incorrect if the app expect two separate swapchains which share a single metal layer.

ANGLE recreates its swapchain on the startup resize (e.g. 1024×768 → native) while the previous swapchain is still alive. Both hold the same singleton view; the old swapchain's destructor then detaches/releases the view the live swapchain is presenting into.

Any source code from ANGLE can prove that?

[seanellul]

Took another look at the ANGLE source to answer this properly. The game ships ANGLE 2.1.23410 (git 1062271d993c), so all links below are pinned to that revision. Short version: my original description got one detail wrong, and the corrected version answers both of your questions. I've updated the PR description and the code comment to match (comment-only commits).

Basically: ANGLE doesn't recreate the swapchain on resize. The game recreates its EGL surface.

At this revision a surface's swapchain is only created once. reset() bails out to resize(), which is a plain ResizeBuffers, whenever a chain already exists:

https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/renderer/d3d/d3d11/SwapChain11.cpp#L600-L604

So the two CreateSwapChainForHwnd calls in my trace have to come from two EGL surfaces. A single surface can't produce them: SwapChain11::reset releases the old chain before creating a new one, and SurfaceD3D::resetSwapChain asserts !mSwapChain. The game apparently destroys and recreates its window surface during startup, which I think is a common pattern in mobile ports.

Why the old swapchain is still alive when the new one is created

This part is required by EGL. Not a timing accident. Destroying a surface that is still current only schedules the destruction: "If the EGL surface surface is not current to any thread, eglDestroySurface destroys it immediately. Otherwise, surface is destroyed when it becomes not current to any thread." (https://registry.khronos.org/EGL/sdk/docs/man/html/eglDestroySurface.xhtml)

In ANGLE:

• eglDestroySurface(old) removes the window from the surface map right away, but the surface itself survives because being current holds a ref:
  https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/Display.cpp#L1795-L1821
  https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/Surface.cpp#L267-L307
• eglCreateWindowSurface(new) on the same HWND passes the duplicate-window check because the map entry is already gone (https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/validationEGL.cpp#L2824-L2828), and creates swapchain chore: update to latest metal-cpp #2 while feat: dxbc to air transpiler #1 is still fully alive.
• eglMakeCurrent(new) drops the old surface's last ref, the deferred destroy runs, and ~SwapChain11 finally releases chain feat: dxbc to air transpiler #1:
  https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/Surface.cpp#L286-L297
  https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/renderer/d3d/d3d11/SwapChain11.cpp#L108-L118

So for any EGL client that recreates its surface while it's current, the new swapchain is created before the old one is released, every time.

I wanted to double check this, so I re-ran an instrumented build, once with the stock view lifetime, and once with this PR's refcount:

Trace with stock view lifetime (probes only, no fix): the detach, then 1000+ presents into the orphaned view

CreateSwapChain hwnd=0x110194 size=1024x768  format=24 bufferCount=1 swapEffect=1 usage=0x70
SwapChain ctor  hwnd=0x110194 view=140676770627888 layer=105553142047984
CreateSwapChain hwnd=0x110194 size=1920x1080 format=24 bufferCount=1 swapEffect=1 usage=0x70
SwapChain ctor  hwnd=0x110194 view=140676770627888 layer=105553142047984   <- same view (winemac singleton)
SwapChain dtor  hwnd=0x110194 view=140676770627888 -> WMT::ReleaseMetalView (unconditional detach)
Present1 #120  view=140676770627888 desc=1920x1080 swapEffect=1 minimized=0
Present1 #240  ...
Present1 #1080 view=140676770627888 desc=1920x1080 swapEffect=1 minimized=0

Window capture at t+30s: 100.0% white pixels, 1 distinct color.

Same run with this PR's refcount: detach intercepted, game renders

CreateSwapChain hwnd=0x9021a size=1024x768  ...
AcquireMetalView hwnd=0x9021a CREATE view=140638694736880 refcount=1
CreateSwapChain hwnd=0x9021a size=1920x1080 ...
AcquireMetalView hwnd=0x9021a REUSE  view=140638694736880 refcount=2   <- overlap, refcount reaches 2
ReleaseMetalView hwnd=0x9021a view=140638694736880 KEPT ALIVE refcount=1   <- old chain's dtor intercepted

Window capture at t+30s: 0.2% white pixels, roughly 4,700 distinct colors (rendered title screen).

Happy to attach the full logs if you need. The repro is a free Steam game, My Singing Monsters (appid 1419170).

Is it legal?

I believe so? It's a single-threaded deterministic sequence, so there is no missing locking. The app never holds two swapchains itself; the overlap comes from the EGL-mandated deferral. The chain is blt-model (BufferCount = 1, DXGI_SWAP_EFFECT_SEQUENTIAL, https://github.com/google/angle/blob/1062271d993c5c84158eb93b4f2641305a232e20/src/libANGLE/renderer/d3d/d3d11/win32/NativeWindow11Win32.cpp#L162-L168). The one-swapchain-per-HWND restriction in the DXGI docs is stated for flip model only ("Because you can associate only one flip presentation model swap chain at a time with an HWND...", https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_2/nf-dxgi1_2-idxgifactory2-createswapchainforhwnd#remarks).

I can't find anything that forbids it for blt chains, and this exact destroy-then-recreate sequence is what the game does on Windows, where it renders fine. So I think the backend does have to tolerate briefly overlapping swapchain lifetimes on the same HWND.

On two swapchains sharing one metal layer

Agreed that would be incorrect if both were actually presenting. But that limitation comes from winemac.drv, which only has one metal view per window (if (_metalView) return _metalView;, https://gitlab.winehq.org/wine/wine/-/blob/1f069f492a487af1fe077f182ae753b58029d67c/dlls/winemac.drv/cocoa_window.m#L710-L722). Two swapchains presenting concurrently to one HWND can't work on macdrv no matter what d3d11 does; supporting that would need per-swapchain layers in winemac.drv itself. What the refcount fixes is the transient case above, where the old chain is already destroyed-pending and never presents again. Today its deferred destructor detaches the view the live chain is presenting into (https://gitlab.winehq.org/wine/wine/-/blob/1f069f492a487af1fe077f182ae753b58029d67c/dlls/winemac.drv/cocoa_window.m#L4019-L4026 does removeFromSuperview + release). With the refcount the live chain keeps its view, and truly concurrent apps are no worse off than before.

Happy to move the refcount into winemetal if you'd rather keep the policy there?