Skip to content

fix(playwright): recognize modern screenshot comparison errors - #2

Closed
dimazollo wants to merge 2 commits into
upstream-base/screenshot-error-detectionfrom
fix/playwright-screenshot-error-detection
Closed

fix(playwright): recognize modern screenshot comparison errors#2
dimazollo wants to merge 2 commits into
upstream-base/screenshot-error-detectionfrom
fix/playwright-screenshot-error-detection

Conversation

@dimazollo

@dimazollo dimazollo commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Playwright 1.59.1 reports screenshot mismatches as:

Error: expect(page).toHaveScreenshot(expected) failed

The adapter only recognizes Screenshot comparison failed, so it marks these as general errors even when expected/actual/diff images are attached. Accept can then leave a stale error instead of resolving the visual failure.

This recognizes the newer page/locator and Buffer matcher headers, checking for matching diff attachments. Capture failures and unrelated assertion errors remain general errors; the old format still works. Only the adapter and its unit tests change. Platform handling is separate.

Reproduction

With Playwright 1.59.1 and html-reporter/playwright:

import {test, expect} from '@playwright/test';

test('screenshot mismatch', async ({page}) => {
    const color = process.env.CHANGED ? 'blue' : 'red';
    await page.setContent(`<div style="width:120px;height:80px;background:${color}"></div>`);
    await expect(page).toHaveScreenshot('example.png');
});
npx playwright test --update-snapshots
CHANGED=1 npx playwright test

Before: error / Error. After: fail / ImageDiffError.

Tests

  • Seven regression cases fail before the fix; all 33 adapter tests pass after it.
  • npm test passes on Node 20.19.6, based on upstream 11.16.2 (a4e797c).
  • Checked actual Playwright 1.59.1 results: page, locator and Buffer diffs are recognized; mixed assertion failures and a capture timeout remain errors.

The full GUI e2e suite and interactive Accept/Undo weren't rerun for this isolated fix.

Draft in the personal fork; upstream submission is pending CLA confirmation.

@dimazollo
dimazollo force-pushed the fix/playwright-screenshot-error-detection branch from e7708a8 to edb83a5 Compare September 7, 2026 21:15
@dimazollo

Copy link
Copy Markdown
Owner Author

Submitted upstream as gemini-testing#808. Closing this preparation draft without merging; the source branch is still in use by the upstream PR.

@dimazollo dimazollo closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant