Skip to content

fix(playwright): recognize modern screenshot comparison errors - #808

Open
dimazollo wants to merge 3 commits into
gemini-testing:masterfrom
dimazollo:fix/playwright-screenshot-error-detection
Open

fix(playwright): recognize modern screenshot comparison errors#808
dimazollo wants to merge 3 commits into
gemini-testing:masterfrom
dimazollo:fix/playwright-screenshot-error-detection

Conversation

@dimazollo

@dimazollo dimazollo commented Sep 7, 2026

Copy link
Copy Markdown

Playwright 1.59.1 reports screenshot mismatches as:

Error: expect(page).toHaveScreenshot(expected) failed

The adapter only recognizes Screenshot comparison failed, so it marks these as general errors even when expected/actual/diff images are attached. Accept can then leave a stale error instead of resolving the visual failure.

This adds support for the newer page/locator toHaveScreenshot header. Playwright uses the same header when it cannot take a screenshot, so the adapter also checks for matching expected/actual/diff attachments. The old Screenshot comparison failed handling stays unchanged.

Only the adapter and its unit tests change. toMatchSnapshot, changes to legacy mixed-error handling, and platform handling are outside this PR.

Reproduction

With Playwright 1.59.1 and html-reporter/playwright:

import {test, expect} from '@playwright/test';

test('screenshot mismatch', async ({page}) => {
    const color = process.env.CHANGED ? 'blue' : 'red';
    await page.setContent(`<div style="width:120px;height:80px;background:${color}"></div>`);
    await expect(page).toHaveScreenshot('example.png');
});
npx playwright test --update-snapshots
CHANGED=1 npx playwright test

Before: error / Error. After: fail / ImageDiffError.

Tests

  • All 30 adapter tests pass, including page/locator diffs, capture failures, snapshot attachment matching, unnamed diffs, ANSI formatting, and multiple soft screenshot assertions.
  • npm test passes on Node 20.19.6: lint, Node and jsdom unit tests, and type checks.

The full GUI e2e suite and interactive Accept/Undo weren't rerun for this isolated fix.

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en.

Comment thread lib/adapters/test-result/playwright.ts Outdated
if (/snapshot .*doesn't exist/.test(message) && message.includes('.png')) {
result.name = ErrorName.NO_REF_IMAGE;
} else if (message.includes('Screenshot comparison failed')) {
} else if (this._testResult.errors.every(error => this._isScreenshotComparisonError(error))) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it not enough to just switch to 2 error options? for example, /Screenshot comparison failed|expect\(.*\)\.toHaveScreenshot\(expected\)failed/

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Playwright uses the same header when it cannot take a screenshot. So I also check that expected, actual, and diff images are attached.

I'll keep toHaveScreenshot support with this check and remove the mixed-error changes from this PR.

Comment thread lib/adapters/test-result/playwright.ts Outdated
return true;
}

if (!/^(?:Error: )?expect\((?:page|locator|Buffer)\)\.(?:toHaveScreenshot|toMatchSnapshot)\(expected\)(?: failed)?$/.test(header)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toMatchSnapshot should be used for non-image comparison, so we don't need this part here

@dimazollo dimazollo Sep 8, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can compare images too, but I agree that we don't need it for this fix. I'll remove it from this PR.

@dimazollo
dimazollo requested a review from sipayRT September 8, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants