feat(document-auto-capture): match legacy desktop UI and fix distance detection#643
Open
Barnabas A Nsoh (ayinloya) wants to merge 116 commits into
Open
feat(document-auto-capture): match legacy desktop UI and fix distance detection#643Barnabas A Nsoh (ayinloya) wants to merge 116 commits into
Barnabas A Nsoh (ayinloya) wants to merge 116 commits into
Conversation
…deploy-preview (#586) * feat: add manual workflow_dispatch trigger with skip_tests option to deploy-preview * feat: add manual workflow_dispatch trigger with skip_tests option to deploy-preview * fix: remove unnecessary if condition from share-preview-url step * refactor: rename step id set_dest_dir_hosted_web to set_dest_dir_embed for clarity * feat: add manual workflow_dispatch trigger to destroy-preview with safe branch handling * chore: limit preview comment step to pull_request events * fix: use safe inputs expression for skip_tests across all trigger types
…oup across 1 directory (#582) chore(deps-dev): bump vite in the npm_and_yarn group across 1 directory Bumps the npm_and_yarn group with 1 update in the / directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite). Updates `vite` from 7.2.2 to 7.3.2 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v7.3.2/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v7.3.2/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 7.3.2 dependency-type: direct:development dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Barnabas A Nsoh <banasco@gmail.com>
The new Preact-based component was missing shadow DOM, which caused
Cypress tests to fail when looking for .back-button in the shadow root.
The original JS component used attachShadow() but preact-custom-element
defaults to no shadow DOM. Adding { shadow: true } option fixes the test.
The embed tests expect an element with id="take-photo" in the document capture instructions component. The new Preact component was missing this ID, causing test failures.
* feat(web-components): update Navigation to match new design - Replace circular colored icon buttons with minimal 40x40px semi-transparent buttons - Use simple white arrow icon for back button (no text label) - Use simple white X icon for close button - Add hover and focus-visible states - Move button labels to aria-label for accessibility - Update stories with dark background to visualize white icons * Update navigation to Figma tokens and parent-controlled padding * Fix dependency * fix(web-components): address PR review comments on Navigation * fix(web-components): improve Navigation hover performance and story visibility * fix(web-components): add appearance resets and decouple theme-color to icon
…ated Side-mounted capture/gallery buttons were keyed on useLandscapeUi, which stays true for landscape doc types (id-card, passport) even on desktop where rotation is suppressed. Switch to shouldRotateUi so the bottom row renders the buttons whenever the UI isn't actually rotated.
Add https local host for mobile testing Show guide throughout capture
Add focusMode continuous as constraints Comment out 4k resolution
* feat(web-components): new document capture instructions screen
* fix(web-components): enable shadow DOM for DocumentCaptureInstructions
The new Preact-based component was missing shadow DOM, which caused
Cypress tests to fail when looking for .back-button in the shadow root.
The original JS component used attachShadow() but preact-custom-element
defaults to no shadow DOM. Adding { shadow: true } option fixes the test.
* fix(web-components): add take-photo id to start button for test compat
The embed tests expect an element with id="take-photo" in the document
capture instructions component. The new Preact component was missing
this ID, causing test failures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(doc-auto-capture): address PR #602 review comments - A4: clamp autoCaptureTimeout to documented 3000-30000ms range - A5: guard ResizeObserver for environments where it is undefined - A6: HeroLottie now destroy()s before flipping hasError - A7: drop selected attr on controlled <select> in previews route - B1: replace contrived useCamera reduce-chain with for-of fallback - B2: gate dev console.info telemetry behind ?debug URL flag - B3: extract safeDelete helper for OpenCV Mat cleanup - B4: friendlier user-facing feedback strings (no raw ratios/percentages) - B5: comment opencvLoader's typeof window guard as SSR-only - C1/C2: explicit prop types for CaptureButton/TuningPanel (FunctionComponent) - C3: relax eslint import/extensions tsx rule with rationale comment - D1: flip sync-roi-to-guide default to false to match README - D2: drop theme-color attribute from props/observedAttributes/README/theme * refactor await in loop to recursive function * fix(doc-auto-capture): restore sync-roi-to-guide default to true Reverts the D1 portion of the previous PR-602 fix-up commit. Flipping the default to false changed runtime behaviour for every consumer that relied on the implicit default; keep the legacy guide-synced ROI as default and document it in the README. * fix(doc-auto-capture): recover from rotation re-encode failure in capture loop Add .catch to the Promise.all rotation path so a canvas 2d context failure resets isCapturingRef and returns to IDLE instead of permanently freezing the detection loop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(doc-auto-capture): restart rAF loop after rotation encode failure The `.catch` handler on the `Promise.all` rotation path reset `isCapturingRef` and set state to IDLE, but left the detection loop dead. The `finally` block runs synchronously (while `isCapturingRef` is still `true`), so no `requestAnimationFrame` was ever requeued after the async rejection settled. The loop will now restart when a rotation encode failure is caught. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf(doc-auto-capture): downscale video to 640px before OpenCV processing All CV ops (GaussianBlur, Canny, findContours, Laplacian, etc.) now run on a 640px-wide canvas instead of the native camera resolution (~1920px). This reduces the pixel count ~9× and should bring per-frame processing time under 10ms on mid-range devices. The full-res canvas is kept for the captured image — only the detection pipeline is downscaled. Detected contour coords are scaled back to full-res before being stored in latestCardRectRef so capture cropping remains accurate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: lint fix * fix(doc-auto-capture): address prfectionist and copilot review comments - safeDelete: wrap each m.delete() in try/catch so one bad mat doesn't abort cleanup of the remaining mats in the array - DocumentCaptureInstructions: guard cleanup animation.destroy() with try/catch since handleError may have already destroyed it - useCamera tryConstraints: thread lastError through recursive calls so the final rejection preserves the real DOMException (e.g. NotAllowedError) instead of swapping it for a generic message Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(doc-auto-capture): address two Copilot review comments - Reset stabilityRef and bestFrameRef in rotation .catch so a failed re-encode doesn't leave the stability counter at threshold, which would immediately re-trigger auto-capture on the next frame and spin in a failure/retry loop - Guard downscaled canvas setup against video.videoWidth === 0 (common early in startup) and null getContext return, both of which would propagate NaN into all downstream ROI math and throw in the hot loop Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: lint fix * fix(doc-auto-capture): scale CV kernels with downscale factor to prevent false distance prompts Fixed-size kernels (5×5 Gaussian, 3×3 morphological close) become proportionally larger as the processing resolution decreases. At 640px (1/3 of 1920px), background texture gaps that were wider than the kernel at full resolution are now narrower, causing edge bridging that produces false large-rectangle contours and triggering "Move document further away" incorrectly. Kernel size and closing iterations are now computed from dsScale so the physical blur radius and bridging distance stay constant relative to the scene content regardless of PROCESS_WIDTH. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(doc-auto-capture): reject ROI-filling contours to prevent background false positives Background patterns (carpet, tile, wood grain) can form large rectangles that pass aspect-ratio and rectangularity checks and win the largest-area selection, triggering false "Move document further away" prompts. A real card at correct distance leaves visible space on at least 2 sides of the ROI; a background rectangle fills all 4 walls simultaneously. Contours whose bounding rect touches 3+ ROI walls (within a 4% margin) are now rejected during selection. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(doc-auto-capture): reject background contours via perimeter compression ratio Background texture (carpet, concrete) forms long jagged edge paths that approxPolyDP collapses dramatically to 4 vertices — original perimeter is 4–10× the approximated polygon perimeter. A real card border is already near-rectangular before approximation, giving a ratio of 1.0–2.0. Contours with peri/approxPeri >= 3.5 are now rejected during selection, stopping textured backgrounds from triggering false "Move document further away" prompts even when no card is in frame. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(doc-auto-capture): reject background contours via polygon-side edge support Previous fixes (wall-hug, perimeter compression) don't catch compact background structures (shadows, tile seams, uniform-texture rectangles) that already have a near-rectangular edge path before approxPolyDP. Add a background-normalised edge support check: sample presenceEdges along each of the 4 approx-polygon sides (84 points total) and compare the hit fraction to the ambient edge density of the whole ROI. A real card border aligns its polygon sides with physical card edges → hit ratio 5–15× above background. Background "rectangles" have side density ≈ background density → ratio ~1×. Threshold: 2.5× relative or 45% absolute (for very plain backgrounds). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(doc-auto-capture): run contour detection at full resolution The 640px downscale (2004999) collapsed the card border to a ~1px line that Canny/findContours could only recover when the card was large in frame. Distance feedback is gated on a validated contour, so detection silently failed: no "move closer/further", eventually "Align document in frame". Every fix since piled on rejection heuristics that also rejected the real, now-faint border. Run the contour pass at native resolution on a crop of just the guide-box ROI (cheap — the ROI is a fraction of the frame and the pass is gated behind the presence/grid check). Keep the cheap gates (presence, texture, grid, blur, glare) on the 640px canvas. Relax the contour gates back to the pre-regression set (4 corners + fill-ratio + angles + aspect + wall-hug), dropping the perimeter-compression and edge-support gates that fought the real border. Distance metrics now divide by the ROI area in their own coordinate space (full-res for the contour, 640px for the presence-edge fallback). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…-old-design Pull latest #602 (incl. #654 review fixes) into #643. Resolved the one conflict in useCardDetection.ts: kept #643's detection tuning (syncRoiToGuide=false, fillRatio>0.75, aspectTol + guide-based minArea) and adopted #602's dsRoiArea fix for the 640px presence-map fill metric. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On desktop, a card filling the visible box was silently rejected by the ROI wall-hug gate before the max-fill gate could say "Move document further away", leaving users stuck on "Align document in frame" - and deadlocking discovery when no document-type is provided, since the discovery timeout only advances on valid contours. Desktop-only changes (gated on skipGridCheck / Desktop defaults): - Treat a quad that fails ONLY the wall-hug check as a "too close" signal: show "Move document further away" in both CAPTURE and DISCOVERY phases instead of the dead-end alignment prompt. - Relax the wall-hug rejection from 3 to 4 simultaneous wall touches; 3 touches is common for legitimate off-center hand-held cards. - Lower desktop minFillPercent from 78 to 70 (area), widening the effective capture band from ~78-90% to ~70-90% fill. - Fix a stale comment claiming desktop fill thresholds are 85/99. - Expose minFillPercent/maxFillPercent sliders in the debug-only TuningPanel for empirical tuning. Mobile behaviour is unchanged. https://claude.ai/code/session_011mDeM9KWiDTC5NR8RPynRx
… detection After widening the desktop capture band (8a9c180), a document-free scene could still auto-capture: the id-card synthetic fallback builds a "card" from the combined bbox of ALL significant contours (face, furniture, window frames), whose aspect/area easily pass its gates. The synthetic then activates fill gating, sails through blur/glare/stability, and captures the whole ROI - the review screen shows a room, not a document. Restore the fallback to its documented purpose - bridging finger/glare contour dropouts of a card that WAS being detected - by requiring a genuine 4-corner card validation within the last SYNTH_BRIDGE_MAX_FRAMES (15 frames, ~0.5s at 30fps). Pristine/reset state is Infinity, so empty scenes never synthesize; resetCapture re-arms the window so it cannot leak across retakes. Book-doc (passport/greenbook) fallback is untouched; mobile behaviour is unchanged (the recency term only tightens an already desktop-only path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recency gate (e29fae6) over-corrected: fingers permanently crossing a card edge - the normal way to hold a card to a webcam - can prevent a clean 4-corner quad from EVER forming, so the bridge window never opens and a well-positioned card stalls on "Align document in frame". Make the synthetic eligible when EITHER a real card was detected within the bridge window OR the scene passes the 7/9 grid-coverage bar that mobile enforces for every capture (passingCells is already computed on desktop; only its gating was skipped). A hand-held card filling the box lights up the whole 3x3 grid, while the document-free scene that previously auto-captured leaves blank wall cells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The capture components enable their tuning panel and verbose telemetry when their own window.location.search contains `debug` — but the embed iframe src never carried the host page's query string, so `?debug` on an integrating page (e.g. the previews app) silently did nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…closer" A card pushed too close overflows the ROI: its outer edges leave the frame, so no 4-corner quad forms and the wall-hug too-close signal never fires (it needs a complete quad). Only inner-content contours remain, clipped at the ROI walls. The synthetic fallback then either fails its aspect check (clipped bbox is too wide) - "Align document in frame" - or forms from the content cluster whose bbox underestimates the card - fill < min - "Move document closer". Both prompts push the user the wrong way. Detect the overflow directly: combined contour bbox clipped at 2+ ROI walls with capture-grade (7/9) grid coverage. On overflow, suppress the synthetic (a capture would clip the card's edges anyway) and show "Move document further away" in both CAPTURE and DISCOVERY no-contour paths. A genuinely far card keeps its small centered bbox and still gets "Move document closer". Desktop-only (skipGridCheck-gated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The overflow heuristic (412935b) counted ANY 2+ ROI wall touches of the combined contour bbox as "card overflows the box". But the hand holding the card is a significant contour too - fingers intrude from one side or corner, dragging the bbox to adjacent walls at a perfectly good distance, suppressing the synthetic and showing a false "Move document further away". Require an OPPOSITE wall pair (left+right or top+bottom): a card overflowing the box spans the ROI along an axis; a hand does not. Also expose the active detection ROI (clamped, mapped back to the video element's CSS box) from useCardDetection in debug mode, and outline it in the desktop layout when ?debug is set, so wall-hug/overflow/fill issues can be judged against what the camera actually analyses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel rendered edge density / texture / blur / glare but never docFill (no row for it) and kept the 3x3 grid densities in a hidden div, so the two signals that drive the desktop fill + coverage gates were invisible during tuning. Add a Doc Fill row, unhide the grid row, and stash docFillPercent in a ref so the per-frame quality payload (emitted after the contour block, where the value is out of scope) reports the live fill at the moment of capture. Debug-only; no behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each gate's setDebugInfo call replaced the whole object, so the panel only ever showed the subset the current frame's early-return path emitted - docFill blanked on the "Align document in frame" path, grid blanked on the move-closer/blur paths, etc. Route all emits through a mergeDebugInfo helper so the last-known docFill / grid / blur / glare stay visible together. Debug-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the hand-rolled back/close <button> pairs in both the desktop and mobile layouts with the shared <smileid-navigation> custom element, the same component used by the selfie capture screens. This brings localised aria labels (navigation.back / navigation.closeVerificationFrame), RTL arrow flipping, and consistent focus styling for free, and drops the duplicated SVG markup and the now-unused roundControlButtonStyle / desktopNavBtnStyle objects. The element's navigation.back / navigation.close events are bridged to the existing onBack / onClose host-event dispatchers via a ref + effect (mirroring SmartSelfieCapture). Per-layout chrome is set through the element's CSS custom properties: dark-on-light pills for the desktop card, translucent-white-on-dark for the fullscreen mobile camera. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
and doc auto-capture screen: replace the hand-rolled back <button> + BackArrowIcon with the shared <smileid-navigation hide-close> element, bridging its navigation.back event to handleBack via a ref + effect. Brings the localised back label and RTL arrow flip for free; the .back-nav class now only positions the element (the circle/hover/focus live in the element's own shadow DOM). Both document capture surfaces now use the same navigation component. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The desktop shutter (DesktopCaptureButton) rendered unconditionally and was only disabled on SUCCESS, so it was always tappable regardless of captureMode and the auto-capture timeout had no effect on it — unlike mobile, which gates its manual control on showManualButton. Render the desktop shutter only when showManualButton is true: shown for manualCaptureOnly, after the autoCaptureTimeout fallback fires in autoCapture, or on CV load failure; never in autoCaptureOnly. Auto-capture state is already conveyed by the video border, so the button is hidden (not just disabled) while unavailable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the manual-fallback timeout default from 10s to 20s so users get more time for auto-capture to succeed before the manual shutter appears. The effective default for the document flow is DocumentAutoCapture's `auto-capture-timeout` prop default (and the invalid-input fallback constant), which always override useCardDetection's own default — update both, plus the hook default for direct callers, the README, and the Storybook arg. Integrators can still override per the 3000–30000ms clamp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wrap document-type, auto-capture, and auto-capture-timeout attribute values in escAttr() with double quotes, matching the escaping already applied to the other forwarded attributes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- DocumentCaptureScreens: validate auto-capture-timeout as a positive number before interpolating into innerHTML (omit otherwise), with double quotes - README: fix broken ./AGENTS.md link and correct the OpenCV runtime URL/CSP origin to web-models.smileidentity.com to match opencvLoader - useCardDetection: gate the remaining MANUAL-capture and [ROI] debug logs behind IS_DEBUG_MODE - add Cypress e2e asserting auto-capture-enabled flips the live-capture element between <document-auto-capture> and legacy <document-capture> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Barnabas A Nsoh (ayinloya)
dismissed
Solomon Nsubuga (solnsubuga)’s stale review
The base branch was changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Summary
Redesigns the desktop layout of
<document-auto-capture>to mirror the legacy<document-capture>element, and fixes desktop distance detection so a card that fills the visible frame is recognized as close enough.UI (desktop)
useCardDetection) is unchanged structurally — only the desktop visual shell differs from the mobile fullscreen view.Distance detection fix
Previously the desktop view reported "Move document closer" even when the card filled the visible frame.
78/98, mobile75/95(mobile behavior unchanged).Misc
Test plan
🤖 Generated with Claude Code
PR Type
Enhancement, Bug fix
Description
Redesign desktop auto-capture UI to match legacy document-capture layout
Fix desktop distance detection by using visible video box as ROI
Add per-device fill thresholds and skip grid check on desktop
Default example app to auto-capture enabled
Diagram Walkthrough
File Walkthrough
main.js
Default auto-capture to enabled in exampleexample/main.js
easier testing
DocumentAutoCapture.tsx
Desktop layout redesign with legacy-style UI componentspackages/web-components/lib/components/document/src/document-auto-capture/DocumentAutoCapture.tsx
minFillPercentandmaxFillPercentto device-specific settings(desktop 78/98, mobile 75/95)
blurThresholdfrom 130 to 60DesktopCaptureButtoncomponent with SVG concentric-circle designand progress ring
title, feedback text, and capture button
skipGridCheckoption touseCardDetectionfor non-mobile devicestheme-colorandtitleprops to the inner componentuseCardDetection.ts
Fix distance detection and add desktop ROI/fill logicpackages/web-components/lib/components/document/src/document-auto-capture/hooks/useCardDetection.ts
skipGridCheckoption to bypass grid-cell validation on desktopskipGridCheckis true, ROI equals the visible video box with 4%margin instead of fixed 64px inset
presence edges
expansion factor
settings.minFillPercent/settings.maxFillPercentCAPTURE_MISS_TOLERANCEfrom 8 to 20 and default fillconstants from 65/90 to 75/95
fillRatiothreshold for valid card outline from 0.65 to 0.75skipGridCheckis active