Snapdom powering AI agent

[pdufour]

I have been interested in WebGPU based llms for a while - transformers.js and webllm. Now with https://github.com/ngxson/wllama things have gotten even better. You can use any gguf on huggingface. I went and built this browser-use library powered by Snapdom, wllama, and some custom-written glue. Check it out here - https://pdufour.substack.com/p/writing-a-browser-use-agent-from. Thanks and let me know any feedback you have!

[tinchox5]

@pdufour Thank you very much for sharing this! I will check it out in detail because it is a field I am very interested in. I made a repository to use SnapDOM as AI Agent helpers (SnapEye - https://github.com/zumerlab/snapeye).

[tinchox5]

Thanks for the writeup and the snapDOM review, Paul. Really nice to see it integrated this thoughtfully, and the drift debugging was a great read.

Reading your project I think that there is one angle that might be worth exploring: you're using snapDOM just as a screenshot tool, but it reconstructs the DOM (full tree, computed styles, embedded assets) before it ever touches the canvas, and that intermediate object can be exposed through plugins.

That might let you sidestep the "send the DOM (too big) vs send an image (no semantics)" tradeoff. A plugin could give you both: the screenshot for the VLA, plus a compact JSON of just the interactive elements with their true bounding boxes. Could be a cleaner path around the pixel-drift you ran into, since the model would be handed the coordinate instead of inferring it.

A few other directions if you ever feel like experimenting: exporting the captured HTML scoped to just the target element (styles inlined) instead of the whole page, to cut down bloat; or, since state is serializable, chaining multi-frame captures for before/after an action, which could help when debugging why a step failed. You could even send animated GIFs or short videos made with a custom plugin to show the real dynamics of an interaction.

The plugin docs are published if you want to dig in.

Thank you again

[tinchox5]

@pdufour, could you take a look at my review? I’d love to hear what you think.

[pdufour]

Thanks so much for the feedback and great ideas, let me take some time to read through it thoroughly!

[pdufour]

That might let you sidestep the "send the DOM (too big) vs send an image (no semantics)" tradeoff. A plugin could give you both: the screenshot for the VLA, plus a compact JSON of just the interactive elements with their true bounding boxes. Could be a cleaner path around the pixel-drift you ran into, since the model would be handed the coordinate instead of inferring it.

This is an interesting idea, I wonder how many situations this would handle. I would think if someone is using a browser-use agent typically they just want to interact with the web page. Could use that as the primary approach and then fallback to the vision approach, that could be cool.

I definitely want to get videos working like you said, snapdom is very quick so it's definitely possible to stream them in realtime. It's just the LLM that is the bottleneck here. Lots of cool things to experiment with!

I think I need some good use cases to anchor the work around, that would probably be when I continue doing this work.

Thanks for all the feedback!