Nothing happens after displaying `nohup: redirecting stderr to stdout

[yudataguy]

this is from run.sh, nothing happens after showing the above text. I've removed --headless but no browser showing on screen. Any suggestion where should I check?

[MinorJerry]

Thank you for your interest in our work. To avoid losing the logs, we used the nohup command. So can you show what is recorded in the log? If you used run.sh, there should be a "test_task.log", and under the results folder there will be the corresponding task folder, in which there is also an "agent.log".

[yudataguy]

Thanks. Missed the log part, here's what I have in the log, looks like its due to Chrome version difference. Trying to fix that right now.

run.py", line 281, in main
driver_task = webdriver.Chrome(options=options)
...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 92
Current browser version is 130.0.6723.58 with binary path /usr/bin/google-chrome

Tried couple solutions, but still not able to get it working yet. Any suggestion?

[MinorJerry]

Sorry for the late reply.
It seems to be a chromedriver version mismatch. I wonder if it is related to the version of Selenium. Because in the latest version it is no longer required to give the path of chromedriver. So please check the Selenium version>4.10.
Or you can try to download the latest chromedriver from https://googlechromelabs.github.io/chrome-for-testing/ , and replace the old chromedriver.

[yudataguy]

@MinorJerry Thank you. That solved the problem for me. Looks like repo has old api

INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 404 Not Found"

I'll try fix this issue and try again. But with the release of Claude computer use API today, wondering if webvoyager method is gonna be replaced? Interesting to hear your thoughts on that.

[MinorJerry]

I see the Claude computer use pattern and am amazed by the rapid development of closed-source models. Claude is actually more general and human-like, not limited to just web pages but capable of interacting with various apps. I think it is a future research trend. Also, It uses coordinate points to operate in a specific position. These points have been mentioned in papers such as AppAgent and CogAgent. However, achieving excellent results remains quite challenging. I am skeptical whether Claude can navigate through a large number of steps on browsers without errors and maintain a good performance.

We design the Observation with set-of-mark in WebVoyager because we find the current abilities (visual grounding and reasoning) of multimodal models are not strong enough, so we have simplified observations as much as possible, allowing the model to simply select the elements it wants to access. From researchers's perspective, we find the multimodal setting is feasible. I also acknowledge that Agent frameworks like WebVoyager should make changes to adapt to new models. If new released models are able to provided the specific coordinate points, we can do more kinds of actions like dragging. However, prompt is easy to change, but achieving a more general framework may require strong engineering capabilities since the model only provides input and output while the implementation of actions still relies on the framework.

Furthermore, for current Open Source multimodal models, WebVoyager remains a good framework. We still do research on training open-source multimodal models with WebVoyager trajectories to break down some barriers of closed-source systems.

[yudataguy]

Thank you for the thoughtful responses. I had an idea of sending both screenshots and HTML code to help multimodal models better understand and interact with web environments(agent-e has the dom distillation to reduce the token). Claude's approach effectively simplifies this process. I'm really looking forward to seeing future research in this field. If your team is open to external contributors, I would be very interested in participating.