Local inference hangs then fails although Inference.Service.Agent is running (Windows 10 22H2)

[cyyssly]

Environment

• OS: Windows 10 22H2 (x64)
• VS Code:
• Foundry Toolkit for VS Code: 1.6.3
• Also reproduced on: 1.6.4
• GPU: NVIDIA CUDA GPU
• Runtime: Microsoft Foundry Local Runtime

---

Problem

Local inference never succeeds.

After selecting a local model and sending a simple prompt (e.g. "hello"), VS Code stays on Thinking... for a long time, then finally returns an inference endpoint error.

The issue is reproducible with multiple models.

---

Models Tested

qwen2.5-0.5b-instruct-cuda-gpu:4

Result:

Unable to call the qwen2.5-0.5b-instruct-cuda-gpu:4 inference endpoint.
Please check if the input or configuration is correct.

---

qwen3.5-9b-cuda-gpu:2

Result:

Failed loading model ...

Error encountered while parsing genai_config.json

Unknown value "patch_size"

---

Steps to Reproduce

1. Install Foundry Toolkit.
2. Download a local model (qwen2.5-0.5b-instruct-cuda-gpu:4).
3. Open VS Code Chat.
4. Select the local model.
5. Ask a simple question such as:

hello

6. Wait.

---

Expected Behavior

The local model should generate a response.

---

Actual Behavior

• Chat stays on Thinking...
• Runtime appears to start correctly.
• GPU memory is allocated.
• After a long delay, VS Code returns:

Unable to call the inference endpoint.

---

Diagnostics Performed

I have already verified all of the following:

• Reinstalled the models.
• Removed incomplete models.
• Verified model files are complete.
• Downgraded Foundry Toolkit from 1.6.4 to 1.6.3.
• Same issue on both versions.
• Runtime starts successfully.
• Inference.Service.Agent process is running.
• Port 5272 is listening.
• Restarted VS Code.
• Restarted the Runtime.
• Tried both the normal Chat window and a new Chat session.

---

Runtime Status

netstat -ano | findstr 5272

TCP    127.0.0.1:5272    LISTENING

tasklist /FI "PID eq <pid>"

Inference.Service.Agent.exe

The Runtime appears healthy.

GPU memory usage is approximately 18 GB while inference is running.

---

Extension Host Error

Developer Console continuously reports:

ERR [Extension Host]
The system cannot find the path specified.

---

AzureFoundryService Log

Failed to process model #0 on page 1.

The input does not contain any JSON tokens.

Expected the input to start with a valid JSON token.

---

Additional Notes

This does not appear to be a model download issue because:

• Runtime launches successfully.
• Port is available.
• GPU memory is allocated.
• Models were reinstalled.
• The issue reproduces on multiple Toolkit versions.

Regression

The issue reproduces on both Foundry Toolkit 1.6.3 and 1.6.4.
Downgrading the extension does not resolve the problem.

It appears that the VS Code extension cannot complete communication with the local inference endpoint even though the Runtime is active.
Any suggestions for collecting additional diagnostics would be appreciated.

[microsoft-github-policy-service]

Thank you for contacting us! Any issue or feedback from you is quite important to us. We will do our best to fully respond to your issue as soon as possible. Sometimes additional investigations may be needed, we will usually get back to you within 2 days by adding comments to this issue. Please stay tuned.