Build error

[Xcc313r4n7]

Running into this error when building:

/home/cinder/git/TensorSharp/TensorSharp.GGML.Native/ggml_ops_mamba2.cpp: In function ‘int TSGgml_NemotronMamba2PrefillF32(tsg::TensorView2DDesc, tsg::TensorView2DDesc, void*, int, void*, int, void*, void*, void*, void*, void*, void*, int, int, int, int, int, int, float)’:
/home/cinder/git/TensorSharp/TensorSharp.GGML.Native/ggml_ops_mamba2.cpp:599:46: error: too few arguments to function ‘ggml_tensor* ggml_ssm_scan(ggml_context*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, int64_t)’
599 | ggml_tensor* scan = ggml_ssm_scan(ctx, ssm_state_t, x, dt, a_t, b, c, ids_t);
| ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/cinder/git/TensorSharp/TensorSharp.GGML.Native/ggml_ops_internal.h:33,
from /home/cinder/git/TensorSharp/TensorSharp.GGML.Native/ggml_ops_mamba2.cpp:11:
/home/cinder/git/TensorSharp/TensorSharp.GGML.Native/../ExternalProjects/ggml/include/ggml.h:2454:35: note: declared here
2454 | GGML_API struct ggml_tensor * ggml_ssm_scan(
| ^~~~~~~~~~~~~
/home/cinder/git/TensorSharp/TensorSharp.GGML.Native/ggml_ops_mamba2.cpp: In function ‘int TSGgml_NemotronMamba2DecodeF32(uint64_t, tsg::TensorView2DDesc, tsg::TensorView2DDesc, void*, int, void*, int, int, int, void*, void*, void*, void*, void*, void*, int, int, int, int, int, int, float)’:
/home/cinder/git/TensorSharp/TensorSharp.GGML.Native/ggml_ops_mamba2.cpp:956:46: error: too few arguments to function ‘ggml_tensor* ggml_ssm_scan(ggml_context*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, int64_t)’
956 | ggml_tensor* scan = ggml_ssm_scan(ctx, ssm_state_t, x, dt, a_t, b, c, ids_t);
| ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cinder/git/TensorSharp/TensorSharp.GGML.Native/../ExternalProjects/ggml/include/ggml.h:2454:35: note: declared here
2454 | GGML_API struct ggml_tensor * ggml_ssm_scan(
| ^~~~~~~~~~~~~
make[3]: *** [CMakeFiles/GgmlOps.dir/build.make:429: CMakeFiles/GgmlOps.dir/ggml_ops_mamba2.cpp.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/Makefile2:209: CMakeFiles/GgmlOps.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:216: CMakeFiles/GgmlOps.dir/rule] Error 2
make: *** [Makefile:170: GgmlOps] Error 2

[Xcc313r4n7]

patched the offending file to complete the build with:
sed -i 's/ggml_ssm_scan(ctx, ssm_state_t, x, dt, a_t, b, c, ids_t)/ggml_ssm_scan(ctx, ssm_state_t, x, dt, a_t, b, c, ids_t, 0)/g' TensorSharp.GGML.Native/ggml_ops_mamba2.cpp

Build completed

But now I'm trying to serve Qwen3.8-27B and getting the following:

info: TensorSharp.Server.Startup[1030]
Logging initialized: minimumLevel=Information fileLogging=True logDir=/home/cinder/git/TensorSharp/TensorSharp.Server/bin/logs
info: TensorSharp.Server.ModelService[1400]
Loading model Qwen3.8-27B-UD-Q4_K_XL.gguf (mmproj=mmproj-F16.gguf, backend=ggml_cuda, fullPath=/home/cinder/models/GGUF/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf, mmprojPath=/home/cinder/models/GGUF/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf)
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 23997 MiB):
Device 0: NVIDIA GeForce RTX 5090 Laptop GPU, compute capability 12.0, VMM: yes, VRAM: 23997 MiB
Backend: GgmlCuda
Model: qwen35, Layers=64, Hidden=5120, Heads=24, KVHeads=4, HeadDim=256, Vocab=248320
RoPE base=10000000, scale=1, eps=1E-06
SSM: dInner=6144, dState=128, nGroup=16, dtRank=48, convKernel=4, fullAttnInterval=4
MRoPE sections: [11, 11, 10, 0]
Layer types: 16 full attention, 48 recurrent (GatedDeltaNet)
NextN/MTP: 1 draft block(s) at layer 64 (excluded from main stack)
Loading model weights... done (360 F32 tensors, 506 quantized tensors)
Quantized: 17072 MB (17072 MB file-backed), F32: 10 MB
Separate projections: 17 mixed-quant Q/K/V sets preserved
Device-resident quantized weights: 16390 MB across 505 tensors
Context length: using GGUF metadata qwen35.context_length=262144.
Initial GgmlCuda KV cache allocation: 8192 tokens (grows on demand up to 262144).
NextN/MTP draft head ready (layer 64, moe=no, ownHead=no)
Qwen3.5 Vision encoder: imageSize=768, patchSize=16, hidden=1152, intermediate=4304, heads=16, blocks=27, projDim=5120, mergeSize=2, gridPerSide=48, ropeTheta=10000
Loading vision encoder weights... done (334 tensors)
info: TensorSharp.Server.ModelService[1401]
Loaded model Qwen3.8-27B-UD-Q4_K_XL.gguf (architecture=qwen35, backend=ggml_cuda, modelBytes=17923394624, mmproj=mmproj-F16.gguf, mmprojBytes=927607488) in 3715.2 ms
info: TensorSharp.Server.Startup[1401]
Startup model loaded: Qwen3.8-27B-UD-Q4_K_XL.gguf architecture=qwen35 backend=ggml_cuda mmproj=mmproj-F16.gguf
Warming up kernels (decode + prefill)...
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at TensorSharp.Models.Qwen35Model.FFNCachedFused(Tensor residual, Tensor postNormW, Int32 layer, Int32 seqLen) in /home/cinder/git/TensorSharp/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs:line 3317
at TensorSharp.Models.Qwen35Model.RecurrentBlock(Tensor hidden, Int32 layer, Int32 seqLen, Int32 startPos) in /home/cinder/git/TensorSharp/TensorSharp.Models/Models/Qwen35/Qwen35Model.GatedDeltaNet.cs:line 1016
at TensorSharp.Models.Qwen35Model.RunPerOpLayerLoop(Tensor hidden, Int32 seqLen, Int32 startPos) in /home/cinder/git/TensorSharp/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs:line 1534
at TensorSharp.Models.Qwen35Model.RunCudaPrefillLayerLoop(Tensor hidden, Int32 seqLen, Int32 startPos) in /home/cinder/git/TensorSharp/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs:line 1590
at TensorSharp.Models.Qwen35Model.ForwardCore(Int32[] tokens) in /home/cinder/git/TensorSharp/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs:line 2057
at TensorSharp.Models.ModelBase.Forward(Int32[] tokens) in /home/cinder/git/TensorSharp/TensorSharp.Models/ModelBase.cs:line 5166
at TensorSharp.Models.ModelBase.WarmUpKernels() in /home/cinder/git/TensorSharp/TensorSharp.Models/ModelBase.cs:line 5291
at TensorSharp.Server.Hosting.StartupModelLoader.LoadIfConfigured(ServerHostingOptions options, ModelService modelService, String configuredBackendInput, ILogger logger) in /home/cinder/git/TensorSharp/TensorSharp.Server/Hosting/StartupModelLoader.cs:line 81
at Program.

$(String[] args) in /home/cinder/git/TensorSharp/TensorSharp.Server/Program.cs:line 274
fish: Job 1, './TensorSharp.Server/bin/Tensor…' terminated by signal SIGABRT (Abort)

[Xcc313r4n7]

Tried with both ggml_cuda and ggml_vulkan as a fallback but this seems to be related to Qwen3.8-27B's GatedDeltaNet

[zhongkaifu]

@Xcc313r4n7 This issue has been fixed by this PR: #141 It has been merged into main.

[Xcc313r4n7]

Got everything built cleanly and serving the model now, but when I access it on the network with nanobot harness via /v1/chat/completions, all I'm getting back is a string of text that looks like this:

using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using TensorSharp.Models;

namespace TensorSharp.Server.RequestParsers
{
    /// <summary>
    /// Parsers for the <c>tools</c> field across all three protocols. Returns
    /// null when no tools are declared so downstream code can short-circuit
    /// (the parsing path is identical, only the wrapping differs: Ollama nests
    /// the function under <c>"function"</c>; OpenAI additionally requires
    /// <c>"type": "function"</c>).
    /// </summary>
    internal static class ToolFunctionParser
    {
        public static List<ToolFunction> ParseOllama(JsonElement body)
        {
            if (!body.TryGetProperty("tools", out var toolsEl) || toolsEl.ValueKind != JsonValueKind.Array)
                return null;

            var tools = new List<ToolFunction>();
            foreach (var toolEl in toolsEl.EnumerateArray())
            {
                if (!toolEl.TryGetProperty("function", out var fnEl))
                    continue;
                var tf = ParseFunction(fnEl);
                if (tf != null)
                    tools.Add(tf);
            }
            return tools.Count > 0 ? tools : null;
        }

        public static List<ToolFunction> ParseOpenAI(JsonElement body)
        {
            if (!body.TryGetProperty("tools", out var toolsEl) || toolsEl.ValueKind != JsonValueKind.Array)
                return null;

            var tools = new List<ToolFunction>();
            foreach (var toolEl in toolsEl.EnumerateArray())
            {
                string type = toolEl.TryGetProperty("type", out var t) ? t.GetString() : "function";
                if (type != "function") continue;
                if (!toolEl.TryGetProperty("function", out var fnEl)) continue;

                var tf = ParseFunction(fnEl);
                if (tf != null)
                    tools.Add(tf);
            }
            return tools.Count > 0 ? tools : null;
        }

        /// <summary>
        /// Parse the Responses API's flat <c>tools</c> shape: each entry is
        /// <c>{type:"function", name, description, parameters}</c> directly
        /// (no nested <c>"function"</c> wrapper like Chat Completions uses).
        /// </summary>
        public static List<ToolFunction> ParseOpenAIResponses(JsonElement body)
        {
            if (!body.TryGetProperty("tools", out var toolsEl) || toolsEl.ValueKind != JsonValueKind.Array)
                return null;

            var tools = new List<ToolFunction>();
            foreach (var toolEl in toolsEl.EnumerateArray())
            {
                string type = toolEl.TryGetProperty("type", out var t) ? t.GetString() : "function";
                if (type != "function") continue;

                var tf = ParseFunction(toolEl);
                if (tf != null)
                    tools.Add(tf);
            }
            return tools.Count > 0 ? tools : null;
        }

        private static ToolFunction ParseFunction(JsonElement fnEl)
        {
            var tf = new ToolFunction
            {
                Name = fnEl.TryGetProperty("name", out var n) ? n.GetString() : "",
                Description = fnEl.TryGetProperty("description", out var d) ? d.GetString() : ""
            };

            if (fnEl.TryGetProperty("parameters", out var paramsEl))
            {
                if (paramsEl.TryGetProperty("properties", out var propsEl) &&
                    propsEl.ValueKind == JsonValueKind.Object)
                {
                    tf.Parameters = new Dictionary<string, ToolParameter>();
                    foreach (var prop in propsEl.EnumerateObject())
                    {
                        var tp = new ToolParameter
                        {
                            Type = prop.Value.TryGetProperty("type", out var pt) ? pt.GetString() : "string",
                            Description = prop.Value.TryGetProperty("description", out var pd) ? pd.GetString() : null
                        };
                        if (prop.Value.TryGetProperty("enum", out var enumEl) && enumEl.ValueKind == JsonValueKind.Array)
                            tp.Enum = enumEl.EnumerateArray().Select(e => e.GetString()).ToList();
                        tf.Parameters[prop.Name] = tp;
                    }
                }
                if (paramsEl.TryGetProperty("required", out var reqEl) && reqEl.ValueKind == JsonValueKind.Array)
                    tf.Required = reqEl.EnumerateArray().Select(e => e.GetString()).ToList();
            }

            return tf;
        }
    }
}

[zhongkaifu]

@Xcc313r4n7 Sorry that I didn't get it. Can you please explain more about what issue did you get?

[Xcc313r4n7]

Seems to be an issue with tool parsing. When the harness sends a request to TensorSharp's open AI compatible endpoint, it doesn't recognize the request format as a valid chat completion. Instead of returning a chat response, it is returning raw HTML/ CSS instead of structured JSON data. The issue seems to come from ToolFunctionParser.cs. the harness is sending an array of objects in an enum block. Tensorsharp is forcing get string on every item in the array, so it can't parse correctly when it encounters a non string value. The result is I'm not getting back valid chat completion responses through the open AI compatible endpoint.

The bug here is an API validation conflict. The harness is sending a completely valid OpenAI tool spec, but TensorSharp's strict endpoint reader hasn't implemented parsing logic for that specific JSON structure yet.

According to my agent, the block in ToolFunctionParser.cs causing the issue is this portion:

   if (prop.Value.TryGetProperty("enum", out var enumEl) && enumEl.ValueKind == JsonValueKind.Array)
                            tp.Enum = enumEl.EnumerateArray().Select(e => e.GetString()).ToList();
                        tf.Parameters[prop.Name] = tp;
                    }
                }
                if (paramsEl.TryGetProperty("required", out var reqEl) && reqEl.ValueKind == JsonValueKind.Array)
                    tf.Required = reqEl.EnumerateArray().Select(e => e.GetString()).ToList();
            }

            return tf;
        }

[Xcc313r4n7]

My agent suggested this as the fix, I don't code so I can't verify but maybe this will give you more information as to what's happening:

                         // FIX: Safely parse array items even if they are numbers, booleans, or raw objects
                        if (prop.Value.TryGetProperty("enum", out var enumEl) && enumEl.ValueKind == JsonValueKind.Array)
                        {
                            tp.Enum = enumEl.EnumerateArray()
                                .Select(e => e.ValueKind == JsonValueKind.String ? e.GetString() : e.GetRawText())
                                .ToList();
                        }
                        
                        tf.Parameters[prop.Name] = tp;
                    }
                }
                
                // FIX: Safely parse required parameter names just in case they aren't explicitly strings
                if (paramsEl.TryGetProperty("required", out var reqEl) && reqEl.ValueKind == JsonValueKind.Array)
                {
                    tf.Required = reqEl.EnumerateArray()
                        .Select(e => e.ValueKind == JsonValueKind.String ? e.GetString() : e.GetRawText())
                        .Where(s => !string.IsNullOrEmpty(s))
                        .ToList();
                }
            }

            return tf;
        }

[Xcc313r4n7]

I haven't tried to patch it myself yet, but I could test the patch and recompile and see if that fixes the issue if that would help.

[zhongkaifu]

@Xcc313r4n7 Just made a fix: #155 for it and merged it. Could you please try it out? Thanks in advance.

[Xcc313r4n7]

Everything working now!