Configuration

Configuration

Memory-Spark uses a 19-block JSON configuration schema.

Configuration Structure

{
  "spark": { ... },
  "embed": { ... },
  "rerank": { ... },
  "fts": { ... },
  "chunk": { ... },
  "search": { ... },
  "hyde": { ... },
  "mmr": { ... },
  "temporal": { ... },
  "pools": { ... },
  "gate": { ... },
  "cache": { ... },
  "ner": { ... },
  "zeroShot": { ... },
  "summarizer": { ... },
  "stt": { ... },
  "ocr": { ... },
  "glmOcr": { ... },
  "reference": { ... }
}

Spark Services

{
  "spark": {
    "host": "10.99.1.1",
    "embed": { "port": 18091, "timeoutMs": 30000 },
    "rerank": { "port": 18096, "timeoutMs": 60000 },
    "llm": { "port": 18080, "timeoutMs": 120000 },
    "ner": { "port": 18112, "timeoutMs": 30000 },
    "zeroShot": { "port": 18113, "timeoutMs": 30000 },
    "summarizer": { "port": 18110, "timeoutMs": 30000 },
    "stt": { "port": 18094, "timeoutMs": 60000 },
    "ocr": { "port": 18097, "timeoutMs": 60000 },
    "glmOcr": { "port": 18080, "timeoutMs": 60000 }
  }
}

Embedding

{
  "embed": {
    "provider": "spark",
    "model": "llama-embed-nemotron-8b",
    "dimensions": 4096,
    "batchSize": 32,
    "instructionAware": true,
    "defaultInstruction": "Represent the semantic meaning of this text for retrieval"
  }
}

Provider Options

Provider	Description	Auth Required
spark	Local DGX Spark	No
openai	OpenAI text-embedding-3	API key
voyage	Voyage AI	API key
cohere	Cohere embed	API key

Instruction-Aware Embedding

The Nemotron-8B embedder is instruction-tuned. Queries should be prefixed:

Instruct: {task}\nQuery: {actual_query}

Documents are embedded without prefix.

Reranking

{
  "rerank": {
    "provider": "spark",
    "model": "llama-nemotron-rerank-1b-v2",
    "topN": 40,
    "batchSize": 8,
    "queryNormalization": true,
    "logitRecovery": true
  }
}

Query Normalization

The reranker is trained on Q&A pairs. Declarative claims are converted:

• "The system uses RRF for fusion" → "Does the system use RRF for fusion?"

Logit Recovery

Applies inverse sigmoid to restore score spread:

logit(p) = ln(p / (1 - p))

Reranker Gate

{
  "gate": {
    "mode": "hard",
    "confidentThreshold": 0.08,
    "tiedThreshold": 0.02
  }
}

Gate Modes

Mode	Description	Use Case
off	Always run reranker	Maximum accuracy
hard	Binary skip/run	Production default
soft	Blend based on σ	Experimental

Threshold Tuning

• Confident (σ > 0.08): Top results already well-separated
• Tied (σ < 0.02): All results have similar scores; reranker adds noise
• Ambiguous (0.02–0.08): Reranker provides discriminative signal

Full-Text Search

{
  "fts": {
    "enabled": true,
    "tokenizer": "stemmed",
    "bm25": {
      "k1": 1.2,
      "b": 0.75
    },
    "sigmoidMidpoint": 10.0,
    "sigmoidSteepness": 0.5
  }
}

Sigmoid Calibration

BM25 scores (5–20+) are normalized via sigmoid:

normalized = 1 / (1 + e^(-steepness × (score - midpoint)))

Midpoint 10.0 calibrated to SciFact distribution.

Hybrid Merge

{
  "hybrid": {
    "method": "rrf",
    "rrfK": 60,
    "vectorWeight": 1.0,
    "ftsWeight": 1.0,
    "adaptiveWeights": true
  }
}

RRF Formula

RRF_score(d) = Σ (weight_i / (k + rank_i(d)))

Scale-invariant fusion that doesn't require score normalization.

HyDE

{
  "hyde": {
    "enabled": false,
    "timeoutMs": 4000,
    "maxTokens": 256,
    "replacement": true,
    "qualityGate": {
      "minLength": 50,
      "rejectPatterns": ["I cannot", "I'm unable", "As an AI"]
    }
  }
}

Replacement vs Averaging

• Replacement: Use HyDE embedding directly
• Averaging: Blend query + HyDE embeddings

Replacement performs better for factual retrieval.

MMR

{
  "mmr": {
    "enabled": true,
    "lambda": 0.9,
    "metric": "cosine",
    "fallbackMetric": "jaccard"
  }
}

Lambda Tuning

• λ → 1.0: Maximize relevance
• λ → 0.0: Maximize diversity
• Default 0.9: Favor relevance with mild diversity

Temporal Decay

{
  "temporal": {
    "enabled": true,
    "alpha": 0.005,
    "referenceField": "timestamp"
  }
}

Decay Formula

decayed_score = score × e^(-α × days_elapsed)

α = 0.005 gives half-life of ~138 days.

Chunking

{
  "chunk": {
    "strategy": "hierarchical",
    "childSize": 512,
    "childOverlap": 64,
    "parentSize": 2048,
    "parentOverlap": 256
  }
}

Hierarchical Chunking

• Child chunks: Retrieved by search (precise)
• Parent chunks: Injected into context (broad)

Best of both: retrieval precision + context breadth.

Vector Search

{
  "search": {
    "topK": 20,
    "minScore": 0.3,
    "indexType": "ivf_pq",
    "ivfPq": {
      "numPartitions": 10,
      "numSubVectors": 64,
      "bitsPerSubVector": 8
    }
  }
}

IVF_PQ Parameters

• Partitions: Cluster count (higher = finer granularity)
• Sub-vectors: Compression ratio (higher = smaller index)
• Bits: Quantization precision (8 = 256 centroids)

Memory Pools

{
  "pools": {
    "agent_memory": { "weight": 1.0, "temporalDecay": true },
    "agent_mistakes": { "weight": 1.6, "temporalDecay": true },
    "shared_rules": { "weight": 1.2, "temporalDecay": false },
    "shared_knowledge": { "weight": 1.0, "temporalDecay": false },
    "sessions": { "weight": 0.8, "temporalDecay": true }
  }
}

Cache

{
  "cache": {
    "embedding": {
      "enabled": true,
      "maxSize": 10000,
      "ttlMs": 86400000
    },
    "query": {
      "enabled": true,
      "maxSize": 1000,
      "ttlMs": 3600000
    }
  }
}