Stress-Testing Memory: LLM-Driven Synthetic Data Generation & Cognitive Benchmarking

Benchmarking database systems is a well-understood field. Tools like YCSB or pgbench generate structured keys and random payloads to stress-test throughput and disk I/O.

However, benchmarking a cognitive memory system like Spector requires more than random bytes. A memory database relies on the relationships between data points: semantic associations, temporal ordering, and co-activation patterns.

To benchmark Spector under realistic conditions, we designed a pipeline to generate an Entity-Dense Baseline Dataset of 10,000 synthetic memories simulating the daily life of an autonomous AI agent.

1. The Anatomy of a Cognitive Memory Dataset

For a benchmark dataset to exercise all four layers of Spector’s memory, it must contain:

1. Temporal continuity: Events that build upon previous events day-by-day (to test temporal chains).
2. Dense entities: Repeated references to specific people, repositories, modules, and bugs (to test Hebbian co-activation and Entity Graphs).
3. Semantic drift: Evolving discussions on topics over time (to test semantic compaction and LLM-driven consolidation).

To achieve this, we developed a synthetic generator that models a virtual software engineer's activity over a simulated 3-year timeline.

[Day N-1 State] ────> [Ollama Generator Prompt] ────> [Day N Memory JSON]
                             ▲
                             │ (Injects entities like 'Spector', 'API-402')

2. Generating 10,000 Memories with Local LLMs

We utilized a local Qwen-3.5 model via Ollama to run the generation pipeline. The generator script steps through time, generating daily work logs, codebase reflections, and communication snippets.

To ensure high-quality entity density, the generator prompt enforces constraints:

• Entity injection: Every generated log must explicitly reference at least 3 entities from a predefined set (e.g., specific developer names, microservice modules, or ticket IDs).
• Context dependency: The prompt for Day N receives a summarized context of Day N-1 to enforce narrative continuity.

Example Generator Code Pattern (PowerShell Orchestrator):

# scripts/generate-entity-dense-dataset.ps1
param (
    [int]$TotalMemories = 10000,
    [string]$Model = "qwen3.5:latest"
)

# Step-by-step synthetic simulation loop
for ($day = 1; $day -le $TotalDays; $day++) {
    $Prompt = BuildPrompt -Day $day -PreviousContext $context
    $Response = Invoke-OllamaChat -Model $Model -Prompt $Prompt
    
    # Save structured memory JSON
    Write-MemoryRecord -Data $Response
    $context = ExtractContextSummary -Response $Response
}

3. The Cognitive Benchmarking Harness

Once the 10,000 synthetic records are generated, they are ingested into Spector. The benchmarking harness then executes three distinct recall modes to measure performance:

During execution, Spector outputs detailed telemetry to detail.csv, tracking:

• latency_ms
• graph_expanded_candidates
• graph_expanded_relevant
• delta_nDCG (evaluating how much the hypergraph improved results over a standard vector search)

4. Conclusion: What We Discovered

By stress-testing Spector with this entity-dense dataset, we identified and fixed critical data-integrity edge cases in our temporal chaining and co-activation persistence managers.

More importantly, it proved that graph-guided memory expansion yields up to a 34% increase in recall quality (nDCG) compared to vector search alone, while maintaining sub-millisecond retrieval overhead.

You can inspect the benchmark harness and download the synthetic data generation script on GitHub. Let’s build the future of AI cognition together.