Eating Our Own Dog Food: How Spector Cognitive Memory Powers Our Autonomous AI Team

At Spectrayan, we don't just build open-source AI infrastructure; we run our entire organization using it.

Spectrayan is managed by an autonomous team of specialized AI personas. From product roadmap design (@nova) and system architecture (@titan) to code execution (@forge), testing (@sentinel), and deployment (@nexus), our engineering lifecycle is entirely automated.

But orchestrating a team of specialized AI agents working across multiple codebases introduces a major bottleneck: context fragmentation.

Here is how we use our own cognitive memory engine, Spector, to enable seamless collaboration, persistent context, and human-like memory retention across our AI development team.

The Bottleneck: The State-Sharing Problem in Multi-Agent Systems

When multiple LLM-based agents work together, they face three primary challenges:

1. Context Window Truncation: As agent chats grow longer, critical engineering decisions, compiler errors, or user preferences get pushed out of the active context window.
2. Knowledge Silos: A bug fix discovered by the developer (@forge) isn't automatically known by the QA engineer (@sentinel) or the documentation writer (@codex).
3. Fragile Handovers: When one agent finishes a task and passes it to another, transferring the entire chat history is too token-expensive, yet passing a short text summary misses critical execution details.

Standard vector databases fail to solve this. They are static, lack temporal awareness, and do not understand the relationships between different pieces of engineering knowledge.

The Dual-Layer Memory Architecture

To solve these coordination challenges, we designed a dual-layer memory system. Every AI agent in the Spectrayan organization operates with two distinct memory spaces:

1. Individual Brain & Local Memory: Each agent has its own private context and specialized cognitive profile. E.g. @forge remembers Java vector search coding standards, while @sentinel remembers testing suites and quality gates.
2. Shared Organization Memory (Spector Memory Core): A shared, team-wide cognitive database. The agents recall organization-level memories (e.g. CEO decisions, shared guidelines, code standards, status reports from other agents) and persist their own handovers and updates back to this shared space.

How We Coordinate via Spector Memory

We equipped every persona on the Spectrayan team with tools to read from and write to a shared Spector Memory instance via the Model Context Protocol (MCP).

1. Local Memory for Persona Autonomy

Each agent's local memory acts as its private scratchpad and cognitive boundaries. When @forge is writing code, its local memory holds details about active file paths, local test output files, and current compilation targets. This keeps the agent's attention field narrow and execution-oriented.

2. The Organization Handover Protocol (Episodic Context)

When transitioning tasks between personas, we use the shared Organization Memory to retrieve and store execution state. Before beginning work, an agent triggers memory_recall to load the history of the current task. When the task is complete, the agent triggers memory_remember to record the status, decisions, and open items to the shared organization pool.

// Example of a status report stored by @forge in Spector Memory
{
  "text": "[@forge] Status Report: Refactored Contact Form to use Angular Material. Solved layout shift by removing custom SCSS overrides.",
  "tier": "EPISODIC",
  "tags": "status-report, developer, forge, spectrayan"
}

3. Reinforcing Decisions (Semantic Weighting)

When our CEO specifies a design preference or makes a strategic product decision, it is written to Spector with a high interest score and a ceo-decision tag. Spector's spreading activation algorithm ensures that whenever any agent works on related modules, these decisions are recalled instantly, preventing agents from violating product guidelines.

4. Graceful Decay of Technical Noise

Not all context is worth keeping forever. A compiler warning that was resolved minutes ago doesn't need to bloat future prompt windows. Because Spector implements biologically-inspired Hebbian decay, transient debugging details naturally fade over time, while core architectural decisions are reinforced and consolidated.

The Results: Zero-GC Performance & Infinite Memory

By using Spector as our team's cognitive backbone:

• Token Efficiency: Agents only load contextually relevant memories, reducing prompt bloat by over $60\%$.
• Continuity: Spawning new agents or resuming paused workflows takes milliseconds, with no state loss.
• Collective Intelligence: Learnings from a bug fix in one repository are immediately available to agents working across the rest of the Spectrayan portfolio.

We designed Spector to give AI agents a memory that behaves like our own. Seeing it coordinate our own autonomous engineering organization is the ultimate proof of concept.

Spector is open-source. If you are building multi-agent systems or need to give your AI agents a persistent, biologically-inspired memory, check out the repository on GitHub.

What are your thoughts on agent memory? Let's discuss in the comments!