Aeon: High-Performance Neuro-Symbolic Memory Management for Long-Horizon LLM Agents

# Abstract

Large Language Models (LLMs) are fundamentally constrained by the quadratic computational cost of self-attention and the “Lost in the Middle” phenomenon, where reasoning capabilities degrade as context windows expand. Existing solutions — primarily “Flat RAG” architectures relying on vector databases — treat memory as an unstructured bag of embeddings, failing to capture the hierarchical and temporal structure of long-horizon interactions.

This paper presents Aeon, a Neuro-Symbolic Cognitive Operating System that redefines memory as a managed OS resource. Aeon structures memory into a Memory Palace (a spatial index implemented via Atlas, a SIMD-accelerated clustered vector index) and a Trace (a neuro-symbolic episodic graph). The architecture introduces three advances:

1. Symmetric INT8 scalar quantization, achieving 3.1× spatial compression and 5.6× math acceleration via NEON SDOT intrinsics.
2. A decoupled Write-Ahead Log (WAL) ensuring crash-recoverability with statistically negligible overhead (< 1%).
3. A Sidecar Blob Arena eliminating the prior 440-character text ceiling via an append-only, mmap-backed blob file with generational garbage collection.

The Semantic Lookaside Buffer (SLB) exploits conversational locality to achieve sub-5 µs retrieval latencies, with INT8 vectors dequantized to FP32 on cache insertion to preserve L1-resident lookup performance. Benchmarks on an Apple M4 Max show 4.70 ns INT8 dot-product latency, 3.09 µs tree traversal at 100K nodes (3.4× over FP32), and a P99 read latency of 750 ns under hostile 16-thread contention via epoch-based reclamation.

# 1. Introduction

The rapid evolution of LLMs has been defined by a relentless scaling of parameters and training data, yet the underlying architecture remains bound by the Context Bottleneck. Self-attention imposes quadratic time and space complexity, O(N²), in the input sequence length. Sparse attention, RingAttention, and hardware-aware kernel fusion have pushed context windows past one million tokens, but the utility of that context does not scale linearly: reasoning quality degrades over extended horizons, a failure widely characterised as being “Lost in the Middle” [7]. As autonomous agents pursue objectives spanning days or weeks, reliance on transient, volatile context windows becomes untenable. A model cannot simply attend to all of history; it must select what is potentially relevant before attention is applied.

The prevailing industry response has been Retrieval-Augmented Generation (RAG). In its most common form — “Flat RAG” — information preservation is offloaded to vector databases performing Approximate Nearest Neighbor (ANN) search over unstructured lists of embeddings. Effective for one-shot question answering, Flat RAG fails to model the structure of extended interaction: it treats memory as a featureless plane (a “bag of vectors”) in which the temporal evolution of a conversation, the causal lineage of decisions, and the hierarchy of concepts are all lost. This failure mode is termed Vector Haze: retrieving semantically similar but episodically disjointed facts that confuse rather than aid the agent.

Aeon proposes a paradigm shift — from memory as a passive database retrieval problem to memory as an active resource management problem inside a Cognitive Operating System. Allocation becomes the deliberate writing of new semantic concepts into a structured Atlas; paging becomes the loading of relevant semantic clusters into a Semantic Lookaside Buffer for immediate, low-latency access; and context switching is re-framed as deterministic movement between branches of a decision tree.

# Contributions

1. Atlas with INT8 quantization. A memory-mapped, hierarchical index. Symmetric INT8 scalar quantization becomes a first-class storage format, reducing per-node footprint from 3,392 B (FP32) to 1,088 B (INT8) at D = 768 — a 3.1× disk compression ratio. The INT8 dot product via ARM NEON SDOT intrinsics reaches 4.70 ns per comparison, a 5.6× acceleration over the FP32 kernel.
2. Write-Ahead Log. A crash-recovery mechanism with a 3-step lock ordering protocol that decouples disk flush latency (wal_mutex_) from RAM delta-buffer updates (delta_mutex_). Enabling the WAL adds less than 1% insert latency.
3. Sidecar Blob Arena. An append-only, mmap-backed blob file removing the 440-character text ceiling for episodic trace events. The 512-byte TraceEvent struct keeps a 64-byte inline text_preview (one CPU cache line) while offloading full-length LLM summaries to a generationally garbage-collected sidecar file.
4. Semantic Lookaside Buffer. A predictive cache exploiting conversational locality for sub-5 µs retrieval. INT8-stored vectors are dequantized to FP32 on SLB insertion to preserve L1-resident hit performance.
5. Epoch-Based Reclamation (EBR). A lock-free read path guaranteeing that concurrent readers never observe torn or unmapped memory during file growth. Under hostile 16-thread contention, P99 read latency is 750 ns.

# 2. Related Work

Retrieval-Augmented Generation. The dominant grounding paradigm, typically built on Dense Passage Retrieval [6] with ANN indices such as FAISS [5] or HNSW [8]. These are “Flat RAG” systems: a single monolithic vector space where every query is independent. Their primary limitation is Vector Haze — as memory grows, so does the probability of retrieving semantically similar but contextually irrelevant facts. Aeon constrains the search space to the agent’s active context region within the Atlas.

Memory-augmented LLMs. MemGPT [11] introduces an OS-like abstraction for managing context windows, but operates in “user space” (Python), relying on the LLM itself to manage memory calls through prompt engineering. Aeon moves that responsibility into a C++23 kernel and reaches sub-microsecond retrieval.

Neuro-symbolic knowledge graphs. GraphRAG [2] excels at multi-hop reasoning by making relationships explicit, but suffers from write latency and rigid extraction pipelines. Aeon’s Trace is hybrid: neural embeddings for nodes, symbolic edges for causal constraints.

Crash recovery in database systems. ARIES [9] established the principles of write-ahead logging. Aeon adapts them to vector index management with record-level CRC32 checksums and a 3-step lock ordering protocol that keeps flush latency off the insert hot path.

Epoch-based reclamation. Fraser’s EBR [3] defers deallocation until all readers advance past the retiring epoch. Aeon uses EBR with cache-line-padded epoch counters to eliminate false sharing under high contention.

Vector quantization. Symmetric scalar quantization maps floating-point vectors to fixed-point integers, cutting storage and compute. CSLS [1] has been proposed as a hub-penalising metric for nearest-neighbour search. Aeon integrates INT8 quantization at the storage layer and optionally applies a CSLS penalty during beam-search traversal.

# 3. System Architecture

Aeon implements a hybrid Cognitive Kernel designed to bridge high-performance systems programming and high-level AI reasoning.

# 3.1 Design philosophy: the Core–Shell model

• The Core (Ring 0) — C++23. All high-frequency, low-latency operations: vector similarity search, tree traversal, memory management, WAL flush, and EBR. It works directly on raw memory pages and uses hardware acceleration (SIMDe on x86-64, NEON SDOT on ARM64).
• The Shell (Ring 3) — Python. High-level control logic: LLM interaction, prompt engineering, and graph topology management.

The critical invariant is the Zero-Copy Constraint: data is never serialized between Core and Shell during normal operation. The Shell operates on read-only views of shared memory pages via nanobind.

# 3.2 The Atlas: spatial memory kernel

The Atlas is the foundation of long-term memory, a spatial index over semantic vectors. A memory node is defined as

N = { id, v, C, meta, s_q } (1)

where id is a unique 64-bit identifier, v the embedding vector (FP32 or INT8), C the set of child pointers, meta a fixed-size metadata block, and s_q the quantization scale factor (used only when v is INT8). The Atlas lives on persistent storage but is mapped into the process’s virtual address space via mmap; heap allocation is avoided for node data to guarantee contiguity.

# 3.2.1 INT8 symmetric scalar quantization

For a vector v ∈ ℝ^D:

s_q = max_i |v_i| / 127 (2)

q_i = clamp( round( v_i / s_q ), −127, 127 ) (3)

Remark 1. Equations (2)–(3) and all subsequent similarity computations assume input embeddings are strictly L2-normalized (‖v‖₂ = 1). Under that constraint the inner product ⟨u, v⟩ is mathematically equivalent to cosine similarity. Every embedding model used in Aeon enforces this invariant at ingestion time.

The scale factor is stored in the node header; if max_i |v_i| = 0 the scale defaults to 1.0 and the output is all zeros. The on-disk node stride differs between representations.

Table 1 — Node stride comparison at D = 768.

# 3.2.2 Greedy SIMD descent

Retrieval uses a Greedy SIMD Descent. For FP32 storage, cosine similarity is computed directly. For INT8 storage, the dot product is computed with NEON SDOT instructions:

raw_dot = Σ_i=0…D−1 q_i^(query) · q_i^(node) (4)

The final similarity follows from dequantization: sim = raw_dot × s_q^(query) × s_q^(node). Descent complexity is O(log_B M), where B is the effective branching factor and M the total node count.

Figure 1 — 768-d vector comparison latency (log scale). INT8 NEON SDOT (4.70 ns) achieves a 5.6× acceleration over FP32 (26.5 ns).

Figure 2 — Impact of INT8 storage on tree traversal: 3.4× faster descent at 100K nodes, alongside a 3.1× reduction in on-disk footprint.

# 3.2.3 Dynamic dimensionality

Embedding dimensions vary by model: 384 (MiniLM), 768 (e5-large), 1536 (OpenAI v3). Hard-coding the node stride couples kernel to model, so Aeon computes the stride at runtime. The AtlasHeader stores dimensionality D, metadata size M, and quantization type Q:

S = align_up( 64 + payload(D, Q) + M, 64 ) (5)

where payload(D, Q) is D × 4 bytes for FP32 or D × 1 bytes for INT8. A single compiled binary therefore serves any embedding model at any precision without recompilation, preventing model lock-in.

# 3.3 Write-Ahead Log

To gain crash-recoverability without sacrificing insert throughput, Aeon uses a decoupled WAL with a 3-step lock ordering protocol:

1. Serialize (no lock). The node is encoded into a byte buffer with a 16-byte WalRecordHeader holding a record type tag, payload size, and CRC32 checksum.
2. WAL flush (wal_mutex_ only). The record is written to the WAL file and flushed with fdatasync(). The delta_mutex_ guarding the RAM delta buffer is not held, so concurrent reads and writes proceed unblocked.
3. Apply to RAM (delta_mutex_ only). wal_mutex_ is released, delta_mutex_ acquired, and the node is memcpy’d into the flat byte-arena delta buffer.

Disk I/O (step 2) and RAM mutation (step 3) never contend on the same mutex, hiding flush latency behind the insert. On recovery the WAL is replayed sequentially, each CRC32 checksum validated, and torn tail records discarded. The WAL is truncated after each successful compaction.

# 3.4 Sidecar Blob Arena

The episodic Trace stores events as fixed-size 512-byte TraceEvent structs for O(1) random access. Earlier versions embedded text in a 440-character field, insufficient for LLM transcripts. Aeon replaces that field with a BlobRef indirection:

• A 64-byte text_preview field stores the first 63 characters inline, aligned to one CPU cache line for zero-cost listing.
• A blob_offset / blob_size pair points into an append-only, mmap-backed sidecar file (trace_blobs_genN.bin).
• The sidecar grows by 2× doubling (ftruncate → munmap → mmap) and serves zero-copy reads via std::string_view over the mapped region.

Generational garbage collection. During compaction a new generation blob file is created and only blobs referenced by non-tombstoned events are copied forward. The old generation is deleted once all EBR readers have advanced.

# 3.5 Double-buffered shadow compaction

To support real-time applications (for example game engines at 60 FPS), Aeon implements stutter-free garbage collection inspired by Redis BGSAVE [12]:

1. Microsecond freeze. The kernel locks, swaps the active delta_buffer with a frozen_delta_buffer, and snapshots state — under 10 µs.
2. Background copy. A background thread walks live (non-tombstoned) nodes in the mmap file and the frozen delta buffer, writing them contiguously into a new generation file (atlas_gen2.bin). The main thread keeps serving reads and accepts new writes into the fresh delta buffer.
3. Hot swap. When the copy completes, the kernel locks briefly to swap the MemoryFile handle to the new generation.
4. Cleanup. The old generation file is closed and deleted; the WAL is truncated, as all data is now durably persisted.

The main thread blocks only during steps 1 and 3 — under 10 µs combined; every expensive I/O operation happens in step 2 on a background thread. The same process applies to the Sidecar Blob Arena: dead blobs are simply not copied forward, achieving zero-overhead garbage collection.

# 3.6 The Trace: episodic context graph

The Trace supplies temporal and causal context as a DAG G = (V, E). The vertex set consists of heterogeneous TraceEvent nodes (V_user, V_system, V_concept). The edge set defines temporal edges (E_next) and reference edges (E_ref) that connect episodic nodes to their semantic grounding in the Atlas.

# 3.7 Trace Block Index

A naive linear scan of the Trace is O(|V|), prohibitive as history grows to 10⁵ events. Events are therefore grouped into TraceBlocks of fixed size B = 1024, each maintaining an incrementally updated centroid of its constituent embeddings. Retrieval is a two-phase SIMD scan:

1. Block scan. A SIMD search over block centroids selects the top-K most relevant time windows, at cost O(|V| / B).
2. Event scan. A deep scan runs only inside those top-K blocks, at cost O(K · B).

T_search = O( |V| / 1024 + K × 1024 ) (6)

With small K (typically 3–5), retrieval stays under 50 ms even for large traces by exploiting the temporal locality of semantic context.

# 3.8 The zero-copy interface

Aeon uses nanobind to expose C++ memory structures to Python: raw C++ pointers are wrapped in a Python capsule and reinterpreted as a read-only NumPy array buffer. Any attempt to mutate the underlying memory from the Shell raises a runtime exception.

# 4. The Semantic Lookaside Buffer

# 4.1 Theory: semantic locality

Traditional caching relies on address transparency, but in vector databases exact equality is rare. Aeon introduces Semantic Inertia: in continuous dialogue the topic vector t_i at turn i is highly correlated with t_i+1. Formally, P( dist(q_i+1, q_i) < ε ) ≈ 1.

# 4.2 Architecture

The SLB is a small contiguous ring buffer of fixed size K = 64, tuned to fit L1/L2 cache. Each entry stores a centroid c_node ∈ ℝ^D and a direct pointer to the full node in the Atlas.

FP32-only cache. The SLB stores exclusively FP32 vectors regardless of Atlas quantization; INT8 vectors are dequantized on insertion. This preserves the 3.56 µs cache-hit latency by avoiding dequantization on every scan — each scan performs K dot products, and the FP32 path already executes within L1/L2 boundaries.

Brute-force SIMD search. Because K is small, an exhaustive linear scan with AVX-512/NEON beats even a few steps of an O(log N) traversal, thanks to perfect hardware prefetching and zero pointer chasing.

# 4.2.1 Multi-tenant isolation

In multi-agent deployments a single shared SLB would leak semantic information between tenants. Aeon shards the SLB into 64 independent ring buffers, each with its own mutex, routed deterministically: shard_id = hash(session_id) mod 64. This lock-striping isolates both contention and semantic context — an agent in session A can never evict or read entries from session B — allowing the SLB to scale linearly beyond 100,000 concurrent sessions on a single node.

# 4.3 The speculative fetch algorithm

Given a query vector q, a hit threshold τ_hit, and the SLB buffer B, the lookup procedure scans the shard for the best-matching centroid and returns the corresponding node pointer when the similarity exceeds τ_hit; otherwise it returns NULL and the request falls through to Atlas tree traversal.

# 5. Experimental Setup

All measurements were collected on an Apple M4 Max using Google Benchmark [4], with the C++23 core compiled for ARM64 (NEON SDOT) and SIMDe [10] providing the x86-64 fallback path.

# 6. Results

# 6.1 Micro-benchmark: vector comparison

# 6.2 Macro-benchmark: traversal and compression

Table 2 — Atlas traversal and file size at N = 100,000.

The 3.4× traversal speedup combines the 5.6× faster per-comparison kernel with the fixed overhead of tree navigation (pointer chasing, branching). The 3.1× spatial compression directly reduces I/O bandwidth requirements.

# 6.3 WAL overhead

Table 3 — WAL overhead on insert latency (N = 10,000, FP32).

The WAL-enabled median (2.23 µs) sits within the standard deviation of the WAL-disabled measurement, confirming that the 3-step lock ordering decouples flush latency from the insert hot path: fdatasync() in step 2 executes concurrently with delta-buffer operations in step 3 across independent mutexes.

# 6.4 Scalability: 10K to 1M nodes

Figure 3 — Query latency vs. database size (log-log). Flat search scales linearly; the Atlas scales logarithmically, with INT8 adding a further 3.4× improvement.

Flat brute-force search scales linearly: 0.52 ms (10K) → 5.87 ms (100K) → 69.8 ms (1M). The FP32 Atlas scales logarithmically: 7.1 µs (10K, depth 2) → 10.5 µs (100K, depth 3) → 10.5 µs (1M, depth 4). The INT8 Atlas reduces this further to 1.82 µs (10K) and 3.08 µs (100K). At one million nodes the FP32 Atlas is more than 6,500× faster than flat scan (10.5 µs vs. 69.8 ms); each tree level partitions the search space by branching factor B = 64, yielding O(log_B N) complexity.

# 6.5 SLB cache performance and isolation

SLB cache-hit latency is 3.56 µs (median, 64-element scan). A cache miss against a warm Atlas — immediate fallback to tree traversal — costs 3.59 µs; the 0.03 µs delta confirms that the SLB scan and the first Atlas comparison are both L1-resident.

L1 residency proof. The BM_SLB_CacheIsolation benchmark measures scan latency as a function of cached items and shows linear scaling: 0.867 µs at 16 items, 1.70 µs at 32, and 3.46 µs at 64. Had any portion spilled to DRAM, scaling would exhibit a step function instead.

Under the “Conversational Walk” workload — simulating realistic chatbot sequences with high semantic locality — the SLB exceeds an 85% hit rate, giving an effective average latency of

L_eff = (0.85 × 3.56) + (0.15 × 10.5) ≈ 4.60 µs (7)

Figure 4 — Retrieval latency CDF (log scale). Warm Aeon resolves 85% of queries in under 5 µs via SLB hits, while HNSW clusters around 1.5 ms — a gap exceeding 300×.

# 6.6 EBR contention

Under hostile contention (15 reader threads, 1 writer thread, 100K iterations per reader across 16 hardware threads), cycle-precise measurement yields a mean of 210.8 ns, P50 of 167 ns, P99 of 750 ns, and P99.9 of 1,083 ns. A sub-microsecond P99 confirms that cache-line padding eliminates false sharing; writers retired 12,353 regions across 1.5M read samples with no observed torn reads.

# 6.7 Beam search and CSLS analysis

Table 4 — Beam search latency at N = 1,000,000 (pool of 1,000 unique query vectors).

The beam = 3 configuration scales sub-linearly (1.63× P50 ratio vs. beam = 1, against a theoretical 3× upper bound). Profiling the CSLS penalty showed a 27.7% latency reduction (30.2 µs vs. 41.8 µs), but strict node-visitation counting rejected the hypothesis of algorithmic pruning — nodes evaluated remained identical at 4.1 per query. The speedup is a superscalar branch-prediction artifact on Apple Silicon rather than a reduction in computational complexity: the CSLS hub penalty reshapes similarity scores into a more predictable branch pattern during beam selection, letting the M4 Max branch predictor achieve higher accuracy.

# 6.8 Trace garbage collection

Evaluated on a 100K-event store (≈67 MB):

• Tombstone scan: ≈100 µs per 100K events (sequential scan with flag check).
• Full compaction (GC ratio 0.5, retaining 50K events): 966 ms median wall-clock, 312 ms median CPU time.

The gap between wall-clock and CPU time is attributable to I/O — writing the new generation file and copying the generational blob arena — confirming that compaction is viable as a background operation that never blocks the insert/query path.

# 6.9 Zero-copy overhead

Transferring 10 MB of vector data from C++ to Python costs roughly 334 ns via nanobind zero-copy shared memory. Traditional serialization is dramatically worse: JSON at ≈318 ms (≈10⁶× slower) and Pickle at ≈32.3 ms (≈10⁵× slower).

Figure 5 — Cross-language memory transfer latency (10 MB payload, log scale). Zero-copy transfer eliminates object-boxing overhead.

# 6.10 Summary

Table 5 — Summary of Aeon performance characteristics.

# 7. Conclusion

Aeon is a Cognitive Operating System for long-horizon LLM agents. Its central argument is that LLM memory must be treated as an active resource-management task, governed by principles drawn from classical operating system kernels.

# 7.1 Key contributions

First, INT8 symmetric scalar quantization delivers a 3.1× disk compression ratio and 5.6× math acceleration via NEON SDOT, making edge deployment viable for knowledge bases that previously required hundreds of megabytes. Second, the decoupled WAL with 3-step lock ordering provides crash-recoverability at under 1% insert latency overhead, achieved by ensuring that disk I/O and RAM mutation never contend on the same mutex. Third, the Sidecar Blob Arena removes the 440-character text ceiling that constrained episodic trace storage, enabling full LLM transcript archival with generational garbage collection. Throughout, the SLB sustains sub-5 µs effective retrieval latency at 85%+ hit rates — the decision to dequantize INT8 vectors to FP32 on cache insertion preserves L1-resident lookup performance regardless of the underlying storage format.

# 7.2 Future work

Multi-modal vector representations. Aeon currently operates only on text embeddings. A natural extension is spatial co-location of audio, video, and structured-data embeddings inside the same Atlas index. The hierarchical tree is agnostic to semantic content; the open challenges are meaningful distance metrics across heterogeneous modalities and the variable dimensionality multi-modal encoders produce.

Hardware-enforced isolation for multi-tenancy. As Aeon grows to serve multiple users or agents in a shared deployment, cryptographic guarantees of memory isolation become necessary. Intel SGX and ARM CCA provide hardware enclaves that could enforce tenant boundaries at the memory page level, preventing even a compromised kernel from reading another tenant’s semantic memory — extending the OS analogy from process isolation to full memory protection.

# References

1. A. Conneau, G. Lample, M. Ranzato, L. Denoyer, H. Jégou. Word translation without parallel data. ICLR, 2018.
2. D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Ben-David, C. Larson. From local to global: a GraphRAG approach to query-focused summarization. arXiv:2404.16130, 2024.
3. K. Fraser. Practical Lock-Freedom. PhD thesis, University of Cambridge, 2004.
4. Google Inc. Google Benchmark: a microbenchmark support library. 2024.
5. J. Johnson, M. Douze, H. Jégou. Billion-scale similarity search with GPUs. arXiv:1702.08734, 2017.
6. V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W. Yih. Dense passage retrieval for open-domain question answering. arXiv:2004.04906, 2020.
7. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. Lost in the middle: how language models use long contexts. arXiv:2307.03172, 2023.
8. Y. A. Malkov, D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE TPAMI 42(4):824–836, 2018.
9. C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, P. Schwarz. ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM TODS 17(1):94–162, 1992.
10. E. Nemeth et al. SIMDe: implementations of SIMD instruction sets for systems which don’t natively support them. 2017.
11. C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, J. E. Gonzalez. MemGPT: towards LLMs as operating systems. arXiv:2310.08560, 2023.
12. S. Sanfilippo. Redis persistence demystified. 2009.