Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory

# Abstract

Agentic large language models (LLMs) on the Model Context Protocol (MCP) re-encode verbose tool schemas every turn, so prefill — quadratic in sequence length — dominates time-to-first-token (TTFT) as the tool registry grows.

Nexus’s primary lever is to decouple routing from the schema-prefill cost: an INT8 semantic lookaside buffer (SLB) with a calibrated cross-encoder margin gate selects tools by retrieval, and arguments are generated over a compressed textual signature (median 19 tokens) rather than over spliced key/value (KV) cache. This path is depth-independent: routing accuracy stays near 89% as the registry scales to 250 tools — where a concatenate-all-schemas baseline overflows the context window entirely — and it reaches a first-argument token 1.66× sooner than a full-schema re-prefill at an ≈80% main-context token saving.

As a secondary, bounded lever we transplant a compiled schema KV block directly into the live context. This is fundamentally limited by rotary position embedding (RoPE) phase drift: an anchored splice is output-exact, but off-anchor placement corrupts attention, so beyond a threshold P = 256 Nexus repairs the seam with a depth-adaptive suffix redecode that escalates to a full re-prefill. The resulting never-regress property is a guarantee on output fidelity (top-1 agreement, D_KL ≈ 0) — not on latency, which can dip to 0.98× before converging to parity — alongside a 1.1–1.7× TTFT speedup at moderate depth that narrows to parity at deep context.

Two negative results bound the design: the off-anchor RoPE fidelity boundary, and the failure of a reference-free drift gate to predict drift (Spearman ρ = 0.193). All measurements are from one model tuple (Qwen2.5-14B-Instruct Q4_K_M) on Apple-silicon unified memory; the qualitative boundaries generalize, while the quantitative envelope is tuple-specific.

# 1. Introduction

Tool-augmented LLM agents increasingly standardize on the Model Context Protocol (MCP), presenting the model with a registry of callable tools, each described by a verbose JSON schema. As the registry scales to hundreds of tools, these schemas dominate the prompt. Because self-attention is O(N²) in sequence length N, re-encoding tool schemas every turn imposes a prefill wall that scales poorly with the number and verbosity of available tools.

An appealing optimization is to compile each schema once into a KV block and transplant it into the live context at inference time, paying the prefill cost only once. We find this is fundamentally constrained by the position dependence of rotary position embeddings (RoPE): a block compiled for one absolute position cannot be relocated to an arbitrary depth without corrupting attention. Prior serving systems sidestep the problem by never moving cache — paged and prefix-shared KV keep blocks at the positions where they were computed. Nexus instead asks how far a relocated schema block can be trusted, and what it costs to repair it.

# Contributions

1. Retrieval-decoupled routing (measured) — the more durable and transferable result. Tool selection runs over an INT8 semantic lookaside buffer with a calibrated cross-encoder gate, and arguments are generated over a compressed textual signature, avoiding the splice seam. It is depth-independent, scales to 250 tools where the concatenate-all baseline cannot run, and delivers a 1.66× first-argument latency win at ≈80% token saving.
2. Depth-adaptive recompute with a never-regress guarantee (measured) — a bounded, characterized secondary lever. Past a 256-token splice threshold, Nexus re-decodes a suffix fraction R(n_past) that scales with depth to 100%, holding output fidelity exact and converging gracefully to prefill parity.
3. Systems mechanisms for unified memory (implemented): transposed-V splicing for soft-capped attention models and cache-line hardening of the block allocator.
4. Two bounding negative results (well-supported): the off-anchor RoPE fidelity boundary and the failure of reference-free drift gating.

Three research questions organize the paper. RQ1: how far can a relocated schema block be trusted before its next-token distribution departs from a full prefill? RQ2: what does repairing that drift cost in TTFT? RQ3: can routing and argument generation avoid the splice path entirely?

# 2. Related Work

Tool retrieval and schema compression. The two standard responses to schema bloat both act before the model. Retrieval-augmented tool selection — exemplified by RAG-MCP — fetches only the relevant MCP schemas prior to prefill. Schema-level compilation and learned latent tool retrieval shrink or replace tool descriptions at the application and embedding levels, the latter requiring trained alignment. Nexus adopts the same retrieval principle to decouple routing but adds a complementary system-level lever: direct KV transplantation for the schemas that remain, with a zero-shot calibrated margin gate over frozen embeddings.

KV-cache management for serving. Paged and prefix-shared KV management and dynamic virtual-memory schemes eliminate fragmentation and reuse cache in place; they do not relocate a block to a new position, which is precisely the regime Nexus characterizes. RedKnot pushes reuse to the granularity of individual attention heads but keeps cache in place, whereas Nexus relocates coarse per-tool blocks.

Non-Nexus numbers are reported on different hardware and workloads, so we treat them as contextual rather than a controlled head-to-head.

# 3. RoPE Phase Drift

RoPE injects position by rotating query/key feature pairs by a position-dependent angle. For feature pair i with base frequency θ_i = b^−2i/d (the runtime model uses base b = 10⁶), a key compiled assuming anchor position m₀ but consumed at position n_past accrues a phase error

Δθ_i = (n_past − m₀) · θ_i

Because attention scores depend on the relative rotation between query and key, a transplanted block whose keys carry the wrong absolute phase is read off-axis by every subsequent query. The error grows with the placement offset and is non-uniform across dimensions (high-frequency pairs drift fastest), so a pre-compiled schema block is faithful only in a neighborhood of its compile anchor.

Nexus counteracts the bulk of this error at splice time by reanchoring: each transplanted key is re-rotated from its compile anchor m₀ to the runtime position n_past — the exact inverse of the offset above — so subsequent queries read the relocated block on-axis. Reanchoring recovers the relative query–key rotation exactly, but the block’s keys and values were compiled under a different preceding context, so a small residual divergence survives the correction. That residual is what §4 measures, and it is what the depth-adaptive repair drives to zero. This is why an off-anchor splice drifts slightly rather than catastrophically, and why the 256-token boundary is a repair-start rather than a hard failure point.

# 3.1 Depth-adaptive recompute

Rather than treat 256 as a hard wall, Nexus repairs drift by re-decoding the trailing tokens of the spliced schema, with a fraction that grows with depth. Let M = 256 be the splice threshold, R_base = 5% the baseline recompute fraction, and K the multiple of M at which recompute reaches 100%. The effective fraction is R(n_past) = R_base for n_past ≤ M, and

R(n_past) = R_base + (n_past − M) / (M(K − 1)) · (100 − R_base), for n_past > M

clamped to 100%. At deep positions the fraction approaches 100%, which is numerically a full text prefill (D_KL = 0); K is the knob that trades TTFT flatness for how early full recompute engages.

# 4. Physical Limitations and Negative Results

We present the negative results first because they motivate the architecture: they fix the shape of the repair curve and force routing to be depth-independent rather than to depend on a deep splice.

# 4.1 The anchored / off-anchor fidelity boundary

At the compile anchor (Δpos = 0) an anchored splice is output-exact: D_KL ≈ 0 with top-1 agreement = 1.0. Once the block is placed off-anchor, the phase error perturbs the distribution.

Two facts stand out. First, the RoPE reanchor is effective: the divergence rises from its ≈0 anchor floor to ~10⁻² nats within the first 64 tokens and then plateaus — it stays in the 0.008–0.038-nat band, with top-1 agreement = 1.0, across the entire 0–2048 range, so a contiguous-suffix bare splice never fails catastrophically at these depths. The threshold P = 256 is therefore a conservative repair-start rather than a sharp fidelity cliff.

Second, the retired scattered partial-recompute configuration (LegoLink) tells the complementary cautionary story: at n_past = 1024 it left the divergence as high as D_KL ≈ 5.7 nats — two orders of magnitude above the contiguous-suffix bare splice at the same depth. Scattered recompute thus actively corrupted the distribution rather than repairing it, which is why the current design repairs a contiguous trailing suffix.

If the contiguous bare splice already holds top-1 agreement = 1.0 out to Δpos = 2048, why repair at all? Because top-1 agreement is a weaker property than distributional identity. The residual ~10⁻²-nat divergence leaves the tail of the next-token distribution perturbed, so under temperature or nucleus sampling the spliced and prefilled paths can still diverge, and that per-token gap compounds across the many turns of an agentic trajectory. Driving the divergence to exactly 0 makes the spliced output bit-identical to a full prefill — an auditable guarantee that holds under any decoding parameters, which is what “never-regress” certifies.

# 4.2 The failure of reference-free drift gating

A natural optimization for the Path-A (≤ 256) to Path-B (> 256) transition is a cheap, reference-free runtime proxy — per-head preceding-context K-variance — that predicts when a deep splice will drift catastrophically. Profiling this proxy against the true per-head drift on the 14B model rules the approach out.

Per-head drift is not itself absent — max drift ranges 175–207 and mean drift 91–105 across depths and both on- and off-topic contexts — but the K-variance proxy fails to rank-correlate with that variation, so a scalar-threshold gate driven by it cannot separate high- from low-drift heads and would fire indiscriminately. An effective online gate would require a true reference (the full prefill cost we seek to avoid) or a custom attention kernel. This justifies keeping the deterministic depth-adaptive curve as the repair mechanism.

# 5. Architecture: Retrieval-Decoupled Routing

The negative results dictate the architecture: routing must be depth-independent rather than depend on a deep splice. Arguments, in turn, are generated over a compact textual signature rather than over spliced schema KV, which both keeps the main-context prompt small and avoids relying on a splice inside the drift-prone regime.

# 5.1 Semantic lookaside buffer and gate

Tool selection runs over a dense SLB: tool embeddings from nomic-embed-text-v1.5 are INT8-quantized and scanned with a branchless SIMD dot-product (NEON / AVX-512-VNNI / AVX2). The embedding of a tool concatenates its name and description under the asymmetric search_document: prefix, mirrored by search_query: on the query side; the tool name carries dominant routing signal. The top candidates pass a margin gate: if the top-two score margin exceeds a calibrated threshold the decision auto-resolves, otherwise it escalates to a fine-tuned MiniLM-class cross-encoder. The threshold is calibrated to the 20th percentile of adversarial-pair margins (τ = 0.0136), so the cross-encoder fires on the lowest-confidence ~20% of decisions. Retrieval depends only on the query embedding and the registry, never on the deep context, so it is inherently depth-independent.

# 5.2 Execution sidecar and compressed IR

When a candidate splice is required to validate routing in a fresh sequence, Nexus allocates a temporary sidecar sequence beginning at position 0, so the anchored splice is always in the output-exact regime. To preserve multi-turn coreference (e.g. “open a pull request there”), the sidecar carries a pruned sliding window: recent turns are kept verbatim while bulky tool payloads are replaced by short semantic surrogates.

Given the resolved tool, arguments are generated in the main context over a semantic compressed intermediate representation: the JSON schema is reduced to a dense, type-hinted function signature with truncated inline descriptions, e.g. create_repository(name: string, private?: boolean). At a median of 19 tokens the IR is far smaller than the full schema, so the main-context prefill avoids schema bloat while retaining enough semantics to bind arguments to the correct fields. A finite-state machine masks logits to the tool-name radix trie; JSON arguments are then constrained by a native GBNF grammar compiled from the schema.

# 6. Systems Implementation

# 6.1 Compiled tool blocks and splice

Each schema is compiled offline into an Aeon Tool Block (.atb): a 128-byte header recording the RoPE anchor and scaling parameters, followed by page-aligned (2 MB) contiguous F16 key and value tensors. The header’s RoPE parameters are validated against the runtime model before any splice, so a block compiled for a different scaling regime is rejected rather than silently corrupting attention. The splice writes K/V rows into the live KV cells at the runtime cursor; a suffix of ⌈S · R(n_past)/100⌉ tokens is then invalidated and re-decoded to stitch the seam.

# 6.2 Transposed-V splicing for soft-capped attention

Attention-logit soft-capping (e.g. Gemma-2) disables FlashAttention, which forces the V cache into a transposed, token-innermost layout. A fixed-stride row-major copy cannot address this layout. Nexus implements a layout-aware routine that performs strided transposed copies directly in the physical cache under unified memory, validates the live V strides before writing, and is fidelity-tested on the soft-capped Gemma-2-9B model.

On non-UMA / discrete-GPU back-ends, where a transposed strided host-to-device transfer would be pathological, the splicer declines and the system falls back to prefill — a deliberate safety boundary, not a general path. Two cases fall back entirely to text-prefill (Path B, 1.0× speedup): cloud/remote execution, because cache transplantation requires direct copies into local physical cache addresses that a token-stream API does not expose; and discrete-GPU back-ends.

# 6.3 Cache-line hardening and concurrency

The Python orchestration layer drives a single native llama_context whose internal decode is guarded by a C++ mutex. Fine-grained Python locking around individual decode steps races that mutex; Nexus instead serializes an entire turn (routing plus argument generation) under one coarse re-entrant context lock, yielding deterministic, bit-stable outputs under concurrency. To remove cross-core false-sharing on the buddy allocator, the allocator is declared alignas(128) and its contended mutex is pinned to its own cache line, enforced at compile time with a static_assert. We report this as implemented hardening with a passing four-thread determinism test; we do not claim a throughput speedup, having measured none in isolation.

# 6.4 L0 exact-token radix arena

A zero-allocation radix cache manages a fixed pool of 32 sequence slots and reuses warm prefixes by exact-token longest-common-prefix matching, copying KV within a pre-reserved arena rather than allocating on the hot path.

# 7. Evaluation

All benchmarks were executed on an Apple M4 Max SoC (16-core CPU, 40-core GPU, 16-core Neural Engine) with 64 GB unified memory and a 1 TB NVMe SSD, running macOS/Darwin 25.5.0 (arm64). All measurements use Qwen2.5-14B-Instruct Q4_K_M with nomic-embed-text-v1.5. Sample sizes are stated per result; they are small, and we treat the numbers as an envelope rather than a population estimate. Every headline proportion carries a Wilson 95% interval and every TTFT median a bootstrap 95% interval.

# 7.1 Depth-adaptive splice: TTFT and never-regress

Two facts hold across every cell: top-1 next-token agreement is preserved and D_KL ≈ 0 — the never-regress guarantee. We stress that never-regress is a guarantee on output fidelity, not on latency: because the repaired suffix adds redecode work, TTFT can dip slightly below parity (0.98× at n_past = 1024, K = 4) before converging to 1.0×. The speedup is not flat: at moderate depth the splice saves 1.1–1.7×, and as R(n_past) escalates toward 100% the repaired splice converges to prefill parity. Flat TTFT and never-regress are in genuine tension, and K is the dial between them.

# 7.2 Routing accuracy and registry scale

Routing is evaluated over 100 GitHub-MCP queries as the registry scales from N = 10 to N = 250 tools. End-to-end routing accuracy is nearly flat with scale: 92%, 90%, 89%, 89% at N = 10, 50, 100, 250 — a spread that sits within overlapping Wilson 95% intervals. At N = 250 the routing accuracy carries a Wilson 95% interval of [81.4, 93.7] (n = 100), with SLB top-1 recall 74% [64.6, 81.6] and top-3 recall 95% [88.8, 97.8].

The contrast with the concatenate-all-schemas oracle is the point: the oracle reaches 98% at N = 10 but overflows the context window at N ≥ 50 and cannot answer at all, whereas Nexus keeps the main-context prompt small and continues to route. Because SLB search depends only on the query embedding and the registry, tool count does not inflate TTFT: the in-situ INT8 scan over 250 real tool vectors completes in 17.6 µs (median, including the Python FFI boundary), while the pure C++ SIMD dot-product scan over 250 unit-norm vectors runs in 8.25 µs.

# 7.3 Argument fidelity and micro-costs

Over the compressed textual IR (median 19 tokens, p99 32), the sidecar routes correctly on 86.7% (Wilson 95% CI [70.3, 94.7]) of the 30-case consistency set and, on the routed cases, fills 100% (95% CI ≥ 91.2% over 40 specified arguments) of query-specified arguments correctly with a 100% JSON validity rate and no placeholder leakage; end-to-end argument accuracy, which charges routing errors as failures, is 80% (40/50 arguments, [67.0, 88.8]). The hybrid path reaches its first-argument token in 443.8 ms versus 737.3 ms for an oracle that re-prefills the full schema every turn — a 1.66× reduction at an ≈80% main-context token saving.

# 7.4 Discussion

The evaluation supports a bounded thesis. Off-anchor KV splicing is fundamentally limited by RoPE phase drift, but the failure envelope is predictable and can be repaired by a deterministic depth-adaptive recompute that never regresses below a prefill. On this workload that buys a 1.1–1.7× TTFT reduction at moderate depth, decaying to parity at deep context. Routing, argument generation, and coreference are deliberately kept out of the splice path — a design forced by the off-anchor fidelity boundary — and run over retrieval and a compact textual IR at microsecond-scale overhead.

# 8. Limitations and Scope

• Single tuple. All quantitative numbers are from one host (Apple M4 Max, UMA) and one measured model tuple (Qwen2.5-14B-Instruct Q4_K_M) with one embedder and one llama.cpp build; the transposed-V splice mechanism is additionally validated on Gemma-2-9B, but generality beyond these is unproven.
• Small n. End-to-end arms use n ≤ 30. Accuracy intervals are correspondingly wide and we do not report them as population estimates.
• Deep splice is micro-validated. The never-regress curve is measured at the fidelity-and-latency level; it is wired into the agent but not exercised as a multi-turn production path.
• Transposed-V is UMA-only. Soft-capped models splice under unified memory; discrete-GPU back-ends decline by design.
• Retrieval ceiling. Dense and cross-encoder recall have a workload-dependent ceiling; the gate improves the lowest-confidence decisions but does not remove hard confusions.

# 9. Conclusion

Nexus reframes tool-schema acceleration from unconditional KV transplantation to retrieval-decoupled routing paired with a depth-adaptive repair of the schema splice. We characterized the RoPE-induced fidelity boundary in output-distribution terms, and established the off-anchor boundary and the reference-free gating failure as well-supported negative results. We design the serving substrate to decouple semantic routing from the attention-level phase drift boundary, executing partial recomputes only where mathematically required to restore output fidelity. On a single-host 14B prototype this delivers a 1.1–1.7× TTFT reduction at moderate depth with high argument validity.

We view the measured envelope and the negative results as the durable contribution, and we separate what generalizes from what does not: the qualitative boundaries — off-anchor relocation drifts, scattered recompute cannot cheaply repair it, and reference-free gating fails to predict it — are properties of RoPE and hold model-agnostically, whereas the quantitative envelope (the P = 256 threshold, the recompute-curve constants, and the specific divergence magnitudes) is calibrated to one model tuple and must be re-measured elsewhere.