TECHNICAL REPORT · JUNE 2026 · SRM IST

MEMORY UTILITY NETS

Utility-based memory retrieval — a strong positive result (MUN v1, label-aware) and a documented negative result (MUN v2, label-free), across thirteen experimental phases.

MEMORY-AUGMENTED AIRETRIEVALEFFECT SIZE d ≈ 28.8NEGATIVE RESULT DOCUMENTED

Technical Report (PDF) ↗Code — not publicly shared

01 Problem

Existing memory-retrieval strategies — cosine similarity, recency ordering, access frequency — are principled but fundamentally retrospective: they measure what a memory has been, not what it will be worth when retrieved in a given context. In few-shot in-context learning, the choice of which examples to retrieve can swing downstream accuracy by 10–40 percentage points — often more than architectural changes to the model itself.

02 Motivation

If a learned function could predict a memory's prospective usefulness — rather than only its similarity or recency — retrieval could be optimized directly for downstream task accuracy instead of a proxy signal. The open question this project set out to answer: does that learned utility function actually beat the heuristics in practice, and does it still work once query labels aren't available at retrieval time (the realistic deployment case for live RAG pipelines and streaming agent memory)?

03 Research Question

Can a learned utility scorer outperform heuristic retrieval strategies (cosine similarity, recency, frequency) in few-shot in-context learning? And if labels are unavailable at retrieval time, can label-free features recover competitive performance?

04 Architecture

MUN v1 (label-aware) is a supervised neural utility scorer mapping a (query, memory) pair to a scalar utility estimate. Five components: a shared frozen SentenceTransformer encoder (all-MiniLM-L6-v2, 384-dim), a context-attention module that produces pool-aware memory representations via self-attention over the whole memory pool, a learnable exponential temporal-decay term, a label-boost mechanism (+0.15 for same-class memories), and a 4-layer MLP scoring head [512→256→128→1] trained with a composite ranking + contrastive + BCE loss under curriculum ordering.

MUN v1 end-to-end neural architecture: query and memory pass through a shared frozen SentenceTransformer encoder and context-attention module, producing five utility features that are concatenated and scored by a four-layer MLP head trained with a composite loss.

Fig. 2 — MUN v1 end-to-end architecture. The label boost (+0.15 for same-class memories) is the dominant performance driver, confirmed by the Phase 12E ablation below. Reproduced from the technical report.

When the Phase 12E ablation showed 34 percentage points of MUN v1's advantage came from label access, the project built MUN v2: a label-free feature-weighted linear scorer, selected over a neural architecture only after an embedding-validation decision gate (Phase 13B.75) failed to clear its own pre-registered threshold.

05 Figures

Research programme timeline showing thirteen experimental phases across four macro-stages: Foundation, Validation and Full-Scale MUN v1, MUN v2 label-free investigation, and Final Controlled Validation.

Fig. 1 — Thirteen experimental phases across four macro-stages, from MUN v1's full-scale validation (Recall@5=0.829) through the MUN v2 label-free investigation to its final controlled negative result.

Bar charts comparing MUN v1 against seven baselines on Recall@5 and NDCG@5, plus MUN v1's full metric suite across Recall@5, Recall@10, NDCG@5, NDCG@10, MAP, and AUC.

Fig. 4 — MUN v1 vs. seven baselines (5 seeds, SST-2): a +0.314 / +61% Recall@5 gap and +0.376 / +70% NDCG@5 gap over the best heuristic (dense cosine similarity), with all seven baselines clustering between 0.481–0.515.

MUN v1 component ablation study showing Recall@5 and NDCG@5 for the full model versus configurations with temporal decay, contrastive loss, curriculum training, context attention, or ranking loss removed, plus a BCE-loss-only configuration.

Fig. 5 — Component ablation (leave-one-out, 5 seeds): removing all ranking-oriented objectives (BCE loss only) collapses Recall@5 by 32.6% — only 16% above random selection — confirming the ranking-based training objective is essential, not incidental.

06 Equations

$$\text{ContextEmb}(m_i) = \text{Attention}(Q{=}\text{emb}(m_i),\, K{=}\text{emb}(M),\, V{=}\text{emb}(M))$$

Pool-aware context attention — lets the scorer down-weight redundant memories relative to the rest of the pool. Ablation: −11.9% Recall@5 upon removal.

$$\mathcal{L} = \mathcal{L}_{rank} + 0.5\cdot\mathcal{L}_{contrast} + 0.3\cdot\mathcal{L}_{BCE}$$

MUN v1 composite training objective: ListMLE ranking loss, pairwise margin (contrastive) loss, and binary cross-entropy on utility labels, under curriculum (easy-to-hard) ordering.

$$U(q,m) = w_1 f_1(\text{overlap}) + w_2 f_2(\text{info\_gain}) + w_3 f_3(\text{uniqueness}) + w_4 f_4(\text{diversity}) + w_5 f_5(\text{entropy})$$

MUN v2's label-free utility scorer: a five-feature weighted linear model, selected after Jaccard text overlap (Pearson r=0.716) dominated a 17-candidate feature-discovery study and no embedding feature cleared the |r|=0.30 neural-architecture decision threshold (best embedding feature: r=0.194).

07 Experimental Setup

Benchmark: SST-2 (GLUE) binary sentiment classification — 67,349 training examples, 872 validation examples. Memory pool: 1,000 training examples (seed=42, ≈37.6% positive / 62.4% negative); queries drawn from the validation set. Downstream classifier: BART-large-MNLI, reformulating sentiment as zero-shot NLI entailment, given the top-k=4 retrieved memories as in-context examples. Retrieval quality measured via Recall@k, NDCG@k, MAP, and AUC. Seven label-free baselines compared: FIFO, LRU, Random, Recency, Frequency (all structural), plus TF-IDF and dense cosine Similarity (content-based).

08 Results

MUN v1 (label-aware): Recall@5 = 0.829, NDCG@5 = 0.914, averaged over 5 seeds — a 60.9% relative improvement over the best heuristic baseline (dense Similarity, Recall@5=0.515). Statistical evidence: t(4)=45.56, p<10⁻¹⁰, Cohen's d ≈ 28.8. A seven-question Phase 12D validation audit confirmed no data leakage, label leakage, or metric error.

Method	Recall@5	NDCG@5	MAP	AUC
MUN v1 (Full)	0.829	0.914	0.800	0.878
Similarity (best baseline)	0.515	0.538	—	—
TF-IDF	0.504	0.535	—	—
Random	0.481	0.523	—	—

MUN v2 (label-free): despite 128-configuration weight optimisation and systematic feature ablation, the final controlled validation (Phase 13C.9, N=20) reached only 35% accuracy — below cosine similarity (65%), below recency (60%), and below random selection (40%). This is reported as a genuine negative result, not a bug: label-aware utility estimation works; label-free utility estimation via the 17 engineered features tested here does not.

09 Ablations

Component ablation (MUN v1, leave-one-out, 5 seeds): removing the ranking-oriented training objectives entirely (BCE loss only) collapses Recall@5 to 0.559 (−32.6%, only 16% above random); removing context attention costs −11.9%, ranking loss −12.9%, curriculum training −9.9%, contrastive loss −7.9%, temporal decay −4.9%.

Label boost ablation (Phase 12E): with the label boost active, accuracy reaches 99.0% (+34.0 pp over Similarity); with it removed (text similarity only), MUN drops to 65.0% — statistically indistinguishable from the cosine-similarity baseline. This single ablation is what motivated the entire MUN v2 label-free investigation.

Feature discovery (Phase 13B.5, N=2,000 triplets): of 17 candidate label-free features, Jaccard text/token overlap dominated (r=0.716), followed by an information-gain proxy (r=−0.450) and memory uniqueness (r=−0.386). Semantic embedding similarity showed only weak signal (r=0.194) — well short of the 0.30 threshold set for justifying a neural (rather than linear) MUN v2 architecture.

10 Failure Cases

MUN v2's weight-optimisation phase (Phase 13C.5) reported a 60% best-configuration accuracy that did not reproduce under the fully controlled Phase 13C.9 validation — a 45-percentage-point discrepancy between intermediate and final phases, investigated and documented as a case study in small-N evaluation fragility rather than swept aside. The root cause traced to implementation variance in pool ordering and random-seed timing across the feature-extraction pipeline. Separately, none of the embedding-based features (real SentenceTransformer vs. random embeddings) reached the pre-registered |r|=0.30 threshold needed to justify a neural MUN v2 architecture, so the simpler linear model was used instead — and still underperformed every heuristic baseline.

11 Limitations

All results are specific to SST-2 binary sentiment classification; its short inputs and two-class label space may systematically favour text-overlap features over richer semantic ones.
MUN v2's final validation rests on N=20 test queries — the 95% CI for the 35% accuracy figure spans roughly 13–57%, so a true accuracy ≥50% for some untested MUN v2 variant can't be definitively ruled out (N≥200 would be needed).
The text-overlap utility proxy used to select MUN v2's features is a rough approximation of true utility, not a validated ground truth.
The decision to use a linear rather than neural model for MUN v2 was based on a single |r|=0.30 correlation threshold, which may have been overly conservative.
Findings are validated only in the SST-2 in-context-learning setting with BART-large-MNLI as the downstream classifier — they should not be extrapolated to RAG, multi-hop reasoning, or code generation without further evaluation.

12 Future Work

The report's own next steps: reframe label-free utility estimation as a sequential decision process, training a retrieval policy end-to-end via policy-gradient methods (REINFORCE, PPO) against downstream accuracy as a non-differentiable reward — decoupling utility learning from label access entirely. Also planned: replicate MUN v1/v2 on additional benchmarks (AGNews, TREC-6, NaturalQuestions) with N≥500 queries and pre-registered designs; integrate MUN-style utility scoring into agent memory systems (ReAct, Reflexion, MemGPT, Voyager); and study utility-based eviction (rather than retrieval) as context windows expand toward 128K–1M tokens. A natural link back to the multi-agent side of this research: whether this retrieval mechanism transfers directly into the mean-field multi-agent setting used in MAMF-RL.