MAMFAC · UNDER REVIEW · ACML 2026

MAMF-RL

Memory-Augmented Mean-Field Actor-Critic — letting large populations of agents coordinate through a shared, utility-weighted mean memory field instead of pairwise messages.

Sreedharshan G J, Dr. V. Ceronmani Sharmila · submission #325, OpenReview

MULTI-AGENT RLMEAN-FIELD SYSTEMSEFFECT SIZE d ≈ 4.9UNDER REVIEW

Paper — pending decisionCode — released on acceptance

01 Problem

Mean-field reinforcement learning lets large populations of agents coordinate without pairwise communication by replacing neighbour interactions with a single mean action field. But standard formulations (MFAC, Mean-Field Q-learning) are memoryless in a specific sense: policies condition only on the current observation and mean action, with no mechanism for aggregating historical information across the population — even in tasks with real temporal structure, like tracking demand history or remembering where prey was last seen.

02 Motivation

Recurrent methods (DRQN-style, mfac-LSTM) extend memory to individual agents; communication-based methods (CommNet, ATOC) let agents exchange messages that can implicitly carry history. No prior mean-field-style method explicitly constructs a utility-weighted aggregate of per-agent memory embeddings as a population-level state variable, analogous to the mean action field itself. MAMFAC targets this narrower, specific gap.

03 Research Question

Does augmenting a mean-field actor-critic with a memory field — where agents write utility scores that a policy later reads from — improve coordination over memory-less mean-field baselines (MFAC, MFQ), and does the conjectured O(1/√N) mean-field approximation-error bound hold up empirically?

04 Architecture

Three components, evaluated together as MAMFAC:

Memory Utility Network (MUN) — a learned module mapping each agent's observation-action-reward trajectory to a memory embedding M_i and an informativeness score u_i ∈ [0,1], via an attention-gated GRU cell.
Mean Memory Field — a utility-weighted aggregate over the k-nearest-neighbour set, computable in O(k·d_m) time (Eq. below).
MAMFAC actor-critic — conditions both actor π_θ(a_i | o_i, ā_i, m̄_i) and a TD3-style twin-Q critic on the mean action field ā_i and the mean memory field m̄_i, using a Tanh-squashed Gaussian policy and prioritised experience replay that factors in memory novelty and MUN utility.

Parameter sharing across homogeneous agents keeps the parameter count independent of population size N.

05 Figures

FIGURES IN MANUSCRIPT, NOT REPRODUCED HEREThe submission includes diagrams of the MAMFAC pipeline (MUN → mean memory field → actor/critic), the internal MUN attention-gated GRU cell, per-environment learning curves (CoopNav, Predator-Prey, MountainCar, Inventory), and the CoopNav/Predator-Prey significance-bracket comparison. These are figures embedded as raster images in the PDF; they are described in the Architecture, Results, and Equations sections here from the manuscript text, and will be swapped in directly once the submission is public.

06 Equations

$$\bar{a}_i = \frac{1}{|N_i|}\sum_{j\in N_i} a_j$$

Mean-field action approximation (Yang et al.), reducing the multi-agent problem to a sequence of two-agent sub-games.

$$\bar{m}_i = \frac{\sum_{j\in N_i(k)} u_j M_j}{\sum_{j\in N_i(k)} u_j + \varepsilon}$$

Mean memory field: a utility-weighted aggregate of neighbour memory embeddings over the k-nearest-neighbour set (ε>0 for numerical stability). Uniform utilities reduce this to a plain neighbour average.

$$\mathbb{E}\big[\lVert \bar m_N - \bar m_\infty\rVert\big] = O\!\left(\frac{1}{\sqrt N}\right) + O(\varepsilon_{mun})$$

Conjectured approximation-error bound (not a proven theorem — two of its three assumptions are stated design choices rather than derived requirements). The statistical term is empirically supported: measured N-scaling exponent −0.459 vs. theoretical −0.5, R² = 0.997.

07 Experimental Setup

Four environments across two categories: Cooperative Navigation and Predator-Prey (Multi-Particle Environments suite, N=10 agents) for standard MARL coordination/pursuit; MountainCar (Continuous) and Inventory Management (N=4 agents) for memory-dependent continuous control. Baselines: MFAC, Mean-Field Q-learning, mfac-LSTM (recurrent), and MAPPO (strong modern CTDE baseline, evaluated only on the two continuous-control tasks). Main results use 500 episodes, 10 seeds; paired t-tests assess significance with Cohen's d effect sizes.

08 Results

Comparison	Effect
vs. MFAC (across environments)	p ≤ 0.034, d ≥ 1.42
vs. MFQ (across environments)	p ≤ 0.01, d ≥ 2.11
Largest gain: CoopNav vs. MFQ	p = 0.0004, d = 4.90, +22.2%
Zero-shot generalisation at N=2 vs. MFAC	+26.2%, p = 0.0016, d = 3.38
Per-step overhead scaling	O(N^0.33), sub-linear to N=200 (vs. MFAC's O(N^0.50))
Mean-field error bound fit	R² = 0.997

On MountainCar and Inventory, MAMFAC outperforms all baselines including MAPPO — notably beating MAPPO's centralised critic on Inventory by 28% (−123.8 vs. −171.2). MAMFAC is slower than MFAC in absolute per-step time at every tested N (2.1–3.3×), but that overhead ratio shrinks as population size grows. Currently under review at ACML 2026 (submission #325).

09 Ablations

The honest headline result here is a surprising one. Removing the memory field entirely causes a significant 10.1% drop on CoopNav (p=0.0455) — some memory clearly helps. But a frozen MUN (random initialisation, never trained) performs identically to the fully trained MUN (d=0) on CoopNav, and neither the learned memory embeddings nor the learned utility-weighting head shows a significant advantage over a non-learned alternative in isolation (both p>0.5, d<0.35 for the utility-head ablation). This is in tension with a simple "the network learns meaningful embeddings, and that's why memory helps" account of the method — it suggests structured, time-varying input may matter more than whether that structure is learned, at least on CoopNav. Neither of the two candidate explanations (architectural inductive bias, vs. CoopNav being insufficiently memory-critical to separate the conditions) is confirmed; this is flagged as an open finding rather than resolved.

10 Failure Cases

MAMFAC trails both MFAC and MFQ for roughly the first 250–280 episodes on CoopNav before overtaking them — a real cold-start cost from jointly learning the MUN and the policy; under a hard episode budget below that threshold, MAMFAC would likely underperform. Zero-shot transfer from a population of N=16 to smaller populations holds up at N=8 but fails significantly at N=2 (a 33.9% deficit vs. training from scratch, p=0.0017) — the transferred memory field provides little benefit once the population has shrunk that far. Table 3's 5-seed CoopNav estimate (−66.5 ± 3.4) is also noticeably worse than the 10-seed estimate for the identical configuration (−54.7 ± 4.2) reported in the main results table — an unexplained cross-table gap the manuscript reports rather than papers over.

11 Limitations

Incomplete CTDE comparison: MAPPO was only evaluated on the two continuous-control tasks, and QMIX/MADDPG were not evaluated at all, so the claim of beating modern CTDE baselines on standard MARL benchmarks is not fully substantiated.
The ablation mechanism is ambiguous — neither learned component beats its non-learned counterpart on CoopNav, and this hasn't been re-run on the memory-critical continuous-control tasks.
All four environments have meaningful temporal structure by construction; MAMFAC has not been tested on a purely reactive, near-Markovian task where its advantage is expected to shrink.
Seed counts are inconsistent across studies (10 / 5 / 3), which is itself flagged as a limitation.
The O(ε_mun) term in the approximation-error bound is not independently bounded in this work.

12 Future Work

The paper's own stated open question: whether the memory field should be learned end-to-end or kept structured, given the ablation result above. Concretely planned next steps include running MAPPO on CoopNav/Predator-Prey under the main protocol, adding a local-information QMIX variant, re-running both ablations on the continuous-control tasks and against a fixed hand-designed aggregation rule, standardising seed counts to 10, and testing the mean memory field under adversarial perturbation.