Dynamical phases of short-term memory mechanisms in RNNs

Dynamical phases of short-term memory mechanisms in RNNs

Bariscan Kurtkaya* 1 2 Fatih Dinc* 2 3 4 Mert Yuksekgonul 5 Marta Blanco-Pozo 2 6 Ege Cirakman 2Mark Schnitzer 2 6 7 Yucel Yemez 1 Hidenori Tanaka† 8 9 Peng Yuan† 10 Nina Miolane† 3

Abstract

Short-term memory is essential for cognitive processing, yet our understanding of its neural mechanisms remains unclear. Neuroscience has long focused on how sequential activity patterns, whereneurons fire one after another within large networks, can explain how information is maintained.While recurrent connections were

shown to drivesequential dynamics, a mechanistic understanding of this process still remains unknown. In thiswork, we introduce two unique mechanisms

that can support this form of shortterm memory: slowpoint manifolds generating direct sequences orlimit cycles providing temporally localized approximations.

Using analytical models, we identify fundamental properties that govern the selection of each mechanism. Precisely, on shortterm memory tasks

(delayed cue-discriminationtasks), we derive theoretical scaling laws for critical learning rates as a function of the delay periodlength, beyond which no learning is possible.

Weempirically verify these results by training andevaluating approximately 80,000 recurrent neuralnetworks (RNNs), which are publicly availablefor further analysis1.

Overall, our work providesnew insights into short-term memory mechanismsand proposes experimentally testable predictionsfor systems neuroscience.

原文链接：https://doi.org/10.48550/arXiv.2502.17433