Sixteen channels carry key swells in from the west. Sixteen more carry teacher swells up from the harbours in the south. Where two channels cross, a small basin of sand feels both swells at once, and the interference between them moves its sand. Store pairs with strong swells, then ping a key faintly and listen at the harbours. How the sand responds decides what kind of memory the beach is.
Blue basins are dug deeper, ochre ones silted up. A magenta ring means the basin hit bedrock or the surface and can't move further. Squares on the west shore are the key, the bars in the south are what each harbour hears (ticks show what it should hear; magenta bars are wrong).
Two beaches, the same swells, opposite order. Do they end up with the same sand?
Each line stores more and more pairs on a fresh beach and scores how many harbour bits a ping gets right, averaged over 6 random sets. Chance is 0.5. It follows the keys look alike, rehearsals and teacher strength settings above.
Beach A stores x → y. Beach B stores y → z. Ping x into A, and A's harbours flow into B's inlets through a puddle. Can B answer z, two hops away? The puddle is the only fast nonlinearity on this page, and there are two kinds.
A crossbar beach is a fast-weight memory: a matrix written by the signals passing through it. Linear attention and DeltaNet are built from exactly this, so none of it is new. What the beach adds is a reason each rule exists, and a picture of it.
| On the beach | In a fast-weight model |
|---|---|
| Key swell along channel j, teacher swell up channel i | key k, value v |
| Basin sand moves with ½(aj + bi)²; its cross term is ajbi | the outer product v kᵀ |
| A faint ping, too weak to move sand, heard at the harbours | a query: y = S q |
| Plain sand: moves whenever it's agitated, so every swell digs every basin | Hebbian writing with a huge bias; the memory fills with noise and saturates |
| Sand with a habit: moves only by how far the stress is from what it's used to | linear attention, S += v kᵀ |
| Habit, and listen first: ping the key, send back only what the harbours got wrong | the delta rule (DeltaNet), S += (v − S k) kᵀ |
| Habit sand: the same swells in either order leave the same beach | linear attention's sums commute; order is lost |
| Listening first: the order of the swells matters | delta writes depend on the state, so they don't commute |
| A puddle between beaches | the nonlinearity between layers |
The wave-equation version of these rules was measured in this repository: plain scour is gradient descent on the swell's own compliance (G6), and the habit rule gives one-exposure gradient steps (G7). On real hardware the crossbar is not a metaphor. Photonic weight banks put a tuned microring at each crossing, its weight set by its detuning. Phase-change photonic memories are written by strong light pulses and read by weak ones, and volume holograms store thousands of pages in one crystal, addressed by angle or wavelength.
node tests/check_crossbar.mjs checks the claims on this page: habit sand writes exactly k vᵀ, plain
sand writes that plus a uniform dig three times larger, pings move exactly no sand, habit writes commute and listening
writes don't, and the delta rule stores 16 orthogonal keys exactly.