How a Spiking Neuron Computes

1. Membrane potential

A real neuron is a cell wrapped in a thin membrane. Signals from other neurons push charge across it, and the voltage across the membrane rises or falls. This voltage is the membrane potential, written V. We count time in steps, one word per step. At each step, the neuron adds its input to the voltage it already has, which is called integrating:

Here I_t is the input at step t. Follow neuron P while it reads The dogs that barked. Its inputs are 0, 1.2, 0, and 1:

At barked, neuron P's voltage is 1.2 + 1 = 2.2. The 1 is the input from barked. The 1.2 is what dogs left in the voltage two words earlier. A number a neuron keeps from one step to the next like this is called its state.

2. Leak and membrane decay

A real membrane leaks. Without new input, charge flows back out and the voltage sinks toward its resting level, which we set to 0. To model this, the old voltage is multiplied by a number β (beta) between 0 and 1 before the new input is added:

β is called the membrane decay. A β near 1 keeps most of the old voltage and forgets slowly, and a β near 0 forgets almost immediately. In our example we set β = 0.8. Each word first becomes neuron P's input, the dot product of the word with P's key, and the input is added to 80% of the old voltage:

Step by step, as a calculation:

Each step keeps 80% of the old voltage. The 1.2 from dogs shrinks to 1.2 × 0.8 = 0.96 at that, and to 0.96 × 0.8 = 0.768 at barked, where the 1 from barked is added on top. This update has the same form as a linear state-space model. That is a kind of recurrent model, a model that reads a text one word at a time and carries a state from each word to the next, and it updates its state h as h_t = A h_t−1 + B x_t: the old state times a fixed number A, plus the new input x_t times a fixed number B. The leaky neuron is that update with A = β and B = 1.

Papers often give the decay as a membrane time constant τ (tau) instead, with β = e^−Δt/τ, where Δt is the length of one step. With Δt = 1, β = 0.8 is τ = −1 / ln 0.8 ≈ 4.5 steps.

The leak also stops the voltage from growing forever. To see this, set our sentence aside for a moment. Take a separate neuron that gets an input of 1 at every step, with the same β = 0.8:

The voltage climbs quickly at first and then more and more slowly, because multiplying by 0.8 takes more away from a larger voltage. At 5, 0.8 × 5 = 4, and the input of 1 brings it back to exactly 5, so the voltage stops changing. In general, with a constant input the voltage settles at I_t / (1 − β), here 1 / 0.2 = 5.

3. Threshold, spike, and reset

What a neuron sends to other neurons is a brief pulse, called a spike (in biology, an action potential), and it sends one only when the voltage reaches a level called the threshold θ (theta). At that step its output S is 1. At every other step it is 0. Right after a spike, the voltage drops, which is called the reset. We use a soft reset, which subtracts θ and keeps whatever was above it:

Adding up input, leaking, and firing together make the leaky integrate-and-fire (LIF) neuron. In our example we set the threshold to θ = 1.5. That is higher than the input any single word gives, at most 1.2 from dogs, so a neuron can fire only when voltage left from an earlier word adds to the new input. Here is neuron P reading The dogs that barked were, where were, like The and that, gives both neurons 0:

At barked, the 0.768 left from dogs and the 1 from barked make 0.768 + 1 = 1.768, which reaches the threshold of 1.5, so neuron P fires. The spike comes at the step where the verb is predicted, and the reset clears most of the voltage once it has been used.

The other common choice is a hard reset, which sets the voltage to 0 after a spike. Neuron P would then hold 0 after barked instead of 0.268. The soft reset keeps the part above the threshold.

4. Spike trains, firing rate, and sparsity

Over the five words, neuron P's output S at each step is [0, 0, 0, 1, 0]. A neuron's outputs over time, read in order, are called its spike train. The firing rate is the share of steps on which the neuron fires. Adding up the outputs and dividing by the number of steps gives (0 + 0 + 0 + 1 + 0) / 5 = 0.2. The sparsity is the opposite, the share of steps on which the neuron stays silent. Neuron P is silent on 4 of the 5 steps, so its sparsity is 4 / 5 = 0.8, which is 1 − the firing rate.

Sparsity matters because a 0 costs nothing. In an ordinary layer, each neuron's input first goes through a function such as ReLU, which keeps a positive number and turns a negative one into 0. The result is the neuron's activation, and the neuron's value is multiplied by its activation before being added to the output. A spike is either 0, and then nothing is added, or 1, and then the value is added as it is, with no multiplication.

5. Rate coding

A single spike says only "yes" or "no", but many signals come in degrees, such as how bright a pixel is or how strongly a neuron responded. Such a strength, scaled to lie between 0 and 1, is called an intensity. A spiking neuron can pass on an intensity only by spreading it over several steps. A rule for turning an intensity into a pattern of spikes, which the receiving neuron can read the intensity back from, is called a code, and sending intensities this way is called coding.

In rate coding, the intensity is held fixed for a stretch of steps, called a window, and fed to the neuron as its input at every step of it. Each step adds the intensity to the voltage again, so a larger intensity reaches the threshold more often, and the receiver reads the intensity back as the share of steps in the window that carry a spike. An LIF neuron needs nothing extra to do this. For this example we choose a window of five steps, no leak (β = 1), a threshold of θ = 1, and an intensity of 0.8 as the input at every step:

Four spikes in five steps is a rate of 0.8, the same as the input. This is no coincidence. With no leak, every bit of input stays in the voltage, so five steps put in 5 × 0.8 = 4. The soft reset, which subtracts θ at each spike and keeps whatever was above it, takes out exactly 1 per spike, so 4 / 1 = 4 spikes come out. In general, with no leak and an input no larger than θ, the rate is the input divided by θ. A larger input cannot push the rate past 1, a spike at every step.

In five steps, an input of 0.81 also gives 4 spikes, the same as 0.8, so the two look identical. After n steps the two inputs have put in totals that differ by 0.01 × n, and the spike counts are sure to differ only once that gap reaches one whole spike, at 0.01 × 100 = 1. With fewer steps the counts may or may not differ, depending on where the window ends. From 100 steps on, where 0.8 gives 80 spikes and 0.81 gives 81, they always do.

6. Temporal coding and latency coding

Rate coding needs many spikes for one intensity, and the receiver has to count them over the whole window. A neuron can instead carry an intensity in when it fires, which is called temporal coding. Its simplest form is latency coding, also called time-to-first-spike coding. The neuron fires once in the window, and the larger the intensity, the sooner it fires, so the intensity travels in a single spike and the receiver can act as soon as it arrives.

For an intensity x, a signal's strength from 0 to 1, and a window of T steps, the spike comes at step:

With T = 10:

The figure puts both codes on the same ten-step window. The top row is the rate-coding neuron run for ten steps, which sends 0.8 as 8 spikes. Latency coding sends it as one spike at step 2, and the scale at the bottom gives the intensity each step stands for. The price is exact timing: a spike one step late, at step 3, reads as 0.7. The steps also limit precision, since 0.81 rounds to step 2 as well. Small intensities arrive late, 0.3 at step 7 and 0 only at step 10, the end of the window.

Both codes send one fixed intensity over a window. Our two-neuron layer uses neither, because each of its steps brings a new word, not the same intensity again. A spike at a step reports on the word just read, combined with whatever the voltage still carries from earlier words.

7. Spiking neurons inside an ordinary model

A model that uses ordinary numbers in most places and spiking neurons in some layers is called a hybrid model. Our spiking layer, placed in an otherwise ordinary language model, makes one. Its keys, its values, and the scoring of y against the verb vectors all work as in an ordinary layer.

Now run both neurons on both sentences with β = 0.8 and θ = 1.5, starting each sentence from a voltage of 0. Each neuron's voltage at barked, where the verb is predicted, is:

In each sentence exactly one neuron crosses 1.5, the one that matches the subject. The other reaches 0.64 × 0.7 + 1 = 1.448, just short. Its 0.7 came from the subject that does not match it (dogs gives neuron S 0.7, and dog gives neuron P 0.7), and two steps of leak left 0.64 of it. The neuron that fires adds its value to y, and y scores the verbs:

To see what the carried voltage contributes, compare three layers at barked: this one, a spiking layer whose voltage is reset to 0 before every word, and an ordinary ReLU layer, which has no state and so also starts fresh at every word.

With the reset, each neuron has only the 1 from barked, stays below the threshold, and nothing fires. The ReLU layer passes both neurons' 1, so y = [1, 0] + [0, 1] = [1, 1], and the verbs tie. Neither layer still knows the subject at barked. Only the carried voltage brings it there.

That is true of this layer on its own. A full language model has other layers, such as attention, that can also carry the subject to barked. What a spiking layer adds is a memory inside its own neurons.

Larger models such as Llama use a gated feed-forward network (gated FFN), in which each neuron has two keys, a gate key and an up key, and so gets two inputs. The gate input z goes through SiLU, a smooth version of ReLU computed as z / (1 + e^−z), and the result multiplies the up input to give the neuron's activation. A hybrid replaces SiLU with a spike, giving the activations a = S ⊙ (x W_up). Here x is the word vector and the columns of W_up are the up keys, so x W_up lists every neuron's up input. S lists every neuron's spike, 0 or 1, and ⊙ multiplies the two lists number by number. Suppose a neuron's gate inputs build its voltage up to 1.768, as neuron P's did at barked, and its up input is 0.5. Against SiLU applied to a gate input of 1.768:

The spike passes the up input at full size and drops the size of the gate input, which SiLU uses to scale the result.

8. Surrogate gradients and backpropagation through time

Training lowers the loss, a number that grows the worse the prediction is. For each weight it finds the gradient, how much the loss changes when that weight is nudged, by working backward from the loss, and then moves the weight a little to lower the loss. In a spiking model the backward path passes through each spike, and a spike is a jump: as the voltage rises, the output stays at 0, jumps to 1 at the threshold, and stays at 1. Nudging the voltage almost never changes the output, so the gradient through a spike is 0 almost everywhere, and the keys that produce the inputs never learn.

A surrogate gradient keeps the real jump in the forward pass, which computes the output. In the backward pass, which computes the gradient, it replaces the jump with a smooth bump, tallest at the threshold and flatter further away. A common bump, with a steepness k, is:

At barked in The dogs that barked, with θ = 1.5 and k = 10:

Neuron S did not fire, but it was only 0.052 below the threshold. A small change would have made it fire, so it gets the larger gradient. Firing would have added the wrong verb, was, so training on this sentence pushes its voltage down. Neuron P fired with 0.268 to spare, so a small change would not stop it, and its gradient is small.

The steepness k is also a training setting. With k = 2 instead of 10:

Neuron P now gets almost six times as much gradient and neuron S about twice as much, because the k = 2 bump is much wider and still high at both voltages. A wider bump keeps more neurons learning, but it also sends gradient to neurons that a small change would not flip.

So far the gradient has taken one path, through the spike at barked to the key that produced the 1 there. The carried voltage gives it a second path, back through time. Neuron P's voltage at barked was 0.8 times its voltage at that, plus 1, and its voltage at that was 0.8 times its voltage at dogs. So the gradient also reaches the dogs step, multiplied by 0.8 × 0.8 = 0.64, and from there P's key, through the dogs vector. This is how training learns that a plural subject two words back should raise neuron P. Following the gradient back through earlier steps is called backpropagation through time.

The carried voltage also slows training down. An ordinary FFN handles each word on its own, so a GPU (graphics processor) can process all the words of a text at once. The leaky update alone could be computed that way too, since it unrolls into a weighted sum of earlier inputs, but the threshold breaks this. Whether a neuron fires, and how much the reset removes, depends on the voltage it has reached, so each step waits for the one before. Backpropagation through time must also keep every step's voltage in memory, so on long texts the gradient is followed back only a fixed number of steps, such as 64. This is truncated backpropagation through time. The voltage itself still carries past the cut and is reset to 0 only at the start of each document, so one document does not leak into the next.

9. Measuring the cost and quality of spiking models

Over the two sentences, the layer runs 4 words × 2 neurons = 8 neuron-steps per sentence, 16 in all. With ReLU, 8 of the 16 are active, both neurons at the subject and both at barked, and each multiplies its two-number value by its activation. The spiking layer fired 2 times out of 16, a sparsity of 87.5%, and each spike added its value with no multiplication:

Computing the inputs costs the same in both layers. The spiking layer, though, pays for its memory, as the last row shows: each neuron multiplies its voltage by β at every step. In this tiny layer the two come out even at 16. With values of d numbers, the ReLU layer needs 8 × d multiplications while the leak stays at 16, so in a real model, whose values hold thousands of numbers, the leak is small next to what the spikes save.

With a gated FFN the saving is smaller. A firing neuron passes on its up input, an ordinary number, so its value, its row of W_down (the matrix whose rows are the neurons' values), is still multiplied by that number. The saving comes from the silent neurons, whose rows are skipped.

Whether the spiking version is worth it is judged against a baseline, the same model without spikes, on three measurements:

• Perplexity measures the quality of the predictions. The loss on one word is −ln p, where p is the probability the model gave the correct word, and perplexity is e raised to the average loss over a text. A perplexity of 20 means the model is, on average, as unsure as if it were choosing evenly among 20 words. Lower is better.
• Inference latency is the time the model takes per word. Saved multiplications do not automatically make a model faster, because GPUs are built for large blocks of multiplication and skip zeros only when the software is written to do so.
• Peak memory is the most memory in use at any moment. When the model generates text, the carried voltage adds one number per neuron, however long the text is. Training must also keep the voltage of every step it will go back through. If every FFN neuron in Llama 3 8B were spiking, that would be 32 layers × 14,336 neurons = 458,752 voltages, 0.9 MB per step at 2 bytes each, or 1.9 GB for a text of 2,048 tokens. An ordinary model already stores one gate activation per FFN neuron per token for its own backpropagation, the same amount, so the voltages add a cost of a size training already pays.

To tell which setting caused a change, each run varies only the settings under study, such as β and θ for the neurons or k and the truncation length for training, and keeps everything else fixed. Three decays × two thresholds is 6 runs. Each run is reported as a relative change, (hybrid − baseline) / baseline. A perplexity of 23.5 against a baseline of 20.0 is 3.5 / 20.0 = +17.5%, which for perplexity means worse predictions.

10. Threshold sweeps and dead or saturated neurons

Changing θ changes which neurons fire, and so the predictions. The two sentences with three thresholds:

With θ = 1, the 1 from barked reaches the threshold on its own, so both neurons fire there whatever the subject, and the verbs tie. The neuron that matches the subject also fires at the subject itself, since 1.2 ≥ 1, which is the third spike in each sentence. With θ = 2, nothing fires. The right verb comes out for both sentences only when the threshold is above 1.448, the wrong neuron's voltage at barked, and at most 1.768, the right neuron's.

The two ends of the table show two ways a neuron can fail. A neuron that never fires, whatever its input, is dead. It adds nothing to the output, and its voltage stays far below the threshold, where the surrogate bump is nearly flat, so training gets almost no gradient to revive it. With θ = 2, neither neuron fires on these sentences, which is how a dead neuron looks from outside. But neuron P is only 0.232 below the threshold at barked, so with k = 10 its surrogate gradient is still 1 / (1 + 10 × 0.232)² = 0.091, enough for training to bring it back.

A neuron that fires at almost every step is saturated, and its spike says nothing about its input. θ = 1 shows the start of this: both neurons fire at barked whatever the subject, so that spike no longer carries the subject. To find both kinds in a real hybrid, one counts every neuron's firing rate over a large body of text.

11. Memory length of a leaky neuron

An input that enters the voltage is multiplied by β at every later step, until a spike removes part of it. After n steps, the share of it that remains is βⁿ:

Adding up the shares of all past inputs, 1 + β + β² + …, gives 1 / (1 − β), the last column of the table. That many steps, each kept in full, would hold the same total, so it serves as a rough count of how many steps a neuron remembers. With β = 0.8, our neurons remember about five words, and the subject, two words before barked, keeps 0.8² = 0.64 of its input there. With β = 0.5, only a quarter would remain, neuron P would reach 0.25 × 1.2 + 1 = 1.3 at barked, below the threshold, and the layer would predict nothing.

A longer sentence needs a β closer to 1. In The dogs in the yard that barked ___, the subject comes five words before barked, and the words in between give both neurons 0. With β = 0.8, neuron P reaches 0.33 × 1.2 + 1 = 1.39 and misses the threshold. With β = 0.9, it reaches 0.59 × 1.2 + 1 = 1.71 and fires, while neuron S stays at 0.59 × 0.7 + 1 = 1.41. But in the short sentence, β = 0.9 lets the neuron that does not match the subject reach 0.81 × 0.7 + 1 = 1.567 and fire too, so the verbs tie. A longer memory therefore also needs a higher threshold. Any θ above 1.567 and up to 1.71, such as 1.65, gives the right verb in both sentences.

The table predicts memory from β alone. To measure how much a trained model actually uses its carried voltage, set the voltage to 0 at some step and see how the loss changes. This is a reset ablation. In The dogs that barked ___, reset both neurons right after that. At barked, each neuron then has only the 1 from barked, nothing fires, and both verbs score 0. Softmax turns scores into probabilities by raising e to each score and dividing by the total, so equal scores give each verb one half. The loss is −ln of the probability given to the correct verb:

The reset raises the loss by 0.693 − 0.313 = 0.38, which measures how much this prediction relied on the voltage carried across that. Repeating the reset at different distances before the prediction, over a large text, shows how far back the model actually reaches. That can be shorter than 1 / (1 − β), because each spike's reset removes voltage, or longer, because a small leftover can tip a neuron sitting just below the threshold.

12. Spiking readout of a recurrent state

A recurrent model, which reads a text one word at a time, keeps its memory in a state of ordinary numbers. Sending that state out as spikes is called a spiking readout. Rounding a state already costs recall. If the state gives the right answer 0.4 and a wrong one 0.2, rounding to the nearest of −1, 0, and 1 turns both into 0, and the model can no longer tell them apart. A spike is coarser still, with only two levels. Send 0.4 into a neuron with θ = 1 for five steps. If the voltage is reset to 0 before every step, 0.4 never reaches the threshold:

The value is lost. If the neuron instead carries its voltage, with a soft reset and no leak, the part below the threshold builds up:

Two spikes in five steps is a rate of 0.4, the value that went in. This is rate coding, with the carried voltage spreading the value over several steps, and like rate coding it needs several steps for the same value. That means either several spiking steps per word, which multiplies the cost, or a state that changes slowly from word to word. With one step per word, as in our layer, each number gets a single bit.

A spiking readout therefore differs from the original model in two ways. The first is precision. One bit per step is like rounding each number to 0 or 1, and a recurrent state also holds negative numbers, which a single neuron cannot send, so a signed value needs two neurons, one for each sign. The second is dynamics. The carried voltage, which plain rounding lacks, can win back some precision by spreading a number over time. The reset-every-step neuron above is a plain rounded model, with lost precision and no dynamics, so comparing a spiking model with a model rounded to the same two levels isolates what the dynamics contribute.

13. Distillation into a spiking model

Training a large spiking model from scratch is expensive, because the carried voltage forces the words through one at a time and backpropagation through time has to store every step's voltage. A cheaper route is distillation. Take an ordinary model that is already trained, the teacher, and train a spiking model, the student, to give the same probabilities for the next word. The student usually starts from the teacher's weights and only has to match its outputs, which takes far fewer training words than learning language from scratch. The gap between two lists of probabilities is measured by the KL divergence (Kullback–Leibler divergence), where p is the teacher's list, q the student's, and i runs over the words:

For The dogs that barked ___, suppose the teacher gives were 0.85, was 0.05, and are 0.10, and a spiking student gives 0.70, 0.10, and 0.20:

The KL divergence is 0 only when the student matches the teacher exactly, and it grows as the two lists move apart. A single term is negative wherever the student gives a word more than the teacher does, as for was and are here, but the total never is. Training lowers it the same way it lowers any other loss.

SpikingMamba applies this to Mamba-2, a state-space model whose state is a matrix, and distills it into a model whose layers receive spikes instead of ordinary numbers. At 1.3 billion weights, the average accuracy on six standard tasks drops from 59.40 to 54.62, and the perplexity on WikiText-2 rises from 10.42 to 15.17. Recall of facts given earlier in a text is not among the reported results, yet it is where two-level steps would most likely hurt. Rounding erases small leads such as 0.4 against 0.2, which recall depends on, while next-word prediction usually survives because the rest of the sentence still points to the right word.

14. Spike-frequency adaptation and the adaptive LIF neuron

Many real neurons become harder to excite the more they fire. Each spike raises the neuron's own threshold a little, and the increase fades over time. This is spike-frequency adaptation, and an LIF neuron with it is called an adaptive LIF (ALIF) neuron. Its threshold has a fixed part θ₀ and a moving part b:

Every step b leaks by a factor ρ (rho), as the voltage leaks by β, and every spike adds α (alpha) to it. With θ₀ = 1, α = 1, and ρ = 0.8, a single spike makes the threshold jump and then sink back toward θ₀:

A neuron that keeps firing keeps its threshold high. Firing at an average rate r adds αr to b per step, while the leak removes (1 − ρ)b, so b settles where the two are equal:

For two firing rates:

The figure follows both neurons for 20 steps. The busy neuron's threshold climbs and then zigzags around 3.5, rising after each spike and sinking in between. The quiet neuron's jumps to about 2 after each spike and sinks back toward 1, averaging 1.5 over a long run. Given the same input, the busy neuron is much further from firing. Adaptation therefore shifts activity away from neurons that respond to everything and toward neurons that respond rarely, and that shift can also decide where a new fact should go.

15. Adaptive thresholds as write allocation

Learning a new fact means changing the values of some neurons, and choosing which ones is called write allocation. The obvious choice, the neuron most active for the new fact, can damage the old facts that neuron already holds. One answer is TF-IDF (term frequency–inverse document frequency), a score taken from search engines, which prefers neurons that are active for the new fact but rarely active for anything else. It pays for this by counting every neuron's activity over a background collection of inputs. An adaptive threshold keeps a similar count on its own, as a side effect of firing.

Take four neurons that have been reading about dogs and cats and now have to learn a fact about quokkas. Each has an adaptive threshold θ = θ₀ + b, with θ₀ = 1, α = 1, and ρ = 0.8, where b jumps by α after each spike and is multiplied by ρ every step. To keep the example short, these neurons carry no voltage: a neuron fires when a word's input reaches its threshold, and the inputs are chosen for illustration. A new fact is written into the neuron whose input is largest compared with its threshold. The layer reads dog cat dog cat and then quokka. Each row shows the thresholds in force when that word arrives, so a spike shows up as a higher threshold in the next row:

Neuron 0 responds most strongly to quokka, at 1.5, but it is also the neuron that fires for dog. By the time quokka arrives, its two dog spikes have faded to 0.8 and 0.8³ = 0.512, so its threshold is 1 + 0.8 + 0.512 ≈ 2.31. Neuron 3 responds only to quokka and has never fired, so its threshold is still 1. A neuron's input divided by its threshold says how close it comes to firing, and is called its excitability:

Neuron 3 is the most excitable, and neurons 1 and 2 are further behind, at 0 and 0.4 / 1.51 = 0.26. So the quokka fact is written into neuron 3, and the neurons that hold dog and cat are left alone. Writing into the most active neuron, neuron 0, would have changed the neuron that answers for dog. Nothing outside the neurons kept count. Each one only tracked its own recent spikes.

The two methods weigh the past differently. TF-IDF counts every background input equally, however long ago it was read. The adaptive threshold weighs recent activity more, because b leaks. That frees neurons whose old facts are no longer in use, but it also stops protecting them. Once neuron 0 has been quiet for a while, its threshold sinks back toward 1, and a new fact can be written over the neuron that answers for dog, where TF-IDF would still steer it away. The timing of activity can also be turned into a rule of its own.

16. Refractory and facilitating allocation

An adaptive threshold rises when its neuron fires. Learning a fact can count as activity in the same way. When a fact is written, add an amount c to the b of the neurons written to, and let it fade by ρ every step like the rest of b. By the next word it has faded once, and after g steps c × ρ^g is left. With c = 0.5 and ρ = 0.8:

Whether that leftover makes the written neurons harder or easier to excite depends on the sign of c:

• Refractory (c > 0): the neurons just written to get a higher threshold and become harder to excite, so a fact that arrives soon after goes to other neurons, and facts learned close together are kept apart. The name comes from the refractory period, the short time after a spike when a real neuron cannot fire again.
• Facilitating (c < 0): the neurons just written to get a lower threshold and become easier to excite, so a fact that arrives soon after tends to share them, and facts learned close together are linked.

Go back to the four neurons that read dog cat dog cat. The first quokka fact went into neuron 3, the only neuron that responds to quokka alone, while neuron 0 responds to both quokka and dog. One step later, a second quokka fact arrives with the same inputs, [1.5, 0, 0.4, 0.9]. Neuron 0's threshold has meanwhile faded to about 2.05, an excitability of 1.5 / 2.05 = 0.73:

The overlap of two writes is the number of neurons they share divided by the number of neurons either one used. Under the refractory rule, the two quokka facts share no neuron, an overlap of 0, though the second lands on the neuron that answers for dog, the price of keeping facts apart. Under the facilitating rule, they share neuron 3, an overlap of 1.

The gap in time decides how much either rule matters. Had the second fact come 10 steps later, with nothing read in between, the leftover c would have faded to 0.05, and neuron 0's threshold would have sunk to about 1.14, an excitability of 1.5 / 1.14 = 1.31. Both rules would then write the fact into neuron 0, the neuron that answers for dog, because the dog cat dog cat reading is too far back for the adaptive threshold to protect it. So the refractory rule pushes facts learned close together apart, the facilitating rule links them, and both effects fade as c × ρ^g does. TF-IDF can produce neither, because it keeps no record of when anything happened.

Experiments on brains point toward facilitation. In mice, two memories formed a few hours apart were stored in overlapping groups of neurons and became linked, while memories formed a week apart were not. The researchers trace this to excitability. Forming a memory leaves its neurons more excitable for some hours, and more excitable neurons are more likely to be recruited for the next memory. That is the facilitating rule, with c < 0 and a fading time measured in hours.