Statistics for Neuroscience
Distributions
Here are four samples of “firing rate” measurements. For each, would you summarize it with the mean or the median — and which choice would mislead a reader?
From left to right these are Gaussian, right-skewed, bimodal, and long-tailed. The mean and median agree only for the first. On the skewed sample the mean is dragged toward the long tail; on the bimodal sample the mean lands in the empty valley between two clusters, describing no real cell. The fourth looks like a narrow spike with empty space to the right — its long tail is invisible on a linear axis. The next figure uses a log scale to reveal what is hiding there, and introduces the closely related log-normal distribution.
The two summaries are defined as
\[\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i \qquad\qquad \text{median} = \text{middle value of the sorted } x_i.\]
The mean uses every value (so outliers move it); the median only cares about rank (so outliers do not).
When you need a log axis
The same two samples, now on a logarithmic x-axis. Top — a log-normal distribution is strongly right-skewed on a linear axis but becomes a symmetric bell once the x-axis is logarithmic. That is exactly what “log-normal” means: the logarithm* of the values is normally distributed (firing rates, synaptic weights, and many biological quantities are log-normal). Bottom — a long-tailed distribution looks like a single spike on a linear axis, but the log axis reveals a tail spanning more than two orders of magnitude. Whenever values are positive and span orders of magnitude, plot them on a log axis — and label it clearly.*
The Sampling Distribution
Almost everything that follows — error bars, confidence intervals, p-values, the bootstrap — is one idea wearing different hats. This section is that idea.
You record 20 neurons and compute their mean firing rate. Tomorrow you run the identical experiment on 20 different neurons from the same population. Will you get the same number? If not, how far off?
Left is the truth you never observe: every neuron in the population. Right is what you actually get — 20 draws from it, and one sample mean that lands near the true mean but not on it. That gap is not a mistake. It is sampling variability, and quantifying it is what statistics is for.
Repeating the experiment
Suppose you could run the experiment hundreds of times. What would the collection of sample means look like?
Population SD (σ): 6.00 Hz
Observed SD of the 5000 sample means: 1.36 Hz
Formula prediction σ/√n = 6.0/√20 = 1.34 Hz
This histogram of sample means is the sampling distribution. Three things to notice: it is centered on the true mean, it is far narrower than the raw data, and its standard deviation matches \(\sigma/\sqrt{n}\) almost exactly. That is what the SEM is — not a property of your neurons, but the width of this distribution: how much your estimate of the mean would bounce around if you repeated the experiment.
Why \(\sqrt{n}\)
More neurons per experiment ⇒ a tighter sampling distribution ⇒ a more precise estimate. But the width falls as \(\sqrt{n}\), not \(n\): quadrupling your sample size only halves your error bar. This is the arithmetic behind every “just add more cells” conversation — going from 20 to 80 neurons buys you a factor of two, and no more.
The Central Limit Theorem
Everything above used a Gaussian population. Real data usually is not Gaussian. Does any of this survive?
The population (left) is emphatically not normal. Yet the distribution of its sample means (right) is very nearly a bell curve. This is the Central Limit Theorem: averages tend toward normality even when the underlying data do not, and faster as \(n\) grows. It is why t-tests are more forgiving than their assumptions suggest — and why that forgiveness runs out when \(n\) is small or the skew is severe, which is exactly when we reach for nonparametric tests later.
The catch: you only ever run the experiment once. You never see the sampling distribution. So how can you know its width?
Two answers, and the rest of this page is built on them:
- Assume a form and use a formula. If the sampling distribution is normal, its width is \(s/\sqrt{n}\) — the SEM — estimated from your single sample. Confidence intervals and t-tests follow directly.
- Simulate it from your own data. Treat your sample as a stand-in for the population and resample it thousands of times. That is the bootstrap, and it works for statistics that have no formula at all.
Everything below is one of these two moves.
Error Bars and Effect Size
Error bars: SD vs. SEM vs. CI
These three bars answer three different questions and are constantly confused.
Before reading on: which of these three describes your neurons, and which describes your estimate of their mean? Which one shrinks as you record more cells?
With the sampling distribution in hand, the difference is easy to state:
- Standard Deviation (SD) — the spread of your data. A property of the neurons. Does not shrink as you collect more. \[s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2}\]
- Standard Error of the Mean (SEM) — the width of the sampling distribution you just saw. A property of your estimate, not your neurons. Shrinks as \(\sqrt{n}\). \[\text{SEM} = \frac{s}{\sqrt{n}}\]
- 95% Confidence Interval (CI) — built by walking out from your estimate far enough to cover 95% of that sampling distribution. \[\bar{x} \pm t_{0.975,\,n-1}\cdot \text{SEM}\]
Same 30 neurons, three error bars. Which one looks the most impressive — and is that a good reason to plot it?
SEM is the smallest bar — but only because it is the SD divided by \(\sqrt{30} \approx 5.5\), not because the data are tight. Nothing about the neurons changed between the three bars. Use SD to describe your sample, CI to express uncertainty about the mean, and always say in the caption which one you plotted.
What a confidence interval actually means
A statement you will hear constantly: “the 95% CI is [18.2, 22.4], so there’s a 95% chance the true mean lies in that range.” Is that correct?
Studies run: 100
CIs containing the true mean: 97
CIs that miss it entirely (red): 3
Each horizontal line is one lab running the same experiment and reporting its 95% CI. The vertical line is the truth. Close to 95 of the 100 intervals catch it — the exact count wobbles from run to run, because coverage is itself a random quantity — and the red ones miss completely. Those labs have no way of knowing they are the red ones.
So the statement above is wrong, subtly. The 95% describes the procedure, not any particular interval: intervals built this way capture the truth 95% of the time. Your one interval either contains the true mean or it does not — the probability is 0 or 1, you just don’t know which. The honest reading of a CI is “values in this range are compatible with my data,” and its width is the useful part: a wide CI is a confession that the experiment could not pin the answer down.
Effect size
A small p-value says an effect is detectable, not large. The standardized effect size, Cohen’s d, reports magnitude independent of sample size:
\[d = \frac{\bar{x}_1 - \bar{x}_2}{s_\text{pooled}}, \qquad s_\text{pooled} = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}}.\]
Conventionally \(d \approx 0.2\) is small, \(0.5\) medium, \(0.8\) large.
Both panels show the same effect size. Only \(n\) differs. Why does the p-value collapse?
Cohen’s d is identical by construction in both panels — the groups overlap exactly as much, and a randomly chosen neuron from B beats one from A just as often. Only \(n\) differs. Yet the p-value falls from “not significant” to vanishingly small. “Significant” answers can we detect it?; effect size answers does it matter?* Report both, always — a p-value alone tells you as much about your sample size as about your biology.*
p-values
Every test in the next section ends in a p-value. Before running any of them, it is worth being precise about what that number is.
The null hypothesis
Every test starts from a specific, quantitative claim of “no effect” — the null hypothesis, \(H_0\). For two groups, \(H_0\) is not a vague default; it says something exact: both samples were drawn from populations with the same mean. The alternative, \(H_1\), is whatever your real question is — that the drug changes firing rate, that the groups differ.
A test never touches \(H_1\) directly. It only asks how surprising would my data be, if \(H_0\) were exactly true? That single question is the entire computation, no matter which test is running.
This produces an asymmetry worth stating plainly: strong evidence against \(H_0\) lets you reject it, but weak evidence does not let you accept it. “p > 0.05” means my data are not surprising under \(H_0\) — never \(H_0\) is true. Absence of evidence is not evidence of absence.
What a p-value is
The definition has a specific and narrow shape:
A p-value is the probability of observing data at least as extreme as yours, if the null hypothesis were true.
It is a statement about data given a hypothesis, never about the hypothesis given the data. We can build one directly: simulate a world where the null is true (both groups drawn from the same population), collect the differences that arise from noise alone, and see where the real result falls.
Observed difference: 2.29 Hz
Fraction of null world at least
this extreme (red tails) = p = 0.1246
t-test p for comparison: 0.1290
The grey histogram is a world with no real effect — every difference in it is pure noise. The red tails are the outcomes at least as extreme as yours. The p-value is simply their share of the total. Note what this does not tell you: nothing here is a probability that the null is true, and nothing is a measure of how big the effect is.
Three things a p-value is not
| Claim | Why it’s wrong |
|---|---|
| “p = 0.03, so there’s a 3% chance the null is true.” | Backwards. p is P(data | null), not P(null | data). Getting the second requires a prior. |
| “p = 0.03, so there’s a 97% chance it replicates.” | Unrelated. Replication probability depends on true effect size and power, and is usually far lower than people assume. |
| “p = 0.001 means a bigger effect than p = 0.04.” | No. p mixes effect size with sample size. A huge n makes a trivial effect tiny in p — see the effect-size figure above. |
The most useful fact about p-values
No real effect: 5.2% of experiments give p < 0.05
Real effect present: 56.1% of experiments give p < 0.05 (this is POWER)
When there is no effect, p-values are uniformly distributed — every value from 0 to 1 is equally likely, so p < 0.05 happens exactly 5% of the time. That flat histogram is the entire reason α = 0.05 means what it means, and the reason 100 tests on noise yield about 5 “discoveries.” When a real effect exists, the distribution piles up near zero, and the fraction below 0.05 is the test’s power.
Choosing a Test
Three questions decide your test: Is the data roughly normal? Are the measurements paired? Are you running many tests at once?
Start with the distribution: is it normal?
A parametric test (t-test, Pearson correlation) assumes the data come from a known distribution family — almost always the normal, defined by two parameters, mean \(\mu\) and SD \(\sigma\). If that assumption is wrong, the test’s p-values are wrong.
Before running a t-test: how would you check whether these data are normal? What tools do you have?
Shapiro–Wilk: W = 0.897, p = 0.0001
D'Agostino–Pearson: K2 = 11.85, p = 0.0027
Lilliefors: D = 0.163, p = 0.0008
Anderson–Darling: A2 = 2.03 (5% crit = 0.74)
The histogram and QQ plot are your first line of defense — the QQ points bend away from the line whenever data are skewed. Formal tests put a number on it:
| Test | Best at | Caution |
|---|---|---|
| Shapiro–Wilk | General-purpose, strong for small n | The usual default |
| D’Agostino–Pearson | Detecting skew & kurtosis | Needs n ≳ 20 |
| Lilliefors (KS, corrected) | Comparing to a normal with unknown \(\mu,\sigma\) | The right way to run KS for normality |
| Anderson–Darling | Sensitive to the tails | Compare statistic to critical value, not a p |
A common trap: running plain Kolmogorov–Smirnov against a normal fit to the same sample. KS assumes the reference distribution is fixed in advance — feed it parameters estimated from the data instead, and it becomes so conservative it almost never rejects, silently failing to catch real non-normality. Lilliefors uses the identical D statistic but simulates the correct null distribution for the estimated-parameter case (this is the test behind MATLAB’s lillietest).
A further caveat: with large n these tests flag trivial, harmless deviations; with small n they miss real ones. Use them to support what the QQ plot already shows, not to replace your eyes.
Paired vs. unpaired
Data are paired when the same unit (neuron, animal, subject) is measured twice. Pairing removes between-unit variability and dramatically increases power.
The two clouds overlap almost completely. Is there no effect here?
Every neuron went up — the change is small but perfectly consistent. The big between-neuron baseline spread is what makes the unpaired view look uninformative. The paired test removes that spread and finds the effect easily. If the design is within-subject, the test must be paired.
Parametric vs. nonparametric
When normality fails (small n, skew, outliers), use a rank-based test: Mann–Whitney U (unpaired), Wilcoxon signed-rank (paired), Spearman (correlation).
The two tests below give different p-values on the same data. Which do you trust?
With skewed data and n < ~15, the nonparametric p-value is the trustworthy one. Rank tests cost a little power when data really are normal, but protect you when they are not — a good default for small neuroscience samples.
Comparing more than two groups
Experiments rarely stop at two conditions — three drug doses, four genotypes, five cortical layers.
With three groups, why not just run three t-tests (A vs B, A vs C, B vs C) and report whichever is significant?
Three groups drawn from the SAME population — no real differences:
'Any of 3 pairwise t-tests significant': 12.3%
One-way ANOVA significant: 4.8%
Three pairwise tests give you three chances to be wrong, so the false-positive rate climbs to roughly 1 in 8. The ANOVA asks one question — “is there any difference among these groups?” — and holds the error rate at 5%.
What the F statistic is
ANOVA compares two estimates of variance:
\[F = \frac{\text{variance between group means}}{\text{variance within groups}}\]
If the groups are truly identical, both numerator and denominator estimate the same noise and \(F \approx 1\). A real difference inflates the numerator. Crucially, the group means alone cannot tell you — the same means can be convincing or meaningless depending on the scatter around them.