Statistical glossary

Describing a group

Median

Sort a group's values and take the middle one. Half the posts are below it, half above. The site reports medians rather than means because the data are heavily skewed: a few posts have tens of thousands of reactions and most have a handful, and a mean of such numbers describes the few. A median of 2 reactions for fully human posts means a typical human-written post gets two.

Quartiles

The values one quarter and three quarters of the way up the sorted list. The range between them holds the middle half of the group. "345 to 1,110 words" for human posts says half of them fall in that range, and it is a better picture of a group than a single number.

Histogram

The plot on every comparison card. Values are counted into bins along the axis and each bin's bar shows how many posts fell in it. Here bar height is the share of the group rather than the raw count, so a group of 1,370 posts and one of 2,600 read on the same vertical scale. The bins are shared across groups, so bars line up. A bar at the far left labelled 0 holds the posts that score exactly zero, so the first real bin is not swamped by them. The axis stops at the 95th percentile of all posts together for rates and the 99th for counts, and the translucent right-most bar absorbs everything beyond, so that one extreme post does not flatten the rest; its tooltip gives the count.

Logarithmic axis

An axis on which equal steps are equal multiples: 1, 10, 100, 1,000 are evenly spaced. It is used for counts and rates, where values span several orders of magnitude; on a plain axis the handful of posts with thousands of reactions would push everything else into a sliver at the left. Zero has no place on a logarithmic axis, which is why zeros are drawn as a bar of their own.

Uncertainty

95% confidence interval

A range built so that, in 95% of samples like this one, it would cover the true value. It is reported after every median and every effect size. A narrow interval says the sample pins the value down; a wide one says it does not. Two intervals that overlap do not prove two groups are the same, and a difference whose interval includes zero is "not distinguished from zero", not "zero".

Bootstrap

How the intervals for medians and effect sizes are made. Draw a new sample of the same size from the posts, with replacement, so some posts appear twice and some not at all; recompute the statistic; repeat a thousand times (three hundred for effect sizes). The middle 95% of those recomputed values is the interval. It needs no assumption about the shape of the data, which matters here because nothing is bell-shaped.

Wilson interval

The confidence interval used for shares, such as "26.5% of scanned posts are fully AI". It is the standard choice for a proportion because it behaves properly near 0% and 100%, where the textbook formula produces intervals that run past the possible range.

Comparing groups

p-value

The probability of seeing a difference at least as large as the one observed if the groups were in truth the same. A p-value of 0.001 means that, were there no real difference, a gap this large would turn up about once in a thousand samples. It measures surprise, not size: with thousands of posts a trivial difference can have a tiny p-value, which is why every test is paired with an effect size.

Mann-Whitney U test

The test behind every pairwise comparison of numbers on this site. Instead of comparing means, it asks: taking one post at random from each group, how often does the first exceed the second? If the groups are the same the answer is about half the time; the test's p-value says how far the observed answer is from that. Because it works on ranks it is unmoved by extreme values and needs no assumption about the shape of the data.

Kruskal-Wallis test

The three-group version of Mann-Whitney: do the human, mixed and AI groups differ anywhere? It is used only as a gate, shown in each card's heading. When it finds no difference the three pairwise comparisons beneath should be read as noise even if one happens to be small.

Chi-square test

For categorical variables, such as whether a title contains a colon or which rhetorical habit a judge found. It compares the table of counts (group by category) with the table expected if group and category were unrelated, and gives a p-value for the gap.

Fisher's exact test

The replacement for the chi-square test when a two-by-two table has a cell expected to hold fewer than five posts, where the chi-square approximation is unreliable.

Cramér's V

The effect size for a table of counts, from 0 (the category mix is identical in every group) to 1 (every group has its own categories). Around 0.1 is small, around 0.3 medium. The AI-label mix across publication categories has V of about 0.3: topic matters a lot.

How big is the difference?

Cliff's δ

The effect size reported for every pairwise comparison of numbers. Take every possible pair of one post from group A and one from group B. Cliff's δ is the share of pairs where A's post is higher minus the share where B's is higher. It runs from −1 (every A below every B) through 0 (no difference) to +1 (every A above every B). It is in the same units for every variable, so word count, em-dashes and reactions can be compared on one chart, and it does not depend on a few extreme posts. Rough reading: 0.15 is small, 0.33 medium, 0.47 large. Reactions per post, fully human against fully AI, has δ of about +0.39: a random human post out-reacts a random AI post in roughly 70% of pairs.

P(a > b)

The same information as Cliff's δ in plainer words: the probability that a random post from the first group scores higher than a random post from the second. 50% is no difference. It is (δ + 1) / 2, so δ = +0.39 is 70%.

Stratified Cliff's δ

Cliff's δ computed over pairs of posts whose publications share a topic, then pooled across topics with each topic weighted by its number of pairs. The groups write about different things (AI-labelled posts lean towards Business and Technology, human ones towards Literature and Culture), so part of any overall difference is topic, not authorship. The stratified value is what remains with topic held fixed. The Topic section of the Comparisons page shows the two side by side; where the stratified value is much smaller, the overall difference was largely topic mix.

Log ratio

Used for posting rates before and after Pangram arrived (21 July 2026, the earliest date Substack scans): the logarithm of after divided by before. Zero is no change, +0.69 is a doubling, −0.69 a halving. Unlike a percentage change it treats rises and falls symmetrically.

Many tests at once

Primary test

Four comparisons were fixed before any data were looked at: word count and reactions per post at the post level, posting rate after Pangram arrived and subscriber count at the publication level. Each was asked once, in advance, so each is reported at the 5% level with no adjustment. The label "significant (primary)" belongs only to these.

Benjamini-Hochberg adjustment

Everything that is not a primary test belongs to one family of secondary tests, around a hundred of them. Run a hundred tests on groups that are truly identical and about five will have p below 0.05 by chance. The Benjamini-Hochberg procedure raises each p-value according to its rank among the family so that, among the results called discoveries, the expected share of false ones is held at 5%. The adjusted p is what the status badges use.

Discovery

Adjusted p below 0.05 and an effect-size interval that excludes zero. A difference that survives the correction for many tests and is of some size.

Borderline

Raw p below 0.05 but adjusted p at or above it. The pattern is there in this sample, and it is the kind of thing a hundred tests would produce by chance now and then. It is a lead for a larger sample, not a finding.

No difference detected

Nothing passed either bar. This is an absence of evidence at this sample size, not evidence that the groups are the same; a small real difference would look exactly like this.

Exploratory

A comparison outside the pre-registered family, reported with no adjustment at all: the within-topic facets, the detection-off group, the extended judge questions, the resemblance score. These were added after the first results were seen, or repeat an already-asked question on a subset, so their p-values are not comparable with the family's. Read them as description with a rough sense of noise, not as tests.

Judges

Cohen's κ

Agreement between two judges beyond what matching by chance would give, used to compare the cheap second judge with the reference judge. Two judges who each say "none" 60% of the time will agree often by accident; κ removes that. 0 is chance agreement, 1 perfect; above 0.6 is usually called substantial. The second judge is extended to the full set only on questions where raw agreement is at least 85%.

Publications

Inverse-probability weighting

Publications enter the sample through a random post, so a publication that posts ten times a week is ten times as likely to appear as one that posts weekly. Unweighted figures therefore describe output ("the typical sampled post comes from a publication that…"); weighting each publication by 1/(its posts after Pangram arrived) describes publications. Both are shown; tests are on the unweighted rows.

The resemblance score

Logistic regression

A model that combines all the text features into one number between 0 and 1: a weighted sum of the features, squashed into a probability. It is fitted to tell fully human from fully AI posts, and the weights (coefficients) say which features pull a post towards the AI side. Applied to the mixed and detection-off groups, it answers "which labelled group does this post resemble?" It is a resemblance score, not a detector.

Standardised

Before fitting, each feature is shifted to mean 0 and scaled to spread 1 (rates are log-transformed first). The coefficients then share a unit, one standard deviation of the feature, and can be compared with each other.

Cross-validation

A model scored on the posts it was fitted to looks better than it is. So the labelled posts are split into five parts; each part is scored by a model fitted on the other four. Every human and AI post's score comes from a model that never saw it. The unlabelled groups are scored by a model fitted on all labelled posts.

AUC

Area under the ROC curve: the probability that the model scores a random AI post above a random human post. 0.5 is coin-flipping, 1 is perfect separation. It summarises how well the surface features separate the two labelled groups at all.

Vocabulary

Rarefaction curve

The number of distinct words in a body of text grows with its size, quickly at first and then ever more slowly, so two groups of different size cannot be compared on their raw vocabulary counts. A rarefaction curve draws posts from a group in random order and plots the distinct words seen so far against the words read so far, averaged over many random orders. Two groups can then be read at the same number of words. A lower curve is a smaller working vocabulary: the same amount of text, fewer different words. The curves stop at the smallest group's total, since beyond it there is nothing to compare.

Jaccard overlap

For two posts, the words they share divided by the words either uses: 0 if they have no word in common, 1 if their vocabularies are identical. Averaged over every pair of posts in a group it measures how much the group's posts draw on the same words. It is computed on each post's first 300 words so that longer posts do not overlap more just by being longer. The interval reported bootstraps each post's mean overlap with the others, which treats the pairs as independent when they are not; read it as a rough guide.

Coverage share

The share of a post's words that are among its own group's 2,000 commonest words. A high share means the post stays within the group's shared stock; a low one means it reaches outside it. Each group's list is computed from that group's posts, so the measure asks how tightly each group's posts cluster around their own common vocabulary.

log2 rate ratio

For a word, its rate per million words in one group divided by its rate in the other, on a log2 scale: +1 means twice as common in human posts, −3 means eight times as common in AI posts. Each count has 0.5 added before the ratio so that a word absent from one group gives a large but finite value. Because the ratio is unstable for rare words, only words found in at least 1% of the two groups' posts (2% within a single topic) are ranked, and the word cloud shows the 60 largest ratios on each side. The topic-adjusted version computes the ratio within each large category and pools the results, weighting each category by the harmonic mean of the two groups' word totals there; it is to the raw ratio what the stratified Cliff's δ is to the overall one.

Cautions

Circularity

Pangram's label and the text features are both computed from the same text. If the detector reacts to, say, em-dash density, then em-dashes will differ by label whether or not they differ by authorship. A text-feature result therefore describes what the label picks up, which is interesting in itself, but it is not independent evidence about how AI-written text differs from human text. Engagement, posting rates and subscriber counts are outside this loop.