Three populations of scanned posts, from Substack's own labels: fully human (Fully Human-Written), fully AI (Fully AI-assisted text), and mixed (the four labels between). This page asks how they differ on what can be measured without reading the posts: length, engagement, punctuation, formatting, vocabulary, and, for a judged subsample, rhetorical habits. Then the same question for publications, grouped by how consistently their posts are flagged. A fourth, grey group is carried alongside: posts whose writer turned detection off, which have no label.
Reading the results. Each variable is drawn as a : one row per group on a shared axis and shared bins, bar height the share of that group's posts in the bin, so groups of different size read on the same scale, and a rule at each group's . Word count and the per-subscriber rate sit on a ; everything else is linear, with one bar per count for reactions, comments and restacks. Where many posts score exactly zero they get the bar labelled 0 at the left edge, cut to fit with its share written in the row label; the translucent bar at the right collects the tail beyond the 95th percentile (99th on the log axes). Hover a bar for its range and count. The table beneath gives the median per group with a interval, then each pair of groups with , the that a random post from the first group exceeds one from the second, and a from the . Four tests were fixed in advance as () and are reported at the 5% level; every other test is adjusted with across all secondary tests. means the adjusted p-value is below 0.05 and the effect-size interval excludes zero. means the raw p-value is below 0.05 but the adjusted one is not: a pattern that is there in the sample and that a hundred tests would also throw up by chance now and then. is an absence of evidence at this sample size, not evidence of absence. marks comparisons outside the pre-registered family, reported with no correction: the detection-off group, the topic facets, the extended judge questions and the resemblance score.
Detection off. public posts in the sample had Pangram detection switched off by the writer, so they carry no label. They are the grey row on every post-level chart, compared against each labelled group with Cliff's δ and an uncorrected p. These writers chose to hide the result, so the group is self-selected, and resemblance on surface features is not detection.
Cliff's δ for every post-level variable, fully human against fully AI. Positive means human posts score higher. Dark points are discoveries or significant primary tests, orange are borderline, grey are not distinguishable from zero. Bars are 95% bootstrap intervals.
- Posts: length, engagement, titles: the four primary tests, reactions, comments and restacks, title length and form.
- Text features: punctuation, formatting and vocabulary measured by script, the vocabulary analyses and word cloud, and the resemblance score.
- Judged features: rhetorical habits scored by a model judge on a subsample, and a second judge on every post.
- Topic: the AI share by publication category and every effect with topic held fixed.
- Publications: posting rates, subscribers, age and engagement by how consistently a publication is flagged.
Generated . Bootstrap intervals use resamples for medians and for δ.