Em Dashes: The Passage of Bias

Introduction

Em dashes (—) seem to be everywhere in LLM-generated writing. Once you start noticing them, they are hard to unsee. And once seen, they grab everyone’s attention, and we ask: why do LLMs seem to be obsessed with em dashes?

Several explanations have been proposed regarding the origins of this phenomenon:

• Prompt priming
• The training corpus
• Something between pretraining and the released instruct model

Although commentary on em dashes as a marker of AI-generated writing is common, evaluating their origins is challenging in the absence of training data. Freeburg’s work infers style shifts from observable fingerprints when model pipelines are inaccessible [1].

The em dash serves as a visible indicator of a broader phenomenon. If this stylistic marker can be transmitted between models via synthetic data, preference tuning, or model-generated demonstrations, the underlying issue goes beyond punctuation. It is necessary to consider what other latent biases, less apparent and more difficult to detect, may propagate through these training pipelines.

Given these constraints, open models provide an opportunity to examine not only whether chat models employ em dashes, but also when this stylistic marker emerges.

In this analysis, an em dash is not the cause of the behavior. It is an easy-to-count, visible marker that lets us track an extended shift in writing style. The study counts exact Unicode em dash characters (U+2014) across open checkpoints and related data artifacts. In OLMo-3, the marker is nearly absent at early instruct-path checkpoints, but increases sharply at the Instruct-DPO and final Instruct stages. OLMo-2 exhibits a noisier version of this pattern. Tulu preference data shows a weak chosen-side skew. K2-V2 indicates that final-stage DPO is not necessary when single-source, model-written demonstrations transmit style directly.

Pretraining results show that em dashes occur at low rates: 0.115 per 1,000 words in Dolma Reddit, 0.589 in DCLM, and 0.964 in Dolma News. These frequencies are substantially lower than those observed in OLMo-3 Instruct-DPO and the final Instruct outputs, showing a shift in the default style, likely introduced after pretraining.

Why is this hard to answer?

With closed models, only the final behavior is observable, necessitating post hoc hypotheses. Consequently, discussions about em dashes often devolve into a series of assumptions: the model may have copied polished editorial text, inherited writing styles from a more advanced teacher model, amplified confident answers through preference tuning, or adopted stylistic tendencies from prompt templates.

Related work has shown that surface writing habits can leave measurable traces in generated text [2], that instruction-tuned outputs can drift away from human baselines [3], and that format preferences can be amplified by alignment procedures [4].

Any of these accounts may be accurate or inaccurate. The main challenge is that the model pipeline is typically inaccessible, making it difficult to validate or refute these hypotheses.

If the closed-source pipeline were accessible, it would be possible to determine where stylistic changes occur. When pretraining corpora contain the marker at modest rates, but released chat models use it more frequently, the significant change likely occurs during intermediate stages such as instruction tuning, preference data integration, synthetic demonstrations, reward modeling, reinforcement learning, decoding defaults, or a combination of these processes.

Checkpoints clarify the issue by enabling comparison of model behavior before and after specific training stages, as well as inspection of related data. OLMo is vital here because AI2 releases unusually open model pipelines: training data, model weights, and intermediate checkpoints across stages of development [5] [6] [7]. Very few institutions make this much of the pipeline visible. That openness lets us move beyond speculation and ask where the signal first appears.

That gives us a practical question. Are these markers already common in the training data? Are they mostly a decoding or conversation-history effect? Or do they become more visible during post-training, when models learn from synthetic demonstrations, preference data, or model-written judgments?

Measurement approach

We use one simple measurement: exact U+2014 em dashes per 1,000 generated words.

Hyphen-minus, en dash, and double hyphen are not converted to em dashes in this measurement. This approach may miss some stylistic variants, but it assures that the metric remains plain and simple. The focus is on tracking a single visible marker through the pipeline, without introducing unnecessary assumptions.

For checkpoint comparisons, the runs use matched prompt sets within each model lineage and fixed decoding settings across comparisons. Rates are normalized by generated words rather than model tokens, so they are comparable across tokenizers but should not be read as tokenizer-level measurements. Greedy runs remove sampling noise; medium-temperature runs use T=0.7, and high-temperature runs use T=1.0. The reproducibility package (details at the end of the post) will pin the top-p, maximum-token, prompt, and word-counting settings for every figure.

The checkpoint rates are estimates based on matched runs and do not provide exhaustive characterizations of each model. Greedy runs often have low counts, with some point estimates based on only a few em-dash events across 4,000 to 5,000 generated words. Where possible, intervals are calculated as exact Poisson intervals over event counts. The decimal values should be interpreted as audit references rather than exact measurements.

We use five evidence sources: OLMo-2 checkpoints [5] [6], OLMo-3 checkpoints [7], Tulu preference data [8] [9], K2-V2 and its SFT path [10] [11], and negative controls using FineWeb-Edu [12] plus multi-turn stripped-history runs.

Tracing the signal

OLMo-3: the tracer appears after DPO

A quick note on terms. Greedy decoding means temperature = 0: at each step, the model takes its highest-probability next token. SFT, or supervised fine-tuning, trains the model to imitate demonstration answers. DPO, or direct preference optimization, trains on chosen/rejected response pairs. Instruct is just a label for the final chat- or instruction-facing release.

Under greedy decoding, the early instruct-path checkpoints are near zero: Base produced 0 em dashes spanning 3,944 words, Think-SFT produced 1 across 5,109 words, and Instruct-SFT produced 0 across 3,709 words. Then the tracer appears strongly: Instruct-DPO produced 17 across 4,357 words, and final Instruct produced 25 across 4,137 words.

The SFT checkpoints remain near zero, while the DPO checkpoint shows a significant increase. The final instruct release increases further.

The RL-Zero controls make the result more useful. OLMo-3 includes RL-Zero branches for math, code, instruction-following, and general reasoning; the OLMo-3 paper provides the full training context [7]. In the logged greedy runs, RL-Zero Math, Code, IF, and General produced zero em dashes. At these sample sizes, zero is still an estimate with an upper interval, not proof of absence. The useful contrast is narrower: in this model family, the DPO/instruct path produced a visible greedy signal while the logged RL-Zero paths did not. “Post-training” is too broad an explanation.

OLMo-2: a noisier cousin

OLMo-2 offers a second model lineage and functions as a cautionary example.

Under greedy decoding, Base and SFT were not distinguishable from zero at these sample sizes. DPO produced 7 em dashes over 4,580 words, or 1.528 per 1,000. Final Instruct dropped to 1 across 4,797 words, or 0.208 per 1,000.

Unlike OLMo-3, this result does not show a clear monotonic increase. In this run, the tracer first becomes visible at DPO under greedy decoding, while SFT and final Instruct remain low-count estimates. Later training stages can change the presence of the marker.

At higher temperatures, more of the distribution becomes visible. Base produced 24 em dashes over 14,049 words, SFT 8 across 9,776, DPO 28 across 13,160, and Instruct 34 across 13,400.

Greedy decoding reveals the default style of the model. Higher-temperature sampling examines how much of that style persists in the larger distribution as sampling explores less-likely alternatives.

Tulu preference data: a weak chosen-side skew

If preference tuning is a potential amplifier, the preference data itself is the obvious next place to check.

In Tulu preference data, chosen responses had 0.306 em dashes per 1,000 words, while rejected responses had 0.231. The audit included 30,158 parsed pairs. A stricter pairwise-delta analysis used 23,499 pairs where both sides met the comparison threshold. These are two different summaries: the pooled rates count all retained words, while the pairwise test asks whether chosen responses tend to run higher than their matched rejected responses.

Most preference pairs do not differ in this feature: chosen responses had more em dashes in 4.4% of usable pairs, rejected responses had more in 3.3%, and 92.3% were equal. The pairwise mean difference was +0.050 em dashes per 1,000 words, with the interval excluding zero. Although small, this result is consistent with the idea that optimization makes weak style-correlated signals more visible in output behavior.

Parser exclusions are a limitation of this analysis. The results should be interpreted as pointing to a potential direction, rather than providing a definitive conclusion. The comparison is useful because it reveals a small chosen-side skew that can be directly audited, but it does not constitute a complete theory of preference data.

K2-V2: elevated rates without final-stage DPO

K2-V2 adds extra complexity to the analysis. It comes from LLM360, a project focused on making large-model development more open than usual, including releases around data, checkpoints, and training process details [10] [11]. It matters here because it gives us a different training path from OLMo, rather than just another final-chat model to sample.

Its final instruct path is SFT-only. There is no final-stage DPO phase that might be amplifying em dashes. But the SFT data, TxT360, was mostly regenerated using GPT-OSS-120B. If those demonstrations carry a concentrated model-written style, supervised fine-tuning can transmit it directly.

K2-V2 greedy produced 6 em dashes spanning 4,587 words, or 1.308 per 1,000 words. Two fixed non-greedy settings produced more: the medium-temperature run at T=0.7 produced 40 across 23,027 words, and the high-temperature run at T=1.0 produced 60 across 22,782.

These results show that DPO is not the only factor producing increased em-dash usage. The K2-V2 findings are consistent with style being transmitted through supervised imitation when demonstrations are synthetic or primarily model-written, although the upstream source of the style remains unidentified.

The em dash constitutes a broader style shift

The em dash is a highly visible marker, while colons are less noticeable. However, in the OLMo-2 greedy runs, colon usage also increased significantly along the same checkpoint path.

This indicates that a more extensive change is occurring. If the effect were limited to em dashes, it would be a simple punctuation issue. Instead, the data suggest a shift in style, modifying how models structure explanations, introduce qualifications, manage transitions, and produce polished content.

The em dash is a useful marker because it is easily observed. This discernibility allows investigation into whether training pipelines can transmit a recognizable style of written content, rather than just a single punctuation habit.

Checking the easy explanations

Three main explanations are considered in this analysis.

The first explanation is that the effect is due to the pretraining process. In sampled open-pretraining-style corpora, em dashes are present but not frequent: Dolma Reddit has 0.115 per 1,000 words, DCLM has 0.589, and Dolma News has 0.964. These baselines make it difficult to attribute the later increases in checkpoint rates to simple exposure during pretraining. The question remains why post-training increases em dash usage in the default model output.

The second explanation is that the effect results from generic educational or quality-filtered web content. However, the FineWeb-Edu tiers do not strongly support this hypothesis. The score tiers are mostly flat, without a clear monotonic increase in em-dash-heavy text: score 2 is 0.942 per 1,000 words, score 3 is 1.074, score 4 is 0.914, and score 5 is 0.812.

The third explanation is that increased em dash usage is a local effect in multi-turn conversations, where the assistant may repeat its own style or be influenced by previous turns. To test this, a stripped-history condition was used, removing prior assistant U+2014 characters before generating the next turn.

The stripped runs did not dependably suppress the marker. OLMo-2 general rose from 0.410 to 0.655 per 1,000 words after stripping. OLMo-3 general fell from 5.015 to 4.248. OLMo-3 factual rose from 1.229 to 1.971.

These controls do not eliminate all alternative explanations, but they weaken the three main ones considered. Broad pretraining exposure, generic quality filtering, and visible em dashes in prior assistant turns do not, by themselves, account for the increases observed at later checkpoints.

What the tracer suggests

Two paths for style movement

A likely explanation is that style moves through two separate mechanisms.

The first path is amplification. In these runs, preference data carries weak style-correlated differences, and DPO is consistent with making those differences more visible in the model’s default output. In OLMo-2, the chosen-side Tulu rate was only 0.306 per 1,000 words, but the DPO checkpoint produced 1.528 per 1,000 under greedy decoding. That does not mean DPO rewards punctuation directly. It means DPO can make features that travel with preferred responses more visible in a trained policy.

The second path is reproduction. Synthetic demonstrations can carry model-written style directly, and SFT can teach the student to imitate that style without a final-stage preference step. K2-V2 is the useful contrast: its GPT-OSS-120B-generated SFT source data sits around 1.0 to 1.5 em dashes per 1,000 words in chat and instruction-heavy subsets, and K2-V2 greedy output lands at 1.308.

In brief, SFT tends to reproduce the style of the source data, while DPO amplifies a weak skew in preference data. Although this is a simplified explanation, it fits the observed cases more closely than either a DPO-only or a generic model-style explanation.

The larger pattern goes beyond punctuation. It involves a style of polished explanation, including the use of caveats, transitions, list structures, and the manner in which a model presents uncertainty.

Discussion

Passage of bias

The main issue is not limited to punctuation, but concerns the transfer of bias from one system to another.

Data generated from GenAI models, model-written demonstrations, distillation, preference optimization, and model-as-judge pipelines all create paths where one system’s bias can become another system’s training signal. Sometimes that is intentional: we want a smaller model to learn from a better one. In other cases, it is not intentional, as when a teacher model, judge, or synthetic data source brings its own biases, blind spots, and shortcomings along with the useful signal.

The above risk is already visible in other synthetic feedback loops. Recursive training on model-generated data can change the learned distribution itself [13]. When those loops touch social or demographic structure, they can also carry mistakes and biases into the surrounding data ecosystem [14].

A tougher case is the model-to-model interactions. The modern training stack is full of them: one model generates demonstrations, another judges preferences, another learns from those signals, and later systems may be trained on the resulting text. Cross-model feedback can homogenize actions across systems [15], and distillation can transmit failure modes from an imperfect teacher to a student [16].

In this analysis, punctuation is used as a visible proxy for broader biases that can propagate downstream across models, whether intentionally or unintentionally. If a visible style marker can be transmitted in this way, it is important to consider whether less visible biases may also be moving through the same process.

Punctuation and uncertainty

Goodfire’s “Features as Rewards” paper has an interesting side note that points to future work [17].

In Figure 5, their reward-probe attention maps show close attention on entity and relation tokens in the top examples. But for a Failed Fix, which should ideally receive a low reward, they report greater attention to punctuation tokens, with similar results for retraction.

This observation does not explain em dash usage, nor does it show that punctuation causes hallucination or uncertainty. However, it suggests that punctuation and style ought to be monitored more closely. In this analysis, em dashes serve as a visible tracer of model-written style as it moves through training. In the referenced figure, punctuation appears in a different context: a reward pipeline predicting whether a hallucinated claim was fixed or retracted.

Based on this result and Goodfire’s observation, the hypothesis is that punctuation-rich, highly structured text may be part of a wider pattern in which models produce polished responses even when uncertain. A model that does not know the answer may still generate text that looks organized, articulate, and confident, thereby satisfying a judge, a preference model, or a human evaluator. In such cases, a fluent answer may be preferred over an explicit admission of uncertainty, as when a student writes around uncertainty to earn partial credit.

For scientific and reasoning use, that’s unacceptable. It is much better if a model simply says, “I don’t know.” A useful system should ask for a missing measurement, retract a weak claim, propose a falsifying test, or say that it does not know. It should not simply become more articulate and confident. This connects toward broader work on hallucination and abstention: the goal is not better-sounding answers, but systems that handle uncertainty more honestly [18] [19].

How to interpret these results

This analysis is not intended to serve as a detection tool. The available evidence does not support the use of em dashes as indicators of synthetic text, as human writers employ them for a range of purposes. Previous research on detection has cautioned against the use of surface markers as reliable classifiers on a per-document basis [20].

The frequency with which human writers use em dashes varies substantially across genres. For example, in a sample from The Atlantic, 13 articles contained 92 em dashes in 16,823 words, corresponding to a rate of 5.469 per 1,000 words. This rate exceeds that observed in OLMo-2 DPO and is comparable to OLMo-3 Instruct. The selection of an appropriate comparison class is therefore essential.

This analysis does not provide direct evidence that DPO specifically rewards punctuation. Rather, the findings are consistent with the amplification of a broader signal correlated with writing style, rather than a targeted objective concerning punctuation.

Furthermore, this analysis does not establish that closed models utilize the same underlying mechanisms. Closed models may incorporate distinct combinations of post-training data, preference modeling, evaluation protocols, prompt selection, and distillation procedures.

The principal contribution of this work is the measurement of the training pipeline. The availability of open checkpoints enables direct identification of the stages at which a visible signal emerges, rather than relying exclusively on analysis of model outputs.

Reproducibility

Acknowledgements

This work depends on teams that make model development fully open. Thank you to the people behind OLMo and K2 for releasing accessible training pipelines, checkpoints, model weights, and supporting artifacts. These releases are a massive contribution to the open research community, and they make analyses like this possible.

References

• [1] E. M. Freeburg. “The Last Fingerprint: How Markdown Training Shapes LLM Prose.” arXiv:2603.27006, 2026. arXiv
• [2] Dmitry Kobak, Rita González-Márquez, Emőke-Ágnes Horvát, and Jan Lause. “Delving into LLM-assisted writing in biomedical publications through excess vocabulary.” arXiv:2406.07016, 2024. arXiv
• [3] Alex Reinhart, David West Brown, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, and Gordon Weinberg. “Do LLMs write like humans? Variation in grammatical and rhetorical styles.” arXiv:2410.16107, 2024. arXiv
• [4] Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. “From Lists to Emojis: How Format Bias Affects Model Alignment.” arXiv:2409.11704, 2024. arXiv
• [5] Dirk Groeneveld et al. “OLMo: Accelerating the Science of Language Models.” arXiv:2402.00838, 2024. arXiv
• [6] Team OLMo et al. “2 OLMo 2 Furious.” arXiv:2501.00656, 2025. arXiv
• [7] Team Olmo et al. “Olmo 3.” arXiv:2512.13961, 2025. arXiv
• [8] Nathan Lambert et al. “Tulu 3: Pushing Frontiers in Open Language Model Post-Training.” arXiv:2411.15124, 2024. arXiv
• [9] Ganqu Cui et al. “UltraFeedback: Boosting Language Models with Scaled AI Feedback.” arXiv:2310.01377, 2023. arXiv
• [10] K2 Team et al. “K2-V2: A 360-Open, Reasoning-Enhanced LLM.” arXiv:2512.06201, 2025. arXiv
• [11] Zhengzhong Liu et al. “LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch.” arXiv:2501.07124, 2025. arXiv
• [12] Guilherme Penedo et al. “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.” arXiv:2406.17557, 2024. arXiv
• [13] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. “AI models collapse when trained on recursively generated data.” Nature 631, 755-759, 2024. DOI
• [14] Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. “Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias.” Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024. DOI
• [15] Hung Anh Vu, Galen Reeves, and Emily Wenger. “What happens when generative AI models train recursively on each others’ outputs?” arXiv:2505.21677, 2025. arXiv
• [16] Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Rame, and Mathieu Blondel. “On Teacher Hacking in Language Model Distillation.” arXiv:2502.02671, 2025. arXiv
• [17] Aaditya Vikram Prasad, Connor Watts, Jack Merullo, Dhruvil Gala, Owen Lewis, Thomas McGrath, and Ekdeep Singh Lubana. “Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability.” arXiv:2602.10067, 2026. arXiv
• [18] Adam Tauman Kalai et al. “Why Language Models Hallucinate.” arXiv:2509.04664, 2025. arXiv
• [19] Christian Tomani et al. “Uncertainty-based Abstention in LLMs improves Safety and Reduces Hallucinations.” arXiv:2404.10960, 2024. arXiv
• [20] Vinu Sankar Sadasivan et al. “Can AI-Generated Text be Reliably Detected?” arXiv:2303.11156, 2023. arXiv

Citation

@online{bressert2026emdashes,
  author = {Eli Bressert},
  title = {Em Dashes: The Passage of Bias},
  subtitle = {A tracer study of how AI writing style moves through model training},
  date = {2026-06-26},
  year = {2026},
  url = {https://2x11.xyz/blog/passage-of-bias/},
}