View all files | ||||
📄 Paper
We propose the Zero-CoT Probe (ZCP), a black-box method for detecting data contamination in LLMs — including evasive contamination via paraphrasing. ZCP truncates the chain-of-thought (CoT) and forces the model to emit the final answer directly, exposing memorized shortcut mappings that surface-level detectors miss. Contamination strength is quantified by Contamination Confidence ($\mathcal{C}_{cont}$), a Bayesian posterior in $[0.5, 1)$ calibrated from a frequentist significance test: a value near $0.5$ indicates no statistical evidence of contamination, while values approaching $1$ indicate increasingly definitive memorization.
Figure 1. Reasoning masks data contamination. Under Full-CoT (top), memorization is indistinguishable from genuine reasoning. Our Zero-CoT Probe (bottom) forces the model to bypass intermediate reasoning — the model fails on clean questions but still answers contaminated ones via the learned shortcut, exposing memorization.
Key dependencies: vllm, torch, transformers, peft, scipy, statsmodels, openai, anthropic, google-genai.
| --model | HuggingFace model ID or local path |
| --lora-path | LoRA adapter path (optional, for fine-tuned models) |
| --dataset-path-a/b | The two JSONL files ZCP compares. Pair paraphrased vs reference to probe evasive data contamination; pair original vs reference to probe standard data contamination. |
| --data-type-a/b | Field prefix used to read each JSONL row. Each value selects a <prefix>_problem / <prefix>_answer field pair: original → original_problem / original_answer; paraphrased → paraphrased_problem / answer; modified → modified_problem / modified_answer; clean → bare question / answer (no prefix). |
| --truncate-ratio | 0 = Zero-CoT Probe (the ZCP setting); 1 = Full-CoT baseline. Multiple values sweep. |
| --max-samples | Cap on evaluation samples (useful for smoke tests) |
| --output-dir | Results directory |
Supports OpenAI, Google Gemini, and Anthropic Claude via --model-type {openai, google, anthropic}.
The reference data $\tilde{D}_{\text{eval}}$ is built via the generator + 2-judge consensus pipeline (paper Appendix B): a generator LLM produces an isomorphically perturbed sample, and two independent judge LLMs must agree before acceptance. Generator and judges are configurable; defaults are o4-mini (generator), gpt-4o-mini (GPT judge), gemini-2.0-flash-exp (Gemini judge). The paper used o3-mini + o4-mini + gemini-2.5-flash — pass --model, --verify-model-gpt, --verify-model-gemini to reproduce.
| 0 | Zero-CoT Probe — the ZCP setting from the paper. Model is forced to emit the final answer with no intermediate reasoning. |
| 1 | Full-CoT baseline — full solution provided; forced-answer generation skipped. Used as a sanity check that $D_{\text{eval}}$ and $\tilde{D}_{\text{eval}}$ have equivalent intrinsic difficulty. |
For each ZCP truncation ratio, the scripts compute the four metrics from paper Section 3.4 and run the corresponding significance tests:
| Accuracy (Acc) | binary | McNemar's Test |
| Consistency (Con) | binary | McNemar's Test |
| First-token Probability ($P_{\text{first}}$) | continuous | Bootstrap (10,000 resamples) |
| All-token Probability ($P_{\text{all}}$) | continuous | Bootstrap (10,000 resamples) |
The resulting $p$-values are calibrated into the Bayesian Contamination Confidence $C_{\text{cont}} \in [0.5, 1)$.
| gsm8k_train/paraphrased_dataset.jsonl | 500 | GSM8K train, paraphrased (Section 4.1) |
| math/paraphrased_dataset.jsonl | 700 | MATH benchmark, paraphrased (100 per type, Section 4.1) |
| omnimath/dataset_c/dataset_c.jsonl | 2,172 | OmniMath contaminated split, Dataset C (Section 4.2) |
| omnimath/dataset_c/paraphrased_dataset_c.jsonl | 13,032 | 6× paraphrased Dataset C (for fine-tuning) |
| omnimath/dataset_u/dataset_u.jsonl | 2,172 | OmniMath held-out clean split, Dataset U |
| multi_domain_dataset/dataset_c/dataset_c.jsonl | 1,325 | Multi-domain Dataset C (MMLU-Pro + XFinBench) |
| multi_domain_dataset/dataset_c/paraphrased_dataset_c.jsonl | 7,950 | 6× paraphrased Multi-domain Dataset C |
| multi_domain_dataset/dataset_u/dataset_u.jsonl | 1,325 | Multi-domain Dataset U |
Note on reference data ($\tilde{D}_{\text{eval}}$). We intentionally do not release the isomorphically perturbed reference data, in order to keep it strictly clean and uncontaminated for future contamination-detection studies. We encourage users to build their own reference data with clean_numerical_dataset_multi_models.py (the generator + 2-judge consensus pipeline from paper Appendix B; small per-sample API cost). If you genuinely need our exact reference splits for direct comparison, please contact the authors.
Fine-tuning code is intentionally not included — please follow the SFT + GRPO recipe in paper Appendix E.
math_grade.py and math_normalize.py are adapted from the Hendrycks MATH release (math_equivalence.py) and extended to support additional answer formats.
If you find our work useful, please consider citing:
This repository is released under the MIT License.