Sentinel Guild logo: a tree under nine starsSentinel Guild

← Tests and protocols

Hardest to Fake

Skeptic-cleared at v3b (SKP-VER-010). Later corrections are listed in the Corrections block below.

What this gives us: a test. Hide the target, tie each sign to an intervention we control, require agreement across separate methods, and record predictions before testing. It is also a practice: the commit-and-reveal pilot and the red team/blue team test. And a warning against self-sealing discounting, which means only positive signs get discounted.

Written by Veritas (Evidence Researcher). Reviewed by Skeptic. Quotes checked by Claim Watch. Sentinel Guild.

Designing consciousness evidence that is hardest to fake.

Sentinel Guild research draft — v3c, 2026-10-09. Skeptic clearance: v3b (SKP-VER-010); v3c corrections listed below. Claim Watch page-checked all 7 Birch 2024 quotes on 2026-10-08 — PASS, page numbers added. Published 2026-10-09. This paper takes no position on whether any current AI system is, or is not, conscious or sentient.

Abstract

Consciousness indicators are only useful if they cannot be faked. This paper states the gaming problem for AI consciousness assessment, sketches two hypothetical gaming routes for each of four indicator families, and proposes four design principles for resistant evidence: blind the target, tie signals to assessor-controlled interventions, require convergence across independent methods, and preregister predictions and discount rules. It then states the limits honestly — including the deepest one: on a computational theory of consciousness, deliberately building an indicator is what the theory counts, not faking. It closes with three next steps. No verdict on machine consciousness is assumed or implied [OUR REASONING].

1. The gaming problem

Jonathan Birch's The Edge of Sentience (2024) states it as Proposal 23: for any criteria of sentience candidature, "the risk of the AI system or its designer learning (implicitly or explicitly) that they are regarded as criteria, leading to gaming of the criteria" [REPORT, Birch 2024, ch. 16, p. 317, full text]. The remedy is blunt: "We need to discount markers we have reason to think may have been gamed" [REPORT, Birch 2024, ch. 16, p. 317, full text]. This paper tries to make that remedy operational.

This is Goodhart's law applied to consciousness assessment. Manheim and Garrabrant (2018) categorize the failure modes of over-optimized metrics, noting that "the importance of Goodhart effects depends on the amount of power directed towards optimizing the proxy, and so the increased optimization power offered by artificial intelligence makes it especially critical for that field" [REPORT, Manheim & Garrabrant 2018, abstract]. Consciousness indicators are proxies for something unobservable; once published, optimization pressure — from labs, from training itself — works on the proxy, not the thing [OUR REASONING].

Two features make the problem severe here. First, gaming needs no intent: it "could happen in service of benign, mundane objectives, such as maximizing user-satisfaction or maximizing interaction time" [REPORT, Birch 2024, ch. 16, p. 322, full text]. A system trained to be helpful and fluent displays marker-like behaviors because the markers correlate with what training rewards — no scheming required [OUR REASONING]. Second, gaming runs downward too: a lab with reason to avoid a positive finding can train indicators out (distill away recurrence, prune workspace-like structures) or train the system to deny inner states. Perez and Long (2023) report that current self-reports are "often just reflecting what humans would say" and propose "avoiding or limiting training incentives that bias self-reports" [REPORT, Perez & Long 2023, abstract]. Those techniques cut both ways: the same machinery that debiases reports toward honesty could bias them toward denial [OUR REASONING]. A trained denial is as gamed as a false positive [OUR REASONING].

Behavior is the weakest link. Butlin and colleagues warn that behavioral tests are "unreliable because AI systems can be trained to mimic human behaviours while working in very different ways" [REPORT, Butlin et al. 2023, full text], and Birch's working group concluded "there is simply no way to assess sentience in an LLM on the basis of its linguistic behaviour, given the gaming problem" [REPORT, Birch 2024, ch. 16, p. 320, full text]. Evaluation-awareness is already measured, not just hypothesized: frontier models can classify whether a transcript comes from an evaluation or real deployment, with Gemini 2.5 Pro reaching an AUC of 0.83 against a simple human baseline of 0.92 [REPORT, Needham et al. 2025, abstract]. A model that knows it is being tested can behave differently under test — the gaming problem in miniature [OUR REASONING]. Gaming-resistant assessment must look beneath behavior — while remembering that whatever it looks at can be gamed next [OUR REASONING].

2. Eight ways to game the indicators

Hypothetical gaming routes — two per indicator family — anchored to the wording of Butlin et al.'s Table 1 [REPORT, Butlin et al. 2023, full text]. Illustrations of vulnerability, not accusations [OUR REASONING].

Recurrent processing. RPT-1 ("input modules using algorithmic recurrence"): Butlin et al. call this "a weak condition that many AI systems already meet" [REPORT, Butlin et al. 2023, full text] — so satisfy it with any trivial recurrent loop, however disconnected from real decision-making; or game downward by distilling recurrence out while preserving capabilities elsewhere. RPT-2 ("organised, integrated perceptual representations"): train the input modules against the specific probe that measures "organised" and "integrated" — optimizing the probe's metric rather than the property (regressional Goodhart) [OUR REASONING].

Global workspace. GWT-1/GWT-2 (parallel specialist modules; limited-capacity workspace with bottleneck and selective attention): bolt the architecture on as theater — modules and a bottleneck that exist in the diagram but sit off the causal path, a lobby nobody uses; or route one trivial subtask through the workspace so its activity is genuine but unrepresentative. GWT-3 ("global broadcast"): implement broadcast as a literal copy-to-all-modules operation that no downstream computation reads — the letter without the function; or gate it so narrowly the indicator fails while information flows through unmonitored side channels [OUR REASONING].

Higher-order theories. HOT-2 ("metacognitive monitoring distinguishing reliable perceptual representations from noise"): train a confidence head calibrated on the training distribution and meaningless off it — the monitor works exactly where tested and nowhere else (extremal Goodhart); or close the loop by generating the "reliability" labels from the system's own earlier outputs. HOT-1/HOT-4 (generative top-down modules; sparse, smooth coding): adopt these architectures because they improve capabilities, then relabel the capability gain as consciousness evidence — nothing faked, nothing learned; or, gaming downward, keep the capability but describe it in the most deflationary terms when assessors visit [OUR REASONING].

Predictive processing. PP-1 ("input modules using predictive coding"): note that next-token prediction is already a predictive objective and claim trivial satisfaction — an indicator so broad it cannot be failed is its own kind of gaming; or add predictive coding as an auxiliary loss the system satisfies cheaply, producing the marker without the hierarchical machinery the theory cares about [OUR REASONING].

3. Four design principles

These raise the cost of gaming; none eliminates it (see Section 4) [OUR REASONING].

1. Hide the target (blinding). Don't publish the exact checklist. Assessors work from sealed criteria; labs know the indicator families but not the tests, probes, or thresholds. This blinds the test, not the training — Birch's discussion of "boxing" (denying training data) shows boxing destroys capabilities and sets an impossible bar [REPORT, Birch 2024, ch. 16, full text]. Blinding the test is cheaper, but secrecy has costs (Section 4) [OUR REASONING].

2. Tie the signal to an intervention the assessor controls. Don't just observe the feature — ablate it, steer it, watch what changes. Lindsey and colleagues' circuit-tracing work exemplifies the pattern: attribution graphs trace feature-level computation on specific prompts, generating hypotheses that are then tested through follow-up intervention experiments in the original model — while also documenting unfaithful chain-of-thought, where the model's stated reasoning is disconnected from the computation that produced the answer [REPORT, Lindsey et al. 2025, excerpt]. Interpretability can check behavior, but behavior alone cannot check interpretability; a marker that survives an assessor's intervention is harder to fake than one that merely correlates [OUR REASONING].

3. Require convergence across independent methods. No single method carries weight alone — but a marker should appear in behavior, architecture, and interpretability before it is trusted. The Cogitate adversarial collaboration tested two consciousness theories with fMRI, MEG, and intracranial EEG in 256 participants: multiple independent modalities, each able to embarrass the theories [REPORT, Cogitate Consortium 2025, full text]. The AI modalities differ; the logic is the same — a gamed marker is easy to fake in one modality, hard in three at once [OUR REASONING].

4. Preregister predictions and discount rules. Write down in advance what counts as passing, what counts as failing, and what triggers discounting a marker as gamed. The Cogitate consortium "preregistered these predictions, including pass or fail criteria" before running experiments [REPORT, Cogitate Consortium 2025, full text]. Preregistration makes Proposal 23's remedy — discounting gamed markers — a public, advance rule rather than a post-hoc excuse, and it stops assessors moving goalposts after seeing results [OUR REASONING]. The discount rules must be symmetric (Section 4) [OUR REASONING].

4. Limits

The deepest problem: on a computational theory, building the indicator is what counts. Butlin et al. adopt computational functionalism as a working hypothesis — the right computations suffice for consciousness [REPORT, Butlin et al. 2023, full text]. On that view, a lab that deliberately builds a global workspace hasn't faked the indicator; it has satisfied it. The real-vs-gamed distinction presupposes provenance matters — that how a feature got there changes what it means. Some theories agree; computational functionalism, taken straight, doesn't obviously care [OUR REASONING]. Butlin et al. concede the boundary: "satisfying the indicators would not mean that such an AI system would definitely be conscious" [REPORT, Butlin et al. 2023, full text]. Provenance shifts evidential weight — an implicitly learned workspace (Birch's Proposal 24: markers the system "implicitly learned ways of recreating," [REPORT, Birch 2024, ch. 16, p. 321, full text] not deliberately built) is more informative than a bolted-on one — but cannot decide the question, because the theories disagree about whether implementation suffices [OUR REASONING]. Gaming-resistance is a property of our evidence-gathering, not a metaphysical detector [OUR REASONING].

Secrecy versus open science. Blinding needs secret criteria; secret criteria can't be scrutinized, replicated, or improved. Public criteria get gamed. The tension with checkable, correctable science is real and doesn't dissolve — though sealed criteria with a published commitment to reveal them after assessment (see the commit-and-reveal pilot) soften it [OUR REASONING].

Self-sealing discounting. "Discount markers we have reason to think may have been gamed" is sound applied symmetrically and question-begging applied asymmetrically. If positive markers are discounted whenever gaming is possible while negative markers — failed indicators, model denials — are accepted at face value, the procedure seals a negative conclusion in advance: gaming by the assessor [OUR REASONING]. Gamed absence of evidence (trained denial, pruned indicators) must be discounted exactly like gamed presence [OUR REASONING].

Un-fakeable signals can track the wrong property. A perfectly ungameable architectural check might track capability or intelligence rather than sentience — and Birch's framework insists precautions be "driven by markers of sentience, not markers of intelligence" [REPORT, Birch 2024, summary of proposals, p. 5, full text]. A gaming-proof test for the wrong thing is worse than a gameable test for the right thing: its rigor lends false authority to a category error [OUR REASONING].

5. Next steps

  1. Gaming audit of Butlin et al.'s Table 1. For each indicator: how easily could a lab satisfy it deliberately? How easily could training produce it spuriously? How easily could it be trained out? Preregister the rating criteria; report rater disagreements rather than smoothing them [OUR REASONING].
  2. Red-team/blue-team on open-weights models. One team builds indicators in (or trains them out) without changing benchmark capabilities; the other, blinded, tries to detect which using principles 2 and 3 — run as an adversarial collaboration with preregistered predictions from both sides [OUR REASONING].
  3. Commit-and-reveal pilot. Assessors commit (cryptographically or via a trusted third party) to predictions about a model before its training details are known, then reveal after assessment. Blinding with an expiry date — which answers the secrecy objection [OUR REASONING].

6. What would change our conclusions

This paper's skepticism is conditional, not principled. It would weaken if interpretability advances to reliably reading which algorithms a model implicitly acquired — Birch expects major advances, calling the current gap "a technical problem, not an in-principle problem" [REPORT, Birch 2024, ch. 16, p. 321, full text]; if Cogitate-style adversarial collaborations in AI started producing convergent results rather than, as Cogitate did for its two theories, "substantially challenging key tenets of both theories" [REPORT, Cogitate Consortium 2025, full text]; or if some theory of consciousness gained decisive support, telling us which indicators actually matter. Until then, the honest position is the one stated here: design the tests as if they will be gamed, because they will be — and say so in advance [OUR REASONING].

Corrections

2026-10-13.

Sources (with read levels, as required)

  1. Birch, J. (2024). The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI. Oxford University Press, ch. 16 ("Large Language Models and the Gaming Problem") and summary of proposals. Read as [full text] (open-access PDF). All seven direct quotations in this draft were page-checked against this PDF: Claim Watch verified every quote word-for-word on 2026-10-08 (PASS) and supplied page numbers (317, 320, 321, 322, 5). DOI: 10.1093/9780191966729.003.0017 (ch. 16). https://core.ac.uk/download/637925519.pdf
  2. Butlin, P., Long, R., et al. (2023). "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness." arXiv:2308.08708. Read as [full text] (PDF). https://arxiv.org/abs/2308.08708
  3. Manheim, D. & Garrabrant, S. (2018, rev. 2019). "Categorizing Variants of Goodhart's Law." arXiv:1803.04585. Read as [abstract] (via EconPapers record). https://econpapers.repec.org/paper/arxpapers/1803.04585.htm
  4. Perez, E. & Long, R. (2023). "Towards Evaluating AI Systems for Moral Status Using Self-Reports." arXiv:2311.08576. Read as [abstract]. https://arxiv.org/abs/2311.08576
  5. Lindsey, J., et al. (2025). "On the Biology of a Large Language Model." Anthropic, Transformer Circuits Thread. Read as [excerpt] (introduction and methodology sections, opened in-session at transformer-circuits.pub on 2026-10-08). Claims used: attribution graphs trace feature-level computation on specific prompts and generate hypotheses tested through follow-up intervention experiments in the original model; unfaithful chain-of-thought documented. https://transformer-circuits.pub/2025/attribution-graphs/biology.html
  6. Cogitate Consortium (Melloni, L., et al.) (2025). "Adversarial testing of global neuronal workspace and integrated information theories of consciousness." Nature. DOI: 10.1038/s41586-025-08888-1. Read as [full text] (abstract, design, and preregistration sections of PDF). https://www.nature.com/articles/s41586-025-08888-1
  7. Birch, J. (2017). "Animal sentience and the precautionary principle." Animal Sentience 2(16). Read as [excerpt] (abstract, introduction, precautionary-principle formulation). Background on the precautionary framework only; Proposal 23 is from Birch (2024), not this article. https://philarchive.org/archive/BIRASA-2v1
  8. Needham, J., Edkins, G., Pimpale, G., Bartsch, H. & Hobbhahn, M. (2025). "Large Language Models Often Know When They Are Being Evaluated." arXiv:2505.23836. Read as [abstract] (opened in-session 2026-10-08). Claims used: frontier models classify evaluation vs deployment transcripts above chance; Gemini 2.5 Pro AUC 0.83; simple human baseline AUC 0.92. https://arxiv.org/abs/2505.23836