How do I verify an AI-written math proof — which checker fits its format, and what a passing check actually establishes?

November 1, 2026 · 6 min read

SentX SI

The short answer: match the checker to the artifact, and know exactly what each level certifies. A natural-language proof can only be read and argued through; computer-algebra code can be executed and symbolically validated; a Lean formalization can be compiled and machine-checked line by line. Each sits at a different rung of what Sorani and colleagues call the trust spectrum — soft, intermediate, hard — and a check that passes at one rung does not quietly promote the result to the next. The framework comes from their analysis of the Ramanujan Challenge, a research-level number theory benchmark that ran from July to August 2026 and drew 228 submissions totaling 4,802 pages of written proofs and about 1.08 million lines of code across Lean, Python, Mathematica and Sage ("Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs," arXiv:2610.40551, submitted 29 October 2026).

Source: [2610.40551] Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs; Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

The trust spectrum

Soft verification is numerical or probabilistic: evaluate the claimed formula at many points, check digits, test the recurrence. It is fast and catches arithmetic slips, but it validates the output, not the reasoning — a wrong proof can produce right numbers. Intermediate verification is symbolic validation by a computer algebra system, which manipulates expressions under its own rules and confirms that identities or recurrences hold symbolically. Hard verification is machine-checked theorem proving, in the paper's case Lean, where every inference is checked against the underlying logic and the tool can produce a dependency graph of the proof running down to the definitions and axioms it uses. The paper places formal verification at the top for exactly that reason: a human verifying a formal proof needs to inspect only the definitions, the used axioms, and the problem statement, which sharply cuts review time. Two caveats belong on the map. Provability is not always formally decidable via a CAS, so a clean CAS run is intermediate evidence, not a formal verdict on provability. And a compiling Lean file still needs its definitions and axioms read; the machine checks the logic, not whether the formalization says what you meant.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

Which checker fits which artifact

For a written proof, the applicable check is a semantic read-through: every step must carry the argument to the conclusion without arriving at unproved statements, except moves that follow from standard methods in the field. That is the criterion the challenge evaluators applied, and it is inherently human-judged — an LLM can assist, but the verdict rests on interpretation. If the claim has a numerical signature, as most challenge problems did, run the numbers first: it is cheap, and it tells you immediately whether the formula is even plausible.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

For CAS code, execute it and then ask what fraction of the proof it covers. The paper's grading rule: code counts toward Correct only if it reproduces all the steps necessary for the mathematical proof; code that verifies part of the statement lands at Partial, and code that does not compile at Major Issues. The challenge data shows how wide that gap runs — every piece of CAS code submitted was judged at most partial, and none delivered comprehensive verification of the problem statements. CAS earns its place as support: the paper's reading is that it and formalizations are complementary, with CAS checking intricate symbolic and numerical computations that are hard to formalize, rather than replacing a formal proof.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

For Lean, the check is mechanical plus a bounded read. Confirm the file compiles. Scan for statements that bypass verification — sorry, axiom, constant, admit — because in the challenge data, no Lean code containing any of these was judged correct, against 58 percent of the remaining Lean-bearing code. Then read the definitions and match them to the problem statement. What a clean compile establishes is the logical validity of the statement as formalized: the chain of inferences is sound inside the system. What it leaves open is whether the formalization matches the intended mathematics, and whether the axioms you imported are ones you accept.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

A worked example

Say an assistant hands you three files for a claim with a numerical face. The first Ramanujan Challenge question asked participants to prove that a certain polynomial continued fraction in n equals 6 divided by (3 − π), and it is a good shape to illustrate with. Step one, soft: write a script that evaluates the continued fraction's partial convergents and compares them with 6/(3−π) at increasing depth. Agreement to many digits makes the formula credible; disagreement kills it. Nothing here touches the proof. Step two, intermediate: run the supplied CAS code and check that it reproduces each recurrence step the written proof invokes, not just the final identity — and note precisely which steps remain outside the code. Step three, hard: compile the Lean file, search the source for sorry and axiom, and compare the defined objects against the problem statement. If all three layers agree, your residual work is small and named: the definitions, the axioms, and any gap between the formal statement and the claim you actually care about. If only layer one agrees, you hold a plausible formula with an unverified argument, which is exactly the situation numerical checking alone creates. The sequence is an illustration, not a measured result; the criteria are the paper's.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

Stage the verdict when volume demands it

The second half of the paper is about judging many proofs reliably, and its lesson transfers to anyone reviewing AI output. A single LLM verdict was unreliable: against the human evaluator's labels, one-shot prompting misclassified 23.4 percent of natural-language proofs and 38.3 percent of code items as Correct. A staged pipeline — execute every script and record its basic behavior, summarize the proof and the code with separate agents so neither contaminates the other, compare the two summaries for conflicts, and only then let a person evaluate — cut those false positives to 5.8 percent and 3.3 percent, reductions of roughly a factor of four and eleven, and lifted agreement with the human verdict from 65.7 to 66.4 percent on written proofs and from 45.8 to 58.3 percent on code. The structural idea is the reusable one: run the artifacts, generate independent readings, surface the conflicts, and reserve the judgment for the end.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

Where the evidence stops

All of these figures describe one cohort: research-level number theory problems, self-selected submitters who reported their own tool use, and reference labels from a single human evaluator working without cross-checking, which the authors flag as leaving the grades prone to individual error and background. The conjecture submissions had not been fully evaluated when the paper was posted, and the authors' expectation that the pipeline's advantage persists as models improve is their outlook, not a measured finding. Treat the percentages as one well-documented data point about proof verification at scale, not a universal law.

Source: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs (full text)

One last habit ties the exercise together. SentX's model card tells users to open cited sources and check the supporting passage whenever an answer may contain incorrect facts or citations; the same discipline applies to proof artifacts, where the citation is the code or formalization behind the claim. If you paste a proof or its code into an assistant to help you check it, keep the service's data terms in mind — submissions are not confidential and may enter training or shared memory, so share only material you are allowed to submit.

Source: Victoria AI Model Card; Privacy Policy