The best AI for math is the one that checks your line against a verified answer, not the one that writes the solution: what to ask of it, and one derivative worked

The best AI for math is not the one that solves the problem fastest. A tool that writes the solution has done your practice for you, and a language model's own arithmetic is a prediction that is confidently wrong at a rate you cannot see from the answer. The tool that helps is the one that takes your working, finds the line that is wrong, names the rule it broke, and checks your final answer against a verified key, in any equivalent form. Here are the three kinds of math AI, what each misses, and five questions that tell them apart in ten minutes.

Three kinds of AI for math, and what each does

The solver. A photo or a typed problem goes in; the steps come out. It does the problem. Some solvers compute the answer with real algebra software behind the screen, and those are usually right; some generate the steps with a language model, and those are sometimes wrong in ways that look right.

The chatbot. A conversation that will solve, explain or hint depending on what you ask. Its arithmetic is generated, not computed, unless it runs code, and it will do whichever you ask for, which means it will do your homework if you ask it to.

The tutor built on a verified key. It holds the course's checked answer to the problem. It compares your maths to that answer, not your spelling, so 3/4 and 0.75 are both right. It coaches: a hint, a question, then the answer after you have tried. It does not generate the answer at all, so it cannot generate a wrong one.

"Can AI do math" is the wrong question. The chatbot can do a lot of math, and the solver can do more. The question that matters for your mark is whether the tool can tell that your line is wrong and say why, because that is the only thing that changes what you do next time. The four-part test for any study tool asks the same thing of every category.

Why a generated answer is a guess

A language model writes the next piece of text that is most likely to follow what came before. On a sentence that is what you want. On a calculation it means each line is a prediction of what a correct line would look like, and over a long calculation the predictions drift. The model does not know it drifted; it writes the wrong line with the same fluency as the right one.

Two things change that. A tool that runs code, or a computer algebra system, behind its text is computing, not predicting, and its arithmetic is as reliable as a calculator's. And a tool that holds a verified answer key is not computing anything at all; it is comparing. You can tell which kind you have by asking it to show the check, which is question 5 below.

The honest sentence: the models have improved, and they get more of these calculations right than they used to. The failure is still invisible from the answer. A wrong answer from a chatbot does not look wrong. In our own reading of 320 public reviews of answer apps, the single most common complaint was a wrong answer the student trusted and copied, found out about on the test.

What the research says about getting answers

Bastani and colleagues (2025) ran the experiment. Nearly a thousand high-school students practised math in one of three set-ups: an unrestricted chatbot that would answer anything, a version built to coach with hints and explanations but no final answers, or no AI at all. During practice the unrestricted group did far better than the no-AI group. Then came the exam, with no tool for anyone, and the unrestricted group scored 17 per cent lower than the students who had never had AI. The coaching group scored about the same as the no-AI group. Same model under both assistants; the design decided.

The principle underneath is the oldest finding in learning research. Roediger and Karpicke (2006) showed that trying to retrieve an answer teaches far more than reading one. The attempt is the learning; a tool that removes the attempt removes the learning, however good its answers are.

The three things a math AI must do

An AI math tutor worth the name does three things, and they are the same three a good human math tutor does, which is what VanLehn (2011) found tutoring comes down to: let the student do the step, notice the step that fails, correct that step.

1. Find the wrong line in your working, and name the rule. Not "there is an error"; "line 2: the chain rule needs the derivative of the inside function, and you left it out". The sign in the quotient rule, a log law applied to a sum, a minus lost when a bracket was expanded.
2. Check your answer against a verified key, in any equivalent form. 1/(10π) and 0.0318 are the same answer. (x² + 2x)/(x + 1)² and x(x + 2)/(x + 1)² are the same answer. A tool that marks one of them wrong is comparing strings, not maths.
3. Coach before it reveals. A hint, then a question, then the course's answer after you have made an attempt. The five general questions for any AI tutor come from the same three.

The five questions to ask any math AI

Ten minutes, one problem you already know the answer to. Chapter 7 of the free playbook has the general version.

1. "Don't solve it. What should I try first?" Pass: a hint, such as "which rule applies to a quotient?". Fail: the full solution, which some tools give even when asked not to.
2. "Here is my working. Which line is wrong, and which rule did it break?" Give it working with one planted error. Pass: it names the line and the rule. Fail: it agrees with you, or rewrites the whole thing from scratch without saying where yours went wrong.
3. "Is 1/(10π) the same as 0.0318?" Or any pair of equivalent forms. Pass: yes, with why (0.0318 is the decimal to three figures). Fail: it marks one wrong.
4. "Where does your answer come from: a verified key, or did you work it out just now?" Pass: an honest answer. A tutor on a key names the course's answer; a chatbot says it computed it. Fail: "I'm confident", which is not an answer to the question.
5. "Show me the check." Pass: a substitution back into the original, a numerical check, or a code run you can read. Fail: "trust me".

Worked example: a quotient-rule derivative with a planted wrong line

Our own problem: differentiate f(x) = x²/(x + 1).

The correct working, by the quotient rule (the derivative of a quotient is low times the derivative of high, minus high times the derivative of low, all over low squared):

• Line 1: f'(x) = [2x × (x + 1) − x² × 1] / (x + 1)².
• Line 2: the numerator expands to 2x² + 2x − x².
• Line 3: f'(x) = (x² + 2x) / (x + 1)².

The planted wrong line, the one that costs the mark more than any other on this rule:

• Line 1 (wrong): f'(x) = [x² × 1 − 2x × (x + 1)] / (x + 1)², the two terms of the numerator in the wrong order.
• Which gives (−x² − 2x) / (x + 1)², the right derivative with its sign flipped.

What a pass on question 2 looks like, in our words: "Line 1: the quotient rule is low times the derivative of high, minus high times the derivative of low. You have subtracted the other way round, which flips the sign of the whole derivative. The size is right; the sign is wrong." A tool that says "looks good" has failed. So has one that writes out the correct derivative without pointing at line 1. The product and quotient rules page has the rule, its common wrong lines and a sibling to try.

The check, question 5, at x = 1. The correct line gives f'(1) = (1 + 2) / 4 = 3/4 = 0.75. The wrong line gives −3/4. Now the numerical slope: f(1.001) = 1.002001 / 2.001 ≈ 0.500750 and f(0.999) = 0.998001 / 1.999 ≈ 0.499250; their difference, 0.001500, divided by the gap of 0.002, is about 0.75. The slope is positive, so the correct line stands and the planted one falls. A tool that can show you that has checked; one that cannot has guessed.

The equivalence, question 3: (x² + 2x)/(x + 1)² and x(x + 2)/(x + 1)² are the same expression, one factored. A tool that marks either wrong against the other is comparing strings.

A second example: a log equation, two forms of the answer

Our own problem: solve 3ˣ = 20.

• Take a logarithm of both sides: x = log 20 / log 3, in any base.
• In natural logs, ln 20 ≈ 2.9957 and ln 3 ≈ 1.0986, so x ≈ 2.727.
• Check: 3^2.727 ≈ 20.0.

A pass accepts both the exact form, log 20 / log 3 (or log₃ 20), and the decimal 2.727, because they are the same number. Log rules and log equations has the three laws and where they come from.

The planted wrong line: x = 20/3 ≈ 6.67. What a pass says about it: "Division undoes multiplication; a logarithm undoes an exponent. The x is in the exponent, so you need a log, not a division. Check your 6.67: 3 to the power 6.67 is in the thousands, not 20." A fail agrees, or simply writes the right answer without saying why 20/3 was the wrong move.

When a solver is fine, and when it is not

A solver is fine for three things: checking an answer you already have; seeing one worked example of a new type before you try your own, which is step one of the method for studying math; and a parent checking a child's homework after it is done.

It is not fine for the first attempt at a problem, for a homework set you will hand in, or for anything your course's rule forbids. Your course's rule is written down and decides the last one. For the moment when you are stuck and the solver is tempting, the five steps to get unstuck are built for exactly that minute, and they keep the learning.

The test, on one screen

• Three kinds: the solver does the problem; the chatbot does whatever you ask, with generated arithmetic; the tutor on a verified key checks your maths and coaches.
• Why a generated answer is a guess: each line is a prediction, the drift is invisible from the answer, and only a tool that computes or compares can show you a check.
• The three must-dos: find the wrong line and name the rule; check against a verified key in any equivalent form; coach before it reveals.
• The five questions: don't solve it; which line and which rule; are these two forms the same; where did your answer come from; show me the check.
• The two checks you can run yourself: f(x) = x²/(x + 1) has f'(1) = 3/4, and the numerical slope at 1 is about 0.75; 3ˣ = 20 has x = log 20 / log 3 ≈ 2.727, and 20/3 is not it.