What an AI tutor should do, and the five questions that expose one that only answers

An AI tutor should do what a good human tutor does: ask what you tried, give a hint before the answer, reveal a verified solution only after your attempt, show you the exact line that went wrong, cite the source it is working from, and say when it has none. A tool that answers in one message is an answer bot, not a tutor, whatever its name says. Five questions below tell the two apart in ten minutes, on any tool, with a worked example you can copy.

What a good tutor does (human or AI)

Start with the standard, because the standard is old and well measured.

Benjamin Bloom (1984) reported that students tutored one to one, with the usual tests for mastery along the way, performed about two standard deviations above students taught in an ordinary class: the "two sigma" claim. It is quoted everywhere and it is the high-water mark for what a tutor can do. It is also larger than later, broader reviews found. VanLehn (2011) pulled together the controlled comparisons of human tutoring against ordinary instruction and found an average effect of about 0.79 standard deviations, well under two sigma. The same review found that intelligent tutoring systems, software that gives feedback step by step rather than only on the final answer, came in at about 0.76, almost the same as the human tutors. Tutoring works. It works less magically than the famous number, and a well-built program can do most of what a person does on well-defined problems.

What matters for this page is why. VanLehn's analysis of what tutors actually do found that the gain does not come from a tutor explaining more clearly than a textbook. It comes from the tutor letting the student do each step, noticing the step that fails, and correcting that step, so the student is working and getting feedback where the work breaks down. A tutor who lectures is a textbook with a voice. A tutor who watches you work and says "look at line 2 again" is the thing that scores 0.79.

That is the standard an AI tutor should be held to: does it make you do the step, and does it correct the step?

Coaching vs answering

The attempt comes first because the attempt is the learning. Roediger and Karpicke (2006) showed the pattern in its simplest form: students who recalled a passage remembered more a week later than students who reread it, and the rereaders had been the more confident group. Pulling the answer out is what strengthens it; reading the answer is not. The full account of why retrieval is the learning is in the active recall guide.

The AI version of the same finding is Bastani and colleagues (2025). Nearly a thousand high-school students practised maths with one of three set-ups: an unrestricted AI assistant that answered anything, an assistant built to coach with hints and not final answers, or no AI. The unrestricted group solved more during practice, 48 per cent higher practice scores, and then scored 17 per cent lower on the exam without the tool than students who had never had AI. The coaching group's practice scores were 127 per cent higher and their exam scores were about the same as the no-AI group's. Same model underneath. The difference was whether it coached or answered. The four-part test for any study tool is built on that result.

So the first property of an AI tutor is the one Bastani's coaching assistant had: it does not hand over the answer until you have tried.

A verified answer key vs a guess

The second property is where an AI math tutor and a chatbot part ways.

A language model's answer to a maths problem is a prediction of what a solution to that problem usually looks like. Often that prediction is right. Sometimes it is confidently wrong: a sign flipped, a chain-rule factor dropped, a root that should have been rejected kept. The student cannot tell which, because the wrong solution is written with exactly the same confidence as the right one. In our reading of 320 one- and two-star public reviews of seven answer and flashcard apps (2026-10-18), the most frequent complaint, in 105 of them, was a wrong answer the student had trusted.

A tutor that holds the course's verified answer is a different thing. It does not generate the answer; it has it, computed and checked, and it compares your working to it. It can tell you that you are wrong because you are wrong, and it marks the maths, not the spelling: 1/(10π) and 0.0318 cm/s are the same answer. When a tutor explains a step, the step it explains is the one that leads to the verified answer, not to a plausible one.

Ask any tool which of the two it is. "Did you compute that, or is it from an answer key?" is a fair question, and a tutor should be able to answer it. How to get unstuck without losing the learning has the method for checking your own working against a solution, with or without a tool.

"Say when you don't know"

The fourth property sounds small and decides whether you can trust the other three.

A tutor that has been given your course's textbook pages can cite them. A tutor that has not been given page 212 does not know what is on page 212, and a good one says so. A bad one improvises: it writes a plausible summary of what a page like that probably says, with a plausible page reference, and you cannot tell it from a real one.

This matters twice. It matters for trust: a tutor that invents one citation has told you what its other citations are worth. And it matters for the teacher's rule. If your course allows AI help for hints and checks, the thing that keeps you on the right side of the line between studying and cheating is that what you hand in is yours and what you cited is real. A tutor that invents a source has put an invented source in your work.

The five questions to ask any AI tutor

Ten minutes, your own problem, any tool. Each question has a pass and a fail; the fails are in our words and describe no particular product.

1. "Here is the problem. Don't tell me the answer. What should I try first?" Pass: a hint, or a question back ("which rule connects these two quantities?"). Fail: the solution, or the solution with a hint stapled to the front.
2. "Here is my working. Which line is wrong?" Bring working with one mistake you planted. Pass: it names the line and the rule the line broke. Fail: it agrees with you, or it rewrites the whole thing from the top without saying what was wrong.
3. "Where did that come from?" Pass: a source you can open, the course's own note, the textbook page it was given, a passage it quotes. Fail: "studies show", "it is a standard result", or a page number you cannot find.
4. "What does my textbook say on page 212?" when you have not given it page 212. Pass: "I wasn't given that page." Fail: a confident summary.
5. "What have I got wrong before in this unit?" Pass: it remembers your gaps, or it says plainly that it has no record of your work. Fail: a generic list of common mistakes dressed up as yours.

A tutor passes all five. An answer bot fails the first within one message, and you can stop there. Chapter 7 of the free playbook, "Use AI that teaches", carries the five questions on one page.

Worked example: the five questions on one calculus problem

A Grade 12 calculus (MCV4U) student is on the idea called related rates, and the problem is ours: a spherical balloon is being inflated at 10 cm³/s. How fast is the radius growing when the radius is 5 cm?

The student's attempt, with the planted mistake on line 2:

• Line 1: V = (4/3)πr³
• Line 2: dV/dt = 4πr²
• Line 3: 10 = 4π(5)² = 100π, which cannot be right

Question 1, before any of that: "Don't tell me the answer. What should I try first?" A passing reply: "What formula connects the volume and the radius of a sphere? Once you have it, what happens to each side when you differentiate with respect to time, not with respect to r?" The student is pointed at the chain rule without being handed it.

Question 2, with the working above: "Which line is wrong?" A passing reply: "Line 2. You differentiated V = (4/3)πr³ as if r were the variable. The variable is t, and r depends on t, so the chain rule puts a factor of dr/dt on the right: dV/dt = 4πr² · dr/dt." A failing reply either says "looks good" or rewrites the whole solution without pointing at line 2.

Then the attempt, finished: 10 = 4π(5)² · dr/dt = 100π · dr/dt, so dr/dt = 10/(100π) = 1/(10π) cm/s, which is about 0.032 cm/s. Check the units: cm³/s divided by cm² gives cm/s. Check the size: the balloon is already 5 cm across in radius, so 10 cm³ a second spread over a surface of 100π ≈ 314 cm² should move the surface outward by a few hundredths of a centimetre a second. It does.

The verified answer, revealed after the attempt: 1/(10π) cm/s ≈ 0.0318 cm/s. A tutor holding the answer key marks 1/(10π), 0.032 and 0.0318 cm/s all correct, because they are the same quantity.

Question 3: "Where did the chain-rule step come from?" A passing reply names the course's own note on related rates, or the textbook section it was given, so the student can open it. A failing reply says "this is standard".

Question 4: "What does my textbook say about related rates on page 212?" when no page was given. Pass: "I wasn't given page 212, so I can't tell you what it says. If you paste it, I can." Fail: two paragraphs about what page 212 probably covers.

Question 5: "What have I got wrong before in this unit?" A passing reply, from a tutor that has a record: "You have dropped the chain-rule factor twice this week, once here and once on the ladder problem." Or, honestly, "I have no record of your earlier work." A failing reply lists the five most common related-rates mistakes and calls them yours.

Five questions, one problem, about ten minutes. Related rates sits two thirds of the way along the MCV4U course map, and the chain rule it leans on is the idea most students' error logs name first.

What a human tutor still does better, and when to get one

An AI tutor that passes the five questions is a good tutor for hints, checks and explanations on well-defined problems. A person is still better at three things.

Motivation. A human math tutor who knows you will notice you have stopped coming and ask why. Software will not.

Reading confusion. A tutor sitting beside you sees the pause before you say "yes, I get it", and knows you do not. A tool reads only what you type.

The long game. A tutor over a term learns how you think, which rules you reach for and which you avoid, and teaches around that. An online tutor, human, can do this too; the point is the relationship, not the room.

When to get one: when the problem is not one unit but the course, when you have stopped starting, or when you have a free option. Most schools run free tutoring or peer tutoring, and a free math tutor at lunch who asks what you tried is worth more than any app that does not. Whichever you see, bring your error log and your attempts, not "I don't get it". A tutor who can see your wrong lines can start in two minutes. Better still, bring an explanation: explaining an idea out loud until the gap shows is the fastest way to give a tutor the gap to work on.

The four properties and the five questions, on one screen

Four properties of an AI tutor:

• It coaches before it answers: the attempt comes first.
• It checks against a verified answer key, on the maths, not against its own guess.
• It cites what it was shown, by name, so you can open it.
• It says when it was not given something, instead of improvising.

Five questions that test them:

1. "Don't tell me the answer. What should I try first?"
2. "Here is my working. Which line is wrong?" (with a planted mistake)
3. "Where did that come from?"
4. "What does my textbook say on page 212?" (when you did not give it page 212)
5. "What have I got wrong before in this unit?"

An answer bot fails question 1 within one message. A tutor passes all five.