A physics tutor more than doubled students’ median learning gains

PS2 Pal, built in TeachGPT. [1]

GPT-4. [1]

GPT-4-0613. [1]

Harvard University developed the tutor. OpenAI supplied GPT-4. [1]

GPT-4 supplied explanations and feedback during physics exercises. [1]

TeachGPT presented each problem in stages. Instructors supplied teaching instructions and worked solutions in its prompts. [1]

The tutor used students’ messages and course material to generate explanations and feedback. [1]

The tutor conversed directly with students within lessons that instructors had prepared and tested. [1]

Students completed a quiz, studied with PS2 Pal at home or attended an active-learning class, and took another quiz. They switched formats for the second lesson. [1]

Instructors wrote the worked solutions and tested the prompts before use. Students then worked with the tutor on their own. A separate team member wrote the tests. [1]

The language model delivered the instruction. The comparison was active learning taught in class. The authors credit the research-based design (prompts, scaffolding) beside the model.

PS2 Pal answered questions and gave feedback within lessons that instructors had prepared. The trial tested this entire teaching format, including videos and self-paced work. [1]

Two outcomes within one study: student learning and the course’s teaching capability.

Students had higher post-test scores after the AI lesson and spent a median of 49 minutes on it. [1]

Improved Learning

Observed in the trial. [1]

People. Organizations.

Students in Harvard’s Physical Sciences 2 course, and the course that provided the lessons. [1]

The course enrolled 233 students, of whom 194 were eligible for the study. The paper does not count individual beneficiaries. [1]

Higher post-test scores occurred among students with lower and higher prior physics scores. The at-home lesson required internet access. [1]

Learning was tested immediately after each lesson. The study does not report later retention. [1]

Yes

Preparing each AI lesson took several days, after several months of platform development. Thirty percent of AI sessions exceeded the classroom’s assumed 60 minutes of learning. [1]

Harvard PS2 tutoring study, protocol IRB23-0797. [1]

A randomized crossover trial across two lessons. Peer groups of two or three students were assigned together, then switched conditions the following week. [1]

Researchers used quizzes before and after each lesson, logged time with the tutor, and asked students about their experience on a five-point scale. [1]

Harvard’s introductory physics course for life-science students. The paper reports 194 eligible students and compares 142 AI post-tests with 174 classroom post-tests. [1]

Editor’s note. Methods reports 194 eligible students. Results gives 142 AI post-tests and 174 classroom post-tests. Whether these counts describe the same analysis population is unclear.

Post-test scores, time spent learning, and students’ ratings of engagement, motivation, enjoyment, and growth mindset. [1]

An active-learning class covered the same material with peer discussion and instructor feedback. The pooled pre-test median was 2.75 points. [1]

Median post-test scores were 4.5 with AI and 3.5 in class. The adjusted advantage was 0.63 standard deviations. [1]

The AI lesson produced higher scores than the classroom lesson. The comparison does not separate the model’s contribution from the videos, prepared solutions, or self-paced format. [1]

The study covered two lessons at one university. High scores limited the test’s ability to distinguish further gains. [1]

Each participant was assigned one AI lesson and one classroom lesson in consecutive weeks. The AI lessons took a median of 49 minutes. [1]

The supplement contains teaching prompts and methods. The authors also published a Stata data file on GitHub. [1]

Not reported. [1]

Not reported. [1]

Not reported. [1]

Not reported. [1]

Not reported. [1]

Not reported. [1]

Not reported. [1]

Lesson preparation took several days, after several months of platform development. Financial costs were not reported. [1]

Not reported. [1]

Not reported. [1]

Not reported. [1]

Developer: Harvard University (Kestin, Miller, Klales, Milbourne, Ponti). Operator: Harvard University, Physical Sciences 2.

Benefit: Cambridge, United States. Developer: United States. Operator: United States.

Not assessed

Students needed internet access to use TeachGPT with GPT-4-0613. Instructors supplied videos, questions, and worked solutions. [1]

Not assessed

Not assessed

Instructors supplied carefully tested prompts, worked solutions, and videos. Students needed internet access and time to work at their own pace. The study covered two introductory topics. [1]

The cited Nigerian program used Microsoft Copilot. It was a related study, not a replication of PS2 Pal.

No later development found in the cited source.

Students used PS2 Pal through TeachGPT for the study. The supplement includes its teaching prompts. Public access to this particular tutor is not established. [1]

Not assessed

Not assessed

Not assessed

Not assessed

AIBD-0039-source-0001

Numbered references below.

English. No translation was used. [1]

Developer

The paper, supplement, and data deposit concern one trial conducted by the tutor’s developers. [1]

The paper and supplement are accessible in full. The authors’ dataset is listed on GitHub. [1]

Students scored higher after lessons with PS2 Pal than after the corresponding classroom lessons. The supplement identifies GPT-4-0613 as the model that they used. [1]

The results establish the test scores and time spent. The supplement identifies the model and teaching prompts. [1]

A peer-reviewed randomized trial by the developers. It supports the teaching format’s effect, without isolating the model’s contribution. [1]

Later retention and financial returns were not reported. [1]

People. Organizations.

Depth 2 for students and depth 2 for the course’s teaching capability.

People: Students achieved a measurable improvement in immediate post-test scores. The trial did not establish a lasting change in their capabilities. Organizations: The course demonstrated a teaching format with better post-test results. This uses the same trial evidence as the student assessment, not a separate organizational study.

The grades describe results observed in the trial.

AIBD’s five-level depth rubric, reviewed on 25 October 2026.

AIBD, 25 October 2026.

Depth 1 would omit the measured learning improvement. Depth 3 would require evidence of a lasting change.

A peer-reviewed randomized trial by the developers.

A randomized study of two undergraduate physics lessons. [1]

Student-facing AI learning assistant: Harvard's GPT-4 tutor doubled learning gains, while LAUSD's AllHere chatbot “Ed” failed to deliver academic support, costing the district up to $6 million and leaving students without it.

AllHere's Chatbot 'Ed' Fails and Costs Los Angeles Unified School District $6 Million

Education. Undergraduate physics.

The tutor was built to teach these physics lessons with research-based pedagogy. The learning gain was that purpose.

The lessons took place in weeks 9 and 10 of fall 2023. The paper was published on 3 June 2025. [1]

A classroom experiment during fall 2023. [1]

SHA-256: 64340e67d651…

AI Incident Database: 793.

Not assessed

Not assessed

7 revisions. Latest: 25 October 2026.