{"id":"AIBD-0618","title":"ChatGPT-written hints in the OATutor system produce the only learning gain significant against a no-hint control in a 274-person online algebra and statistics experiment (UC Berkeley, 2023)","summary":"In a UC Berkeley experiment with 274 Mechanical Turk workers in 2023, ChatGPT-written math hints gave a 17% pre-to-post gain, the only gain significant against a no-hint control.","date_observed":"2023-05-10","date_basis":"event","date_precision":"day","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"Zachary Pardos and Shreya Bhandari of the University of California, Berkeley ran an online experiment from 28 February to 10 May 2023 with 274 Amazon Mechanical Turk workers who held at least a high school degree. The authors randomly assigned each participant to one of four subjects drawn from OpenStax textbooks, elementary, intermediate or college algebra or statistics, and to one of three hint conditions in the open-source OATutor system: hints written by human tutors, worked-solution hints generated by the 15 December version of ChatGPT 3.5, or no hints. A three-item pre-test and post-test measured learning. The ChatGPT group showed a 17.00 percent learning gain from pre-test to post-test. The authors report that only the ChatGPT condition produced gains significant against the no-hint control, and that gains did not differ significantly between ChatGPT and human-tutor hints. ChatGPT hints failed the authors' quality checks on 32 percent of problems, a rate that a self-consistency technique cut to nearly zero for algebra. The learners were adult crowd workers in short sessions, and problems with figures were left out. PLOS ONE published the study on 24 May 2024.","category":"general benefit","unexpected":false,"purpose":"ChatGPT is a general-purpose chatbot. The authors used it to write hints for tutoring software, and learning from those hints was the purpose of the study.","system":"ChatGPT 3.5 (\"Dec 15 Version\"), prompted to write worked-solution hints shown in the OATutor tutoring system","model_version":"ChatGPT 3.5, \"Dec 15 Version\"","developer":"OpenAI","deployer":"University of California, Berkeley (Pardos and Bhandari), in an online study on Amazon Mechanical Turk","location":"Online (Amazon Mechanical Turk); study run from Berkeley, California, United States","domain":"Education","segment":null,"benefit_type":"Improved Learning","taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"personal","depth":2,"reason":"Participants given ChatGPT hints showed a 17.00 percent learning gain from pre-test to post-test, significant against a no-hint control."}],"ai_role":{"status":"assessed","statement":"Participants were randomly assigned to ChatGPT hints, human-tutor hints or no hints on the same problems, so the comparison isolates the effect of the ChatGPT hints."},"evidence":[{"kind":"peer-reviewed study","citation":"Pardos ZA, Bhandari S. ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLoS One. 2024;19(5):e0304013. doi:10.1371/journal.pone.0304013","title":"ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/","date":"2024-05-24","written_by":"independent","quote":"We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help."}],"replication":{"produced_by":["ChatGPT wrote worked solutions that served as hints on textbook problems.","Learners saw the hints in a tutoring system during short practice sessions."],"must_hold":["Hints must be checked or generated several times to catch errors.","Problems must be text only."],"echoes":[{"date":null,"setting":"The authors' earlier pilot with 77 Mechanical Turk participants, where ceiling effects limited ChatGPT gains","model_version":null,"outcome":"did not recur","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/"}]},"paired_incident":null,"headline":"*ChatGPT* hints help adults learn algebra and statistics online","metric":"The ChatGPT group showed a 17.00 percent learning gain from pre-test to post-test.","comparator":{"against":"Participants who received no hints","before":"1.85 percent gain in the no-hint control (not significant)","after":"17.00 percent gain with ChatGPT hints","basis":"measured","quote":"We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/"},"later":null,"value":null,"developer_country":"United States","deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/","editor":"AIBD"}],"confidence":"studied","shade":"high","claim_status":"Independently studied","claim_basis":"Sources: one independent peer-reviewed study. The cited evidence includes no finding by a regulator or court.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0618","catalog_version":"1.0.0","source_fingerprint":"ec4e2f4a78e203434794b8e7c9092aa5b115757ef307feef27c61f3a20a41f7c","collection_refs":[{"collection_id":"aibd","version":"2026-10-29"}],"source_ids":["AIBD-0618-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0618-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":[],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"partial","record_ids":["AIBD-0618-evaluation-1"]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"OpenAI","deployer":"University of California, Berkeley (Pardos and Bhandari), in an online study on Amazon Mechanical Turk"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"Participants were randomly assigned to ChatGPT hints, human-tutor hints or no hints on the same problems, so the comparison isolates the effect of the ChatGPT hints."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"general benefit","ai_role":{"status":"assessed","statement":"Participants were randomly assigned to ChatGPT hints, human-tutor hints or no hints on the same problems, so the comparison isolates the effect of the ChatGPT hints."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Participants given ChatGPT hints showed a 17.00 percent learning gain from pre-test to post-test, significant against a no-hint control."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"studied","claim_status":"Independently studied","basis":"Sources: one independent peer-reviewed study. The cited evidence includes no finding by a regulator or court.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Participants given ChatGPT hints showed a 17.00 percent learning gain from pre-test to post-test, significant against a no-hint control."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Participants given ChatGPT hints showed a 17.00 percent learning gain from pre-test to post-test, significant against a no-hint control."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"value_present","value":{"kind":"structured","data":{"classification":"evaluation or study","answer":"The case describes an evaluation or a study. The case is a randomized online experiment.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The case describes an evaluation or a study. The case is a randomized online experiment.","inferred":false}},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Compared with the incidents in the AI Incident Database snapshot of 26 October 2026.","record_fields":["paired_incident"],"source_refs":[]},"reason":"No incident in the AI Incident Database snapshot of 26 October 2026 concerns the same kind of system harming the same kind of stakeholder.","review_state":"reviewed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Independently studied. Sources: one independent peer-reviewed study. The cited evidence includes no finding by a regulator or court.","evidence_word":"studied","claim_status":"Independently studied"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Independently studied. Sources: one independent peer-reviewed study. The cited evidence includes no finding by a regulator or court.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"laboratory test","answer":"The system was tested in a laboratory. The hints were tested in a short online experiment with crowd workers.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was tested in a laboratory. The hints were tested in a short online experiment with crowd workers.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"value_present","value":{"kind":"text","text":"ChatGPT hints produced a gain significant against no hints and similar to human-tutor hints."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"ChatGPT hints produced a gain significant against no hints and similar to human-tutor hints.","inferred":false}},"evaluations[].comparator":{"state":"value_present","value":{"kind":"text","text":"No-hint control and human-tutor hints."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"No-hint control and human-tutor hints.","inferred":false}},"evaluations[].design":{"state":"value_present","value":{"kind":"text","text":"Randomized."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].effect":{"state":"value_present","value":{"kind":"text","text":"17.00 percent gain with ChatGPT hints against 1.85 percent with no hints and 11.62 percent with human-tutor hints."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"17.00 percent gain with ChatGPT hints against 1.85 percent with no hints and 11.62 percent with human-tutor hints.","inferred":false}},"evaluations[].exposure":{"state":"value_present","value":{"kind":"text","text":"One session of about 10 to 20 minutes."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"One session of about 10 to 20 minutes.","inferred":false}},"evaluations[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0618-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].measurement_method":{"state":"value_present","value":{"kind":"text","text":"Three-item pre-test and post-test on the assigned skill."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Three-item pre-test and post-test on the assigned skill.","inferred":false}},"evaluations[].metric":{"state":"value_present","value":{"kind":"text","text":"Percentage-point learning gain from pre-test to post-test."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Percentage-point learning gain from pre-test to post-test.","inferred":false}},"evaluations[].reproduction_materials":{"state":"value_present","value":{"kind":"text","text":"Participant data at doi:10.6084/m9.figshare.23935269."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Participant data at doi:10.6084/m9.figshare.23935269.","inferred":false}},"evaluations[].setting_sample":{"state":"value_present","value":{"kind":"text","text":"274 Amazon Mechanical Turk workers with at least a high school degree, 28 February to 10 May 2023."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"274 Amazon Mechanical Turk workers with at least a high school degree, 28 February to 10 May 2023.","inferred":false}},"evaluations[].uncertainty_limitations":{"state":"value_present","value":{"kind":"text","text":"Adult crowd workers; short sessions; three-item tests; 32 percent of ChatGPT hints failed quality checks before mitigation."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0618-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Adult crowd workers; short sessions; three-item tests; 32 percent of ChatGPT hints failed quality checks before mitigation.","inferred":false}},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0618"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Zachary Pardos and Shreya Bhandari of the University of California, Berkeley ran an online experiment from 28 February to 10 May 2023 with 274 Amazon Mechanical Turk workers who held at least a high school degree. The authors randomly assigned each participant to one of four subjects drawn from OpenStax textbooks, elementary, intermediate or college algebra or statistics, and to one of three hint conditions in the open-source OATutor system: hints written by human tutors, worked-solution hints generated by the 15 December version of ChatGPT 3.5, or no hints. A three-item pre-test and post-test measured learning. The ChatGPT group showed a 17.00 percent learning gain from pre-test to post-test. The authors report that only the ChatGPT condition produced gains significant against the no-hint control, and that gains did not differ significantly between ChatGPT and human-tutor hints. ChatGPT hints failed the authors' quality checks on 32 percent of problems, a rate that a self-consistency technique cut to nearly zero for algebra. The learners were adult crowd workers in short sessions, and problems with figures were left out. PLOS ONE published the study on 24 May 2024.","benefit_type":"Improved Learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Participants given ChatGPT hints showed a 17.00 percent learning gain from pre-test to post-test, significant against a no-hint control."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"ChatGPT is a general-purpose chatbot. The authors used it to write hints for tutoring software, and learning from those hints was the purpose of the study."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Online (Amazon Mechanical Turk); study run from Berkeley, California, United States","developer_country":"United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"ChatGPT is a general-purpose chatbot. The authors used it to write hints for tutoring software, and learning from those hints was the purpose of the study."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2023-05-10","date_basis":"event","date_precision":"day","revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["ChatGPT wrote worked solutions that served as hints on textbook problems.","Learners saw the hints in a tutoring system during short practice sessions."],"must_hold":["Hints must be checked or generated several times to catch errors.","Problems must be text only."],"echoes":[{"setting":"The authors' earlier pilot with 77 Mechanical Turk participants, where ceiling effects limited ChatGPT gains","outcome":"did not recur","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["ChatGPT wrote worked solutions that served as hints on textbook problems.","Learners saw the hints in a tutoring system during short practice sessions."],"must_hold":["Hints must be checked or generated several times to catch errors.","Problems must be text only."],"echoes":[{"setting":"The authors' earlier pilot with 77 Mechanical Turk participants, where ceiling effects limited ChatGPT gains","outcome":"did not recur","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Education"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Pardos ZA, Bhandari S. ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLoS One. 2024;19(5):e0304013. doi:10.1371/journal.pone.0304013","title":"ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/","date":"2024-05-24","written_by":"independent","quote":"We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0618-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Pardos ZA, Bhandari S. ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLoS One. 2024;19(5):e0304013. doi:10.1371/journal.pone.0304013","title":"ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/","date":"2024-05-24","written_by":"independent","quote":"We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In a UC Berkeley experiment with 274 Mechanical Turk workers in 2023, ChatGPT-written math hints gave a 17% pre-to-post gain, the only gain significant against a no-hint control.","account":"Zachary Pardos and Shreya Bhandari of the University of California, Berkeley ran an online experiment from 28 February to 10 May 2023 with 274 Amazon Mechanical Turk workers who held at least a high school degree. The authors randomly assigned each participant to one of four subjects drawn from OpenStax textbooks, elementary, intermediate or college algebra or statistics, and to one of three hint conditions in the open-source OATutor system: hints written by human tutors, worked-solution hints generated by the 15 December version of ChatGPT 3.5, or no hints. A three-item pre-test and post-test measured learning. The ChatGPT group showed a 17.00 percent learning gain from pre-test to post-test. The authors report that only the ChatGPT condition produced gains significant against the no-hint control, and that gains did not differ significantly between ChatGPT and human-tutor hints. ChatGPT hints failed the authors' quality checks on 32 percent of problems, a rate that a self-consistency technique cut to nearly zero for algebra. The learners were adult crowd workers in short sessions, and problems with figures were left out. PLOS ONE published the study on 24 May 2024."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"value_present","value":{"kind":"record_fields","fields":{"model_version":"ChatGPT 3.5, \"Dec 15 Version\""}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["model_version"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"ChatGPT 3.5 (\"Dec 15 Version\"), prompted to write worked-solution hints shown in the OATutor tutoring system"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"ChatGPT-written hints in the OATutor system produce the only learning gain significant against a no-hint control in a 274-person online algebra and statistics experiment (UC Berkeley, 2023)","headline":"*ChatGPT* hints help adults learn algebra and statistics online"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["187ea83521fb7c43be57a38d9e4bc599983cac96af4510fb73e240520cfba300"],"role":"SHA-256 of the complete public case and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Zachary Pardos and Shreya Bhandari of the University of California, Berkeley ran an online experiment from 28 February to 10 May 2023 with 274 Amazon Mechanical Turk workers who held at least a high school degree. The authors randomly assigned each participant to one of four subjects drawn from OpenStax textbooks, elementary, intermediate or college algebra or statistics, and to one of three hint conditions in the open-source OATutor system: hints written by human tutors, worked-solution hints generated by the 15 December version of ChatGPT 3.5, or no hints. A three-item pre-test and post-test measured learning. The ChatGPT group showed a 17.00 percent learning gain from pre-test to post-test. The authors report that only the ChatGPT condition produced gains significant against the no-hint control, and that gains did not differ significantly between ChatGPT and human-tutor hints. ChatGPT hints failed the authors' quality checks on 32 percent of problems, a rate that a self-consistency technique cut to nearly zero for algebra. The learners were adult crowd workers in short sessions, and problems with figures were left out. PLOS ONE published the study on 24 May 2024.","replication":{"produced_by":["ChatGPT wrote worked solutions that served as hints on textbook problems.","Learners saw the hints in a tutoring system during short practice sessions."],"must_hold":["Hints must be checked or generated several times to catch errors.","Problems must be text only."],"echoes":[{"setting":"The authors' earlier pilot with 77 Mechanical Turk participants, where ceiling effects limited ChatGPT gains","outcome":"did not recur","source":"https://pmc.ncbi.nlm.nih.gov/articles/PMC11125466/"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":43,"reviewed":15,"reviewed_complete":14,"reviewed_partial":1,"record_mapped":29,"review_not_assessed":47,"not_assessed":0,"complete":false,"by_state":{"value_present":43,"not_reported":0,"unresolved":48,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"45fc0d29bd355b2e36c9b8ecf79f5e1906bbe03dd3b4e3fcce83269e803b4174"}