{"id":"AIBD-0617","title":"Georgia State's TA Pounce course chatbot raises the share of students earning an A or B by 4 percentage points in randomized trials in two online courses (2021-2023)","summary":"In trials with 2,483 Georgia State students in online Government and Microeconomics courses from 2021 to 2023, the TA Pounce text chatbot raised the share earning an A or B by 4 points.","date_observed":"2023-01-01","date_basis":"event","date_precision":"year","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"Georgia State University tested a course chatbot, called TA Pounce, that ran on the text-messaging platform of Mainstay, in two large asynchronous online courses. In Introduction to American Government (fall 2021, spring 2022 and fall 2022) and Principles of Microeconomics (2022-23 academic year), researchers randomly assigned half of 1,568 and 915 enrolled students to receive two or three scheduled text messages a week. The messages listed due dates, prompted students to finish assignments and encouraged them. The chatbot answered questions from a course knowledge base and passed other questions to the teaching assistant. Katharine Meyer of the Brookings Institution, Lindsay Page of Brown University and coauthors report that the chatbot raised the likelihood that students earned an A or B by four percentage points, from 61 percent in the control group, with similar results in both courses. In Microeconomics, women gained most and men showed no effect. The estimated gain in the numeric final grade, 1.58 points on a control mean of 71.07, was not statistically significant. The authors describe the chatbot as non-generative AI. Several coauthors work at Georgia State. The study appeared as an NBER working paper in July 2026 and in Economics of Education Review.","category":"general benefit","unexpected":false,"purpose":"The chatbot was built to send course reminders and answer students' questions in large online courses. Better course performance was that purpose.","system":"TA Pounce, a course chatbot on Mainstay's text-messaging platform with non-generative AI question answering and scheduled outreach","model_version":null,"developer":"Mainstay (formerly AdmitHub)","deployer":"Georgia State University (Political Science and Economics departments)","location":"Atlanta, Georgia, United States","domain":"Education","segment":null,"benefit_type":"Improved Learning","taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"personal","depth":2,"reason":"Students assigned to the chatbot were four percentage points more likely to earn an A or B in the course."}],"ai_role":{"status":"assessed","statement":"Students were randomly assigned to receive the chatbot or not. The estimate covers the whole service, including messages written by course staff and questions that the teaching assistant answered, so the share due to the chatbot's automated answers is not isolated."},"evidence":[{"kind":"report","citation":"Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. NBER Working Paper 35397, July 2026; published in Economics of Education Review vol. 115 (2026), doi:10.1016/j.econedurev.2026.102816.","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance","url":"https://www.nber.org/papers/w35397","date":"2026-07-01","written_by":"independent","quote":"We find the chatbot messaging increased students' final grades and engagement with academic supports, such as tutoring."},{"kind":"peer-reviewed study","citation":"Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. Economics of Education Review. 2026;115:102816. doi:10.1016/j.econedurev.2026.102816","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance","url":"https://doi.org/10.1016/j.econedurev.2026.102816","date":null,"written_by":"independent","quote":null},{"kind":"report","citation":"Meyer KE, Page LC, et al. NBER Working Paper 35397, full text (results and sample).","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance (PDF)","url":"https://www.nber.org/system/files/working_papers/w35397/w35397.pdf","date":"2026-07-01","written_by":"independent","quote":"the academic chatbot significantly shifted students' final grades, increasing the likelihood that students earned an A or B by four percentage points."}],"replication":{"produced_by":["The chatbot sent two or three scheduled messages a week, tailored to whether a student had finished the previous week's work.","A teaching assistant answered the questions that the chatbot could not.","The courses were large, asynchronous and online, with few chances to meet the instructor."],"must_hold":["Students must read text messages on their phones.","Course staff must keep due dates and the knowledge base current."],"echoes":[]},"paired_incident":{"database":"AI Incident Database","identifier":"793","title":"AllHere's Chatbot 'Ed' Fails and Costs Los Angeles Unified School District $6 Million","url":"https://incidentdatabase.ai/cite/793","severity":null,"group":"personal","basis":"Both records concern an AI chatbot that an educational institution deployed to give its students academic support, and students are the stakeholders in each."},"headline":"*TA Pounce* chatbot helps Georgia State students earn higher grades","metric":"raised the likelihood that students earned an A or B by four percentage points","comparator":{"against":"Students in the same course sections who did not receive the chatbot","before":"61 percent earned an A or B (control group)","after":"Four percentage points more likely to earn an A or B","basis":"measured","quote":"the academic chatbot significantly shifted students' final grades, increasing the likelihood that students earned an A or B by four percentage points.","url":"https://www.nber.org/system/files/working_papers/w35397/w35397.pdf"},"later":null,"value":null,"developer_country":"United States","deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://www.nber.org/papers/w35397","editor":"AIBD"},{"date":"2026-10-29","change":"Linked a comparable incident in the AI Incident Database (incident 793).","source":"https://incidentdatabase.ai/cite/793","editor":"AIBD"}],"confidence":"studied","shade":"high","claim_status":"Independently studied","claim_basis":"Sources: one independent peer-reviewed study and two independent reports. The cited evidence includes no finding by a regulator or court.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0617","catalog_version":"1.0.0","source_fingerprint":"51c15b06822dbc32bcf3c61bb8fb038ac5b6140357435ba68411368c73ac75c0","collection_refs":[{"collection_id":"aibd","version":"2026-10-29"}],"source_ids":["AIBD-0617-source-0001","AIBD-0617-source-0002","AIBD-0617-source-0003"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0617-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":[],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"partial","record_ids":["AIBD-0617-evaluation-1"]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Mainstay (formerly AdmitHub)","deployer":"Georgia State University (Political Science and Economics departments)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"Students were randomly assigned to receive the chatbot or not. The estimate covers the whole service, including messages written by course staff and questions that the teaching assistant answered, so the share due to the chatbot's automated answers is not isolated."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"general benefit","ai_role":{"status":"assessed","statement":"Students were randomly assigned to receive the chatbot or not. The estimate covers the whole service, including messages written by course staff and questions that the teaching assistant answered, so the share due to the chatbot's automated answers is not isolated."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students assigned to the chatbot were four percentage points more likely to earn an A or B in the course."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"studied","claim_status":"Independently studied","basis":"Sources: one independent peer-reviewed study and two independent reports. The cited evidence includes no finding by a regulator or court.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students assigned to the chatbot were four percentage points more likely to earn an A or B in the course."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students assigned to the chatbot were four percentage points more likely to earn an A or B in the course."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"value_present","value":{"kind":"structured","data":{"classification":"evaluation or study","answer":"The case describes an evaluation or a study. The case is a pair of preregistered student-randomized trials.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0617-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The case describes an evaluation or a study. The case is a pair of preregistered student-randomized trials.","inferred":false}},"case_links[]":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Both records concern an AI chatbot that an educational institution deployed to give its students academic support, and students are the stakeholders in each.","links":[{"database":"AI Incident Database","identifier":"793","title":"AllHere's Chatbot 'Ed' Fails and Costs Los Angeles Unified School District $6 Million","url":"https://incidentdatabase.ai/cite/793","group":"personal","basis":"Both records concern an AI chatbot that an educational institution deployed to give its students academic support, and students are the stakeholders in each.","relation":"comparable_harm"}]}},"provenance":{"kind":"curatorial","basis":"Compared with the incidents in the AI Incident Database snapshot of 26 October 2026; a second reading confirmed that the incident concerns the same kind of system and stakeholder.","record_fields":["paired_incident"],"source_refs":["https://incidentdatabase.ai/cite/793"]},"review_state":"reviewed","coverage":"complete","display":{"answer":"Both records concern an AI chatbot that an educational institution deployed to give its students academic support, and students are the stakeholders in each.","inferred":false}},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Independently studied. Sources: one independent peer-reviewed study and two independent reports. The cited evidence includes no finding by a regulator or court.","evidence_word":"studied","claim_status":"Independently studied"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0001","AIBD-0617-source-0002","AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Independently studied. Sources: one independent peer-reviewed study and two independent reports. The cited evidence includes no finding by a regulator or court.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"pilot","answer":"The system was used in a pilot. The chatbot ran in selected course sections during randomized trials over four semesters.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0617-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was used in a pilot. The chatbot ran in selected course sections during randomized trials over four semesters.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"value_present","value":{"kind":"text","text":"The chatbot service as a whole; the automated answering is not separated from staff-written messages."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The chatbot service as a whole; the automated answering is not separated from staff-written messages.","inferred":false}},"evaluations[].comparator":{"state":"value_present","value":{"kind":"text","text":"Students randomly assigned to no chatbot."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Students randomly assigned to no chatbot.","inferred":false}},"evaluations[].design":{"state":"value_present","value":{"kind":"text","text":"Randomized."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].effect":{"state":"value_present","value":{"kind":"text","text":"Four percentage points from a control rate of 61 percent."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Four percentage points from a control rate of 61 percent.","inferred":false}},"evaluations[].exposure":{"state":"value_present","value":{"kind":"text","text":"Two or three text messages a week for a semester."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Two or three text messages a week for a semester.","inferred":false}},"evaluations[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0617-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].measurement_method":{"state":"value_present","value":{"kind":"text","text":"Final course grades and grade categories from university records."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Final course grades and grade categories from university records.","inferred":false}},"evaluations[].metric":{"state":"value_present","value":{"kind":"text","text":"Probability of earning an A or B."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Probability of earning an A or B.","inferred":false}},"evaluations[].reproduction_materials":{"state":"value_present","value":{"kind":"text","text":"Registry of Efficacy and Effectiveness Studies 8160 and 13760."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Registry of Efficacy and Effectiveness Studies 8160 and 13760.","inferred":false}},"evaluations[].setting_sample":{"state":"value_present","value":{"kind":"text","text":"1,568 students in Introduction to American Government (fall 2021 to fall 2022) and 915 in Principles of Microeconomics (2022-23) at Georgia State University."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"1,568 students in Introduction to American Government (fall 2021 to fall 2022) and 915 in Principles of Microeconomics (2022-23) at Georgia State University.","inferred":false}},"evaluations[].uncertainty_limitations":{"state":"value_present","value":{"kind":"text","text":"Numeric final grade effect not significant; men in Microeconomics showed no effect; several coauthors work at the deploying university."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0617-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Numeric final grade effect not significant; men in Microeconomics showed no effect; several coauthors work at the deploying university.","inferred":false}},"external_references[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"paired_incident":{"database":"AI Incident Database","identifier":"793","title":"AllHere's Chatbot 'Ed' Fails and Costs Los Angeles Unified School District $6 Million","url":"https://incidentdatabase.ai/cite/793","group":"personal","basis":"Both records concern an AI chatbot that an educational institution deployed to give its students academic support, and students are the stakeholders in each."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["paired_incident"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0617"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Georgia State University tested a course chatbot, called TA Pounce, that ran on the text-messaging platform of Mainstay, in two large asynchronous online courses. In Introduction to American Government (fall 2021, spring 2022 and fall 2022) and Principles of Microeconomics (2022-23 academic year), researchers randomly assigned half of 1,568 and 915 enrolled students to receive two or three scheduled text messages a week. The messages listed due dates, prompted students to finish assignments and encouraged them. The chatbot answered questions from a course knowledge base and passed other questions to the teaching assistant. Katharine Meyer of the Brookings Institution, Lindsay Page of Brown University and coauthors report that the chatbot raised the likelihood that students earned an A or B by four percentage points, from 61 percent in the control group, with similar results in both courses. In Microeconomics, women gained most and men showed no effect. The estimated gain in the numeric final grade, 1.58 points on a control mean of 71.07, was not statistically significant. The authors describe the chatbot as non-generative AI. Several coauthors work at Georgia State. The study appeared as an NBER working paper in July 2026 and in Economics of Education Review.","benefit_type":"Improved Learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students assigned to the chatbot were four percentage points more likely to earn an A or B in the course."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"The chatbot was built to send course reminders and answer students' questions in large online courses. Better course performance was that purpose."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Atlanta, Georgia, United States","developer_country":"United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"The chatbot was built to send course reminders and answer students' questions in large online courses. Better course performance was that purpose."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2023-01-01","date_basis":"event","date_precision":"year","revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://www.nber.org/papers/w35397","editor":"AIBD"},{"date":"2026-10-29","change":"Linked a comparable incident in the AI Incident Database (incident 793).","source":"https://incidentdatabase.ai/cite/793","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["The chatbot sent two or three scheduled messages a week, tailored to whether a student had finished the previous week's work.","A teaching assistant answered the questions that the chatbot could not.","The courses were large, asynchronous and online, with few chances to meet the instructor."],"must_hold":["Students must read text messages on their phones.","Course staff must keep due dates and the knowledge base current."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://www.nber.org/papers/w35397","editor":"AIBD"},{"date":"2026-10-29","change":"Linked a comparable incident in the AI Incident Database (incident 793).","source":"https://incidentdatabase.ai/cite/793","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Education"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"report","citation":"Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. NBER Working Paper 35397, July 2026; published in Economics of Education Review vol. 115 (2026), doi:10.1016/j.econedurev.2026.102816.","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance","url":"https://www.nber.org/papers/w35397","date":"2026-07-01","written_by":"independent","quote":"We find the chatbot messaging increased students' final grades and engagement with academic supports, such as tutoring."},{"kind":"peer-reviewed study","citation":"Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. Economics of Education Review. 2026;115:102816. doi:10.1016/j.econedurev.2026.102816","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance","url":"https://doi.org/10.1016/j.econedurev.2026.102816","written_by":"independent"},{"kind":"report","citation":"Meyer KE, Page LC, et al. NBER Working Paper 35397, full text (results and sample).","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance (PDF)","url":"https://www.nber.org/system/files/working_papers/w35397/w35397.pdf","date":"2026-07-01","written_by":"independent","quote":"the academic chatbot significantly shifted students' final grades, increasing the likelihood that students earned an A or B by four percentage points."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0617-source-0001","AIBD-0617-source-0002","AIBD-0617-source-0003"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"report","citation":"Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. NBER Working Paper 35397, July 2026; published in Economics of Education Review vol. 115 (2026), doi:10.1016/j.econedurev.2026.102816.","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance","url":"https://www.nber.org/papers/w35397","date":"2026-07-01","written_by":"independent","quote":"We find the chatbot messaging increased students' final grades and engagement with academic supports, such as tutoring."},{"kind":"peer-reviewed study","citation":"Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. Economics of Education Review. 2026;115:102816. doi:10.1016/j.econedurev.2026.102816","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance","url":"https://doi.org/10.1016/j.econedurev.2026.102816","written_by":"independent"},{"kind":"report","citation":"Meyer KE, Page LC, et al. NBER Working Paper 35397, full text (results and sample).","title":"Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance (PDF)","url":"https://www.nber.org/system/files/working_papers/w35397/w35397.pdf","date":"2026-07-01","written_by":"independent","quote":"the academic chatbot significantly shifted students' final grades, increasing the likelihood that students earned an A or B by four percentage points."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In trials with 2,483 Georgia State students in online Government and Microeconomics courses from 2021 to 2023, the TA Pounce text chatbot raised the share earning an A or B by 4 points.","account":"Georgia State University tested a course chatbot, called TA Pounce, that ran on the text-messaging platform of Mainstay, in two large asynchronous online courses. In Introduction to American Government (fall 2021, spring 2022 and fall 2022) and Principles of Microeconomics (2022-23 academic year), researchers randomly assigned half of 1,568 and 915 enrolled students to receive two or three scheduled text messages a week. The messages listed due dates, prompted students to finish assignments and encouraged them. The chatbot answered questions from a course knowledge base and passed other questions to the teaching assistant. Katharine Meyer of the Brookings Institution, Lindsay Page of Brown University and coauthors report that the chatbot raised the likelihood that students earned an A or B by four percentage points, from 61 percent in the control group, with similar results in both courses. In Microeconomics, women gained most and men showed no effect. The estimated gain in the numeric final grade, 1.58 points on a control mean of 71.07, was not statistically significant. The authors describe the chatbot as non-generative AI. Several coauthors work at Georgia State. The study appeared as an NBER working paper in July 2026 and in Economics of Education Review."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"TA Pounce, a course chatbot on Mainstay's text-messaging platform with non-generative AI question answering and scheduled outreach"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Georgia State's TA Pounce course chatbot raises the share of students earning an A or B by 4 percentage points in randomized trials in two online courses (2021-2023)","headline":"*TA Pounce* chatbot helps Georgia State students earn higher grades"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["36f93bdb719a2751144935f8a55c2fd24fd723c98cde8fe5f5a828d975f10359"],"role":"SHA-256 of the complete public case and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Georgia State University tested a course chatbot, called TA Pounce, that ran on the text-messaging platform of Mainstay, in two large asynchronous online courses. In Introduction to American Government (fall 2021, spring 2022 and fall 2022) and Principles of Microeconomics (2022-23 academic year), researchers randomly assigned half of 1,568 and 915 enrolled students to receive two or three scheduled text messages a week. The messages listed due dates, prompted students to finish assignments and encouraged them. The chatbot answered questions from a course knowledge base and passed other questions to the teaching assistant. Katharine Meyer of the Brookings Institution, Lindsay Page of Brown University and coauthors report that the chatbot raised the likelihood that students earned an A or B by four percentage points, from 61 percent in the control group, with similar results in both courses. In Microeconomics, women gained most and men showed no effect. The estimated gain in the numeric final grade, 1.58 points on a control mean of 71.07, was not statistically significant. The authors describe the chatbot as non-generative AI. Several coauthors work at Georgia State. The study appeared as an NBER working paper in July 2026 and in Economics of Education Review.","replication":{"produced_by":["The chatbot sent two or three scheduled messages a week, tailored to whether a student had finished the previous week's work.","A teaching assistant answered the questions that the chatbot could not.","The courses were large, asynchronous and online, with few chances to meet the instructor."],"must_hold":["Students must read text messages on their phones.","Course staff must keep due dates and the knowledge base current."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":43,"reviewed":15,"reviewed_complete":15,"reviewed_partial":0,"record_mapped":28,"review_not_assessed":48,"not_assessed":0,"complete":false,"by_state":{"value_present":43,"not_reported":0,"unresolved":48,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"0fe697744b0cebbc13ac180a00bd1167b1370ef0376575db28d83dcf4526df07"}