Benefits
- Benefit
-
Raised the likelihood that students earned an A or B by four percentage points
Awaiting review
- Outcome identifiers
-
Unresolved
Not reviewedUnresolved
- Category
-
Improved Learning
Awaiting review
- Status
-
Reported benefit
Awaiting review
- Beneficiary groups
-
People.
Awaiting review
- Beneficiaries
-
Unresolved
Not reviewedUnresolved
- Reach
-
Unresolved
Not reviewedUnresolved
- Distribution and access
-
Unresolved
Not reviewedUnresolved
- Duration
-
Unresolved
Not reviewedUnresolved
- Expected?
-
Yes
Awaiting review
- Costs and harms
-
Unresolved
Not reviewedUnresolved
AI system
- System
-
TA Pounce, a course chatbot on Mainstay's text-messaging platform with non-generative AI question answering and scheduled outreach
Awaiting review
- Model
-
This is an explicit unresolved assignment: the sources on record do not settle this question.
Not reviewedUnresolved
- Release
-
Unresolved
Not reviewedUnresolved
- Provider
-
Unresolved
Not reviewedUnresolved
- Role
-
Unresolved
Not reviewedUnresolved
- Configuration
-
Unresolved
Not reviewedUnresolved
- Inputs and outputs
-
Unresolved
Not reviewedUnresolved
- Autonomy
-
Unresolved
Not reviewedUnresolved
- Workflow
-
The chatbot sent two or three scheduled messages a week, tailored to whether a student had finished the previous week's work. A teaching assistant answered the questions that the chatbot could not. The courses were large, asynchronous and online, with few chances to meet the instructor.
Awaiting review
- Human oversight
-
Unresolved
Not reviewedUnresolved
- Relation to AI
-
Students were randomly assigned to receive the chatbot or not. The estimate covers the whole service, including messages written by course staff and questions that the teaching assistant answered, so the share due to the chatbot's automated answers is not isolated.
Awaiting review
- AI contribution
-
Students were randomly assigned to receive the chatbot or not. The estimate covers the whole service, including messages written by course staff and questions that the teaching assistant answered, so the share due to the chatbot's automated answers is not isolated.
Awaiting review
Measurement
- Studies
-
Randomized. [3]
Fully reviewed: supporting details for Studies
Review date: 29 October 2026.
Sources: [3]
- Study design
-
Randomized. [3]
Fully reviewed: supporting details for Study design
Review date: 29 October 2026.
Sources: [3]
- Measurement
-
Final course grades and grade categories from university records. [3]
Fully reviewed: supporting details for Measurement
Review date: 29 October 2026.
Sources: [3]
- Setting and sample
-
1,568 students in Introduction to American Government (fall 2021 to fall 2022) and 915 in Principles of Microeconomics (2022-23) at Georgia State University. [3]
Fully reviewed: supporting details for Setting and sample
Review date: 29 October 2026.
Sources: [3]
- Measures
-
Probability of earning an A or B. [3]
Fully reviewed: supporting details for Measures
Review date: 29 October 2026.
Sources: [3]
- Comparison
-
Students randomly assigned to no chatbot. [3]
Fully reviewed: supporting details for Comparison
Review date: 29 October 2026.
Sources: [3]
- Results
-
Four percentage points from a control rate of 61 percent. [3]
Fully reviewed: supporting details for Results
Review date: 29 October 2026.
Sources: [3]
- Additional value from AI
-
The chatbot service as a whole; the automated answering is not separated from staff-written messages. [3]
Fully reviewed: supporting details for Additional value from AI
Review date: 29 October 2026.
Sources: [3]
- Limitations
-
Numeric final grade effect not significant; men in Microeconomics showed no effect; several coauthors work at the deploying university. [3]
Fully reviewed: supporting details for Limitations
Review date: 29 October 2026.
Sources: [3]
Money
- Kind of value
-
The kind of monetary value is unclear.
Not reviewedUnresolved
- Amount
-
The monetary value is not established.
Not reviewedUnresolved
- Currency and prices
-
The currency and price basis are unclear.
Not reviewedUnresolved
- Period and unit
-
The period and unit of the monetary estimate are unclear.
Not reviewedUnresolved
- Realization
-
Whether the monetary value was realized is unclear.
Not reviewedUnresolved
- Recipients and payers
-
The recipients and payers are unclear.
Not reviewedUnresolved
- Gross or net
-
Whether the monetary figure includes costs is unclear.
Not reviewedUnresolved
- Included costs
-
The costs included in the estimate are unclear.
Not reviewedUnresolved
- Comparison
-
The financial comparison is not established.
Not reviewedUnresolved
- Method
-
The method used to calculate monetary value is unclear.
Not reviewedUnresolved
- Return and payback
-
The return on investment or payback period is not established.
Not reviewedUnresolved
People and places
- Participants
-
Developer: Mainstay (formerly AdmitHub). Operator: Georgia State University (Political Science and Economics departments).
Awaiting review
- Jurisdictions
-
Benefit: Atlanta, Georgia,
United States. Developer: United States. Operator: United States.Awaiting review
- Ownership and control
-
Unresolved
Not reviewedUnresolved
- Dependencies
-
Unresolved
Not reviewedUnresolved
- Cross-border connections
-
Unresolved
Not reviewedUnresolved
- Geopolitical implications
-
The geopolitical implications are unclear.
Not reviewedUnresolved
Follow-up
- Conditions for replication
-
Students must read text messages on their phones. Course staff must keep due dates and the knowledge base current.
Awaiting review
- Replication attempts
-
Unresolved
Not reviewedUnresolved
- Later findings
-
Unresolved
Not reviewedUnresolved
- Availability
-
Unresolved
Not reviewedUnresolved
- Institutional decision
-
A relevant institutional decision has not been established.
Not reviewedUnresolved
- Grounds
-
The grounds for an institutional decision are unclear.
Not reviewedUnresolved
- Action and result
-
The result of an institutional decision is not established.
Not reviewedUnresolved
- Success criterion
-
The criterion for a successful institutional decision is unclear.
Not reviewedUnresolved
Evidence
- Source numbers
-
AIBD-0617-source-0001–0003
Awaiting review
- Citations
-
Numbered references below.
Awaiting review
- Language
-
Unresolved
Not reviewedUnresolved
- Source relationships
-
Independent
Awaiting review
- Original evidence
-
Unresolved
Not reviewedUnresolved
- Access and versions
-
Unresolved
Not reviewedUnresolved
- Supported findings
-
Unresolved
Not reviewedUnresolved
- Evidence links
-
Unresolved
Not reviewedUnresolved
- Evidence strength
-
Independently studied. Sources: one independent peer-reviewed study and two independent reports. The cited evidence includes no finding by a regulator or court. [1] [2] [3]
- Missing evidence
-
Unresolved
Not reviewedUnresolved
Grades
- Assessed benefits
-
People.
Awaiting review
- Depth
-
People: 2.
Awaiting review
- Reasons
-
People: Students assigned to the chatbot were four percentage points more likely to earn an A or B in the course.
Awaiting review
- Observed or intended
-
Unresolved
Not reviewedUnresolved
- Rubric
-
Unresolved
Not reviewedUnresolved
- Assessor and date
-
Unresolved
Not reviewedUnresolved
- Alternative assessments
-
Unresolved
Not reviewedUnresolved
- Fill
-
Independently studied
Awaiting review
Entry details
- Kind
-
The case describes an evaluation or a study. The case is a pair of preregistered student-randomized trials. [2]
Fully reviewed: supporting details for Kind
Review date: 29 October 2026.
Sources: [2]
- Related entries
-
Both records concern an AI chatbot that an educational institution deployed to give its students academic support, and students are the stakeholders in each.
Fully reviewed: supporting details for Related entries
Compared with the incidents in the AI Incident Database snapshot of 26 October 2026; a second reading confirmed that the incident concerns the same kind of system and stakeholder.
Sources: incidentdatabase.ai
- Field
-
Education.
Awaiting review
- Purpose
-
The chatbot was built to send course reminders and answer students' questions in large online courses. Better course performance was that purpose.
Awaiting review
- Dates
-
Observed: 2023. Updated: 29 October 2026.
Awaiting review
- Stage
-
The system was used in a pilot. The chatbot ran in selected course sections during randomized trials over four semesters. [2]
Fully reviewed: supporting details for Stage
Review date: 29 October 2026.
Sources: [2]
- Version
-
SHA-256: 36f93bdb719a…
Awaiting review: supporting details for Version
SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.
References
- Other identifiers
-
AI Incident Database: 793.
Awaiting review
- Category mappings
-
Unresolved
Not reviewedUnresolved
- Citable summary
-
Unresolved
Not reviewedUnresolved
- Revisions
-
2 revisions. Latest: 29 October 2026.
Awaiting review
Sources
-
Peer-reviewed study Independent author
Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. Economics of Education Review. 2026;115:102816. doi:10.1016/j.econedurev.2026.102816
-
Report Independent author
Meyer KE, Page LC, Mata C, Smith E, Walsh BT, Fifield CL, Tyson M, Eremionkhale AE, Evans M, Frost S, Jung EE. Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. NBER Working Paper 35397, July 2026; published in Economics of Education Review vol. 115 (2026), doi:10.1016/j.econedurev.2026.102816.
We find the chatbot messaging increased students' final grades and engagement with academic supports, such as tutoring.
-
Report Independent author
Meyer KE, Page LC, et al. NBER Working Paper 35397, full text (results and sample).
the academic chatbot significantly shifted students' final grades, increasing the likelihood that students earned an A or B by four percentage points.