Benefits
- Benefit
-
Raised essay scores by 0.094 standard deviations
Awaiting review
- Outcome identifiers
-
Unresolved
Not reviewedUnresolved
- Category
-
Improved Learning
Awaiting review
- Status
-
Reported benefit
Awaiting review
- Beneficiary groups
-
People. Organizations.
Awaiting review
- Beneficiaries
-
Unresolved
Not reviewedUnresolved
- Reach
-
Unresolved
Not reviewedUnresolved
- Distribution and access
-
Unresolved
Not reviewedUnresolved
- Duration
-
Unresolved
Not reviewedUnresolved
- Expected?
-
Yes
Awaiting review
- Costs and harms
-
Unresolved
Not reviewedUnresolved
AI system
- System
-
Letrus automated writing evaluation (AWE) platform, with machine-learning essay scoring and automated feedback (tested alone and with human graders)
Awaiting review
- Model
-
This is an explicit unresolved assignment: the sources on record do not settle this question.
Not reviewedUnresolved
- Release
-
Unresolved
Not reviewedUnresolved
- Provider
-
Unresolved
Not reviewedUnresolved
- Role
-
Unresolved
Not reviewedUnresolved
- Configuration
-
Unresolved
Not reviewedUnresolved
- Inputs and outputs
-
Unresolved
Not reviewedUnresolved
- Autonomy
-
Unresolved
Not reviewedUnresolved
- Workflow
-
Students wrote practice essays for the ENEM on a platform that scored them at once and flagged errors. Teachers used the time saved on grading for individual feedback. The state gave the schools laptops before the program began.
Awaiting review
- Human oversight
-
Unresolved
Not reviewedUnresolved
- Relation to AI
-
Schools were randomly assigned to the automated system alone, to the system with human graders, or to no system. The fully automated arm produced the same gain as the arm with human graders, which isolates the contribution of the automated scoring and feedback within the program. The gain also runs through changes in what teachers did with the time saved.
Awaiting review
- AI contribution
-
Schools were randomly assigned to the automated system alone, to the system with human graders, or to no system. The fully automated arm produced the same gain as the arm with human graders, which isolates the contribution of the automated scoring and feedback within the program. The gain also runs through changes in what teachers did with the time saved.
Awaiting review
Measurement
- Studies
-
Randomized. [2]
Fully reviewed: supporting details for Studies
Review date: 29 October 2026.
Sources: [2]
- Study design
-
Randomized. [2]
Fully reviewed: supporting details for Study design
Review date: 29 October 2026.
Sources: [2]
- Measurement
-
ENEM essay scores from the official 2019 exam and an independently administered essay with the same structure, graded on the ENEM's five competencies. [2]
Fully reviewed: supporting details for Measurement
Review date: 29 October 2026.
Sources: [2]
- Setting and sample
-
178 public high schools in Espírito Santo, Brazil, with about 19,000 senior students; 55 schools per treatment arm and 68 comparison schools. [2]
Fully reviewed: supporting details for Setting and sample
Review date: 29 October 2026.
Sources: [2]
- Measures
-
Full ENEM essay score in standard deviations. [2]
Fully reviewed: supporting details for Measures
Review date: 29 October 2026.
Sources: [2]
- Comparison
-
Comparison schools without the platform. [2]
Fully reviewed: supporting details for Comparison
Review date: 29 October 2026.
Sources: [2]
- Results
-
0.094 standard deviations for both the automated-only and the automated-plus-human-grader versions. [2]
Fully reviewed: supporting details for Results
Review date: 29 October 2026.
Sources: [2]
- Additional value from AI
-
The automated-only arm matched the arm with human graders, so the gain does not depend on human grading. [2]
Fully reviewed: supporting details for Additional value from AI
Review date: 29 October 2026.
Sources: [2]
- Limitations
-
Working paper; pooled outcome across two essays; the company is not named in the paper. [2]
Fully reviewed: supporting details for Limitations
Review date: 29 October 2026.
Sources: [2]
Money
- Kind of value
-
The kind of monetary value is unclear.
Not reviewedUnresolved
- Amount
-
The monetary value is not established.
Not reviewedUnresolved
- Currency and prices
-
The currency and price basis are unclear.
Not reviewedUnresolved
- Period and unit
-
The period and unit of the monetary estimate are unclear.
Not reviewedUnresolved
- Realization
-
Whether the monetary value was realized is unclear.
Not reviewedUnresolved
- Recipients and payers
-
The recipients and payers are unclear.
Not reviewedUnresolved
- Gross or net
-
Whether the monetary figure includes costs is unclear.
Not reviewedUnresolved
- Included costs
-
The costs included in the estimate are unclear.
Not reviewedUnresolved
- Comparison
-
The financial comparison is not established.
Not reviewedUnresolved
- Method
-
The method used to calculate monetary value is unclear.
Not reviewedUnresolved
- Return and payback
-
The return on investment or payback period is not established.
Not reviewedUnresolved
People and places
- Participants
-
Developer: Letrus. Operator: Espírito Santo State Education Department (SEDU/ES) with Letrus, in 110 public high schools.
Awaiting review
- Jurisdictions
-
Benefit: Espírito Santo,
Brazil. Developer: Brazil. Operator: Brazil.Awaiting review
- Ownership and control
-
Unresolved
Not reviewedUnresolved
- Dependencies
-
Unresolved
Not reviewedUnresolved
- Cross-border connections
-
Unresolved
Not reviewedUnresolved
- Geopolitical implications
-
The geopolitical implications are unclear.
Not reviewedUnresolved
Follow-up
- Conditions for replication
-
Schools need working computers and an internet connection. Teachers must assign the practice essays and discuss the results with students.
Awaiting review
- Replication attempts
-
1 follow-up entry. Outcome unresolved: 1.
Awaiting review
- Later findings
-
Letrus states that its program is public policy in the public school systems of Mato Grosso, Goiás and Espírito Santo.
Awaiting review
- Availability
-
Unresolved
Not reviewedUnresolved
- Institutional decision
-
A relevant institutional decision has not been established.
Not reviewedUnresolved
- Grounds
-
The grounds for an institutional decision are unclear.
Not reviewedUnresolved
- Action and result
-
The result of an institutional decision is not established.
Not reviewedUnresolved
- Success criterion
-
The criterion for a successful institutional decision is unclear.
Not reviewedUnresolved
Evidence
- Source numbers
-
AIBD-0614-source-0001–0003
Awaiting review
- Citations
-
Numbered references below.
Awaiting review
- Language
-
Unresolved
Not reviewedUnresolved
- Source relationships
-
Independent
Awaiting review
- Original evidence
-
Unresolved
Not reviewedUnresolved
- Access and versions
-
Unresolved
Not reviewedUnresolved
- Supported findings
-
Unresolved
Not reviewedUnresolved
- Evidence links
-
Unresolved
Not reviewedUnresolved
- Evidence strength
-
Independently studied. Sources: one independent report, one independent news report and one independent preprint. The cited evidence includes no finding by a regulator or court. [1] [2] [3]
- Missing evidence
-
Unresolved
Not reviewedUnresolved
Grades
- Assessed benefits
-
People. Organizations.
Awaiting review
- Depth
-
People: 2. Organizations: 1.
Awaiting review
- Reasons
-
People: Senior students in schools with either version of the system scored 0.094 standard deviations higher on the ENEM essay than students in comparison schools. Organizations: The authors report that teachers moved time from routine grading toward individual discussions with students about essay quality.
Awaiting review
- Observed or intended
-
Unresolved
Not reviewedUnresolved
- Rubric
-
Unresolved
Not reviewedUnresolved
- Assessor and date
-
Unresolved
Not reviewedUnresolved
- Alternative assessments
-
Unresolved
Not reviewedUnresolved
- Fill
-
Independently studied
Awaiting review
Entry details
- Kind
-
The case describes an evaluation or a study. The case is a school-randomized trial by academic researchers. [2]
Fully reviewed: supporting details for Kind
Review date: 29 October 2026.
Sources: [2]
- Related entries
-
No incident in the AI Incident Database snapshot of 26 October 2026 concerns the same kind of system harming the same kind of stakeholder.
Partly reviewedUnresolved: supporting details for Related entries
Compared with the incidents in the AI Incident Database snapshot of 26 October 2026.
- Field
-
Education.
Awaiting review
- Purpose
-
The system was built to grade practice essays for the ENEM and give students feedback, so that teachers could spend less time on routine grading. Higher essay scores were that purpose.
Awaiting review
- Dates
-
Observed: November 2019. Updated: 29 October 2026.
Awaiting review
- Stage
-
The system was used in a pilot. The system ran in 110 public schools for one school year as part of a randomized evaluation before any statewide rollout. [2]
Fully reviewed: supporting details for Stage
Review date: 29 October 2026.
Sources: [2]
- Version
-
SHA-256: 14cab0e38372…
Awaiting review: supporting details for Version
SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.
References
- Other identifiers
-
Unresolved
Not reviewedUnresolved
- Category mappings
-
Unresolved
Not reviewedUnresolved
- Citable summary
-
Unresolved
Not reviewedUnresolved
- Revisions
-
1 revision. Latest: 29 October 2026.
Awaiting review
Sources
-
Report Independent author
J-PAL. The Impact of Artificial Intelligence on Learning in Brazil. Evaluation summary of AEARCTR-0003729 (researchers Ferman, Lima, Riva).
In schools where AI technology was introduced, teachers were able to provide more frequent individualized feedback to students, and students' essay scores improved as a result.
-
Preprint Independent author
Ferman B, Lima L, Riva F. Artificial Intelligence, Teacher Tasks and Individualized Pedagogy. SocArXiv working paper, current draft 15 February 2021. doi:10.31235/osf.io/qw249
Both technologies significantly improved students' essay scores in a large college admission exam
-
News report Independent author
Balago R. Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação. Exame, 13 June 2024.
Um primeiro ECR financiado pelo laboratório J-PAL do MIT e conduzido pela FGV-SP em 2019 com estudantes do Espírito Santo