{"id":"AIBD-0614","title":"Letrus automated essay scoring and feedback in 110 Espírito Santo public high schools raises ENEM essay scores by 0.094 standard deviations in a 2019 randomized trial","summary":"In a 2019 trial in 178 public high schools in Espírito Santo, Brazil, Letrus automated essay scoring and feedback raised ENEM essay scores by 0.094 standard deviations.","date_observed":"2019-11-01","date_basis":"event","date_precision":"month","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"Bruno Ferman, Lycia Lima and Flavio Riva of the São Paulo schools of economics and business administration at FGV ran a randomized trial from March to November 2019 with the Espírito Santo state education department and an education technology company. Exame reports that the company was Letrus and that J-PAL funded the trial. The trial covered 178 public high schools with about 19,000 senior students. In 55 schools, students wrote practice essays for the national college entrance exam (ENEM) on a platform whose automated writing evaluation system gave instant feedback and a predicted score. Another 55 schools received the same system plus grades and comments from human graders, and 68 schools served as the comparison group. Both versions raised essay scores by 0.094 standard deviations, pooled across the official 2019 ENEM essay and an independently administered essay of the same format. The human graders added no further gain. The authors report that teachers shifted time from routine grading toward individual discussions with students about their essays. The paper is a working paper that the authors posted in February 2021, and it does not name the company.","category":"general benefit","unexpected":false,"purpose":"The system was built to grade practice essays for the ENEM and give students feedback, so that teachers could spend less time on routine grading. Higher essay scores were that purpose.","system":"Letrus automated writing evaluation (AWE) platform, with machine-learning essay scoring and automated feedback (tested alone and with human graders)","model_version":null,"developer":"Letrus","deployer":"Espírito Santo State Education Department (SEDU/ES) with Letrus, in 110 public high schools","location":"Espírito Santo, Brazil","domain":"Education","segment":null,"benefit_type":"Improved Learning","taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"personal","depth":2,"reason":"Senior students in schools with either version of the system scored 0.094 standard deviations higher on the ENEM essay than students in comparison schools."},{"group":"organizational","depth":1,"reason":"The authors report that teachers moved time from routine grading toward individual discussions with students about essay quality."}],"ai_role":{"status":"assessed","statement":"Schools were randomly assigned to the automated system alone, to the system with human graders, or to no system. The fully automated arm produced the same gain as the arm with human graders, which isolates the contribution of the automated scoring and feedback within the program. The gain also runs through changes in what teachers did with the time saved."},"evidence":[{"kind":"preprint","citation":"Ferman B, Lima L, Riva F. Artificial Intelligence, Teacher Tasks and Individualized Pedagogy. SocArXiv working paper, current draft 15 February 2021. doi:10.31235/osf.io/qw249","title":"Artificial Intelligence, Teacher Tasks and Individualized Pedagogy","url":"https://osf.io/qw249/download","date":"2021-02-17","written_by":"independent","quote":"Both technologies significantly improved students' essay scores in a large college admission exam"},{"kind":"report","citation":"J-PAL. The Impact of Artificial Intelligence on Learning in Brazil. Evaluation summary of AEARCTR-0003729 (researchers Ferman, Lima, Riva).","title":"The Impact of Artificial Intelligence on Learning in Brazil","url":"https://www.povertyactionlab.org/evaluation/impact-artificial-intelligence-learning-brazil","date":null,"written_by":"independent","quote":"In schools where AI technology was introduced, teachers were able to provide more frequent individualized feedback to students, and students' essay scores improved as a result."},{"kind":"news report","citation":"Balago R. Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação. Exame, 13 June 2024.","title":"Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação","url":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/","date":"2024-06-13","written_by":"independent","quote":"Um primeiro ECR financiado pelo laboratório J-PAL do MIT e conduzido pela FGV-SP em 2019 com estudantes do Espírito Santo"}],"replication":{"produced_by":["Students wrote practice essays for the ENEM on a platform that scored them at once and flagged errors.","Teachers used the time saved on grading for individual feedback.","The state gave the schools laptops before the program began."],"must_hold":["Schools need working computers and an internet connection.","Teachers must assign the practice essays and discuss the results with students."],"echoes":[{"date":"2024-06-13","setting":"A planned three-year randomized trial of Letrus with about 40,000 high school students in Ceará, Brazil, reported by Exame","model_version":null,"outcome":"unknown","source":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/"}]},"paired_incident":null,"headline":"*Letrus* essay feedback lifts Brazilian students' exam essay scores","metric":"raised essay scores by 0.094 standard deviations","comparator":{"against":"Students in 68 comparison schools without the platform","before":"Comparison schools' ENEM essay scores (the reference level)","after":"0.094 standard deviations higher in schools with either version of the system","basis":"measured","quote":"the enhanced and the pure AWE ed techs had almost identical effects on the full ENEM essay score, at 0.094σ","url":"https://osf.io/qw249/download"},"later":{"status":"scaled","date":null,"note":"Letrus states that its program is public policy in the public school systems of Mato Grosso, Goiás and Espírito Santo.","quote":"A Letrus está presente na rede pública de três estados brasileiros como política pública: Mato Grosso, Goiás e Espírito Santo.","url":"https://www.letrus.com/"},"value":null,"developer_country":"Brazil","deployer_country":"Brazil","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://osf.io/qw249/download","editor":"AIBD"}],"confidence":"studied","shade":"high","claim_status":"Independently studied","claim_basis":"Sources: one independent report, one independent news report and one independent preprint. The cited evidence includes no finding by a regulator or court.","case_jurisdictions":["Brazil"],"profile":{"case_id":"AIBD-0614","catalog_version":"1.0.0","source_fingerprint":"e507611f0ad8ebd175cb751d0d0df42039e4752bb1bdc90855c155de4070df02","collection_refs":[{"collection_id":"aibd","version":"2026-10-29"}],"source_ids":["AIBD-0614-source-0001","AIBD-0614-source-0002","AIBD-0614-source-0003"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0614-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":[],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"partial","record_ids":["AIBD-0614-evaluation-1"]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Letrus","deployer":"Espírito Santo State Education Department (SEDU/ES) with Letrus, in 110 public high schools"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"Schools were randomly assigned to the automated system alone, to the system with human graders, or to no system. The fully automated arm produced the same gain as the arm with human graders, which isolates the contribution of the automated scoring and feedback within the program. The gain also runs through changes in what teachers did with the time saved."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"general benefit","ai_role":{"status":"assessed","statement":"Schools were randomly assigned to the automated system alone, to the system with human graders, or to no system. The fully automated arm produced the same gain as the arm with human graders, which isolates the contribution of the automated scoring and feedback within the program. The gain also runs through changes in what teachers did with the time saved."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Senior students in schools with either version of the system scored 0.094 standard deviations higher on the ENEM essay than students in comparison schools."},{"group":"organizational","depth":1,"reason":"The authors report that teachers moved time from routine grading toward individual discussions with students about essay quality."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"studied","claim_status":"Independently studied","basis":"Sources: one independent report, one independent news report and one independent preprint. The cited evidence includes no finding by a regulator or court.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Senior students in schools with either version of the system scored 0.094 standard deviations higher on the ENEM essay than students in comparison schools."},{"group":"organizational","depth":1,"reason":"The authors report that teachers moved time from routine grading toward individual discussions with students about essay quality."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Senior students in schools with either version of the system scored 0.094 standard deviations higher on the ENEM essay than students in comparison schools."},{"group":"organizational","depth":1,"reason":"The authors report that teachers moved time from routine grading toward individual discussions with students about essay quality."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"value_present","value":{"kind":"structured","data":{"classification":"evaluation or study","answer":"The case describes an evaluation or a study. The case is a school-randomized trial by academic researchers.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The case describes an evaluation or a study. The case is a school-randomized trial by academic researchers.","inferred":false}},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Compared with the incidents in the AI Incident Database snapshot of 26 October 2026.","record_fields":["paired_incident"],"source_refs":[]},"reason":"No incident in the AI Incident Database snapshot of 26 October 2026 concerns the same kind of system harming the same kind of stakeholder.","review_state":"reviewed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Independently studied. Sources: one independent report, one independent news report and one independent preprint. The cited evidence includes no finding by a regulator or court.","evidence_word":"studied","claim_status":"Independently studied"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001","AIBD-0614-source-0002","AIBD-0614-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Independently studied. Sources: one independent report, one independent news report and one independent preprint. The cited evidence includes no finding by a regulator or court.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"pilot","answer":"The system was used in a pilot. The system ran in 110 public schools for one school year as part of a randomized evaluation before any statewide rollout.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was used in a pilot. The system ran in 110 public schools for one school year as part of a randomized evaluation before any statewide rollout.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"value_present","value":{"kind":"text","text":"The automated-only arm matched the arm with human graders, so the gain does not depend on human grading."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The automated-only arm matched the arm with human graders, so the gain does not depend on human grading.","inferred":false}},"evaluations[].comparator":{"state":"value_present","value":{"kind":"text","text":"Comparison schools without the platform."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Comparison schools without the platform.","inferred":false}},"evaluations[].design":{"state":"value_present","value":{"kind":"text","text":"Randomized."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].effect":{"state":"value_present","value":{"kind":"text","text":"0.094 standard deviations for both the automated-only and the automated-plus-human-grader versions."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"0.094 standard deviations for both the automated-only and the automated-plus-human-grader versions.","inferred":false}},"evaluations[].exposure":{"state":"value_present","value":{"kind":"text","text":"Five practice essays over the 2019 school year."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Five practice essays over the 2019 school year.","inferred":false}},"evaluations[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0614-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].measurement_method":{"state":"value_present","value":{"kind":"text","text":"ENEM essay scores from the official 2019 exam and an independently administered essay with the same structure, graded on the ENEM's five competencies."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"ENEM essay scores from the official 2019 exam and an independently administered essay with the same structure, graded on the ENEM's five competencies.","inferred":false}},"evaluations[].metric":{"state":"value_present","value":{"kind":"text","text":"Full ENEM essay score in standard deviations."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Full ENEM essay score in standard deviations.","inferred":false}},"evaluations[].reproduction_materials":{"state":"value_present","value":{"kind":"text","text":"AEA RCT Registry AEARCTR-0003729."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"AEA RCT Registry AEARCTR-0003729.","inferred":false}},"evaluations[].setting_sample":{"state":"value_present","value":{"kind":"text","text":"178 public high schools in Espírito Santo, Brazil, with about 19,000 senior students; 55 schools per treatment arm and 68 comparison schools."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"178 public high schools in Espírito Santo, Brazil, with about 19,000 senior students; 55 schools per treatment arm and 68 comparison schools.","inferred":false}},"evaluations[].uncertainty_limitations":{"state":"value_present","value":{"kind":"text","text":"Working paper; pooled outcome across two essays; the company is not named in the paper."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0614-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Working paper; pooled outcome across two essays; the company is not named in the paper.","inferred":false}},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"later":{"status":"scaled","note":"Letrus states that its program is public policy in the public school systems of Mato Grosso, Goiás and Espírito Santo.","quote":"A Letrus está presente na rede pública de três estados brasileiros como política pública: Mato Grosso, Goiás e Espírito Santo.","url":"https://www.letrus.com/"}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["later"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0614"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Bruno Ferman, Lycia Lima and Flavio Riva of the São Paulo schools of economics and business administration at FGV ran a randomized trial from March to November 2019 with the Espírito Santo state education department and an education technology company. Exame reports that the company was Letrus and that J-PAL funded the trial. The trial covered 178 public high schools with about 19,000 senior students. In 55 schools, students wrote practice essays for the national college entrance exam (ENEM) on a platform whose automated writing evaluation system gave instant feedback and a predicted score. Another 55 schools received the same system plus grades and comments from human graders, and 68 schools served as the comparison group. Both versions raised essay scores by 0.094 standard deviations, pooled across the official 2019 ENEM essay and an independently administered essay of the same format. The human graders added no further gain. The authors report that teachers shifted time from routine grading toward individual discussions with students about their essays. The paper is a working paper that the authors posted in February 2021, and it does not name the company.","benefit_type":"Improved Learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Senior students in schools with either version of the system scored 0.094 standard deviations higher on the ENEM essay than students in comparison schools."},{"group":"organizational","depth":1,"reason":"The authors report that teachers moved time from routine grading toward individual discussions with students about essay quality."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"The system was built to grade practice essays for the ENEM and give students feedback, so that teachers could spend less time on routine grading. Higher essay scores were that purpose."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Espírito Santo, Brazil","developer_country":"Brazil","deployer_country":"Brazil"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"The system was built to grade practice essays for the ENEM and give students feedback, so that teachers could spend less time on routine grading. Higher essay scores were that purpose."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2019-11-01","date_basis":"event","date_precision":"month","revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://osf.io/qw249/download","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Students wrote practice essays for the ENEM on a platform that scored them at once and flagged errors.","Teachers used the time saved on grading for individual feedback.","The state gave the schools laptops before the program began."],"must_hold":["Schools need working computers and an internet connection.","Teachers must assign the practice essays and discuss the results with students."],"echoes":[{"date":"2024-06-13","setting":"A planned three-year randomized trial of Letrus with about 40,000 high school students in Ceará, Brazil, reported by Exame","outcome":"unknown","source":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Students wrote practice essays for the ENEM on a platform that scored them at once and flagged errors.","Teachers used the time saved on grading for individual feedback.","The state gave the schools laptops before the program began."],"must_hold":["Schools need working computers and an internet connection.","Teachers must assign the practice essays and discuss the results with students."],"echoes":[{"date":"2024-06-13","setting":"A planned three-year randomized trial of Letrus with about 40,000 high school students in Ceará, Brazil, reported by Exame","outcome":"unknown","source":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://osf.io/qw249/download","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Education"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"preprint","citation":"Ferman B, Lima L, Riva F. Artificial Intelligence, Teacher Tasks and Individualized Pedagogy. SocArXiv working paper, current draft 15 February 2021. doi:10.31235/osf.io/qw249","title":"Artificial Intelligence, Teacher Tasks and Individualized Pedagogy","url":"https://osf.io/qw249/download","date":"2021-02-17","written_by":"independent","quote":"Both technologies significantly improved students' essay scores in a large college admission exam"},{"kind":"report","citation":"J-PAL. The Impact of Artificial Intelligence on Learning in Brazil. Evaluation summary of AEARCTR-0003729 (researchers Ferman, Lima, Riva).","title":"The Impact of Artificial Intelligence on Learning in Brazil","url":"https://www.povertyactionlab.org/evaluation/impact-artificial-intelligence-learning-brazil","written_by":"independent","quote":"In schools where AI technology was introduced, teachers were able to provide more frequent individualized feedback to students, and students' essay scores improved as a result."},{"kind":"news report","citation":"Balago R. Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação. Exame, 13 June 2024.","title":"Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação","url":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/","date":"2024-06-13","written_by":"independent","quote":"Um primeiro ECR financiado pelo laboratório J-PAL do MIT e conduzido pela FGV-SP em 2019 com estudantes do Espírito Santo"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0614-source-0001","AIBD-0614-source-0002","AIBD-0614-source-0003"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"preprint","citation":"Ferman B, Lima L, Riva F. Artificial Intelligence, Teacher Tasks and Individualized Pedagogy. SocArXiv working paper, current draft 15 February 2021. doi:10.31235/osf.io/qw249","title":"Artificial Intelligence, Teacher Tasks and Individualized Pedagogy","url":"https://osf.io/qw249/download","date":"2021-02-17","written_by":"independent","quote":"Both technologies significantly improved students' essay scores in a large college admission exam"},{"kind":"report","citation":"J-PAL. The Impact of Artificial Intelligence on Learning in Brazil. Evaluation summary of AEARCTR-0003729 (researchers Ferman, Lima, Riva).","title":"The Impact of Artificial Intelligence on Learning in Brazil","url":"https://www.povertyactionlab.org/evaluation/impact-artificial-intelligence-learning-brazil","written_by":"independent","quote":"In schools where AI technology was introduced, teachers were able to provide more frequent individualized feedback to students, and students' essay scores improved as a result."},{"kind":"news report","citation":"Balago R. Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação. Exame, 13 June 2024.","title":"Letrus recebe R$ 5 milhões do governo dos EUA para estudo sobre IA na educação","url":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/","date":"2024-06-13","written_by":"independent","quote":"Um primeiro ECR financiado pelo laboratório J-PAL do MIT e conduzido pela FGV-SP em 2019 com estudantes do Espírito Santo"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In a 2019 trial in 178 public high schools in Espírito Santo, Brazil, Letrus automated essay scoring and feedback raised ENEM essay scores by 0.094 standard deviations.","account":"Bruno Ferman, Lycia Lima and Flavio Riva of the São Paulo schools of economics and business administration at FGV ran a randomized trial from March to November 2019 with the Espírito Santo state education department and an education technology company. Exame reports that the company was Letrus and that J-PAL funded the trial. The trial covered 178 public high schools with about 19,000 senior students. In 55 schools, students wrote practice essays for the national college entrance exam (ENEM) on a platform whose automated writing evaluation system gave instant feedback and a predicted score. Another 55 schools received the same system plus grades and comments from human graders, and 68 schools served as the comparison group. Both versions raised essay scores by 0.094 standard deviations, pooled across the official 2019 ENEM essay and an independently administered essay of the same format. The human graders added no further gain. The authors report that teachers shifted time from routine grading toward individual discussions with students about their essays. The paper is a working paper that the authors posted in February 2021, and it does not name the company."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Letrus automated writing evaluation (AWE) platform, with machine-learning essay scoring and automated feedback (tested alone and with human graders)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Letrus automated essay scoring and feedback in 110 Espírito Santo public high schools raises ENEM essay scores by 0.094 standard deviations in a 2019 randomized trial","headline":"*Letrus* essay feedback lifts Brazilian students' exam essay scores"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["14cab0e38372e23d7df7caf70c82d3b86ac9f304808c5c079be18209edea7859"],"role":"SHA-256 of the complete public case and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Bruno Ferman, Lycia Lima and Flavio Riva of the São Paulo schools of economics and business administration at FGV ran a randomized trial from March to November 2019 with the Espírito Santo state education department and an education technology company. Exame reports that the company was Letrus and that J-PAL funded the trial. The trial covered 178 public high schools with about 19,000 senior students. In 55 schools, students wrote practice essays for the national college entrance exam (ENEM) on a platform whose automated writing evaluation system gave instant feedback and a predicted score. Another 55 schools received the same system plus grades and comments from human graders, and 68 schools served as the comparison group. Both versions raised essay scores by 0.094 standard deviations, pooled across the official 2019 ENEM essay and an independently administered essay of the same format. The human graders added no further gain. The authors report that teachers shifted time from routine grading toward individual discussions with students about their essays. The paper is a working paper that the authors posted in February 2021, and it does not name the company.","replication":{"produced_by":["Students wrote practice essays for the ENEM on a platform that scored them at once and flagged errors.","Teachers used the time saved on grading for individual feedback.","The state gave the schools laptops before the program began."],"must_hold":["Schools need working computers and an internet connection.","Teachers must assign the practice essays and discuss the results with students."],"echoes":[{"date":"2024-06-13","setting":"A planned three-year randomized trial of Letrus with about 40,000 high school students in Ceará, Brazil, reported by Exame","outcome":"unknown","source":"https://exame.com/brasil/letrus-fara-estudo-com-40-mil-alunos-para-avaliar-impacto-da-ia-para-melhorar-escrita/"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":43,"reviewed":15,"reviewed_complete":14,"reviewed_partial":1,"record_mapped":29,"review_not_assessed":47,"not_assessed":0,"complete":false,"by_state":{"value_present":43,"not_reported":0,"unresolved":48,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"be39776b6c329851eab6bef57c1abc0082d9b4bce381ba653c75c0388d9e8169"}