Lexi Local captions meet Canada's live accuracy standard in Lethbridge

Sources

1. Peer-reviewed study, by authors with no tie to the system.

   Romero-Fresco P, Fresno N. The accuracy of automatic and human live captions in English. Linguistica Antverpiensia, New Series: Themes in Translation Studies 22 (2023): 114-133, section 3.4 (case study 4: live TV captions in Canada). Published 13 December 2023. doi:10.52034/lans-tts.v22i.774.

   for the first time, broadcasters such as Jim Pattison Broadcast Group and The Miracle Channel Associations consistently reach accuracy rates above the required threshold in 2020-2021. Impressively, of the 55 samples analysed by these two broadcasters in 2020-2021, 52 reached 98%.

   In 2019-2020, their cloud-based version of the automatic captioning software EEG reached the 98% accuracy rate in 8/31 samples and 10/31 scored below 97.5%. For the 2020-2021 round, they decided to use the local server of the software (called Lexi Local) and to have a dedicated member of the staff compile glossaries with their phonetic spellings to improve the accuracy of the captions. This enabled them to reach the NER threshold consistently, with only 1/24 samples below par but pretty close to the minimum accuracy rate (97.92%).

   They are also more accurate than automatic captions, which feature more samples below 98% accuracy and fewer samples exceeding a 99% accuracy rate (60% of the automatic samples did not reach the minimum threshold and only 1% featured an excellent accuracy).

2. Evaluation report, by an evaluator that the makers or users commissioned.

   The Miracle Channel Association (CJIL-DT). NER98 Accuracy Standards Summary for Miracle Channel (CJIL-DT) 2020-21 and Live-Captioning NER Score Report for 2020-2021, with scores by a certified NER evaluator that the broadcaster engaged. Annual report to the CRTC under Broadcasting Regulatory Policy CRTC 2019-308. Archived copy of https://crtc.gc.ca/eng/BCASTING/ann_rep/miracle_cc2021.pdf.

   We are maintaining consistent NER98 accuracy through the use of an on-premise speech-to-text service called Lexi Local, from EEG Enterprises.

3. Regulatory document, by authors with no tie to the system. Context or contrary evidence.

   Canadian Radio-television and Telecommunications Commission. Broadcasting Regulatory Policy CRTC 2019-308: English-language closed captioning mandatory quality standards relating to the accuracy rate for live television programming. Ottawa, 30 August 2019. Archived copy of https://crtc.gc.ca/eng/archive/2019/2019-308.htm. Appendix 1, Quality Standard 3.

   Beginning 1 March 2020, broadcasters must reach an accuracy rate of at least 98 for English-language closed captioning of live television programming

   Further, E-ILTS acknowledged that voice recognition software, which is sometimes used by independent television stations and allows for costs savings, is improving year by year, but noted that based on preliminary testing, voice recognition-based captioning could have difficulty meeting the proposed NER 98 accuracy standard in the short term.

   Jim Pattison Broadcast Group Limited Partnership, operating in Kamloops and Prince George, British Columbia, and Medicine Hat, Alberta; Stingray Digital Group Inc. (now Stingray Group Inc.) in Lloydminster, Alberta; Thunder Bay Electronics Limited in Thunder Bay, Ontario; The Miracle Channel Association in Lethbridge, Alberta;

4. Report, by the organization that used it. Context or contrary evidence.

   The Miracle Channel Association (CJIL-DT). Summary of The Miracle Channel Association (CJIL-DT) Efforts to Comply with BRP 2019-308 and NER98 Accuracy Standards, with the Live-Captioning NER Score Report for 2019-2020. Annual report to the CRTC, 30 November 2020. Archived copy of https://crtc.gc.ca/eng/BCASTING/ann_rep/miracle_cc2020.pdf.

   The review process involved reviewing the video segment that was captioned and fixing all the mistakes that were presented in the exported log file and then uploading that back into the cloud service to help improve the AI and its dictionary in the hope of improving the accuracy moving forward.