{"id":"AIBD-1010","title":"ANAO finds that 64.7% of the classifications made by IP Australia's Patent Auto Classifier were correct on average as of November 2025","summary":"The ANAO analyzed IP Australia's data and found that 64.7% of the Patent Auto Classifier's classifications were correct as of November 2025. Another 27.5% were missing and 7.8% incorrect.","date_observed":"2026-06-29","date_basis":"source publication","date_precision":"day","benefit_status":"mixed","source_collection":null,"source_entry":null,"account":"IP Australia has used the Patent Auto Classifier (PAC) since 2018. The Australian National Audit Office (ANAO) describes it as an AI system that uses natural language processing to read the text of each patent application. PAC scores how relevant each International Patent Classification symbol is and sends the application to the examination queue with the highest score. Examiners redirect the applications that it sends to the wrong queue. In a performance audit published on 29 June 2026, the ANAO analyzed IP Australia's data on PAC's classifications. It found that, as of November 2025, 64.7% of PAC's classifications were correct on average, 27.5% were missing and 7.8% were incorrect. The ANAO found that accuracy had stayed consistent from December 2022 to November 2025, and IP Australia retrains the model every year. The ANAO also found that IP Australia had not linked its AI tools to changes in the efficiency of patent examination. The audit reports no measured time saving from PAC and does not define a missing classification.","category":"aggregate result","unexpected":false,"purpose":"IP Australia built the Patent Auto Classifier to classify each incoming patent application by type of invention and send it to the right examination queue.","system":"Patent Auto Classifier (PAC)","model_version":null,"developer":"IP Australia (built in-house and retrained every year)","deployer":"IP Australia (patent examination)","location":"Australia (national)","domain":"Intellectual property administration (patent examination)","segment":null,"benefit_type":"Repetitive Task Automation","work_activities":["4.A.2.a.2","4.A.1.b.1"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":1,"reason":"PAC routes incoming applications automatically, and 64.7% of its classifications were correct, though the time that it saves was not measured."}],"ai_role":{"status":"unassessed","statement":"The ANAO measured the correctness of PAC's own classifications, so the figures concern the system itself. It did not compare classification with and without PAC or measure the time that PAC saves. The ANAO describes PAC as an AI system that uses natural language processing and is retrained every year."},"evidence":[{"kind":"audit report","citation":"Australian National Audit Office. Artificial Intelligence Use in IP Australia. Auditor-General Report No. 43 of 2025-26, published 29 June 2026. Paragraphs 3.25 and 3.26 and Figure 3.3 (archived copy of the report page).","title":"Artificial Intelligence Use in IP Australia","url":"https://web.archive.org/web/20260712121955/https://www.anao.gov.au/work/performance-audit/artificial-intelligence-use-ip-australia","date":"2026-06-29","written_by":"independent","quote":"The ANAO analysed this data and as at November 2025, the model's classifications are on average 64.7 per cent correct, with 27.5 per cent of its classifications missing and 7.8 per cent of its classifications being incorrect"}],"replication":{"produced_by":["PAC read the text of each application and scored how relevant each International Patent Classification symbol was.","It sent each application to the examination queue with the highest score, and examiners redirected the applications that it misrouted."],"must_hold":["IP Australia retrains the model every year so that its performance does not drift.","Examiners must catch and redirect the applications that PAC sends to the wrong queue."],"echoes":[]},"paired_incident":null,"headline":"*Patent Auto Classifier* gets about two in three classifications right","metric":"64.7% of the Patent Auto Classifier's classifications were correct","comparator":null,"later":null,"value":null,"developer_country":"Australia","deployer_country":"Australia","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the Australian National Audit Office.","source":"https://web.archive.org/web/20260712121955/https://www.anao.gov.au/work/performance-audit/artificial-intelligence-use-ip-australia","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by an auditor","claim_basis":"Sources: one independent audit report. The cited evidence includes no independent study or report.","case_jurisdictions":["Australia"],"profile":{"case_id":"AIBD-1010","catalog_version":"1.0.0","source_fingerprint":"6d1ce9f79f5fb246d7def06e8f62b88fe5bf0466440595c54316bb1c6dd61efc","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-1010-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"IP Australia (built in-house and retrained every year)","deployer":"IP Australia (patent examination)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"unassessed","statement":"The ANAO measured the correctness of PAC's own classifications, so the figures concern the system itself. It did not compare classification with and without PAC or measure the time that PAC saves. The ANAO describes PAC as an AI system that uses natural language processing and is retrained every year."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"PAC routes incoming applications automatically, and 64.7% of its classifications were correct, though the time that it saves was not measured."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by an auditor","basis":"Sources: one independent audit report. The cited evidence includes no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"PAC routes incoming applications automatically, and 64.7% of its classifications were correct, though the time that it saves was not measured."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"PAC routes incoming applications automatically, and 64.7% of its classifications were correct, though the time that it saves was not measured."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by an auditor. Sources: one independent audit report. The cited evidence includes no independent study or report.","evidence_word":"found","claim_status":"Found by an auditor"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-1010-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by an auditor. Sources: one independent audit report. The cited evidence includes no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"routine use","answer":"The system was in routine use. IP Australia has classified incoming patent applications with PAC since 2018.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-1010-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was in routine use. IP Australia has classified incoming patent applications with PAC since 2018.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-1010"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Repetitive Task Automation"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"IP Australia has used the Patent Auto Classifier (PAC) since 2018. The Australian National Audit Office (ANAO) describes it as an AI system that uses natural language processing to read the text of each patent application. PAC scores how relevant each International Patent Classification symbol is and sends the application to the examination queue with the highest score. Examiners redirect the applications that it sends to the wrong queue. In a performance audit published on 29 June 2026, the ANAO analyzed IP Australia's data on PAC's classifications. It found that, as of November 2025, 64.7% of PAC's classifications were correct on average, 27.5% were missing and 7.8% were incorrect. The ANAO found that accuracy had stayed consistent from December 2022 to November 2025, and IP Australia retrains the model every year. The ANAO also found that IP Australia had not linked its AI tools to changes in the efficiency of patent examination. The audit reports no measured time saving from PAC and does not define a missing classification.","benefit_type":"Repetitive Task Automation"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"PAC routes incoming applications automatically, and 64.7% of its classifications were correct, though the time that it saves was not measured."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"mixed"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"IP Australia built the Patent Auto Classifier to classify each incoming patent application by type of invention and send it to the right examination queue."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Australia (national)","developer_country":"Australia","deployer_country":"Australia"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"IP Australia built the Patent Auto Classifier to classify each incoming patent application by type of invention and send it to the right examination queue."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2026-06-29","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the Australian National Audit Office.","source":"https://web.archive.org/web/20260712121955/https://www.anao.gov.au/work/performance-audit/artificial-intelligence-use-ip-australia","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["PAC read the text of each application and scored how relevant each International Patent Classification symbol was.","It sent each application to the examination queue with the highest score, and examiners redirected the applications that it misrouted."],"must_hold":["IP Australia retrains the model every year so that its performance does not drift.","Examiners must catch and redirect the applications that PAC sends to the wrong queue."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the Australian National Audit Office.","source":"https://web.archive.org/web/20260712121955/https://www.anao.gov.au/work/performance-audit/artificial-intelligence-use-ip-australia","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Intellectual property administration (patent examination)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"audit report","citation":"Australian National Audit Office. Artificial Intelligence Use in IP Australia. Auditor-General Report No. 43 of 2025-26, published 29 June 2026. Paragraphs 3.25 and 3.26 and Figure 3.3 (archived copy of the report page).","title":"Artificial Intelligence Use in IP Australia","url":"https://web.archive.org/web/20260712121955/https://www.anao.gov.au/work/performance-audit/artificial-intelligence-use-ip-australia","date":"2026-06-29","written_by":"independent","quote":"The ANAO analysed this data and as at November 2025, the model's classifications are on average 64.7 per cent correct, with 27.5 per cent of its classifications missing and 7.8 per cent of its classifications being incorrect"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-1010-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"audit report","citation":"Australian National Audit Office. Artificial Intelligence Use in IP Australia. Auditor-General Report No. 43 of 2025-26, published 29 June 2026. Paragraphs 3.25 and 3.26 and Figure 3.3 (archived copy of the report page).","title":"Artificial Intelligence Use in IP Australia","url":"https://web.archive.org/web/20260712121955/https://www.anao.gov.au/work/performance-audit/artificial-intelligence-use-ip-australia","date":"2026-06-29","written_by":"independent","quote":"The ANAO analysed this data and as at November 2025, the model's classifications are on average 64.7 per cent correct, with 27.5 per cent of its classifications missing and 7.8 per cent of its classifications being incorrect"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"The ANAO analyzed IP Australia's data and found that 64.7% of the Patent Auto Classifier's classifications were correct as of November 2025. Another 27.5% were missing and 7.8% incorrect.","account":"IP Australia has used the Patent Auto Classifier (PAC) since 2018. The Australian National Audit Office (ANAO) describes it as an AI system that uses natural language processing to read the text of each patent application. PAC scores how relevant each International Patent Classification symbol is and sends the application to the examination queue with the highest score. Examiners redirect the applications that it sends to the wrong queue. In a performance audit published on 29 June 2026, the ANAO analyzed IP Australia's data on PAC's classifications. It found that, as of November 2025, 64.7% of PAC's classifications were correct on average, 27.5% were missing and 7.8% were incorrect. The ANAO found that accuracy had stayed consistent from December 2022 to November 2025, and IP Australia retrains the model every year. The ANAO also found that IP Australia had not linked its AI tools to changes in the efficiency of patent examination. The audit reports no measured time saving from PAC and does not define a missing classification."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Patent Auto Classifier (PAC)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"ANAO finds that 64.7% of the classifications made by IP Australia's Patent Auto Classifier were correct on average as of November 2025","headline":"*Patent Auto Classifier* gets about two in three classifications right"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["3a10e713a2382a966bc883707113d6bffb4234834e8c47c39c266c1bc5a64fac"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"IP Australia has used the Patent Auto Classifier (PAC) since 2018. The Australian National Audit Office (ANAO) describes it as an AI system that uses natural language processing to read the text of each patent application. PAC scores how relevant each International Patent Classification symbol is and sends the application to the examination queue with the highest score. Examiners redirect the applications that it sends to the wrong queue. In a performance audit published on 29 June 2026, the ANAO analyzed IP Australia's data on PAC's classifications. It found that, as of November 2025, 64.7% of PAC's classifications were correct on average, 27.5% were missing and 7.8% were incorrect. The ANAO found that accuracy had stayed consistent from December 2022 to November 2025, and IP Australia retrains the model every year. The ANAO also found that IP Australia had not linked its AI tools to changes in the efficiency of patent examination. The audit reports no measured time saving from PAC and does not define a missing classification.","replication":{"produced_by":["PAC read the text of each application and scored how relevant each International Patent Classification symbol was.","It sent each application to the examination queue with the highest score, and examiners redirected the applications that it misrouted."],"must_hold":["IP Australia retrains the model every year so that its performance does not drift.","Examiners must catch and redirect the applications that PAC sends to the wrong queue."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"effb56f2fc3053cfbba8bdf435d6a1be52ef3334827a3c315722ffbfe23d998f"}