Public AI Procurement Should Include a Language Rights Test
Governments are beginning to use conversational artificial intelligence as a primary point of access to public information. A person asks about a benefit, a permit or an administrative procedure and receives a fluent reply. The promise is compelling: services available beyond office hours, in accessible language and at potentially lower operating costs. But the design of such systems exposes a problem that public procurement often overlooks. A chatbot can be useful in one language and misleading in another.
A language rights test should be a standard part of any public-sector AI procurement that serves a multilingual population. The test would ask whether people can obtain accurate information in the languages they actually use, whether important distinctions survive translation and whether a human review path exists when the system fails. It would be more demanding than a vendor claim that a model “supports” dozens of languages.
Language support is not binary. A model may recognize a language yet struggle with regional terms, bureaucratic wording or the way a person describes a problem informally. In public services, those limitations can have consequences. Someone may ask a question about eligibility without knowing the legal terminology. If the chatbot misinterprets a local expression as an unrelated category, the user might leave with false confidence about their options. Research on multilingual AI systems has documented differences in performance across languages, raising concerns about whether users receive equally reliable information. Public authorities should therefore evaluate accuracy separately for each supported language rather than relying on an overall performance score.
The test should begin with the specific service. A system answering general library questions can tolerate different risks from one explaining legal deadlines or health services. Procurement documents should identify those risks explicitly and require vendors to demonstrate performance on representative tasks. Evaluations should record language-specific accuracy, the severity of errors and the proportion of questions requiring human assistance. Systems should be reassessed after significant model updates or changes to the underlying public-service rules. Sample questions should be developed with local speakers, public servants and independent evaluators, not drawn solely from standardized benchmark sets.
Next comes transparency. Public authorities should publish the scope of tested languages, the kinds of queries evaluated and the limitations discovered. They need not disclose private data or security-sensitive system details. Citizens should nonetheless be able to know whether an advertised service has been evaluated for their language or whether they are effectively participating in an experiment.
The human review path is equally important. Where an automated answer is consequential, the system should link to the controlling official source and provide a way to contact a qualified staff member. If an error is reported, the institution should be able to trace it, correct it and determine whether other users may have received similar misinformation. Accountability belongs to the public institution, not to an abstract model.
There are legitimate objections. Governments face budget constraints, and testing every linguistic variety to the same depth may be impossible. The answer should be prioritization, not indifference. High-stakes services and widely used languages merit early attention; smaller language communities should have a transparent route to identify risks and request evaluation. Procurement can require incremental coverage rather than an unsupported claim of universal language coverage.
There is also a danger of treating translation as the whole problem. Language access does not guarantee access for people with disabilities, low digital literacy or intermittent connectivity. A language rights test should sit beside accessibility standards, privacy safeguards and an option to use a non-digital service. The aim is public access in practice, not a multilingual interface that serves only the confident.
International principles already offer a useful frame. UNESCO’s guidance on AI in education and the OECD AI Principles emphasize human oversight, responsibility and inclusive approaches. Public procurement can turn such values into obligations that are checked before a service is launched. The key move is from broad aspiration to verifiable requirements.
Procurement officials can translate this principle into contract clauses. Vendors might be required to document evaluation datasets, support periodic independent testing and provide a clear process for investigating materially incorrect responses. The public agency should retain enough access to the system’s outputs and records to review complaints. Without such rights, an institution can become dependent on a vendor’s assurances while remaining responsible for decisions it cannot fully audit.
Performance should also be evaluated across time. A system that passed a language test before launch may change when a model is updated or the underlying government rules are revised. Contracts should require reassessment after significant changes and the publication of corrected information when an earlier response was consequentially wrong. The issue is not that every change is hazardous. It is that public accountability cannot be a one-time certification attached to a moving product.
Communities can help identify what ordinary benchmarks miss. Test participants may ask questions in informal registers, switch languages within a sentence or describe an administrative difficulty through family circumstances rather than official terms. These are realistic interactions, not edge cases to be discarded for convenience. Testing with them creates a clearer picture of the service the public will actually encounter.
For multilingual states, this is an issue of democratic fairness. Access to official information should not depend on whether a person happens to speak the language for which a vendor’s model performs best. Public AI can widen participation, but only if institutions are willing to test the boundaries of that promise. A government that buys a multilingual chatbot should be able to show what multilingual actually means.
(The opinion expressed in this article belong only to the author and do not necessarily reflect the views of The Convention).
Illustration: AI-generated for The Convention
