Greater visibility on the use of artificial intelligence and machine technology by government agencies is needed, according to the NSW Ombudsman.
In a report tabled in parliament this week, NSW Ombudsman Paul Miller warned agencies that the use of machine technology in ways that do not follow standards of lawfulness could lead to findings of maladministration or unlawful conduct.
“Public authorities exercise powers that impact virtually all aspects of an individual’s life,” the report said.
“The inherently ‘public’ nature of such functions…and the specific focus of some government service provision on vulnerable groups means that the government’s use of machine technology will necessarily, and often significantly, impact most of society.”
The report, The new machinery of government: using machine technology in administrative decision-making, was written as a response to recent findings regarding Revenue NSW.
The state’s debt-collection agency was found to have been using an automated technology system to garnishee the bank accounts of people who had failed to pay fines.
A garnishee order, an indirect method of recovering a debt from someone, is legally permissible to be issued by the head of Revenue NSW, the Commissioner of Fines Administration, in certain circumstances to recover debts.
The NSW Ombudsman’s Office first became aware that there could be a problem when it received complaints from financially vulnerable individuals who, in some cases, had their bank accounts emptied.
Upon further investigation, it found that Revenue NSW had been using machine technology in the exercise of those garnishee powers.
“As we learned more about how Revenue NSW was using machine technology to issue garnishee orders, we became increasingly concerned about the lawfulness of its conduct,” the report said.
Revenue NSW is currently considering further changes to its garnishee system, according to the report.
“The government’s use of machine technology will necessarily, and often significantly, impact most of society.”
What is machine technology?
Not much is known about how NSW government agencies are using machine technology because there is currently no mandatory requirement or means of tracking technology use.
Machine technology is a broad term that doesn’t always include a computer or a device, the report said.
“In this report we have chosen to use the term machine technology to refer to a broad cluster of current and future systems and processes that, once developed, run with limited or no human involvement.
“[Their] output can be used to assist or even displace human decision-making (and specifically in the context of this report, within a public sector administrative context).”
Establishing a multi-disciplinary team
The report provides practical guidance to agencies to help them reduce the risk of machine technology.
The first step recommended by the report is for agencies to establish a multi-disciplinary design team that includes lawyers, policymakers, operational experts and technicians.
“Adopting machine technology to support a government function should not be viewed as simply, or primarily, an information technology project,” the report said.
“It is, rather, a coordinated exercise of legal, policy and operational process development, aided by technology.”
“Machine technology [refers] to a broad cluster of current and future systems and processes that, once developed, run with limited or no human involvement.”
Appropriate degree of human involvement
The second step is to assess the appropriate degree of human involvement in the decision-making processes.
“Discretionary decision-making requires some degree of human involvement – it can never be fully automated,” the report said.
However, this is more than just adding a human on top of a process, according to the report.
“Even if a person officially ‘signs off’ at the end of a process, the decision-making process may still be unlawful if in reality that person is merely acting as a rubber stamp, accepting the outputs of a machine… ‘without engaging in a mental process to justify that conclusion’.”
Agencies need to ensure that meaningful reasons are provided to those whose legal or other significant interests may be affected by decisions, the report said.
At a minimum, this should state that machine technology was involved, the nature and extent of that involvement, the information processed by the machine, the version of the technology used and an explanation about how the technology works.
Although these statements of reasons can be generated by the technology, the report advises against this.
“We suggest that it is safer, if a machine technology process is to be used in the decision-making process, that this process not also be tasked with generating a statement of reasons for the decision,” it said.
“Adopting machine technology to support a government function should not be viewed as simply, or primarily, an information technology project.”
Testing and legislative amendments
The fourth step involves agencies identifying ways of testing that go beyond checking that the machine technology is performing according to its programming.
“It is essential that machine technology be subject to ongoing monitoring, review and periodic evaluation to ensure that the technology continues to support lawful decision-making consistent with principles of good public administration,” the report said.
Finally, agencies should consider whether legislative amendment is necessary.
“It may be that, after applying the steps…the design team concludes that it would be unlawful or legally risky,” the report said.
“That raises the question: can and should the statute be amended to expressly authorise the use of machine technology?”
The NSW Ombudsman’s Office will work with relevant bodies, including Digital NSW and the Office of Local Government, to map the types of machine technology currently in use across state and local government.
“Following that work, we will explore whether there is a need for a centralised registry or other approaches to enhance transparency on an ongoing basis,” the report said.
“Even if a person officially ‘signs off’ at the end of a process, the decision-making process may still be unlawful if in reality that person is merely acting as a rubber stamp.”
Comment below to have your say on this story.
If you have a news story or tip-off, get in touch at email@example.com.
Sign up to the Government News newsletter