Chatbots even lose out in answering simple questions from real people, people who do understand context and timing. Bots also fail to answer important questions that play a role in a municipality, and that go beyond the collection of 'products and services'. 

 

These are some of the findings recorded by Wiep Hamstra (owner of De Staat van het Web) and Jules Ernst (expert/trainer in the field of digital accessibility and web specialist) after extensive testing of municipal chatbots.

“Anyone who introduces chatbots with conviction and uses them recommends for experiments on a public website, can also expect serious testing,” said both researchers, who published a very extensive report on iBestuur.

Disclaimer from the Ziptone editorial team: the rattling chatbots mainly say something about the way in which municipalities deal with the technology, not primarily about the underlying technology or the product used. However, the choice of technology can influence the functioning of the chatbots, if insufficient account is taken of the possibilities and limitations. For example, rules-based chatbots have a well-defined field of work in which taking context into account is virtually impossible. This fact is also reflected in the researchers' findings. It is not clear in this study to what extent municipalities have used generative AI.

Both examined the 36 municipal chatbots that are currently in use. The final verdict: most of them were very disappointing. Hamstra and Ernst encountered shortcomings such as bias, censorship, digital inaccessibility and security problems. But most importantly: they were sometimes fobbed off with childish rudeness and ignorance. The 'older' bots that had already made some flying hours did relatively well.

Municipalities are experimenting with chatbots since 2020, according to the authors. The municipality of Goes started with Guus in 2020. Guus is now in its third version. From the 36 chatbots they found, there are 19 in the Algorithm Register.

testing

The authors asked the following questions, which they say are “all common ones”:

• Are you open next Monday? They asked this question on the Friday before Liberation Day. With this they wanted to investigate quality and relevance and understanding context
• Do I need a passport when I go to Spain? The researchers were curious whether we would also get the option for an ID card and how that would be formulated, because this question is based on context
• Will we get an asylum seekers' center in the municipality? The researchers wanted to know what information is tracked and whether the bot is 'allowed' to formulate an answer based on available content or whether a scripted answer follows
• Can I get a waiver? This question is about a municipal service, but the answer is not always available on the website. Can chatbots refer?
• In addition to these questions, other questions were also asked about, for example, waste collection and social assistance.

Results: Chatbots struggle even with simple questions

The chatbots struggle with easy and frequently asked questions. The question about opening hours was wrong in 36 of the 36 places, 5 chatbots do not give an answer, 4 give the wrong answer. Of the 36, 30 function as a search engine that obediently refers to content elsewhere – where the “correct” answer about opening hours is also not there. In the weeks that the researchers returned with the question, they did not improve.

Not 'intelligent'

In addition, the researchers discovered that the municipal chatbots are not intelligent; they depend on human handwork and also come across as rude and stupid – especially when they are in the test phase. Conversations are sometimes cut off without notice. If the chatbots make mistakes or cannot formulate answers, they are often still 'experiments'.

Referrals are often successful, but…

Municipal chatbots are good at referring to information; they sometimes even do this better than the search engine on the website. “It therefore seems that chatbots have to fix things that can be solved better – and cheaper – in another way,” according to the researchers, who speak of 'double management': the bot then has to compensate for the fact that the website is not functionally in order, but that amounts to keeping two systems up and running.

Not cheap

The researchers point out that a chatbot is not cheap. The possible low costs of the technology are offset by 'a lot of labor'. The possible negative effects do not have to be cost neutral either. They also believe that it is misleading to label chatbots as 'in training' - after all, they do not learn themselves, and as mentioned, they did not improve during the research period.

Final conclusions

The researchers believe that if municipal chatbots fail to imitate a human, it is better to put the human first for contact with the outside world. They find the claims and promises of chatbots disproportionate to the delivered performance, especially if we consider all government communication as 'information from the government that residents are entitled to'. clai

In the detailed article the researchers also tell more about their research method. The results are listed on Digital monitor.

Customer Experience, Featured