A New York business chatbot is sending out some particularly bad information

a face with a blue filter and covered in words
Image Credit: Pixabay (Image credit: Geralt / Pixabay)

An AI chatbot released by the New York City government designed to assist business owners in accessing information has come under scrutiny for sharing inaccurate and misleading guidance.

A report by The Markup, co-published with local nonprofit newsrooms Documented and The City, reveal multiple instances where the chatbot provided wrong advice about legal obligations.

For example, the AI chatbot claimed that bosses could accept workers’ tips and that landlords are allowed to discriminate based on source of income – both wrong pieces of advice.

Chatbot fail?

Launched in October 2023 by Mayor Adams’s administration as an extension of the MyCity portal, the chatbot, described as “a one-stop shop for city services and benefits,” is powered by Microsoft’s Azure services. Despite its intention to serve as a reliable source of information sources directly from the city government’s websites, the pilot program has been found to generate flawed responses.

One example given by The Markup sees the chatbot asserting that businesses could operate as cashless establishments, despite the New York City’s 2020 ban on such practices.

Responding to the report, Leslie Brown, spokesperson for the NYC Office of Technology and Innovation, acknowledged the chatbot’s imperfections, emphasizing ongoing efforts to refine the AI tool:

“In line with the city’s key principles of reliability and transparency around AI, the site informs users the clearly marked pilot beta product should only be used for business-related content, tells users there are potential risks, and encourages them via disclaimer to both double-check its responses with the provided links and not use them as a substitute for professional advice.”

After a months-long honeymoon period, the cracks are beginning to show as businesses and government agencies start to question the reliability, safety and security of artificial intelligence, with many imposing bans and others introducing strict regulations.

More from TechRadar Pro

Craig Hale

With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!

Read more
ChatGPT app on an iPhone
ChatGPT and Google Gemini are terrible at summarizing news, according to a new study
Sam Altman and OpenAI
OpenAI launches a version of ChatGPT just for governments
DeepSeek on a mobile phone
Australian and Indian governments block DeepSeek from worker devices
A scale with AI on one side and a brain on the other
What is AI bias? Almost everything you should know about bias in AI results
AI Education
The AI lie: how trillion-dollar hype is killing humanity
A person using DeepSeek on their smartphone
DeepSeek ‘incredibly vulnerable’ to attacks, research claims
Latest in Pro
Protection from AI hacker attacks
Maintaining SAP’s confidentiality, integrity, and availability triad
A trough sensor at Overbury farm
“It's wildlife working for you” - how Agri-Tech can help revolutionize British farming as we know it
Epson EcoTank ET-4850 next to a TechRadar badge that reads Big Savings
I found the best printer deal you won't see in the Amazon Spring Sale and it's got a massive $150 saving
NVIDIA RTX PRO 6000 Blackwell Server Edition
Nvidia's most expensive Blackwell card gets massive price cut but it is not the RTX 5090
Microsoft Copiot Studio deep reasoning and agent flows
Microsoft reveals OpenAI-powered Copilot AI agents to bosot your work research and data analysis
Group of people meeting
Inflexible work policies are pushing tech workers to quit
Latest in News
Buzz Lightyear Space Ranger Spin Rennovations
Disney’s giving a classic Buzz Lightyear ride a tech overhaul – here's everything you need to know
Hisense U8 series TV on wall in living room
Hisense announces 2025 mini-LED TV lineup, with screen sizes up to 100 inches – and a surprising smart TV switch
Nintendo Music teaser art
Nintendo Music expands its library with songs from Kirby and the Forgotten Land and Tetris
Opera AI Tabs
Opera's new AI feature brings order to your browser tab chaos
An image of Pro-Ject's Flatten it closed and opened
Pro-Ject’s new vinyl flattener will fix any warped LPs you inadvertently buy on Record Store Day
The iPhone 16 Pro on a grey background
iPhone 17 Pro tipped to get 8K video recording – but I want these 3 video features instead