A "Once-in-a-Generation Opportunity"

On October 16, 2023, New York City Mayor Eric Adams stood at a press event and announced what he called "a once-in-a-generation opportunity to more effectively deliver for New Yorkers." The centerpiece was MyCity, a Microsoft Azure-powered AI chatbot designed to help small business owners get fast, reliable answers about operating in the city. The bot would provide "trusted information from more than 2,000 NYC Business web pages" on topics like regulatory compliance, available incentives, and best practices for avoiding violations and fines.

The city's Department of Small Business Services was behind the rollout. Commissioner Kevin D. Kim described the chatbot as a "baby step" into AI, noting that "trust is the most important thing" when governments deploy these tools. The framing was optimistic: AI would cut through the complexity of doing business in New York, sparing entrepreneurs from sorting through thousands of pages of city regulations.

Five months later, that optimism looked badly misplaced.

The Markup's Investigation

On March 29, 2024, reporter Colin Lecher published an investigation in The Markup (co-published with THE CITY) documenting that the MyCity chatbot was routinely dispensing advice that violated New York City law. The investigation wasn't a matter of nitpicking edge cases or tricky legal gray areas. The bot was confidently handing out guidance that directly contradicted well-established municipal protections for workers, tenants, and consumers.

The examples were specific and damning. When asked whether employers could take a cut of workers' tips, the chatbot said yes. Under New York labor law, this is illegal. When asked whether landlords could refuse tenants who pay rent through housing vouchers, the bot said they could. New York City's source-of-income discrimination protections explicitly prohibit this - landlords cannot reject tenants solely because they use rental assistance programs like Section 8 vouchers.

The chatbot also told users that employers could fire workers for complaining about sexual harassment. That directly contradicts New York's extensive whistleblower and anti-retaliation protections, which make it illegal to terminate an employee for reporting workplace misconduct. On paid sick leave, the bot gave answers that misrepresented city policy. On building codes and consumer protections, the answers were similarly unreliable.

Each of these responses carried the implicit authority of being on an official city government website. The MyCity bot wasn't positioned as a novelty or an experiment. It was presented to business owners as a compliance resource - a place to learn how to follow the rules. Instead, it was teaching them how to break them.

The City's Response: Keep It Running

The story gained traction fast. Reuters, the AP, Fast Company, Engadget, and numerous other outlets picked up the findings. The question turned from "what went wrong" to "what will the city do about it."

Mayor Adams addressed the situation at a press conference on April 2, 2024. He acknowledged that the chatbot's answers were "wrong in some areas." But he declined to take it offline. Instead, he defended the technology, reiterating his earlier position that AI represented a major opportunity for the city. The bot would stay up while the city worked on fixes.

New York City's Office of Technology and Innovation released a statement saying that "as soon as next week, we expect to significantly mitigate inaccurate answers." The response framed the problem as a bug to be patched rather than a systematic failure of the deployment model.

Microsoft, for its part, told Reuters it was working to fix the chatbot. The company had provided the Azure AI Services infrastructure powering the bot, but the specifics of how the city's content was being indexed, retrieved, and presented to users through the language model were less clear. The bot drew on those 2,000-plus city business web pages, but the retrieval and answer-generation pipeline was producing outputs that flatly contradicted the source material.

The city also updated its disclaimers, adding language noting that the chatbot's answers should not be treated as legal advice. That disclaimer was a quiet concession that the original positioning of the bot - as a source of "trusted information" on compliance and regulations - had been overstated.

Expert Criticism

The decision to keep the bot online drew sharp criticism from AI researchers and policy experts. Julia Stoyanovich, a computer science professor and director of the Center for Responsible AI at New York University, told the AP that the city's approach was "reckless and irresponsible." She said New York was "rolling out software that is unproven without oversight" and had "no intention of doing what's responsible." The criticism cut at the core of the city's strategy: deploying a public-facing AI system on sensitive legal and regulatory topics without adequate testing, validation, or human review.

Suresh Venkatasubramanian, director of the Center for Technological Responsibility, Reimagination, and Redesign at Brown University, told the AP that New York's chatbot should warn other municipalities considering similar deployments. The concern wasn't just about one malfunctioning bot. It was about a pattern forming across government agencies eager to adopt AI without fully understanding the failure modes.

Andrew Rigie, executive director of the NYC Hospitality Alliance, took a more pragmatic view. He acknowledged that AI could be a useful tool for small businesses but warned it was "also a massive liability if it's providing the wrong legal information." His position: the chatbot needed to be fixed immediately, and errors of this nature could not continue.

Why a City Chatbot Gets the Law Wrong

The MyCity chatbot was built on a retrieval-augmented generation architecture. In principle, this means the language model draws its answers from a defined set of source documents - in this case, the 2,000+ pages of city business guidance. The idea is that grounding the model in official documents prevents it from fabricating information. In practice, retrieval-augmented systems can still produce answers that misinterpret, selectively quote, or contradict their source material. The model synthesizes text that sounds authoritative regardless of whether the underlying retrieval step pulled the correct passages or interpreted them correctly.

Legal and regulatory content poses a particular challenge. Municipal codes contain cross-references, exceptions, conditions, and amendments that change meaning depending on context. A question like "can a landlord refuse a tenant who pays with a housing voucher?" has a simple answer under city law - no. But the relevant regulations may also discuss narrow exceptions for certain building types or scenarios. A language model scanning those documents may latch onto an exception clause and present it as the general rule, or miss the prohibition altogether.

The bot also lacked any mechanism for flagging uncertainty. When a human city employee doesn't know the answer to a regulatory question, they can say so and direct the caller to someone who does. The chatbot had no equivalent safeguard. Every response came with the same confident tone, whether the underlying retrieval had found relevant content or not.

The Broader Pattern

New York's experience wasn't isolated. By early 2024, multiple government entities were experimenting with AI chatbots for public services, and several were running into similar problems. The pattern was consistent: an eagerness to deploy AI for efficiency gains, combined with insufficient testing on the kinds of questions real users would actually ask.

The MyCity incident also revealed a tension in how governments frame AI adoption. Adams had positioned the chatbot as part of a broader "MyCity" IT modernization effort. The language around the launch emphasized accessibility and empowerment - giving small business owners tools to navigate city bureaucracy more easily. But that framing created expectations the technology couldn't meet. A chatbot that sometimes gets regulatory guidance right and sometimes advises breaking the law is worse than no chatbot at all, because users have no way to distinguish the accurate answers from the illegal ones.

Fast Company reported that even after the initial Markup investigation, the chatbot continued to produce problematic answers. When reporters tested it again in early April 2024, it falsely suggested that employers could legally fire workers who complain about sexual harassment. The errors weren't a one-time glitch; they reflected something structural about how the system processed and presented legal information.

Disclaimers as a Band-Aid

The city's decision to strengthen the chatbot's disclaimer - noting that responses should not be treated as legal advice - was a revealing move. It acknowledged the problem without actually solving it. A small business owner looking up whether they can require employees to work on a public holiday is unlikely to treat a chatbot disclaimer the way they would a lawyer's caveat. The entire design of the system was built to look like a reliable, official information source. Adding fine print at the bottom doesn't undo that framing.

This gap between presentation and capability is a recurring failure mode in government AI deployments. The technology often works well enough in demonstrations and controlled tests that decision-makers approve deployment. But the range of questions real users bring - and the consequences of wrong answers in domains like labor law and housing policy - create risks that prelaunch testing rarely captures.

The MyCity chatbot remained live through the spring of 2024 with the added disclaimer. Whether the city's promised improvements actually reduced the rate of legally incorrect answers was not independently verified. The bot continued to carry the implicit brand authority of an official New York City government resource, now with the asterisk that its answers might be wrong.