In what might be a first, a programmer added a feature to a piece of software because ChatGPT hallucinated it, and customers kept attempting to force the software to do it.. The developers of the sheet music scanning app Soundslice, a site that lets people digitize and edit sheet music, added additional functionality to their site because the LLM kept telling people it existed. Rather than fight the LLM, Soundslice indulged the hallucination.
Adrian Holovaty, one of Soundslices’ developers, noticed something strange in the site's error logs a few months ago. Users kept uploading ASCII tablature—a basic system for notating music for guitar, despite the fact that Soundslice wasn’t set up to process it, and had never advertised that it could. The error logs included pictures of what users had uploaded, and many of them were screenshots of ChatGPT conversations where the LLM had churned out ASCII tabs and told the users to send them to Soundslice.
“It was around 5-10 images daily, for a period of a month or two. Definitely enough where I was like, ‘What the heck is going on here?’” Holovaty told 404 Media. Rather than fight the LLM, Soundslice decided to add the feature ChatGPT had hallucinated. Holovaty said it only took his team a few hours to write up the code, which was a major factor in adding the feature.
“The main reason we did this was to prevent disappointment,” he said. “I highly doubt many people are going to sign up for Soundslice purely to use our ASCII tab importer […] we were motivated by the, frankly, galling reality that ChatGPT was setting Soundslice users up for failure. I mean, from our perspective, here were the options:
“1. Ignore it, and endure the psychological pain of knowing people were getting frustrated by our product for reasons out of our control.
“2. Put annoying banners on our site saying: ‘On the off chance that you're using ChatGPT and it told you about a Soundslice ASCII tab feature, that doesn't exist.’ That's disproportional and lame.
“3. Just spend a few hours and develop the feature.”
There’s also no way to tell ChatGPT the feature doesn’t exist. In an ideal world, OpenAI would have a formal procedure for removing content from its model, similar to the ability to request the removal of a site from Google’s index. “Obviously with an LLM it's much harder to do this technically, but I'm sure they can figure it out, given the absurdly high salaries their researchers are earning,” Holovaty said.
He added that the situation made him realize how powerful ChatGPT has become as an influencer of consumer behavior. “It's making product recommendations—for existent and nonexistent features alike—to massive audiences, with zero transparency into why it made those particular recommendations. And zero recourse.”
This may be the first time that developers have added a feature to a piece of software because ChatGPT hallucinated it, but it won’t be the last. In a personal blog, developer Niki Tonsky dubbed this phenomenon “gaslight-driven development” and shared a recent experience that’s similar to Holovaty’s.
One of Tonsky’s projects is a database for frontends called Instant. An update method for the app used a text document called “update” but LLMs that interacted with Instant kept calling the file “create.” Tonsky told 404 Media that, rather than fight the LLMs, his team just added the text file with the name the systems wanted. “In general I agree `create` is more obvious, it’s just weird that we arrived at this through LLM,” he said.
He told 404 media that programmers will probably need to account for the “tastes” of LLMs in the future. “You kinda already have to. It’s not programming for AI, but AI as a tool changes how we do programming,” he said.
Holovaty doesn’t hate AI—Soundslice uses machine learning to do its magic—but is mixed on LLMs. He compared his experience with ChatGPT to dealing with an overzealous sales team selling a feature that doesn’t exist. He also doesn’t trust LLMs to write code. He experimented with it, but found it caused more problems than it solved.
“I don't trust it for my production Soundslice code,” he said. “Plus: writing code is fun! Why would I choose to deny myself fun? To appease the capitalism gods? No thanks.”
