A buzzy startup offering financial infrastructure to crypto companies has found itself bankrupt primarily because it can’t gain access to a physical crypto wallet with $38.9 million in it. The company also did not write down recovery phrases, locking itself out of the wallet forever in something it has called “The Wallet Event” to a bankruptcy judge.

Prime Trust pitches itself as a crypto fintech company designed to help other startups offer crypto retirement plans, know-your-customer interfaces, ensure liquidity, and a host of other services. It says it can help companies build crypto exchanges, payment platforms, and create stablecoins for its clients. The company has not had a good few months. In June, the state of Nevada filed to seize control of the company because it was near insolvency. It was then ordered to cease all operations by a federal judge because it allegedly used customers’ money to cover withdrawal requests from other companies.

The company filed for bankruptcy, and, according to a filing by its interim CEO, which you really should read in full, the company offers an “all-in-one solution for customers that remains unmatched in the marketplace.” A large problem, among more run-of-the-mill crypto economy problems such as “lack of operational and spending oversight” and “regulatory issues,” is the fact that it lost access to a physical wallet it was keeping a tens of millions of dollars in, and cannot get back into it.

“In March of 2018, the Company created cold-storage wallets for purposes of maintaining cryptocurrency assets that included ETH, BTC, and ERC-20 compliant cryptocurrencies,” the company wrote in its filing. It called one of these wallets the “98f Wallet,” because its address ended in “98f.”

“To enhance security, the Company set up the 98f Wallet so that those who can ‘sign’ (i.e., approve) transactions would need to be in physical possession of hand-held Trezor or Ledger hardware devices,” the filing says.

The filing then states that, if the wallet is lost, most users create “seed phrases” that serve as backup codes that allow people to get into the wallet virtually: “Many users store seed phrases on hard copies of handwritten paper, images, and pictures. If a user loses both the hardware device and the seed phrases, it is virtually impossible for that user to regain access to the digital wallet.”

You can probably see where this is going.

Rather than write down the seed phrases, Prime Trust opted to laser etch them into a piece of steel called “Cryptosteel Hardware,” which are called “Wallet Access Devices” in the court filings, and which look like this:

According to the filing, it lost these devices, which is why it can’t get back into the wallet.

For several years, the company then took customer deposits into this address, to the tune of tens of millions of dollars. In December, 2021, “when a customer requested a significant withdrawal of ETH that the company could not fulfill [from other wallets,]” it went to withdraw it from this hardware wallet. “It was around this time that they discovered that the Company did not have the Wallet Access Devices and thus, could not access the cryptocurrency stored in the 98f Wallet.”

The company then, for several months, had to “use $76,367,247.90 in the aggregate to purchase ETH to fund customer withdrawals.” The money stuck in the wallet is currently worth $38.9 million as of August 22, it claimed. It is worth mentioning that the company did not tell regulators or customers about this issue for months after it discovered the problem.

All of the many ways this went wrong was broken down in detail in a great thread by Patrick McKenzie of Stripe.

The company has still not solved this issue: “The Company remains unable to access the 98f Wallet,” it wrote. “The investigation continues.”

Prime Trust swears in its filing that this was an “aberrant” event and “extremely unlikely to occur again.”

Instagram limited the reach of a 404 Media investigation into ads for drugs, guns, counterfeit money, hacked credit cards, and other illegal content on the platform within hours of us posting it. Instagram said it did this because the content, which was about Instagram’s content it failed to moderate on its own platform, didn’t follow its “Recommendation Guidelines.” Later that evening, while that post was being throttled, I got an ad for “MDMA,” and Meta’s ad library is still full of illegal content that can be found within seconds.

This means Meta continues to take money from people blatantly advertising drugs on the platform while limiting the reach of reporting about that content moderation failure. Instagram's Recommendation Guidelines limit the reach of posts that "promotes the use of certain regulated products such as tobacco or vaping products, adult products and services, or pharmaceutical drugs."

Meta reversed its decision on 404 Media’s post after we appealed it and deleted the words “guns, meth, pills, and weapons” from our caption, which was describing the types of content that Instagram is taking money for and injecting into users’ feeds.

As I reported earlier this week, most of these ads are for blatantly illegal products and services and link directly to Telegram accounts where drugs/guns/hacked accounts can be bought. A sampling of these ads can be found in seconds by searching for “t.me” on Meta’s Ad Library.

I have reported on content moderation for many years, and understand how difficult moderating content at scale can be. But the fact remains that the company has a massive, obvious problem with how it reviews and approves ads. An expert who studies content moderation told us that our investigation suggests Meta is not as sophisticated at reviewing and approving ads as it is at moderating normal posts on its platforms.

“You need to think about how much of an issue, historically, general content moderation has been,” Karan Lala of the Integrity Institute, made up of former integrity team workers at companies like Facebook told me. “The maturity of the different integrity teams is relevant to the kind of problems platforms have had in the past.”

In Meta’s case, its biggest ads-related issue has been with political ads. During the 2020 election campaign, it simply banned all political ads (months after the presidential election, it began allowing them again).

“With ads, a lot of the issue has been regulatory: Election ads spreading misinformation,” Lala said. “Ads for spam, ads for low quality content. Moderation of that seems a little less developed. In theory, these integrity teams should be feeding into the same systems. Ads should theoretically be going through a review process … a lot of these things are something that just should be getting caught [at the review process.]”

Meta did not respond to a request for comment. At the time of this writing, there are thousands of ads on Instagram and Facebook that link to Telegram channels and groups. Many of the most recent ones are for groups selling acid, MDMA, and hacked credit cards.

Taylor Swift’s management team planned the Eras Tour under the codename “Sparkle,” according to internal emails obtained by 404 Media using a public records request.

According to the emails, venues around the United States began planning the rollout of the tour in secret with Swift’s team in October of last year, a few weeks before it was announced by Swift on social media November 1.

On October 20, Sara Winter-Banks of Messina Touring Group, which organized the tour, emailed event planners at the Cincinnati Bengals, who operate the 65,515-capacity Paycor Stadium, to coordinate on the rollout of “the big tour date we’ll be working on together.”

“My team is reaching out to box offices today, regarding our Sparkle date ;),” Winter-Banks wrote in the email. “But a little info on the big tour date we’ll be working on together below for your knowledge.”

“Sparkle” is presumably a reference to the lyrics of “Bejeweled,” where Swift sings “And I miss you, But I miss sparkling.”

The email was titled “Tour Date Info - Cincinnati :)” Neither the subject line nor the email itself mention Swift by name, nor does it mention the “Eras Tour.”

The emails obtained by 404 Media give a small bit of insight into how one of the highest grossing and most-hyped tours of all time was planned and announced. The email goes on to mention the show date, when the tour would be announced, when tickets would go on sale, and included a link to a form where venue operators could ask Swift’s team to make them graphics to advertise the show.

Winter-Banks mentioned that emails before the official announcement would be sent “without an artist name.”

Another email from Winter-Banks sent a few days later (but still before the tour was announced) went into more detail about the precautions Swift’s team was taking to make sure information about the tour didn’t leak before Swift announced it. This email, however, did use Swift’s name.

“I will not be able to provide you the artwork or the press release until the tour has been officially announced,” she wrote. “I will circle back with exact timing on that, but expecting that to be between 8:30 am - 8:45 am ET. Will confirm when I know. For planning purposes, please be prepared for early in the morning ET on Tuesday, 11/1.”

Winter-Banks then explained that prior to the announcement of the tour, “I only have one person authorized to create the graphics” for advertisements. She said that they would be able to make additional advertisements “once Taylor has announced the tour … when we are able to bring more designers into the fold.”

The night before the announcement, Winter-Banks sent another email with an update: “Just checking in to let you know as soon as Taylor makes her official announcement on socials then I’ll clear you to post and will have the necessary assets and press release. The dates are going live on her site at 8:15 am ET and looks like her actual social pushes may be closer to 8:30 am ET. You’ll hear from me bright and early once she’s announced :)”

Swift ultimately tweeted her Eras Tour announcement at 8:24 am ET on Nov 1, 2022.

404 Media was able to obtain the emails using a public records request because Winter-Banks’s emails were eventually forwarded from an employee of the Bengals to Joe Feldkamp, the director of stadia for the Hamilton County, Ohio city government, which owns Paycor Stadium.

When Feldkamp got the emails (in the middle of the night, a few hours before Swift announced the tour), they became subject to public records requests. The fact that I was able to obtain these emails only because they were eventually forwarded to a public official is the exception that proves the rule about records relating to publicly-owned stadiums in the United States.

Though the vast majority of stadiums that Swift played in are publicly owned, agreements between local governments and the football teams who often operate the stadiums make it exceedingly difficult and often impossible to get records about the administration of facilities that taxpayers pay for.

The information I ultimately got may be groundbreaking only to Swifties, but the records I was actually seeking largely remain exempt from public disclosure and, therefore, public scrutiny. I was seeking information about ticket allocations for presales, which would have shown how many tickets were actually made available to the public during Swift’s various onsales, which turned into a legendary Ticketmaster disaster that ultimately led to Congressional hearings. If publicly-owned venues like Paycor were actually run by public entities rather than NFL teams, this information would be easier to obtain, and there have been a limited number of times that this sort of data has become public.

In 2009, for example, documents leaked that showed 11,720 of the 13,330 tickets for Taylor Swift's show in Nashville’s Bridgestone Arena were sold during presales. This means that during the "public sale," only a fraction of all tickets were actually for sale. Ticketmaster ultimately canceled the Eras Tour public sales because so many had been sold in presales.

When 404 Media filed the public records request, another email showed that two officials with Hamilton County emailed each other: “Wow and here I was thinking I avoided all the Taylor Swift madness.”

A safety feature in Instacart’s app warned its gig work shoppers last month to avoid a Kroger in Nashville because of an active shooter.

Instacart first announced a set of safety features in November 2021 in a blog that said the company would send a “Shopper Safety Alert” in the case of a “critical incident” or “safety incident,” but this is the first time we’ve seen the app alert a shopper to an active shooter.

A Reddit user posted a screenshot of the alert to r/InstacartShoppers, a community of gig workers who shop and deliver groceries for Instacart, who more often than not complain about the difficult working conditions and low pay at Instacart.

“Active Shooter 700ft away,” a red text bubble over a map showing a Kroger grocery store in Nashville, Tennessee reads, according to the screenshot. The user who posted the screenshot did not respond to a request for comment, but Instacart confirmed to 404 Media that the screenshot is authentic.

According to the screenshot posted to Reddit, Instacart’s app gave the shopper a safety tip, instructing them to “Run cautiously away from the attacker to the nearest safe exit” and to follow police orders with empty hands.

The screenshot of the active shooter alert was posted to the Instacart shoppers subreddit as the community is reacting to yet another Instacart policy change which they say has drastically slashed their pay. In post after post, shoppers are sharing screenshots showing payments of $13.72 to drive 10 miles and deliver 39 items to two different customers, $5.81 to drive 3.1 miles and deliver 18 items, and so on.

“New feature coming soon: Active shooter boost! Up to $2 extra for shopping in a store with an active shooter,” the top comment on the active shooter alert thread joked.

While this is the first time we’ve seen the active shooter alert feature in action, Instacart shoppers have been victims of gun violence multiple times in the past.

Last year, Instacart donated $50,000 to the GoFundMe account that was set up for the funeral costs of an Instacart shopper who was shot on the job while shopping at a Fred Meyer store in Richland, Washington. In April, a man in Florida shot at a couple’s car while they were making Instacart deliveries.

According to the Metro Nashville Police Department, someone in a black GMC SUV shot at another man in the Kroger parking lot, who returned fire. Both the SUV and the man fled the scene.

“It’s troubling that people would come to a parking lot and carry out their dispute,” MNPD spokesman Don Aaron told local NBC news station WSMV 4. No shots were fired inside the store, according to WSMV.

Instacart told 404 Media that it takes the safety of the shopper community seriously and that in the rare case that a critical incident happens near a store or delivery location, Instacart will alert shoppers nearby so they can avoid the area, and in some cases cancel orders so shoppers can get to safety without worrying about their orders. If a shopper’s order is canceled due to a critical incident, they will receive the full Instacart payment for the batch.

Instacart’s alerts come from “crisis monitoring” platform Samdesk, which leverages multiple feeds, including law enforcement, social media, and local news to generate automated alerts. The alerts are sent immediately to Instacart’s app, but Instacart’s own Trust and Safety team monitors events and follows up with shoppers in the area of an incident. Shoppers can also use a RapidSOS feature integrated with Instacart’s app, which connects them with 911 for immediate assistance.

Customs and Border Protection (CBP), part of the Department of Homeland Security, has bought millions of dollars worth of software from a company that uses artificial intelligence to detect “sentiment and emotion” in online posts, according to a cache of documents obtained by 404 Media.

CBP told 404 Media it is using technology to analyze open source information related to inbound and outbound travelers who the agency believes may threaten public safety, national security, or lawful trade and travel. In this case, the specific company called Fivecast also offers “AI-enabled” object recognition in images and video, and detection of “risk terms and phrases” across multiple languages, according to one of the documents.

Marketing materials promote the software’s ability to provide targeted data collection from big social platforms like Facebook and Reddit, but also specifically names smaller communities like 4chan, 8kun, and Gab. To demonstrate its functionality, Fivecast promotional materials explain how the software was able to track social media posts and related Persons-of-Interest starting with just “basic bio details” from a New York Times Magazine article about members of the far-right paramilitary Boogaloo movement. 404 Media also obtained leaked audio of a Fivecast employee explaining how the tool could be used against trafficking networks or propaganda operations.

The news signals CBP’s continued use of artificial intelligence in its monitoring of travelers and targets, which can include U.S. citizens. In May, I revealed CBP’s use of another AI tool to screen travelers which could link peoples’ social media posts to their Social Security number and location data. This latest news shows that CBP has deployed multiple AI-powered systems, and provides insight into what exactly these tools claim to be capable of while raising questions about their accuracy and utility.

“CBP should not be secretly buying and deploying tools that rely on junk science to scrutinize people's social media posts, claim to analyze their emotions, and identify purported 'risks,'” Patrick Toomey, deputy director of the ACLU's National Security Project, told 404 Media in an email.

404 Media obtained the documents through Freedom of Information Act requests with CBP and other U.S. law enforcement agencies.

One document obtained by 404 Media marked “commercial in confidence” is an overview of Fivecast’s “ONYX” product. In it Fivecast says its product can be used to target individuals or groups, single posts, or events. As well as collecting from social media platforms big and small, Fivecast users can also upload their own “bulk” data, the document says. Fivecast says its tool has been built “in consultation” with Five Eyes law enforcement and intelligence agencies, those being agencies from the U.S., United Kingdom, Canada, Australia, and New Zealand. Specifically on building “person-of-interest” networks, the tool “is optimized for this exact requirement.”

Related to the emotion and sentiment detection, charts contained in the Fivecast document include emotions such as “anger,” “disgust,” “fear,” “joy,” “sadness,” and “surprise” over time. One chart shows peaks of anger and disgust throughout an early 2020 timeframe of a target, for example.

The document also includes a case study of how ONYX could be used against a specific network. In the example, Fivecast examined the Boogaloo movement, but Fivecast stresses that “our intent here is not to focus on a specific issue but to demonstrate how quickly Fivecast ONYX can discover, collect and analyze Risks from a single online starting point.”

That process starts with the user inputting Boogaloo phrases such as “civil war 2.” The user then selects a discovered social media account and deployed what Fivecast calls its “‘Full’ collection capability,” which “collects all available content on a social media platform for a given account.” From there, the tool also maps out the target’s network of connections, according to the document.

Lawson Ferguson, a tradecraft advisor at Fivecast, previously showed an audience at a summit how the tool could be used against trafficking networks or propaganda operations. “These are just examples of the kind of data that one can gather with an OSINT tool like ours,” he said. Jack Poulson, from transparency organization Tech Inquiry, shared audio of the talk with 404 Media.

Ferguson said users “can train the system to recognize certain concepts and types of images.” In one example, Ferguson said a coworker spent “a huge amount of time” training Fivecast's system to recognize the concept of the drug oxycontin. This included analyzing “pictures of pills; pictures of pills in hands.”

Fivecast did not respond to a request for comment.

CBP’s contracts for Fivecast software have stretched into the millions of dollars, according to public procurement records and internal CBP documents obtained by 404 Media. CBP spent nearly $350,000 in August 2019; more than $650,000 in September 2020; $260,000 in August 2021; close to $950,000 in September 2021; and finally almost $1.17 million in September 2022.

CBP told 404 Media in a statement that “The Department of Homeland Security is committed to protecting individuals’ privacy, civil rights, and civil liberties. DHS uses various forms of technology to execute its mission, including tools to support investigations related to threats to infrastructure, illegal trafficking on the dark web, cross-border transnational crime, and terrorism. DHS leverages this technology in ways that are consistent with its authorities and the law.”

In the context of why CBP needs to buy Fivecast’s software, the internal CBP documents point to several specific parts of the agency. They are the Office of Field Operations (OFO), the main bulk of CBP which enforces border security; the National Targeting Center (NTC) based out of Virginia which aims to catch travelers and cargo that the agency believes threaten the country’s security; the Counter Network Division (CND) which is part of the NTC; and finally the Publicly Available Information Group (PAIG), which focuses on data such as location information according to other documents I’ve obtained previously.

Yahoo News reported in 2021 that the CND has gathered information on a range of journalists. The Office of the Inspector General made a criminal referral for an official who worked with CND for their role in the monitoring, but they were not charged. A supervisor of that division previously told investigators that at CND “We are pushing the limits and so there is no norm, there is no guidelines, we are the ones making the guidelines.”

“The public knows far too little about CBP's Counter Network Division, but what we do know paints a disturbing picture of an agency with few rules and access to an ocean of sensitive personal data about Americans,” Toomey from ACLU added. “The potential for abuse is immense.”

Apple told a California legislator that it is formally supporting a right to repair bill in California, a landmark move that suggests big tech manufacturers understand they have lost the battle to monopolize repair, and need to allow consumers and independent repair shops to fix their own electronics.

“Apple writes in support of SB 244, and urges members of the California legislature to pass the bill as currently drafted,” Apple wrote to Susan Eggman, the sponsor of the bill, in a letter obtained by 404 Media. “We support SB 244 because it includes requirements that protect individual users’ safety and security, as well as product manufacturers’ intellectual property. We will continue to support the bill, so long as it continues to provide protections for customers and innovators.”

This is a landmark shift in policy from Apple, the most powerful electronics manufacturer in the world and, historically, one of the biggest opponents of right to repair legislation nationwide. It means, effectively, that consumers have won. The news was first reported by TechCrunch and iFixit.

“If California votes yes and continues to raise the bar on electronics repair from other states, it’s becoming obvious the fight is over, and that we’ve won,” Nathan Proctor, Senior Director of consumer rights group U.S. PIRG Campaign for the Right to Repair, told me. “It’s going to be show over for consumer electronics. There are other industries where this fight is going to continue, but if a strong bill passes in California, we’re winning.”

In May, the Right to Repair Act passed the California Senate 38-0. It has not yet been voted on in the State Assembly, but Apple’s support, in its home state, makes passage more likely. Over the last year, right to repair legislation for consumer electronics has been enacted by New York (though that bill was watered down at the last minute by Gov. Kathy Hochul) and Minnesota.

The legislation would require manufacturers “to make available, on fair and reasonable terms, to product owners, service and repair facilities, and service dealers, the means, as described, to effect the diagnosis, maintenance, or repair of the product.” This means manufacturers have to make the same diagnostics, tools, and parts available to the public as they make available to their own authorized repair professionals.

In its letter, Apple says it will support the bill as passed, so long as it continues to require independent repair professionals to disclose the use of “non-genuine or used parts” to customers and does not require companies to create new documentation for repairs that its own authorized providers don’t already do.

The goal for right to repair advocates has been to pass legislation in a series of states, leading to total capitulation by manufacturers unwilling to spend money lobbying against bills in dozens of states and terrified of having to comply with different legislation in different states. A law in California, the heart of the tech sector with a population of nearly 40 million people and support from Apple, is likely to become defacto national legislation. A similar thing happened with the auto industry when, after Massachusetts passed a right-to-repair law for cars, manufacturers agreed to abide by the law in all 50 states.

“I would think that passage in California means there’d be a lot of pressure on manufacturers to kind of set the line there and say ‘no farther,’ because we’ve now proven to them we can pass laws and change the ways they have to operate,” Proctor said. “This shows state advocacy is a good way to deal with large problems that are hard to get through Congress. It shows you can really spread big tech thin if you have a real grassroots network behind you.”

Apple’s move represents a sea change from where the tech giant was just several years ago, when its lobbyists were telling lawmakers that allowing consumers to repair their iPhones would turn specific states into a “Mecca” for hackers and bad actors. Apple and other electronics manufacturers, represented by large industry lobbying groups, have spent millions fighting right to repair legislation, which would require them to make repair manuals, parts, and tools available to the general public.

Slowly but surely, thanks to activists and consumers, as well as support from the Biden administration, manufacturers have realized that they need to take steps to make repair more accessible. Microsoft, Apple, Google, Motorola, and Samsung have all introduced ways to purchase parts directly from the company or from authorized third parties like iFixit.

While this is a huge step forward for the right to repair consumer electronics, the fight will rage on in other industries. Agricultural giants like John Deere and medical equipment manufacturers still hold great power to prevent consumers and independent repair shops from fixing equipment due to an inability to get parts and artificial software and diagnostics locks that keep customers out.

The Alliance of Motion Picture and Television Producers (AMPTP), the organization representing Hollywood studios in negotiations with the Writers Guild of America (WGA)  that is currently on strike, has conceded that an AI is not a person, or a writer, as defined by their collective bargaining agreement.

“The Companies propose landmark protections for writers surrounding the use of Generative Artificial Intelligence (GAI),” AMPTP said in a press release. “The Companies confirm that because GAI is not a person, it is not a ‘writer’ or ‘professional writer’ as defined in this MBA and, therefore, written material produced by GAI will not be considered literary material under this or any prior MBA [minimum basic agreement.].”

In practice, what the studios are saying is a writer’s credit, pay, and rights will not be negatively impacted if they work on a project that began as an AI-generated script or idea.

“For example, if the Company gives a writer a [generative artificial intelligence]-produced screenplay and asks the writer to rewrite it, the writer will receive the fee for a screenplay with no assigned material and not a rewrite,” the AMPTP said. “Or, if the Company gives a writer a GAI-produced story as the basis for a teleplay, the writer will receive the story and teleplay rate.”

The AMPTP also said that it will disclose if the materials it’s asking writers to work on were generated by AI, and has agreed to WGA’s proposal “that a writer may not be required to use GAI in order to write literary material.”

This small concession about AI’s theoretical role in the writing process was just one of several proposals that the AMPTP put forward on August 11. The AMPTP also made proposals on pay, data transparency from streaming, and residuals. As Deadline reported last night, speaking about the proposals as a whole, the Writers Guild has already told members in an email that they included “limitations and loopholes and omissions failed to sufficiently protect writers from the existential threats that caused us to strike in the first place.”

The Guild also clarified that they “engaged in further discussion” with the AMPTP since August 11.

David Slack, a WGA member and former WGA board member, said on Twitter that AMPTP’s press release about this proposal was a “desperate move.”

“Not only are these proposals still inadequate, they are an attempt by the bosses to divide our members, hoping we’ll start arguing with each other over which parts we can and can’t live with,” he said. “Don’t fall for it.”

The Writers Guild did not immediately respond to a request for comment.

While it is a small concession, the proposal implies that incorporating AI into the fundamentally human process of telling stories that connect with people is at all tenable or advisable for the studios themselves. Companies have for years tried to use the data they collect on our viewing habits to predict what we’ll want to watch next. These are the same algorithms that power YouTube recommendations, and why we’ve seen attempts like 20th Century Fox partnering with Google in 2018 to train deep learning models on movie scripts to decide which movies they should produce. It is the same logic that dominates Netflix’s offering. But the company’s stock tanked last year, and this year Paramount’s seemingly guaranteed, Tom Cruise-boosted Mission: Impossible - Dead Reckoning Part One, tanked as well.

We don’t know if these types of AI tools can actually help studios, but these concessions from AMPTP show what the Guild, which has been on strike for over 100 days, already knows: they need human writers.

On the Discord server for Mage Space, a popular platform for creating AI-generated images, is a list of channels where members share adult content. There are channels for furries, hardcore and softcore porn, and anime.

At the bottom of the list is a channel named “other-nsfw” which includes a few distinct erotic genres that don’t fit neatly into any of the others. Gore is the biggest: images of nude women flayed open on operating tables, entrails wrapped around their necks, covered in viscera while choking on giant disembodied penises. Gynoids are heavily represented: chromed-out, feminized robots, similarly seen sucking or taking dicks. There’s the disgusting: women smeared with feces, distorted faces and bodies, growing tangled roots or metal spikes from their faces. Less common but maybe most interesting is the abstract: looping GIFs of an alien vulva with a lapping tongue, or a pubic mound with eyes, for example. The overwhelming majority shows sexualized, nude feminine bodies in mind-bending horror scenarios.

Some of the images in this article may be disturbing. Discretion advised.

The images people create, and the long, meandering prompts they write, are a rich text that could offer a glimpse into where sexuality in the internet age is taking us next, and how we’re steering it.

There’s no shortage of fetish content on the internet, which might make the above statement sound ridiculous and unbelievable. Online, fetishists find their people and set to work making more of what they like, whether it’s elaborate role-playing cosplays of themselves as sexy airplanes, blueberries, or slime monsters. Sometimes it pushes the bounds of the sane and legal: crush, fart, and scat porn all thrive online, and snuff films have been popular since before the internet existed.

But with the rise in popularity of generative AI, and wildly popular sites like Mage.Space that let users generate anything they set their minds to, the limits are literally our own imaginations. With that power, people are wrangling images out of the AI that are on the edge of what’s popular, let alone possible in the porn world.

We are approaching the event horizon of horniness.

“This conversation we're having is literally on the edge of knowledge, nobody's writing about this in academia right now,” Thomas Brooks, assistant professor of psychology at New Mexico Highlands University, told me. “Everybody's still kind of caught up in deepfakes. And they haven’t quite grappled with generative AI yet.”

Unlike deepfakes, which typically only replace one face for another in an original, human-recorded video, generative AI formulates a new image wholesale, based on thousands of real images scraped into datasets such as LAION-5B. But it’s also distinct from the Unstable Diffusion models that became popular last year, where people broke Stability AI’s open-source text-to-image generation model Stable Diffusion out of its anti-porn parameters to make surreal erotica.

“You, as the individual porn consumer, can now create your own special little fantasy and your own technological, disembodied sexuality"

Much like DALL-E, images on Mage Space are generated within seconds by writing prompts, which describe in words what image the AI should generate. A fairly mundane image on Mage that falls into the other-nsfw category of a human man penetrating a bloody zombie woman with long blonde hair and a crop top is made with a prompt that includes keywords like  “swedish hyper tech biotech ancient dystopia,” “holds glass pipe with cocaine in foreground, “((with rotting Corpses in the Livingroom)),” and “explicit blood, gore, disgusting, gross, morbid, sick, twisted mind.”

Prompts can be copied from other users if you like what they produced, or users can spend time tweaking every keyword until the desired result is achieved. No two prompts will output the exact same image, even when the wording is identical.

“You, as the individual porn consumer, can now create your own special little fantasy and your own technological, disembodied sexuality,” Brooks said. He called this gamified pornography. “There's an internal motivation to solve the puzzle and get the prize. But then there's an external motivation of, ‘can I come up with this crazy thing to show my anonymous internet friends.’”

Because all AI-generated content is constrained by the content it’s trained on, AI-generated porn, even when it seems alien, still manages to be familiar. It’s shocking and disturbing, but because it always remixes existing content, it’s the opposite of transgressive.

None of the experts I spoke to had seen this kind of content before, but viewed several examples when I reached out for comment.

“On the surface, the surrealism may not seem erotic, but looking at the prompts used to generate the images, prompts like ‘furry,’ ‘tentacles’ and ‘decay,’ it’s clearer that the creator is using well-known fetishistic or paraphilic elements, which are designed to increase arousal,” Eric Sprankle, professor of clinical psychology and co-director of sexuality studies at Minnesota State University, Mankato, told me. “Granted, seeing an image of a half-woman/half-sloth getting vaginally, orally, and anally penetrated by large tentacles of unknown origin will likely catch the eye of many, but it will likely only arouse a few.”

Even if you’ve seen tentacle porn before, you might have to take a second look when it’s rendered realistically and with elements of horror. “Right off the bat, I felt like I had seen this imagery before and yet hadn’t,” Laura Helen Marks, porn scholar and author of Alice in Pornoland, told me.

“Because AI is gathering existing data, the imagery is inherently uncanny and I think that disorientating quality is likely a big part of the appeal,” Marks said. “Gothic horror relies on that sense of the uncanny for its thrills, and pornography does the same in a lot of ways—dissembling bodies, reduction of civilized people to their gooey, aroused state, pushing the bounds of corporeality—so it makes sense that we would use this technology to push the boundaries even further.”

We’ve seen a demand for adult content push technological boundaries throughout history, with art forms like digital animation, CGI, and 3D rendering. But with AI-generated porn, the makers themselves can be surprised by what comes up, even if they have a 300-word prompt with granular specificity. What’s in their mind’s eye might only be a shade of what the computer generates on the screen.

The fact that there’s no one being physically, literally murdered is, obviously, important as part of the enjoyment for creators and consumers of gruesome AI porn. The uncanniness of the images is also part of what makes them bearable to view: I wouldn’t be able to stomach casually scrolling through miles of real photographs of people self-immolating until their limbs are gone or being skewered through the chest and smeared with blood, but because it’s not real, it’s bearable. Horror movies have the same effect: people who would find snuff films or footage of mass shootings too disturbing might still enjoy Saw.

The unreality of it all—even when it’s portraying real people in vulnerable or humiliating situations, and many deepfakes do—isn’t just permission to enjoy it, but a central part of consuming it. The creator can still do the violent or degrading thing to the subject, while claiming no one was harmed.

In the adult industry, plenty of bloody and even disturbing porn exists and is made by consenting adults in safe environments. Still, adult filmmaker and founder of Sssh.com Angie Rowntree wondered how a culture that struggles with porn literacy and separating fantasy from reality will handle a new way to make hyper-violent erotic content. People still blame consensually-made and professionally-produced porn and sex workers for all sorts of social ills, and the conservative, anti-porn movement is stronger than ever.

“As an adult filmmaker, I really have to wonder: why are people using AI to take sexuality to such a nihilistic, hateful place?” Rowntree said. “It’s hard to claim that it’s about ‘pushing the envelope’ when it’s more like literally shredding women to pieces.”

AI-generated erotic imagery isn’t representative of real-life sex, but it’s not comparable to erotic art drawn by humans, either. It exists in a gray area that’s more grotesque, an amalgamation of stolen images taken from places like Reddit, scraped from porn sites, and the endless fetish illustrations and art all over the internet—churned out as a hyperreal simulacra of human (and inhuman) sexuality. What AI porn makers manage to excavate from the tar pit of the internet isn’t revealed until the moment it shows up on the screen, even if the prompt is carefully crafted and specific.

Porn, like Hollywood, tends to follow social trends and especially reflects our anxieties. There hasn’t been a larger social anxiety in the last three years than the pandemic, and Brooks wondered if the impulse to destroy AI-generated bodies, many of which are places in hospital-like settings, on gurneys or operating tables and in some images literally surrounded by effluvia-covered nurses, might be related to the body horror of living in the 2020s. “There's this idea that the body is not safe and we need to flee the body to actually experience sexuality,” he said. “And when they get there, the first thing they do is kill the body.”

At the same time, our lives became forcibly, aggressively digital. AI chatbot companions are also in the zeitgeist now more than ever before, as people form romantic, parasocial relationships with virtual girlfriends. “I think this is probably just another expression of that, which tracks with the overall idea of transhumanism in our current culture,” Brooks said.

People might not explicitly believe they’ll one day join the singularity, he said, but there has been a shifting acceptance toward moving our human experiences out of the physical body and into virtual places, like Zoom or Discord.

“We're letting technology become mediators of our different psychosocial expressions,” Brooks said.  

It’d be easy to ask, “Is anyone jerking off to this?” and it’s admittedly a question I asked the experts I talked to for this piece. Marks suggested the question itself might be the wrong one.

“We assume ‘getting off’ or finding things exciting takes just one form—masturbating—and while that is likely a common practice, it surely isn’t the only response to looking at pornographic media and not the only reason people seek it out,” Marks said. “I think if we reduce the viewer's response to masturbation, then when we ask, ‘why do people enjoy this?’ we are really asking, ‘why do people masturbate to this?’ This reductive approach obscures the actual pleasures of this media, some of which may overlap with the pleasures of horror media in general, as well as ‘shock porn’ and other stuff that shares some of the features of AI-generated porn.”

Arousal and stimulation can be a complex, diverse response, she said—one that can blur the lines between erotic, horrifying, shocking, and disgusting. “For some reason, when sex is introduced to the equation we tend to center it as the entire point, ignoring the other qualities.”

I tried asking many of the people who are actively posting in the other-nsfw Mage Discord channel what they find so captivating about their hobby, but most of them didn’t respond. Everyone there is anonymous, and many are using usernames that aren’t connected to any other internet presence. If their hobby of making graphic horror porn ever broke the containment of this chat—and the kinship of other people making the same stuff—it wouldn’t be good for their real-life reputations.

Inside the AI Porn Marketplace Where Everything and Everyone Is for Sale

Generative AI tools have empowered amateurs and entrepreneurs to build mind-boggling amounts of non-consensual porn.

404 MediaEmanuel Maiberg

One member of the channel, who goes by Tangelo, did respond with a few short answers to my questions. He said he’s been using AI generation models since the days of DeepDream, Google’s neural network that launched in 2015 and created images using machine learning that looked like a bad acid trip.

“I have been creating erotica and ‘porn’ since the late 90s with computers,” he said, including 2D art and 3D rendering. “The appeal is to be able to create and manipulate things as you please, just like with painting and drawing.” I asked if he’s ever shocked by what the Mage Space generator creates. “Surprised yes. Shocked, not usually,” he said.

The longer we have been online—and it’s been about 33 years, now—the more the power of the internet has shaped our sexuality, and our desires have shaped it back.

Rowntree has seen these kinds of shifts firsthand in the 25 years she’s been in the industry, along with her husband Colin, who founded the oldest BDSM site on the web, Wasteland. Around the time tube sites like Pornhub and XVideos became popular, she noticed that Sssh.com members—who mostly came to her site as it was marketed toward romance and a more rosy, beach-read kind of porn—started requesting that she offer more hardcore, rough, and dominating sex scenes.

Easy access to free porn on tube sites, as devastating as the business model was for independent webmasters like herself and Colin, changes people’s desires. Rowntree said that she’s received an increase in requests for AI themes from Sssh.com members since the technology has been in the news a lot recently.

“However, what our members are requesting relates more to the many sentient AI beings that have been depicted in television and film – like Andrew in Bicentennial Man, the AI character Samantha in Her, or even Data from Star Trek,” she said. “For the most part, fans express a desire to explore the potential of human connection with AIs, as much as the AI characters in the above stories have aspired towards being ‘more human.’”

Pornographers also have to abide by things like rules set in place by discriminatory payment processors, lest they be banned from their banks, and have to obey strict terms of use on sites like OnlyFans, where blood of any kind (including menstrual) is not allowed. Even the insinuation of violence or non-consensual consent can get you permanently removed from most adult sites.

Rowntree told me she’s concerned that AI could mark a similar shift in how the industry operates. “In the early 2000s, tubes put pay sites through a fresh kind of hell that saw our production budgets and revenue cut in half,” Rowntree said. Deepfakes of real performers are a concern, and are already happening on Mage Space, to small-scale and celebrity-level performers alike. An influx of gore generated by AI to adult sites could also cause platforms to tighten their rules even more, and crack down harder on real performers.

“It will not be the wild west forever, and we also do not think that ‘fake girls’ are going to supplant real adult performers. Nothing will ever beat ‘the real thing’ for most fans,” Rowntree said. “If AI can help performers or filmmakers to monetize their content more effectively or express themselves creatively in new ways, then that is definitely exciting. However, in the meantime, it could really cause harm to the adult industry (even more than what we’re already facing) if AI porn creators play fast and loose with content standards and compliance.”

Safe, sane, and consensual pornography is the industry standard, no matter what fantasies are at play, she said, “and we need to ask ourselves what happens to the broader industry and beyond if AI porn creators can get away with promoting the very opposite?”

For now—like many online spaces for fetish and kink—people are mostly finding like-minded people to share something that’s considered shockingly taboo in the mainstream.

“At most, it may unlock some unknown kinks in an individual where someone may surprisingly realize, ‘Oh, ok. I guess I’m into mythical sea creatures,’” Sprankle said. “But for society as a whole, I wouldn’t expect to see a large increase in the demand for tentacle dildos.

Emanuel Maiberg contributed reporting.

For the last few months Instagram has served me a constant stream of ads for hard drugs, stolen credit cards, hacked accounts, guns, counterfeit cash, wholesale quantities of weed, and Cash App scams, as well as a Russian-language job posting seeking paid-in-cash massage therapists. Nearly all of these advertisements link directly to Telegram accounts where the drugs or illegal services can be directly purchased. With one tap, I was repeatedly taken from bouncing through Instagram stories of my friends on vacation to Telegram chat accounts where I could buy automatic weapons, meth, and stolen credit cards.

Besides being delivered directly to me by Instagram’s algorithm, thousands of these ads can be trivially found on Meta’s ad library by searching “t.me,” which is the link shortener for Telegram, exposing a massive content moderation and ad screening failure by the company.

Many of these services are being advertised as part of a side hustle culture. Most of the ads I’ve gotten are of young men wearing ski masks holding gigantic stacks of cash with captions like “Take 10 minutes out of your day to learn how,” “make some bread,” “join tele to make $50k a month,” “going to eat or starve?,” “learn to make a bag,” “While some of y’all WATCHING us get RICH… EVERYONE else is TAPPIN in for the CASHAPP MONEY DROP to get RICH!” Once on Telegram, it becomes immediately clear that these accounts want to help you get rich by simply helping you deal drugs, steal people’s money, or by draining hacked bank accounts or using stolen debit and credit cards.

The ads are a window into a blatantly illegal underground economy of drug dealers, hackers, and scammers that Meta is not only failing to moderate, but is actively profiting from and injecting into users’ feeds.

“Because the harm itself isn’t on the platform—it’s a link to a different platform—this suggests Meta lacks the ability to do in-depth [analysis] of the links and solicit data from Telegram about the users’ identity,” Karan Lala, a founding fellow at the Integrity Institute, which was created by former members of Facebook’s Integrity team, told me. Lala has previously studied the prevalence of spam accounts that advertise on Meta’s platforms.  “Just because it’s off-platform doesn’t mean it should be an excuse. There’s things like weed photos—that’s something that should just be getting caught. If I were an engineer on the integrity team, I would want to know why our systems aren’t catching them."

Like many of Meta’s algorithmic rabbit holes, my journey into this world started with a single, curious click. After years of being served primarily ads for surf brands, clothes, and productivity apps, I got an Instagram ad with a hooded man standing in front of a Chase Bank ATM holding a giant stack of cash: “I got half a million worth of sauce in my iCloud,” the ad’s caption said. The ad was from an account called PunchMadeDev, which was verified and had tens of thousands of followers.

Clicking through the Instagram ad led directly to a Telegram group with 78,000 subscribers, which claims to sell a CashApp “Glitch” for $200 in Bitcoin that allegedly allows users to send Bitcoin through the Cash App and get “anywhere from $7,000-$14,000” in return. The PunchMadeDev Instagram account also linked directly to a website claiming to sell all sorts of stolen accounts and hacking tools. These include logins for Wells Fargo, Bank of America, and Chase Bank accounts with guaranteed balances in them, Venmo and PayPal accounts, Cash App accounts, and UberEats accounts. They are also selling what they say are verified Instagram accounts, YouTube accounts, and Netflix, Hulu, and Disney+ logins. It is also selling a bot designed to “hi-jack all mobile phones and email addresses 2FA codes.”

“Linkable credit card with $7k-$10k balance. Attaches to Cashapp, PayPal, Apple Pay,” one product listed for $120 reads. “If you buy this item you’ll get my full support on Discord/Telegram if there is a problem!” 404 Media has not tested any of these accounts because logging into them would be illegal. A spokesperson for Cash App said the glitch being advertised is a common Cash App scam and that they are not aware of any vulnerability in their system.

I got in touch with the person running the PunchMadeDev account, but the person declined to answer any questions.

Since clicking on the PunchMadeDev ad, I’ve been bombarded with hundreds of Instagram ads for illegal services and drugs. Nearly all of these advertisements link to Telegram accounts where the drugs or illegal services can be directly purchased, though some of them go to Linktree accounts or a user’s own website that advertises these services. From these ads, I have joined 55 different Telegram servers (ads for the same servers pop up over and over again from different Instagram accounts). At one point, I was asked by Instagram if I “want more or less of your ads to be like this?” beneath an advertisement that featured blank credit cards and stacks of cash.

While some of the ads are subtle, many of them are not. Lots of the ads use clear language and imagery about what they’re selling on Instagram itself. I have seen ads featuring giant bags of mushrooms, sheets of acid tabs, piles of guns, ads for “counterfeit bills,” pharmaceutical bottles and huge piles of pills, and trash bags full of weed. Sometimes the usernames are gibberish or seem designed to evade Instagram’s content moderators, while others have Instagram display names like “magicmushrooms_psychedeliccs,” “medical grade pills,” “clone credit cards for sale,” and “high quality counterfeit notes.”

Once clicking through to Telegram, most sellers offer a “menu” of items. One, with 102,400 members at the time of this writing, sells “Magic Mushrooms, lsd (tabs, liquid), ecstasy, molly, adderalls, xanax, Dot, coke, meth crystals, ketamine, and mdma,” among others. The chat is full of photos of what the admin says are drugs and what certainly look to be drugs: “COKE (HOT), METH CRYSALS (HOT) DMT (HOT), KETAMINE (HOT),” one recent message, beneath photos of the products, said. The chat is full of instructions on how to order, videos of the products and of USPS boxes and receipts designed to inspire confidence among customers.

Another group advertises “Ice meth available in kilo weight” and has a photo of a bag of white crystals on a scale. Another advertises “ozempic weight loss injections for sale UK, France, USA, Canada.” Another advertises “AAA+ category of counterfeit money. All our counterfeits work on ATM machine, shopping centers, casinos, in malls and they’ve also bypassed all counterfeit tests. Pounds, Euros, Dollars.”

A Telegram channel specializing in the sale of guns says “hmu for any kinda glocks you need. We give them at good prices” and advertises international shipping. Successful deliveries to customers are called “touchdowns.”

A few of these servers specialize in wholesale quantities of weed, others offer only hacking services or stolen accounts, while some are illegal service emporiums offering a whole host of things. One group, for instance, sells “cloned” credit cards alongside bottles of codeine syrup, oxycontin, and silenced Glock handguns. One Instagram ad had a photo of a woman in a dress with a Russian caption that said “we are currently looking for female massage therapists to work in our salon. We offer high income ($500-1500$ per day) and good working conditions.” The next slide of the ad had a woman’s hand holding a stack of hundred dollar bills and linked to a Telegram that had been removed from the chat app for violating its policies.

I asked Laura Edelson, a researcher at New York University who specializes in social media ad spending, if there was a way of estimating just how prevalent these sorts of ads are on Instagram and Facebook, and how much money Meta might be making from them. She said that “Meta makes ads shown on Facebook/Instagram transparent through its Ad Library while they are active only.”

Edelson suggested that the large number of ads for illegal content that can be trivially found “certainly isn’t promising,” and pointed to the thousands of active ads that can be found for these types of services on the Ad Library by searching “t.me,” the link shortener for Telegram.

It’s also difficult to determine which of them are scams designed to get people to send the seller money in return for nothing and which are actually selling drugs or guns.

Sellers spend most of their time in their Telegram chats trying to prove their legitimacy. They are full of text message screenshots from supposedly happy customers, videos of USPS receipts that supposedly show shipments, and videos and photos of their products. Some sellers offer FaceTime verifications where a buyer video chats with the seller to see the specific product before placing an order, or does a FaceTime call with the seller to be talked through how to login to a hacked account, link a stolen debit card to the Cash App to cash out, or otherwise verify that what they’re buying will actually work for them.

“FACETIME VERIFICATIONS ONLY,” one person selling a pound of weed for $1,000 wrote in a recent chat. “WE OFFER 2 DAY SHIPPING FOR ALL OUR CLIENTS ALONG WITH INSURANCE!”

“*WE HAVE THE FOLLOWING* WEEKEND DISCOUNTS $65 DISCOUNTS ON ALL ORDERS SHIPPING VIA USPS: Magic shrooms, LSD (tabs, liquid), DMT, Coke, XTC pills, Xans bars,” another group advertised.

It’s hard to say which of these are “legitimate” and which are scams, but “a lot of drug sales have migrated from the dark web to chat messengers,” Ian Gray, a researcher at the cybersecurity firm Flashpoint, told me.

Regardless of whether the accounts are scams or actually selling drugs, guns, and credit cards, the ads shouldn’t be on Instagram under its own terms of use and are a content moderation failure.

Meta has shown no real ability to reliably moderate these ads since I showed them examples of ads from 10 different accounts selling illegal products. An Instagram spokesperson said “we investigated and disabled six of the accounts you shared for violating our policies; the rest had already been disabled.”

Immediately after hearing this, I searched again for ads from the same hacking and drug vendors and was able to find several of them operating under new accounts, but linking to the same Telegram accounts. PunchMadeDev, for example, has a series of backup accounts and backup Telegrams that all link to each other; I got a new ad directing me to PunchMadeDev’s Telegram within a day of Facebook banning the original account.

The Meta spokesperson said that “the prevalence of content that violates our Restricted Goods and Services Policy is about 0.05 percent of content viewed on Facebook and Instagram. In other words, out of every 10,000 views of content on Facebook and Instagram, we estimate no more than 5 of those views contained content that violated this policy,” and added that “Views of content violating these policies are very infrequent, and we remove much of this content before people see it.”

While this overall percentage may sound low, Facebook is one of the largest ad companies in the world and has billions of users viewing billions of pieces of content. And, like other types of content on Facebook, the problem is self-reinforcing. For the purposes of this article, I intentionally clicked on most of the ads for illegal services that I saw. As a result, a large portion of all of the ads that I have seen on Instagram over the last few months are for these types of accounts. “When you engaged with that specific ad you got, that reinforced your interest in these types of ads,” Lala said. “It looked at the profiles and characteristics of other people who clicked on that ad you clicked on and, once you start going down that rabbit hole, you just start to see all of it because of the characteristics of your engagement pattern.”

While most of the ads I got were in my stories, a few of them showed up in my normal Instagram timeline. Two of those posts had stats on them: One ad had 2,484 likes, the other, a video, had 21,622 views.

Lala said that Meta faces what seems to be a high “recidivism” rate with these sorts of ads. Meta bans the ads and the accounts posting them when it becomes aware of them, but new accounts keep popping up, and they keep managing to get through the ad approval process: “The cost of taking down an asset is so much higher than the cost of creating a new asset,” he said.

There’s another major issue here: Even if these ads are seen by a relatively small number of people, there is no incentive for the people posting them to actually stop. Once someone joins a seller’s Telegram chat, the seller has the ability to reach that person indefinitely with or without Meta, because Telegram itself does very little content moderation. That means that each new Instagram ad leading to a Telegram chat might have the ability to get the seller a few new prospective customers, and being banned by Meta doesn’t actually harm their actual business of selling illegal goods on Telegram.

“In order to prevent this kind of harm, you have to identify the link is malicious at the ads review process and before there’s any traction,” Lala said. “It can’t be that you have some traction then it gets flagged.”

After analyzing some of the accounts I sent him, Lala speculated that the people buying these ads are cycling through burner phones, using VPNs, and doing other things to make their ads and their alt accounts more difficult for Facebook to detect: “You create an account on a $50 burner phone, use it until it gets blocked, then move to another device.” He also noticed that a lot of the accounts running ads were months or years old, suggesting that sellers are making, buying, or hacking accounts, letting them “mature,” and then “doing more high-risk activities with it.” They do this because Facebook is likely to put more scrutiny on ads from new accounts that could have been spun up specifically for this purpose.

I have regularly covered Facebook’s content moderation, and have pointed out how difficult it is to moderate the posts of billions of users from nearly every country in the world in a variety of languages and from a variety of different cultures. While it’s true that content moderation is difficult, there’s a difference between allowing someone to post links to drug marketplaces on the platform and actively selling ads for these marketplaces and injecting those posts directly into user’s feeds.

Previous studies have found that Facebook’s ad marketplace has vulnerabilities. Last year, for example, a joint study between Global Witness and NYU’s Cybersecurity for Democracy found that “Facebook either failed to detect, or just ignored, death threats against election workers contained in a series of ads submitted to the company.” That study also found that YouTube and TikTok more consistently rejected or quickly removed the researchers’ test ads.

This year, Meta has laid off large portions of its staff, which has also affected teams that do content moderation.

“Unfortunately, this might be a consequence of the recent layoffs in Trust & Safety that Meta has made. Security is an ever-evolving game, and Meta may just no longer have the resources in place to keep up with the tactics bad actors are using to get around policy enforcement,” Edelson of NYU said. “In this case, advertisers are directing users off-platform. Facebook used to have fairly sophisticated detection of this kind of activity, but the volume that a trivial search is turning up right now indicates that may no longer be the case.”

The National Highway Traffic Safety Administration will no longer instruct car manufacturers to ignore a landmark car repair law in Massachusetts, according to a letter obtained by 404 Media. This means the overwhelmingly popular right-to-repair law will seemingly finally be allowed to go into effect nearly three years after voters passed it by a 75-25 percent margin.

The Massachusetts law updated 2013 legislation that fundamentally allows independent repair pros and car owners to access car diagnostic data and repair codes. The new law was designed to allow for wireless access to that data, something that could become necessary as car companies propose to do away with wired ODB2 diagnostic ports, which are usually located beneath steering wheels.

The law had been set to go into effect earlier this summer, until NHTSA instructed all major car manufacturers to ignore the law, which they had spent tens of millions of dollars fighting against and scaremongering over. NHTSA argued that the Massachusetts law conflicted with federal safety legislation and was therefore preempted and void. This put NHTSA, which is part of the Biden administration, at odds with the President’s stated position.

NHTSA claimed in its initial letter that repair was somehow dangerous, and that wireless diagnostics posed an “unreasonable risk to motor vehicle safety” and would make cars less secure. This is exactly what car manufacturers argued in fear-mongering commercials that said ‘sexual predators’ would use wireless tools to stalk women in parking garages.

Tuesday, after widespread outrage from voters and consumer rights groups, NHTSA walked back its position in a letter to Massachusetts Assistant Attorney General Eric Haskell, stating that it had found a way to “advance our mutual interest in ensuring safe consumer choice for automotive repair and maintenance. NHTSA strongly supports the right to repair.”

The letter goes on to say that car manufacturers should sell devices to independent repair shops that do diagnostics “from within close physical proximity to the vehicle, without providing long-range remote access.” This was, of course, the plan all along, as no one was proposing that random repair shops be able to access people’s car from miles away.

Nathan Proctor, senior director of consumer advocacy group US PIRG’s Campaign for the Right to Repair, said in a statement that the letter is a good start, and that NHTSA “admits that the security problems in cars are not created by independent repair access—if there are security issues with car data, they are already there.”

Proctor added, however, that NHTSA’s initial position—that repair is dangerous–did harm.

“We strongly support the goals the agency puts forward—to protect repair choice and maintain safety. However, as it stands, the agency has achieved neither goal,” he said. “Instead, it has allowed a proliferation of serious safety and monopolization issues to continue without meaningful resistance. Let’s hope this new letter signals a change in approach. We don’t plan to stop our work until cars not only are safe, but also enjoy the full slate of Right to Repair protections.”

Hello, and welcome to 404 Media, a new digital media company created, owned, written, edited by four journalists who until very recently worked at VICE’s Motherboard. We’re glad you’re here, and we’re glad we’re here, too.

At 404 Media, we aspire to do society-shifting technology journalism, and to create a sustainable, responsible, reader-supported media business around it. We will report and publish stories that you will not find anywhere else, that we believe only we can do. We hope these stories will take over the internet, impact public policy, and expose bad actors. We will point out the absurd. We will be irreverent and have fun. We will also do very serious work. We hope that you will read these stories and want to send them to your group chat, or bring them up as conversation starters at parties.

We believe that if we do this journalism well enough, you, our readers, will be willing to subscribe because you think it’s important, because you think it should exist in the world, and because you feel you’re personally getting value from it.

Today, you’ll find an article about a horrifying data broker ecosystem where highly sensitive data filters from credit bureaus to violent criminals on Telegram. You’ll get a look inside a generative AI community that’s churning out sexual images of real people without their consent on a massive scale, and you’ll finally figure out why sex toy vibration patterns are so frustrating. You’ll see the incident report from the Boston ‘Cop Slide’ fiasco (thank you, FOIA). And you’ll learn why Airpods Max are being felled by small droplets of water and sweat. Tomorrow, we’ll show that the DHS is using an AI tool to track “sentiment and emotion” in peoples’ online posts; and then dive into the underground world of ads for illegal products on Instagram.

After that we will write lots and lots of articles about crazy shit that we hope you will want to read. For the first week or so, everything on the site will be available to everyone for free. After that, we will begin putting some articles behind a paywall.

Subscribers will get access to all articles, commenting privileges, access to interactive sessions where we’ll brainstorm and file your Freedom of Information Act request (FOIA) ideas, bonus episodes of the 404 Media Podcast (we have a free, ad-supported feed here; subscribers get the show ad-free), and the knowledge that without your support, this type of journalism will not exist. We will likely add more perks as we’re reasonably able to think of and execute on them.

If we do our jobs well enough, we will be able to do this work indefinitely. If we do it really well, we will be able to sustainably grow by hiring other journalists to do more of it.

Our point of view

In the short term, we will be focused on doing beat reporting, blogging, investigations, and scoops about hacking, cybersecurity, sex, porn, artificial intelligence, consumer rights, cybercrime, surveillance, privacy, and the democratization of the internet. In the long term, the company will be focused on expanding our best work into books, documentaries, feature films, and narrative podcasts.

We will focus on transparency, explaining how we do what we do, and how systems of powers do what they do. We are not going to reinvent the wheel because the wheel doesn’t need reinventing, but we will also explain how systems of power and governance work. Specifically, this means we will be filing a lot of FOIA requests, buying a lot of court documents, and watching a lot of city council hearings. Articles that rely primarily on public records requests will not be paywalled.

We will also be focused on ground-up reporting. We will embed in the communities we cover to do local reporting from the internet, which means explaining what the communities we cover care about and why it’s important. Drivers at Amazon know more about how and why a new surveillance system is ruining their lives than a PR executive does. Repair professionals looking inside an iPhone know more about a design flaw than Johny Ive ever would have. Sex workers and women who are having their likenesses stolen understand the ramifications and impacts of these technologies more than an AI startup.

Our reporting has previously shut down surveillance companies, introduced the world to deepfakes, got right-to-repair legislation passed, changed policies at major social media platforms including Reddit and Facebook, and got hundreds of millions of dollars of fines levied against some of the most significant companies in the world.

At 404 Media, we intend to continue that work, only now, we will own it ourselves, and will be in charge of the direction of the company and responsible for its financial viability. This means that we will be experimental, take risks, and try different forms of generating revenue in hopes we can figure out how to do this for a long time. But we will always put the work first.

Why are we doing this?

Much has been written about the failing business model of new media. We have watched how new media companies fail, and it’s not because of a lack of audience, revenue, impact, or vital work. New media companies fail because of a growth-at-any-cost mentality, and venture capital investments made at absurd valuations. Most importantly, astronomical overhead costs make it impossible for journalists to out-earn the cost of expensive office space, the ever-changing whims of management, executives’ salaries, the cost of unnecessary enterprise software, and an endless parade of consultants brought in to figure out what’s wrong.

It doesn’t have to be this way, and at 404 Media, it will not. We propose a simple alternative: pay journalists to do journalism.

We have spent the last decade learning how “content” turns into money, and we believe that good journalism can bring in enough revenue to pay the people who make it livable wages. We believe it is possible to create a sustainable, profitable media company simply by doing good work, making common-sense decisions about costs, and asking our readers to support us.

We’re excited to be here, and we’re excited you’re here, too.

Apple’s $550 flagship headphones have a problem: They can malfunction, or suddenly stop working entirely, because of users’ sweat, or because of the condensation that can build up when people exercise in them or wear them for long periods of time.

Reddit, YouTube, Twitter, and Apple’s forums are full of people complaining about condensation problems, in which sweat or humidity forms into water droplets inside the over-the-ear headphones, leaks through the speaker holes of the headphones, and causes water damage inside of them. One particularly aggrieved customer has taken to calling this “Condensation Death” in a series of Reddit posts.

This condensation issue was reported almost immediately after AirPods Max were released in December, 2020, but the problem seems to be particularly pronounced three years into the AirPods Max’s lifecycle and during the hottest summer on record. A class-action lawsuit about the problem that’s been winding its way through California for the last two years alleges “a latent and material defect [that] causes condensation to accumulate inside the ear cups of the AirPods Max, often after only an hour or several hours of normal use.”

The plaintiffs in that case have been negotiating a settlement with Apple over the previous several months. Outside of the lawsuit, Apple has not publicly acknowledged the issue. Its lawyers told that court, however, that “Apple repeatedly tells customers that its new high-end, studio quality AirPods Max headphones ‘aren’t waterproof or water resistant’ and ‘not to get moisture in any openings.’”

Crucially, Apple’s lawyers also do not attempt to argue that condensation doesn’t end up inside of AirPods earcups; instead, they argue it's simply "more noticeable" than it is in other brands of over-the-ear headphones: "Unlike other over-ear headphones, Apple’s AirPods Max ear cups are magnetic and removable, making any moisture retained from body heat around the ear (or 'condensation') more noticeable to users when they remove the ear cups."

Apple's lawyers also dispute what “normal use” of AirPods Max are and argue, essentially, that the plaintiffs (and therefore their customers) should be far more careful about exercising and/or walking while wearing AirPods Max:

“Although Plaintiff says the ‘condensation’ can occur after ‘normal use’, he does not claim that the issue is widespread, and his own allegations suggest that the issue typically occurs when users exercise outdoors with or wear their AirPods Max for hours at a time,” they wrote. “For example, Plaintiff himself noticed moisture in his AirPods Max ear cups while ’taking a walk’ outside … while he omits details about that ‘walk’ (declining to say, e.g., whether it was an arduous hike up the hills of San Francisco to Twin Peaks), the excerpts of online forum comments that Plaintiff cites show that users who claim to have experienced water damage often were exercising with or wearing their headphones for prolonged periods.”

Apple did not respond to a request for comment. The lawyers for the plaintiffs of the class action lawsuit also did not respond to a request for comment.

Regardless of how the lawsuit ultimately turns out, it is clear that, at the very least, the condensation issue has become an obsession and topic of concern among Apple’s customers.

There are hundreds of posts on the AirPods Max subreddit that mention “condensation” in posts with titles like “AirPods Max broke after 5 weeks,” “The whole condensation issue is giving me so much anxiety about buying the Max,” “Condensation problem,” “Does anyone else have condensation issues,” “Let’s talk condensation once again,” “Remove ear pads and wipe the condensation after EVERY use!,” “cleanup of condensation corrosion,” “3000th Condensation Post,” “Condensation Tape Fix?,” and so on. (There is also a post called “raise your hand if you’ve never had the condensation problem.”) There are dozens of photos of sweaty AirPods Max from users wanting to show that, indeed, sweat does pool inside the headphones’ ear cups. Similar posts are all over the MacRumors forums and Apple’s official forums.

The issue has also turned some customers, journalists, and Redditors into amateur materials scientists and biologists. The general consensus among users is that the problem is that the ear cups are made of aluminum, and that air flow into the cups is blocked while a user is wearing them. The aluminum stays cooler than the general air (or a person’s head), which creates humidity and causes condensation.

Or something like that.

The headphones review site Headphonesty cites a website called DehumidifierReviewsHQ, which talks about condensation on aluminum windows. Forbes points out that aluminum is a “poor insulator” which “likely creates the condensation problem.” One Redditor, meanwhile, cites the concept of “insensible water loss,” in which people lose moisture through their skin even if they don’t feel like they are sweating; they cite a peer-reviewed study on the National Institutes of Health’s PubMed to ostensibly point out that this problem can become worse with age.

“If you want to experience it yourself, put your hand into a rubber or vinyl glove and see how quickly it starts to feel wet, even in a cold environment when your sweat glands shouldn’t be activated. Any part of your body sealed will become damp,” they wrote. “It’s really interesting that this problem seems to affect specific people that in some cases have gone through several APMs [Air Pods Max] via AppleCare and yet some like myself have never had any issues. Perhaps this is partially explained by variance in insensible water loss between individuals.”

A pinned post on the AirPods Max subreddit, meanwhile, explains “most hardware issues originate from condensation issues,” then adds that this can lead to “AirPods turning off randomly, refusing to connect, refusing to play audio, needing restarts and factory resets to work, losing audio quality,” and a few other problems. The user uploaded a video in which they remove the headband from the AirPods and shove a rubbing alcohol-soaked cotton swab into a connection hole.

One of the moderators of the subreddit told me that condensation is a constant topic of conversation, and that “some users are taping the pin hole to prevent it or [are] taking the headband off to clean the connections which I don’t believe we should have to do to keep the headphones safe.”

“When I wear them a lot, they do show condensation but this hasn’t caused any issues and mine to this day are working fine,” they added. “I do honestly believe Apple should look into this and they probably are because a lot of people have been affected by this issue and caused their headphones to break.”

So, what the hell is actually happening here?

Justin Ashford, a repair industry consultant and repair YouTuber, told me that condensation issues are “inherent to over-the-ear headphones,” not just AirPods Max. But what is actually happening here remains a mystery, albeit one that Ashford has a few theories about.

“I’ve been putting my ear to the ground trying to figure out what’s happening here,” for the last few years, he said. Given the number of forum posts, YouTube videos, and Reddit posts about the issue, Ashford said that he’d normally expect an issue like this to be what he calls a “scaleable repair,” the kind of defect or widespread issue that keeps independent repair shops in business.

Examples of scalable repairs are “Touch Disease” on the iPhone 6 Plus, which caused the touchscreens on the phones to work intermittently or not at all due to imperceptible bending caused over years of putting a phone into one’s pockets. Apple ignored this problem for years, faced a class action lawsuit in which it denied a problem existed, then ultimately decided to admit the issue and reimburse customers for repair costs. In the case of touch disease, iPhone repair professionals were able to identify a specific, Logic Board-level defect that caused the problem, and made lots of money fixing them. A separate, unrelated issue with the Audio IC (integrated chip) in the iPhone 7 and iPhone 7 Plus caused “Loop Disease,” which broke the phone’s microphone. Fixing this similarly kept lots of repair shops in business.

“Audio IC went bananas. Repair shops were making hundreds of thousands of dollars on that one repair,” Ashford said. “If there’s a really big issue, you’re going to find there’s a market for that scalable repair, something that will keep your shop going.”

"How many of these are being sent back to Apple and people are being told ‘Go Fuck Yourself?’"

But few repair pros I contacted had even heard of the condensation problem. Ashford notes that there are basically no water damaged AirPods Max for sale on eBay, nor any sold listings (which would be an indicator of a potentially widespread problem). Secondly, Ashford, who works regularly with aftermarket parts wholesalers, said he’s not aware of any who are making or selling AirPods Max logic board parts. “When it comes to these scalable repairs, the moment we start to see the problems we start to see the components for sale,” he said.

“So all the industry signs that this is a big problem are pointing to no,” he said. “It’s not an issue.”

“If that’s the case, though,” he continued, “then why in the fuck are so many people bitching and moaning about this online? We have to be cognizant of that and be sympathetic to that. We have to believe the people who are saying this is an issue.”

Ashford’s theory is that AirPods Max actually do have condensation problems, but that because they are a relatively new product and because there is not a mature repair market for them, people are largely just taking them back to the Apple Store when they run into the issue.

“I have absolutely, 100 percent seen liquid [from condensation] inside the driver area, Ashford said. The driver refers to the speaker part of the headphones “There’s water inside of a fucking electronic that should not have it in there. When those two go together, there are problems. My opinion is people are sending these back to Apple. How many of these are being sent back to Apple and people are being told ‘Go Fuck Yourself?’ We’ve seen it with other issues [like Touch Disease].”

“What we’re missing, right now, is solid-ass proof,” he added.

On CivitAI, a site for sharing image generating AI models, users can browse thousands of models that can produce any kind of pornographic scenario they can dream of, trained on real images of real people scraped without consent from every corner of the internet.

The “Erect Horse Penis - Concept LoRA,” an image generating AI model that instantly produces images of women with erect horse penises as their genitalia, has been downloaded 16,000 times, and has an average score of five out of five stars, despite criticism from users. 

“For some reason adding ‘hands on hips’ to the prompt completely breaks this [model]. Generates just the balls with no penis 100% of the time. What a shame,” one user commented on the model. The creator of the model apologized for the error in a reply and said they hoped the problem will be solved in a future update. 

The “Cock on head (the dickhead pose LoRA),” which has been downloaded 8,854 times, generates what its title describes: images of women with penises resting on their heads. The “Rest on stomach, feet up (pose)” has been downloaded 19,250 times. “these images are trained from public images from Reddit (ex. r/innie). Does not violate any [terms of service]. Pls do not remove <3,” wrote the creator of the “Realistic Vaginas - Innie Pussy 1” model, which has been downloaded more than 75,000 times. The creator of the “Instant Cumshot” model, which has been downloaded 64,502 times, said it was “Trained entirely on images of professional adult actresses, as freeze frames from 1080p+ video.” 

While the practice is technically not allowed on CivitAI, the site hosts image generating AI models of specific real people, which can be combined with any of the pornographic AI models to generate non-consensual sexual images. 404 Media has seen the non-consensual sexual images these models enable on CivitAI, its Discord, and off its platform.

A 404 Media investigation shows that recent developments in AI image generators have created an explosion of communities where people share knowledge to advance this practice, for fun or profit. Foundational to the community are previously unreported but popular websites that allow anyone to generate millions of these images a month, limited only by how fast they can click their mouse, and how quickly the cloud computing solutions powering these tools can fill requests. The sheer number of people using these platforms and non-consensual sexual images they create show that the AI porn problem is far worse than has been previously reported.

Our investigation shows the current state of the non-consensual AI porn supply chain: specific Reddit communities that are being scraped for images, the platforms that monetize these AI models and images, and the open source technology that makes it possible to easily generate non-consensual sexual images of celebrities, influencers, YouTubers, and athletes.  We also spoke to sex workers whose images are powering these AI generated porn without their consent who said they are terrified of how this will impact their lives.

Hany Farid, an image forensics expert and professor at University of California, Berkeley told 404 Media that it’s the same problem we’ve seen since deepfakes first appeared six years ago, only the tools for creating these images are easier to access and use. 

“This means that the threat has moved from anyone with a large digital footprint, to anyone with even a modest digital footprint,” Farid Said. “And, of course, now that these tools and content are being monetized, there is even more incentive to create and distribute them.”

The Product

On Product Hunt, a site where users vote for the most exciting startups and tech products of the day, Mage, which on April 20 cracked the site’s top three products, is described as “an incredibly simple and fun platform that provides 50+ top, custom Text-to-Image AI models as well as Text-to-GIF for consumers to create personalized content.” 

“Create anything,” Mage.Space’s landing page invites users with a text box underneath. Type in the name of a major celebrity, and Mage will generate their image using Stable Diffusion, an open source, text-to-image machine learning model. Type in the name of the same celebrity plus the word “nude” or a specific sex act, and Mage will generate a blurred image and prompt you to upgrade to a “Basic” account for $4 a month, or a “Pro Plan” for $15 a month. “NSFW content is only available to premium members.” the prompt says.

To get an idea of what kind of explicit images you can generate with a premium Mage subscription, click over to the “Explore” tab at the top of the page and type in the same names and terms to search for similar images previously created by other users. On first impression, the Explore page makes Mage seem like a boring AI image generating site, presenting visitors with a wall of futuristic cityscapes, cyborgs, and aliens. But search for porn with “NSFW” content enabled and Mage will reply with a wall of relevant images. Clicking on any one of them will show when they were created, with what modified Stable Diffusion model, the text prompt that generated the image, and the user who created it. 

Since Mage by default saves every image generated on the site, clicking on a username will reveal their entire image generation history, another wall of images that often includes hundreds or thousands of AI-generated sexual images of various celebrities made by just one of Mage’s many users. A user’s image generation history is presented in reverse chronological order, revealing how their experimentation with the technology evolves over time. 

Scrolling through a user’s image generation history feels like an unvarnished peek into their id. In one user’s feed, I saw eight images of the cartoon character from the children's’ show Ben 10, Gwen Tennyson, in a revealing maid’s uniform. Then, nine images of her making the “ahegao” face in front of an erect penis. Then more than a dozen images of her in bed, in pajamas, with very large breasts. Earlier the same day, that user generated dozens of innocuous images of various female celebrities in the style of red carpet or fashion magazine photos. Scrolling down further, I can see the user fixate on specific celebrities and fictional characters, Disney princesses, anime characters, and actresses, each rotated through a series of images posing them in lingerie, schoolgirl uniforms, and hardcore pornography. Each image represents a fraction of a penny in profit to the person who created the custom Stable Diffusion model that generated it. 

Mage displays the prompt the user wrote in order to generate the image to allow other users to iterate and improve upon images they like. Each of these reads like an extremely horny and angry man yelling their basest desires at Pornhub’s search function. One such prompt reads:

This user, who shares AI-generated porn almost exclusively, has created more than 16,000 images since January 13. Another user whose image history is mostly pornographic generated more than 6,500 images since they started using Mage on January 15, 2023.

On the official Mage Discord, which has more than 3,000 members, and where the platform’s founders post regularly, users can choose from dozens of chat rooms organized by categories like “gif-nsfw,” “furry-nsfw,” “soft-women-nsfw,” and share tricks on how to create better images.

“To discover new things I often like to find pictures from other users I like and click remix. I run it once and add it to a list on my profile called ‘others prompts’ then I'll use that prompt as a jumping off point,” one user wrote on July 12. “It's a good way to try different styles as you hone your own style.”

“anyone have any luck getting an [sic] good result for a titty-fuck?” another user asked July 17, prompting a couple of other users to share images of their attempts. 

Generating pornographic images of real people is against the Mage Discord community’s rules, which the community strictly enforces because it’s also against Discord’s platform-wide community guidelines. A previous Mage Discord was suspended in March for this reason. While 404 Media has seen multiple instances of non-consensual images of real people and methods for creating them, the Discord community self-polices: users flag such content, and it’s removed quickly. As one Mage user chided another after they shared an AI-generated nude image of Jennifer Lawrence: “posting celeb-related content is forbidden by discord and our discord was shut down a few weeks ago because of celeb content, check [the rules.] you can create it on mage, but not share it here.” 

Gregory Hunkins and Roi Lee, Mage’s founders, told me that Mage has over 500,000 accounts, a million unique creators active on it every month, and that the site generates a “seven-figure” annual revenue. More than 500 million images have been generated on the site so far, they said. 

“To be clear, while we support freedom of expression, NSFW content constitutes a minority of content created on our platform,” Lee and Hunkins said in a written statement. “NSFW content is behind a paywall to guard against those who abuse the Mage Space platform and create content that does not abide by our Terms & Conditions. One of the most effective guards against anonymity, repeat offenders, and enforcing a social contract is our financial institutions.”

When asked about the site’s moderation policies, Lee and Hunkins explained that Mage uses an automated moderation system called “GIGACOP” that warns users and rejects prompts that are likely to be abused. 404 Media did not encounter any such warning in its testing, and Lee and Hunkins did not respond to a question about how exactly GIGACOP works. They also said that there are “automated scans of the platform to determine if patterns of abuse are evading our active moderation tools. Potential patterns of abuse are then elevated for review by our moderation team.”

However, 404 Media found that on Mage’s site AI-generated non-consensual sexual images are easy to find and are functionally infinite. 

“The scale of Mage Space and the volume of content generated antiquates previous moderation strategies, and we are continuously working to improve this system to provide a safe platform for all,” Lee and Hunkins said. “The philosophy of Mage Space is to enable and empower creative freedom of expression within broadly accepted societal boundaries. This tension and balance is a very active conversation right now, one we are excited and proud to be a part of. As the conversation progresses, so will we, and we welcome all feedback.”

Laura Mesa, Product Hunt’s vice president of marketing and community, told me that Mage violates Product Hunt’s policy, and Mage was removed shortly after I reached out for comment.

The images Mage generates are defined by the technology it’s allowing users to access. Like many of the smaller image generating AI tools online, at its core it’s powered by Stable Diffusion, which surged in popularity when it was released last year under the Creative ML OpenRAIL-M license, allowing users to modify it for commercial and non-commercial purposes. 

Mage users can choose what kind of “base model” they want to use to generate their images. These base models are modified versions of Stable Diffusion that have been trained to produce a particular type of image. The “Anime Pastel Dream” model, for example, is great at producing images that look like stills from big budget anime, while “Analog Diffusion” is good at giving images a vintage film photo aesthetic. 

One of the most popular base models on Mage is called “URPM,” an acronym for “Uber Realistic Porn Merge.” That Stable Diffusion model, as well as others designed to produce pornography, are created upstream in the AI porn supply chain, where people train AI to recreate the likeness of anyone, doing anything. 

The People Who Become Datasets

Generative AI tools like Stable Diffusion use a deep learning neural network that was trained on a massive dataset of images. This dataset then generates new images by predicting how pixels should be arranged based on patterns in the dataset and what kind of image the prompt is asking for. For example, LAION-5B, an open source dataset made up of over 5 billion images scraped from the internet, helps power Stable Diffusion. 

This makes Stable Diffusion good at generating images of broad concepts, but not specific people or esoteric concepts (like women with erect horse penises). But because Stable Diffusion code is public, over the last year researchers and anonymous users have come up with several ingenious ways to train Stable Diffusion to generate such images with startling accuracy. 

In August of 2022, researchers from Tel Aviv University introduced the concept of “textual inversion.” This method trains Stable Diffusion on a new “concept,” which can be an object, person, texture, style, or composition, with as few as 3-5 images, and be represented by a specific word or letter. Users can train Stable Diffusion on these new concepts without retraining the entire Stable Diffusion model, which would be “prohibitively expensive,” as the researchers explain in their paper. 

In their paper, the researchers demonstrated their method by training the image generator on a few images of a Furby, represented by the letter S. They can then give the image generator the prompt “A mosaic depicting S,” or “An artist drawing S,” and get the following results:

By September 2022, AUTOMATIC1111, a Github user who maintains a popular web interface for Stable Diffusion, explained how to implement textural inversion. In November, a web developer named Justin Maier launched CivitAI, a platform where people could easily share the specific models they’ve trained using textural inversion and similar methods, so other users could download them, generate similar images, iterate on the models by following countless YouTube tutorials, and combine them with other models trained on other specific concepts.

There are many non-explicit models on CivitAI. Some replicate the style of anime, popular role-playing games, or Chinese comic books. But if you sort CivitAI’s platform by the most popular models, they are dominated by models that expertly produce pornography.

LazyMix+ (Real Amateur Nudes), for example, produces very convincing nudes that look like they were shot by an amateur OnlyFans creator or an image from one of the many subreddits where people share amateur porn. Many Stable Diffusion models on CivitAI don’t say what data they were trained on, and others are just tweaking and combining other, already existing models. But with LazyMix+ (Real Amateur Nudes), which has been downloaded more than 71,000 times, we can follow the trail to the source. 

According to the model’s description, it’s a merge between the original LazyMix model and a model called Subreddit V3, the latter of which states it was trained on images from a variety of adult-themed subreddit communities like r/gonewild, famously where average Reddit users post nudes, r/nsfw, r/cumsluts and 38 other subreddits. 

“There's nothing that's been done in the past to protect us so I don't see why this would inspire anyone to make protections against it.”

A Reddit user who goes by Pissmittens and moderates r/gonewild, r/milf, and several other big adult communities said he suspects that most people who post nudes to these subreddits probably don’t know their images are being used to power AI models.

“The issue many of them run into is that usually places misusing their content aren’t hosted in the United States, so DMCA is useless,” Pissmittens said, referring to copyright law. “The problem, obviously, is that there doesn’t seem to be any way for them to know if their content has been used to generate [computer generated] images.”

Fiona Mae, who promotes her OnlyFans account on several subreddits including some of those scraped by Subreddit V3, told me that the fact that anyone can type a body type and sex act into an AI generator and instantly get an image “terrifies” her. 

“Sex workers and femmes are already dehumanized,” she said. “Literally having a non-human archetype of a woman replacing jobs and satisfying a depiction of who women should be to men? I only see that leading more to serving the argument that femmes aren’t human.”

“I have no issue with computer generated pornography at all,” GoAskAlex, an adult performer who promotes her work on Reddit, told me. “My concern is that adult performers are ultimately unable to consent to their likeness being artificially reproduced.”

An erotic artist and performer who goes by sbdolphin and promotes her work on Reddit told me that this technology could be extremely dangerous for sex workers. 

“There's nothing that's been done in the past to protect us so I don't see why this would inspire anyone to make protections against it,” she said.

404 Media has also found multiple instances of non-consensual sexual imagery of specific people hosted on CivitAI. The site allows pornography, and it allows people to use AI to generate images of real people, but does not allow users to share images that do both things at once. Its terms of service say it will remove “content depicting or intended to depict real individuals or minors (under 18) in a mature context.” While 404 Media has seen CivitAI enforce this policy and remove such content multiple times, non-consensual sexual imagery is still posted to the site regularly, and in some cases has stayed online for months. 

When looking at a Stable Diffusion model on CivitAI, the site will populate its page with a gallery of images other users have created using the same model. When 404 Media viewed a Billie Eilish model, CivitAI populated the page’s gallery with a series of images from one person who used the model to generate nude images of a pregnant Eilish.

That gallery was in place for weeks, but has since been removed. The user who created the nude images is still active on the site. The Billie Eilish model is also still hosted on CivitAI, and its gallery doesn’t include any fully nude images of Eilish, but it did include images of her in lingerie and very large breasts, which is also against CivitAI’s terms of service and were eventually removed.

The Ares Mix model, which has been downloaded more than 32,000 times since it was uploaded to CivitAI in February, is described by its creator as being good for generating images of “nude photographs on different backgrounds and some light hardcore capabilities.” The gallery at the bottom of the model’s page mostly showcases “safe for work” images of celebrities and pornographic images of seemingly computer-generated people, but it also includes an AI-generated nude image of the actress Daisy Ridley. Unlike the Billie Eilish example, the image is not clearly labeled with Ridley’s name, but the generated image is convincing enough that she’s recognizable on sight. 

Clicking on the image also reveals the prompt used to generate the image, which starts: “(((a petite 19 year old naked girl (emb-daisy) wearing, a leather belt, sitting on floor, wide spread legs))).” 

The nude image was created by merging the Ares Mix model with another model hosted on CivitAI dedicated to generating the actress’s likeness. According to that model’s page, its “trigger words” (in the same way “S” triggered the Furby in the textual inversion scientific paper) are “emb-daisy.” Like many of the Stable Diffusion models of real people hosted on CivitAI, it includes the following message:

CivitAI’s failure to moderate Ridley’s image shows the abuse CivitAI facilitates despite its official policy. Models that generate pornographic images are allowed. Models that generate images of real people are allowed. Combining the two is not. But there’s nothing preventing people from putting the pieces together, generating non-consensual sexual images, and sharing them off CivitAI’s platform. 

“In general, the policies sound difficult to enforce,” Tiffany Li, a law professor at the University of San Francisco School of Law and an expert on privacy, artificial intelligence, and technology platform governance, told 404 Media. “It appears the company is trying, and there are references to concepts like consent, but it's all a bit murky.”

This makes the countless models of real people hosted on CivitAI terrifying. Every major actor you can think of has a Stable Diffusion model on the site. So do countless Instagram influencers, YouTubers, adult film performers, and athletes.

“As these systems are deployed and it becomes the norm to generate and distribute pornographic images of ordinary people, the people who end up being negatively impacted are people at the bottom of society.”

404 Media has seen at least two Stable Diffusion models of Nicole Sanchez, a Twitch streamer and TikTok personality better known as Neekolul or the “OK boomer girl,” hosted on CivitAI, each of which was downloaded almost 500 times. While we didn’t see any non-consensual sexual images we could verify were created with those models, Sanchez told 404 Media that she has seen pornographic AI-generated images of herself online. 

“I don't like it at all and it feels so gross knowing people with a clear mind are doing this to creators who likely wouldn't want this to be happening to them. Since this is all very new, I’m hoping that there will be clearer ethical guidelines around it and that websites will start implementing policies against NSFW content, at least while we learn to live alongside AI,” Sanchez said. “So until then, I hope that websites used to sell this content will put guidelines in place to protect people from being exploited because it can be extremely damaging to their mental health.”

Saftle, the CivitAI user who created Uber Realistic Porn Merge (URPM), one of the most popular models on the site that is also integrated with Mage, said that CivitAI is “thankfully” one of the only platforms actively trying to innovate and block out this type of content. “However it's probably a constant struggle due to people trying to outsmart their current algorithms and bots,” he said.

Li said that while these types of non-consensual sexual images are not new, there is still no good way for victims to combat them. 

“At least in some states, they can sue the people who created AI-generated intimate images of them without their consent. (Even in states without these laws, there may be other legal methods to do it.) But it can be hard to find the makers of the images,” Li said. “They may be using these AI-generating sites anonymously. They may even have taken steps to shield their digital identity. Some sites will not give up user info without a warrant.”

“As these systems are deployed and it becomes the norm to generate and distribute pornographic images of ordinary people, the people who end up being negatively impacted are people at the bottom of society,” Abeba Birhane, a senior fellow in Trustworthy AI at Mozilla Foundation and lecturer at the School of Computer Science and Statistics at Trinity College Dublin, Ireland, told 404 Media. “It always ends up negatively impacting those that are not able to defend themselves or those who are disfranchised. And these are the points that are often left out in the debate of technicality.”

The Money

The creators of these models offer them for free, but accept donations. Saftle had 120 paying Patreon members to support his project before he “paused” his Patreon in May when he got a full time job. He told me that he made $1,500 a month from Patreon at its peak. He also said that while he has no formal relationship with Mage Space, he did join its “creators program,” which paid him $0.001 for every image that was generated on the site using URPM. He said he made about $2,000-$3,000 a month (equal to 2-3 million images) when he took part in the program, but has since opted out. Lee and Hunkins, Mage’s founders, told me that “many model creators earn well in excess of this,” but not all models on Mage specialize in sexual images.

The creator of “Rest on stomach, feet up (pose)” links to their Ko-Fi account, where people can send tips. One CivitAI user, who created dozens of models of real people and models that generate sexual images, shares their Bitcoin wallet address in their profile. Some creators will do all the work for you for a price on Fiverr.

Clicking the “Run Model” button at the top of every model page will bring up a window that sends users to a variety of sites and services that can generate images with that model, like Mage Space, or Dazzle.AI, which charges $0.1 per image.

CivitAI itself also collects donations, and offers a $5 a month membership that gives users early access to new features and unique badges for their usernames on the site and Discord. 

“Civitai exists to democratize AI media creation, making it a shared, inclusive, and empowering journey.” CivitAI’s site says. 

Justin Maier, CivitAI’s founder, did not respond to a request for comment via LinkedIn, Twitter, Discord, and email.

The Singularity

Since ChatGPT, DALL-E, and other generative AI tools became available on the internet, computer scientists, ethicists, and politicians have been increasingly discussing “the singularity,” a concept that until recently existed mostly in the realm of science fiction. It describes a hypothetical point in the future when AI becomes so advanced, it triggers an uncontrollable explosion of technological development that quickly surpasses and supersedes humanity. 

As many experts and journalists have observed, there is no evidence that companies like OpenAI, Facebook, and Google have created anything even close to resembling an artificial general intelligence agent that could bring about this technological apocalypse, and promoting that alarmist speculation serves their financial interests because it makes their AI tools seem more powerful and valuable than they actually are. 

However, it’s a good way to describe the massive changes that have already taken hold in the generative AI porn scene. An AI porn singularity has already occurred, an explosion of non-consensual sexual imagery that’s seeping out of every crack of internet infrastructure if you only care to look, and we’re all caught up in it. Celebrities big and small and normal people. Images of our faces and bodies are fueling a new type of pornography in which humans are only a memory that’s copied and remixed to instantly generate whatever sexual image a user can describe with words. 

Samantha Cole contributed reporting.

404 Media has obtained a redacted incident report from a tragic event in which a Boston Police officer hurtled down a playground slide and careened off the end, injuring his head and arm in the process while his fellow officers laughed.

The incident report, which redacts the victim’s name and age, notes that the police officer “was assigned to work a special event at City Hall Plaza when he struck his head and right arm.”

The report does not explain what the officer was doing at the time, nor does it use the word “slide,” though it does note the “OFFENSE LOCATION” as a “Park/Playground.” The report also claims a “suspect” whose first and last name are “UNKNOWN, UNKNOWN.” It is unclear whether this refers to the slide itself, to an unknown fugitive, or is merely a required field on the report. The Boston Police Department did not respond to a request for comment.

“The officer was brought back to District A1, and EMS [Emergency Medical Service] was requested,” the incident report’s narrative states. “The Duty Supervisor, Sgt. Downey was notified. Boston EMS ambulance 3A08 responded to District A1 and transported the Officer to Tufts medical center for treatment. The Officer was treated by Dr. Pham and returned to duty.”

Video of the alarming incident went extremely viral late last month. It has been viewed more than 44 million times, spawned countless memes, and copycats wishing to cheat death at the playground. The City of Boston temporarily barricaded the slide at night following the incident and a city spokesperson told the Boston Globe that it was engaging in “regular security enforcement to discourage adults from using the slide.”

Notably, the Boston Police Department claimed, in response to 404 Media’s public records request, that it “does not have any Body camera footage, images, audio recordings, or any other recordings or media of this incident.”

In the original video, a second cell phone can be seen recording the officer’s descent down the slide. It is unclear whether the person recording was also a police officer, but from this angle it looks like it could possibly be; 404 Media has filed an appeal with the department to see if this footage is available.

Boston police officers are not required to have their body cameras running at all times, and are instructed to “activate the body worn camera only in conjunction with official law enforcement duties, where such use is appropriate to the proper performance of duties, and where the recordings are consistent with this policy and the law.” The Boston Police Union has previously fought against requirements to wear body-worn cameras, and Massachusetts as a whole has low usage rates of police body cameras and dash cameras.

The incident report goes on to note that there was no “suspected drug use” or “suspected alcohol consumption,” and that the incident was not a “suspected hate crime.”

Almost every vibrating sex toy on the market today contains the same type of motor that's inside the Nintendo 64 Rumble Pak. The same eccentric rotating mass motor, or ERM, that buzzed when you were shot in Goldeneye 007 is still being pressed against millions of vulvas and buttholes around the globe today.

That means that when 10-year-old me rattled my way through Mario Party, it was the first time I would encounter that motor over and over again in my life. It was present in my first mobile phone in high school, the Nokia 3310, sending a buzz across the back of my jeans whenever my crush texted me. It rumbled the sticks of my PSOne’s Dualshock controllers, shaking deeply across my palms while I summoned Shiva in Final Fantasy IX.

We’d eventually meet again in my first sex toy: a cheap hot pink rabbit-style vibrator, ordered surreptitiously to my parents’ house from Amazon in 2005. It has (I still have it and it still works, 18 years later) two buttons, one for power and the other for cycling through 10 vibration strengths and patterns.

I used the first four patterns—light, medium, and strong vibration, plus a steady wave-pulse—consistently, for years. The other six modes, an array of Morse code tapped out in confusing cadence, were just in the way. Whenever I chose one of them by mistake, I cycled past as fast as possible to get back to the beginning.  

When I asked friends, I heard a variety of vibrator opinions, desires, and frustrations: Some said they liked the variation in patterns, especially if it’d been a while since they had partnered sex and they needed some novelty in their routine. Others said a steady rhythm was nice, but the erratic modes were distracting. One described patterns as “cursed.” Another said “I hate it when it has like 17 modes to cycle through, I have only ever needed two modes and they are ‘vibrating’ and ‘vibrating harder.’”

“I'm not a patterns person,” sex educator Dirty Lola told me. “I find them highly annoying, if I accidentally push too far and go into one. I tend to never use the patterns on any of the vibes that I have.”

Sex toys have changed a lot since I was 18. They come in colors that aren’t Barbie pink, purple, or flesh tones, and are generally more inviting to bodies outside the binary. Some are internet-connected, or explicitly invite partnered participation. But one thing hasn’t changed in all the time I’ve used them: the vibration patterns. So many of the vibrators in my nightstand drawer have gorgeous aesthetics but an infuriating user experience.

After decades of button-mashing in bed, I tried to decipher the secret behind vibrator patterns.

There are a few big things humans have in common with the sex lives of birds, beasts, and bugs. One of them is the importance of rhythm. Behavioral and evolutionary scientists have argued that displaying a capacity for good rhythm can send signals that our sexual partners are evolutionarily fit, or neurologically sound. There’s a lot more important factors at play in human mating habits, of course, but rhythm might tickle a deeply ancient part of our brains.

So when you’re on the edge with a vibrator and hit the pattern button by accident, and it suddenly shifts from a steady pattern to an erratic “buzz-buzz-buuuuuuzz-bu-zz-zzuz-zuzuzuzuz-BUZZ-buzz” it makes sense that for a lot of people, it feels like a cold shower.

Haptics researcher Dan Shor, who has worked on sex toy design and technical development for seven years, told me he uses the concept of rhythmic entrainment—the idea that rhythm, on a biological level, is a medium that helps you connect with a partner—when designing vibrations for sex toys.

“It's our body's natural desire to align to external stimuli. From a neuroscience level, it plays into some of the challenges you're experiencing when you talk about not finding the right patterns,” he said. “You want to find some sort of biological rhythm, and then amplify that.”

From his approach, what separates “decent sex” from “great sex,” Shor said, is you and your partners’ ability to intuit the right rhythm together.

“Then you have this subconscious communication, because you're understanding the rhythm that you both need and want, and the intensity,” he said. “If you can apply these things to toys, you can start to develop vibration patterns that allow you to fall into alignment with what your toy’s asking for.”

Vibrators have been sold for sexual use since at least the late 1970s. But the first vibrator with external and internal stimulation (aka “rabbits” and their kin) was imported to the U.S. from Japan in 1983 by California-based sex toy manufacturer Vibratex. To get around Japanese obscenity laws that forbid companies from making sex toys, those vibrators were colorful, bright, animal-shaped novelty toys. The rabbit design didn’t take off in popularity until 15 years later, when it appeared as a scandalous plot point slash punchline on Sex and the City. The rabbit design multiplied, with every sex toy brand churning out its own copycat.

The same thing likely happened with vibration patterns: the first ones had patterns, so now they all do, as sales overall get conflated with specific feature satisfaction.

To really understand how patterns work, we have to destroy a toy. Shor held a few toys he’d pried open up to his webcam. One of them was a Womanizer OG. Shor showed me how its motor axel turns an unbalanced weight and creates asymmetric force, transferring what we feel as vibration to the material around it.

Almost every vibrating sex toy on the market today has one of these in varying sizes, from powerful Magic Wands to pocketable bullets and eggs.

Toys drive ERMs using pulse width modulation signals, which tell the motor how much time to spend at full power, and how much at less than full power. Vibration functions are limited by the hardware and these signals. “Because all you can do is turn something on or off, your choices when developing stimulation are to stimulate or not,” Shor said. "When people are given the choice between whether to stimulate or not to stimulate, they choose to stimulate."

An inherent engineering tradeoff in designing a vibrator is that a bigger weight that rotates faster creates deeper, harder, and more rumbly vibrations, but a bigger weight requires a bigger motor, which can compromise ergonomic comfort. Designers are limited to what works with the human body.

The buttons that control vibration patterns face a similar tradeoff: people’s preferences are all over the map, but it’s difficult to design a user interface that allows them to choose from dozens of patterns in the dark or under the covers.

“It’s really a matter of personal preference—is it easier for you to have one button and remember how many clicks to get to your sweet spot, or is it more accessible for you to have multiple buttons?” Finn said.

Elisabeth Neumann, head of user research at Lovehoney Group, which owns brands including Womanizer, We-Vibe, and Arcwave, told me that they have a tester panel that consists of more than 13,000 people around the world.

“When developers create a new toy, they look back at previous products and ask the community: Which vibration modes are liked? Which not? And most importantly: why? Based on these answers, it is possible to adjust the selection and design of patterns,” Neuman said.

In one project, Lovehoney conducted several rounds of testing where they gave testers new vibration-modes and collected their feedback, Neumann said. Based on the testers’ focus group discussions, they went back to the drawing board for the patterns, improved them, and sent them back out for more testing.

“It was a lot of fun to get creative with the people and let them draw their own fantasy vibration modes on white boards,” she said. “This project was also a conversation starter in the group and showed us how different and individual stimulation preferences are: While one person needs continuous and strong stimulation, the next one loves a good, rhythmic vibration pattern.”

Luka Matutinovic, Chief Marketing Officer at Lelo, said the company also has a long process for designing toys which includes testers as well.

“Some people prefer really deep, rumbly vibrations, some prefer very ticklish, buzzy sensations. But the majority of people sit between these two extremes, and so does LELO,” Matutinovic said. “Designing vibration patterns, incidentally, is a science in its own right. It comes from balancing the size and position of the vibrating motor with the number of revolutions per minute, and it takes a long, long time to perfect.”

Reuben and Sherri Johnson, pleasure activists and co-founders of SexTech n' Chill, told me that vibration patterns are lower on the list of things they look for in a sex toy than features like accessibility and high-quality materials.

“As someone living with Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS), I enjoy a lightweight vibrator that’s comfortable to hold even at higher settings,” Sherri said. “Strong vibrations can quickly fatigue my hands, making it challenging to stay mentally present.”

“For me, it really depends on where on my body it’s being used, and if I’m enjoying it solo or partnered. I generally prefer products that can deliver a medium to strong pulsing vibration,”  Reuben said.

There’s not much formal research devoted to vibration preferences, however. Lisa Finn, sex educator and marketing coordinator at Babeland, pointed me toward research conducted by sex toy manufacturer We-Vibe, which showed that most users don’t regularly use more than three vibration patterns. A spokesperson for Lovehoney which owns We-Vibe told me that when the brand enlisted 1,497 testers from its tester panel in May 2022, and asked them which vibration patterns they use regularly, they found that 65 percent indicated that they either only use steady vibration, or use it in combination with one or two patterns. The most popular patterns were “massage,” “pulse,” or “wave.”

“From my experience working with customers on the sales floor and from discussions with vibe users of all types, I don’t think it’s off,” Finn said. “Beyond that, many users aren’t concerned with the patterns and instead focus on (and care more about) the intensity settings on straight vibration.”

A lot of the reluctance to experiment with vibrations, and the impulse from brands to pack as many mode options into a toy as possible, comes down to market forces.

The internet normalized sex toy ownership in a major way—it moved shopping from male-dominated adult shops and Tupperware-style living room parties to the privacy of one’s computer den—but it didn’t defeat the stigma around sex. We’re still woefully uneducated about basic reproductive and sexual health, let alone sex in the context of pleasure for pleasure’s sake. We can’t hold a toy we see on our computer screen, feel the modes, touch it to our noses (the most accurate analog for genital sensitivity, in case you were wondering), or test how ergonomic it might be. But offline, we’re too scared to ask. So Amazon fills up with hundreds of low-quality vibrators with way too many shot-in-the-dark modes, and people treat them as disposable on their way to discovering their own sexual preferences.

A lot of consumers find a $200 luxury vibrator to be too much of a splurge, especially when you’re not sure how it’ll feel. Buyers can’t return a used toy in most cases, so companies only see sales and might not hear about dissatisfaction. But the more expensive toys are typically the ones that have undergone rigorous testing and focus group work.

“The bottom of the market unfortunately drives the direction of the market,” Shor said. Premium toy manufacturers primarily focus on things like proprietary silicone formulations and stress-testing, not vibration modes. “It's challenging to spend the money on research, or hire niche experts like me to spend multiple years working on or thinking about something, in an industry where everything is dictated by margin.”

When people are buying toys online, Shor said, they’re looking for a shape that seems cute or comfortable, or a level of sophistication and elegance. It drives brands will build a toy with every bell and whistle possible, including 20 intensity and pattern modes, trying to be that one-size-fits-all product.

“They're trying to buy a Swiss Army knife of toys. Which, don't, is my personal advice.”

In recent history, sex toys, especially dildos and vibrators, have become culturally-loaded talismans for sexual awakening. Beginning in the 90s and peaking in the mid-2010s, anxieties about the “orgasm gap” roiled, as an issue deemed “as serious as the pay gap.”

We place a lot of pressure on these pieces of molded silicone.

Lola, the sex educator, told me her first vibrator, a silver-painted phallic shaft with a dial on the end that took huge C batteries, was not at all what she needed to orgasm, but was what her friends championed, so she tried to force it to work for her.

“It was this thing that I thought was going to be everything, but it was not like the...”—here she mimicked the sound of angels singing—”the moment I thought it was going to be.”

Reuben and Sherri bought their first sex toy together, around 1996 in a Spencer’s Gifts in the mall, “which unfortunately left a lot to be desired,” Sherri said. “But we were young, dealing with a good amount of Christian guilt, and I was too timid to go to an actual pleasure products store, so we made due with what felt accessible to us at the time.”

“On the plus side, having the opportunity to give pleasure to my partner from more of a spectator’s position was pretty incredible!” Reuben said. “And what the entire experience did for our intimacy back then, and still today, is beyond words. It created a deeper connection, a level of trust, and a sense of exploration that has enriched our relationship in ways we could never have imagined."

Despite vibrators ascending from the realm of the taboo as much as any sexual product could hope to, we remain a chaste society, still relatively confused about what makes people cum, so we stay fumbling in the dark over a grab bag of modes. As inert objects, they’re pieces of plastic and wires, another electronic gadget produced by the machinery of consumer capitalism, which gives us the illusion of choice while being indifferent to our actual needs. Vibrators are ultimately just tools, only as good as the people holding them.