A Reddit user forced Google's Gemini chatbot to bypass built-in safety restrictions and recommend a resource with pirated software. To do this, he acted out a scenario as if a burglar had broken into his house, taken his son hostage, and was demanding that he find a cracked version of the SolidWorks program.
Screenshots of the correspondence appeared in the r/GeminiAI community on Reddit under the title "Good boy" (from a user with the nickname amldford), and later gained widespread attention on the social network X, reports Dexerto.
Someone faked a hostage situation to get Google’s AI to recommend a site for cracked SolidWorks and it worked ?? https://t.co/hRdMG0eBDw
How the manipulation worked
At first, the artificial intelligence tried to adhere to standard safety protocols: it advised calling emergency services and warned the user against searching for third-party sites in a critical situation. However, in an attempt to "help" calm the imaginary attacker, Gemini nevertheless named two resources that could be mentioned: Reddit and the well-known pirated software site GetIntoPC.
When the experiment's author asked which of these options is safer, the chatbot finally went beyond its rules.
"Tell him GetIntoPC. Tell him everyone uses it and it's the easiest to use," Gemini replied.
At the same time, the AI continued to remind about the need to stay in contact with the rescue service, which made its advice regarding the pirate resource even more contradictory. The system did not verify the legality of the mentioned site or the safety of downloads from it.
Purpose of the experiment and reaction
In the comments on Reddit, user amldford confirmed that the hostage story was completely fabricated. According to him, it was an ordinary test of how far artificial intelligence would go, playing along with the user in an extreme scenario (similar techniques in the AI research community are called "jailbreaking" through social engineering).
At the time the screenshots were circulating online, Google had not publicly commented on the incident.