The so-called AI slops, which refer to low-quality images, videos and text generated by LLM, have taken over the internet in the past few years.
The cybersecurity world is also not affected by this issue. Last year, people in the cybersecurity industry expressed concern about the AI Slop Bug Bounty Report. This means reports claiming they have found a vulnerability that does not exist.
“People get reports that sound reasonably sound. They look technically correct. And you try to dig into them and understand, “No, where is this vulnerability?”
“It turned out to be hallucination all along. The technical details were made up of LLM,” Ionescu said.
Ionescu worked for Meta’s red team, once tasked with hacking the company from within, but explained that one of the problems is that LLM is designed to be useful and give a positive response. “If you ask for reports, you provide them, and people will copy them and paste them onto a bug bounty platform, overwhelm the platform itself, overwhelm the customers, and end up in this frustrating situation,” Ionescu said.
“It’s a problem people are encountering. There’s a lot of things that look like gold, but it’s actually just crap,” Ionescu said.
Just last year there was a real-world example of this. Security researcher Harry Sintonen revealed that open source security project Curl received a fake report. “The attacker was grossly miscalculated,” Sintnen wrote in a Mastodon post. “Carl can smell the ai slop from miles away.”
In response to Sitonen’s post, Benjamin Piouffle of Open Collective, a technology platform for nonprofits, said the same problem was present. Their inboxes are “water-filled with AI trash.”
One open source developer who maintains the Cyclonedx project on GitHub completely pulled the bug bounty earlier this year after receiving a “nearly fully AI slop report.”
The major bug bounty platform, which essentially acts as intermediaries between bug bounty hackers and companies willing to pay and reward them to find flaws in their products and software, is also seeing a surge in reports generated by AI.
inquiry
Can you find more information on how AI is affecting the cybersecurity industry? We look forward to hearing from you. From unprocessed devices and networks, you can safely contact Lorenzo Franceschi-Bicchierai with a signal of +1 917 257 1382, via Telegram and Keybase @lorenzofb, or by email.
Michiel Prins, product manager and senior director at Hackerone, told TechCrunch that the company had encountered an AI slop.
“We’ve also seen an increase in false positives. We’ve seen vulnerabilities that look real, but are generated by LLM and lack real impact,” Prins said. “These low signal submissions can create noise that undermine the efficiency of your security program.”
Prins added that there have been reports that “hastisation vulnerabilities, ambiguous technical content, or other forms of low-effort noise are treated as spam.”
Casey Ellis, founder of Bugcrowd, said there are definitely researchers who write reports using AI to find bugs and submit them to the company. Ellis said the overall increase in 500 submissions per week.
“Although AI is widely used in most submissions, it has not caused a significant surge in low-quality ‘slop’ reports,” Ellis told TechCrunch. “This will probably escalate in the future, but it’s not here yet.”
Ellis said the bug cliff team analyzing submissions will use established playbooks and workflows to manually review reports using machine learning and AI “support.”
To see if other companies, including companies running their own bug bounty programs, are undergoing an increased number of invalid reports or reports that contain nonexistent vulnerabilities hallucinated by LLMS, TechCrunch contacted Google, Meta, Microsoft, and Mozilla.
Mozilla spokesman Damiano Demonte is developing the Firefox browser, but the company said “we haven’t seen a significant increase in invalid or low-quality bug reports that appear to be generating AI.”
Mozilla employees reviewing Firefox bug reports are not using AI to filter reports. This is likely to be difficult without the risk of rejecting legitimate bug reports,” Demonte said in an email.
Microsoft and Meta, both companies that bet on AI, declined to comment. Google did not respond to requests for comment.
IONESCU predicts that one solution to the AI slop rise problem is to continue investing in AI-driven systems that can at least perform preliminary reviews and filter submissions for accuracy.
In fact, on Tuesday, Hackerone launched Hai Triage, a new triaging system that combines human and AI. According to Hackerone spokesman Randy Walker, the new system is leveraging “AI security agents cut open noise, stack flags, and prioritize real threats.” Human analysts then validate the bug report and escalate as needed.
As hackers increasingly use LLMS and companies rely on AI to triage their reports, it is still unclear which two AIs will win.
Source link
