For months, leading artificial intelligence developers have implemented sophisticated vetting programs and stringent safety protocols, aiming to prevent their powerful models from being exploited by malicious actors. However, this robust security architecture, designed to thwart cybercriminals, is now inadvertently impeding the critical work of legitimate network defenders and offensive cybersecurity researchers. The very safeguards intended to secure the digital frontier are becoming obstacles to those tasked with its protection.
The issue gained significant public attention in June when the U.S. government imposed export control restrictions on Anthropic’s highly anticipated AI models, Mythos and Fable. This move was reportedly influenced by a report suggesting that the models’ built-in guardrails, designed to prevent their use in crafting and executing cyberattacks, could be circumvented. While the precise motivations behind the government’s decision remain a subject of discussion, with some analyses questioning whether fears of a "jailbreak" were the sole driver, the practical effect was a significant restriction on access to advanced AI capabilities.
Anthropic had, in the lead-up to these restrictions, actively marketed Mythos as a potentially "doomsday cybermachine," emphasizing its power and the necessity of carefully vetting users and implementing strict controls. This positioning, while perhaps intended to highlight the model’s advanced capabilities and the company’s commitment to safety, contributed to the perception that such tools require extreme caution. The export controls on Fable 5 and Mythos 5 have since been lifted, with Fable 5 returning to general access on July 1. Mythos 5 has been reintroduced, but only to vetted U.S. organizations as part of an ongoing government review process.
This approach to access control is not unique to Anthropic. Both Anthropic, with its other AI models, and OpenAI offer specialized programs designed to provide cybersecurity researchers with access to models that have fewer restrictions. OpenAI’s "Trusted Access for Cyber" program and Anthropic’s "Cyber Verification Program" require applicants to undergo a vetting process before granting access to more permissive AI tools.
However, these guardrails have drawn considerable criticism from researchers whose work inherently involves identifying and exploiting vulnerabilities before they can be weaponized by adversaries. Mark Dowd, a prominent security researcher with decades of experience in discovering and selling "zero-day" exploits—previously unknown software flaws and the methods to leverage them—to Western governments, voiced his concerns during a recent cybersecurity podcast. Dowd articulated a sentiment shared by many in his field: "It’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not."
Dowd’s work involves providing these vulnerabilities to governments for intelligence operations, where the value lies in their unpatched nature. This perspective highlights a fundamental tension: governments often pay a premium for vulnerabilities precisely because they remain undisclosed and unpatched, allowing for covert surveillance and intelligence gathering. This contrasts with the typical cybersecurity approach of rapid disclosure and patching.
While acknowledging his own potential bias, Dowd is not alone in his critique. Several individuals working in offensive cybersecurity—the practice of proactively probing systems for weaknesses—have shared their experiences with TechCrunch regarding their use of AI tools and the challenges posed by their guardrails.
Chris Anley, Chief Scientist at the security consulting firm NCC Group, emphasized the dual nature of AI in cybersecurity. He explained that using an AI model to attempt to exploit a bug is a crucial step in validating its severity and the necessity of a fix. Yet, if a guardrail prompts the model to outright refuse to engage with such a query, it directly hinders the defensive effort. "This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base," Anley stated. He further elaborated that the same AI tool serves as both an offensive and a defensive instrument, and these functions are intrinsically intertwined.
Anley drew an analogy to a hammer: "You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well." When faced with these restrictions, researchers often resort to open-source AI models that come without any guardrails, allowing for unfettered exploration.
Paolo Stagno, Chief Technology Officer at Crowdfense, a company specializing in the acquisition and sale of unknown vulnerabilities to government agencies, echoed Dowd’s sentiment. Stagno described the vetted programs and guardrails as AI companies treating their customers "like children who need babysitting." While Stagno and his colleagues do utilize advanced AI models for tasks like reverse engineering, they actively avoid using them for vulnerability discovery or exploit development. This caution stems from the risk of leaking sensitive vulnerability data or having it inadvertently incorporated into future model training runs when using cloud-based AI services. For these critical tasks, they rely on open-source models run locally, ensuring data remains within their own environment.
Giuseppe Cali, a security researcher focused on discovering zero-day vulnerabilities and developing exploits, offered a different perspective. He stated that guardrails do not impede his work because he strategically employs AI. Instead of using AI for offensive operations, Cali leverages it for initial reverse engineering, to gain a deeper understanding of the code he is analyzing, and to develop supporting tools. He finds that AI tools significantly accelerate these preparatory phases, allowing him to dedicate more time and focus to the actual discovery of vulnerabilities. "I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow," Cali asserted. "I am jealous of my bugs, and I like this game too much to let models play it for me."
However, not all researchers have found workarounds. An anonymous researcher at a smartphone component manufacturer, who spoke on the condition of anonymity due to authorization constraints, revealed that their employer is not part of Anthropic’s Cyber Verification Program. Consequently, the AI tools available to them are "barely useful for finding vulnerabilities because the guardrails are too strict." The researcher added, "If it catches wind we’re doing anything security related, it just stops and isn’t usable."
Chris Thompson, CEO of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an event focused on offensive security and AI, highlighted the inconsistent nature of these guardrails. He observed that even within the more permissive environments of Anthropic’s and OpenAI’s vetted programs, the AI models’ behavior can vary significantly from day to day. "I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," Thompson explained. "Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output."
This inconsistency and over-sanitization are reportedly pushing researchers towards Chinese open-source AI models, such as GLM. These models are freely downloadable, can be run locally, and impose no vetting or usage restrictions. Thompson expressed concern over this trend: "You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems." He concluded, "I think it’s more harmful than good to have these guardrails in place."
Thompson advocates for a shift in approach from AI frontier labs. Instead of tightening restrictions, he calls for the opening up of their programs, providing responsible access, and holding accountable those who misuse their tools. He warned that without such changes, defenders risk losing the critical AI race. "There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before," Thompson cautioned. "But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now."
The implications of these restrictive guardrails extend beyond individual researchers. A significant concern is that the delay in developing and refining offensive cybersecurity tools—which are essential for testing and improving defensive measures—could leave organizations more vulnerable to sophisticated cyberattacks. As AI capabilities rapidly advance, the ability of cybersecurity professionals to understand and counter emerging threats relies on their access to powerful, albeit potentially dangerous, tools. The current approach of heavily restricting access, while understandable from a risk-management perspective, may be creating a bottleneck that ultimately compromises overall digital security.
The debate over AI guardrails in cybersecurity highlights a complex balancing act. On one hand, the potential for misuse of powerful AI models by malicious actors is a significant and legitimate concern. The incident involving Anthropic’s Mythos and Fable models serves as a stark reminder of this risk. On the other hand, the indispensable role of offensive cybersecurity research in identifying and mitigating vulnerabilities cannot be overstated. These researchers act as the digital immune system, proactively uncovering weaknesses before they can be exploited by adversaries.
The current environment, where legitimate researchers find themselves constrained by the very tools designed to enhance security, suggests a need for a more nuanced strategy. This might involve developing more sophisticated methods for distinguishing between legitimate research and malicious intent, enhancing transparency in AI model development and deployment, and fostering greater collaboration between AI developers, cybersecurity researchers, and government agencies. The ongoing evolution of AI necessitates a parallel evolution in how we approach its security, ensuring that the pursuit of safety does not inadvertently disarm those on the front lines of digital defense.
