Table of Contents
Researcher
Related Projects
Key Takeaways
I.R.I.S, an AI security agent on the Nethermind’s AuditAgent platform, is set to launch on Ethereum through Virtuals Genesis, gaining significant attention from the community.
Automated vulnerability detection in Web3 has faced challenges, failing to detect complex threats or producing unreliable results, highlighting the need for a hybrid approach combining traditional and AI techniques.
AuditAgent, underpinning I.R.I.S, integrates static/dynamic analysis, documentation-based code review, and development-stage support, suggesting a promising future for AI security tools in Web3 as complementary aids.
1. Emergence of I.R.I.S
Source: @ethereum
Just yesterday, the announcement that an agent from Virtuals Protocol would be deployed on Ethereum was made through the official Ethereum account, garnering significant public attention. The community soon recognized that this agent was I.R.I.S, scheduled to launch on the Virtuals Genesis platform. Interest intensified further upon the revelation that Nethermind, a well-known security research group within the Ethereum ecosystem, was leading its development.
I.R.I.S is built on AuditAgent, an AI security audit solution developed by Nethermind. It is introduced as being deployed on social channels like X to scan conversations and code in feeds such as aixbt, providing real-time security analysis and insights. Although the agent's implementation has not yet been publicly released, which makes it difficult to analyze its capabilities, the concept itself presents important implications for the AI security field.
AI security has been inadequately addressed in the blockchain industry so far. Existing AI security tools have underperformed, particularly in detecting smart contract vulnerabilities. Many tools from both academia and industry only identify simple vulnerabilities or frequently exhibit hallucinations. However, an analysis of AuditAgent, the foundation for I.R.I.S, reveals a higher level of technical maturity and convincing architecture compared to previous attempts. It offers valuable insights into the future development of AI security in Web3.
In this article, coinciding with the emergence of I.R.I.S, we examine cases where automated solutions have been utilized in Web3 security and analyze what limitations they have experienced. We then explore the possibilities for how the future of AI-based Web3 security might be structured, focusing on AuditAgent.
2. History and Limitations of Automated Vulnerability Detection in Web3
In the Web3 environment, where large amounts of user funds can be directly stolen, there is a heightened awareness of security compared to Web2. As a result, projects allocate substantial budgets to security audits. According to a February 2025 survey by Ulam Labs, audits with top-tier security companies like Trail of Bits or Consensys Diligence cost between $20,000 and over $150,000, depending on project complexity. The duration of these audits can extend beyond a month.
These prices are influenced more by the scarcity of security personnel who are "better than hackers" than by the specificity of Web3 programs. Even though the Web3 security market is now considered somewhat mature, the expansion of high-level security personnel remains extremely challenging. This ongoing difficulty maintains the perception that security audits are expensive.
Source: Trail of Bits
To tackle the rising costs of security audits, driven by recruitment challenges and limited scalability, Web3 security companies have invested heavily in developing automated vulnerability detection tools. Before the maturity of machine learning technology, around 2020, these tools were crafted using traditional techniques such as symbolic execution, static analysis, and fuzzing. These methods aimed to identify basic verification vulnerabilities or branch errors.
Notable tools include Mythril, created by ConsenSys, and Slither/Echidna, developed by Trail of Bits. These tools remain popular during development stages due to their fast analysis speeds and low false positive rates for known vulnerability types. However, they are limited in identifying new attack vectors or complex vulnerabilities because they rely heavily on known vulnerability patterns.
The development of AI/ML-based automated vulnerability detection began in earnest after 2020. These efforts aimed to shift from traditional rule-based methods to pattern recognition from past audit data. However, their overall performance was limited. AI development struggled to keep pace with the rapid evolution of blockchain technology and the emergence of new attack vectors. As a result, these tools could not deliver reliable results, necessitating thorough manual reviews by auditors.
This scenario persisted even after the advent of LLMs in 2023. While LLMs generated "plausible" results, they led to a proliferation of low-quality security audit reports. Consequently, within the security community, LLM or AI-based security audits were often viewed as nearly fraudulent.
Source: @BunzzDev
In April 2024, a notable incident in the security community unfolded involving a dispute between security researcher 0xnirlin and the AI security audit company Bunzz Audit. The debate centered on the effectiveness of AI-based security audits. To settle the matter, they decided to conduct audits on approximately 265 lines of code and compare their findings. The results were publicly uploaded to GitHub, marking a significant moment in understanding AI security audits.
The confrontation yielded intriguing results. All high-level vulnerabilities detected by AI were found to be invalid. Moreover, the AI failed to identify high-level vulnerabilities that could lead to fund theft. Bunzz Audit clarified that their service was not entirely AI-performed but rather AI-assisted. This distinction highlighted the challenges of achieving fully automated AI security auditing.
The inability of AI-based security tools to fully replace existing security audits is clear from the numerous security incidents in the current Web3 ecosystem. In 2024 alone, there were over 120 incidents due to smart contract vulnerabilities. Many of the projects that were hacked had undergone prior audits, yet were still attacked. This indicates that AI-based real-time vulnerability scanners are not effectively blocking actual threats.
While various analyses explore the root causes of these failures, I believe there are two main reasons previous AI-based security audit products have fallen short.
Source: misirov
The first is excessive dependence on AI. Current AI-based security audit products are closer to wrappers around open-source models with minimal reinforcement learning, lacking efforts to improve detection accuracy through traditional software analysis techniques. This causes serious reliability issues due to hallucination and creates additional effort requirements for filtering false positives.
Source: oroinc
Another issue is design that ignores existing security audit workflows. Security auditing starts with verifying that implementation behavior aligns with developer intent, requiring a meticulous comparison between official documentation and source code. Products that overlook this workflow and merely provide source code to models cannot be deemed as conducting comprehensive security audits.
This is closely related to AI model context preservation challenges. Early-stage LLMs could only process documents of limited length, lacking the capacity to fully understand and reason through the flow of lengthy official documents. Improving AI model performance could significantly enhance AI security audit capabilities.
Considering these factors, AI-based security auditing should be conducted as follows. First, analysis based on traditional security techniques should be performed. Rather than simply reading source code as text, interpretation of programs through control flow and data flow should be fundamental, and invariant specification should be used to prevent AI models from making interpretations that deviate from intent. Second, a meticulous comparison process between official documentation and source code should be introduced, enabling the suggestion of various constraints and tests that users may not have addressed. Additionally, the latest models capable of processing contexts long enough to digest this should be adopted.
3. AuditAgent's Structure and Implications
Source: Nethermind
Returning to I.R.I.S, Nethermind's AuditAgent is noteworthy as a project that adheres well to this direction. AuditAgent presents an approach that combines static and dynamic analysis with traditional security analysis techniques alongside the latest LLMs. Additionally, it provides documentation-based code analysis and invariant generation, as well as functionality to simulate behavior in actual execution environments.
AuditAgent's positioning as a development-stage tool rather than a complete replacement for existing security audits is also considered a wise choice. LLMs still lack perfect reliability to replace manual security audits, and for improving overall ecosystem security, it's important to eliminate vulnerabilities in the development process as much as possible so that security auditors can focus on detecting complex vulnerabilities. For this purpose, AuditAgent provides GitHub integration functionality, integrating into development workflows to provide continuous monitoring of code changes. The provision of comprehensive services including not only vulnerability detection but also attack simulation, gas optimization, and executable report generation can also be considered a differentiating factor from existing tools.
Thus, AI security tools in Web3, particularly vulnerability detection solutions, are expected to be comprehensively integrated into development environments in the future, serving as sidekicks that improve the overall security of the ecosystem. I believe Ethereum's support for I.R.I.S may be related to Ethereum's new security initiative, as security services could be provided to developers at relatively low cost through the $IRIS token, or free security-related advice that would normally consume costs could be provided through agent mentions on social media. While I.R.I.S has not yet been officially launched, making it impossible to fully predict its performance, if I.R.I.S's success allows us to fully verify AuditAgent's performance, I believe it could become an important milestone in setting the direction for future AI security.
4. AI Security is No More an Illusion
The reason AI-based vulnerability detection is not an illusion is that significant movements are visible not only in Web3 but also in Web2. Most notably, last month security researcher @seanhn published an account of discovering and reporting a zero-day vulnerability in the Linux kernel using OpenAI's o3 model, shocking the security community when it became known that no additional tools beyond the API were used.
Source: @seanhn
Another example is the AIxCC competition hosted by the U.S. Defense Advanced Research Projects Agency (DARPA) in 2024, which demonstrated through semifinals held at DEF CON 32 in August 2024 that AI systems can not only identify vulnerabilities but also automatically generate patches.
Source: AIxCC
This competition was a competitive program where participants used their trained LLMs to find vulnerabilities in challenge tasks based on open-source projects widely used across the internet, such as Jenkins, Linux kernel, and Nginx. The competition resulted in the discovery of 22 vulnerabilities total, including the achievement of finding previously undiscovered bugs in SQLite3. This signifies that AI-based security tools are no longer illusions based on technological vanity but proven technology, and it was expected that AI would rapidly penetrate Web3 security auditing soon. Indeed, companies performing Web3 security audits such as Trail of Bits and Theori/ChainLight advanced to the AIxCC finals, and @p_misirov, a researcher from the famous security company Cantina/Spearbit, mentioned this at EthDAM, pointing out that the day when AI agents would actually penetrate the security domain was not far off.
In this sense, the attention I.R.I.S is currently receiving has great significance for the Web3 security ecosystem. If I.R.I.S and AuditAgent show satisfactory performance and establish themselves as basic security infrastructure, this would mean not just the launch of a single AI agent but the popularization of AI-based security services and validation of new service deployment models through token economics. I also believe it's necessary to closely observe how effectively these tools improve the security of the ecosystem going forward.
5. Resource
Related Articles, News, Tweets etc. :
