Detection Performance (internal)

Security Performance Metrics / Detection Performance Overview / Accuracy & Trust

Introduction

This page outlines Venn’s detection accuracy, showing how often Venn accurately identifies malicious transactions and how rarely Venn mistakenly flags legitimate ones. These factors demonstrate Venn’s reliability in protecting your protocol, users, and assets.

---

Key Metrics

• True Positive: 60 out of 63 known hacks detected, >~95%

• False Negative: 3 out of 63 known hacks not detected, > ~4%

• False Positive: ~0.08% of ~3.1 million inspected transactions flagged

• False Negative: Malicious transactions that Venn failed to detect.

• False Positive: A legitimate transaction that the Venn system incorrectly flags as malicious.

---

Methodology

What We Did: We forked Ethereum’s mainnet to create a controlled backtesting environment that mirrors real-world conditions. This allowed us to replay historical blocks and transactions as they originally occurred, ensuring our detection metrics reflect real on-chain activity.

How We Did It:

• Timeframe & Blocks:

  • The backtest spanned from block 16,308,190 (Jan 1 2023 00:00 UTC) to block 20,838,190 (Sep 26 2024 23:51 UTC).

  • This period lasts about 1.5 years and includes more than 4.5 million blocks.

• Protocol & Assets:

  • We included 64 protocols representing a total of 280 assets.

    • Of these, 63 protocols had a verified exploit on record. One protocol was kept as a control group (with no known hack).

• Transactions Inspected:

  • We examined every transaction in which at least one of the selected protocol assets appeared in the call trace.

  • In total, this amounted to over 3.1 million transactions.

Why We Did It: By replaying a large, realistic slice of mainnet history and focusing on transactions tied to our selected protocols, we can accurately measure Venn’s ability to detect and block real exploits. This approach ensures our reported FP and FN numbers reliably reflect Venn’s performance in a production-like environment.

---

Results & Interpretation

1. High True Positive Rate (> 95%)

   • Detecting 60 of the 63 hacks showcases strong coverage of attack patterns.

2. Low False Positive Rate (< 0.08%)

   • This low false positive rate means that legitimate transactions are rarely disrupted.

   • Caveat: Some of the flagged TXs may stem from exploits that have not been publicly disclosed.

3. Impact

   • Protocols: minimize disruption by producing a minimal amount of false alarms while maintaining robust protection against actual exploits.

---

See Venn in Action

Check out the Hack Simulator, where you can:

• Explore a list of historic hacks.

• Replay each hack under a Venn-enabled environment.

• Observe how Venn would have flagged and blocked the malicious transactions.

• Use the simulator to explore individual exploits and understand Venn.

---

Note: All metrics on this page apply to Venn’s root network under the default security configuration. Subnets that utilize custom or specialized security models may exhibit different performance.