All Episodes
Your AppSec Tools Find 15% of Defects, Not 95% (Chris Near)
· 34:08 · Season 1, Episode 34

Your AppSec Tools Find 15% of Defects, Not 95% (Chris Near)

0:00 / 34:08

// TRANSMISSION

The application security industry spent years believing its tools found roughly 95 percent of the defects in code. Chris Near studied 34 million findings across 57 tools and put the real number closer to 15 or 20 percent.

In this episode of We Built It Because We Had To, host Jonathan W. Buckley is joined by Chris Near, founder of CyberSagacity, the company he built to sit on top of application security tooling as an intelligence layer rather than as another scanner. Chris traces where the 95 percent number came from: a pre-published artificial test suite that vendors could design against. Run the same tools against real code and the results collapse. He connects that gap to zero days, which by his account drive around 80 percent of breaches, defects nobody knew were there because no single tool was ever going to find them.

The backstory runs long. A PhD in electrical engineering at Cornell after an undergraduate degree at Northwestern, then close to seven years at Bell Labs, then a decision in 1992 to leave and chase the mathematics of software on his own. He self-funded that work until 2007 on 120-hour weeks, picked up an angel backer, lost him, self-funded again, and got him back. He pitched venture capital roughly ten times across those years with different products out of the same research and never closed one. His own explanation is blunt: he was a technologist with no partner who understood go-to-market.

Underneath the product is a statistical engine. Defect-level attack and consequence probabilities are matched against a cybersecurity insurance database of historical loss events, which turns a bucket of ten thousand equally urgent defects into a ranked list and reframes the question as return on risk reduction rather than raw defect count. SATriage does the triage inside the development process, sorting what is true, what is false, and what actually matters. SATraits is the planning tool that compares which tools fit a given environment before a program is built.

Chris is candid about the missteps. The first product was built for developer usability but carried management information, so it fit neither audience, and rebuilding it for developers, management and the C-suite fixed the alignment while lengthening the sale. He also gives the cleanest positioning lesson on this show in a while. He described his coverage numbers as coverage. The industry says false negatives. Same concept, opposite framing, and only the second one landed.

His advice to a founder six months into a security startup and drowning in scope: find the one feature with the biggest business impact, not the biggest technical impact. He notes it took him decades to learn it.

Every episode on Apple, Spotify, YouTube, Amazon and more: https://www.WeBuiltItBecauseWeHadTo.com

Connect with Chris Near: https://www.linkedin.com/in/chris-near-82abb591
CyberSagacity: https://cybersagacity.com

Enjoying the show? Follow We Built It Because We Had To wherever you listen and leave a rating. It helps other founders find these stories. For more on how The Artesian Network helps early-stage tech companies get to market and scale, visit https://www.artesiannetwork.com/?utm_source=podcast&utm_medium=referral&utm_campaign=ep34&utm_content=shownotes

Key Takeaways

  • Application security tools find approximately 15-20% of defects, not the widely believed 95%, based on Chris Near's analysis of 34 million findings across 57 tools.
  • The 95% figure originated from pre-published artificial test suites that vendors could design against, but real-world code produces drastically different results.
  • Zero days drive around 80% of breaches and represent defects that no single tool was designed to find.
  • CyberSagacity uses a statistical engine matching defect-level attack and consequence probabilities against a cybersecurity insurance database to rank defects by return on risk reduction rather than raw count.

Frequently Asked Questions

Where did the 95% defect detection claim come from?

It came from a pre-published artificial test suite that vendors could design their tools against, which does not reflect performance on real code.

What are the two main products CyberSagacity offers?

SATriage performs triage within the development process to sort true, false, and material defects; SATraits is a planning tool that compares which tools fit a given environment before a program is built.

What is Chris Near's background?

He has a PhD in electrical engineering from Cornell, an undergraduate degree from Northwestern, and spent close to seven years at Bell Labs before leaving in 1992 to pursue software mathematics research.

How does CyberSagacity rank defects?

It matches defect-level attack and consequence probabilities against a cybersecurity insurance database of historical loss events to create a ranked list based on return on risk reduction.