That "attack" is what happened with coldcard.. it wasn't an actual attack (hopefully), but it was something no AI could find until somebody actually spend the time and money to find and exploit it... It went hidden for 5 years, most of which already had AI-based bug discovery tools in the hands of consumers and researchers. And for the last 1.5 years those tools were good enough to replace developers entirely.
Nobody could find it until the attacker spend a good amount of tokens to discover it.
Keep in mind that number generation is the first thing every crypto auditor looks at in any project. So, pointing an LLM at it is the first step that most people did with Coldcard's code for months before the attacker actually found what was wrong.
Login to reply
Replies (2)
well, if we're gonna talk about this bug,
it was introduced by doc-hex aka switck aka peter gray, who wrote both the lib and the integration, though he wrote the lib under a pseudonym for whatever shady reason, and did a whole thatre talking to himself in the prs when doing "reviews"..
they were warned by multiple people (incl. jamesob) that this setup looks buggy, if not straight up dangerous, but they ignored it, waved it off, either due to hubris or malice.
It sounds to me at least 50% likely it was an attack.
but either way, you say the hacker found it with AI, but the devs couldn't? what's the logic here?
Correct. Many experts had a *feeling* for it, but nobody actually found the issue. They all thought the problem was a lot smaller than it is. Any of these humans would have gone bat shit crazy if they knew the complete findings.
And even with all that info, even with AIs being able to find that "suspicion" out there, they never did or they did and just like those humans didn't think too much on it.
That happens everywhere with AIs, the just glance over everything because the context window is small. They never see everything.