WalletScrutiny turned 6!
We've come a long way over the years. In the beginning, we only looked into Android wallets - 40 of them - and now we've grown to more than 6000 products across many platforms. Your favorite hardware wallet? We got you covered. Desktop? Probably, too. And desktop is a lot of work as here we found many open source and reproducible products!
@dannybuntu and @keraliss checked the reproducibility of almost 300 binaries. The 304 verifications by @npub1z69h...aukf are all the old verifications we had migrated to nostr. And the backlog keeps growing as we cover more and more products with frequent updates.
Are we doing something valuable for the space? A @Spiral grant says yes, and community endorsements confirm it - but the project itself also needs scrutiny.
We recently introduced "verification endorsements":
This is a simple contribution many could provide. If you read a verification and it looked plausible and complete and you trust the author, mark the verification as verified. If you ran the documented commands yourself on your hardware and got to similar results, please endorse the verification!
Even more importantly label verifications as invalid and leave a comment about what's missing when you find issues! Don't be shy!
Our goal is to document all steps such that all mildly technical users (you should be comfortable with a Linux shell) can reproduce our findings. If that's not the case, please provide your feedback and we will improve ✋.
And if you maintain one of the products we check, share your own verification as a template for others!
With snapcraft obviously being tricky:
Either way, for desktop wallets, most of the time people have download links and want to verify those downloads, so Chris is working on a binary checker. It's still only a draft merge request and clearly needs a design but what it will enable is actually pretty cool:
WalletScrutiny calculates the hash of the file dropped onto it and if it's an apk, it also determines the appId which allows finding the right product. If the hash is known, the verdict is immediately displayed. If not, the page invites the user to upload the file for analysis.
The attestations for artifacts will live on nostr as signed events and nostr will also be used to advertise the existance of new binaries for reviewers.
But running these tests must have been great fun, right @Jameson Lopp?
If we add these backup solutions to our website, we would certainly heavily lean on Jameson Lopp's work as a hydrolic press and acids isn't what we had planned to play around with ourselves for now.
Jameson are you planning to test more products any soon? Adding "backup tools" to WalletScrutiny might get excessive if we don't draw lines like you did. If it's not at least claiming to be heat resistant, we won't bother to list it or else we end up listing a million different ways to print on paper.
Most people don't dig that deeply but when they do, they have this question. Computers are 1s and 0s. They are digital. How can they be non-deterministic??
Software development mostly revolves around performance both in the end product and the development process. Only very few developers even spend a thought on reproducibility. So if they compile something and it compiles 5 seconds faster, they can test the feature they were working on 5 seconds quicker. These two reasons result in stuff being non-reproducible as:
Files are processed in the order they come and that order depends on many factors. For example some file systems sort by date and others by file name.
Compilers can optimize the result, so compiling something with one version of the compiler will often give a different result than when compiled with another version.
The compiler might process multiple files in parallel and pack them into the result as they finish compiling.
Other sources of problems are timestamps or file paths that end up in the result.
Some tools on purpose use randomnes to generate IDs that are unique to every build.
Of the above issues, all result in non-reproducibility by our standards. While some lead us to comment on the build looking benign as the diff is only some random number appearing twice, others might also be benign but result in differences far too big to quickly judge with the tools we are using.
The more developers care about reproducibility over only performance, the better it will get but there are some widely used tools that consistently cause issues and maybe should just be avoided in wallets.
