crates.io: Rust Package Registry

crates.io serves as a central registry for sharing crates, which are packages or libraries written in Rust that you can use to enhance your projects

Second Rust crate, re-implementing a utility I had written several times and see. This one takes a more thorough approach and handles corner cases a bit better. I also added percentile support which is actually quite cute. Basically, if you have a pile of text lines, which contains some numbers, something like you've run a benchmark 20 times, produced a pile of output, line stats will gather similar lines and show you the stats on the numbers.

Sometimes people speak of Bitcoin and I get a moment of disorientation before I realize they're speaking of it as an *investment*. Like gold, oil, Nvidia stock, etc. There's nothing wrong with that, but it's a single dimension: up good, down bad. Corollaries: 1. This is fungible: there are other investments, some of which may be better at going up. 2. They may be "really into" Bitcoin, but it's a phase: some other financial winner will come along. 3. There are no deep insights here, just a lot of effort to predict numbers. 4. They're not wrong: finance is important. But for me it's like doing my taxes: I do it, but I kinda resent the time I spend on it.

Paused my reading of The Mandibles for Xmas day: It's not exactly holiday reading. It's one thing to know nothing stops this train. It's another thing to read through a detailed description of what the slow inevitable crash looks like: It's like the right brain equivalent of When Money Dies.

My second week of Rust (only a few hours a day though: I am on vacation after all) has me reproducing and enhancing a tool I wrote in C. It takes multiple lines and runs stats on the numbers in it. This is useful for quick benchmarks: `for i in $(seq 10); do /usr/bin/time myprog; done | linestats`. Each line with the same literal parts is combined, with numbers replaced by "min-max(mean+/-stddev)". The C one wasn't very clever about decimals, so it needed a good rewrite. The new code works, but needs polish, more options, optimization, tests and documentation before I release it. The good thing about these small projects is they don't get hamstrung by Rust's glacial build times!

I have published my first Rust crate. A the library that I've always wanted to exist and would normally have written in C, so it was a good chance to experiment with rust.

syncless - Rust

syncless: ordered, atomic storage without durability guarantees.

1. No, I'm not reading your article on quantum. 2. Yes, all choices are bad. That's what "breaking" means. 3. I'm glad smart people are thinking about technical mitigations.

Since I'm taking a few weeks vacation, I've decided to seriously try to learn rust. My method in this case is to ask ChatGPT to guide me (but not write for me!) a library ("crate") that I've always wanted to write and never got around to. Of course, I get a lot of feedback on appropriate rust styling, but some of it veers into things I feel are deeper constraints. In this case, I had an open function, which took a struct containing some flags, such as "writeable", "create if didn't exist". It didn't like the fact that I asserted if you set create and didn't set writeable. Here is my response: --- Re: assert!(). I dislike APIs which allow misuse. Callers of a library should not rely on such checks in the library, in fact the concept of adding an InvalidOptions error type is offensive. A recoverable error is strictly a worse api than an unignorable error. But a compile time error is better. We should use an enum READONLY, WRITE_MUST_EXIST, WRITE_MAY_CREATE. --- Of course, it's a waste of time for me to lecture it on style, but I can't help myself!!

"Bitcoin is taking over the world"? Tether is many things, but it's not Bitcoin.

Happy VPN-day Australia, to those kids who celebrate!

In preparation for onboarding new core lighting developers are preparing a series of videos. So I've been asking ChatGPT about CLN developer features, particularly with comparison with other projects people might be familiar with. Of course, I compare myself with Linux, but it's interesting to see comparisons against other projects: **Type-safety**: OpenBSD High Bitcoin Core High Core Lightning Very high for C Nginx Low curl Moderate MySQL Moderate SQLite Moderate CLN sits near the top among major C codebases for safety discipline. **PR Submission** Core Lightning’s PR flow is unusually strict, slow-moving, and review-heavy compared to most open-source C projects — closer to Bitcoin Core or OpenBSD than to typical GitHub projects. --- Compared to “average OSS” Most projects: Feature-oriented PRs Informal review Few required reviewers Patch squashing common Tests sometimes optional Architectural discussion often post-merge CLN: Patch-first culture Pre-merge architectural scrutiny Extremely high reviewer expectations Tests are mandatory Clean, narrative commit history matters

Seriously considering putting two RTX 6000 in my upcoming build machine. Puts the price up an order of magnitude, but truly private AI might be a worthwhile investment. Never played with GPUs before, so informed thoughts welcome?

The latest (final?) #CLN release candidate fixes a long-standing bug where we could forget UTXO spends when we restart. This explains a variety of bug reports we have seen and been unable to reproduce over the years: the most recurrent being gossipd telling peers about channels which are long closed. We no longer make this mistake, but we also have to walk back and revisit old UTXOs. We do this while running, but for older nodes (like mine!) that can be a lot of blocks. In fact, and I only vaguely recall this, my node tracks back to block 500 (!) so it's going to take a while. Of course we remember progress, so you can restart slike normal during this process. Other than higher CPU consumption you shouldn't notice anything.

With my bike being repaired, I've been doing more walking. Australian cities are really optimised for driving, but I must say it gives me a lot of time to reflect on broader issues, probably a decent way to increase my nostr posting!

@Jameson Lopp recently posted on X about the danger of "store and forget" for Bitcoin over decades. Unfortunately he's right. Originally I stored my raw private keys and UTXOs (on paper, care taken) figuring that was standard. Then bitcoin core stopped supporting them! Other wallets tend only to support them for sweeping, and I wonder how long. If I were storing funds today I would use BIP39. BIP93 is cool and more general, but not widely supported, and I don't know what support will look like in a decade.

Christian Decker made a side comment on my PR about a potential further optimization for Postgres. I spent an hour on it, trying to get it to fit in our db API. Since my wife was out with friends last night, so I dived into the implementation in earnest after the kids in bed. Man, this is a horrible API which forces you to understand the design and history of PostgreSQL. Until v14, the async API only supports one request at a time. (Surprise!!) You must explicitly turn on pipelining. Then you get one or more responses per request, with NULL between them. This is because you can have multiple SQL statements per string, or other modes which split responses like this (?). You can no longer use non-pipelining APIs, once you turn this on. You can't use PQsendQuery in pipeline mode. You need to use the lower level PQsendQueryParams with NULL params. This is documented as literally "You can't use this in pipeline mode" without explanation (the docs otherwise imply it's just a convenient wrapper). This is why you should always check your error returns! And your code will still block forever. You need to explicitly flush the pipeline, otherwise it's cached locally. There are multiple different APIs to do this, and I'm not sure which to use yet. Also, if you get an error in a SQL query, you need to drain the pipeline, turn pipeline mode off and on again. Finally, the documentation warns that you are in danger of deadlock unless you turn on non-blocking mode. This makes some sense: the server won't read more commands if you're not reading, but I would prefer it to buffer somewhere. This whole API seems to be implemented by and for people who have deep familiarity with PostgreSQL internals. Hope the latency gain for CLN is worth it!

I think I got nerd sniped into implementing online compaction for CLN's gossip store. Mainly from people agitating that we should use a DB for gossip messages. This will speed startup and reduce memory usage. And it's only going to take me a couple of days' work, depending on how many side-cleanups I do. It can be hard to tell a flawed implementation of a good idea from a bad idea. But trust me, this is gonna be great!

The last CLN release was terrible for really large nodes: the bookkeeper migrations were exponentially slow on Postgres, then the first call to bookkeeper would grind the node and even OOM it. Some quickbfixes went in, and there will probably be another point release with those. Thanks to those who donated their giant accounting dbs! But the exciting thing is all the optimization and cleanup work that came from me profiling how we handle this large data. Not just reducing the time (those single linked lists being abused!) which I've been spending the last weeks on, but last night I started measuring *latency* while we're under stress. 1. Anyone can spray us with requests: the initial problem was caused by bookkeeper querying a few hundred thousand invoices. 2. A plugin can spew almost infinite logs. 2. xpay intercepts every command in case xpay-handle-pay is set, so it can intercept "pay" commands. Fixes: 1. Yield after 100 JSON commands in a row: we had logic for autoclean, but that was 250ms, not count based. We now do both. 2. Fix our I/O loop to rotate through fds and deferred tasks, so yield is fair. 3. Yielding on plugin reply RPC too, which definitely helps. 4. Hooks can now specify an array of filter strings: for the command hook, these are the commands you're interested in. As a result, my hand-written sleep1/lightning-cli help test spikes at 44ms, not 110 seconds! My next steps are to use synthetic data (say, 10M records to allow for future growth) and formalize latency measurements into our CI. Next release looking amazing!