The last CLN release was terrible for really large nodes: the bookkeeper migrations were exponentially slow on Postgres, then the first call to bookkeeper would grind the node and even OOM it.
Some quickbfixes went in, and there will probably be another point release with those. Thanks to those who donated their giant accounting dbs!
But the exciting thing is all the optimization and cleanup work that came from me profiling how we handle this large data. Not just reducing the time (those single linked lists being abused!) which I've been spending the last weeks on, but last night I started measuring *latency* while we're under stress.
1. Anyone can spray us with requests: the initial problem was caused by bookkeeper querying a few hundred thousand invoices.
2. A plugin can spew almost infinite logs.
2. xpay intercepts every command in case xpay-handle-pay is set, so it can intercept "pay" commands.
Fixes:
1. Yield after 100 JSON commands in a row: we had logic for autoclean, but that was 250ms, not count based. We now do both.
2. Fix our I/O loop to rotate through fds and deferred tasks, so yield is fair.
3. Yielding on plugin reply RPC too, which definitely helps.
4. Hooks can now specify an array of filter strings: for the command hook, these are the commands you're interested in.
As a result, my hand-written sleep1/lightning-cli help test spikes at 44ms, not 110 seconds!
My next steps are to use synthetic data (say, 10M records to allow for future growth) and formalize latency measurements into our CI.
Next release looking amazing!
I think finding a bug where printf("%*.s") was used instead of printf(".*s") was the point at which I realized that some C issues cannot be mitigated with better tooling...
I had way too much fun optimizating CLN due to a Boltz report recently. Like most engineers, I love optimizing, but bitter experience means I only allow myself to do so when I have an actual problem to solve.
My favourites:
1. Adding 1M entries to a sqlite3 table with two indexes: 81 seconds. Adding entries *then* creating indices: 17 seconds.
2. Optimization of O(n²) cleanup code by realizing it could be deleted entirely.
I admit I continued to optimize long after their performance problem was fixed, but you gotta have *some* fun!
I hate price talk, but if you're going to do it, please understand that "market cap" is a very rough *ceiling* on current value.
It's neither the amount of money which has gone in, nor the amount of money which can come out.
So the order of magnitude is useful to compare against other assets. But abusing it in terms of profits and losses is a category error, and I assume done mainly because it's so easy to measure.
Grump over.
In two days' time Julian and I will be doing an open Jitsi meeting to discuss the work on Script Restoration. Come and ask questions!
1pm Berlin time:
https://www.timeanddate.com/worldclock/meetingdetails.html?year=2025&month=10&day=15&hour=11&min=0&sec=0&p1=5&p2=37I needed some code in a third place, so I decided to extract the ~100 lines it into a single function in common code (and make it more efficient, because I had seen it in profiles under stress testing).
I often write code by starting it and seeing where it leads: this is helped by modern revision control and fast compile times (I point this out because we take these for granted).
First extraction went fine. Then I tried to use the same code in the second place. By the time I'd had to add four callback hooks (three of which were unneeded in the first case) I stopped. The code was different enough that the "framework with callbacks" made the code far *less* clear.
Threw it away, in favor of a handful of helper routines. More code for the first case, but much more natural for the second.
This took a day, on and off. But the off was important: the breaks gave me time to ask "now it's almost finished, do I like it?" and avoid being too goal driven.
.
I'm going to do an open hour on the Script Restoration BIPs: All questions welcome, even (especially!) if you're not a developer.
About 24 hours from now (I'll comment with the Jitsi link)
https://www.timeanddate.com/worldclock/meetingdetails.html?year=2025&month=10&day=4&hour=22&min=0&sec=0&p1=5&p2=224&p3=37&iv=1800Should I run open hours for the script BIPs? Like, one a week for four weeks, you can show up and ask questions and I'll try to answer?
That might help me get a feel for what is unclear or confusing.
If so, what platform do I use? Never done anything like this before đŹ
Finally posted the 4 script restoration BIPs to the mailing list.
Sorry it's taken so long!
I strongly suspect that the bitcoin experience, even for veterans, will be quite different in ten years.
BIP 353 brings names: ACINQ have a spec for supporting contact lists too. I expect vanity addresses to follow as providers get onboard.
Silent payment addresses should replace all other on-chain addresses, especially deposit to exchanges. Though you'll probably just send to e.g. <acctname>@client.river.com (will this allow probing of account names? Will they use random ones instead? Or accept anything and if they get typo payments sort it out in customer service?)
And BOLT12 provides reusable lightning addresses, which provide the off-chain analog of silent payments. Doesn't matter to you if the recipient is using some weird layer 2, either. BIP 353 returns both this and a silent payment address, so the sender wallet chooses.
It's been two years since the twenty-weeks-of-friends-coming-over that was Pandemic Legacy. Now the second edition of Gloomhaven has arrived, and we are planning another epic:
If you haven't played this kind of thing before: the game is a progressive cooperative affair, designed to be played in two(?) hour long sessions. Obviously there are weeks you miss, but the overall experience is a season or two of regular game play. There are highs and lows as things twist and turn, and I always feel the discomfort of doing something complex and unfamiliar, but overall it's about shared experience.
Must get the script BIP drafts published *before* we start this!
Don't suggest a developer resign unless you're the one who will shoulder the load.
And if any bitcoin developer is thinking of stepping down over the current drama, *please * reach out to me: I've been a FOSS dev full time for over 25 years and I've been there! I'm not going to judge, but I may have a useful perspective. â¤ď¸
I love BIP 93, with its ability to store any secrets with super-high redundancy and natively handle multiple shares by hand.
But I also love BIP 39 for the simplicity of 12 words, and the universality of the standard.
I want a word scheme for BIP 93. It would be 20-23 words(depending on what scheme is used), but highly redundant. You could recover with 3 or 4 unknown words.
Ideally the word list would avoid near-miss words, and be distinct from BIP-39 enough that it would likely be distinguishable, rather than relying on the strange number of words.
Should this exist?
