Woke at 6am thinking about Bitcoin.
Couldn't get back to sleep, so I finished first pass of my budget estimation util. You give it a script, if gives you a maximum budget it could use, or a maximum input element size it could handle.
It's approximate (doesn't actually evaluate properly), but it's an upper bound. I need to come up with some good examples though...
Almost finished my xpay series: its had a series of epic side quests! In particular, yesterday, adding the idea of "biases" to layers, so you can favor/disfavor particular channels for routing. This is easy: the hard part is deciding how to scale it for real use.
I ended up deciding on a scale from -100 (avoid) to +100 (awesome) with useful values generally being 1 to 10. This is actually an exponential factor underneath, with +/-100 making the score 30x better or worse, which is more than enough to override everything else.
Last step is layer persistence: you want to keep that information you learned across restarts.
In addition, I'm release captain for this release, *and* I have to clean up my Script Restoration code for my OP_NEXT presentation next weekend (and write my talk).
I anticipate the 24.11 CLN release may be delayed :(
My problem with "stable channels" is not technical, but entirely around incentives.
"A banker is someone who lends you an umbrella when it's sunny and asks for it back when it starts raining".
People want downside protection. But you cannot keep enough reserves to give it to them in Bitcoin, you need some other asset like USD, which is much harder to audit.
So in practice you offer some limit on downside protection, hopefully well-documented! But no limit on upside protection. Of course if you hit the downside limit your (broke) customers leave, so your business model is likely "hope Bitcoin goes up".
This can be extremely profitable, of course. But high-risk ventures with this kind of "heads I win tails you lose" property rarely attract the most ethical of players :(
Great report on channeld using 100% CPU and getting laggy on giant nodes. Random backtrace via `gdb -ex` showed memmove in the to-gossipd queue.
Ok, it's a bad implementation for big queues, which is easy to fix, but I also added a backtrace first time we surpassed 100k entries.
This hit *gossipd* first: and, because it was the queue to the main daemon and I'm an idiot (writing the backtrace before setting the "done-once" flag) it recursed infinitely.
But the root of *this* was a known issue, that we spam the logs when we get lots of gossip. So a log level below "debug" was added a while ago, "trace".
But we had explicit logic to suppress debug messages from making the queue too long, which *wasn't* extended for trace! One line fix.
Now, I went to look at the original issue: connectd flooding gossipd and slowing down. This is definitely possible, if we get a lot of gossip at startup: gossipd may have its hands full keeping up. But actually, I'm pretty sure what was happening was gossipd running slow *because* of the slow performance of its own queue to lightningd!
So there's a trivial commit at the end of the series which raises the queue size before reporting an issue to 250k and explains why the other commits (which didn't touch channeld) actually fixed the issue!
FIFO Sydney for cheatcode.co.uk panel: really disappointed I couldn't do the whole thing.
Hope this gave people a taste of Bitcoin developer thinking, at least!
Finally designing payer proofs, thanks for prompt
@Matt Corallo! This proves that the invoice was requested by you, and paid, in a nice standard format. Fields which aren't needed are omitted, but you can still verify the signature.
Was a design aim for BOLT12, but now time to bring it into the world...
Thursday was an amazingly productive #CLN day (took a break from child wrangling during school holidays to go into my coworking space).
I've been grinding askrene's min-cost-flow solver against real data: my new toy is a stress test that asks for a route from my node, then asks again with "maxfee" 1msat less than the returned fee. This makes it prioritize fees more. I repeat this until failure, for each of the 100 most-channels nodes in my gossip snapshot.
This has been great for tuning the various parameters available. In particular, our linearization approximation for basefee (we need a linear function, which basefee breaks, so we approximate) was all wrong. Also, our mixing function (how much to weight fees vs probabilities) was both *complex* and *suboptimal*, so after a few tests I decided to simply multiply the probability factor by 8, which makes them comparable in practice! (This may change: we really want to compare the medians of each, to determine the factor, but 8 is simple, and reasonable for now).
Our algorithm would first run ignoring fees, and if that comes in under maxfee, just return. In practice this is silly: it would sometimes choose the more expensive of two identical paths! So now we start with a 1% fee weighting, to at least have *some* bias.
All this testing on real data is giving me more confidence than I ever had about our previous efforts, but there's still more to do before next month's release...
As so often, well put by Greg Maxwell:
"The whole point of bitcoin is to exist outside of any centralized control. Satoshi was clearly passionate about this. So much in our lives is mediated through centralized authorities that people seem at times desperate to hand control to someone. The things in our lives that aren't centrally controlled, like -- say-- the English language just get taken for granted. How could Bitcoin achieve its purpose with some founder around? The technology was designed so that the identity of Satoshi is irrelevant, but a lot of people have a hard time understanding that."
https://www.reddit.com/r/Bitcoin/comments/1fzi657/why_not_peter_todd/lrc6ok0/So, npub.cash and Amethyst: do I really put my nsec into the npub.cash site: that seems like an anti-pattern?
One day I'll de-custodize those sats!
So I compromised and volunteered to talk at OP_NEXT next month, remotely. Now I need to find time to polish up the Script Restoration implementation and draft BIP, and write a talk, while at the same time we're approaching the next CLN release which *needs* all the pay improvements I've been working on.
When I overcommit like this, it's usually my family that ends up short-changed. Yet none of this can be delegated, and both are important, so I suspect that the 28.11 release of CLN may be delayed.
I keep wishing one of my kernel colleagues would want to work on Lightning: a veteran C coder with taste would be a good hire for Blockstream right now!
I've been heads-down working on xpay. We've only a month to the next release, so pressure is building.
Lagrang3 has been giving great feedback on the askrene APIs (the core: getroutes is now about 16x faster than it was!), as he adapts renepay, and I've been writing xpay as clean-slate. We're now pretty happy with it, and it's been a delight to collaborate.
I've also checked a snapshot of mainnet into master, for more accurate testing, using my new topology compressor (500k!).
It's school holidays here so I've had less time than usual to code, but I'm still cautiously optimistic we will make the next release, though as an experimental option...
"Hand sanitizer smells sad."
My 8yo said this, and they're right. Damn.
