> tcp connections are cheap
I would disagree but we could be on different scales, that's at least two extra OS handlers, at least 16k (usually 64k) of kernel space (x2 for rx and tx) and often userspace (usually shared) buffers, per connection, per LB. L7 lbs generally multiplex tcp connections and dramatically cut down on memory consumption. While it can be a bit aggressive. For every websocket that's opened in my http server, tuned to 16k, thats about 256k after the upgrade is established (because websockets are usually fully buffered) and http buffers are freed just in userspace. For 5000 connections that's over 1.2G of committed system memory minimum just hanging about. I would expect software like nginx to be more optimized in comparison but that still hogs up LBs and I know I've heard other sysadmins share their stories of websocket overhead.
L7 lbs multiplex these http/1.1 connections, generally cutting down to double digit connections to service 5-6 digit ingress traffic.
I agree with the basic architecture, that is, a highly functional with a "popularity decay" model. I can't remember the scientific name.
