Luckily we have relays that can do some scanning for us...

Also you need to just scan from the time when you created the wallet. So the first time right after creation this is fast...