Very interesting, the reason I haven't tried silent payments yet is exactly what you mentioned, it needs to check every block looking for a payment, I also have my bitcoind pruned so I think that would be a problem too in case it gets desynchronized?
