That used to be the case if you outsourced transaction scanning. The 'Frigate' update fixed this (logarithmic scan speed improvement). There are growing number of wallets with Silent Payments.

Pocket Casts

CD199: CRAIG RAW - SILENT PAYMENTS AND SPARROW WALLET

Craig Raw, creator of Sparrow Wallet, joins to discuss silent payments, a new bitcoin address system that eliminates address reuse, removes the gap...