You should believe what your brain tells you. In this case, you can see that your Dojo server stores your xpubs and addresses and that's because it makes sense to do so. The same applies to the Samourai server because it had to manage a much higher user base and that makes caching even more important.