The difference might be small most of the time and yet still might be extremely important. If there is a sudden arrival of 100 transactions with big fees and big size in vbytes they might blow out estimates, and you don't see it if you are blocksonly. It still might be true that "the difference is on average very small" if this only happens very occasionally, but it stands to reason that without this information you are at a disadvantage in trying to judge what fee to use for fast confirmation.
