The issue seems to be that you cannot subscribe to all the other pubkeys in active use at the same time. This works for your contact list, but you won't see a stranger's DM.