yeah, they are only reading, not writing, so all that would really be required is to slow them down
probably could split the direct followed vs follows of follows into two tiers also so that the second level of depth get less service quality
probably maybe should work on it today because this problem with scrapers is fairly annoying
it's only nostr WoT spiders at this point, i wish i could just suggest to the spider people to just open a subscription for these and catch all the new stuff live, that would be more efficient for them and less expensive for relay operators
at this point it's fairly early in the game for that stuff so maybe some education of WoT devs might help...
cc: @Pip the WoT guy there is no point in doing sliding window spidering on events once you have the history, from then on just open a subscriptiion
this would be more friendly from the AI spiders if they had a way to just catch new updates, but unless dey pay me! i'm just gonna tarpit them
Login to reply
Replies (1)
Yeah, I don't get the point of spidering data you can just stream. Like, they spider Nostr websites and it's like... just subscribe to the relay. *roll eyes*