Yes I’m processing the data from the public data which has been archived by The Archive Team in 2013 Only usernames so far, which is already a ton of data

Replies (3)

Cool! What relay implementation are you using? Also, if you are not doing it already you could have your migration scraper write directly to the db in batches, instead of publishing to a relay. It should be much faster. What is the migration doing exactly? Can I read the code somewhere? Maybe a second look might find some low hanging fruits in terms of perf