wasnt p2p yacy search engine doing archival and browsing and spidering stuff in the old days?
Login to reply
Replies (1)
web archiving is a childs play.
what we want to do is decentralize web crawling data. nobody uses yacy which implicate it failed.
what is the total size of latest common crawl? (estimate)
Compressed size (gzip‑ed WARC) 250‑350 TB
solve this.