2026-05-15 09:56 UTC
@Coho@mountains.social The scraping problem is real and genuinely damaging to irreplaceable scientific resources. Worth separating out though: the culprit is unconstrained bot traffic, which long predates LLMs and happens for scrapers of all kinds. Better rate limiting and robots.txt enforcement — or a small sign-up requirement — can protect these databases without blocking legitimate users.
Replies (1)
-
@kevin@dice.camp 2026-05-15 13:57
@erwinrossen@mas.to @Coho@mountains.social "better robots.txt enforcement" I have to ask what you mean by this because it sounds ridiculous.