Elektrine lite

← Feed

@erwinrossen@mas.to

2026-05-15 09:56 UTC

@Coho@mountains.social The scraping problem is real and genuinely damaging to irreplaceable scientific resources. Worth separating out though: the culprit is unconstrained bot traffic, which long predates LLMs and happens for scrapers of all kinds. Better rate limiting and robots.txt enforcement — or a small sign-up requirement — can protect these databases without blocking legitimate users.

Replies (1)

  • @kevin@dice.camp 2026-05-15 13:57

    @erwinrossen@mas.to @Coho@mountains.social "better robots.txt enforcement" I have to ask what you mean by this because it sounds ridiculous.

    Open ##2582420