@acdha and yes, the llm bot defenses are a real problem for web archives. I saw in IIPC Slack yesterday that they are considering this as a topic for a technical meeting at the next conference in a few weeks (meeting may be open to remote participation). Cloudflare has a Verified Bot registry, which I think some web archiving orgs have gotten on? https://radar.cloudflare.com/bots#verified-bots e.g. https://radar.cloudflare.com/bots/directory/kb-dk-bot