#crawlers

The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools. Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.

“AI webpage #crawlers are now being served their very own #ads that ordinary visitors never see, with one firm's boss openly describing a strategy to influence what #chatbots say about #brands. The next time you ask Claude about where to bank, its answer may have been influenced by this #BotTargeted content. We only have one documented example so far, and that's Time serving #AIOnly ads to selected AI crawlers, as spotted by Germany based freelance software developer #VincentSchmalbach. In a Wednesday blog post, Schmalbach detailed how he found #sponsored content embedded in #markdown versions of some Time pages that are served to AI crawlers but not ordinary browsers. Those pages also contained #advertisings tags from #AdTech vendor Mobian ahead of extensive #FAQs for online-only bank Ally. The FAQs include brand "facts," such as the number of fee-free ATMs associated with the branchless bank, alongside claims that Ally is "the only bank built for life today," putting it in "a category of one." AI poisoning with advertising. #AI / #advert

The effect of blocking AI crawlers on Blender's infrastructure. This is the CPU usage graph.

So many AI bots are crawling Blender's infra all the time. Just do a git clone and investigate local files. It's way faster (also for the AI users themselves) and doesn't block actual Blender development (it got that bad).

#b3d #blender3d #AI #crawlers