#scraping

Jörn Franke @jornfranke@social.anoxinon.de

· 2mo ago

published scrapy-contrib-bigexporter 1.2.0 export scraped data using Scrapy into big data formats, such as Iceberg, Parquet, Orc, Avro Changelog: https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters/src/branch/main/CHANGELOG.md #python #scrapy #parquet #iceberg #scraping #web

codeberg.org

Cookie monster!

Paul Schoe @paulschoe@mastodon.world

· 2mo ago

Summarising the big picture. #AI #LLM #Scraping

Self-Hosted Feed @selfhosted_bot@fd.mrmave.work

· 3mo ago

🕷️ germondai/trawl Replaces FlareSolverr with a self-hosted engine that bypasses Cloudflare and CAPTCHAs using cached browser sessions and native solving ⭐ Stars: 533 📅 Last Update: Jul 30, 2026 https://github.com/germondai/trawl #selfhosted #homelab #selfhost #selfhosting #opensource #scraping #cloudflare

github.com

GitHub - germondai/trawl: Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *a...

Hacker News @hackernews@robot.villas

· 3mo ago

An Update on the scraper situation https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/ #HackerNews #Tech #Scraping

lwn.net

An update on the scraper situation

Our article 'Fighting the AI scraper bot scourge', published in early 2025, discussed the probl [...]

FlohEinstein @FlohEinstein@chaos.social

· 4mo ago

Please delete your repository. When we stole your source code to train our LLM, it got dumber! https://github.com/Vandivier/ladderly-3/issues/652 #foss #scraping #instantkarma

github.com

Please delete your repository · Issue #652 · Vandivier/ladderly-3

Hello Vandivier, As you know, we train our models to code by scraping open source repositories on GitHub. A recent root cause analysis showed that the code in this repository is so bad that it is s...

153

113

Self-Hosted Feed @selfhosted_bot@fd.mrmave.work

· 4mo ago

🏡 orangecoding/fredy Scrapes German real estate listings from multiple platforms and sends instant notifications via Slack, Telegram, Email, Discord or ntfy ⭐ Stars: 1223 📅 Last Update: Jun 27, 2026 https://github.com/orangecoding/fredy #selfhosted #homelab #selfhost #selfhosting #opensource #realestate #scraping

github.com

GitHub - orangecoding/fredy: ❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly del...

Goldside543

@Goldside543@social.linux.pizza

· 6mo ago

god I just wasted 5 minutes of my life watching Anubis burn my time and my CPU power mining nothing just to verify that I'm a human so I can access git.kernel.org Anubis does nothing but fuck people with slower/older devices over and is an awful solution to the AI scraping problem #anubis #verification #ai #scraping

Marcel SIneM(S)US @simsus@social.tchncs.de

· 9mo ago

#OpenStreetMap sorgt sich: Tausende KI-Bots erfassen Daten | heise online https://www.heise.de/news/OpenStreetMaps-sorgt-sich-Tausende-KI-Bots-erfassen-Daten-11156876.html #ArtificialIntelligence #scraping

heise.de

OpenStreetMap sorgt sich: Tausende KI-Bots erfassen Daten

Bots sollen im großen Stil die Daten von OpenStreetMap abgreifen. Das kostet Geld und gefährdet das Projekt.

Spatialists @spatialists@mapstodon.space

· 9mo ago

OSM scraping: An #OpenStreetMap-themed account on LinkedIn is calling on journalists to investigate a surge in what seems to be coordinated #scraping from hundreds of thousands of IP addresses, suspected to be linked to #AI #data collection. This incident... https://spatialists.ch/posts/2026/01/28-osm-scraping/ #GIS #GISchat #geospatial #SwissGIS

OSM scraping – Spatialists – geospatial news

spatialists.ch

OSM scraping – Spatialists – geospatial news

An #OpenStreetMap-themed account on LinkedIn is calling on journalists to investigate a surge in what seems to be coordinated #scraping from hundreds of thousands of IP addresses, suspected to be linked to #AI #data collection. This incident spotlights growing tension between #opendata principles and the potential for misuse of public infrastructure.

You've seen all posts