What web scraping and selectors are
Web scraping is when a program automatically pulls just the information you want out of web pages a person would otherwise open in a browser. Recording shop prices every day, gathering reviews, or collecting notices from many sites into one place all fall under it.
The core tool is the selector. A web page looks like text and images, but underneath it is built from a tag structure called HTML. A selector is like an address that points at where to grab within that structure. For example, the CSS selector .price means 'grab the element with the class price,' and reading that element's text gives you the price.
So a scraper splits into two parts: fetching the page, and parsing the fetched HTML to pick out the pieces you want with a selector. Scrapling handles both.
Why scrapers break on a redesign
The problem is that a selector is tied to the target site's current structure. If I wrote it to grab .btn-primary and the site renames that class to .button-main during a redesign, my selector now points at something that no longer exists. Not a single character of my code changed, but the result is empty.
This is not a rare accident; it happens constantly. The target site has no reason to accommodate my scraper. Class names and structure shift all the time for marketing refreshes, A/B tests, framework swaps. Each time, a human reopens the site, finds the new class name, and edits the selector back in. Most of the maintenance time teams spend on long-running scraping comes from exactly this.
That is why an unbreakable scraper was nearly impossible for a long time — it fundamentally depends on someone else's screen structure. Scrapling's adaptive selector is a way to loosen that dependency.
How the adaptive selector relocates elements
The idea behind the adaptive selector is simple. When it first grabs an element successfully, it stores the element's fingerprint alongside the data — not just the class name but the tag type, parent and sibling relationships, surrounding text, and on-screen position. You save this with the auto_save option.
Next time, when the site has changed and the original selector fails, with the adaptive option on Scrapling searches the whole page by similarity for the element closest to the saved fingerprint. Even if one class name changed, as long as the other clues remain it grabs the moved element again. The step where a human hunts for the new class name and edits the code disappears.
from scrapling.fetchers import StealthyFetcher
StealthyFetcher.adaptive = True
# First run — auto_save=True stores each element's fingerprint
page = StealthyFetcher.fetch('https://example.com')
products = page.css('.product', auto_save=True)
# Later, after the site's structure changed — adaptive=True relocates the moved element
products = page.css('.product', adaptive=True)Install it and try it without writing code
You install it with pip, the Python package manager. One line, pip install scrapling, gives you the parser and HTTP requests. To also use browser automation (dynamic sites, anti-bot bypass), run scrapling install to fetch the needed browsers.
pip install scrapling
# When you also want browser-based features (StealthyFetcher, etc.)
scrapling install# Save a page's body as Markdown
scrapling extract get 'https://example.com' content.md
# Only a specific CSS selector, impersonating Chrome
scrapling extract get 'https://example.com' content.txt --css-selector '.article' --impersonate 'chrome'
# Extract a page that Cloudflare blocks, bypassing it
scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' out.html --solve-cloudflarefrom scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)Beyond adaptive — anti-bot bypass, large crawls, MCP
The other three things Scrapling bundles into one library matter just as much in practice. First, StealthyFetcher bypasses anti-bot defenses like Cloudflare Turnstile out of the box. It spoofs TLS fingerprint and headers to look like a real browser and, when needed, clears the block screen with the solve_cloudflare option. A wall you used to need a paid service to cross became a single library option.
Second, there is a Spider framework for large-scale collection. Like Scrapy, you define a crawler with start_urls and a parse function, and it supports concurrency limits, automatic proxy rotation, pause and resume (stop with Ctrl+C, restart to continue), and streaming results in real time. The same tool scales from a few pages of one site to crawls of millions of pages.
Third, a built-in MCP server ties it to AI. Connect Scrapling to an AI tool like Claude or Cursor over MCP, and instead of the AI swallowing an entire web page, Scrapling extracts just the parts you need first and hands those over. That cuts the tokens the AI has to process, so speed and cost drop together. It is a particularly useful point from an AX standpoint.
Before you use it — legality and courtesy
An easier tool doesn't mean you may scrape anywhere. Scraping is subject to the target site's terms of service, login and paywalls, personal data, and copyright. Scrapling includes safeguards: an option to respect robots.txt (robots_txt_obey), delays that space out requests, and domain blocking. Turning them on is both courtesy to the other server and risk management.
In short, Scrapling is an attempt to move scraping from 'write once, keep fixing' to 'write once, leave it alone.' The adaptive selector cuts maintenance, anti-bot bypass lowers the barrier to entry, and MCP reduces the cost of connecting to AI. If you run collection today or plan to add it, it is an open-source project worth a look.