searchts

The problem

Ask an AI agent to go read a web page and watch what happens. Cloudflare, PerimeterX, or DataDome takes one look at its naive fetch, decides it is a robot, and slams the door. The agent gets a CAPTCHA or an empty shell, shrugs, and quotes some third-party summary instead of the source. I got tired of watching that happen.

The trick paid unlockers do not spell out

Commercial unlockers charge real money to punch through bot-walls, but what you are actually renting is their pool of millions of clean residential IP addresses. Here is the joke: you already have one. searchts runs on your machine, from your home connection, at personal volume. The single most expensive piece of the paid product is sitting in your house.

What a bot-wall actually checks

A bot-wall is a bouncer with a checklist, and each line falls to a different trick:

• Do you look like a browser? Headers. Trivial.
• Do you sound like a browser? The TLS and HTTP/2 handshake, the JA3 fingerprint. Real Chrome has a distinctive accent; scripts sound robotic. This is the key lever.
• Can you run JavaScript? Needs a real engine.
• Can you press and hold like a human? An interactive CAPTCHA. No free robot beats this, and searchts admits it instead of faking success.
• Which neighborhood are you from? Datacenter IPs get flagged. Your home IP walks right in.

The escalating ladder

So a fetch walks a ladder, cheapest tier first, and stops at the first real content:

• curl_cffi puts on a real Chrome's exact TLS fingerprint in a single call. Fast, local, private: the URL never leaves your machine.
• Jina Reader, a JavaScript-rendering relay, for pages that only exist after the JS runs.
• A stealth browser (patchright), launched lazily only when the cheap tiers fail. You pay its 300-600 MB on hard pages, never at idle.

The ladder remembers which tier worked per domain, so the second visit starts at the cheapest thing that works, and everything comes back as clean Markdown.

The bugs that taught me block detection

Deciding real page or wall is where naive implementations die, and every rule here was paid for with a real bug. Zillow's genuine homepage ships the PerimeterX sensor script, so matching vendor names falsely flagged 432 KB of real content: match the wall's interstitial phrases, never its vendor. A 500-character minimum called example.com blocked, because short is not blocked, short is an escalation hint. And one relay returned HTTP 200 with a body politely explaining the upstream 403, a failure dressed as success, straight onto the block list.

Proof

One benchmark row says it all. Zillow: naive fetch, 403. Fingerprint tier, a genuine 200 with 422 KB of real listing data. Same request, same machine, same afternoon. And g2.com, sitting behind DataDome's interactive CAPTCHA, was reported blocked honestly instead of returning junk, because a tool that cannot be trusted to say no cannot be trusted to say yes either.

The pages the ladder could not read

Then something beat the whole ladder with no bot-wall in sight. Paste a ChatGPT or Claude share link and all three tiers come back with a thin shell or a conversation cut off mid-sentence. Nothing was blocking me. A chat share page is a single-page app, and the transcript never lands in the page as text worth extracting, so there was simply nothing there to read.

The fix runs ahead of the ladder instead of inside it: recognize the share URL, then read the provider's own data channel rather than the page it paints. Eight are handled now, in two shapes.

• Five hand the transcript over with no browser at all. ChatGPT and Poe bury it in the page payload (a React Router turbo-stream, a __NEXT_DATA__ blob); Claude, Grok and Gemini answer a keyless endpoint that the Chrome-impersonated fetch already clears.
• Three are JavaScript shells: DeepSeek, Perplexity and Copilot. Those borrow the stealth tier's browser, wait on a ready selector, scroll until the page height stops moving so virtualization cannot truncate the chat, then click the collapsed sections open before reading.

Each provider is one auto-discovered module, so adding the ninth is adding a file, and an extractor that fails drops through to the normal ladder rather than failing the read. The benchmark covers the five that need no browser and passes all five. The three that need one are not in it yet.

Beyond the unlocker

Reading is a third of it. searchts also searches (keyless, multi-provider, results fused with reciprocal rank) and transcribes video, subtitles-first with a Whisper fallback. It ships as a CLI, an MCP server with six tools, a Claude Code skill, and a plain Python library. Fetched content is scrubbed for invisible-character tricks and prompt-injection tells before it reaches an agent, and every read comes with a receipt (which tier, when, final URL), so what an agent read becomes a citation another agent can replay.

There is no per-platform router. `searchts doctor` only probes whether optional CLIs like gh or opencli are on PATH and logged in, and being on PATH is not a claim that searchts reads those sites through them. I retracted the older line about installed CLIs unlocking native GitHub, X, Reddit, LinkedIn and RSS channels, because it was not true and the README says so now.

The honest ceiling

An interactive CAPTCHA still needs a human. Instead of pretending otherwise, a --human flag opens a real browser, you solve it once, and the fetch continues. Personal scale only: one home IP at low volume, not a mass scraper. Built on Agent-Reach (MIT), shipped MIT on PyPI. No API key, no proxy bill, no subscription.