SEO work needs alot of “what ranks for this and why”, not “give me a keyword list”.
More of you are running Claude or Codex to do the research part.
The actual problem
Scraping at volume is messy.
Captchas, ads, engine modules (videos, images, news) mixed in with real results.
Same page showing up twice under different URLs.
So the url tool does this all for you now.
What changed
Every result row expanded: query, rank, url, title, snippet.
All queries in a batch go to 1 CSV (search-results.csv).
MCP returns csv results
What you get
- site: and “exact phrase” pass through as typed
- Free engines: Google, Bing, DuckDuckGo, Baidu
- Cleaner results:
- no jump links or video timestamps as extra rows
- ads and engine modules skipped
- snippets without the date stuck on the front
- Chinese and other non-Latin URLs readable
- DuckDuckGo now goes past the first 10
- Baidu pages like a real person, clicks next instead of loading URLs
- Captcha shows up? The window opens so you can solve it, same as Google already did
- Stop a batch halfway, it resumes where it left off
MCP
- New tool: url_finder_create
- Your agent sends a list of queries, runs it with task_run, reads it back with task_results
- task_results gives 1 status row per query plus the CSV rows
- So you can ask Claude “top 30 results for these 200 keywords, which domains show up most” and it works off real SERPs
Where this goes
- SERPs data is now MCP ready
- With title and snippet in the file, your agent can find gaps, group keywords by the pages ranking and compare competitors