Do I need to add proxies for the tool to run all the scrapping tasks?
Proxies are only used for search engine scraping. ie Google.
Proxies are not used when scraping websites.
What happens if I don’t use proxies?
If you start requesting more than 6 searches a minute, Google will start showing captchas.
SCM has its own captcha breaking service or you can unblock results manually by going into Settings > Test Google and completing captchas there.
If they are not solved, eventually Google stops serving results to you for 1 hour or so and starts displaying 503 forbidden errors.
What kind of proxies should I use?
Google is good at blocking proxies, so they aren’t always the first and easiest solution.
The best proxies are private residential proxies IPs. These will cost at least $1 per IP if not double or triple that amount.
Rotating proxies etc don’t work as they are shared and people burn through them with Google searches and never solve the captcha requests. They end up getting flagged by Google.
You waste money and more time waiting for 1 proxy to work when you could have gotten there faster by just waiting 1 minute between each search on your direct IP.
You can also use a VPN to change your IP.
The SCM captcha breaking service may also not work properly behind proxies. If so you can turn it off in the settings menu.
How to view your search usage
SCM tracks all your Google search requests over the hour and displays it as searches per minute.
Keep this number under 0.50 for a reliable service.
How to space out your searches
Settings > Search
Set a wait time that lets you keep under 0.5 search rpm.
eg 60s
If you only do searches here and there you can set wait time to 0
Search engine fallback
SCM has a handy feature where if Google search doesn’t work it will try Bing and then Duckduckgo for you automatically!
Do I need poxies to scrape urls of my competitors content?
Proxies are not used by SCM for scraping websites.
You should instead add wait times between requests.