1. The problem
Would you mind updating the program to limit searches to 6/minute, and also tell the user it’s doing that?
Yes, Please.
-
The problem
To set max searches per 10 mins, requires manual adjustment of the search pause time in settings. -
Simple solution
Automatically slow down searches to keep under 5 searches per 10 mins.
If you can provide an option to keep the searches under a safe limit, that will be helpful to avoid captchas.
I have been thinking about this recently, part of the problem is that once you are issued captchas, even if you solved them after 4-5 times of that Google will start to automatically reject all your searches (even if captcha is solved)
There is a hard limit to the number of searches you can do per hour.
Solution ideas:
- Limit searches to xx in 1 minute, remove the existing delay based option and instead let SCM slow down each request so its only xx per minute
- On captcah request, slow down the next requests with an additional time penalty, eg +2 minute wait on next requests.
Captchas are not a long term solution because Google only allows you to solve xxx captchas an hour before rejecting your solves.
I think option 1 is better.
Limit searches to xx in 1 minute, remove the existing delay based option and instead let SCM slow down each request so its only xx per minute
1 Like
This is still in progress.
Turns out my method is not thread safe.
So multiple tasks starting at the same time both end up waiting and there is a race condition.
Turns out the fix is to redo all the task code and move it from running in the web browser to running it in node instead.
This might even finally solve problems where tasks go out of memory and the UI becomes white and unresponsive.
This is a big refactor and will require a lot of work and debug, basically most of the codebase needs to be touched up.
Rate limiter working.
Each task will wait for its turn to access the search engine.
If there is a free slot it proceeds.
If there isn’t a free slot it will wait.
With multiple queued tasks, each one will wait as required.
Only one active search engine window can be opened at the same time.
So SCM will never open 2 windows to Google at the same time.
Even if you allow 3 requests in 60 seconds, each request has to be completed one at a time to avoid looking like a bot.
