SCM Freezes during Google Maps Scraper

support
1

1. The problem

For some reason … Google Maps Scraper freezes on every single time when it gets to a certain listing or set of listings. I think it might be something in the listing’s website that hangs up SCM. I’m not sure which website it is but maybe you can tell from the freeze screenshots. I had to force-close SCM and reboot to restart scraping.

2. Screenshot or task log of the problem

Freeze 1

Freeze 2

Freeze 3

I hope it can give you some hint as to why SCM freezes. I am going to try skipping the keyword for now. if it happens again .. i will update you.

2

We need to isolate the steps.

Set google map threads to 1

image
image970×859 36.3 KB

Then run the task again.

Send me screenshot of task log where it gets stuck.

Or export the task and send it to me.

3

Turns out it was just on 1 company website that hung up SCM .. not sure which but I believe its one of the ones in the screenshots I sent you. Hasn’t happened since.

However, I’ve noticed a huge slowdown since last week. My Task settings are the same (5 gmaps threads /10 total) .. as before .. but it is taking 3-4 times as long for same settings. Not sure why. was there something in one of the updates

Take a look at my proxy provider logs:

You’ll notice I went from 70,000 requests per day to around 20,000 per day scraping Gmaps .. almost overnight. Not sure if it’s SCM or not. Was anything updated in SCM last week that might have effected gmaps speed?

4

I added proxy support to the scraping of websites, not sure if you are using the new proxy options?

5

This slowdown happened overnight .. but I think it was before you added the 2nd proxy box. according to the dates in my proxy provider log .. the slowdown started around Aug 17 .. compare to your release changelog. I remember there was a different update LAST week .. I’m not sure what actually changed on that update .. but I think that’s when the slowdown started happening.

6

For sites, it has image search, meta scraping and contact scraping.

Each of those steps takes extra time.

If you disabled all site scraping is it faster?

Also with proxy update, the max retries now uses the value in the settings box. Before it was hard coded to 3.

If you have a high setting it might be stuck doing retries,even if you don’t have proxies enabled.

If you could barrow down the site it would be helpful.

SCM has timeouts when scraping external sites, however it seems that here it is getting stuck somewhere.

7

Yes, I know .. but it was scraping those same features before. none of the settings changed before or after overnight slowdown.

Retries are set to 1 as they were before. if anything, if it was hardcoded to 3 and is now only 1 retry, if anything, it should be faster .. not 3 times slower … right?

I used to be able to fill a 100MB CSV with all fields in 36-48 hrs .. now it takes a week. I’m baffled. I can try to remove the features .. but i need them. Not sure I wish there was timing indicators in the CSV to say how long each of the the scrapes took. Then I could find out if the proxy service was throttling me .. that’s the only other thing I think it is .. my ISP connection is just as fast as before.

8

Can you run it and send me task log?

I will have a look see where its slowing down.

9

When I watch it the speed doesn’t seem much different … but looking in log .. i notice 8min - 10min gaps between entries that I haven’t seen before .. (which I would think would be highly unlikely with my threads set at 5 map /10 task )

8/23/2025, 4:55:46 PM	26 of 41 opening https://www.google.com/maps/place/Affordable+Mattress+Inc/data=!4m7!3m6!1s0x52b32df296d131c1:0xb3fef3443d4c0c4c!8m2!3d45.0046858!4d-93.3079924!16s%2Fg%2F1td587th!19sChIJwTHRlvIts1IRTAxMPUTz_rM?authuser=0&hl=en&rclk=1
8/23/2025, 4:55:45 PM	25 of 41 opening https://www.google.com/maps/place/J+%26+J+Furniture+Plus/data=!4m7!3m6!1s0x52b332507a65beeb:0x7c96a32ade042103!8m2!3d45.0057498!4d-93.3114728!16s%2Fg%2F1vh_80fx!19sChIJ675lelAys1IRAyEE3iqjlnw?authuser=0&hl=en&rclk=1
8/23/2025, 4:55:45 PM	22 of 41 done - Sleep Concepts, Mattress and Futon Factory XP46+PJ Minneapolis, Minnesota
8/23/2025, 4:55:44 PM	20 of 41 done - Slumberland Furniture XG9W+XQ Minnetonka, Minnesota
8/23/2025, 4:55:39 PM	22 of 41 Unable to find contact at https://www.sleepconcepts.com/...
8/23/2025, 4:55:38 PM	20 of 41 looking for contact at https://stores.slumberland.com/ (200) ...
8/23/2025, 4:55:33 PM	22 of 41 Unable to find website meta, Error: ERR_CERT_AUTHORITY_INVALID (-202) loading 'https://www.sleepconcepts.com/'...
8/23/2025, 4:55:28 PM	20 of 41 Found website meta at https://stores.slumberland.com{"metaTitle":"All Slumberland Furniture & Mattress Store Locations","metaDescription":"Shop furniture, mattresses, home décor and more at a local Slumberland Furniture showroom. Find the best service and selection available in the Midwest.","metaKeywords":"","metaImage":""}
8/23/2025, 4:55:24 PM	20 of 41 looking for website meta at https://stores.slumberland.com (200) ...
8/23/2025, 4:52:21 PM	23 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:52:16 PM	24 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:52:14 PM	23 of 41 opening https://www.google.com/maps/place/Mattress+Firm+Knollwood/data=!4m7!3m6!1s0x87f62066a8ffff0d:0xc05808d45d785e71!8m2!3d44.9363198!4d-93.3860093!16s%2Fg%2F11b75gdx_z!19sChIJDf__qGYg9ocRcV54XdQIWMA?authuser=0&hl=en&rclk=1
8/23/2025, 4:52:14 PM	24 of 41 opening https://www.google.com/maps/place/The+Original+Mattress+Factory/data=!4m7!3m6!1s0x52b2d393df527b9b:0x55e8adf1125ad461!8m2!3d45.0125694!4d-93.0477218!16s%2Fg%2F1hc1tlqw_!19sChIJm3tS35PTslIRYdRaEvGt6FU?authuser=0&hl=en&rclk=1
8/23/2025, 4:42:28 PM	18 of 41 done - Mattress Firm Maple Grove II 3HV5+M5 Maple Grove, Minnesota
8/23/2025, 4:42:26 PM	19 of 41 done - Geneva Discount Furniture 3J3M+57 Crystal, Minnesota
8/23/2025, 4:42:11 PM	19 of 41 found Genevadiscountfurniture@gmail.com
8/23/2025, 4:42:07 PM	18 of 41 looking for contact at https://www.mattressfirm.com/ (200) ...
8/23/2025, 4:42:07 PM	19 of 41 looking for contact at https://www.genevadiscountfurniture.com/ (200) ...
8/23/2025, 4:32:13 PM	19 of 41 Found website meta at https://www.genevadiscountfurniture.com{"metaTitle":"Geneva Discount Furniture","metaDescription":"From living rooms and bedrooms to dining spaces, and beyond, our carefully curated selection includes classic, traditional, and modern pieces, all offered at discounted prices. Geneva Discount Furniture also carries a variety of home accents and accessories, including rugs, lighting, wall art, and more decorative items","metaKeywords":"","metaImage":"http://www.genevadiscountfurniture.com/cdn/shop/files/studio_6691e706cd56a4356571720841494_d10af401-2ea6-47c2-a3cb-97dbfc2b23bc.jpg?v=1751562748"}
8/23/2025, 4:32:13 PM	18 of 41 Found website meta at https://www.mattressfirm.com{"metaTitle":"Mattress Firm | Best Prices-Top Brands-Fast and Free Delivery","metaDescription":"Mattress Firm has the best sales on mattresses, beds, adjustable bases, bedding and more from top brands like Tempur-Pedic, Purple, Serta and Beautyrest.","metaKeywords":"","metaImage":""}
8/23/2025, 4:32:09 PM	22 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:32:06 PM	21 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:32:04 PM	22 of 41 opening https://www.google.com/maps/place/Sleep+Concepts,+Mattress+and+Futon+Factory/data=!4m7!3m6!1s0x52b336e555555555:0xb343e892588b8e1b!8m2!3d44.956785!4d-93.2884475!16s%2Fg%2F1tfl0s5_!19sChIJVVVVVeU2s1IRG46LWJLoQ7M?authuser=0&hl=en&rclk=1
8/23/2025, 4:32:04 PM	21 of 41 opening https://www.google.com/maps/place/Online+Mattress+Showroom+Minneapolis+Minnetonka/data=!4m7!3m6!1s0x52b34b9b437b8285:0xcc448518adc21123!8m2!3d44.9699545!4d-93.4538679!16s%2Fg%2F11q4bzb8l7!19sChIJhYJ7Q5tLs1IRIxHCrRiFRMw?authuser=0&hl=en&rclk=1
8/23/2025, 4:32:04 PM	19 of 41 looking for website meta at https://www.genevadiscountfurniture.com (200) ...
8/23/2025, 4:32:04 PM	18 of 41 looking for website meta at https://www.mattressfirm.com (200) ...
8/23/2025, 4:32:01 PM	17 of 41 done - Sleep Number 3HW9+2V Maple Grove, Minnesota
8/23/2025, 4:31:38 PM	16 of 41 done - Mattress Firm 245 Atlantic Avenue M2Q5+QH Brooklyn, New York
8/23/2025, 4:31:26 PM	17 of 41 looking for contact at https://www.sleepnumber.com/ (403) ...
8/23/2025, 4:31:20 PM	16 of 41 looking for contact at https://www.mattressfirm.com/ (200) ...
8/23/2025, 4:31:05 PM	17 of 41 looking for website meta at https://www.sleepnumber.com (403) ...
8/23/2025, 4:31:04 PM	16 of 41 Found website meta at https://www.mattressfirm.com{"metaTitle":"Mattress Firm | Best Prices-Top Brands-Fast and Free Delivery","metaDescription":"Mattress Firm has the best sales on mattresses, beds, adjustable bases, bedding and more from top brands like Tempur-Pedic, Purple, Serta and Beautyrest.","metaKeywords":"","metaImage":""}
8/23/2025, 4:29:50 PM	16 of 41 looking for website meta at https://www.mattressfirm.com (200) ...
8/23/2025, 4:29:32 PM	20 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:29:27 PM	20 of 41 opening https://www.google.com/maps/place/Slumberland+Furniture/data=!4m7!3m6!1s0x52b34ae61d4451ef:0x22c4e74d42debc9f!8m2!3d44.969965!4d-93.4530376!16s%2Fg%2F1tr176gf!19sChIJ71FEHeZKs1IRn7zeQk3nxCI?authuser=0&hl=en&rclk=1
8/23/2025, 4:29:25 PM	15 of 41 done - Sleep Number 3HW9+2V Maple Grove, Minnesota
8/23/2025, 4:29:21 PM	15 of 41 looking for contact at https://www.sleepnumber.com/ (403) ...
8/23/2025, 4:29:10 PM	15 of 41 looking for website meta at https://www.sleepnumber.com (403) ...
8/23/2025, 4:28:26 PM	19 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:28:24 PM	19 of 41 opening https://www.google.com/maps/place/Geneva+Discount+Furniture/data=!4m7!3m6!1s0x52b336db86e27dbd:0xb082fa9d34527ece!8m2!3d45.0529062!4d-93.3668469!16s%2Fg%2F1tr7cg14!19sChIJvX3ihts2s1IRzn5SNJ36grA?authuser=0&hl=en&rclk=1
8/23/2025, 4:28:17 PM	18 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:28:15 PM	14 of 41 done - Mattress Firm Coon Rapids 5JXV+92 Coon Rapids, Minnesota
8/23/2025, 4:28:15 PM	18 of 41 opening https://www.google.com/maps/place/Mattress+Firm+Maple+Grove+II/data=!4m7!3m6!1s0x52b337fe641e0a65:0x9fd249eb4e045520!8m2!3d45.0941613!4d-93.4421037!16s%2Fg%2F11bv3vkbnm!19sChIJZQoeZP43s1IRIFUETutJ0p8?authuser=0&hl=en&rclk=1
8/23/2025, 4:28:12 PM	13 of 41 done - Loyal Home Center 3PCH+GX Fridley, Minnesota
8/23/2025, 4:20:19 PM	14 of 41 looking for contact at https://www.mattressfirm.com/ (200) ...
8/23/2025, 4:20:07 PM	13 of 41 looking for contact at https://loyalhomecentermn.com/ (200) ...
8/23/2025, 4:19:58 PM	17 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:19:56 PM	17 of 41 opening https://www.google.com/maps/place/Sleep+Number/data=!4m7!3m6!1s0x52b34801e243ddb9:0x1d05179794866b5d!8m2!3d45.095039!4d-93.430284!16s%2Fg%2F1vzv39n4!19sChIJud1D4gFIs1IRXWuGlJcXBR0?authuser=0&hl=en&rclk=1
8/23/2025, 4:19:55 PM	14 of 41 Found website meta at https://www.mattressfirm.com{"metaTitle":"Mattress Firm | Best Prices-Top Brands-Fast and Free Delivery","metaDescription":"Mattress Firm has the best sales on mattresses, beds, adjustable bases, bedding and more from top brands like Tempur-Pedic, Purple, Serta and Beautyrest.","metaKeywords":"","metaImage":""}
8/23/2025, 4:19:45 PM	16 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:19:45 PM	12 of 41 done - The Sleep Loft - Downtown Brooklyn Mattress Showroom M2Q6+7G Brooklyn, New York
8/23/2025, 4:19:43 PM	16 of 41 opening https://www.google.com/maps/place/Mattress+Firm+245+Atlantic+Avenue/data=!4m7!3m6!1s0x89c25bef10c7b943:0x66e92618f30e5f79!8m2!3d40.6894294!4d-73.9910165!16s%2Fg%2F11l6yvyjh3!19sChIJQ7nHEO9bwokReV8O8xgm6WY?authuser=0&hl=en&rclk=1
8/23/2025, 4:19:43 PM	13 of 41 Found website meta at https://loyalhomecentermn.com{"metaTitle":"Furniture & Mattress Store In Fridley MN | Royal Home Center","metaDescription":"Discover top-quality furniture and mattresses at Royal Home Center in Fridley, MN. Explore our wide selection of affordable home furnishings and enjoy exceptional customer service. Visit us for great deals.","metaKeywords":"","metaImage":"http://loyalhomecentermn.com/cdn/shop/files/ROYAL_HOME_CENTER_GOLD_LOGO_f26d11b5-60f9-4aad-808e-7c0ca5470e60.png?v=1711163496"}
8/23/2025, 4:19:43 PM	14 of 41 looking for website meta at https://www.mattressfirm.com (200) ...
8/23/2025, 4:19:28 PM	11 of 41 done - The Original Mattress Factory 4G2X+9R Maple Grove, Minnesota
8/23/2025, 4:19:27 PM	12 of 41 found cs@thesleeploft.com
8/23/2025, 4:19:23 PM	11 of 41 looking for contact at https://www.originalmattress.com/ (200) ...
8/23/2025, 4:19:23 PM	12 of 41 looking for contact at https://www.thesleeploft.com/ (200) ...
8/23/2025, 4:19:23 PM	13 of 41 looking for website meta at https://loyalhomecentermn.com (200) ...
8/23/2025, 4:11:01 PM	11 of 41 Found website meta at https://www.originalmattress.com{"metaTitle":"","metaDescription":"","metaKeywords":"","metaImage":""}
8/23/2025, 4:11:00 PM	12 of 41 Found website meta at https://www.thesleeploft.com{"metaTitle":"The Sleep Loft - Online Mattress Showroom - Mattress Store In NYC, Brooklyn & NJ","metaDescription":"Top Rated Mattress Store In New York City, Brooklyn & New Jersey Stores . Voted the #1 Mattress Store in NYC, Brooklyn & NJ. Try and Buy Top online mattresses in our showrooms.","metaKeywords":"","metaImage":"https://thesleeploft.com/cdn/shop/files/32x32.jpg?v=1712949543"}
8/23/2025, 4:10:56 PM	11 of 41 looking for website meta at https://www.originalmattress.com (200) ...
8/23/2025, 4:10:56 PM	12 of 41 looking for website meta at https://www.thesleeploft.com (200) ...
8/23/2025, 4:10:49 PM	15 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:02:54 PM	15 of 41 opening https://www.google.com/maps/place/Sleep+Number/data=!4m7!3m6!1s0x52b34801e243ddb9:0x1d05179794866b5d!8m2!3d45.0950391!4d-93.4302841!16s%2Fg%2F1vzv39n4!19sChIJud1D4gFIs1IRXWuGlJcXBR0?authuser=0&hl=en&rclk=1
8/23/2025, 4:02:52 PM	10 of 41 done - Becker Furniture 4PGQ+9F Blaine, Minnesota
8/23/2025, 4:02:49 PM	10 of 41 looking for contact at https://www.beckerfurnitureworld.com/ (200) ...
8/23/2025, 4:02:41 PM	10 of 41 Found website meta at https://www.beckerfurnitureworld.com{"metaTitle":"","metaDescription":"","metaKeywords":"","metaImage":""}
8/23/2025, 4:02:37 PM	10 of 41 looking for website meta at https://www.beckerfurnitureworld.com (200) ...
8/23/2025, 4:01:54 PM	14 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:01:51 PM	14 of 41 opening https://www.google.com/maps/place/Mattress+Firm+Coon+Rapids/data=!4m7!3m6!1s0x52b33c3c4567a4d5:0xdd8452d62bea5848!8m2!3d45.1983959!4d-93.3574506!16s%2Fg%2F11b5wqjxjy!19sChIJ1aRnRTw8s1IRSFjqK9ZShN0?authuser=0&hl=en&rclk=1
8/23/2025, 4:01:49 PM	6 of 41 done - Liam Furniture 4J57+QW Brooklyn Park, Minnesota
8/23/2025, 4:01:44 PM	13 of 41 Getting next proxy: dc.decodo.com:10000 ...
8/23/2025, 4:01:42 PM	13 of 41 opening https://www.google.com/maps/place/Loyal+Home+Center/data=!4m7!3m6!
10

Very big 8min gaps.

Can you set threads to 1,then run it again.

I need to know at what steps it is getting stuck on.

Paste in log again once you run it.

11

I’m halfway through a job that started 3.5 days ago .. as soon as its done, i can set to 1 thread … in meantime, maybe you can try to duplicate issue.

12

Can you email me proxy details for test?

13

right now i’m using same rotating decodo datacenter proxies in both proxy boxes (1 retry per proxy). try your test without proxies to see if you can duplicate. I will do same as soon as this job is done to see if it is related to proxies or not.

14

I ran it without proxies and it works.

I’m thinking it might be proxy related, either cause its rotating proxies until it works or proxies are slow.

15

Okay .. I am running it on single thread and I didn’t see the 10m gaps yet … but I did notice .. that the “skips” from duplicates take 1 minute each to process now. if it skips 40 items, then it takes 40 min to skip. But once I increased task threads to 2 .. this situation fixed itself and only takes 1 second per skip.

8/25/2025, 3:49:13 PM	Skip (existing url) https://www.google.com/maps/place/Randy%27s+Electric+%26+Plumbing/data=!4m7!3m6!1s0x52b339c4e37e4867:0xc558759ca53d4723!8m2!3d45.11043!4d-93.385201!16s%2Fg%2F1hc3p___6!19sChIJZ0h-48Q5s1IRI0c9pZx1WMU?authuser=0&hl=en&rclk=1

8/25/2025, 3:48:13 PM	Skip (existing url) https://www.google.com/maps/place/Plus+Maintenance+MN/data=!4m7!3m6!1s0x52b330c1292f3791:0x8bbe66dd4560981a!8m2!3d45.0794939!4d-93.338201!16s%2Fg%2F11ddyh_1t5!19sChIJkTcvKcEws1IRGphgRd1mvos?authuser=0&hl=en&rclk=1

8/25/2025, 3:47:13 PM	Skip (existing url) https://www.google.com/maps/place/Personal+Pride+Construction/data=!4m7!3m6!1s0x52b335a40a8d56b1:0x2520dd9ee97ad6aa!8m2!3d44.992221!4d-93.399158!16s%2Fg%2F1tg4wh50!19sChIJsVaNCqQ1s1IRqtZ66Z7dICU?authuser=0&hl=en&rclk=1

8/25/2025, 3:46:13 PM	Skip (existing url) https://www.google.com/maps/place/Windsor+Gates/data=!4m7!3m6!1s0x52b33754b9591ab7:0x98823a4a7296ba54!8m2!3d45.0970144!4d-93.3585248!16s%2Fg%2F1v9lbf4k!19sChIJtxpZuVQ3s1IRVLqWcko6gpg?authuser=0&hl=en&rclk=1

8/25/2025, 3:45:13 PM	Skip (existing url) https://www.google.com/maps/place/LeBrun+Electric,+Heating+%26+Cooling/data=!4m7!3m6!1s0x52b49de4138c0001:0x66dd38378b3e0868!8m2!3d45.0915665!4d-93.3912614!16s%2Fg%2F11j03btncl!19sChIJAQCME-SdtFIRaAg-izc43WY?authuser=0&hl=en&rclk=1

8/25/2025, 3:44:13 PM	Skip (existing url) https://www.google.com/maps/place/Service+Star+Plumbing/data=!4m7!3m6!1s0x52b33183bb236d1f:0x31d66cdbd78dfbea!8m2!3d45.1072292!4d-93.3270625!16s%2Fg%2F11j81khv86!19sChIJH20ju4Mxs1IR6vuN19ts1jE?authuser=0&hl=en&rclk=1

8/25/2025, 3:43:13 PM	Skip (existing url) https://www.google.com/maps/place/Jewell+Scanning+%26+Coring+LLC/data=!4m7!3m6!1s0x52b331ebb95ca96f:0x4aa46d878894b9dc!8m2!3d45.0960845!4d-93.2826631!16s%2Fg%2F11pxs425dl!19sChIJb6lcuesxs1IR3LmUiIdtpEo?authuser=0&hl=en&rclk=1

8/25/2025, 3:42:13 PM	Skip (existing url) https://www.google.com/maps/place/Goodwill+-+Brooklyn+Park+Outlet/data=!4m7!3m6!1s0x52b3376b0f928123:0xf2cb4c992dadb6e7!8m2!3d45.0818811!4d-93.378218!16s%2Fg%2F11b764q9c_!19sChIJI4GSD2s3s1IR57atLZlMy_I?authuser=0&hl=en&rclk=1

8/25/2025, 3:41:13 PM	Skip (existing url) https://www.google.com/maps/place/Pillar+Construction/data=!4m7!3m6!1s0x52b337724b08c5a3:0x88a6a772303e1caa!8m2!3d45.0865957!4d-93.3888458!16s%2Fg%2F11fdwt31zr!19sChIJo8UIS3I3s1IRqhw-MHKnpog?authuser=0&hl=en&rclk=1

8/25/2025, 3:40:13 PM	Skip (existing url) https://www.google.com/maps/place/Scheff+Design+%26+Remodeling+Inc./data=!4m7!3m6!1s0x52b334f702429c83:0x171255acfc0a6580!8m2!3d45.1027374!4d-93.496705!16s%2Fg%2F1hd_cm5rd!19sChIJg5xCAvc0s1IRgGUK_KxVEhc?authuser=0&hl=en&rclk=1

8/25/2025, 3:39:13 PM	Skip (existing url) https://www.google.com/maps/place/Magic+Carpets,+Your+Shaw+Flooring+Center/data=!4m7!3m6!1s0x52b33768afe1cafd:0xfa5482c47a2dd100!8m2!3d45.0896219!4d-93.3798878!16s%2Fg%2F1tdyqfmq!19sChIJ_crhr2g3s1IRANEtesSCVPo?authuser=0&hl=en&rclk=1

8/25/2025, 3:38:13 PM	Skip (existing url) https://www.google.com/maps/place/The+Market/data=!4m7!3m6!1s0x52b337047255b299:0xde4a9da209258e34!8m2!3d45.0846749!4d-93.3771181!16s%2Fg%2F11h4djb9z2!19sChIJmbJVcgQ3s1IRNI4lCaKdSt4?authuser=0&hl=en&rclk=1

8/25/2025, 3:37:13 PM	Skip (existing url) https://www.google.com/maps/place/VANGUARD+HOMES/data=!4m7!3m6!1s0xa9653537d3c3a097:0xc2b5c45d97210610!8m2!3d46.4419165!4d-93.3614534!16s%2Fg%2F11sj98xdfh!19sChIJl6DD0zc1ZakREAYhl13EtcI?authuser=0&hl=en&rclk=1

8/25/2025, 3:36:13 PM	Skip (existing url) https://www.google.com/maps/place/Century+Drywall+%26+Painting/data=!4m7!3m6!1s0x52b33052a3edfc63:0xc0160dff98888c92!8m2!3d45.0915665!4d-93.3912614!16s%2Fg%2F1tgpxvf1!19sChIJY_zto1Iws1IRkoyImP8NFsA?authuser=0&hl=en&rclk=1
16

This is not makin sense .. even with 3 threads going now .. its doing the same thing … taking EXACTLY 1 minute each to skip over each duplicate. these slowdown issues are intermittent. very hard to determine a pattern. but they ARE happening.

8/26/2025, 3:16:15 AM	Skip (existing url) https://www.google.com/maps/place/Office+Depot/data=!4m7!3m6!1s0x52b337f1416084d9:0xb2f54a98eed11006!8m2!3d45.0928984!4d-93.422259!16s%2Fg%2F1tdqzc5r!19sChIJ2YRgQfE3s1IRBhDR7phK9bI?authuser=0&hl=en&rclk=1
8/26/2025, 3:15:15 AM	Skip (existing url) https://www.google.com/maps/place/Trusted+Semiconductor+Solutions/data=!4m7!3m6!1s0x52b337f9e47ac06f:0x55551663990e9522!8m2!3d45.083342!4d-93.390721!16s%2Fg%2F12q4rgmsr!19sChIJb8B65Pk3s1IRIpUOmWMWVVU?authuser=0&hl=en&rclk=1
8/26/2025, 3:14:15 AM	Skip (existing url) https://www.google.com/maps/place/Antares/data=!4m7!3m6!1s0x52b3383207352073:0xe93d2f85b9751dcc!8m2!3d45.1084868!4d-93.4029969!16s%2Fg%2F1tjdqzxz!19sChIJcyA1BzI4s1IRzB11uYUvPek?authuser=0&hl=en&rclk=1
8/26/2025, 3:13:15 AM	Skip (existing url) https://www.google.com/maps/place/DataBank+Data+Centers+-+Brooklyn+Park/data=!4m7!3m6!1s0x52b339fcef4f646b:0xb114458ac42f843e!8m2!3d45.1437775!4d-93.3877138!16s%2Fg%2F11tsk5_hlp!19sChIJa2RP7_w5s1IRPoQvxIpFFLE?authuser=0&hl=en&rclk=1
8/26/2025, 3:12:15 AM	Skip (existing url) https://www.google.com/maps/place/Pabak/data=!4m7!3m6!1s0x52b3314f7b2f4fcd:0x36a7c824cdfa203c!8m2!3d45.096407!4d-93.2964767!16s%2Fg%2F11j7jnv1q7!19sChIJzU8ve08xs1IRPCD6zSTIpzY?authuser=0&hl=en&rclk=1
8/26/2025, 3:11:15 AM	Skip (existing url) https://www.google.com/maps/place/Point+B+Solutions/data=!4m7!3m6!1s0x52b33463ab42a1d7:0xdb5ecd7de7146966!8m2!3d45.1253174!4d-93.3740235!16s%2Fg%2F11b6bqf3s_!19sChIJ16FCq2M0s1IRZmkU533NXts?authuser=0&hl=en&rclk=1
8/26/2025, 3:10:15 AM	Skip (existing url) https://www.google.com/maps/place/POSproviders/data=!4m7!3m6!1s0x52b338323b85bbe9:0xdb1b8ee46db7100f!8m2!3d45.1122945!4d-93.403298!16s%2Fg%2F11c3kcflbn!19sChIJ6buFOzI4s1IRDxC3beSOG9s?authuser=0&hl=en&rclk=1
8/26/2025, 3:09:15 AM	Skip (existing url) https://www.google.com/maps/place/Veracity+Technologies+-+Minneapolis+IT+Services/data=!4m7!3m6!1s0x87f621e7f849e863:0xa787d914359f568e!8m2!3d44.8959401!4d-93.4466716!16s%2Fg%2F1tmpl1p9!19sChIJY-hJ-Och9ocRjlafNRTZh6c?authuser=0&hl=en&rclk=1
8/26/2025, 3:08:15 AM	Skip (existing url) https://www.google.com/maps/place/Blessed+Business+Solutions/data=!4m7!3m6!1s0x52b3375258e96455:0xc30baf904deadcfa!8m2!3d46.423669!4d-129.9427086!16s%2Fg%2F11cmtb038t!19sChIJVWTpWFI3s1IR-tzqTZCvC8M?authuser=0&hl=en&rclk=1
8/26/2025, 3:07:15 AM	Skip (existing url) https://www.google.com/maps/place/Cybertrol+Engineering/data=!4m7!3m6!1s0x52b34a230d6fc9cd:0x8a683c87e1fae0b6!8m2!3d45.0738294!4d-93.4473248!16s%2Fg%2F1wj_sxx5!19sChIJzclvDSNKs1IRtuD64Yc8aIo?authuser=0&hl=en&rclk=1
8/26/2025, 3:06:15 AM	Skip (existing url) https://www.google.com/maps/place/Brooklyn+Park+Small+Business+Center/data=!4m7!3m6!1s0x847082f4bad0cb1:0x1a511471136c4869!8m2!3d45.0955659!4d-93.3751705!16s%2Fg%2F11vdm74fdh!19sChIJsQytSy8IRwgRaUhsE3EUURo?authuser=0&hl=en&rclk=1
8/26/2025, 3:05:15 AM	Skip (existing url) https://www.google.com/maps/place/Three+North+Agency/data=!4m7!3m6!1s0x52b332ee2e02b66f:0x5f2700ee3059fbe3!8m2!3d45.1240748!4d-93.37382!16s%2Fg%2F11c0v3vg6f!19sChIJb7YCLu4ys1IR4_tZMO4AJ18?authuser=0&hl=en&rclk=1
8/26/2025, 3:04:15 AM	Skip (existing url) https://www.google.com/maps/place/Techmate/data=!4m7!3m6!1s0x52b339128105074b:0xa637523502b323ec!8m2!3d45.1175322!4d-93.4018537!16s%2Fg%2F11rfdfghlb!19sChIJSwcFgRI5s1IR7COzAjVSN6Y?authuser=0&hl=en&rclk=1
8/26/2025, 3:03:15 AM	Skip (existing url) https://www.google.com/maps/place/Bidias+Systems+Inc/data=!4m7!3m6!1s0x52b33b49209feec3:0x784c5661d7fcc126!8m2!3d45.108522!4d-93.3354967!16s%2Fg%2F11qqv0db86!19sChIJw-6fIEk7s1IRJsH812FWTHg?authuser=0&hl=en&rclk=1
8/26/2025, 3:02:15 AM	Skip (existing url) https://www.google.com/maps/place/a+COUPLE+of+GURUS/data=!4m7!3m6!1s0x52b3328ac3b26e8d:0xc5c01292696fdb28!8m2!3d44.988663!4d-93.281582!16s%2Fg%2F1tf310zn!19sChIJjW6yw4oys1IRKNtvaZISwMU?authuser=0&hl=en&rclk=1
8/26/2025, 3:01:15 AM	Skip (existing url) https://www.google.com/maps/place/Nurzee+Inc/data=!4m7!3m6!1s0x52b337bf769aeb0b:0xd52a4115790e398!8m2!3d45.0940669!4d-93.3700215!16s%2Fg%2F11fp3qmqw3!19sChIJC-uadr83s1IRmOOQVxGkUg0?authuser=0&hl=en&rclk=1
8/26/2025, 3:00:15 AM	Skip (existing url) https://www.google.com/maps/place/RYMARK+IT+Support+%7C+Minneapolis+IT+Services+and+Cybersecurity+Company+in+Twin+Cities/data=!4m7!3m6!1s0x52b3315f5bf79683:0x13d4a581e2ed7385!8m2!3d45.0672774!4d-93.3075111!16s%2Fg%2F11rg_s4x_n!19sChIJg5b3W18xs1IRhXPt4oGl1BM?authuser=0&hl=en&rclk=1
8/26/2025, 2:59:15 AM	Skip (existing url) https://www.google.com/maps/place/Axiom+IT+Services/data=!4m7!3m6!1s0x52b336634b13a16b:0xe5aa42fde704b3bf!8m2!3d45.0830123!4d-93.3889858!16s%2Fg%2F11b5pjhz6m!19sChIJa6ETS2M2s1IRv7ME5_1CquU?authuser=0&hl=en&rclk=1
8/26/2025, 2:58:15 AM	Skip (existing url) https://www.google.com/maps/place/Point+B+Solutions/data=!4m7!3m6!1s0x52b33463ab42a1d7:0xdb5ecd7de7146966!8m2!3d45.1253174!4d-93.3740235!16s%2Fg%2F11b6bqf3s_!19sChIJ16FCq2M0s1IRZmkU533NXts?authuser=0&hl=en&rclk=1
8/26/2025, 2:57:15 AM	Skip (existing url) https://www.google.com/maps/place/Organic+Navigation+LLC/data=!4m7!3m6!1s0x52b33bee1c5d4531:0xdf7d3b69b0d5e064!8m2!3d45.1077501!4d-93.3350647!16s%2Fg%2F11w8zbh14s!19sChIJMUVdHO47s1IRZODVsGk7fd8?authuser=0&hl=en&rclk=1
8/26/2025, 2:56:15 AM	Skip (existing url) https://www.google.com/maps/place/Three+North+Agency/data=!4m7!3m6!1s0x52b332ee2e02b66f:0x5f2700ee3059fbe3!8m2!3d45.1240748!4d-93.37382!16s%2Fg%2F11c0v3vg6f!19sChIJb7YCLu4ys1IR4_tZMO4AJ18?authuser=0&hl=en&rclk=1
8/26/2025, 2:55:15 AM	Skip (existing url) https://www.google.com/maps/place/Techmate/data=!4m7!3m6!1s0x52b339128105074b:0xa637523502b323ec!8m2!3d45.1175322!4d-93.4018537!16s%2Fg%2F11rfdfghlb!19sChIJSwcFgRI5s1IR7COzAjVSN6Y?authuser=0&hl=en&rclk=1
8/26/2025, 2:54:15 AM	Skip (existing url) https://www.google.com/maps/place/CFC+Technology+Corporation/data=!4m7!3m6!1s0x52b34a3134cb3dcb:0xd52f3358dcd01e7e!8m2!3d45.0829972!4d-93.3891534!16s%2Fg%2F1ptvt8nn5!19sChIJyz3LNDFKs1IRfh7Q3FgzL9U?authuser=0&hl=en&rclk=1
8/26/2025, 2:53:15 AM	Skip (existing url) https://www.google.com/maps/place/Blessed+Business+Solutions/data=!4m7!3m6!1s0x52b3375258e96455:0xc30baf904deadcfa!8m2!3d46.423669!4d-129.9427086!16s%2Fg%2F11cmtb038t!19sChIJVWTpWFI3s1IR-tzqTZCvC8M?authuser=0&hl=en&rclk=1
8/26/2025, 2:52:15 AM	Skip (existing url) https://www.google.com/maps/place/Axiom+IT+Services/data=!4m7!3m6!1s0x52b336634b13a16b:0xe5aa42fde704b3bf!8m2!3d45.0830123!4d-93.3889858!16s%2Fg%2F11b5pjhz6m!19sChIJa6ETS2M2s1IRv7ME5_1CquU?authuser=0&hl=en&rclk=1
8/26/2025, 2:51:15 AM	Skip (existing url) https://www.google.com/maps/place/MNTech/data=!4m7!3m6!1s0x52b3315850ee1cf3:0x2f41f8d55a2dd5fb!8m2!3d45.092151!4d-93.3545098!16s%2Fg%2F11h6rn48dr!19sChIJ8xzuUFgxs1IR-9UtWtX4QS8?authuser=0&hl=en&rclk=1
8/26/2025, 2:50:15 AM	Skip (existing url) https://www.google.com/maps/place/Dedicated+Networks+Inc/data=!4m7!3m6!1s0x52b33fbbc4d1a52b:0xa5d4bed62b949891!8m2!3d45.1259978!4d-93.394969!16s%2Fg%2F1260l_1vq!19sChIJK6XRxLs_s1IRkZiUK9a-1KU?authuser=0&hl=en&rclk=1
17

How many other tasks are you running simultaneously in SCM?

18

I looked closely and there is a 1sec delay when posting log messages from google maps browser.

Not sure if it makes any difference, but I removed that delay.

How skip code works is that it loops an array and prints out any urls that already exist in that array.

Unless that array is massive, I don’t see why it would be so slow.

Can you open the task data folder.

image
image510×443 12.9 KB

Look for the partial plusurls file

image
image963×207 9.6 KB

How big is that file in the task you are running?

19

only 1 task at a time. I pick 1 town and scrape 200 business categories in the town. Usually It works out well, because there are alot of overlap duplicates from Google between the categories and SCM auto-removes the duplicates.

The plusurls file is current 1.6MB … what does this file do?

What was strange was just how it slowed down overnight .. going from 70k requests per day to 20k requests per day literally the next day. This didn’t matter when I changed tasks.

20

Can you export that task and also send me the plusurl file?

I will check if that is causing the slowdown or not

Did you try the new SCM update too?