logo
Welcome to our new AbleCommerce forums. As a guest, you may view the information here. To post to this forum, you must have a registered account with us, either as a new user evaluating AbleCommerce or an existing user of the application. For all questions related to the older version of Gold and earlier, please go to AbleCommerce Gold forum. Please use your AbleCommerce username and password to Login. New Registrations are disabled.

Notification

Icon
Error
Managing bots with large catalogs
Options
Go to last post Go to first unread
Previous Topic Next Topic
Joe Payne2  
#1 Posted : Sunday, July 11, 2021 7:23:43 AM(UTC)
Joe Payne2

Rank: Advanced Member

Groups: HelpDesk, Developers
Joined: 11/9/2018(UTC)
Posts: 564

Thanks: 122 times
Was thanked: 26 time(s) in 25 post(s)
We are finding that bots are causing major issues in an Able store with a large catalog i.e. 200,000+ products.

The bots don't persist the session cookie. So every bot page request creates a new Able user record. After a few days, we have millions of records in ac_Users. This slows down the site overall. Able maintenance routines just don't clean up anonymous users fast enough to keep up.

I'm thinking about adding a store setting for bot user agent strings. In WebUserLocator(), I'll check the request user agent against the bot list. If it's a bot match, I will locate an existing ac_Users record reserved just for bots, instead of creating a new user.

Do you foresee any problems with that implementation? I've never worked that low in the application before so I am unsure if I would be impacting other areas.
Back to top

Wanna join the discussion?! Login to your AbleCommerce Forums forum account. New Registrations are disabled.
Back to top  
judy at Web2Market  
#2 Posted : Monday, July 12, 2021 6:04:44 AM(UTC)
judy at Web2Market

Rank: Advanced Member

Groups: Developers
Joined: 11/7/2018(UTC)
Posts: 303

Thanks: 21 times
Was thanked: 5 time(s) in 5 post(s)
I would be interested in this also. I see that AC9 still has the code for the BOT_USER_GUID but we found that in Gold sites this didn't always work.
Back to top
mazhar  
#3 Posted : Tuesday, July 13, 2021 8:58:07 AM(UTC)
mazhar

Rank: Administration

Groups: Admin, Administrators, HelpDesk, System, Authorized User, Developers, Registered
Joined: 10/5/2018(UTC)
Posts: 175

Thanks: 8 times
Was thanked: 17 time(s) in 15 post(s)
If BOT doesn't have cookie support they must be handled through a shared user instance. I wonder if bots in your case do support cookies but throw it away after few requests. If you are certain about user agents then I think what are you suggesting with User Locater may work.
Back to top
Joe Payne2  
#4 Posted : Tuesday, July 13, 2021 10:48:13 AM(UTC)
Joe Payne2

Rank: Advanced Member

Groups: HelpDesk, Developers
Joined: 11/9/2018(UTC)
Posts: 564

Thanks: 122 times
Was thanked: 26 time(s) in 25 post(s)
Everything I've seen so far in IIS logs shows both the bing bot and google bot do not keep the cookie between requests. From what I have found online, others say the same thing.
Back to top
mazhar  
#5 Posted : Tuesday, July 13, 2021 11:33:06 AM(UTC)
mazhar

Rank: Administration

Groups: Admin, Administrators, HelpDesk, System, Authorized User, Developers, Registered
Joined: 10/5/2018(UTC)
Posts: 175

Thanks: 8 times
Was thanked: 17 time(s) in 15 post(s)
Which version of AbleCommerce is used on this store?
Back to top
Joe Payne2  
#6 Posted : Tuesday, July 13, 2021 11:43:01 AM(UTC)
Joe Payne2

Rank: Advanced Member

Groups: HelpDesk, Developers
Joined: 11/9/2018(UTC)
Posts: 564

Thanks: 122 times
Was thanked: 26 time(s) in 25 post(s)
Gold R12 SR2
Back to top
Joe Payne2  
#7 Posted : Tuesday, July 27, 2021 6:31:32 AM(UTC)
Joe Payne2

Rank: Advanced Member

Groups: HelpDesk, Developers
Joined: 11/9/2018(UTC)
Posts: 564

Thanks: 122 times
Was thanked: 26 time(s) in 25 post(s)
So we took the bot code from Gold R12 and merged it back into WebUserLocator() in the Gold R10 install. The result was a huge improvement. We're no longer seeing massive growth in user records ( +250,000/day ) and maintenance routines are able to keep up once more.

Big relief seeing this issue put to bed once and for all. It's been a huge hassle keeping the db clean.
Back to top
mazhar  
#8 Posted : Tuesday, July 27, 2021 9:58:17 AM(UTC)
mazhar

Rank: Administration

Groups: Admin, Administrators, HelpDesk, System, Authorized User, Developers, Registered
Joined: 10/5/2018(UTC)
Posts: 175

Thanks: 8 times
Was thanked: 17 time(s) in 15 post(s)
Perhaps the version number you mentioned earlier was mistaken for some other install. If it was R12 install it would already have the bot protection, glad that you were able to handle it using code from R12.
Back to top
Joe Payne2  
#9 Posted : Tuesday, July 27, 2021 10:06:47 AM(UTC)
Joe Payne2

Rank: Advanced Member

Groups: HelpDesk, Developers
Joined: 11/9/2018(UTC)
Posts: 564

Thanks: 122 times
Was thanked: 26 time(s) in 25 post(s)
Yikes, you're right I did mention the wrong version. My apologies!

Ugh I need a vacation.
Back to top
Users browsing this topic
Guest (5)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Privacy Policy | Powered by YAF.NET 2.2.4.14 | YAF.NET © 2003-2024, Yet Another Forum.NET
This page was generated in 0.207 seconds.