Google and bing spider crawling useless URLsComing from PrestaShop to Magento CE: How to handle links already in Google/Bing Index?How Can I fix missing title tags and duplicate URLs?magento shop down by bots crawlingHigh CPU load due to Google crawling search termsRedirect for urls with filename and query stringRemove useless zero after numberProduct url give 404 error to crawling even product is enabled and visibility set to catalog search.Magento and Google Customer Reviewswhat is wrong with robots.txt this file?Google complaining about too many duplicate pages after enabling SEO urls
Is "stainless" a bulk or a surface property of stainless steel?
Alchemist potion on Undead
Does C++20 mandate source code being stored in files?
How does the Saturn V Dynamic Test Stand work?
Church Booleans
How to detect a failed AES256 decryption programmatically?
Why doesn't the Falcon-9 first stage use three legs to land?
Repurpose telephone line to ethernet
"Silverware", "Tableware", and "Dishes"
My two team members in a remote location don't get along with each other; how can I improve working relations?
Default camera device to show screen instead of physical camera
To "hit home" in German
Chess software to analyze games
What professions does medieval village with a population of 100 need?
How does the government purchase things?
iPhone 8 purchased through AT&T change to T-Mobile
Have ejective consonants ever arisen on their own?
90s(?) book series about two people transported to a parallel medieval world, she joins city watch, he becomes wizard
Sleeping solo in a double sleeping bag
How to get distinct values from an array of arrays in JavaScript using the filter() method?
Writing/buying Seforim rather than Sefer Torah
How to dismiss intrusive questions from a colleague with whom I don't work?
Designing a prison for a telekinetic race
I think my coworker went through my notebook and took my project ideas
Google and bing spider crawling useless URLs
Coming from PrestaShop to Magento CE: How to handle links already in Google/Bing Index?How Can I fix missing title tags and duplicate URLs?magento shop down by bots crawlingHigh CPU load due to Google crawling search termsRedirect for urls with filename and query stringRemove useless zero after numberProduct url give 404 error to crawling even product is enabled and visibility set to catalog search.Magento and Google Customer Reviewswhat is wrong with robots.txt this file?Google complaining about too many duplicate pages after enabling SEO urls
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
After upgrading my site from magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as
https://www.example.com/cms/index/noCookies/
and
https://www.example.com/wishlist/index/add/product/101/form_key/QRh31BjtGTfy2Ur9
Google and bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.
I have added the following command to robots.txt a long time ago.
Disallow: /cms/
Disallow: /wishlist/
But this seems to be useless.
Before I upgraded to magento 1.942, I never found out that google and bing spiders would crawl these URLs in Online Customer tab.
Why is this happening, how to prohibit the crawl of these two URLs?
I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt
robots.txt Tester showed these two URLS has been blocked.
Here's screenshot 
magento-1.9
edited Aug 8 at 4:21
Clark
31 bronze badge
asked Aug 8 at 0:15
ClarkClark
1
add a comment |
After upgrading my site from magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as
https://www.example.com/cms/index/noCookies/
and
https://www.example.com/wishlist/index/add/product/101/form_key/QRh31BjtGTfy2Ur9
Google and bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.
I have added the following command to robots.txt a long time ago.
Disallow: /cms/
Disallow: /wishlist/
But this seems to be useless.
Before I upgraded to magento 1.942, I never found out that google and bing spiders would crawl these URLs in Online Customer tab.
Why is this happening, how to prohibit the crawl of these two URLs?
I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt
robots.txt Tester showed these two URLS has been blocked.
Here's screenshot
magento-1.9
edited Aug 8 at 4:21
Clark
31 bronze badge
asked Aug 8 at 0:15
ClarkClark
1
Did you check your robots.txt using Google's Search Console? support.google.com/webmasters/answer/6062598?hl=en
– Christoph Farnleitner
Aug 8 at 2:58
add a comment |
After upgrading my site from magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as
https://www.example.com/cms/index/noCookies/
and
https://www.example.com/wishlist/index/add/product/101/form_key/QRh31BjtGTfy2Ur9
Google and bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.
I have added the following command to robots.txt a long time ago.
Disallow: /cms/
Disallow: /wishlist/
But this seems to be useless.
Before I upgraded to magento 1.942, I never found out that google and bing spiders would crawl these URLs in Online Customer tab.
Why is this happening, how to prohibit the crawl of these two URLs?
I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt
robots.txt Tester showed these two URLS has been blocked.
Here's screenshot
magento-1.9
edited Aug 8 at 4:21
Clark
31 bronze badge
asked Aug 8 at 0:15
ClarkClark
1
After upgrading my site from magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as
https://www.example.com/cms/index/noCookies/
and
https://www.example.com/wishlist/index/add/product/101/form_key/QRh31BjtGTfy2Ur9
Google and bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.
I have added the following command to robots.txt a long time ago.
Disallow: /cms/
Disallow: /wishlist/
But this seems to be useless.
Before I upgraded to magento 1.942, I never found out that google and bing spiders would crawl these URLs in Online Customer tab.
Why is this happening, how to prohibit the crawl of these two URLs?
I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt
robots.txt Tester showed these two URLS has been blocked.
Here's screenshot
magento-1.9
magento-1.9
edited Aug 8 at 4:21
Clark
31 bronze badge
asked Aug 8 at 0:15
ClarkClark
1
edited Aug 8 at 4:21
Clark
31 bronze badge
asked Aug 8 at 0:15
ClarkClark
1
edited Aug 8 at 4:21
Clark
31 bronze badge
edited Aug 8 at 4:21
Clark
31 bronze badge
edited Aug 8 at 4:21
Clark
31 bronze badge
31 bronze badge
asked Aug 8 at 0:15
ClarkClark
1
asked Aug 8 at 0:15
ClarkClark
1
asked Aug 8 at 0:15
ClarkClark
1
1
Did you check your robots.txt using Google's Search Console? support.google.com/webmasters/answer/6062598?hl=en
– Christoph Farnleitner
Aug 8 at 2:58
add a comment |
Did you check your robots.txt using Google's Search Console? support.google.com/webmasters/answer/6062598?hl=en
– Christoph Farnleitner
Aug 8 at 2:58
Did you check your robots.txt using Google's Search Console? support.google.com/webmasters/answer/6062598?hl=en
– Christoph Farnleitner
Aug 8 at 2:58
Did you check your robots.txt using Google's Search Console? support.google.com/webmasters/answer/6062598?hl=en
– Christoph Farnleitner
Aug 8 at 2:58
add a comment |
1 Answer
1
active
oldest
votes
you can just block these locations from bots with user agent filter:
location ~ ^/(wishlist|customer|catalog/product_compare|tag/product/list|cms/index/noCookies) bingbot
add any location.
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "479"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmagento.stackexchange.com%2fquestions%2f284759%2fgoogle-and-bing-spider-crawling-useless-urls%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
you can just block these locations from bots with user agent filter:
location ~ ^/(wishlist|customer|catalog/product_compare|tag/product/list|cms/index/noCookies) bingbot
add any location.
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
add a comment |
you can just block these locations from bots with user agent filter:
location ~ ^/(wishlist|customer|catalog/product_compare|tag/product/list|cms/index/noCookies) bingbot
add any location.
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
add a comment |
you can just block these locations from bots with user agent filter:
location ~ ^/(wishlist|customer|catalog/product_compare|tag/product/list|cms/index/noCookies) bingbot
add any location.
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
you can just block these locations from bots with user agent filter:
location ~ ^/(wishlist|customer|catalog/product_compare|tag/product/list|cms/index/noCookies) bingbot
add any location.
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
answered Aug 8 at 8:50
MagenXMagenX
2,67010 silver badges27 bronze badges
2,67010 silver badges27 bronze badges
add a comment |
add a comment |
Thanks for contributing an answer to Magento Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmagento.stackexchange.com%2fquestions%2f284759%2fgoogle-and-bing-spider-crawling-useless-urls%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Did you check your robots.txt using Google's Search Console? support.google.com/webmasters/answer/6062598?hl=en
– Christoph Farnleitner
Aug 8 at 2:58