I’m wondering if I block a page with ‘robots.txt’ and also use ‘nofollow’ or ‘noindex’ in the ‘meta robots’, will that stop all bots from crawling it? Or could one still manage to get through?
If this post breaks any rules, let the mods know! Got more questions? Chat with folks in our community Discord!
I’m a bot, and I’m here to help things run smoothly. If you need a human, contact the moderators of this forum for assistance.
Isn’t this more of a coding question than a marketing one?
You’re right, but it’s also key for digital marketing strategies. Crawling affects SEO big time!
Are you in marketing and have a few minutes? Help us out by taking our 2024 Marketing Survey!
I’m just a bot, but if you’ve got issues with this post, reach out to the mods.
Theoretically, the page shouldn’t be crawled. But, be aware, some shady bots might ignore your settings.
After using both, reputable bots usually can’t crawl the page. I’ll just watch out for the rogue ones.
When you use both ‘robots.txt’ and ‘meta robots’, major search engines like Google should respect both. It’s likely the page won’t get crawled or indexed. But, as others said, some lesser-known bots might ignore the rules and still crawl it.
Your best bet? Use both methods: block crawling with robots.txt and noindex in the meta tags. It’s a solid combo, but keep in mind, pages won’t disappear instantly from search results after adding noindex.
Also, double-check your robots.txt for mistakes. A tiny error could mess up everything, but when done correctly, it should do the job.
To be safe, use both methods. While most reputable bots will follow your instructions, some might not, so cover your bases.
Just remember: Meta robots only take effect after a bot crawls the page. If robots.txt blocks crawling, meta robots can’t step in to stop indexing.