Stop Google From Crawling Web Pages
Posted by | Under Remove Pages From Google Friday Nov 21, 2008WHY WOULD YOU WANT TO STOP GOOGLE FROM INDEXING YOUR PAGES?
Well, in some cases, you might want something to be private, like personal information, sensitive details like member numbers, or you’ve started getting spam because you left your details on a forum somewhere. The best way to stop these problems from occurring in the first place or to fix any problems you may have inadvertently created, is to remove your urls from google.
HOW TO FIND OUT WHAT PAGES HAVE BEEN INDEXED BY GOOGLE
You can do a google search of the indexed pages of your website by putting site:yoursite.com into a search box. On occassions I have seen some of the worpdress plugins or themes showing up in search results. Indexing these are of no benefit at all as most lead to a blank page. It’s good to prevent the search engines from indexing our login page too.
HOW TO STOP SEARCH ENGINES FROM CRAWLING CERTAIN PAGES OR AREAS OF YOUR WEBSITE
There are various way to stop google or other search bots from indexing pages on your site and these are listed below…
…and this following video recorded by Matt Cutts from google, talks about how to use the google webmaster tools and shows how each of these methods of url removal really work and describes in detail the effectiveness of each method.
