“My site isn’t appearing in Google searches, what can I do?”
“What keywords are you using?”
“Everything I can think of…”
The customer had initially asked me to help map her blog sudbdomain (thebusinessname.typepad.com) to a domain name owned by the business. If you are serious about your website and blog, this is one of the first things you want to do. It allows you to build traffic to a domain name you control, not to one controlled by Blogger, Typepad or whatever other platform you’re using.
But, the business owner was now telling me that the site, which had been live for a year, updated on a weekly basis and linked to by several other sites, simply did not appear in Google; and she wanted me to fix it.
I hung up the phone and immediately Googled [“The Business Name”]. Enclosing the keyphrase with quotation marks forces Google to return only results containing the exact phrase. This was an uncommon phrase so Google returned only one page of results.
To my disbelief, all but a couple of the results contained links to the site I was looking for, but none of them was THE site.
I typed the website URL into my address bar to check it was working and found myself at the business homepage. I scanned the page to confirm the business name featured in the text on the page, and it did, a number of times.
Back to Google for another search.
This time I performed a site specific search in the format [keyword site:www.thebusinessname.typepad.com].
Using this format tells Google to search only the URL you specify and return any page it has indexed on that site which contains the specified keyword.Not a web robot
Result: “Your search… did not match any documents”.
So far I had worked out:
* The URL was valid.
* The keywords existed on the homepage.
* Google wasn’t indexing the page at all.
I stared into space for a couple of minutes, trying to work out reasons for this. The business certainly hadn’t been spamming, trading links, duplicating content or doing any of the other things which may make Google remove the site from their index, so why wasn’t it showing up in searches, even very narrow searches like I had performed?
I went back to the homepage, right-clicked on the body of the page and brought up the page source. There was the answer, the “Robots” tag. Because the blog had never been marked as public in Typepad, webcrawlers (programs used by search engines crawl the web, also known as robots or webbots) were being told to ignore the site with the “noindex, nofollow” command.
What is the “Robots” tag used for? If you have files or directories in your site which you DON’T want indexed by search engines, you can tell the webcrawlers about this using the “noindex, nofollow” commands.
But, if you’re happy for all files on your site to be indexed, you can do without a “Robots” tag entirely. Your site won’t be missed by the robots because the default for all of the major search engines’ webcrawlers is to index the entire site unless told otherwise.
However, if you still want to use the tag the correct format is –
<meta name=”robots” content=”index, follow” />
If in doubt, leave it out…
