You have created your first website, put some pages on it, and are waiting for google to come crawling. Soon you will be indexed pretty good and the visitors will come by the dozens. At least that’s the schedule you had planned. Unfortunately, it doesn’t happen. Your main index page gets a visit now and then, but google leaves your other pages uncrawled. What is happening? Google doesn’t like your new and informative site?

Let’s take a look at a summary of a session of Search Engine Strategies that was held in 2006 in New York. In an answer to one of the questions, Matt Cuts (representing google), seems to have answered something along the following lines:

Your site may not have enough PageRank for us to do a deep crawl.

Now that’s interesting. Many of us already knew about this one, but it is always nice to have this from the “mouth of google” itself. Let’s rephrase that statement: if your site doesn’t have high enough points for google, not all pages will get indexed. With high enough points, I mean PageRank (PR) of course.

That’s hard to swallow for any beginning webmaster. If you are lucky enough to avoid the sandbox, you still face another problem: nearly non-existence. You did a hard job on creating some pages that you think others might find useful, and now google tells you it doesn’t care about them. And how can you make google care? By having enough others link to them. But how are those others going to find your pages and find them useful enough to link to, if they can’t be find through google? Guess the url that might be interesting for their search phrase?

The problem we have here does seem to have some resemblance to the problem of the rich getting richer and the poor getting poorer. It doesn’t actually mean the poor are getting poorer, but that the distance between the rich and the poor is getting bigger and bigger. And that is what seems to be happening for the smaller guy on the web. While the big are getting bigger because they have enough PR and therefore google crawls their pages almost before they are written, the small guy faces the hard task to get off the ground. It’s far from impossible. That’s not what I’m saying. What I am saying is that for a well established site it is a lot easier to get the content indexed and found by people. This leads to an increasing change to get links back, and therefore even getting better points. So the gap between big sites and small sites becomes larger and larger.

To get out of this circle you have to become a good marketer, or write truly great content. Then, eventually, you will be noticed and a similar (though initially much smaller) spiral will happen to your site. But for most of the people out there it won’t happen. That’s the nature of the web google is currently endorsing.

Some might wonder how people know that their pagerank actually changed on some Google datacenters. Or maybe you are just curious to know what yours is.

If you just want to check your current pagerank in a simple way an option is to install the google toolbar. Allow it to give the pagerank and then you’ll see a green bar on it (or a white bar if there is no pagerank for the given page).

If you want to check your pagerank on all datacenters a nice little tool can be found here.

Some people report that some datacenters report updated pageranks. This will stirr emotions and cause nausea for some, happiness for others. Personally, I prefer high pagerank above a lower one. Because pagerank is still some factor in the algorithm. But it doesn’t seem to be as great a factor as it used to be. If you have good anchor text for example you might do better then a page with higer pagerank. So I don’t think one should stare blindly at it: the value of a page in a search is determined by more than incoming links. Get those other factors right and you should do well.

One of the strangest phenomena in the Google world is the sandbox. Say you create a new site and are able to get it indexed in Google. A link to it from the right place is sometimes all you need for a page. So you see in your stats that Google has visited your site and a few days later you are about to check your site in the Google index. You carefully type in http://www.yoursite.com/, hit enter and yup, there’s your puppie! Awesome, in just about the blink of an eye you made it into Google’s index. Naturally, you are also eager to know what place you are in the search engine results. Take as an example that you opitmised the site for a three keyword phrase “word1 word2 word3″. So you type those three words in and let the results display. It appears you are not on the first page. That’s OK, since there is some strong competition going on for this keyword combo. You hit for the next results page. Again, your site isn’t there. In fact, it appears not to be in the first 100 of results. Guess things will be a little tougher than you thought. But you keep on searching and finally hit the last page. Your site isn’t there also. In fact it didn’t come up in the search results at all? Why is that?

The answer is: you’ve been sandboxed by Google. For some undetermined period, at least a couple of months, your site won’t appear in the search engine results for some more competitive keywords. Mind you, it is there. It is indexed and Google knows about your site. But it’s algorithm has decided not to show you for some terms while for others (the least interesting) you do show up.

Am I making this up? Not at all. Matt Cutts aknowledged there is such a thing as the sandbox on a PubCon conference. Brett Tabke, webmaster of webmasterworld asked the question there directly to Matt. His answer was that (and I quote from a thread at webmasterworld)

that there wasn’t a sandbox, but the algorithm might affect some sites, under some circumstances, in a way that a webmaster would perceive as being sandboxed.

I know from experience that it doesn’t have to be a whole site that is affected but can just be one page. I have done some testing where I put up a bunch of articles. A couple of genuine reviews I wrote together with some scraper reviews. I got a decent link pointing at it making sure Google would visit the site often and all pages would be indexed. All of the articles were indexed, but one genuine review got sandboxed. This review was in a really competitive niche and about buying something. I used the keyword “buy” in another review but as that was sort of the only real review of that product to be found on the internet it takes the first place in the results. Clearly it was not the specific keyword that triggered this.

Nobody really can tell whether your site will be sandboxed or not. There is speculation however, like when you point a link to it from a webpage with high pagerank you will avoid the sandbox. I can’t tell whether this is actually true.

And why does it exist? Debate about that too. Clearly, the Google folks are happy with this effect. Most webmasters aren’t and I am one of them. It is a lot less fun working on a website you know the next year or so noone will look at.