Suppose you need to evaluate a person. Not just for a job or something like that, but you have to see whether he would make a good friend. One that is reliable and subscribes to the same high values you do: honesty, integrity, friendliness, caring, etc. Now the only problem is that you can’t see nor hear the person. The only clue you have is what other people he visits or what people visit his house. Take a moment and think about this problem.

I think most of you would agree that the only way we could guess whether the person would be a good friend or not is to look at what the people he visits and receives are like, and the quantity of those. If every week a drugs dealer comes by, that would be better than having one to visit him every day. On the other hand, it would still be worse than seeing one of your best friends visit him two times a year. But if your best friend visits him almost every day, that would be far better. Or if all of you friends pay a regular visit, you would think there’s a reasonably good chance the guy/girl would make a nice friend. And if noone would visit this guy, and the guy would visit noone himself, you would be able to tell very little about him.

I think this must be somewhat like the task of a search engine. Part of it is determining whether the page is a reliable candidate to deliver good info for the user. They need to do this without being able to actually read the page. They know what words are on them. I can fill an entire page with the search phrase and this would not be your best friend. Most probably this page would have not many other links to it. It wouldn’t have a lot of friends. Or if it has a lot of links to it, they are most likely to be spam links, the drugs dealers of the not-so-best friend.

If you are that guy who needs to be evaluated, you best avoid visiting drugs dealers or letting them into your house. If you have a website, you best avoid linking to pages with dubious intentions or avoid you become the target of them.

The problem is, a search engine has to see at least some people initially to determine whether they are good people or bad people, and go on from there, inferring about other people who visit them or get a visit by them. What pages could a search engine take as a starting point? What pages are most likely to be informative and not to contain spam? If I were to choose those would be government sites and education sites. Some people think links from these domains get you more points (in Google) than links from other sites. I think that from my perspective this makes sense.

Some people report that some datacenters report updated pageranks. This will stirr emotions and cause nausea for some, happiness for others. Personally, I prefer high pagerank above a lower one. Because pagerank is still some factor in the algorithm. But it doesn’t seem to be as great a factor as it used to be. If you have good anchor text for example you might do better then a page with higer pagerank. So I don’t think one should stare blindly at it: the value of a page in a search is determined by more than incoming links. Get those other factors right and you should do well.

One of the strangest phenomena in the Google world is the sandbox. Say you create a new site and are able to get it indexed in Google. A link to it from the right place is sometimes all you need for a page. So you see in your stats that Google has visited your site and a few days later you are about to check your site in the Google index. You carefully type in http://www.yoursite.com/, hit enter and yup, there’s your puppie! Awesome, in just about the blink of an eye you made it into Google’s index. Naturally, you are also eager to know what place you are in the search engine results. Take as an example that you opitmised the site for a three keyword phrase “word1 word2 word3″. So you type those three words in and let the results display. It appears you are not on the first page. That’s OK, since there is some strong competition going on for this keyword combo. You hit for the next results page. Again, your site isn’t there. In fact, it appears not to be in the first 100 of results. Guess things will be a little tougher than you thought. But you keep on searching and finally hit the last page. Your site isn’t there also. In fact it didn’t come up in the search results at all? Why is that?

The answer is: you’ve been sandboxed by Google. For some undetermined period, at least a couple of months, your site won’t appear in the search engine results for some more competitive keywords. Mind you, it is there. It is indexed and Google knows about your site. But it’s algorithm has decided not to show you for some terms while for others (the least interesting) you do show up.

Am I making this up? Not at all. Matt Cutts aknowledged there is such a thing as the sandbox on a PubCon conference. Brett Tabke, webmaster of webmasterworld asked the question there directly to Matt. His answer was that (and I quote from a thread at webmasterworld)

that there wasn’t a sandbox, but the algorithm might affect some sites, under some circumstances, in a way that a webmaster would perceive as being sandboxed.

I know from experience that it doesn’t have to be a whole site that is affected but can just be one page. I have done some testing where I put up a bunch of articles. A couple of genuine reviews I wrote together with some scraper reviews. I got a decent link pointing at it making sure Google would visit the site often and all pages would be indexed. All of the articles were indexed, but one genuine review got sandboxed. This review was in a really competitive niche and about buying something. I used the keyword “buy” in another review but as that was sort of the only real review of that product to be found on the internet it takes the first place in the results. Clearly it was not the specific keyword that triggered this.

Nobody really can tell whether your site will be sandboxed or not. There is speculation however, like when you point a link to it from a webpage with high pagerank you will avoid the sandbox. I can’t tell whether this is actually true.

And why does it exist? Debate about that too. Clearly, the Google folks are happy with this effect. Most webmasters aren’t and I am one of them. It is a lot less fun working on a website you know the next year or so noone will look at.

I once started testing how well google, MSN and Yahoo would index pages from Phorum, a bulletin board system written in PHP. I happen to have a test phorum installation somewhere on this site that is a showcase for my martha theme I once made for it. Now phorum uses dynamic urls by default though there are some plugins available that can transform these into neat urls. It turned out that google and MSN indexed those neat urls pretty good. Yahoo didn’t seem to like me and had only a handfull of pages indexed. Afterwards I removed the plugin that generated the neaty urls and left my test board alone, with the broken urls and the new (normally default) ugly dynamic urls with parameters in place.

Today I did a search on what pages of this site were indexed. It turns out that MSN and Google made the jump quite easily. All obsolete urls seem to be gone, and in its place are now the new urls you always get warned about. I must add however that google hasn’t indexed all of them properly. Some of them are just indexed with no description at all, just giving the url in its results. I don’t know whether this is due to them being dynamic or caused by my switching. I’ll leave them as is for some more and look again later.

Yahoo still doesn’t like me. It has indexed about the same pages it had when I first looked.

In another post on this blog I talked about a little experiment of mine. Let’s start by recapitulating what was the goal there. Phorum is a discussion board that uses urls with parameters by default. On my own test forum (http://www.dissurion.com/phorum/) I didn’t have much success in getting those urls indexed by the search engines yahoo and google. MSN was no problem at all. So in order to try to get better results in those first two, I tried a plugin that transformed those ugly urls into clean ones.

Now it’s time for the sequel since google seems to have updated datacenters. Let’s start with MSN, since they were the first to have indexed my poor PR0 site to a decent degree. When I ask them to show my pages, they show 32 results. All my blog posts I have published so far are there. Some, but not all, of my phorum posts. When I search msn.com for my magical-test-word (you’ll have to find it in the phorum pages), MSN finds the post, but this post is not listed when I ask to show all the pages of my site. MSN did index the clean urls, though I could find one ugly url together with it’s nice form. This ugly one didn’t show up consistently thoug (most times I repeated the search it didn’t show up).

How’s Yahoo doing? Sigh, my site only comes up on the second page for they keyword “dissurion”. When I click on “more from this site”, displayed when my site shows up in the results, I get … 3 results. Who wouldn’t be disappointed by such search results? What comes up are my main index page, the phorum index page, and my latest blog post. So I decided to search again, but this time with yahoo.fr, which is closest to me (Belgium). Now my page comes up first in the search results (hmm, have to find out more about this). Unfortunately only the same 3 pages are indexed. Well, at least others seem to have the same experience: Yahoo indexing is very slow.

Hello google! Where am I? Seems like I’m in there with 38 pages. All blog posts, all phorum posts, all clean urls. Well, almost: exactly one forum post is missing. When I use the test datacenter at 64.233.179.104 things get more messy. Not all of my posts are indexed, my forum lists are also there with ugly urls (so twice: with nice and ugly urls), and some posts are indexed in a way I can’t figure out how google got it in there that way. If this test center is an indication of future google updates this doesn’t predict future results as good as they are now.

I have to be honest and say I can’t really tell whether google indexed my results when I wrote my first post about this experiment. The forum got into existance probably after an update and before a new update. So to be sure the nice urls really helped, and to what extent, I will have to remove them and see what google will say after the next update. I expect the nice urls to be still there, as google doesn’t seem so fast in deleting obsolete urls. I still have some old urls with ‘dissurion’ in google from the time I still looked at free webhosting, although those sites are completely gone. Since the forum is just sitting there I might as well take advantage of the opportunities it gives me. I have nothing to loose in terms of dropping in the search engines. So I’ll remove those nice urls, and we’ll see what happens next.

So again, to be continued…