Hard at work in modern computing are algorithms. Like cells in the body, algorithms perform many behind-the-scenes vital functions.

Other than tirelessly executing basic tasks, they can also be used for sophisticated pattern recognition.

It is algorithms that websites employ to work out what sorts of things we might like to buy. Take holidays for example. The algorithms on travel sites are able to tell which sort of places a user has researched, and come up with tailored suggestions on which hotel to stay at, a suitable car to hire, or which restaurants to eat in.

Again it’s algorithms that allow sites such as Amazon to recommend books or music to its customers based on their previous buying or browsing habits.

So far, so good. But unfortunately, the problem is, algorithms are created by humans. Just like humans, algorithms can be fallible. And when they fail, the consequences can be harmful to sites and their users.

One of the biggest website owners to use algorithms is Google. Programmed by Google staff, they determine which websites are seen by users when they search the internet. They control which websites get the all-essential traffic, and which are buried far down the stack of results pages and effectively hidden from the Web.

As you are aware, there are two well-known aspects to Google’s use of algorithms. First, it is renowned for constantly tweaking its algorithms. Google regularly refreshes its search algorithms because it has to keep up to date with an ever expanding Web, with millions of new sites and blogs created each year, the vast majority have dynamic, constantly changing content.

The second well-known thing about Google is how notoriously secretive it is about protecting its algorithms. Google considers its algorithms an important part of its intellectual property (IP), into which many tens of billions of dollars have been invested, and which have helped Google garner around 80% of the world market in internet search.

And let’s be frank, Google’s overwhelming dominance in internet search is what enabled it to make more than $10 billion profit in 2010. We do not quibble with its right to protect its algorithms which are a key part of its intellectual property. And we agree that it has to work hard to stop unscrupulous website owners (of which there are plenty) from gaming the system so their sites are found on the first page of search results while legitimate ones are marooned in Google’s lower ranked pages.

But because of its success and dominance, Google must remember its duty to be open, reactive and transparent with website owners, especially if they have inadvertently incurred a ranking penalty and wish desperately to address the reason why.

Because of Google’s complete lack of transparency over penalties, it is impossible for website owners to know precisely how to rectify problems when Google’s algorithms get it wrong.

This situation became even more marked a few months ago when Google, rather than tweaking its algorithms, introduced a major change in how it ranks many websites. Such was the scope of the change, made under the name of ‘Panda’, Google made the rare move of publicly discussing the reasons for the initiative on its blog.

The intention behind Panda, it said, was ‘... to reduce rankings for low-quality sites—sites which are low-value add for users, copy content from other websites or sites that are just not very useful.’

It is churlish to argue against Panda being well intentioned. However, with its implementation many previous popular sites suddenly disappeared from the important first few pages of Google’s search results. And while some of these were smaller sites, there is no dispute about their quality or ethics. In short, the quality sites, as well as the targeted dross, slipped massively down the search rankings.

A good example is eHow.com. This very popular site provides more than 2 million articles and videos on "how to do things". Post the second roll out of Panda in June, eHow’s traffic was 40% lower.
(See traffic graph at http://www.alexa.com/siteinfo/ehow.com#)
Alexa traffic chart for eHow.com

And what hope does a small site have of growing to be the giant of tomorrow, if it has been doomed to anonymity by Panda? The answer is virtually none.

An official Google blog post prompting for website owner reactions to Panda has generated more than 6,000 replies:

The comments on the forum are telling. Here are just a few examples:

Homeconstructionimprovement.com, an original content site almost 5 years old, said:
‘I've felt a HUGE impact to my site and I have to say the results in the SERPS are disturbing at best. I would VERY much like to speak with someone at Google to find out why....I'm an expert in construction, DIY and home improvement. I'm a respected member of the profession and I travel all over at the request of many large corporations. So when it comes to quality content I'm very certain I offer that. I also receive thousands of comments from folks thanking me for my content and wishing that had found the site sooner.’

Another non-plussed owner said:
‘I am totally baffled as to what has occurred with my site (www.proposalwriter.com) since this change. I have lost at least 2/3 of my traffic and revenue. I have had my website for over 10 years, with the objective of providing relevant, useful, and high-quality information for people who are seeking information on subjects such as government grants, government contracting, proposal development, small business, and international development.’

The owner of charlesandhudson.com had this to say:
‘My site Charles & Hudson (http://www.charlesandhudson.com) has felt the negative impact of this algorithm change. We've been around since 2005 and have always followed the recommendations of Google in regards to doing things the right way and not being selfish with how we build content and share information.

Our Google SERPs are now down at least 35% and we've seen sites that have copied our content or produce URL's strictly for SEO ranked higher. We also receive a healthy amount of traffic from direct and referral sources but almost all of our search traffic is from Google so this algorithm change is of a HUGE concern to us’.

The frustrating thing is that these website owners were not been given any help or advice by Google about what they needed to do.

And it also begged the question, if Panda is a quality filter, what are its criteria of what constitutes a good site? At least if this was made plain, honest sites could make sure they complied with the quality thresholds.

Naturally, it would have been fairer if Google had warned site owners about the real ranking criteria behind Panda and allowed them time to prepare and comply.

Right now, website owners have no idea how improve their site content in the light of Panda. This sort of lack of transparency in web search is precisely why we launched our “Have I Been Penalized..?” campaign last year at http://www.haveibeenpenalized.com.

Currently, if a site has been given a search penalty, some search engines including Google make it virtually impossible for owners to find out what sort of penalty has been imposed on their site, and why.

Like with the lack of clarity over Panda, we want the leading search engines to allow us to know if we have faced a penalty and, if so, what we can do to quickly put things right.

We say if the search engines operated more transparently, it would make the online market place fairer, which is better for all site owners, and ultimately users.

Panda is a prime example of how high quality sites, large and small, were affected, and the net result is that Web users have less chance to visit perfectly good sites. We think that Google hiding behind its "algorithm" excuse is no longer acceptable. Quite simply much more transparency across Web search is needed. Not tomorrow, but now.