Search is not about getting a list of web pages. The structure of online information is much more complex. If you’re looking for some help on a particular Java library, you wouldn’t expect Google to return you a link to every single page of a 50-page online manual: the whole manual is the information unit you’re looking for. If you’re looking for opinions on a new Wii game, you should expect a list of forums, with for each forum, some insight into how many posts refer to the game, what the overall sentiment is, etc. The posts may be on one same web page, or they may not, and this does not matter.

The concept of relevancy is changing. Google’s strengths was to give you the best results quickly. But if you want to know what is being said about a particular brand, product or person - who’s to say which mention is more relevant than the others? Shouldn’t the relevancy of a mention be a subjective mixture of the credibility, reach, and popularity of the mention’s author? It seems impossible to reduce such a mixture into a single number - and wrong to assign such a number to each and every web page.

The Google index proudly boasts of returning you millions of results, but only ever shows you a thousand (quick tests on the News and Blog searches show you that these numbers are misleading). But the key assumption is that only the top results are interesting. Aren’t they all? Don’t most users want to get a complete picture of the information out there, including information from smaller, less ‘relevant’ sources?

Social content evolves, too quickly for a fat search index like Google’s. Google did create separate products for News and Blog searches, and that is revealing about the limitations of their main search index. Besides, Google Blog search is packed with non-blog content, and full of spam.

The future really seems about extracting ‘keyword mentions’ from web pages, as often as possible, then identifying the mentions’ authors and sentiment, aggregating them in various ways, and so on.

You could argue that Google Search answers a different need: it gives a rather static picture of the knowledge accumulated on the web. This may be why, for half of my searches, the first result is from Wikipedia - in which case, do we really need Google in the first place?

After a few of these, I’m beginning to get frustrated and bored. The other issue of course is spam - it’s everywhere. For a typical search, when I look beyond the first 20 results or so, it feels like almost half the rest are spam - auto-generated sites that infect the search indices. Blog search is particularly bad. Some of this spam is so well disguised, again it takes time to sift and filter.

What I do find though are pockets of goodness. Places you’ll find people in the know. They visit them directly or subscribe to their RSS feeds. I’m researching a company called Nobel Biocare and I find this site osseonews.com with the best site tag I’ve ever seen “The World of Implant Dentistry onlnine”. Fantastic. It reminded me of a guy I met years ago who just said when I asked him what his business did “carbonmonoxidekills dot com”. OK, it was a short conversation after that, but he hit the information thirst like a bacon sandwich hits hangover-hunger.

Back to dental implants - my point is that if you want to know about dentistry you go to osseonews.com. If you want to find something out about mortgages, you go to moneysavingexpert.com, not Google although if you check this out you can see that Google is making it easy to search MSE for you. (and try to get you to click somewhere else before you go so they can get some PPC revenues)

What does this mean long term? It means that Page Rank (and the other associated ways Google determines relevance) needs to be improved if search is Google isn’t going to start losing traffic share. How? By understanding more about what the user is trying to find so it can serve up more relevant information. That’s why Google encourages you to log in so it can track what you have been searching for in the past, get to know you better and give you more relevant results in the future. Can this rather subtle, behind-the-scenes approach deliver? Maybe, but it doesn’t seem to be working so far. It may be that a different approach would be better. Could a combination of some social search approaches (recommendations and crowd-ranking for instance) coupled with better data linking based on topics as well as hyperlinks deliver higher quality search results? Is this an opportunity for a Google killer?

When I got to the gym, I found a locker in the men’s changing rooms and unfurled my towelling bundle. Out rolled a little pink rabbit. Oh shit I thought, I must have rolled the bunny up in it by mistake. Slightly odd to have a small pink fluffy rabbit in a men’s locker room, but hey, if I throw it into the locker quickly, no-one will notice and all will be forgotten.

But after about a minute a guy wrapped in nothing but a white towel came up to me

“Was that a small pink toy you found in your locker?” he asked “it’s just that a friend of mine’s kid left a little pink toy in the changing room here, so could I take it?”

My first reaction was a small panic.

“Um, no, actually that is MY friend’s kid’s small pink rabbit” I said. And as I said it I became completely aware of how ridiculous that sounded

I started to blush.

He looked at me with a strange quizzical look on his face. Then another guy who was seriously buff and standing right there turned around and looked at me.

They think I’m trying to steal a small pink fluffy rabbit that I found in the gym, I thought.

Then the words of my friend Damian flew through my head “…thank God you found it, it’s Lola’s favourite toy….”

“oh” said my half-naked inquisitor, “really?” He’s offering me a way out, I thought. I could laugh and say “course not, here you go”, hand him the toy and we could all laugh in a slightly embarrassed way and they would think “he’s a bit odd, and hey, it takes all sorts”

But NOOO, I can’t just hand over Lola’s favourite toy to a towelled up toy thief I’ve never met before, so I try again.

“Yes”, I said, trying to be all casual, but feeling the tension rise up my through my stomach “she left it at my house yesterday, and I meant to leave it in my car, but hahahaha, it got wrapped up in my towel by mistake and wouldn’t you believe it when I unrolled my towel, out popped flopsy” [smile like an idiot]

No smiles coming back - just more “I don’t believe you, you weirdo” looks.

Plan B

“OK, guys, it’s my bunny. I don’t usually bring it out with me but I feel safe when I have Nu-nu with me”

Now they are smiling, but in a ‘I need to get away from you’ kind of way. I lock the locker and go and jump in the pool and try to swim off the embarrassment.

Lola will never realise the trauma that bunny caused me.

Not the Marks and Spencer that is constantly talked about in the financial pages and business sections – but the Marks and Spencer that brought us Chicken Kiev (with real chicken), sandwiches with exotic fillings (like avocado and walnuts!) and stuffing with real fruit (amazing).

Anyone who has watched “The Royle Family” will know what I am talking about.  Marks’ (pronounced “Markses”) is revered alongside the Church, “Who wants to be a Millionaire” and the Queen.

This was typified at Christmas time in our house – my mum would faithfully put her order into Marks and Spencer for the Christmas lunch (typically ordering enough to keep the Russian army well fed through a long Moscow winter), but would always have a list as long as her arm to get when she went to pick up the order.  We didn’t shop weekly at Marks, so mum was always there first thing (which was around 6am).

And this is where I come to my point – the perception of Marks and Spencer in the British psyche is incredibly strong and you have to sometimes feel sorry for the other retailers. Below is the simple equation my mum employed when she turned up at 6am:

Marks and Spencer: Food she wanted was already gone = her fault > “I should have got there earlier”
Tesco: Food she wanted was already gone = completely their fault > “Bloody Tesco’s never got any stock in”

Not particularly fair, but there you go.

(I have to show some basic science or equations in my blog to keep up with my PhD colleagues here at Brandwatch.)

But is the view of a 30-something-year-old man (and his mum) represented in the 21st century with the wealth of information and discussion available now on mainstream and social media?

I put it to the test.

Having recently joined Brandwatch, I need to get to grip with all this “Social Media Analytics” stuff and prove to myself that “Buzz Monitoring” really works (and is easy to navigate through). So, I did some analysis in Brandwatch on Marks and Spencer vs. the other big retailers to see if my view holds true on the Internet – and value can be derived quickly and easily.

Brandwatch uses clever technology to look at documents and understand the sentiment - i.e. are you liked or disliked?  It uses machines trained by people and I won’t even pretend to understand the maths behind it at this stage.

The overall Brandwatch sentiment scores for the big UK retailers for the past 12 months are:  Marks and Spencer +0.3, Waitrose -0.1, Asda and Morrisons -0.2, Tesco and Sainsbury’s -0.3 and Somerfield -0.4.  To put simply – a positive score is good, a negative score is bad.

If the great British public are really represented on Blogs and Forums – then this is where Marks and Spencer performs well. However what strikes me is their relative performance compared to the other retailers – their score is 400% better than the next best (percentages are so evil). Yes, OK … Marks is mainly a fashion retailer whereas the others are grocers – but this line is now blurred because the others devote more and more of their retail space to clothing and non-food.  And my love for Marks and Spencer is their food, not their socks and jumpers (which at my age is the vast majority of my Christmas presents).

I don’t know if I am too far programmed but I am pretty sure that Marks’ food just tastes better than even the posh ranges in Tesco and Sainsbury’s.  I’m not sure Brandwatch can help me with that. I’m sorry – but “Finest” and “Taste the Difference” just don’t measure up. (I’m pretty sure my mum will say Marks’ water tastes better than the others).

Anyone who has had Marks and Spencer custard with cream will understand. Looks like the Internet agrees too.

And I subscribe to several blogs including Jason Yip’s which has some nuggety insights.

I was at Reboot this year where Eelco Rustenburg gave a great talk on Agile, Software, Management, India collaboration in which he spoke about the benefits of a more agile approach. During his talk something crystallised in my mind.

Non-Agile development

There are two big problems with setting long release cycles and doing thorough detailed design and specification documents that require the developer to think through every aspect of a development before starting.

1. We think in terms of days and weeks not months and years

Try to think of something that might take a year to do. Build a house maybe? Or write a book? These are broad tasks and it’s difficult to know whether they will take a year or maybe go on for three years. The act of thinking through what needs to be done is extremely difficult and even when that’s done, you may still end up being 100% over your estimates. We don’t like thinking in terms of years, because evolution hasn’t had a need to teach us those skills. Getting a little anthropological for a moment, for most of history, for us humans job #1 has been to eat enough to get through the day. Short term. Job #2 has been to find a partner and reproduce - weeks, maybe months at worst. Until big projects came along, which in our evolution is recently, there hasn’t been the need to think in years so we’re not very good at it.

2. We like gratification

You sometimes have to wait a long time in non-Agile development to get this fix. But you get it much more with an Agile approach. Outline a feature, plan it quickly, prototype it, check it with colleagues and customers, amend it, ship it. 2 weeks or maybe a month, then satisfaction and the sense of job done. That’s a great feeling. My friend Katja puts it nicely - ‘we all need some sugar from time to time’

Agile give us what we like

So agile development is more in line with our natural event horizons and it gives us the kind of feedback that we crave. What crystallised in my head during Eelco’s talk was that because of these two things, Agile is a primarily a great motivational tool. And motivation is a big factor in productivity.

How hard can that be? It turns out, way harder than it should be. One word aarrrgghh

Here’s what I did

The data is the number of pages on the internet as advertised on the Google homepage

* this one is a guess as they stopped publishing the numbers in sept 05 (8.1bn)

Open Office
First up - I tried to generate a chart using Open Office - I’m a Mac user and I don’t have Excel installed as it’s not great on the Mac. My approach was to save as html and go and find the jpg/png that is generated.

Here is the output for open office

One word: BLURRY. And I saved this as the highest quality I could.

Time: That took me about 30 minutes once I had uploaded it to wordpress and previewed etc. Also, the scale on the x axis did not work properly (the gap between 04 and 07 looks way too smail), but I can fix that in the application.

Verdict: 2/10 (once i fixed the x axis issue)

Excel
Next up was Excel - I used one of the other machines in the office which runs XP (SP2) and Excel 2000. Same data. Same approach. Different result

One other word - DARK

Time: about 25 mins including getting the data onto the XP machine

Verdict: 0/10 Unusable

Google Docs
What about the promise that is Google Spreadsheets I hear you cry (I have good ears)

The approach was a little different. Once I created the chart which is called a ‘Gadget’ in Google docs, I ‘published’ it and they gave me a piece of code to put into my page. Not that difficult, but of course the Wordpress WYSIWYG editor didn’t like it so I’m forced into editing html directly which is OK for me as I’m down on that stuff, but it’s not accessilbe for non-html folks.

There were several issues with this one. The biggest was that there are very few layout controls. Look at the dates - there’s no way to fix that. The title has disappeared and we have the same x axis scale problem as the Open Office chart but unlike OO, with Google Docs there is no way to fix it.

Time: 30 minutes a lot of which was spent cleaning up Wordpress-inserted html

Verdict: 1/10 quite pretty, but not very good and the html issue was a pain
(would have been 2/10, but I just found out that as the graph is stored by Google and linked with the scipt, it doesn’t show up in the RSS feed which sucks)

What about some online app that could generate a flash file? That thought flashed through my head and I let it fly away without trying to stop it and find out more. Bad idea - just a bad idea.

So what do I do?
I get frustrated. I speak to our designer who says she can do it in Illustrator. Illustrator?? This is data!

Not a good use of her time.

Then another colleague suggested taking a screenshot of the Open Office or Excel Version. For some reason that felt all wrong. I had the same feeling I used to get when I taped the top 40 off radio 1 with my radio/cassette player on a Sunday evening 20 years ago.
But, you know, it worked.

The chart is actually a screenshot of the Open Office graph taken with my trusty Mac.

Time: 5 mins plus the original 30

Verdict: 5/10. OK, but not very C21.

Machines and the size problem

The Internet is big and it’s growing. I went to the WayBackMachine to try to find out by how much. The WayBackMachine is fabulous. It saves a copy of all the webpages it has found during its crawl and lets you go and have a look at how things were. It’s a digital archive of the early days of the web and looking back at some of the sites I was involved with 5 or 6 years ago is a cringe-inducing exercise. I’m just glad they aren’t showing up in Google these days!

I used the WayBackMachine to visit the Google homepage over a few dates. Here’s an example of the Google homepage from 2005.

Google stopped publishing the size of their index in Sept 05, so I’m projecting after that, but there is hearsay evidence online for something like 20bn pages right now.

A word of warning

I am sceptical about these numbers. I wrote in a previous post about Google’s bogus results count and I’m pretty sure that these numbers are also inflated, but I think we can conclude that the internet is pretty big.

So what?

The so what is that with this vast vat of data, one would expect that only the powerful processing of a (or many) computers would be able to make sense of it all. But that turns out not to be true. The reasons are that:

1. There is an army of professionals out there trying to screw with the google rankings - some using nasty ‘Black Hat’ tactics
2. As volume grows, even machines have a harder time finding the ‘right’ content

As a result, Google results are getting worse. Google has segmented its dataset into news, blogs, books etc but that has added complexity to the interface and has not solved the problem.

The human touch

Enter Social Search - Wikipedia for example. If you know what you want to find out about, just go to the Wikipedia page and you will find generally high quality information on the subject. No spam, No machine-generated rankings, just people-edited goodness. Mahalo is a spin on this. Professional editors this time and a news feed to pull through more up-to-date info. But under the skin, Mahalo is really another wiki.

And there are more bizarre social search examples - take twitter. It’s not immediately obvious that Twitter is social search, but it can be. My friend Simon Grice told a story at Being Digital in which Loic Lemeur from Seesmic was looking to hire a raccoon (Seesmic’s logo) in San Francisco. Not having had much luck through more traditional channels, Loic posted a request on Twitter and received a number of replies. [ref: Andrew Whitehouse's weblog]

Google itself uses another type of social interaction for it’s paid search or PPC ranking. According to Google, the number one factor in Ad Rank is the historical clickthrough rate (CTR) of the ad and of the matched keyword on Google. So people’s behaviour this time rather than their words, but it’s still social.

Then there are the discovery engines such as stumble upon and digg that are also human-powered. These are gaining popularity as it’s very rare to get something from them that is irrelevant or spammy. Due to the nature of the service, the network is self-managing.

One final example from the Arts. thepaintingfool.com will create art for you or rather is ‘a computer program that aspires to be an artist’ and Koan is software that will write music. But students at universities like Sussex are using this software in a more directional model. That’s to say they aren’t using the machines to generate music or art, they are directing them. They are tools. It’s like having an intelligent amplifier rather than having an amplifier that creates music all by itself.

The big picture take away for me is that a combination of people and technology is very powerful.

The best way I can figure to describe why this is that whilst machines create efficiency people add purpose.

The second paragraph goes on “Social media use the ‘wisdom of crowds’ to connect information in a collaborative manner. Social media can take many different forms, including Internet forums, message boards, weblogs, wikis, podcasts, pictures and video”.

Robert Scoble says “When I say “social media” or “new media” I’m talking about Internet media that has the ability to interact with it in some way”.

That’s the definition I like the most.

Interactivity is the key - like a conversation. Our friends at iCrossing have written a whole ebook on Social Media Others call it, User Generated Content (UGC), Consumer Generated Media (CGM) or even Consumer Generated News (CGN) but a s far as we know it all means the same thing. The one addition we would add is that this information needs to be available on the internet.

So now we understand the term Social Media, Social Media Analysis is just what is say on the virtual tin. But hang on a second – how exactly do you analyse forums, blogs, and things like Facebook? Well the simple answer is it’s not easy.

The best way to think about it is it’s as if you could take all the results for a search on Google, figure out when they were posted, pull out the main topics of conversation add sentiment for each topic and put all that together you would have some sort of answer.

We explain how we analyse social media on our technology page. Brandwatch is a tool that is specifically designed to help you understand social media - not what it is so much as what it means to you.

If by watching a brand we mean the measurement and tracking of its image online, then we need a clear and operational definition of what a brand image is. We propose that a brand’s image is best construed as an aggregate of how that brand is perceived along multiple related topical dimensions.

This way of defining brand image as a multi-dimensional entity and working with individual dimensions explicitly has several important benefits, in both practical and conceptual terms.

Definition of Brand
It is essential to first elucidate the meaning of the concept Brand. Once there is a clear definition of Brand, we can work our way upwards to developing the rest, all the while using our understanding to resolve conceptual and practical ambiguities.

We can start with what it is not: a brand is not a real entity.

Corporations, organisations, companies, services, products, ranges, people, news items are real entities. A brand, on the other hand, is a symbolic concept used to group all the related information under one umbrella.

The definition from Wikipedia makes the above point more clear: ”A brand is a symbolic embodiment of all the information connected to a company, product or service.”

This concise statement clarifies the two defining features of the Brand concept:

1. it is a symbolic entity
2. it groups together all the relevant information connected to another entity.

Definition of Topic
Topic is a generic term we use to refer to the subject of any identifiable conversation/information associated with a brand. Taking the “Barack Obama” brand as an example, its possible topical dimensions are [according to our Brandwatch, at the beginning of April 2008]:

• Jeremiah Wright
• Trinity United Church of Christ
• Contraversial former pastor
• Democratic nomination
• Phil Gramm
• Infinitesimal substantive differences
• White House hopeful
• Economic speech
• Kenyan father
• Kansan mother
• Student scholarship
• State convention
• Law professor
• Passport records
• Former classmates

It is easy to see that Brand and Topic are inevitably defined in terms of each other: a brand consists of its associations with a set of topics, and topics exist with reference to a given set of brands.

Definition of Brand Image
Once the definitions of Brand and Topic are clarified as above, it becomes easy to define Brand image: A brand image is how a brand is perceived along the different related topical dimensions.

An immediate advantage of this definition is that it is readily operational: if the set of topics related to a brand is known, the monitoring of the brand image consists of evaluating how that brand is perceived in the context of these topics.

I supplied Alan with some of the data he used for the article. I used the Superbrands top 100 UK consumer brands as the starting point and plugged them into Brandwatch and grabbed the data for a week.

Once you strip out internet-specific influences such as the web is skewed towards the young and towards men (for various reasons), you can see that FMCGs don’t get much chat, and nor do brands or products that don’t change much - beer brands for example.

As Alan says, I guess this isn’t so much of a surprise and in all liklihood it’s probably no bad thing for them - as lots of chat might well spell a change in attitude towards what are after all very sucessful brands and products.

Marketing for these guys therefore is more about lodging a deep idea in people’s brains that these products are somehow better that their competition - they taste better, feel better, smell better or make you look better. And these are multi-year efforts that only the very biggest organisations can afford.

Brandwatch is not really designed to help these brands. The system needs a reasonable quantity of data to function well - we reckon this is about 30 or so posts a week. When volumes are less than that, good old Google alerts are probably the best way to go - they are free after all.

The one area that might be more interesting for low volume brands is reputation management. That’s to say if I’m the brand manager at Cobra beer, I want to know if there is a big change to the volume or sentiment of chat around my brand online - or my competitors. If it’s not good, I’ll want to know about it as soon as I possibly can. We’re going to be adding this to the Brandwatch Alerts in July.