LinkedIn’s short URL fail

Here’s a tip for anyone developing a URL shortener – blacklist ‘naughty’ words.

I’m talking to YOU LinkedIn. Check out the short URL you generated for an event I created recently.

Now you’d have to say, as far as words that might be filtered as inappropriate, it’s a pretty obvious one. If you missed that, you probably don’t have any kind of exception mechanism at all. You might want to fix that.

By the way, I didn’t notice this until after I had pasted the short URL into an email and widely distributed it. A mate of mine, Vaughan, messaged me saying the link was busted - Erin (his girlfriend) was redirected to a ‘scary bunny’ website. No doubt I was being blocked, warned about and denounced as a pornographer and spammer by institutional firewalls all over town. 

The Voice of the Customer in Agile

The ‘Voice of the Customer’ is the weakest link in Agile web development.

New advances – which make Choice Modelling cheap, fast and relevant – solve this problem.

The ‘Voice of the Customer’ is a poorly taught and poorly understood component of Agile methodologies. There is no consensus on the tools that should be used to capture it.

Of the methods that are in use, the most rigorous are complex, costly and time-consuming, and can’t be easily and quickly repeated at each iteration. The less rigorous methods are cheap and quick, but the information they produce is actually worthless (we have proven this, as I demonstrate below).

So, measuring the ‘Voice of the Customer’ in Agile web development is generally done either:

-          not at all; or

-          poorly; or

-          fairly rigorously at the outset, then poorly or not-at-all throughout.

But there are now Choice Modelling methods available that allow us to quickly, cheaply and accurately measure the Voice of the Customer, both initially and throughout development.

Let me illustrate by example.

On a recent web project we simply wanted to know which features our customers wanted most. So we conducted a basic online survey to build a choice model. In order to prove this was a superior approach, we conducted a parallel online survey using the common ‘rating’ approach (where respondents rate each feature on a scale of 1 to 5).

Both approaches involved exactly the same cost (cheap), time (2 days) and sample (40 people). The results are summarised below.

The Rating results weren’t statistically meaningful. The features all scored about the same average, and the highest scoring feature was actually the number 6 ranked feature in the Choice Model. These results were either useless, or - if you ignored statistical significance and took notice of the highest average rating – wrong.  

Because the choice survey was so quick and cheap, we’ll be able to repeat it between each sprint, in order to see if (and how) customers’ preferences change as they became exposed to more fully developed iterations of some of the features.

Having seen the benefits of this approach, I think you’d have rocks in your head to use anything else.  The adoption of these new Choice Modelling tools can only be a Good Thing for product innovation online.

I am happy to point anyone who wants to know more in the right direction.

Social Media: It’s Social AND it’s 'The Media'

Social Media is a behaviour not a technology. Individuals are simultaneously consumers and publishers of social information (location, mood, likes, dislikes, events, photos, videos, reviews, opinions, news). Furthermore, your social network is your personal editor-in-chief and indexer of the internet.

Social Media was born on the day in 2006 that Facebook launched their News Feed - the perfect merger of Social Networking and Online Media.

Think of Facebook as the world’s largest portal business, with an unpaid editorial staff of 500 million (and counting). Facebook + Twitter adds 200 million (or so) unpaid reporters in every corner of the globe.

 Brands wanting to engage using Social Media should not be asking ‘How do I make friends?’ They should be asking ‘What do I have to offer that’s newsworthy?’

By 2005 individuals had the ability to create and distribute media at no (or low) cost, with the same presentation quality and reach as governments and media corporations. There was probably no precedent for this phenomenon (certainly not since the days of the early 20^th century revolutionary pamphleteer – just before radio arrived).

There were millions of blogs that looked at least as good as one of Rupert Murdoch’s newspaper sites, and were accessible instantly, everywhere, on really cool devices. There were millions of YouTube videos with higher production values and merit than an episode of Big Brother or Funniest Home Videos.

But the problem was one of discovery – it cost time and effort to seek out the content that was truly interesting to you. Google couldn’t help you because its algorithms don’t index based on characteristics of subjectivity or sentiment (interesting, cool, funny, relevant etc.).

The big media companies had mastheads – brands you recognised and trusted. They had editorial staff to aggregate and package content nicely for you. So you the first website you visited in the morning, or whenever you had some time to kill, was probably a news website or media portal.

Then, in 2006, Social Networking and the new online media landscape merged to become Social Media.  Facebook launched its News Feed feature, and the average amount of time spent on the site quickly tripled. Increasingly, the first website you visited whenever you had time to kill was Facebook. And the online media you consumed was increasingly via links shared on Facebook.

What Facebook had done was enabled ‘social search’ (or what I’ve called elsewhere ‘collaborative indexing’).  Your social network seeks out and shares news, videos, blog posts etc. that are interesting and relevant to them, therefore (in many cases), interesting and relevant to you – because these people are your friends, and you share the same interests.  Of course, you return the favour. The larger and more diverse your social network, the more interesting, frequent, eclectic, and broadly topical your news feed becomes. The odd quirky item may come as a pleasant surprise. Even if it strikes a bum note, you don’t blame the mechanism. When a search engine throws up a useless link, you think ‘this search engine is broken’, but when social search turns up a dud you think ‘I have some weird (or boring) friends’. And if you wish, you can call them out by posting a comment.

The News Feed becomes compelling and super-convenient. In one place you can catch up on what’s happening in the Middle East, why your team’s coach is an idiot, a great music video, and who your friend Kylie is dating this week – all probably of equal interest to you. In the same place you can fulfil all your needs as a social being. Extroverts are busy posting chirpy comments and passing on links; introverts and curmudgeons enjoying the fact they can lurk, passively consume, and achieve technical compliance with their social obligations by pressing the ‘Like’ button a few times and sending brief birthday greetings when prompted.   

Social Media (Facebook and its adjuncts) consumes a lot of spare time for a lot of people. And it’s not just eating into ‘other media consumption’ – the time people used to spend watching TV, reading magazines, listening to radio, using internet portals. It’s also replacing ‘social interaction’ – the time people used to spend talking on the phone, chatting over the back fence, hanging around the water cooler. For daily users of Social Media it is ‘The Media’ – the place they get all their information about the world around them, and form their world view. Unlike ‘The Media’ in the traditional sense, there is no central, controlled editorial point-of-view. The ‘spin’ is provided by individuals’ social networks. This is why governments around the world are scared of the internet and want filters, kill-switches, or simply to turn it off.

Is there a place in here for brand marketing? Yes, of course. You may have come across the acronym SMO (Social Media Optimisation). Once you understand that ‘social search’ relates to your ‘brand’ content (which is about subjectivity, sentiment and brand engagement), in the same way that Google search relates to your ‘product’ content (which is about features, price, offers, and making a sale), you’ll realise you need to pay a lot of attention to SMO.

The challenge for a brand is to be more than just present and active on Social Media. If a brand wants to become an influential member of an individual’s social network, it has to be useful. It has to contribute something newsworthy.

Brands need to push content which is relevant, personal, timely, bite-sized, convenient. It can be mundane – 90% of Social Media is completely mundane – along as it’s about the customer.  For example: “your pay is in your account”, “you have reached your savings goal”, “you have earned 100 loyalty points” “you sent 3,000 text messages last month”. ‘About the customer’ is always better than ‘about the brand’. I like it when my cable TV provider tweets about handy shortcuts on the remote. I don’t like it so much when they re-tweet gormless generic praise like “company X has awesome service”.

An aside - in my opinion, one of the best ways for a fearless brand to use Facebook or Twitter is for publicly visible customer service enquiries and resolution.

Anyway, the lesson for brand marketers is this: when you’re thinking about Social Media, don’t overlook the ‘Media’ aspect. Be useful. Be newsworthy. I’m certainly not saying to ignore the ‘Social’ aspect. Yes, develop a persona, develop a tone of voice, and strive for authenticity. But remember in Social Media, as in life, a good friend is not just there to hang out; he’s there to help out. 

It's Banking Jim, but not As We Know It

I’m sceptical about the predicted rise of Social Banking, peer-to-peer banking or ‘Facebank’.

However I think that disintermediation in financial services markets, driven by a combination of social and algorithmic enablers, will be a powerful and disruptive reality in the next few years.  

‘Disintermediation’ was a buzzword of the late 1990s, the fall of the traditional brokerage to low-cost online stock trading was touted as its first scalp, but it seemed to run out of steam and drop off the hype radar.

In other words, Disintermediation was one of those trends that appeared on the cover of Wired Magazine one month, hadn’t transformed the world by the following month, so was quickly overlooked. But while the hype curve subsided, the reality was (and is) slowly, inexorably building.

Disintermediation has at its core the open, free access to individual market participants of: market information (data); and the tools to perform aggregation, search, filtering and analysis on that data. In the pre-digital world, this access was expensive in terms of effort and money, and restricted to intermediaries or agents. It should not be so in the digital world. Although there are regulatory barriers protecting agents in many markets, I believe these will fall as individuals realise they can bypass costly intermediaries, and governments accept that potential gains in productivity far outweigh the interests of the intermediaries.

On the extinction list are the usual suspects - recruiters, real estate agents, etc. But also, more tantalisingly, I think the process of disintermediation may completely transform the global banking system as we know it.

A new financial paradigm will require BOTH ‘social’ and algorithmic governance. The social and altruistic elements of peer-to-peer online banking and microfinance will combine with: tools to perform real-time analysis of risk/return across large, live datasets; and willingness to make personal finances semi-public in order to significantly reduce financing costs.

We’ll see completely new businesses, and new disintermediated relationships between depositor/borrower, insured/insurer in services like student loans, business receivables finance, consumer credit, who knows what else... 

What's eating online publishers?

Published in NETT magazine May 2010

Internet publishers bemoan the ‘commoditisation’ of the banner ad. What really happened is hyperinflation. And it’s the publishers’ own fault. But there is some good news, especially for small players.

Major online publishers complain about the recent ‘commoditisation’ of the banner ad.

They paint ‘commoditisation’ as a pestilence striking the crops of simple, honest farmers of online eyeballs.

As I see it, there are two things wrong with this view. One, the obvious one, is that publishers brought this phenomenon upon themselves. The second is the misuse of the word ‘commoditisation’.

Now, there’s nothing intrinsically wrong with a commodity. Just ask anyone in Western Australia. They love the commodities they’re sitting on, and can’t dig them up and ship them off to China fast enough.

What has blighted the price of ad impressions is better described as hyperinflation.

Hyperinflation is what happens to a currency when a government simply prints more banknotes, without a corresponding growth in the real product or productivity of the economy. Pretty soon you need to take a wheelbarrow full of hundred dollar bills down to the corner store for the loaf of bread that used to cost pocket change. Internet publishers found that internet penetration became saturated and the growth in their audience started to plateau. Addicted to years of double and triple digit growth, publishers started to manufacture more ad impressions, while the underlying commodity that advertisers remain happy to pay good money for – viewing time multiplied by size of audience – remained flat or grew slowly.

How did the publishers manufacture more impressions? A whole grab bag of tricks – photo galleries, multiple-page articles, aggressive auto-refresh, anything that increased the amount of deliberate or inadvertent page requests in a typical visit.

The advertisers are not stupid, nor the publishers immune from the simple laws of economics. The price of impressions (CPM) plummeted.

Large publishers have been heard to argue that the price plunge shows that what advertisers now want, and what will save the publishers from their current predicament, is a more accountable pricing model (Cost Per Action) and better use of data-driven targeting.

They’re effectively saying they need to become Google - targeted online advertising that offers tactical marketers proven results. Google works. Businesses know what they’re getting, they’re happy to pay for it. There’s no question whether the commodity Google sells is value for money. The price is set by the market in real time, via auction.

If these publishers aspire to even approach Google in their ability to use data and achieve goal-based performance... they’re dreaming.

The good news for online publishers is that they don’t actually need to become Google, or compete with Google, in order to survive.  The proof of this lies in the one ray of sunshine for online publishers in the current market – strong demand for video inventory. Video inventory is not somehow more demographically targeted, or more conducive to clicks and conversions than banners. It largely serves the same purpose – eyeballs, recognition, brand awareness. Strong demand for video inventory shows continuing faith in online advertising whose main function is strategic and awareness-focused, rather than tactical and performance-focused.  

The challenge for publishers is what it always has been. Recognise your audience and create content which appeals to that audience. This is particularly heartening for small publishers, who tend to be close enough to their audience to know them extremely well.  And small publishers can build really successful businesses. Sound Alliance is the outstanding example. They started a dance music website ten years ago. Last year they enjoyed double-digit growth, $15 million in revenue, and the longest average session times of any online publisher measured by Nielsen.

Clicks, shmicks – their formula is only about compelling and relevant content. It’s not rocket science, or applied mathematics – leave that to Google.

Social Media Rules: Rule 2 - Listen

First published on the Hyro blog Jan 14 2010.

The previous post in this series [Rule 1 – Don’t’ Believe The Hype], made the case that Online Social Media represents ‘effect’ more than ‘cause’. In social media opinions are aired, shared and confirmed, but not originated. Social media is the water cooler conversation of the digital age, and even though brands and celebrities are allowed to overhear, and even join the conversation, the reputations of these brands and celebrities come largely pre-formed.

Social media does not represent the uniquely powerful new means of manipulating opinion promised by some*.

However, Social Media is a great place to simply listen.

There are many ways to ‘listen’. You can measure aggregate sentiment, gauge the success or failure of targeted marketing and communications activities,  gather feedback from individual customers on products and features, find out what your competitors are doing right and wrong.

The good news: the data is rich, high volume, real-time and 100% free.

 The even better news: the opinions of social media users seem to be a very accurate measure of the opinions of the general population.

Previously, I poked fun at those drawing a very long [and very wrong] bow based on research data. But in doing so, I found that the same research data showed the responses of active social media users did not materially vary from the responses of the general population, including infrequent users. In other words – what active social media users think and say is very close to what everyone is thinking and saying.

Brands can use social media as a real-time, unprompted focus group, tracking actual, intimate, and detailed conversations about their products, price and service.
Brands can measure sentiment, and derive Net Promoter Scores, across large populations, and track how these measures change over time.

Brands can collect immediate feedback on a product launch or marketing activity, and react rapidly.

These possibilities, and the appetite for analytics tools they will create, have not gone un-noticed by the global technology giants. I have seen a few sneak previews of Social Media Monitoring software to be released in 2010 – including Microsoft’s LookingGlass.

Access to these tools will result in an increasing sophistication and subtlety in the way that marketers address social media.

In the next post, I’ll consider whether the best way respond to social media is by using social media, and discuss the pros and cons of active participation.

*Nor is any new medium likely to deliver on this promise. Ever again. The 20th century, one-to-many model of media is in decline, and will continue to decline as long as large numbers of individuals can easily produce content of good-enough presentation quality and access distribution networks like the internet.

Social Media Rules. Rule 1: don’t believe the hype

From Hyro blog, October 1 2009. A caution on confusing cause with effect.

Much of the social media hype emanating from agencies and consultants is based on an astonishing confusion between cause and effect.

To use the analogy of social media as the ‘water cooler conversation’ of the digital age - People don’t formulate the opinion that Bank A has great service, Politician B can be trusted, or Kyle Sandilands is a goose, purely through the mechanism of a debate around the water cooler. These opinions are formed elsewhere, and brought to the water cooler.
The following assertion, published by a leading agency, epitomizes the confusion between cause and effect. 

“It’s important to note that we found no variations in the responses among the people who identified  themselves as active users of social networks and those who use social media less frequently. In other words, as you study the survey responses, note that social influencers and social media have an impact on the general consumer population – not just a small elite of social media enthusiasts.” Shiv Singh Vice President & Global Social Media Lead, Razorfish Social Media Labs. Fluent: The Razorfish Social Influence Marketing Report, 13 July 2009, p 9

Using the fact that research shows no variations in response between those who are active social media users, and those who aren’t, to conclude that social media has a uniform effect on those who use it and (somehow) on those who don’t (by some kind of spooky osmosis?) is wonky logic and wonky science.

Possible explanations for the research observations include -

1. The opinions of those who actively use social media are influenced by their use of social media, and these opinions in turn (by a mechanism unknown) uniformly influence those who don’t actively use social media

2. There is an influencing mechanism, outside of social media, to which both groups are uniformly exposed, and by which both groups are uniformly affected

In the absence of a proven hypothesis explaining the mechanism by which the opinions of social media users influence non-users, the first explanation should be rejected. (Why? Read Wikipedia entries on The Scientific Method and Occam’s Razor).

There we go again - cause and effect.

Social media use is not the mechanism that causes users’ responses. The cause originates in the media in general. What is observed amongst social media users is the effect. The real good news from Razorfish’s research is that the opinions of social media users seem to be a very accurate measure of the opinions of the general population. But more on this in the next post: “Rule 2 – listen”

Before I sign off, one more hype-puncturing factoid.

Dell is one of the poster children for Social Media strategy, especially when it comes to Twitter. Dell have invested properly in their Twitter strategy, and do a very good job at it. In June Dell announced that Twitter had contributed to $3 million in sales revenue over a 2 year period. Dell’s turnover for the same period was $122.2 Billion – so that’s 0.00002% of sales.

- - -

A comment:

Peter Darke Says:
February 3rd, 2010 at 12:51 am
3 Million is 0.00245% of 122.2 Billion.
Just to be pedantic.

Mea Culpa, Peter

Customer Experience vs. User Experience

This is a post from the Hyro blog, from August 14 2009 - another one with a useful comment wasn't forwarded to me at the time. I respond belatedly.

I’ve heard a lot of answers to the question “What is the difference between Customer Experience and User Experience?”, all of them long-winded.

Some of them have been very good answers – but needlessly complex, since there is, in fact, a very simple answer.

The difference between Customer Experience and User Experience is the difference between a Customer and a User, and the difference between a Customer and a User is that a Customer has a choice.

Your digital business is competing moment-by-moment for the customer’s attention with other digital businesses, other channels (TV, Radio, iPod, billboards), and now (on mobile internet) with the good-looking girl/boy sitting near them on the bus. Before the potential customer even thinks about doing anything an usability expert can measure, they are making a split second, emotional decision to give their attention to you.

I particularly like sharing this insight with senior marketers. It instantly transforms them from slightly-intimidated-by-Digital to smartest-guy-in-the-room. Because if there’s one thing marketers know about, it’s Customers, and the Customer Decision Journey. By contrast, we in Digital Services were calling them ‘users’ a year ago - and still routinely refer to them as ‘visitors’ or ‘browsers’. Quaint, really.

“Oh, we have some visitors!
Should we make them a cup of tea?
No don’t worry, they just want a look around, they’ll be off soon.”

Anyway, the point is that the Customer Experience point-of-view allows us to access and apply years of excellent learning from the offline world and marketing science. It allows us to ask how a customer feels about completing an online task, rather than just worrying about how many clicks they have to make. More importantly, it allows us to stop feeling guilty about using emotive words like ‘cool’ or ‘kick-arse’ when talking about the visual and tactile interface.
 
- - -
1 Comment:

Bora Says:
December 11th, 2009 at 11:56 pm
Hello Michael,
I am not sure I understand and agree (from what I understand) with your differentiation of user/visitor vs. customer.
For me a customer is simply a visitor that is either going to buy right now or at the minimum has the potential to buy in the future.
Sounds simple but it seems that many companies and digital agencies forget the word “buy” very often. Instead of creating a website that makes everybody happy, it is in my mind more important to concentrate on the buyer and give them the necessary information that is necessary (providing for their stages from information seeker to actual buying decision).
All this depends on who ever your buyers are, consumers, corporates, investors or even the press.
I do support that a customer makes a split second decision if they find you interesting and trusting enough. But real customers who actually look to spend money to solve a pain/problem are usually easier to convince to stay compared to the browsers who are jumping from site to site.
In my experience the conversion part is more important, because here it will make the difference if one can even collect the low hanging fruits - the customers with their wallet open ready to buy a solution to their problem.
That is also where usability plays a big role - how easy is it for the customer to buy or convert to a lead. One of the studies from Stompernet show that 70% of customers ready to buy from a site never finish their purchase. Imagine that in a supermarket - people leaving in droves because they found it to hard to buy.
I would be interested in your response Michael.
Cheers,
Bora

- - -

Hi Bora

Thanks for waiting patiently for 18 months for a response.

In short, I think we are both right.

I'm arguing here against the past tendency of the usability discipline to be doctinaire and humourless - much like Bauhaus-inspired architects of the 1950's decrying bourgeious ornamentation, and shoe-horning American office workers into buildings designed for 1920's European public housing.

But you are right. The converse - the sole emphasis on 'cool' - is just as unsatisfactory.

There has to be a balance in user interface design - between desirable/emotional considerations, and usable/rational considerations. Ideally, this balance should be weighted according to what stage of the Customer Decision Journey we can infer a user has reached.

As for the store where people are leaving in droves because they find it hard to buy - that store exists in the real world, in high streets and malls across the world. It looks good from the outside, and you want what they sell, but the music's too loud, and the shop assistant is too busy texting her friends to even look at you.

I agree, you don't want your website to be that store.

Erase the Lines – with VIM! (Vertically Integrated Marketing)

Another one from the Hyro blog. Posted July 15 2009

Marketing blogs everywhere call for us to tear down the traditional divisions between Above-The-Line, Below-The-Line, and Direct marketing. The future is integrated, they say. The future is digital.

These calls are admirable; and not to be discouraged. Digital channels can be the fulcrum of accountability, interactivity and personalisation in a multi-channel, multi-device marketing campaign. But if you stop there, you’re missing a large and important part of the Integrated Marketing picture.

Let’s call the integration of campaign messaging across traditional and digital channels Horizontally Integrated Marketing. Digital channels also deliver the promise of Vertically Integrated Marketing. I’m talking about the seamless transition between delivering a message and delivering a service. Digital channels let you build customer experiences around the entire life cycle of the customer experience – through the stages of awareness, preference formation, purchase, consumption, confirmation, and back through the loop.

Which leads us to two questions:

Why confine your interactive marketing to activities around building awareness?

And

Is the notion of the ‘campaign’ relevant anymore?

Now, the answer to the first question may be – “we have to focus solely on awareness because there are real constraints to fulfillment via digital channels”- for example if you’re an alcohol brand. But where there is any opportunity for clients to configure, purchase or consume online, there is no excuse for confining interactive marketing to building brand awareness.

 The answer to the second question - is the ‘campaign’ relevant anymore? No. Not if you are using Vertically Integrated Marketing to build experiences around the core of the customer relationship cycle – configuration, purchase, consumption, service, confirmation, retention. Your marketing activities should be ‘always-on’ and constantly optimised.

Chris Maloney of the Commonwealth Bank knows what this is all about. He is ADMA’s Young Direct Marketer of the Year, and recently took a victory lap in the form of an extremely popular presentation – so popular it was repeated three times to cope with audience demand. His presentation didn’t use exactly the same language as I have, but the principles were the same – and he executed these principles to increase the number of home loan applications referred from online from 1% of applications to 13% of applications in less than 12 months. This is an extremely impressive headline, also the main reason people flocked to Chris’ presentation - to find out how he did it. Chris pulled together a roster of agencies to implement his plans - although there are agencies combining all the necessary elements under one roof, like (ahem) Hyro (excuse the shameless self-promotion)*. 

Chris described the traditional notion of the marketing campaign thus: “It’s like the hokey pokey. You put your campaign in, you pull your campaign out, you put your campaign in, and you shake it all about.” He’s obviously not a big fan of the idea either.

*Hyro was not one of agencies used by the Commonwealth Bank, so maybe I’m not such a shameless self-promoter after all!

---

I got this comment from Chris Maloney, but it was never forwarded to me. Chris, I apologise for not acknowledging it.

Chris Maloney Says:
September 3rd, 2009 at 12:37 am
Totally agree with your thoughts on integration Michael.

We need to move away from the world of ‘matching luggage’ advertising integration and move to thought integration that carries right from the marketing through to the customer experience.

Glad to here you liked my presentation, thanks for the plug! Keep an eye out over the next few months, I’ve got a few digital integration game changers in the works*…

*Shameless self promotion

Chris

The Goldilocks Formula and the Web of Possibilities

First published on the Hyro blog May 21 2009. The rambling epic braindump I referred to in my last post

Search and Market Research will converge. They will become indistinguishable from one another. We will arrive at a point where many products, services and content are created on a just-in-time basis. Exactly at the moment you realise you need it, the perfect music compilation, job offer, electronic device, documentary video, or dinner date will appear right in front of your face.

Maybe even in 3D. Make sure you have the special glasses ready, just in case.
Let me explain.

Market research is all about finding out what people want, and getting into their heads, so you can become better at building products they want - or at persuading them to want the products you build (AKA advertising). Search is also all about finding out what people want, and getting into their heads, so you can find and deliver what they want.  Wait a second, you say – is that last statement really true?
In order to justify my argument, it looks like I’ll need to digress into the recent past and near future of search.

Here goes…

The Next Big Thing in Search

In the mid 1990’s, the internet comes along. It’s like the biggest library ever assembled, the store of all human knowledge, profanity, trivia and vanity. 1 million monkeys with a million typewriters, Shakespeare, the lot. But it is too large for a single human brain or single human lifetime to comprehend or conquer. It is functionally infinite. It is labyrinthine. It is unruly chaos – with no order, classification or government. It is nearly impossible to locate the Shakespeare within the nonsense of the million monkeys. The only way to handle it is to find a little corner and build a pleasant walled garden in which to while away your days… or… Google.

Google comes along. Well, Google and others, but let’s simplify history. Google comes along and solves the problem. Hurrah. The garden walls are trampled, needles are found in haystacks everywhere, the internet thrives; Google lists on the stock exchange and reaches a market cap of around $200 billion.

Fast forward to today – internet speeds are much, much faster, and there’s a lot more and different types of content. It’s not just words and pictures and animations. The internet has punched the music industry inside out and is just getting started on the film and television industry. People are accessing the internet on their phones and TVs as well as their computers.

Today - the analogy for the internet is no longer ‘the world’s largest library’. Today - we’re faced with the world’s largest record shop, the department store where the aisles stretch off into infinity, the cable TV network with 99 million channels (and still nothing on).

So we’re back to square one. And Google, as we know it, can’t help us.

The simple reason is that the songs, videos and products we want to find aren’t composed of text or consistently described by the text they contain (even considering metadata). We can’t rely on the object of desire (be it a song, a movie, or a bargain) to effectively describe itself in words and the user can’t effectively articulate his or her desire in words “I wanna watch a movie that’s really, umm… good”.

So how do we solve this problem?

There are a number of promising approaches, and I’ll go through a few of them in a moment. But at the crux of them all are these two principles:
1.       content is best described and classified by the way humans use it and behave around it; and
2.       the desires of humans can be predicted by their past behaviour and current state 

By the way, in case you haven’t worked it out yet. ‘Search’, ‘Recommendations’ – they’re the same thing. I make no distinction between the two here.

Anyhow, let’s look at a few interesting approaches to the problem.

Collaborative Filtering

Collaborative Filtering is the science behind many of the most successful recommendations engines. The best known users include Amazon, eBay, iTunes and TiVo.

From Wikipedia:
“Collaborative filtering systems usually take two steps:
1.       Look for users who share the same rating patterns with the active user (the user whom the prediction is for).
2.       Use the ratings from those like-minded users found in step 1 to calculate a prediction for the active user.

Alternatively, item-based collaborative filtering popularized by Amazon.com (users who bought x also bought y) and first proposed in the context of rating-based collaborative filtering by Vucetic and Obradovic in 2000[citation needed], proceeds in an item-centric manner:
1.       Build an item-item matrix determining relationships between pairs of items
2.       Using the matrix, and the data on the current user, infer his taste”

Amazon and eBay use the second method, which relies on what people bought.

iTunes and TiVo use the first method, which relies on what people liked.

This points to something really interesting which scientists have observed about consumer preference.  We used to talk about ‘confirmation bias’. People took a bit of time to decide whether to buy something, and once they got it home and used it, they were more likely than not to convince themselves they’d made the right decision.

Confirmation bias simply does not apply when we’re talking about music and video. The mere fact that someone bought a piece of music or rented a movie, or invested time listening/watching makes no difference to whether they’ll like it or not. That’s why user ratings are so important to music and video sites.

I predict we’re heading into a world where confirmation bias is less and less prevalent in all sorts of purchases and markets. Therefore, it will be even more crucial to deliver what people want rather than convince people they want what you deliver. I’m going out on a limb here, there’s no real data to back this up yet. But if I’m right, it’s another nail in the coffin of advertising-as-we-know-it. Yes, I’ll happily add my voice to that chorus of doomsayers. Google does not advertise. But Google has a brand and people trust the Google brand. Google got there by giving people really useful stuff.
Anyway, let’s crack on…

Choice Modelling

In terms of practical applications Choice Modelling is still very much under the radar.
The inventor of Choice Modelling, Dan McFadden, won the Nobel Prize for economics in 2000.

Choice Modelling is highly accurate, much more so than current Collaborative Filtering methods, but operates within certain limitations. It is excellent for predicting individual and segment preference for things you can describe as a series of attributes. For example you can describe and compare cars by make, colour, number of doors, fuel economy, and price. You could similarly describe and compare mobile phones, holidays, job offers, shares in a company, web pages.

You couldn’t easily break down music or video into comparable attributes. Nor does Choice Modelling recognise ratings – it’s about a binary decision to purchase/consume or not. Choice Modelling is used to mine existing sets of data and identify optimum product configurations. It is also used to conduct experiments which accurately predict demand for hypothetical new products and configurations. Choice Modelling the science is behind the Accenture XoS software, which automatically varies and optimises webpage content and configuration in real time.

I can imagine an application whereby consumers invest 10 minutes of their time completing a choice experiment (survey) on certain type of product – say mobile phones. They learn about features, and accept or reject a number of hypothetical alternatives. The application then builds a choice model for the individual, and trawls the internet to find the best deal for them, based on their unique preference structure. (Anyone has who has a bit of money to play with – contact me, let’s build this thing, I know how.)

Alternatively we could coerce Warren Buffett into completing a choice experiment on stock selection, and build a Warren Buffett ‘bot to buy shares for us and make us rich. Then again, maybe not.

I reckon Choice Modelling will come strongly into play once we move from a universe in which consumers are faced with a functionally infinite range of existing things to choose; to a universe where consumers are faced with a literally infinite range of ‘possibilities’ - options that do not exist yet but could be quickly assembled. More on that later.

Collaborative Indexing

Collaborative Indexing is a term I made up to describe how humans actually operate to solve the problem of functionally infinite choice.

It’s what ‘the kids’ do when they talk about and classify music and subcultures. I have no idea what ‘New School Speed Garage’ means; nor ‘Electroclash’. I’m not exactly sure what an ‘Emo’ is or does.  But everyone who needs to know seems to know. A new musical genre or youth subculture is invented, tagged and propagated in the flicker of an eyelid. No-one is in charge, there’s no formal process, and no authoritative lexicon exists. Yet it works – flawlessly, and instantly. Every teenager across the globe gets the memo. And they all conspire to keep it from their parents (who rely on the tabloid press for alarming misinformation).

The person who can figure out what’s happening here and put it in algorithm will be fabulously rich. He or she won’t even need to put it into an algorithm. The kids are doing a perfectly good job as is. Developing a means to facilitate and accelerate this process online would be powerful enough. 

When you’re online, you can easily shift persona without changing your wardrobe or getting a new haircut. You could use multiple different ‘personalities’ to tap into the Collaborative Indexing power of whatever subculture or interest group would be most useful for a certain search or type of content. 

Paying Smart People a Million Dollars (to work it out for you)

This is how Netflix is going about solving the problem. It’s an approach I particularly like, and would recommend to anyone sitting on a million dollars and a set of data potentially worth a million times that.

Netflix have offered a cash prize of one million US dollars to whoever can come up with a recommendation algorithm that represents a significant improvement in prediction success compared to their current one. Groups and individuals that wish to participate are given a huge set of real data to play around with. This data is more of a motivation than the prize for some participants - it’s the kind of real data that would cost much more to collect than any academic research budget would allow. Participants are encouraged to collaborate and share details of their approach. The winner of the grand prize, as well as winners of progress prizes, are required by Netflix to publish a description of their algorithm, which becomes public domain science, accessible to all.

In effect, Netflix will pay a million dollars for a piece of intellectual property that they will not own, and that anyone, including a competitor, is free to use.  Google’s algorithm is a more closely guarded secret than the Da Vinci Code. Netflix will let anyone who wants to use theirs.

But this is not an act of philanthropy, or of idiocy. It highlights an essential feature of the next big thing in search: the data is more important than the algorithm (or at least as important).

In the world of Google-as-we-know-it, the dataset is the zillions of words on the internet that anyone can look at, and the algorithm is the super-clever, highly-valuable thing that makes sense of the data. In the world of the next-big-thing, the dataset consists of highly valuable bits of information on what humans buy, like and do, and where they are. The bigger the data set the better, and these data sets are precious, closely guarded property. Without access to the data, the new algorithms, no matter how good, are useless.

So, am I saying that Google is dead? (OMG!) No, I am not saying that. Larry Page and Sergey Brin had probably thought this far ahead by the time they were 12. Google has more data, and more relevant data, than anyone else. Do not sell your Google shares. Buy more.

And, by the way, the datasets that Netflix gives away to prize participants are real, but masked. They’re great for testing theories, but you can’t do anything commercially useful with them.

As an aside – if you are in a business that looks anything like Netflix (i.e. if you deliver music or video, or books, or anything similar) I strongly advise you to collect data in the same format as Netflix – a rating scale of 1 to 5. For the blindingly obvious reason that you, too, will be able to use the winning algorithm.

Back to the Point

Well, that was a considerable digression, but I think I’ve explained what I mean when I say that search is (or will be) all about working out what people want, and getting into their heads, so you can find and deliver what they want. In that regard, search is (or will be) similar to market research, which is all about working out what people want, and getting into their heads, so you can become better at building products they want - or at persuading them to want the products you build (AKA advertising).

We’ve already killed off advertising. In the new world, we get this powerful information about what people want instantaneously, and can act on it instantaneously. We’re not sitting on a warehouse full of red widgets. In fact, all known things are just a click away. So, why would we use this information to try to convince someone what they really want is a red widget? If what they want is out there, we get it for them.

And if what they want is not out there, even given the hundreds, or thousands, or millions of options?

Well, if we could make it for them, we would, right? Given this powerful insight into a consumer’s desires, surely we’d take advantage of it…
…as long as we could make some money out of it. So we’d need a matching algorithm to work out if we could make a margin, and how to optimise that margin.

The products and services we offer would then exist as myriad permutations of configuration and price. Some permutations would be profitable for us, some would not. Where the desires of a consumer or a segment of consumers matches a profitable permutation, we make that permutation, and deliver it. Depending on what business we’re in, this loop could be completed almost instantaneously.

In any case we wouldn’t make something unless we were sure there was demand for it.
So -  the Next Big Thing in web search is algorithms fed with rich behavioural data. And these algorithms then enable the Big Thing After That – which is the Web Of Possibilities
But this is all a bit abstract. Let me give you a few examples of how it might pan out.

The Big Thing After The Next Big Thing

Your flight has levelled out and the seatbelt light is off. You pull out your laptop. YouTV knows what you like to watch, knows that you’re travelling for pleasure, and that the flight time means you’ll have 25 minutes to watch video. During the taxi ride to the airport, YouTV has downloaded three alternative viewing packages based on: your preferences; this morning’s zeitgeist; analysis of YouTube sessions; et cetera. All of the packages are automatically created mini-documentary ‘riffs’ – related videos strung together. Your options are: heavyweight title-fight mismatches; risqué European TV commercials; and great jazz performances. You buy the jazz. YouTV rates this as a normally low probability choice, but it knows that you are a single male, and that another of its customers, a female around your age, will be sitting in the adjacent seat. YouTV algorithms accommodate situational shifts in persona. You make a good impression.

The last time you bought a phone on the internet was only a year ago. From your purchasing history, and preferences inferred from the way you drilled down to find more information about certain options presented, and ignored others, YouPhone knows you are most interested in stylish form, low weight, and music features. Their R&D department has just come up with some new case designs, and the means to produce more gigabytes of memory at a lower size, weight and cost. You are pushed an offer to upgrade to a stylish, feature-packed new handset, which has never been made. You buy it. The factory in China starts making it. You have it within a week.

You’re cycling in the park, listening to music on your new phone. YouTunes knows who you are, what you like, what your friends are listening to, where you are, and infers by your velocity that you’re on a bike. YouTunes programs and delivers a playlist perfect for you at that moment. Some of the songs you’ll have purchased licences for already, some will be new songs that YouTunes knows you’ll probably buy, some will be songs you’ll be predicted to like but not buy, one might be a song that your new girlfriend (the one you met on the plane) is listening to at the same time, and has decided to share with you.

You’re looking for a new job, and have decided to invest 15 minutes doing a simple survey on a job site, so that it can create a model of your preferences. YouCorp is looking for someone with your skills, and has posted a vacancy to the same site. The job vacancy is in the form of a series of trade-offs that YouCorp is willing to make - in wages, fringe benefits and working hours. The job site crunches the data makes an automated offer on behalf of YouCorp. The job site knows that you attach a value equivalent to $10,000 salary to a company dental plan; you have a similar strength of preference for a location close to the city, and couldn’t care less about a company car. You’re offered a job in YouCorp’s city office, with a dental plan. The money is seems good (the savings YouCorp will make by not providing a car have topped up the salary). You take the job.

This is the Web of Possibilities. Enabled by detailed insight into the behaviour and preferences of groups and individuals, the internet connects humans with a humungous matrix of complex and abstract things, and an even more humungous super-matrix of complex and abstract possibilities - things that might exist, and will exist as soon as they are desired.

The Web of Possibilities can’t encompass everything, or every market. Certain things, like human beings to meet, or holiday locations to visit, will always be limited to a finite set of existing options.

Or will they? Better hang on to those 3D glasses after all.

As an epilogue, my prediction of the death of advertising is probably exaggerated. Advertising in some form will always be around. Not only because we’ll always need information about what’s available, and what’s new, but because we actually need to be sold the implicit benefits. Reassurance that the blandishments of consumer society are in fact deeply fulfilling is the opiate of the post-industrial masses. Every now and then the buzz wears off and we crave new and better ways to revive it.

Fast and Upside Down

First published on the Hyro blog 29 Jan 2009. Here's me purchasing a petard with a piece on concise and single-minded communication. Yes, one of my very next posts was a rambling epic braindump.

It strikes me that the ‘reverse pyramid’ principle of web copywriting is an excellent metaphor for the world we now live and work in. The ‘reverse pyramid’ turns the orthodox narrative structure upside down. The punchline comes first. Draw the conclusion, then build the case. 

The reason behind this principle: people are less able to read large amounts of text on a screen versus a book or magazine. You need to make your point immediately, and hope that it’s compelling enough to keep some of your audience reading.

But I think this principle goes beyond web copywriting – it’s an insight to something pervasive in the digital world, and into the way the digital generation behaves.
It’s often said that our attention span is shrinking, that young people can absorb less information, and are less capable of understanding complex concepts. The implication; people are dumber than they used to be, especially ‘young people’.
But is this what’s really happening? From where I stand, the generation which has grown up on the internet is getting smarter, and their ability to quickly grasp complex things is increasing.

‘Quickly’ is the key word here – the digital world is fast, and getting faster. Its inhabitants realize that, in the course of a day, they will be exposed to much more information than they can possibly absorb. Their minds have been trained to rapidly process snippets of information and decide what to ignore, and what to persist with.

So, our attention span may appear to be shrinking, but it’s not because we’re losing the ability to concentrate, it’s because we’re learning the ability to rapidly and consciously decide which information is useful, and which information is not – and we’re doing so almost every waking moment of our lives. The kids are alright – their brains haven’t been dulled by years of internet and video games, they’ve been sharpened.

Think about this the next time you stand up to address a room full of people with mobile phones and laptops. The first thing you say will have to both encapsulate your purpose, and convince your audience that it’s worthwhile to keep listening. Same goes for just about every interaction in your professional life – the interactions you conduct personally, and those you engineer via digital channels on behalf of your brand or business.

If you have any questions, please feel free to call me. But make it fast. And upside down.
Next week – I expose the dangers of Binge Thinking
- - -
BTW, there was no post on 'Binge Thinking'. I was having a subtle dig at a pointless gabfest of a management conference. Binge Drinking was a tabloid preoccupation at the time, hence the pun.

Here's one of the comments on the original post. I reproduce it because the link's a good one, and on topic.

Tim Says:
January 29th, 2009 at 8:53 pm
Nicholas Carr’s article from last year “Is Google Making Us Stupid?” deals with this and related topics in detail.
Among other things, Nick claims that there is an implicit trade-off between breadth and depth of knowledge taking place - we are skimming more, but reading in detail very little.http://www.theatlantic.com/doc/200807/google
-Tim

A Brand New Blog

This is a new home for all my writing on my professional interests. Which are, in short, technology and human behaviour.

First up, I intend to re-post a selection of my blog-posts and articles from various sources, then post new material as and when the inspiration strikes.

I will continue to submit to corporate or industry blogs and magazines from time to time, and certainly value the larger audience and informed conversation they offer. However by running this baby myself, I won't have to worry about other parties archiving poorly, failing to forward comments, or retrospectively changing my bylines to generic attributions (you know who you are!).

So, from this day forward, all my professional writings will start or end up here. This is more for my own convenience than for the edification of an admiring audience - although that would be nice ;)