The next big thing: The pulse of the world

If you want to know what the Next Big Thing with the Internet is going to be, then your money would be pretty safe on something called real-time search.

But what is it, and why is everyone getting excited about it?

Well, cast your minds back a few years, to the pre-Google times.

If you wanted to search for something on the Internet, you needed to be patient.

Not overly so - it was still faster than anything we'd come across before - but the Internet was kind of slow.

We assumed that if we wanted to find something online, it would take a while.

A minute, maybe.

We'd enter our search term into a search engine like Yahoo, or AltaVista, or Excite, and expect to get an answer of sorts - all in good time.

Now we go to Google - well, two-thirds of us do - and expect to get an answer - the right answer - in a fifth of a second.

That's pretty fast.

That feels like real time.

But it's not.

Why? Well, that search is culled from Google's servers - a sort of snapshot of the Internet that's been tweaked so your search comes back super fast.

That's an extraordinary achievement in itself. But the results are culled from a collection of data that's already old.

Old, of course, being relative.

Google's index is based on millions of bots - short for robot, and in reality just a slither of code that Google sends off around the web - being sent out to web pages to see what has changed.

If something has, then a bot would report back with the changes, and the Google index would update.

In other words, how good, or our contemporary, our search results are depends on how frequently Google's bots are sent out.

Ten years ago, they might visit a page once a month.

Now, 10 years on, they could be visiting a page every minute or so.

That means your search results are fresher, more up-to-date, than before. A lot more.

So, two key advances: faster results, more up-to-date results.

But what about all the changes that happened in the minutes since the bot last visited a page?

How about all those twitterers twittering news about Michael Jackson?

How about all the gigabytes of information that are added to the Internet every minute?

This is the thin layer atop the 500 billion gigabytes that already exist that people are calling the real time web.

It may represent a tiny proportion of the overall web, but in some ways it's the most important.

At least, it's of the greatest interest.

After all, we are primarily interested in new stuff, right?

We could call it news, but that's maybe defining it too narrowly.

It's the chatter about Michael Jackson that I've talked about before; the hour or so between the small websites reporting his death and the conventional media "confirming" it.

But it's bigger than that.

I guess you could call it "the pulse of the world".

It's part of a bigger shift, of course. One that started with instant messaging - swapping messages in real time - but has since expanded to embrace all social media.

Facebook, for example, is about what people we know are doing.

And we don't expect to hear about it later. Its appeal lies in the fact that it's the here and now.

(Twitter, by comparison, is in what people we may not know are talking about, reading or thinking. Also in real time.)

Put all this together and we can see a huge shift in the way the Internet is used.

A few years ago it was a ramble through stuff that was, largely, of historical interest. The past. Recent, or old. Think of it as browsing in a library.

An interesting library, but a library nonetheless.

Now that is giving way to something more dynamic.

Now we expect our web pages to move, courtesy of web tools Web 2.0 folk call AJAX.

No more refreshing of whole pages when we enter text in forms, or watch stock market charts update.

This is the real time web.

Think of it as having access to everything that is going on - in the world, in people's heads - as it happens.

The web, in short, is no longer just a repository of information but a reflection of our lives.

But a key bit is missing.

For it to be really reflective of our world, we need to be able to trawl through it and find what we're looking for.

And to a certain extent we can.

We can now search twitter to see what people are saying. This was why news of Michael Jackson's death moved so quickly.

As I showed, people were updating their twitter feeds or Facebook feeds with "RIP Michael Jackson" before the hospital had pronounced him dead.

Rightly or wrongly, correctly or incorrectly, the world was hearing about Jackson's demise in real time.

But this, in fact, isn't really real time search.

It's real time filtering.

We were able to filter our twitter messages so that we could follow the Jackson saga.

(As I've mentioned before, twitterers have gotten into the habit of labeling their tweets with hashtags, or labels, which make this filtering easy and effective.)

But search isn't quite the same as filtering.

Google doesn't just filter information, it creates huge indexes in which the data are cross-referenced, ordered and assigned relevance.

So you search for "Thriller video", and it knows you're not just looking for any video that's thrilling, but you're probably looking for Michael Jackson's groundbreaking 1983 mini-movie.

This is called "relevancy". And it's a lot trickier to do than just filtering all messages which have "Thriller" and "Video" in them.

Especially if you want to do it in real time.

So while there are lots of cool real time search engines out there - check out my blog at loosewireblog.com for a list - none have so far really come close to this new Holy Grail.

But they will. Google has made clear it's moving that way.

Twitter will get better at it.

And if you're not convinced, but are game for a try, I suggest you do this: Install the Greasemonkey script in your Firefox browser at http://bit.ly/4ayYQa.

This little tweak means that every time you search Google, the results will be topped by a set of results from twitter too.

Sometimes they're not much use. But sometimes they give you a glimpse of the power of the real time web.

Sometimes they show you what other people are thinking about what you're interested in. Right now.

It's something we haven't quite got our heads around.

And for many of us it could be a step too far.

But I confidently predict we will find it as compelling, as useful, as liberating as the non-real time web has been up until now.

As Wikipedia has given millions their first access to a library of learning, so the real time web will give millions access to events and wisdom as things happen, irrespective of where they are.

Even if 99 percent of it is fatuous and inane, the one percent may end up toppling murderous regimes, teaching those who want to learn, giving opportunities to those who want to earn and sharing where once we hoarded.