After many months of operation, TagWalk has collected over 100 million tweets, and tracked over 40 million URLs. There are increasing numbers of Foursquare URLs popping up, and due to good RESTful design from the Foursquare team, I noticed that when users tweet their checkins, it is possible to pattern match their venue URLs.

In true mashup fasion, after a quick nosey through the Foursquare API documention, I was quickly able to fetch additional meta data about venues and present them nicely in the links section. In the same way image thumbnails are shown TwitPic and the other twitter photo services.

I ♥ Recommendation Engines

I am a massive fan of recommendation engines. One of my goals for building TagWalk was to create a recommendation engine that would suggest users with the most reputation for particular subjects or #hashtags, and show popular links being tweeted by users. My inspiration is probably the oldest and most successful example I can think of, Amazon’s “Customer who bought this also bought…”

Location-based Recommendations

Treating Foursquare URLs as symbolic locations instead of web-pages unlocks lots of new meaning burried in Twitter data. It makes location based analytics and recommendations possible. By crunching some historic data it was possible to create a very crude location recommendation engine, or rather, “people who went here, also went there”

Some locations of interest and their related “people who went here, also went there” recommendations are: San Francisco Airport (SFO) ✈, Twitter HQ, Foursquare HQ, Facebook HQ, Tech Crunch HQ , Y Combinator, The White House, and the San Francisco Apple Store.

Sample Size

Foursquare recently trumpeted their 100,000,000th checkin. This dataset covers less than 0.2% of this; just over 150,000 checkins and approximately 80,000 locations. This is a large enough sample size to see some interesting relationships start to emerge. As the population of Twitter users that TagWalk follows has a statistical bias to San Francisco and other tech areas, it figures that these tech-savvy areas have better coverage by my data set than others areas.

What next?

Whilst the data set is not (and can never be) complete, a large enough sample size can yeild “good enough” results. By selectively loading more data, the data set can be adapted to improve the quality of results. Over time, TagWalk will continue to collect small volumes of checkins, but specific search filtering could by used to prioritise certain regions or businesses.

With further digging around in the datamine, it is possible to produce location-based recommendations, or rather, “people who went here, also went there” and for more considerable computation effort, “people who went here, also have the following interests, reputation, talk with these users, and visit these web sites.”

If you run a business, particularly with many locations and would like to work with me to get to know your customers better, then please get in touch.

You can follow me here: @timhastings

In the example below, there is a very shy but talented Elvis impersonator.

By examining the tweets, we can see three cases where “ronnie” is mentioned with Elvis, yet he himself never mentions Elvis at all.

Here’s another illustration of the difference, who is failing and who is a troll?

In the title of this post, I’ve related interest and reputation to ego and trust. Going off what an individual says only takes one voice into account, their ego. Many services use follower count as a messure of reputation, but in my opinion, most importantly, conversation matters! Listening to what many people say gives a more trustworthy answer.

TagWalk associates what people are saying in their tweets (interest) and who they are saying it to (passing reputation along). This means that TagWalk can make recommendations on a certain hashtag or word. In doing so, it also derives relationships between words, hashtags and URLs. Allow you to browse through the relationships discovering new people and content.

Here are some example hashtags to explore: #followfriday, #socialmedia, #wine, #nodejs, and #beer.

You can also explore an individuals reputation by checking out their own pages. Here are the pages of some prominent Twitter users: @BillGates, @Scobleizer, @guykawasaki, @pistachio and @arrington.

There is wisdom in them thar crowds!

User Generated Directories

There are directories like wefollow.com where users add themselves by choosing which categories they belong. This is effective, but I see there are some flaws with this approach for use as a reputation system:

• The user’s motives for listing themselves in the directory is self-promotion.
• It is open to bogus claims. For example: I could tag myself as an #seo expert when I may know very little on the subject.
• Relevance and ranking. What determines which user is higher in a category than another user? Connectedness and number of followers can be used, but my single-malt whiskey followers also boost my rank in the PHP developers category.
• Staleness. The oldest users may well have the most followers and influence. How can a new-comer hope to gain rank?

Twitter Lists

Fairly new on the scene is Twitter Lists. This feature of Twitter allows users to build arbitrary public and private lists and add users to them. This could be viewed as tagging by another name. This is a great usability enhancement to the Twitter service, and they make it much easier to partition friends into subject groups. I imagine a valuable side-effect (and end-game) for Twitter is a new metric: how many lists does a user belong to? Or alternatively “how many people think this user is worth adding to a list?” Emphasis on “worth”, as in, how many users value this user?

It is early days for Twitter Lists. They are clearly a usability win! Whether they get used to build a reputation system is yet to be seen, but I think they are definitely helpful in when screening for news feed bots or spammy accounts who are unlikely to appear on any one’s lists.

My concern around any reputation system built around Twitter Lists is that it costs very little effort to add a user to a list, merely the motivation to do it. Also list membership is black and white; a user cannot be “very in a list” or “a little bit in a list” – a user either belongs to a list or does not. Michael Gray posted about How to Use Twitter Lists To Create Reputation Management Problems which illustrated how a simple act of adding a user to a list can taint a user’s reputation.

Update 14-Jan-2010: MustExist’s List Tag is a working reputation systems which derives from reputation from list membership for a given user name. Given a user’s screen name, it processes their list memberships and produces a tag cloud. For example: this is what is says about timhastings. This is good example of list-based reputation, if I was to suggest an improvement, it would be to allow the discovery of users within any given #hashtag. Thanks to @eric_andersen for the tip.

Wisdom of Crowds

In my opinion, a good reputation system should be derived from user activity and the relationships a user has with other users. I want a system which observes Twitter activity and then auto classifies users based on evidence. Each time somebody talks to me and uses a particular tag, it should increase my score for that tag. The system should be able to differentiate somebody who just talks a lot (self promotion) from somebody who is mentioned a lot (reputable). The number of different talkers using a tag, defines the size of that community.

Reputation emerges from monitoring Twitter activity and aggregating statistics.

Many Wisdom of Crowds systems are well known and very successful. The two best examples I can think of are Amazon’s Recommendation Engine and the Delicious Bookmarking system.

Amazon aggregates sales information, and computes things shopper A would also like to purchase knowing what users with similar purchase histories have also bought. Delicious is far simpler, it relies on users ‘tagging’ bookmarks; if lots of users tag a bookmark as ‘webdesign’, then it probably has a lot to do with ‘webdesign’.

The Delicious model is the closest fit for a Twitter Reputation system. With Delicious, the aggregate value is gained from individuals selfish desire to organise their bookmarks. In our Reputation system, the aggregate value is derived from individuals selfish desire to communicate, it is given a turbo boost when they use #hashtags, hyperlinks and the user names of their friends.

Demonstrating this Approach

This blog is about a project of mine to build a great Reputation and Recommendation system for Twitter. TagWalk analyzes tweets and keeps track of who said what to who. It maintains a data mine of relationships. Each relationship gets strengthened each time it sees a tweet. This is then available to explorer and discover online.

For example, if we take a well known Twitter phenomenon, #followfriday and use this as a basis for computing reputation, we can see some leaders emerge, as well as related hash-tags, and words used.

Our metrics around the volume of tweets and number of talkers helps to define the size of the audience (or niche). It also shows how a new leader can quickly emerge.

I am also fairly certain that a new breed of SEO will evolve which will specialise in Social Media Reputation Optimization (SMRO).

In Conclusion

The number of friends, followers and how many lists you belong to are not enough to build a great reputation system. To be truly great, we must pay attention to the content and flow of the messages between users.

Wisdom of Crowds – it’s the only way to be sure.

Once the decision was made, it was very quick and painless to secure a new dedicated host. The hardest part was moving the TagWalk data mine from ‘the cloud’ to a real server.

It has been an interesting exercise, and an excellent opportunity to validate many aspects of the design, identify weaknesses, and learn some interesting and painful lessons — every day’s a school day right?

Non-technical:

• Migration is an excellent rehearsal for disaster recovery.
• Restoring a read-only mirror of the site was a good start goal.
• Create a journal file with all your command snippets.

Technical:

• Ubuntu Server 9.10 rocks.
• DNS can be very useful. Subdomains are great.
• Moving large volumes of data from one data center to another sounds daunting, but that was the easiest of jobs. The transfer rate was about 1.4MB/s and the generally speaking, the data could arrive faster than it could be dealt with.
• Sharding was a good idea which pays huge dividends (never in doubt)
• Waiting 20+ hours for a single table to restore really makes you question alternative methods of storage.
• Mono 2.4.3.4 has some issues which required VB.Net stuff to require “Option Strict”

MySQL Specific:

• MySQL imports can be *REALLY* slow. Under certain circumstances restoring large tables that have a UNIQUE INDEX can be very slow. I had heard anecdotal evidence but never really paid attention to it. Due to some rookie mistakes made along the way, it took me several attempts to restore my largest table. This problem alone set me back at least 2 days.
• Giving MySQL as much memory as possible can considerable improve import times. Optimize MySQL for the task in hand.
• MySQL’s innodb_file_per_table setting is the only way to go.
• The Linux nohup command is the only way to run long MySQL restore operations.
• Not using MySQL for the bulk of the data mine meant that for 70% of the data it was a zip-and-ship migration strategy. +1 for NoSQL!
• Representing a MD5 digest as 4 unsigned integers is much more efficient than a 32 character string.
• Restoring the data is easy, it is the indexes and constraint which take the time. A read-only version of your site can be up very quickly, the constraints are only necessary when you switch to read/write.
• MySQL table switcheroos with the RENAME TABLE command allows your site to use temporary copies of your tables which do not have all the indexes or use the right storage engine. The most effective large-table restore strategy I arrived at involved restoring MySQL tables into a MyISAM table without expensive indexes. Getting the site to use these for read only. Meanwhile, populate expensive versions of these tables with the full set of indexes (InnoDB + Unique indexes).
• Some big MySQL operations cannot be canceled. If an script load or INSERT/SELECT dies then MySQL want to “close tables” of “clean up”, this can take forever. In many cases, this required server reboots, and MySQL remove/installs to overcome.

The new server has more than double the memory which will give memcache and MySQL a much needed RAM massive boost. I also plumped for RAID as moving away from EC2′s Elastic Block Storage means a loss of redundancy on disk; a level of risk I am uncomfortable with, given the volumes of data

TagWalk has already settled into it’s new environment very quickly. The performance benefits are immediately apparent. As I begin to pay more attention I will be able to optimise this performance further. More importantly, the additional machine capacity will enable me to develop the next set of features.

Profiles are refreshed if a user posts or is mentioned in the tweets proceeds by the tweet processors and the profile held in the database is older than 7 days.

The other user information pulled from the Twitter API is useful for me to determine how many of the user’s tweets have been processed by TagWalk versus how many they have tweets altogether. At a later date, I can backfill a user’s tweets from the API, although rumour has it that the Twitter API only supports access to the last 3200-tweets of a user.

So far, the tweet processors have fetched 382,000 user profiles. Which is nice.

In other news, I’ve also added a retweety button which allows a user to easily post a tweet for the page they are looking at. This is done by linking to the Twitter site and populating the message box.

I have more to be getting on with, but if you have any suggestions for TagWalk, please let me know.

This evening I have mostly been doing front end work on TagWalk, exposing the behind-the-scenes computed numbers. These numbers very useful, but previous attempts to display them had looked ugly. I took some inspiration from the way that Twitter displays their follower count information, and I am very pleased with the result.

TagWalk breaks tweets apart into things, and computes the relationship between the things that appear together in messages. Depending on what thing you are looking at, the statistics can be interpreted differently.

Like most statistics, they are open to interpretation, so I would like to explain what is shown, and most importantly, the meaning.

Tweets: this is the sample size. This shows the number of tweets that have been processed and flagged with this term. Very importantly, TagWalk goes to a lot of trouble to ensure that any tweet is only counted once, so this figure does not include any duplicates. This is controlled by the Twitter allocated message ID (the one which is about to overflow in the Twitpocalypse). Duplicated user content in not filtered, so if a user does repost the same message, this will be counted. This is important to identify spammy users.

Retweets: What percentage of these tweets were a retweet? For TagWalk, the definition of a retweet is whether the message begins with RT, or Retweet, or contains a “via username” pattern towards the end of the tweet.

With Links: The percentage of the tweets which contained a link to a valid website. This is useful to identify whether a user or hashtag is particularly spammy. Especially when used in conjunction with the Web Sites metric.

Hashtags: How many other hashtags have been used in conjunction with this thing?

Talkers: This identifies the number of people talking about this thing. It is derived from the distinct users, so it is the population size of talkers.

To Users: This is different from By Users, because it analyses the users who appear in the message body. It is useful to have this as a separate measure because it helps to determine the target or focus of the conversation or thing. A small number is a narrowly focused laser beam, versus a widely scattered meme.

Web Sites: When links are included in tweets, TagWalk tries to determine the destination web site. This number shows the distinct host names of links (after redirects through URL shortening services). To try to get a more accurate figure, www is dropped from the host name.

For any stato boffin, sample size is very important. TagWalk statistics are based on a sample of the Twitter-sphere, and are not real time. The messages are fetched using “search seeds” which are used to hit the Twitter Search API, at the moment the stream APIs are not being used. If you would like to suggest seed terms or would like some comprehensive research, please get in touch.

I hope you find this statistics enhancement useful, any feedback or suggestions you may have would be very welcome!

Last Saturday I attended the BarCamp Leeds and gave a talk about Data Mining Twitter to talk about the theory behind TagWalk and then gave a demonstration. After a shakey start with projector difficulties, the session got back on track and there was a good number of questions and recommendations.

In a nutshell, TagWalk is a Recommendation Engine which uses Twitter messages as its input. In the same way that Amazon suggests books you might like by examining the purchase history of other shoppers, TagWalk is able to recommend related hashtags, users and URLs.

Messages are decomposed into ‘tags’. These are not just explicit #hashtags, but also ordinary words, URLs, @usernames. For any given tag, TagWalk will show commonly used other tags, words, users and links.

Motivation

The motivation to build TagWalk came from three four main factors:

• A desire to retain links from messages with the hashtags – users of Delicious explicitly tag for themselves, whereas Twitter users are sharing links. These links must be worth keeping.
• A love of recommendation engines – my day job doesn’t require building one, and I fancied building one as a side project.
• A love of Twitter
• An API must be called – an unexplored API is like a big red button – it has to be pushed.

All in all, there has been about four months of late nights gone into TagWalk as a side project and there is still lots more to do.

Computing Reputation and Authority

Twitter directory sites such as WeFollow rely on users to tag themselves with hashtags. This is flawed. Just because I tag myself as #studly does not make it true. Likewise, I could post lots of messages containing #boffin as a hashtag, but this relates to the content of my message, and say little about me. What is very relevant however, is when other users use a username in a message alongside a hashtag — because these hashtags relate to that user.

For this reason, TagWalk differentiates the user who tweeted from a user mentioned in the tweet. When aggregated we are able to see hashtag experts emerging

URL Shortening

Whenever a URL is tweeted, it is usually shunk using a URL shortening service such as tinyurl. As many users may use a different service to shorten their links, TagWalk spiders the links to resolve the final URL. This also allows the hostname of the site to be included as an analysis tag.

Architecture

Unlike tradition LAMP stack applications, TagWalk is implemented using LAAMMMP. Which is Linux, Apache, Amazon EC2, MySQL, Mono, Memcache, and PHP. The algorithms and data structures behind the scenes are inspired by map/reduce, and will enable distribution across many servers (if it can generate income to pay for them)

Challenges Ahead

• Increasing the data processing capacity. At present seeds are used to focus attention, instead a shift towards the real time hose-feeds would give a fairer slant.
• Bootstrapping and monetization. TagWalk runs on Amazon EC2 infrastructure which has costs money to run. Whilst it is a labour of love, generating income would enable more computing horsepower and therefore more features and bigger data sets.
• Boring and stale links. As the site nears its second month of unattended operations, the link sorting algorithms have already needed tweaking to ensure a balance of popular and fresh links. There will be little incentive to return to the site if the same URLs are always at the top.
• Spam. There are a lot of spam tweets out there, but fortunately I have the stats and algorithms to find them

Credits

• Thanks to @MikeNolan for the photograph (via flickr)
• Thanks to @foodiesarah for lending me her laptop and rescuing me from projector problems
• Thanks to @perki for the excellent TagWalk logo
• Thanks to everyone at BarCamp Leeds and Geekup for being a friendly audience

The fopen function creates an internal data structure to keep track of the file and the current read position. On a 32-bit platform, the default PHP build uses a signed integer as the file cursor so this gives a maximum addressable file size of 2.1GB. If you try to open a file greater than this limit, you’ll get this error:

function.fopen: failed to open stream: File too large

Apparently, there is a build option for PHP which can be used to compile PHP with a 64-bit integer in the file handling data structure, but I want to use standard “found anywhere” PHP rather than start tinkering. Using standard PHP makes it easier for me to move to new servers, either on Amazon EC2 or elsewhere.

Plus, this is a trigger point for me to implement some of the sharding functionality I had planned that would enable the scaling of TagWalk across muliple servers and the parallel processing of separate shards.

Tonight’s tasks include:

• Writing and running the data splitter to create the shards. This will be in the Mono backend tools.
• Creating the PHP shard switching classes. This will require the shard lookup data required to resolve each tag to the appropriate shard. Ultimately this will live in memcache.

If I can get all this done tonight – I’ll be very pleased.

TagWalk has been a side project of mine which I started around February 2009. On Monday night (20th April), I demonstrated progress-so-far to a small and friendly crowd at Geekup Preston. It was well received and I got lots of feedback and ideas.

It prompted lots of discussion around hashtags, which is a main focus of TagWalk. In particular, the ambiguity of hashtags and how different people will use different hashtags to classify the same thing which can make them a difficult mechanism for retrieval unless you have pre-arranged what tag should be used. This happens alot at conferences etc.

This blog’s purpose is to capture the progress of TagWalk and to provide some background information into what’s going on here.