For Culture Hack Scotland this year I worked on an idea I came up with that week. I feel I didn’t really do the hack day properly; I ended up working on my own and not using any of the data. But the event was the perfect opportunity to test the idea.
Flocklight pulls all the twitter data for those attending or mentioning an event like a conference, and tries to highlight useful data about the connections between people. The aim is to provide you with information so you make the most of an event. For instance, are others you know going? Do you have something in common with a stranger there who you can go and chat to?
(The aim is not to produce some kind of score and crown someone the “winner”. I’m sure a gamification expert could do that and lots of people would love that – and indeed I was told about a recent conference with a score board for those that tweeted the most – but I personally think that’s meaningless, and I’m much more interested in exposing information to help you have a better time in real life.)
By 4pm on Sat (the deadline) we were tracking 148 users, 2272 follows between those users, 9354 tweets and 1417 instances of people mentioning another user in those tweets.
For each user, we had a page that exposed who they followed there and who they had conversations with. To be clear, we only tracked follows and conversations between two users we were already tracking; if a user followed someone who wasn’t there we didn’t care.
We pulled in data on everyone who tweeted with the hashtag and everyone on the curated list set up before the event. This means we had people who weren’t actually at the event but had tweeted because they were interested in it. I don’t think that’s a problem; there are still links there that may be interesting.
But I did add an “attending” flag on the system, so people could be marked as attending or not. At the hack day this was filled by marking those on the twitter list as attending and everyone else not attending.
So, what’s next for this? The main thing for the idea is to explore, probably by trial and error over more events, what information is signal and what is noise. What do people really want to know?
But here is other stuff you could add, apart from improving what’s already there and some good design:
- Allow users to set the attending flag. Logging in with Twitter is already implemented; users could set the attending flag for themselves. If a person hasn’t set the attending flag for themselves, others could be allowed to set it. (Some events may need multiple attending flags; the recent WhiskyWeb conference had two days and many people attended the first but not the second.)
- Show common relationships between two people; is there a third party they both know that could help with introductions? The problem here is that unlike LinkedIn (another service that does that) the definition of a connection on twitter is more vague. Just because Bob follows Alice, you can’t assume Alice has even heard of Bob.
- Start using follows from Twitter lists, and data from favorited tweets.
- We could end up pulling private data (more on that later) so we have to be careful. At the moment we discard such data but can we use it in any way whilst respecting the private flag?
- We track followers; can we track old follows as distinct to any new follows made during the event?
- Pull attendee lists from places like Lanyard.
- Search for links between people using others not at the event. For instance, maybe Bob and Alice are linked by Jane, but Jane’s hasn’t even heard of the event – but knowing they have a connection in common is still good.
- Graph analysis; can we identify clusters of people at events who know each other?
- Can we use other social networks like LinkedIn or Facebook? There may be less useful data there, and it’s harder to get.
Technically, here are the details: It’s done in PHP and MySQL and there isn’t much to say except Twitters Rate Limiting can be a giant problem. (I wasn’t the only one using Twitter and the venue’s IP was blocked almost as soon as the hack day started. Almost all my API developing had to be done on a remote server over SSH; great fun!)
At the moment it’s all done by pulling from the REST API and you have to do that a lot. You are constantly searching for new mentions of the hashtag. For ever user you care about, you are making two calls regularly; one for the tweets and one for the followers. It all adds up, and we haven’t started polling lists yet.
You are allowed many more calls as an authenticated user to a Twitter app and we started doing that. Apparently calls from authenticated users are measured per user; so if you can encourage users to log in you can make calls as them (however then you start seeing private data and you have to be careful).
Using realtime API’s may help a bit – and anyway a service that updates in realtime is much better than a slow one.
The end result? Several people really liked the idea and I know of one business that is going to be shown it in the coming weeks. The code I did is on GitHub, and I want to thank Rohan for coming up with the name and Carol for a rushed yet great job on the logo.
While there are lots of technical things that could be done, next is proving the central idea by asking is any of this data actually useful to attendees? I think lots of trials are needed to find out (and if anyone wants to invite me to a conference to try it out, get in touch!)