The idea is to store acceleration data for the last couple of seconds and examine that. To trigger a shake event we look for the acceleration changing direction on one axis a couple of times. This isn’t a perfect system, as someone moving the phone in one direction only in stops and starts could produce an acceleration and deceleration that would trigger the algorithm.

But with this problem it’s about striking a balance between making it easy to shake and trigger but not so sensitive that just passing the phone to someone else or putting it down triggers it.

After we detect a shake, we ignore any events for a second. We don’t want two shakes to close together.

One thing to note is that when the screen shuts off due to user inactivity, your application is still in the foreground and is still detecting shakes which may make for a slightly odd user experience. To get round this, when the activity is active I’ve just locked the screen on. See the onResume() and onPause() methods on this class.

The final class is here and the listener class is here. To use it, simply have your activity create this class then pass a listener. Make sure you call onPause() and onResume(). Below is the minimum code needed.

import uk.co.jarofgreen.lib.ShakeDetectActivity;
import uk.co.jarofgreen.lib.ShakeDetectActivityListener;

public class MainActivity extends Activity {

    ShakeDetectActivity shakeDetectActivity;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        shakeDetectActivity = new ShakeDetectActivity(this);
        shakeDetectActivity.addListener(
        new ShakeDetectActivityListener() {
            @Override
            public void shakeDetected() {
                MainActivity.this.triggerShakeDetected();
            }
        });
    }

    @Override
    protected void onResume() {
        super.onResume();
        shakeDetectActivity.onResume();
    }

    @Override
    protected void onPause() {
        shakeDetectActivity.onPause();
        super.onPause();
    }

    public void triggerShakeDetected() {
        // do something!
    }

}

It’s on github as part of this app and I’ve used it in other apps. Feedback and patches welcome!

EDIT 22nd Feb: I made library easier to use, updated instructions.

For starters, something like  hello@mydomain.co.uk or  info@mydomain.co.uk looks much more professional.

But mainly, it’s insane because your tying yourself to one email provider and account for ever more. Once you start handing out that email address, you have to make sure you keep it.

What happens if something happens to your account and you get locked out? Or what happens if the service goes away or starts charging? If the service already charges, are you happy to pay that for ever more?

In contrast, you can have hello@mydomain.co.uk and move the actual provider of email servers between the best of thousands of companies. When you do this, your email address will not change – and thus all the people you have ever given your address to will still be able to reach you. Now you aren’t at the mercy of one company any more!

(I was at an event recently and politely told this to two people. They were grateful so I thought I’d blog it.)

It all started with NESTA’s Make It Local competition. This required councils to open some new data and start a new digital service with it. The code should also be Open Source. Me and Edinburgh Council entered, and won the prize: a grant to fund development! Today the result Edinburgh Outdoors was launched (Article and video).

After that the Edinburgh International Science Festival got interested. With any public charity money is tight, and they couldn’t have afforded to develop from scratch. But by building on the Open Source code we managed to put a project together and yesterday the festival programme was announced with this: the City Sci-Quest! Some new features are being developed for this and they will be released Open Source to.

The Open Source software is released on Github (Wed and Android, iPhone/iOS coming soon) and at the moment it’s more of a code dump than a full project. Things like documentation are needed, and making the code more modular would be good.

So what does this all do anyway?

It lets you run a website and native apps that list points of interest on a map and in categories and lets people find them. Users and anonymous browsers can comment or upload photos (moderation is an option).

There is also a game element. Administrators can set questions that users have to answer, with points being awarded and a leader-board being shown. Different types of questions can be set:

• Freetext lets the user type in anything as an answer. The administrators can set multiple correct answers so if the answer has two spellings that are popularly used then both can be recognised as correct.
• Multiple Choice lets points be awarded based on how many tries the user takes, with several answers having the capability to get points. One answer can get more points than others.
• Higher Or Lower is a question with a numerical answer. As the user guesses they are told “to high” or “to low”. Points are awarded based on the number of tries taken. A range of numbers (eg 3.5 to 4.1) can be set as the correct answer .
• Content questions ask the user to send a photo or some text to be displayed publicly, and the moderator can award points.

For every question, an additional explanation can be shown to the user after they have got it correct. This allows the game to be an educational experience.

Users can currently log in with an email address and password or Twitter.

There is a web admin interface and a very basic CMS is included. One way of showing static content from this is by month; so you have one page with all the months in the submenu and a different static content page for each one.

The Website is PHP & MySQL.  A lot of data is versioned in the database, although there aren’t friendly user interfaces to view that history.

I’m really looking forwards to seeing this idea develop, and also hopefully seeing this develop into a full Open Source platform.

Others involved are Sally Kerr at Edinburgh Council, Matt Wright at Edinburgh International Science Festival, Marius Ciocanel who is coding the iOS/iPhone version and Eskimo who did the Edinburgh Outdoors design.

You see, most programmers are self taught. Which means that if they know something can be done and they want to do it, they’ll just sit down with Google search results until they crack it.

So teach concepts instead. The idea will stick in the programmers mind and when they need it, maybe years later, it’ll pop up again.

For example, I found learning Android programming easier because I had seen a talk that explained the basic concepts of the activity life cycle and the intent system that sends messages between activities.

Or the talk about messaging systems that I saw was perfect for web developers. I think when I finally came to use a messaging system I used a different one from the one that was demoed (I used Beanstalkd and I can’t even remember for sure which one was demoed) but what was important is that the talk taught me the concept of moving the workload from the front of your web app to the back end.

So when I was finding talks for Tech Meetup, I would find talks like an introduction to user testing not because I thought all programmers should be experts in it (I wouldn’t call myself an expert) but because I thought everyone should know the basic concepts of user testing.

I’m not saying you should never have any technical details in a talk; sometimes a technical detail or demo communicates the idea much more effectively than words ever will, and its usually more entertaining. But always remember to concentrate on the concept.

I just gave a talk on running local meetups at Oggcamp, and this is the first blog post from that. Also check out Open Tech Calender -listing tech events and bringing them together.

How can we do this? We could write an open source app that works with notes and talks to an open source server, but setting up a server is still a pain and a barrier for some people.

Here’s a simple suggestion: Let’s use git as the backend storage and transport mechanism. You write your note in a text editor, commit it and push and your done!

Setting up a server is now easy because there are tons of git providers out there. Bitbucket give you private repositories for free for instance.

And look at all the features you get out of the box!

• History
• Merging
• Off-line access – I thought this would be standard for notes apps but I’ve seen a review of one that didn’t have it, yet was still praised. Really?
• Team work – just share a git repo between several users
• Desktop tools
• Backups – any repository is self-containing and can be used to replace a lost one, and if you want more security it’s easy to take a backup elsewhere!

The only flaw I can see is the requirement to store a copy of the repository on the device with full history, which could take up a lot of space. But we are talking simple text notes here, so hopefully not to much space would be used. And if you don’t care about history you can always regularly start a new repository and copy your notes across to reclaim some space.

Sure, this is a bit of a geeky solution and would probably never go mainstream – but sometimes there is nothing wrong in aiming for the geek audience.

Simply by putting text in the email subject that gets turned into active links in the tweet.

So when I announced Open Tech Calendar I used this email subject “Announcing Open Tech Calendar http://opentechcalendar.co.uk/ @OpenTechCalUK”. And this is the tweet that resulted:

The URL and the twitter username became clickable, meaning readers had one less click to reach the real content.

Before you get to carried away, let me just remind you that Tech Meetup’s email list is moderated and all spam or stupidity will be dumped.

So, please keep your posts to the community useful and informative – but here’s how you can get that little bit more out of them when you do.

When the internet first started, every computer attached to it had a number and people had to remember specific numbers. However, it quickly got to the stage where that was unworkable. Who can remmeber the number for Google is “http://173.194.34.101/“? So we invented the Domain Name System (DNS), which converts a name like “http://google.com” into the right number. (Every computer attached to the internet still has a specific number, but we don’t have to care about that anymore!)

Pretty cool huh? This is a problem we solved for website names decades ago. So why haven’t we solved it for phone numbers yet? Why are we still printing phone books so people can look up numbers of local taxi firms? Why do people still have to ask for their friends phone number’s a second, third or fourth time if they lose their phone? Why isn’t there a system to convert names into numbers for us?

Let’s go for a really quick win here with a simple, open, worldwide, scalable system; let’s just stick phone numbers in a DNS record.

Then you can have a simple Android app – enter a URL:

and get a list:

click on a number to jump to the dialer, check the number, and make your call! Download the app now from the app store or if you don’t have an android phone, there are some PHP scripts that let you play with this online.

Here’s what a DNS entry looks like – a simple country code, number and optional description:

v=phone1 44 3700100222 BBC Scotland

(Detailed instructions on this format are in github.)

One website can have several phone numbers and let the user choose which one, aided by the description. An open system means there are lots of tools people can build on top of this. Reusing DNS means a whole bunch of tricky technical problem are solved for us; uniqueness of names, caching, etc. Subdomain’s can be used so many users can be under one domain (eg jeff@gmail.com can choose to publish his numbers under jeff.gmail.com)

Does this mean numbers have to be public?

Answer #1: Actually, I could publish my work phone on “jamesbaster.co.uk” and my personal phone on “personal.jamesbaster.co.uk”. And yes, that is security through obscurity. Oh noes! But that’s how the current system works. What’s stopping you from calling Brian Cox right this second? Only the fact that you don’t know his number. Once you know his number, there’s no security. You can call him any time of the night or day – or at least until he gets a restraining order.

Answer #2: Well, yeah. But if we can solve the problem for public numbers with such a simple system that would still be fantastic. There are plenty of businesses that would benefit from having their numbers easier to find.

But no-one wants to type a domain name into an app!

The screenshots above are just a demo. What if you were browsing a website on your phone, and you could press the phone button to instantly get a list of numbers for that site to choose from? There are lots of possibilities. Apps could warn the user if it was a premium or international call.

How much are you charging?

It’s all Free and Open Source – both the Android code and some PHP scripts. The only way this will take off is if everyone can get stuck in. Download the app from the Android app store or play with the website.

What next?

Let’s see what people think of this …..please comment or blog/tweet/share if you like it!

(And if it’s popular, there are lots of improvements to be made to the sample apps – more robust parsing, save to address book, documentation on how to use libraries from them in your apps, etc)

Sometimes it’s done with good intents; a rich web app may decide a toolbar should always be visible, for instance. Usually it’s done with scummy intents; “If I make the viewer see SHARE ME! buttons wherever they scroll, eventually they’ll give up and do it!”.

Either way, read on for a simple visual guide as to why this must stop.

(I don’t mean to pick on these sites especially, and I’m sorry. Your just one of many that do this, and you were a handy example. Nothing personal.)

If you scroll down, the menu stays with you, like this:

Neat, huh?

Now, do you all remember that browsing on mobile devices is on the rise? Good. What do many mobile devices have in common? A small screen. What do people do on small screens? They zoom in to read the text. What happens with your clever floating menu then?

Congratulations. Your website is now completely unreadable. Here are some more examples:

In summary: use only when appropriate (clue: almost never), and test carefully.

But the problem with that is that firstly it doesn’t work if the user is off line, and secondly, you make the user perform an extra action to get to anything good.

What’s the alternative? You can embed several pieces of information including a URL in one code with a contact record (VCARD).

This is what comes up when you scan the QR code on my business card:

The user is instantly presented with options to email, call or visit my website – not to mention saving the whole lot to their phone book for later use.

I just wanted to briefly mention this as today someone said it was the first good use of a QR code they had seen. See this? This is my smug face.

Remember, the aim of the hack day is to explore an idea with other people, to see what is possible and hopefully kickstart the project into life after the hackday. You’ll probably end up building a prototype of the idea, to demonstrate how it works. The energy and ideas that are unearthed as people from different backgrounds work together is amazing.

[Note I'm talking about hackdays where a team gets together to explore a new idea. Some hackdays are about playing with a new technology, and those are different.]

Normally when computing programming, you have to think about the fact that your software will live on and hopefully grow and change. Other programmers will come in later and have to pick up the pieces and work out what is going on – and that includes you, looking at your work from a year ago wondering what on earth you thought you were doing. (There’s nothing worse after a long debugging session than discovering the moron responsible was you.)

To support that, we have come up with a whole load of professional “best practices” we follow when writing software.

• Requirements
• Architecture & Design
• Modularity
• Code Building
• Commenting
• Naming conventions
• Data validation
• Testing
• Automated Testing (eg Unit Testing)
• Continuous Integration
• Source Code & Release Management

However, all that is because we expect our software to live on into the future.

Let me make a prediction now about the software you will write over the hackday:

“Most code written at this event while developing ideas will be rewritten from scratch or almost completely rewritten before it is used for real.”

With that in mind, can we come up with a list of professional “best practices” for programming at a hackday?

• Just make the protoype work!

That’s it. Really. If you write a Unit Test at a hackday, you’re probably doing it wrong.

At one hackday, I did a site for finding dates at a festival. When you signed up there was a field to type your birthday in. During the final presentation, my team mate Sarah typed in a random string as she didn’t want to tell everyone her birthday. The whole site crashed; I hadn’t even bothered to write the error-handling code to deal with what would happen if users didn’t put in their birthday. Quickly, I told her to go back and put a real date in and it was fine.

I suggest you think about what will make an effective protoype – what will really get the idea across and impress others? At one hackday, we put together a messaging system for carers. We secretly found out the phone numbers of half the judges and organisers, and during our final presentation sent a message on our website. 20 seconds later 10 phones went off around the room as our message was texted out – a humorous and very effective demonstration of our idea.

So have fun!

This weekend I’ll be a mentor at this hackday in Pristina – looking forward to it!

[Photo by Oolong.]

Flocklight pulls all the twitter data for those attending or mentioning an event like a conference, and tries to highlight useful data about the connections between people. The aim is to provide you with information so you make the most of an event. For instance, are others you know going? Do you have something in common with a stranger there who you can go and chat to?

(The aim is not to produce some kind of score and crown someone the “winner”. I’m sure a gamification expert could do that and lots of people would love that – and indeed I was told about a recent conference with a score board for those that tweeted the most – but I personally think that’s meaningless, and I’m much more interested in exposing information to help you have a better time in real life.)

By 4pm on Sat (the deadline) we were tracking 148 users, 2272 follows between those users, 9354 tweets and 1417 instances of people mentioning another user in those tweets.

For each user, we had a page that exposed who they followed there and who they had conversations with. To be clear, we only tracked follows and conversations between two users we were already tracking; if a user followed someone who wasn’t there we didn’t care.

You could click on “relationship” to see the relationship between any two users:

We also did some really basic word analysis so you can search for people who are talking about things you are interested in. At Culture Hack, sleep was mentioned a lot:

We pulled in data on everyone who tweeted with the hashtag and everyone on the curated list set up before the event. This means we had people who weren’t actually at the event but had tweeted because they were interested in it. I don’t think that’s a problem; there are still links there that may be interesting.

But I did add an “attending” flag on the system, so people could be marked as attending or not. At the hack day this was filled by marking those on the twitter list as attending and everyone else not attending.

So, what’s next for this? The main thing for the idea is to explore, probably by trial and error over more events, what information is signal and what is noise. What do people really want to know?

But here is other stuff you could add, apart from improving what’s already there and some good design:

• Allow users to set the attending flag. Logging in with Twitter is already implemented; users could set the attending flag for themselves. If a person hasn’t set the attending flag for themselves, others could be allowed to set it. (Some events may need multiple attending flags; the recent WhiskyWeb conference had two days and many people attended the first but not the second.)
• Show common relationships between two people; is there a third party they both know that could help with introductions? The problem here is that unlike LinkedIn (another service that does that) the definition of a connection on twitter is more vague. Just because Bob follows Alice, you can’t assume Alice has even heard of Bob.
• Start using follows from Twitter lists, and data from favorited tweets.
• We could end up pulling private data (more on that later) so we have to be careful. At the moment we discard such data but can we use it in any way whilst respecting the private flag?
• We track followers; can we track old follows as distinct to any new follows made during the event?
• Pull attendee lists from places like Lanyard.
• Search for links between people using others not at the event. For instance, maybe Bob and Alice are linked by Jane, but Jane’s hasn’t even heard of the event – but knowing they have a connection in common is still good.
• Graph analysis; can we identify clusters of people at events who know each other?
• Can we use other social networks like LinkedIn or Facebook? There may be less useful data there, and it’s harder to get.

Technically, here are the details: It’s done in PHP and MySQL and there isn’t much to say except Twitters Rate Limiting can be a giant problem. (I wasn’t the only one using Twitter and the venue’s IP was blocked almost as soon as the hack day started. Almost all my API developing had to be done on a remote server over SSH; great fun!)

At the moment it’s all done by pulling from the REST API and you have to do that a lot. You are constantly searching for new mentions of the hashtag. For ever user you care about, you are making two calls regularly; one for the tweets and one for the followers. It all adds up, and we haven’t started polling lists yet.

You are allowed many more calls as an authenticated user to a Twitter app and we started doing that. Apparently calls from authenticated users are measured per user; so if you can encourage users to log in you can make calls as them (however then you start seeing private data and you have to be careful).

Using realtime API’s may help a bit – and anyway a service that updates in realtime is much better than a slow one.

The end result? Several people really liked the idea and I know of one business that is going to be shown it in the coming weeks. The code I did is on GitHub, and I want to thank Rohan for coming up with the name and Carol for a rushed yet great job on the logo.

While there are lots of technical things that could be done, next is proving the central idea by asking is any of this data actually useful to attendees? I think lots of trials are needed to find out (and if anyone wants to invite me to a conference to try it out, get in touch!)

You see, all of these games I’ve seen before just take a maze and put it down wherever you happen to be at the moment, like this one:

I don’t mean to pick on this one especially (sorry!) but I think that’s a bit rubbish. For one, it doesn’t take account of any natural barriers or features that might make the maze more interesting. And it could be actively dangerous – any roads around?

So here’s the twist: I want people to be able to bring up a map and draw a maze on top of it, using their local knowledge of the area to incorporate natural features. On this website, there would be a everyone’s mazes with ratings and comments. You would be able to download a maze to your phone, and play!

It could be a great way to explore a new area, or see a familiar area in a different light. And it could be fun to draw mazes on areas you know, and set people puzzling.

I’ve just spent a day at the WhiskyWeb conference hackday, and managed to make a good start on this project. There’s a website where you can design your map and export the data (The maze traces are a bit hard to see at the moment – click for full version):

And a native Android app where a map centered on where you are shows you the maze (with only one wall around the hackday venue, this one is a bit of an easy maze):

I was sitting on this idea for a bit, but my experiences today again taught me the value of being open – from talking to people today I realised that one big and hard feature I thought was essential actually isn’t. (Thanks Kat!)

So this is clearly early days – and I have no idea when I’ll find time to work on it – but let’s see what happens. Check the app and website out on github but don’t expect much yet!

If your Twitter app could work equally well with both read-only or write permissions, what do you do? If you ask for read permissions part of your app won’t work, and if you ask for write permissions then some users may be scared to give them to you and just abandon your app.

For ShowShift, I had this problem and I decided to try a different approach. I made 2 Twitter apps, one with read-only permissions and one with read and write permissions.

When the user signs up

The login link they first click is for the read-only app; then they are logged into the app and can use it at will. But they are prompted that if they let us have write permissions, they get more features.

The yes button takes them to Twitter to authorise the write app.

Behind the scenes

You now may have two sets of tokens for each user; one for the read-only app and one for the write app. Store them in separate columns in your user table and then use what ever is appropriate for what your doing.

CREATE TABLE user_account (
  id BIGINT NOT NULL,
  ....
  read_only_oauth_token VARCHAR(250),
  read_only_oauth_secret VARCHAR(250),
  write_oauth_token VARCHAR(250),
  write_oauth_secret VARCHAR(250),
  PRIMARY KEY(id)
);

When the user returns

The login link they first click is for the read-only app; so the login and sign up link is the same. This is enough to authenticate them and let them into the app.

Is this worth it?

Is having two apps, and the fuss of having to ask the user for authorisation twice to much? Is the proportion of users who are put off by your app asking for write permissions to start with so small that this added complication isn’t worth it? For some apps, maybe.

However, the users of ShowShift are going to be quite geeky to start with because of what it does, so I thought I’d give it a go and see what happens. The challenge now is to tweak the user interface to make it easy to understand. Comments welcome!

FaceBook seems like a good basis for this as then you can build on top of the existing social graph.

You could log in, and tell it what presents you were getting for someone. Then if a mutal friend logged in, they could see that list. So if I said I was getting my sister a cat, my brother could log in and see that and not get her a 2^nd cat, and instead plan to buy her some cat toys. (We’re not getting her a cat. A pet is for life, etc …)

You would need to be able to specify the present was for more than one person, so you can buy for couples. A mobile interface you can use when out shopping would be essential to.

You could say what events the present was for, and while a person would have obvious events pre-filled like Christmas (or other suitable holidays for them) and their birthday, if someone added a special event like “Retirement party” then others could see that, and know you were planning on getting them that comedy pipe and pair of slippers.

People could have wish lists of things they wanted, with links for purchasing. These could even be pulled automatically from places like Amazon.

I found one site from 2005, but it doesn’t seem to let you see what other family members have brought for someone, which seems a central feature. Amazon also have a Gift Organizer, but again, as far as I can tell it doesn’t have that social element.

Of course, some would say that this ruins the spirit of christmas and presents should be a more personal thing. Think of the competitve anxieties this might start! Up to you. But some would use it.

Ironically, I would have realised this could be a good app earlier if I hadn’t left my christmas present shopping till the last minute. So anyone want to write this? I’m off shopping.

Got a comedy phone call from a recruiter last month. He asked if I was looking for work and I explained I was a freelancer and looking for contracts.

He got very truculent, saying “It says here your looking for work!”

“Where did you get my details from?”

“CV Library*”

“That’s 2 years old!”

He paused for a second, then exclaimed “It says here it’s a year old!”

Seriously people, if the information you have is a year out of date then check before acting on that information. It’s not hard.

But then he calmed down and got very morose and sad, complaining how hard it was these days – “finding you guys is like gold dust”. Bless.

* These fools also just sent me a newsletter without an unsubscribe link, not recommended.

On Friday and Saturday, the Moving Targets event brought together developers and people from industry to work on the problem of audience engagement. This is the idea our team talked about with industry people on friday afternoon, the idea I pitched to the judging panel, one of two ideas they picked and the idea me and Amy Guy spent the night hacking up.

The ideal situation is to watch an event on TV (Film, Show, Sporting match) while taking in the social media commentary from others, so you get a full experience as part of a community of fans (some BBC shows broadcast Twitter details at the start to encourage this).

The problem is so many people watch after the initial broadcast, either from on-demand services like the BBC iPlayer or a recording device. When you do this, you miss out on the social media experience – and you have to be careful not to read any spoilers beforehand.

So why not have a service that re-plays social media messages as they happened, but at a later date? Any messages you add as you watch can be seen by others watching later on.

Here’s the end result; a web service that can import tweets for any event and make them available.

Ideally, you’d have better login mechanisms than this:

Then you can list the videos we have a conversation for.

You can open the video in a new window or “Stream” the messages – this shows you some controls to control the timeline, the messages and a box for you to post your own.

The blue marks above the progress bar indicate when messages appear; so if there was a rush of messages about the 14 minute mark (maybe that’s when the first goal was scored?) you can jump straight there.

One pain point is that you have to manually sync the message feed to what your watching; it would be better if we could use a microphone or camera to see the broadcast and do the synchronisation ourselves. Sadly, that was a bit beyond us in one night.

This page was designed with mobile devices in mind, and was tested all night in iPhone and Android. At one point, we even broke out Amy’s kindle for a laugh, and were amazed to see it working perfectly.

The watch option on the home screen appears only for YouTube videos and embeds the player right in the page.

You can add any video you want.

We talked about options for

* filtering out the noise of the conversation by having some indicator of how long ago the message was posted or only showing messages from your friends.

* allowing spam messages to be reported to admins for review.

* adding notifications so if you replied to a message the original author could be notified and carry on the conversation.

* collecting some amazing data about which bits of a programme people paused at, fast-forwarded or re-watched.

All night, other coders kept remarking how similar the UI looked to SoundCloud – a service I have heard of but still haven’t seen.

Oh, and it was done in PHP & MySQL with thanks to jQuery, Smarty and Bytemarks Bigv.io – thought their beta programme, I was able to get a massively over-specced Debian+Apache+APC virtual machine for free that worked perfectly.

Special thanks to my team mate Amy, Jonathan Galea for making us some great graphics and Pockets for producing the video we used all night as a test piece.

What else is there?

We did a quick google search on Friday afternoon and found nothing. However, a more leisurely search just now found Tweet Rewinder – a service that replays your tweets for you. We did say at the hackday that it seemed like such a simple idea and we were surprised no-one had done it before, and it turned out they had – even if literally only by 1 month.

It was still fun ….

It was still a fun hackday, and hopefully demonstrates how even one day and nights work can produce an idea and a solid proof of concept you can show to people to develop the idea further. Thanks to all involved.

It was based on color.com, the photo sharing service that shows you photos that others have taken around you. They thought this was daft – after all you know what’s around you, your already there. What you don’t know is what’s at the other side of the world. They wanted a service were you could take a photo, and you would get one back from the other side of the world.

They said they would be happy with just a mere fraction of the mind-boggling $41Million that color.com raised – really, just a tiny fraction and they’d be very content. (Same goes for me please!)

They then tried to write iPhone and Android apps, via an amusing journey during which they fired one of their team mates by twitter half way through, who then turned out to be their flatmate. “You fired your own flatmate via twitter? Couldn’t you have just left a note?” “That’s how we roll.”

Why not do a proof of concept with a simple webpage and the Flickr API, I thought? And finally I did. You can play with it here, and the code is on github.

So how does it work? To display the map it uses OpenLayers, a Javascript library that allows you to put maps from many sources on a webpage and allow an user to interact with them. It has a built in class for showing OpenStreetMap, the world map that anyone can edit in a wikipedia fashion.

Then we use a service from MaxMind to locate the user and set the initial position on the map. You can include a Javascript file in your code, which will contain several functions that return geographical information. All the information is hardcoded at the server side so the resultant file is only about 500 bytes in size and will work in any Javascript capable browser.

When the user clicks go, a simple bit of PHP converts the coordinates to the other side of the world then sets up the 2nd page. This calls another bit of PHP by AJAX to actually load the photos. This means the page displays quickly to the user.

The Flickr API is nice and clear, and returns XML that is easy to turn into useful JSON. It does seem to error a lot tho – it’ll say it has 200 pages of results but the results XML will be empty. We do a bounding box search, set quite wide to ensure it finds some results. That is a bit kludgy – it could do something like start with a narrow bounding box, but if no results come back you could expand to a much larger one. You wouldn’t want to do that more than once tho, for the sake of Flickrs server load and your page load times.

Then some simple Javascript helped by Jquery lets the user explore the photos!

What could be improved? We already have groupings so if some photos are on exactly the same location they will be shown with one marker, but what if two photos are very close together but the user is zoomed out? It will look like there is only one marker there – some clustering would help with that.

A slideshow mode that jumped from location to location to show all the photos would be cool to.

But there you go. When working on this I found a whole batch of photos from New Zealand that made it look fantastic: mountains, lakes, beaches and 20 men sitting on brightly painted tractors at some kind of parade – I want to go!

You can play with it here, and the code is on github.

The problem is that many insist you point your DNS records at them, then set up the options for the custom domain. They have checks that won’t allow you to do it the other way round. This means that in the time between you changing your DNS records and setting up your custom domain, you have unavoidable downtime.

I’ve previously ranted about WordPress.com, which wont let you continue until you point your name servers at it. This means that every service on that domain, including those that have nothing to do with websites and are considered vital, like, oh, email, will have unavoidable downtime.

Last week I moved a site into DrupalGardens.com, and they insist that a CNAME record exists before you continue. Which means some unavoidable downtime for your website, but at least your email survives.

I haven’t tried other popular services like Tumblr and Posterous, but looking at the instructions it implies they have the same problem.

I guess it’s a result of lazy design; after all you should really check the DNS is set up correctly and if you do it when the user sets up the domain it’s a nice simple workflow for all concerned.

So who gets this right? Google Apps. You can set up an Google App control panel, add all your user accounts and aliases, wait for the users to log in, run training sessions (if needed) and check they are happy and then, and only then, must you change your MX records. Which means that you can move an existing domain into Google Apps with no disruption to your incoming email at all. Brilliant!

FakeFringe challenges players to pick out the fake fringe show from the real one. You can submit a fake show and see how many players you can fool.

FringeWars pits fringe shows head to head and asks you to vote.

Both these sites compile charts of the best and worst shows, and are intended in a playful spirit. But they have a serious side; the chart of best shows should be valuable data for people writing Fringe shows marketing in the future.

I worked on both these sites before the hack day, and they are open now. At the introduction to the hack day, I announced an idea and showed all the work I had done on it so far: a 10 minute mock up of the home page:

The fantastic Sarah Drummond from WeAreSnook joined me, and we made a full set of mockups and a working prototype over one hectic day. The slideshow with our final presentation is here:

We hope to take all these ideas forward in the future: Watch this space!

Every time a coder commits something into a source control repository, they make  information available. Ohloh is one site that attempts to extract information from that. To start with, I just wanted to look at the time stamps.

Firstly, looking at the commits I made on Elastik in a more date-orientated manner: lets put them into an iCal file, so we can view them on a calendar.

A couple of commits Monday evening, then lots on tue and wed during office hours? We’ll come back to that. How about aggregating them by hour of the day?

No great surprises there, although even I’m surprised I committed something at 6am in the morning (you can’t see it in the graph because it only happened once, but I did).

Why is there a marked increase on Tue, Wed and Thu? If you know I only work part time, and those are my normal work days, then this graph and the calendar screenshot make sense.

The basically working proof-of-idea PHP scripts are here.  I did try to build a very modular class structure so it’s easy to add other methods of collecting or writing data, but the code is quick and dirty. Be warned.

I did it all on a netbook over one weekend. The 1st version was started on the train on Friday afternoon, and was got working when I got home late on Friday night. On Sat and Sun I was out all day, but I did some coding on it both mornings before I went out and both evenings when I came back.

It’s not perfect information. It doesn’t tell us how long a programmer spent working on a task before committing it, and with systems like Subversion that need network access you can’t tell if the programmer worked off-line and then committed later when they got access. But still, turns out it’s pretty trivial to get information about a programmers routine.

This could be extended to other open source activities; looking for comments in an issue tracker for instance, all of which are time-stamped. Or people other than programmers; anyone who contributes to an web forum regularly could find their routine analysed.

Comments welcome!

So you run your blog on myblog.com, a little dinky server somewhere. It has a contact form and user@gmail.com comes along to send you a message. They fill out the form and press submit, and your server at myblog.com puts together an email message and sends it to you at me@myemail.com.

However, the operators of myemail.com have had enough of spam and are fighting back. One way is by using data from Sender Policy Framework records. This is a system where by the owners of a domain can specify which servers are allowed to send email for it. It is designed to prevent spoofing, where an email claims to come from a user but doesn’t really. Here’s what happens.

The server at myemail.com gets the email that claims to come from user@gmail.com. They look up the SPF records for gmail.com

james@ubuntu:~$ dig gmail.com TXT
....
gmail.com.    300    IN    TXT    
      "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19"
....

We can see that Google has specified a list of IP blocks that are allowed to send email. At this point myemail.com notices that the email apparently from user@gmail.com really comes from myblog.com, a dinky little server that is not in one of those IP blocks. Having determined it to be a spoof email, it marks it as spam and you never see the message. Meanwhile, the user who sent the message gets disillusioned by your lack of response and never visits your site again.

So what do you do about it? Simple: don’t put the email address the user supplies as the “From” address. The “From” address should be an email address on the server, such as “blog@myblog.com”. What you can do is set the users email address as the “Reply-To” header (after carefully checking it is actually a email address to avoid header injection attacks) so that when you press reply in your email client the reply automatically goes to them and not blog@myblog.com.

I sat down to write this after seeing this mistake many times, including in a WordPress plugin with almost 10,000 downloads. My patch for the plugin is: patch.replyto, there is a discussion thread for it here.

ps. I’ve simplified the actual SPF record from Google for a simpler example.

To use an existing domain name on your blog, they ask you what the domain name is. They then check the name servers for that domain name, and they refuse to continue until the name servers are pointing at them. When you have done all this, apparently you can add extra DNS records for your domain – for example, an MX record for your email server.

Firstly, why does wordpress.com insist on being the name server for my domain? I already have a perfectly good name server. Why can’t I just change my A record to point to wordpress.com?

However, the thing that pushed me into self-hosting was that while I was going through this sign-up procedure (they warn it could take several hours), my email and all my sub-domains would be off-line, and there was nothing I could do about it.

It basically means that if your trying to move an existing domain onto wordpress.com and downtime is a problem for you, you can’t do it. I’m about to look at CMS’s for several local charities. I would like to consider wordpress.com because small charities tend not to have a staff member who can look after software installs, and if you have to pay someone to look after your WordPress then wordpress.com is a decently priced option from the experts. But this one issue may mean I can’t.

So I ended up hosting this blog on a Debian server, where a fantastic package made setting it up easy (as always).

There are weekend long events, like the Social Innovation Camp I did in summer 2009 where teams work to make the first version of a social enterprise over a weekend. Just like Launch48 for good causes, basically. And recently I’ve done several Social Media Surgeries in Edinburgh, where volunteers teach local groups about twitter and the like. There are websites where charities and community groups can post requests and get people to work on small projects, like Sparked and it4Communities.

With these, I think the elephant in the room is what happens next – it’s easy to get people to sign up for a fun weekend, but ongoing support or development work is more important and that’s harder to find volunteers for.

For instance, setting up a WordPress blog for someone is one thing but if they don’t keep up to date with security releases they will get hacked eventually (This is true of all software; not picking on WordPress). Fortunately, there is http://wordpress.com/ – if you start a blog there, they will take care of all the system administration for you.

Also, how does a charity who doesn’t really know what is possible decide what they want? And how do they judge the quality of the work? Both seem to require a more general technology consultant to help them, or some kind of feedback loop.

In the mean time, I’ve signed up for more Social Media Surgeries and have volunteered for my first project with it4Communities – so we’ll see how that goes …

Suppressed errors

In PHP, you can indicate that some code should suppress any errors with an @ sign.

@print_r($this_variable_does_not_exist);

I personally don’t think you should do this – are you really sure you don’t care about any of the possible warnings that a function may generate? However I accept some programmers may think this is ok, and more to the point, you may have to collect errors from an application you did not write that does this.

Note that the error handler is still called in this case. But the error_reporting() function will return 0, so you can test whether errors were suppressed.

function error_handler($number, $message, $file, $line, $vars)  {
   if (error_reporting() == 0) return;
   die($message);
}
set_error_handler('error_handler');
@print_r($this_variable_does_not_exist);
print "OK";

Now ‘OK’ will be printed, as you would expect.

However there is a gotcha: If error_reporting is set to zero anyway, then all errors will be ignored. So make sure that error_reporting is set to a non-zero value before using this trick. If error_reporting is set to zero then I don’t know of a way to tell if errors are suppressed – if you do please let me know.

More information from errors

The debug_backtrace() function will provide you with a full stack-trace which can be very helpful for working out how the code got to the error.

<?php
function e($number, $message, $file, $line, $vars) {
   $x = debug_backtrace();
   print_r($x);
   die();
}
set_error_handler('e');
function b() { $hteu->oueoe(); }
function c() { b(); }
c();

Array
(
   [0] => Array
   (
      [file] => /media/692A-1D57/PHP Talk On Errors/debugBacktraceSample.php
      [line] => 8
      [function] => e
      [args] => Array
      (
         [0] => 8
         [1] => Undefined variable: hteu
         [2] => /media/692A-1D57/PHP Talk On Errors/debugBacktraceSample.php
         [3] => 8
         [4] => Array
         (
         )
      )
   )

The output is truncated to the 1st line only, but you can see it includes file, line, function and args – lots of data. But if you look closely you’ll see the function is set to ‘e’ – in other words, the exception handler itself is included. I find you can get a more accurate stack trace by taking off the top line and putting back a line with the information passed to the error handler.

<?php
function e($number, $message, $file, $line, $vars) {
   $x = debug_backtrace();
   array_shift($x);
   array_unshift($x, array('function'=>'?','file'=>$file,'line'=>$line));
   print_r($x);
   die();
}
set_error_handler('e');
function b() { $hteu->oueoe(); }
function c() { b(); }
c();

Array
(
   [0] => Array
   (
      [function] => ?
      [file] => /media/692A-1D57/PHP Talk On Errors/debugBacktraceSampleImproved.php
      [line] => 10
   )
   [1] => Array
   (
      [file] => /media/692A-1D57/PHP Talk On Errors/debugBacktraceSampleImproved.php
      [line] => 11
      [function] => b
      [args] => Array
   (
 

Finally, we previously talked about to how to catch FATAL errors with a shutdown function. In this case, debug_backtrace() will return the shutdown function only. I haven’t found a way to get a stacktrace when a fatal error occurs, and would love to know of one.

Multiple Error handers

You can only have one error exception at a time. When you call the set_error_handler() function it will return the name of the old error handler. You can store this and call it yourself from your error handler – thus allowing you to have multiple error handlers.

function e($number, $message, $file, $line, $vars) {
   global $old_error_handler;
   call_user_func_array($old_error_handler, array($code, $message, $file, $line, $vars));
}
$old_error_handler = set_error_handler('e');

Handling Errors and Exceptions leaving you puzzled? We work on an open source project to log problems, display a nice message to the user and email developers. We detect duplicates and let you track errors in tickets. Have a look at the ErrorReportingService module in Elastik!

PHP Exceptions are a object-orientated way of triggering errors and dealing with them.

try {
   throw new Exception();
   print "This line is never run";
} catch (Exception $e) {
   print "error";
}

You can create a new exception object, and throw it. When thrown PHP will work back up the code stack looking for a catch block that can handle it. You can have as much code as you want in a try or catch block.

Because exceptions are just objects, you can extend them

class MyError extends Exception {}
try {
   throw new MyError();
} catch (MyError $e) {
   print "error";
} catch (Exception $e) {
   print "error";
}

Note you can have multiple catch blocks and the appropriate catch block will be used. Inheritance works; so any error that extends the Exception class will be caught by the catch(Exception $e) block.

PHP has a number of built in predefined exception classes you can use.

You can also re-throw an exception:

try {
   throw new Exception();
} catch (Exception $e) {
   throw $e;
}

So in your catch block you can run some code (maybe clean up a resource, like close a file you have opened and were half way through reading) then re-throw the exception for it to be dealt with by other code.

Any exceptions that aren’t caught will kill your script. You can catch them and do something with them before PHP dies. This example is directly from the set_exception_handler() manual.

function exception_handler($exception) {
   echo "Uncaught exception: " , $exception->getMessage(), "n";
}
$old_exception_handler = set_exception_handler('exception_handler');
throw new Exception('Uncaught Exception');
echo "Not Executedn";

You can only have one exception handler at a time, but you can store the name of the old exception handler and call it yourself in your exception handler – more on this later.

Getting information out of Exceptions

This is easy with the getMessage(), getFile(), getLine() and getTrace() functions.

However note that these don’t give you the place that the error was thrown from but the place the exception was created (contrary to documentation). Seeing as almost all of the time people create and throw the exception on the same line, it’s not a problem and is in fact is a good thing – if you re-throw the exception then the information in it will be from the original problem, not when it was re-thrown. But if you create an exception then pass it around for a bit then watch out for this.

When to use exceptions?

That is a matter of personal choice, but here are some suggestions to get you thinking.

If a function returns a value, or 0 or -1 for an error, that could be a candidate. Especially if there are multiple errors: you may think it’s easy to return -1, -2 and so on but that makes it very easy for a programmer to forget to check for an error state (and they will probably have to refer to the manual all the time). Instead, throw an Exception if there is an error. This is harder to ignore and much more readable.

What if a function returns a failure state that evaluates to the same as a valid state when doing a lazy comparison? intval() returns “The integer value of var on success, or 0 on failure.” Which means that these 3 statements all return the same!

intval('0')
intval('')
intval('oeuoehtu')

If your writing a function for a web-app which has a text field to collect an integer input, you want the 1st to return 0,the 2nd null (to indicate the user did not put anything in), and the last to raise an error or exception.

Handling Errors and Exceptions leaving you puzzled? We work on an open source project to log problems, display a nice message to the user and email developers. We detect duplicates and let you track errors in tickets. Have a look at the ErrorReportingService module in Elastik!

This should set alarm bells off with everyone.

They aren’t using an SSL certificate. An SSL certificate means that all traffic between your web browser and the web server is encrypted and no-one can listen in. (If your using an open wifi spot at a coffee shop, listening in is very easy). Someone listening in could get your credit card details, so using a SSL certificate for important data like credit card details is essential.

Except they are: If you use a tool to investigate the AJAX calls, you will see that every single one is to an https server. They use encryption for all your data.

So that’s fine then? Umm, no.

Web browsers provide tools to users to show them when an encrypted connection is being used. We should be training users to look for them and to refuse to use websites that don’t have them. Amiando undermines this education effort.

The 1st time I used Amiando was for a big conference, a couple of hundred people. When I saw this I stopped my order and complained to the organisers. After a bit of to-ing and fro-ing I payed by bank transfer instead. But when I was there I had a chat to the event organisers, trying to educate them nicely, and the event organizers informed me in a slightly exasperated manner that I was the only one to complain. So much for the education effort then. The argument about how you educate users effectively is a big one.

Still, I would really like Amiando to sort this out. We should be educating users about security.

Type 1. Most errors

This is the example you usually see in tutorials. Most errors can be caught by setting a custom error handler using the set_error_handler function. The example on the official PHP manual page is very long but here are the basics:

function error_handler($number, $message, $file, $line, $vars)  {  
     .... do something with error ...
}
set_error_handler('error_handler');

Note the PHP process won’t die – after calling your error handler, it will carry on where it left off.

Type 2. Exceptions

You’ll also want to catch any exceptions using the set_exception_handler function. This time the example on the official manual page is short and to the point.

Type 3. Fatal Errors

Finally, how do you catch Fatal Errors? Because they are fatal, a custom error handler isn’t called. But you can do this:

function x() {
   $error = error_get_last();
   if ($error) {
      // deal with $error['file'],$error['line']
      // $error['message'],$error['type']
   }
}
register_shutdown_function('x');

This is called when PHP shuts down. It will only get the last error, but by definition a fatal error will be the last error to occur so that’s fine.

How do you know it doesn’t get the same error you may have seen earlier? It would be annoying to get the same error reported twice. Fortunately, errors you catch with a custom error handler won’t appear in the results of the error_get_last() function.

Note that what you can do in a shutdown script can be limited – PHP is shutting down after all. So don’t try anything fancy.

What to do with these errors?

Now we have caught these errors, what do we do with them? Ideally you want to log them, display a nice message to the user and email some developers. Which is one of several things our open source project does with errors – it also detects duplicates and lets you track errors in tickets. Have a look!

It just smacks of bigger counties getting all whiny about the existence of smaller countries and having to scroll. It just rings of arrogance about how the small countries aren’t important and just get in the way.

And from an UI point of view, it seems like bad advice.

Your list is suddenly out of order, Sure, if you do happen to live in the biggest country your country is now at the top and it’s great. If not, you suddenly have 2 places your country could be – annoying.

So what’s the alternative?

You don’t have to scroll. You can type the first few letters and the computer will scroll for you. But I’ll accept that not everyone knows that. So maybe you could have an empty text box and when you start typing, an auto-complete popup box suddenly appears to help you.

Or maybe have a select box with a sensible default selected. What should that sensible default be?

If your a website that mostly serves one country it’s easy. Or if your a website that has separate domains for each country, it’s easy. Someone on http://www.amazon.fr/ is going to want France, 99% of the time.

So what if your a genuinely international website that serves many different countries? Use IP Lookup. Or use the language the user has chosen to browse the website in as a hint.

Now I’m not an UI expert and I’d love to hear from one on this topic. It just seems to me that “Just put the most popular one at the top!” is a really lazy solution that smacks of arrogance on the part of big countries, and there must be a better way.

Each error report contains full information (stacktrace, server variables, user submitted data – click the screenshot below for a sample error).  Users can be informed by email as soon as an error occurs, and a ticket can be created so you can track the process of fixing the error. We attempt to catch duplicate errors and tie them together so one error occurring many times does not flood your inbox.

There is a PHP class you can add to any project to collect errors, and I’m also working on a WordPress plugin.

For more see the documentation on the ErrorReportingService module here. Any feedback or ideas on what you would like to see is appreciated!

An Error Report in Elastik (Ver 0.3.1)

It’ll sit there saying “5 minutes” till the bus, but if that information is 3 minutes old then you have much less time than you think you did (I’ve already missed one bus!). A simple addition of a small text label saying “this is 3 minutes old” would enable the user to quickly work out how long they really had.

It does have an auto-update feature, but this would still be good. What if the user had a data connection then loses it? (They may leave a wifi spot for instance).

Apart from that, brilliant app

Edit 28th Jan: Got a tweet from the developers saying this will be considered, here’s hoping.

Edit 19th March: This issue can be tracked at http://code.google.com/p/androidedinburghbustracker/issues/detail?id=16 and will apparently be in the next version.