How to stop Google listing your site; the insane truth.

How do you stop Google listing your site?

What many people think the answer is:

Put a robots.txt file in the root of your website with:

User-agent: *
Disallow: /

Job done!

What actually happens if you do this:

Your site may still appear in Google with a natty little “A description for this result is not available because of this site’s robots.txt – learn more” message next to it.

Google Example Screenshot

(To be clear, just using this as an example. Not trying to pick on them at all.)

What just happened?!

Ok, some history. The robots.txt standard was started in 1994 and was intended to deal with the problems of bad robots causing problems for servers.

In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren’t welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren’t suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).

These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed. This standard addresses this need with an operational solution.

Note there is nothing there about SEO. That was not the original purpose of the standard. The original purpose was for web owners to have a way to indicate that robots should not go to certain pages.

So Google, it can be argued, is technically following the original intent and purpose of this. If you have a robots.txt file blocking robots, then the Google Bot will not visit your site*.

But – and this is the crucial bit – Google will still list your site with information from other sources, like links.

Ok, how do I actually stop Google from listing my website?

Go and read this official Google page carefully. It’s short, I’ll wait.

In effect, you have to add a tag to your page:

<meta name="robots" content="noindex">

And also – note the “Important” box on that Google page – you have to make sure the Googlebot is not blocked from seeing that meta tag by something, like, say, for the sake of example, a robots.txt file.

So wait, in order to stop Google listing my site …

… you have to make damn sure Google can crawl your site. Exactly.

That’s Freaking Insane!

Glad I’m not the only one that thinks that.

TL;DR

The robots.txt standard was created for the purpose of stopping bots from accessing parts of your site, nothing to do with SEO. Over the years, this point got confused. Now Google, while arguably following the standard to the letter, has created a slightly insane situation in regards to what you must actually do to make sure your site isn’t listed in Google.

 

 

* Really, Google do honour robots.txt. To the point that if you try and import a ics feed from a URL to your Google Calendar, and that URL is covered by a robots.txt file that bans robots …. Google Calendar will just point blank refuse.

Do people want a list of hustings for the Scottish 2016 election? (MeetYourNextMSP)

At the General Election 2015, I ran MeetYourNextMP with Democracy Club. This was a crowd sourced list of hustings around the country, and we made the listings we collected available for others as Open Data. Our mission was to get more people involved in politics by spreading knowledge of hustings and encouraging more people to go to hustings.

[If your wondering what a hustings is, that only goes to show how unaware of these events many people are. It’s a public event where you can question a panel of candidates standing for an election.]

With help from many contributors, we collected details of over 1,000 events in over 400 seats. We were cited by all kinds of people, including Citizens Advice Bureau, MIND for better mental health, The Campaign against the Arms trade, MacMillian Cancer Support and senior politicians across the party spectrum.

It wasn’t perfect (there are several long blog posts if you are interested in some of the issues), but for a small project with zero marketing budget it did very well. I would argue it showed that there is a need for hustings listings, and that this kind of project can go a long way towards meeting that.

However, it was a pretty gruelling time. I was working mainly by myself alone in Edinburgh, flat out and I was pretty burnt out by the time it was all over. We weren’t paid in any way for it, had costs to cover, and couldn’t do other paid work at the same time.

So, as the Scottish elections of 2016 approach, the question is do people want to see a MeetYourNextMSP? A website listings hustings in Scotland?

Is anyone so keen to see this they are prepared to help out? Either with a commitment of time, resources or money? For example, if there was a crowd funder, would they contribute? Could they contribute skills? Or even just time to go round and find events and add them to the site? (There is an easy process to do this, no tech skills required!)

Basically, I am trying to gauge the support a Scottish site in 2015 would get before I commit. To be really honest, without promises of substantial support from others I won’t be running this project. (I now have a 3 day a week job that this would have to work around to.)

Please comment below or if you’d rather comment in private an email address can be found here.

Thanks,

James

ps. Yes, I’m aware there are other big elections in 2016 (eg, Wales, London mayor). However, I strongly feel a local connection to the area is important to get the best out of community event sites like this. For that reason I would not get involved in hustings projects for those elections – but if other local groups wanted to do this I would happily share expertise, lessons learnt and Open Source software with them.

pps. Democracy Club’s core project is a Open Data set of all candidates standing with information about them. This was very heavily used during the General Election and they will be repeating this in Scotland in 2016.

How do we know what events people want to see on MeetYourNextMP?

MeetYourNextMP was a crowdsourced calendar of over 1000 independent events at the general election. This was to make these events easier to find so voters could go and question their candidates, and also to try and make more voters aware these events exist and are open to all.

When people came to the site, how did we know what events to show them? We asked people what their postcode was, and directed them to a page for each constituency. But what events did we show? Continue reading How do we know what events people want to see on MeetYourNextMP?

The problems finding details of independent hustings for MeetYourNextMP

“now that I follow my candidates, I’m really surprised how many hustings have happened / are happening that are terribly badly advertised” – Democracy Club Volunteer

MeetYourNextMP was a crowdsourced calendar of over 1,000 independent events at the general election. This was to make these events easier to find so voters could question their candidates, and also to try and make more voters aware these events exist and that they are welcome at them. However, before we could start on the second goal we had to crack the first goal. Continue reading The problems finding details of independent hustings for MeetYourNextMP

Why MeetYourNextMP only listed independent events

MeetYourNextMP was a crowdsourced calendar of over 1,000 independent hustings or candidate forum events at the general election. Its purpose was to make these events easier to find so voters could go and question their candidates, and also to try to make more voters aware that these events exist and are open to all.

We chose to only list independent events. This was much discussed at the start and now the election’s over, it’s time to take a good look at this policy. Continue reading Why MeetYourNextMP only listed independent events

Made a Prototype maze game with a Twist!

So 3 years ago I wrote up this idea for a GPS Maze game I wanted to make. I played with the idea at a hackathon then blogged about it … then did nothing more as there was one piece of it that was really hard to do. But then just last week I searched and found a library for that, and now after about 10 hours work I have a working prototype!

The idea is to have a GPs maze game, but based around a map that someone has drawn on a particular place. You draw a maze on a map:

c799d14a414d0ed573656eb252760f65

Then you download the maze onto your phone, go out to the actual place and play it! We’ve just been out for a walk to test it in an Edinburgh park.

43e928cda6bbcbaf7b2cef9e6b8790f0

The advantage of this over the usual maze games that just drop down a maze wherever you happen to be is that the maze designer can take account of any natural barriers or features that might make the maze more interesting, and they can avoid things that would be actively dangerous like main roads.

It could be a great way to explore a new area, or see a familiar area in a different light. And it could be fun to draw mazes on areas you know, and set people puzzling.

Lots to do on this still, so please excuse me not putting the code/app online yet – but let me know if you want to play! Android V4+ on phones with GPS only at the moment I’m afraid.

Are you doing a church website? UI protip

I’ve been looking at a lot recently (guess who tends to hold hustings) and the number that don’t have the actual full address and postcode of the church is crazy. A lot have the full address for the office, with no indication if that is also the address of the church or not. I guess they assume anyone looking at their website already knows where it is – but great way to reach out to new people, organisations that are currently worrying about a lack of new people!

Mind you, I’ve long been of the opinion that any website for a physical place (church,restaurant, whatever) that doesn’t have a big “How to find us” menu option that links to a page with a map, address, public transport and parking notes, etc was designed by morons.