How to stop Google listing your site; the insane truth.

How do you stop Google listing your site?

What many people think the answer is:

Put a robots.txt file in the root of your website with:

User-agent: *
Disallow: /

Job done!

What actually happens if you do this:

Your site may still appear in Google with a natty little “A description for this result is not available because of this site’s robots.txt – learn more” message next to it.

Google Example Screenshot

(To be clear, just using this as an example. Not trying to pick on them at all.)

What just happened?!

Ok, some history. The robots.txt standard was started in 1994 and was intended to deal with the problems of bad robots causing problems for servers.

In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren’t welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren’t suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).

These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed. This standard addresses this need with an operational solution.

Note there is nothing there about SEO. That was not the original purpose of the standard. The original purpose was for web owners to have a way to indicate that robots should not go to certain pages.

So Google, it can be argued, is technically following the original intent and purpose of this. If you have a robots.txt file blocking robots, then the Google Bot will not visit your site*.

But – and this is the crucial bit – Google will still list your site with information from other sources, like links.

Ok, how do I actually stop Google from listing my website?

Go and read this official Google page carefully. It’s short, I’ll wait.

In effect, you have to add a tag to your page:

<meta name="robots" content="noindex">

And also – note the “Important” box on that Google page – you have to make sure the Googlebot is not blocked from seeing that meta tag by something, like, say, for the sake of example, a robots.txt file.

So wait, in order to stop Google listing my site …

… you have to make damn sure Google can crawl your site. Exactly.

That’s Freaking Insane!

Glad I’m not the only one that thinks that.


The robots.txt standard was created for the purpose of stopping bots from accessing parts of your site, nothing to do with SEO. Over the years, this point got confused. Now Google, while arguably following the standard to the letter, has created a slightly insane situation in regards to what you must actually do to make sure your site isn’t listed in Google.



* Really, Google do honour robots.txt. To the point that if you try and import a ics feed from a URL to your Google Calendar, and that URL is covered by a robots.txt file that bans robots …. Google Calendar will just point blank refuse.

Made a Prototype maze game with a Twist!

So 3 years ago I wrote up this idea for a GPS Maze game I wanted to make. I played with the idea at a hackathon then blogged about it … then did nothing more as there was one piece of it that was really hard to do. But then just last week I searched and found a library for that, and now after about 10 hours work I have a working prototype!

The idea is to have a GPs maze game, but based around a map that someone has drawn on a particular place. You draw a maze on a map:


Then you download the maze onto your phone, go out to the actual place and play it! We’ve just been out for a walk to test it in an Edinburgh park.


The advantage of this over the usual maze games that just drop down a maze wherever you happen to be is that the maze designer can take account of any natural barriers or features that might make the maze more interesting, and they can avoid things that would be actively dangerous like main roads.

It could be a great way to explore a new area, or see a familiar area in a different light. And it could be fun to draw mazes on areas you know, and set people puzzling.

Lots to do on this still, so please excuse me not putting the code/app online yet – but let me know if you want to play! Android V4+ on phones with GPS only at the moment I’m afraid.

Are you doing a church website? UI protip

I’ve been looking at a lot recently (guess who tends to hold hustings) and the number that don’t have the actual full address and postcode of the church is crazy. A lot have the full address for the office, with no indication if that is also the address of the church or not. I guess they assume anyone looking at their website already knows where it is – but great way to reach out to new people, organisations that are currently worrying about a lack of new people!

Mind you, I’ve long been of the opinion that any website for a physical place (church,restaurant, whatever) that doesn’t have a big “How to find us” menu option that links to a page with a map, address, public transport and parking notes, etc was designed by morons.

Learn Tech Edinburgh

Several months back I wrote this blog post “Adults wanting to learn to code?“. It got a great response and we ran 4 trial sessions over June and July.

We wanted to run an informal group, where people wanting to learn could meet up and chat, help each other or just have a place to work to make them actually do it. There would be several programming mentors on hand to help people if they got stuck.

There are several programmes with set courses going around, and in contrast this was to be a group where people worked on what they wanted to. Most people worked on Python, mostly through Codecademy.

This wasn’t exclusively aimed at under-represented groups in programming, but we do want to make sure we have a very inclusive atmosphere for learners where they can feel very welcome. We briefly discussed various rules or guidelines for this.

We wanted to run some trial sessions quietly just to see how it went with a small crowd so we didn’t advertise to heavily. It seemed to go pretty well – we may be having a feedback session to discuss more. I could write more on how I think it went, but I want to hear more feedback from others first.

(And we need to think of a better name – that common problem of new things!)

We’re now going to try opening it up to more people as we carry on. If you’re interested in attending drop me a email and I’ll add you to the email discussion list. [ james at jarofgreen dot co dot uk ]

Thanks to everyone who has encouraged this, been keen about coming and the other mentors!

Open Source projects and Git self hosting

A mailing list I’m on has been discussing where Open Source projects should host Git repositories. I wrote up these rough thoughts and have been asked to post them for others:

Been thinking about this, and I realised Open Source projects have to be very careful self-hosting.

The whole model of Open Source on Git for welcoming contributors is Fork-and-pull-request. As in Random Person A forks your projects, makes changes, makes pull request, you check,¬† accept and merge – great. But that also means you have to allow Random Person B to fork, change, request and you to realise that B’s code changes are totally wrong¬†(or worse, B themselves is an idiot) and politely refuse.

But this means if you self-host your Git repository for an Open Source project, you ideally have to allow random person A and B (who may be an abusive moron) to make accounts and start putting whatever code they want up there. You basically have to became a code host and start watching your hosting very carefully, otherwise before you know it someone is abusing your servers or has uploaded objectionable material. That’s not a welcoming prospect at all for an already overworked Open Source project.

The alternative is not to allow others to make new repos on your own Git hosting, but then suddenly you’ve made it harder for new contributors to contribute. You also lose visibility; if a random person forks your code most places like GitHub will track that with pretty graphs but if they clone your project and push that elsewhere you don’t see that. This matters because even if someone forks your project and changes it for their own personal use, it’s still interesting to see what they have done.

I don’t know where Open Source should host Git projects; comments and more blog posts welcome.


Adults wanting to learn to code?

Recently lots of clubs where volunteers teach kids to code have sprung up like CoderDojo and Prewired, and I know several adults have been jealous.

Often the best thing to do is start small and see how it goes, so let’s suggest this: a regular evening slot, maybe twice a month, where adults who want to learn to code can meet up and work together. They would work online on a course of their choice (Codeacademy for instance) but be with other people they could swap tips with and get encouragement from. We would try and encourage a couple of programmers along to help out people who get really stuck.

What do you think? Would people be interested? Comment below ….

You shouldn’t be able to downvote without comment

Discuss: You shouldn’t be able to downvote someone else’s content on a community moderated site without also having to explain why.

A downvote without explanation may deal with trolls but it does a terrible job of encouraging newcomers to a community who just get something slightly wrong. It basically sends the message “f* off” with no explanation or clear path on how they can engage better in the future.

For this reason, it also encourages groupthink not discussion. Anyone who’s opinion differs slightly from the prevailing opinion is down voted. The downvote offers no chance to engage in discussion to explore the issue; it simply says “You’re wrong – end of”.

Basically, a downvote without comment is a door slammed in someone’s face which offers them no path to discuss the issue more, no path to further learning and no path to re-engage with the community. The person who gets it is shut out, and the person who does it just reinforces their own views and reinforces the groupthink of the community.

Of course, it’s not all bad. Some people are just trolls who need many doors slammed in their faces and also communities need a bit of cohesion to survive. But it seems to be an unchallenged assumption that community moderated sites should allow members with a high score to downvote freely, and I think it should be considered more carefully.

Moving on from TechMeetup, but still want to help the tech community

I’ve been involved with TechMeetup for a while now. In 2010 I presented a couple of evenings then in 2011 I was the main presenter in Edinburgh, finding speakers and presenting the evening. Since then I haven’t needed to do much – others like Douglas, Dale, Marius and Phil have taken over active roles and done a great job at organising Edinburgh Events.

I’ve decided it’s time to officially move on tho. Many people still associate me with Tech Meetup and I feel I should be clear that I’m not involved anymore. I don’t expect this to change anything – every TechMeetup event I’ve been to recently has been great, and I expect that to continue.

So what will I be doing? I’ll carry on working on Open Tech Calendar. I’m working on a spin off platform called Has A Calender that lets anyone have the same wiki calendar platform for their own group or interest. This has driven lots of improvements, and I hope to launch a new version of Open Tech Calendar in a month or so (Late Dec 2013).

I also want to encourage the tech community as much as possible, and especially those who don’t normally feel part of the tech community. If anyone wants to talk about tech events or how we can build the local community for everyone please get in touch – I’m happy to talk to anyone about what we can all do to support each other.

So thanks to everyone at TechMeetup – I learnt a lot in my time there and had fun, and I look forwards to a bright future for TechMeetup and the tech community in general.