I was on holiday last week and it ended badly with a phone call on Friday afternoon from one of our developers telling me that Clear Books was down.
There is nothing that makes my stomach turn more than unexpected downtime. It’s demoralising for our team and frustrating for our customers.
We are truly sorry to our customers for any disruption caused on Friday afternoon.
Our aim is to provide 100% uptime but things sometimes go wrong. For the record here are our uptime statistics:
- 1 Year: 99.7%
- 6 Months: 99.52%
- This Month: 98.48%
This month, skewed by yesterday, simply isn’t good enough.
Our hosting provider, CatN, had their team of engineers working on the problem immediately and were providing updates on twitter. We also announced the problem on our system status page. There was no data loss or corruption associated with this downtime.
With CatN we will now review the cause and measures we need to take to prevent this happening again and provide an update shortly.
Thank you for your continued support.
@Michael
This is something we definitely need to improve.
With communication in mind the system status site was recently introduced to provide a focused place for updates. However, it’s probably not widely known about at the moment and we didn’t utilise it enough.
We need to get better at passing on updates from our host, catn. Perhaps we should incorporate their twitter feed into our system status blog and/or we could auto retweet their messages in our twitter feed so that the latest developments are passed on immediately.
@Andrew
Thanks for your comments to which we will be better placed to respond once we have had a full debrief.
@Flamefix
We will explore your auto email idea further.
@Anonymous
I don’t think any of our customers would post anonymously so I am not sure who you are or what your agenda is.
For what it’s worth I co-own Clear Books. I don’t work at Fubra (about 8 years ago I did).
Fubra never ‘bought’ Clear Books. They are an original 50% shareholder.
Yes CatN is owned by Fubra, 100%.
For the above reasons we have a very close relationship when it comes to our hosting as Fubra has a vested interest in making sure Clear Books runs smoothly on their CatN platform (it’s not ours).
When Clear Books has upset customers due to unscheduled downtime then so does Fubra and they share our pain. That makes them more resolved to improve and address issues.
We are not tied down to CatN as a host provider. As Andrew Taylor suggests we could move elsewhere, however, there are many benefits to the relationship we have.
My post is reaching out to our customers to apologise for the inconvenience caused yesterday.
@Reece
It’s a fair question but if we were to refund everyone for the 1.52% of downtime this month we would need to payout 8p on the growth plan, 15p on the established and 23p on the premium.
I appreciate your request to be compensated but it would require a lot of effort to put in place for not much gain to anyone. It would also set a precedent for future downtime. We cannot guarantee that there will not be downtime in the future. We can guarantee that we will review what happened and do out best to prevent it happening again.
@Reece
We’ve been talking about this internally and I think I was too hasty in my reply dismissing your suggestion.
We are going to investigate how we can incorporate a refund credit system into our subscription system. Every penny counts after all. More to follow later.
@Flamefix @Steve
The great thing about twitter is it lends itself to rapid updates.
There isn’t much time to organise email campaigns when everyone is focused on solving the problem and getting the application back up and running as soon as possible.
We certainly need to do more to raise awareness about our system status page though so please bookmark http://www.accountingsystemstatus.co.uk/ and visit that any time there is an issue.
We also need to do better at using http://www.accountingsystemstatus.co.uk as a focused area for clear and regular updates.
Given that you are running a SaaS business, you need a proven process for this type of event. Because even with a 99.9% performance, it will keep happening (hopefully just less). Such process needs to include clear communication to customers, and a communications channel which is not caught up in the outage (e.g. your website has been unreachable, let alone the accounting system). So Twitter or other posting mechanism would be good.
I also think you should drop your formal corporate speak down a notch. You have a number of customers who really believe in your product and what you are doing, and your system stands out above other boring tired dated systems. (But at least they seem to run!)
With such users, they will stick with you, support you and assist you in growing the business ….. but only if you play fair with them. Which includes frank information about errors, and not glossed-over corporate speak to excuse and transfer blame.
And I am guessing you need to look seriously at investing in better infrastructure with mirrors, failover and no single point of failure. That’s what SaaS is all about.
Guys,
I’ve posted a brief udpdate here after this morning: http://www.clearbooks.co.uk/2011/10/31/update-on-downtime/
A more detailed update will follow shortly.
Thanks
Tim