Yodlee issues: an explanation

We are as concerned as our users about the repeated Yodlee issues.

I feel that an elaborate explanation is appropriate, explaining why the service is unreliable and perhaps more importantly— why Yodlee will always be unreliable (if it wasn’t — we wouldn’t need it).

I’ll start with our own strategic dilemma at Clear Books and why we’re between a rock and a hard place.

The ubiquitous way for online systems to communicate is by using an API (application programming interface) — a standard protocol allowing them to do so. Clear Books offers an API so other systems (e.g. ChannelGrabber) can interface with it. Clear Books connects to other systems using their API — most notably and importantly for this topic: PayPal.

As at 1st Jan 2014: (and I believe until this very day), not a single bank in the UK offers an API allowing online systems like Clear Books (or Yodlee) to retrieve bank records from their clients or businesses accounts. There are two main reasons for this:

1. Banks are big corporate organisations with complex structures. Nearly all had the bulk of their history long before online systems existed (in stark contrast to Clear Books or PayPal). Like many other big companies whose principal operation is not internet-based, most of their online technology is the outcome of outsourcing and agency staff. They have limited internal expertise and vision when it comes to online technology — the people in charge are often managing executives. They care a great deal for their customers (and pay a fortune to get their online systems to tick), but they have little interest in companies like Clear Books because we’re only a bridge to a small and specialised section of their customers. (Would anyone pick a bank because it offers API and seamless integration with online systems? Would any of you leave bank X for bank, because bank Y has seamless integration with Clear Books?)

1. Banks are a bit security-mad; the very notion of anyone but a human being who is the account holder accessing bank records seems to them like madness. Needless to say that if PayPal offers an API (allowing secure, limited and read-only access), this isn’t as big an issue as banks make it. Ironically, the very fact banks do not offer an API, means their security is, in practice, compromised (although only a little, for the reasons described below).

Had any bank offered an API, Clear Books would interface with that bank directly, cutting the middle man — Yodlee. But no bank does, and that’s where Yodlee comes into play.

Since, like Clear Books, Yodlee can’t use an API, they employ a different strategy called ‘scraping’. Slightly more advanced than the crawlers Google use to feed their search engine, a scraper is a piece of software that simulates the actions of a real person using a web site — so essentially, the scraper pretends to be a human, doing what humans do.

In simple terms, you program it like so: go to Natwest.com; click the login button; enter the customer number in field X; on the next page read the first letter to the left of the first field; if it says ‘4th’ type the 4th digit of the password… and so on. Scrapers are not intelligent, they only do what a human programs them to.

So for scrapers to work, the site they visit needs to be identical each and every time they visit it. It is enough for an online chat message to suddenly pop up, or for the bank to add another step in the login sequence requiring the user to confirm they have read a message, and the scraper can no longer execute the next action, and so it terminates.

To combat this, each scraper logs (for a short period) each page it visits, so if it terminates, a human can look at the last page in the page log, see what has changed and program the scraper to respond to such change in the future (should it happen again).

The problem is that they have no way to test the new program unless the bank site behaves exactly the same way it did the last time the scraper visited it (which is only sometimes the case). In practice, pretty much any change on an online banking site breaks the scraper and requires reprogramming that often cannot be tested.

I can only guess that Yodlee gets thousands of scraper terminations each hour, so no human looks into the issues without a support request. When the scraper terminates, Yodlee fails, and you guys see this in Clear Books as a failed import.

I don’t know about you guys, but my online NatWest business account is inaccessible about 3 out of  every 4 times I try it (but I’m a night owl). When I do manage to login, boy oh boy… their system makes BT phone boxes look modern. You first have to request a statement, then wait an unknown period of time before it shows in some link on the side. It’s the most unusable thing ever. It’s hard enough for a human to use the site — I pity the scraper. Many other banking systems have their occasional glitches and downtimes — all stop the scraper from working.

So any change or a glitch on the bank site means Yodlee doesn’t work. Clear Books integration with Yodlee was hardly ever the source of the real problem (we did, should and will improve where we can, like providing better feedback when possible, but if Yodlee fails on its part, there’s nothing we can do about it). Nor is Yodlee to blame —they look at themselves as an ambitious and brave challenger, and they really do all they can, often as quick as they can, to get things fixed. Nor are the banks to blame — the last thing on their mind is to make their sites scraper-friendly, and if anything, they would surely prefer offering an API than having machines pretending to be humans on their sites (which is less secure than an API solution since a scraper can in theory perform much more than read actions; and since a third party company holds the credentials to your online banking account, which an API wouldn’t. I must emphasise here that Yodlee is highly secure).

As far as Clear Books goes, it is sometimes better not to offer users something than offer something that annoys them. As a user experience expert, Yodlee would be thrown out of the window long ago unless:

1. doing so would mean that we take away a feature that does work sometimes, and people do find useful despite the glitches

1. it wasn’t the only product on the market (all companies similar  to Clear Books get from Yodlee what we do and when Yodlee fails, it fails for them the same as for us)

We are also comfortable with scrapers, as we use them internally to gather data on our own system e.g. for finding which pages have a particular component,  which is much quicker than asking a human to search through all the pages the Clear Books app has.

Trying to replace Yodlee and attempt to scrape online banking sites would only mean we would have to deal with exactly the same problems as them but with less experience. That’s pretty much what Yodlee does —writes and fixes scrapers. Our developers can deliver better value to our customers doing other things. Although cutting the middleman may mean faster fix times, the problems will always be there.

So I hope that explains it.

While we actively investigate if there is anything we can do better, neither we nor Yodlee can make this work seamlessly. If anyone can propose a way to make things better — please please do so. The only thing I can think of that may solve this issue once and for all is if the accounting community in the UK together with Clear Books and similar companies sign a petition and send it to the banks asking for an API.