Bank statements, credit card statements, and tax forms all contain valuable data, but it's trapped on paper and in PDFs. We humans recognize the ink patters them as letters, but they contain no instructions for the computer. Optical Character Recognition (OCR) is how machines learn to read. We explore the mechanics of OCR - the scale of the paper problem in financial services and why paper-based data is so difficult for computers to extract. We look at how accuracy statistics for machines can be misleading and why that results in people - lots of people - staying involved in the digitization process.
.png)
Hey friends -
My apologies for this letter not being published on the regular Wednesday morning schedule. While others are enjoying 20 Days of Christmas, I'm experiencing 20 Days of Jury Duty. It’s more time-consuming than I had anticipated. Please forgive me if other letters this month are also later than usual.
Make sure to check out last week’s podcast! We revisit a topic from a previous letter - benchmarking the size and growth of the crypto market relative to GDP against oil, gold, and stocks. The data doesn't lie - crypto is truly in a league of its own!
This week we're tackling a topic near and dear to my heart, deep in the bowels of financial services where I've spent significant time. It’s the dirty secret of fintech: for all the talk about technology, there is a truly staggering number of people who are involved in highly manual tasks. Nowhere is this more true than when the data is stuck on paper but the machines that analyze it need it to be digitized.
In this week's letter:
Total read time: 14 minutes, 41 seconds.
Imagine you and I wanted to launch a new financial services startup that will loan money to consumers. One of the most basic - and most critical - challenges will be our ability to determine if the loan applicants will actually pay back the loans.
The beauty of technology is that it can crunch through lots and lots of data. Our startup's data-crunching machine will determine if the applicants will pay back their loans by analyzing data like individuals' income, spending habits, and whether they pay their bills on time. But first, we have to get ahold of all this data. That's where stuff gets tricky.
The data doesn't actually exist.
At least not in such a way that our data-crunching machine can make use of. It mostly lives on pieces of paper like printed-out bank and credit card statements. Paper and pictures contain analog data. Machines understand digital data. Something has to translate from paper to digital.
Welcome to the world of Optical Character Recognition.
The fundamental challenge with a piece of paper or a "digital" PDF is that those ink marks we recognize as letters contain no instructions for the computer. The computer quite literally doesn't know how to read them.
This problem is extremely common across financial services because of the volume of paperwork. Just think about how much paperwork you deal with across your credit card statements, bank statements, and tax forms. When you multiple that by millions of people, it becomes a truly staggering amount of paper. A bank I previously worked with explained that they spend over $20 million per year on printer ink alone.
The basics of every computer system used to "read" the paper are the same. The systems stores ink blotch patterns in its memory. Those patterns are associated with letters and words. Each pattern is just a collection of pixels, such as knowing that a pattern of pixels laid out with a horizontal line atop a vertical line is probably a "T".
The systems also have a scanner-esque component that takes pictures of the documents and compares the ink blotches to what it has stored in its memory. When it finds matches - like for the letter "T" - it records the match. It's this process that gives rise to the name for these technologies - optical character recognition - or OCR for short.
The challenge of using technology to convert data on paper to digital breaks down into two considerations:
Document content can be broken down again. It's one challenge to extract the data, such as converting a handwritten note to digital words. It's often a different challenge for the computer to "understand" the digitized data. For instance, if a form field labeled First Name is filled in with "Jared," then the computer should store both the value "Jared" and denote that it is the First Name. We call the latter metadata - additional data that describes the data we care about.
Metadata can become voluminous very quickly. Recording that Amazon Web Services always appears capitalized as a triplet is just the start. We might also record its place in the sentence, which paragraph, and the location on the page. In the case of Amazon Web Services, such metadata helps the machine differentiate between the capitalized brand name "Amazon Web Services" and web services in the Amazon. Digital storage is cheap and metadata doesn't take up much room, so we generally try to gather as much metadata as possible and figure out if it's useful later.
Different types of paper documents present different challenges. Let's look at a couple of examples.
Structured text, like forms, is notoriously difficult for humans. We're bad at understanding where to fill in answers. Just look at the signatures from Japan's surrender at the end of World War II.
Canada's representative signed in the wrong place! It created a hilarious ripple as each subsequent representative had to amend the document so they too could sign.
While such forms are complex for humans, they're more straightforward for computers.
Forms like the I-9 come with a massive built-in advantage for computers - the form structure is the same, every time regardless of the answers to the questions. Where there is consistency, we can build rules so the machines can understand the document structure:
To help the machine extract data, we can divide the page into a Cartesian plane (X + Y axes). We'll label coordinates such as (10,14) and program the machine to record the ink blotches found there as Last Name (Family Name). This works even if the page was scanned in crooked. We can use markers - like the seal in the upper left and the page number in the bottom right - to help the computer reorient as needed.
It gets more complicated when we don't know what to expect.
Unstructured text is relatively easy for us humans to understand but devilishly difficult for computers, the reverse of structured data. Regulations are good examples of unstructured text and are the problem I spent significant time solving.
As people, a couple of things are immediately apparent:
That's a lot of structural elements, but it is by no means an exhaustive list. We could have additional structural elements like sub-bullets, numbered lists, and block quotes as well. All of this needs to be programmed into the computer.
Unlike with a form, the computer also needs to "understand" the structure without the aid of a pre-built graph to extract content. Content starts with the basics - recognizing the shape of the letters, punctuation, and symbols such as §. It gets more complex as we want the system to error-check and make sure what it "thinks" it sees are actual words. There's as much art as science here. While a dictionary can help, acronyms like TILA and formal names like Regulation Z require a more complex understanding of the content.
This really becomes a pain with tables.
Tables don't come with "how to read" instructions. The first row may be a header to data displayed in rows, or, the first column may be a header to data displayed in columns. Or, we might even have a combination of the two!
This is a good example of a table that can appear straightforward in both structure and content but be a nightmare. Some of the potential challenges include:
While we can build computers to understand this table, imagine if this was included in a document with many other tables each of which had its own nuances.
Or imagine if this was part of a more complex document.
When you combine all these problems in a single document, they become very difficult to solve.
Here you have a one-off form that's specific to a single company. It includes a freeform address section, fill-in-the-blanks that appear on the left of the text and in the middle of sentences, booleans structured as paragraphs... in short, a mess. This will be a real pain to have a computer understand.
But there's always an alternative - people.
Rather than use technology, a company can pay rooms full of people to read the documents and retype the answers into databases. Most firms outsource this to specialist companies based in India.
The specialists usually charge by the page for standardized forms or by kilo-character (per 1000 characters) for unstructured text. There are additional charges for accuracy, such as a contract that guarantees 99.9% accuracy and includes an optional upcharge to get 99.999% accuracy. Keep in mind here - while 99.999% accuracy sounds impressive, it means you should expect a mistake every 10,000 characters. That's a mistake every 3.5 pages!
How do we get more accuracy in a people-based system? More people! In the banking compliance world, we call this checkers checking checkers checking checkers. The more layers of review, the higher the degree of accuracy that can be guaranteed.
These layers of manual review and a pricing model based on volume create incentives for the specialists to use machines and automate people out of the process. But it's not just the traditional startups anymore - startups have also woken up to the opportunity.
The technologies these startups have built have varying degrees of smarts. Those smarts typically vary along the same document structure and document content challenges highlighted earlier. Smarter systems are not always better - it depends on the problem we're trying to solve.
The most simple systems scan documents regardless of layout and pull out all of the known words. Any contiguous group of letters that is not found in the machine's dictionary is flagged for manual review. If a letter overlaps with a line on the page and can't be accurately deciphered, it'll be flagged.
Such technology is regularly used in trade finance, the financial services business where global trade is insured and sellers are assured they'll be paid. A bill of lading, like the one above, is a key document that describes the contents being shipped.
Trade finance can be highly risky. There is financial risk alongside a world of sanctions and tariffs to navigate. You don't want a system that's 99.999% accurate - you need 100% accuracy, every time.
The technology used is deliberately brittle - most of the words it flags as "not found in the dictionary" are false positives. At the bottom of the page, you'll notice the signature covers the text "CANADA INC. AS AGENTS FOR." While we as humans can decipher the text, the technology will flag it for manual review. This isn't poor design - it's exactly what the system is designed to do. The financial services company wants a person to review anything with even small probabilities of being incorrectly read by the machine.
One trade finance bank I worked with reviews hundreds of thousands of pages of trade documentation per month. Their system not only detects words and names but also compares the content to terrorist watch lists. In a single month, the system flagged over 10,000 "hits" from the OCR system that had to be manually reviewed. Only 3 were accurate.
Most uses of OCR don't require the accuracy of trade finance. If you're trying to digitize a book, for instance, you can generally tolerate more mistakes.
I searched for tortoise in Google's digitization of The Tortoise and the Hare, and their technology successfully identified the word. With older books or when pages are blurry, the technology will have a more difficult time.
Artificial intelligence has helped make great strides here. The challenge is that we need training data to teach the machine. Training data is the formal term for a straightforward concept - if the machine is going to learn how to identify tortoise correctly every time, it needs a lot of examples of tortoise in different fonts, in varying degrees of blurriness, and so forth. The machine also needs to be told - manually - that each of the examples is tortoise. With that training dataset, it can use artificial intelligence to "guess" at new incoming scans of tortoise that are different than the training data, but close enough that it can recognize the word with high degrees of precision.
How does Google get the training data? Well, you know those obnoxious "prove you're not a robot by guessing the letters" we fill out online?
You're creating training data for Google which they feed back into their OCR technology so it does a better job recognizing words! The new ones involving pictures ("identity all of the stop signs") are similar but for Google Maps.
In the case of forms where we know where to look for the data, humans will annotate a sample form to instruct the system from where on the page to extract data. Annotating is quite similar to highlighting. Think drag-and-drop boxes and a form field where you input "this box is labeled First Name."
Most OCR technology has a rules-based understanding of document structure - they follow exactly what the annotations indicate. Some cutting-edge technologies use artificial intelligence to "learn" how to read forms they have never seen before. This can work well with forms that vary slightly but have predictable content like bank statements.
Even if we gathered ten more bank statements, they will look somewhat similar. Each will have a similar layout that includes the account owner's name, their address, the period, and a table of the account activity during that period. Once humans have annotated a broad sample of bank statements to tell the technology what to look for, the technology can often "guess" at how to read a new statement with a novel layout.
This is why I call it fintech's dirty secret. In every example, people are intimately involved in translating analog paper and PDFs to digital data. So much for technology.
We have not yet invented a system that can do the job unaided. Once trained, these systems can analyze truly staggering quantities of data - Google alone has digitized over 25 million books. But all of the systems are staggeringly inaccurate, and none are general-purpose for all types of paper. Accuracy and adapting to novel document structure continues to be firmly a people problem.
And yet, even with these limitations, these OCR startups are extremely well-positioned to build valuable businesses that will stand the test of time.
Just think for a moment about some of that data that's collected. A startup that's digitizing bank statements has data on all of the money moving in and out of someone's bank account. If the startup's analyzing credit card statements as well, they understand daily purchasing decisions. Add in tax forms, and you have an almost complete picture of most individuals' finances.
That's a powerful position to go beyond just digitizing data. An OCR startup can build a business by analyzing the data to create new, novel insights.
It’s where we’ll go next week!
"To be taken before 11AM, or whenever steam or energy is needed"
1.5oz Cognac
0.5oz Laird's Bonded Apple Brandy
1.0oz Dolin Sweet Vermouth
Maraschino cherry for garnish
Pour all of the ingredients, except for the cherry, into a mixing glass. Add ice until it comes up over the top of the liquid. Stir for 20 seconds (~50 stirs) until the outside of the glass is frosted. Strain into a coupe glass. Garnish with the cherry and enjoy!
The morning drinking instructions come from the much renowned Savoy Cocktail Book published in 1930. As the name implies, the Corpse Reviver #1 is part of a family of drinks designed to be had the morning after drinking too much. While I've made the most famous of the bunch - the Corpse Reviver #2 - many times, this one somehow escaped me. It ended up being a lovely drink for a cool fall evening. All three liquors are distinctive fruity but none overly sweet which creates a mellowness that's surprising given the alcohol-only make up. While I'm unable to recommend this as a breakfast drink, I thoroughly enjoyed it as an aperitif.
Cheers,
Jared
