How to automate processes with unstructured data

Filippo Nigro

Many people regard processes as "impossible to automate" because some of the workflows require too much human intervention. And in some cases, this is true. But in many cases, this human intervention can be traced back to the presence of unstructured data in the process.

Unstructured data accounts for about 80% of data generated by the average business – emails, presentations, audio, video, and certainly all kinds of documents and images. So if your business has a lot of it, you are in good company.

In the past, companies would respond to the increasing workload by adding more resources to it. Today, we have machine learning to structure data effectively – the emerging backbone of the business process automation.

What is unstructured data?

Since different people have different opinions about it, let's get a bit more specific about what we mean with unstructured data in a business context.

Any data that cannot be stored in a key-value relation and lacks an associated data model is classified as unstructured data. And while many files may possess an internal structure, they are still regarded as unstructured as the data fails to fit neatly in a database.

It doesn't fit into specific formats or sequences nor does it abide by any semantic or rules. Its lack of an easily identifiable structure makes it more difficult to search, manage, and analyze. This requires more specialized skills and tools to utilize effectively.

For instance, a list of music tracks artist names and classified by genre can be classified as structured data wheres the raw sound file itself is unstructured data.

We are talking about success stories for many of the above in other places, so we will not go into greater detail here. But what is important to note is that only when unstructured data is made accessible, searchable, and relevant, it can be used to effectively automate business processes.

Challenges of working with unstructured data

To build upon what you already know, working with unstructured data is not an easy task. That is primarily due to two major reasons:

  1. Unstructured data cannot be integrated into existing information systems...
  2. ... and because of that, traditional automation solutions cannot process unstructured data

To extend on the second point: Technologies like Robotic Process Automation (RPA) are great tools for automating rule-based processes. But unlike the term "Robotic" may imply, these tools are far from interpreting, let alone understanding if unstructured data comes into play. And the same goes for code-based automation as well as integration tools.

Conventional approaches to automation fail when unstructured data is involved

How to overcome these challenges and improve processes with unstructured data? Behold the power of machine learning

As this article (and really our whole website) suggest, machine learning is the answer to about everything. On a more serious note, a word of caution before we go deeper into the matter:

If you can avoid adding machine learning into the process, you probably should!

That is because no matter what tools promise, they usually need to be tuned to your specific business with training data and that can be tedious, even if packed in a nice user interface.

But if you don't, applying machine learning to your processes with unstructured data can bring that cognitive element you need to automate them. So let's jump right in!

Machine Learning technologies such as computer vision (CV) and Natural Language Processing (NLP) are able to understand and classify unstructured data points such as images, text, documents, and audio. This unlocks a huge and previously untapped potential for process automation. By drawing upon large amounts of data, in particular, deep learning can arm your business to take over tasks that are commonly regarded as impossible to automate.

Deep learning is a good solution to get from unstructured to structured data
Machine learning closes the gap and allows the automation of processes involving unstructured data

AI/ML models can interface with your existing applications for processing unstructured data and triggering responses to optimize your workflow. Each ML model is built through a learning phase and an execution phase to resolve specific pain points in your business operation chain.

The user first trains the model to 'observe' how to troubleshoot a particular decision point in your workflow and then replicate it further as necessary, limiting the manual load on your workforce.  Models can also be trained to trigger a human response in case it classifies a certain data input with an accuracy lower than a threshold determined by the user.

Let’s take a quick look at which deep learning models you can train and some services they can provide using your unstructured data:

After you have trained one of these ML models, you just have to integrate it into you workflow.

End-to-end workflow automation

If you’re willing to embrace deep learning as a silent coworker, then it will streamline your workflows like never before. Merging ML models with data sources and automation tools, you can automatically label unstructured-data inputs, and trigger different actions based on the assigned label. Like dominoes in a chain.

Example of an end-to-end workflow with a PDF classifier. See details here.

As you could see, machine learning can enable you to automate tasks with unstructured data. However, deriving true value out of your unstructured datasets depends on how well you approach the task using your available resources while customizing your ML automation tools.

Finding the right approach to cognitive automation

Possibilities to automate processes with AI exist for all types of businesses. Indeed, you don't need a full-blown IT overhaul or trade one precious resource for another to benefit from deep learning.

Your ideal approach to cognitive automation is a classic case of "it depends". As part of our own market research, we identified a few discrete segments along two dimensions: Process complexity and the amount of technical expertise required.

The map below shows the main types of cognitive automation solutions available on the market and we will later touch upon internal routes for the sake of completeness. Knowing where you stand on this map will help you to quickly find the best approach to cognitive automation fitting your needs.

Now that you have an idea of which approach could be the best for you, let's dig deeper into it. Below you can find a more detailed description for each solution illustrated on the map.

1. Self-service tools

Effective self-service platforms providing cognitive workflow automation empower you to deal with the majority of your complex multi-stage processes involving unstructured data. They can be defined as no-code or low-code solutions that allow building customized workflows and machine learning algorithms through a friendly UI. They are usually available for an affordable price point compared to other automation solutions. Being no-code and affordable, the best providers allow for fast experimentation and implementation.

Self-service tools Profile & Examples
Self-service tools Profile & Examples

2. AutoML solutions

Auto ML are platforms that allow you to create machine learning models from scratch, with no or low coding knowledge required. However, while they can help you to give structure to unstructured data, they fail to automate end-to-end workflows since they lack the workflow builder feature.

AutoML solutions - Profile & Examples
AutoML solutions - Profile & Examples

3. Vertical-specific AI automation solutions

These solutions provide intelligent automation tools that are highly specialized within a specific vertical. While they allow handling complex processes with unstructured data, their area of application is limited. In addition, they are usually offered at high price points.

Vertical-specific AI automation solutions - Profile & Examples
Vertical-specific AI automation solutions - Profile & Examples

4. Intelligent RPA

With the development of new machine learning technologies, traditional RPA players are introducing AI capabilities to their process automation solutions. If you are a medium-large company with RPA developers in your team, you might think to go for these solutions.

Intelligent RPA - Profile & Examples
Intelligent RPA - Profile & Examples

5. AI consultancies

They help companies in implementing digital transformation projects. They usually develop tailored intelligent automation solutions for their clients or suggest to adopt existing providers that fit clients' needs.

AI consultancies - Profile & Examples

6. In-house AI experts

The "traditional" approach would be to create a solution using in-house engineers. If you are a company with a deep tech department, your team can probably handle intelligent automation. However, you might need to hire additional developers or shift resources from your core tech activities. If you don't have any ML capabilities yet, be advised that finding the right people can be expensive and time-consuming.

In-house AI experts - Profile & Examples
In-house AI experts - Profile & Examples

Conclusion

While the size of clear unstructured data is bound for exponential expansion, data efficiency is still in its early stages of maturity in the industry. However, technological paradigm shifts are known to come and go as blink-and-you-miss-it events. Intelligent automation seems to be one of them.

According to a study from the International Data Corporation, by the end of 2020, organizations that upgrade their data processing and analysis capabilities with AI will achieve an extra $430 billion in productivity gains.

If data is the new oil then machine learning tools are the refinery you use to extract true value from it – if you pick the one that fits your business well. We hope that this overview could assist you in finding the right tool according to your resources, capabilities, and process complexity.

Share on social media: 

More from our Blog

The value chain of machine learning

The value chain of machine learning consists of more than algorithm programming. Here's an overview of the different players and their roles.

Read story

How to successfully implement AI-powered automation

AI is a powerful tool to automate processes but it requires careful implementation to reap all its benefits. This article guides you on how to do it.

Read story

Why decision fatigue is hurting your productivity and how to prevent it

Feeling stressed by the many decisions you have to take every day? Let’s discover how productivity tools can help you reduce your decision fatigue.

Read story

Stay inspired.

Sign up and get thoughtfully curated content delivered to your inbox.
We will never share your email address with third parties. And we don't like spam either.