Selecting the Right AI/ML Problems

Economic Value of the Problem

A second criterion in problem selection involves the economic rationale – the business case – and an analysis of the potential value that could be unlocked if the problem were addressed.

This is often a crucial step for most enterprises because the number of potential applications of AI in an organization is enormous. Most companies are at the very start of their AI business transformations, which means that almost all business processes potentially can be honed using AI.

The following figure shows an illustrative distribution of value and number of AI use cases we typically see at large enterprises.

Usually, however, only a few business cases are likely to result in vastly disproportionate high returns – and it is those business cases that warrant immediate consideration. A rough order of magnitude economic value calculation before embarking on a specific use case can help focus and prioritize efforts.

To determine how much value can be unlocked by machine learning and AI applications, it is particularly important to understand and articulate the baseline performance (e.g., efficacy, efficiency) of the business function that the AI application is seeking to augment. In most business use cases, the baseline performance directly reflects the problem that the company is seeking to address with AI/ML. For example, an equipment operator who wants to use machine learning to predict failures may have a baseline known as “run to failure,” where the operator uses the equipment until it breaks down. It is important to evaluate baseline performance in order to understand the current economic performance (or other performance measures) of the business use case today, as well as the benefits from the application of AI/ML techniques.

Consider an example from the financial services industry. C3 AI Anti-Money-Laundering (AML) is one of the AI/ML applications offered by C3 AI that applies machine learning to identify whether a banking customer is committing money-laundering fraud. At most financial institutions, the baseline financial crimes process draws on a library of rules that flag suspicious client behavior.

The costs of such a baseline system to banks are two-fold. First, financial regulators impose fines and penalties when banks fail to catch money launderers; these fines and penalties drive significant reputational and personal risk to bank executives. Fines totaled more than $8 billion globally in 2019. Second, banks hire thousands of analysts to manually review and investigate potential money laundering cases each year.

In one example, a division of a bank evaluated the baseline operations of their rules-based system. The rules identified 7,300 cases of potential money laundering, but only 33 of those cases were verified by an analyst to be true. Upon later review, the financial institution discovered that there were 110 actual cases of money laundering in the data set, meaning that 87 cases went undetected by the existing rules. As shown in the following figure, the baseline precision of the rules-based system was just 0.5%. And the baseline recall of the rules-based system was 30%.

An AI/ML system that improves on these baseline numbers could add significant value to a bank by both increasing efficiency (smaller staff to review alerts) and effectiveness (be able to catch more money launderers).

You may often hear comments to the effect that an organization will not accept the output of AI/ML systems if the “accuracy” of such systems is not in the “90% range.” There are many problems with this statement, including often a vague definition of accuracy

However, the primary issue with this statement (as in the case above) is that there are many business use cases where the performance of existing rules-based, physics-based, or business-logic-based systems is far from “90%.” And, in most of those business use cases, even modest improvements to the baseline performance numbers of an organization can unlock significant economic, social, and environmental benefits

The right question is not: “Does the AI/ML algorithm reach 90%?” Instead, it should be: “What is the business performance gain that the AI/ML algorithm delivers?” We should also determine whether that performance gain is worth the investment.

Through first evaluating the economic value of a business problem, the machine learning project can be prioritized for development and accurately recognized for the economic value it will create. Demonstrating value helps drive adoption and change management among the end users who will deploy the machine learning program to make data-based decisions.