Best Practices in Prototyping

When evaluating and scaling machine learning systems, managers are faced with constraints: people, technology, budget, and time. To profoundly impact the organization, managers must balance the need to enable data science experimentation with the realities that business value is usually only captured after models are deployed to production and integrated into business processes.

If the prototyping phase is mismanaged or cut short, immature models can succumb to real-world complexities and be rejected by business teams and end users. If the experimentation phase is allowed to drag on, the business burns through precious budget and wastes time on “science experiments.”

This chapter focuses on best practices for the prototyping phase: how to set it up for success, what to watch out for, and how to know when a model is good enough.

Each organization will vary in its capacity to provide leeway for data science teams to identify solutions for complex business problems. However, it is universal that rapid demonstration of success and value generation is key to ongoing funding and resourcing of AI use cases.

In our ten years deploying AI systems at C3 AI, we have seen a common pattern across business teams. Most teams are interested in the capabilities of AI/ML and are looking to verify the potential for algorithms to demonstrate operationalizable business value, usually within 8 to 16 weeks or, at most, 24 weeks. Following this, business teams either decide to double-down and operationalize the AI capabilities, or move on to different problems.

In order to demonstrate the value of AI/ML, and to accelerate adoption and ensure future investments, we recommend carefully managing the scope of the problem based on two criteria:

1. Reduce the scope of initial work to create boundaries around the problem so that it can be solved in a short period of time, usually no more than 8 to 16 weeks, while at the same time ensuring sufficient complexity to convince decision makers of the algorithms’ benefits.
2. Ensure that if the initial work is successful, it can be rapidly transitioned into production to unlock significant economic value.

The following figure demonstrates a typical timeline for a small team to start and complete the experimentation phase of AI use case development.

Note that managers should ensure that prototype efforts are conducted with a view towards a rapid transition to production with minimal additional effort, and the corresponding ability to capture significant economic value.

A good rule of thumb is to seek to operationalize into production an AI/ML prototype within six months of the prototype effort’s start. This focus on value and time to production can greatly accelerate the organization’s excitement and interest around AI/ML and its ability to scale up its AI digital transformation efforts.