Alternative Title: Evaluation Strategy for AI Initiatives

There’s a classic game played in classrooms and corporate retreats: a line of people all face the same direction. The person at the back taps the shoulder of the one in front, turns them around, and performs a simple sequence of actions—maybe a clap, a hop, a funny dance move. The second person then turns the third person around and repeats what they think they saw. This continues until it reaches the person at the front of the line. When the final move is performed, everyone turns around to watch the original person redo the initial actions. Hilarious laughter usually follows.

This is what happens when information travels through interpretation, memory, and context. It’s also what happens in AI initiatives when the status quo is clear at the start, and everyone anticipates there will be improvements by the end of the initiative, but success becomes fuzzier with each layer of execution. The idea that a process can be understood, then improved by inserting technology in just the right place, is compelling—but the moment of “tool meets world” is rarely clean. And with AI, the challenge only grows. The shift from rules-based “app intelligence” to generative, probabilistic “artificial intelligence” makes evaluation harder and more subjective. The additional cost and complexity has to be justified—not just over status quo human-managed processes, but over simpler, digitized ones.

Planning for Success That’s Hard to Define: AI Initiative Mindsets

Many organizations investing in AI aren’t just looking to do what they’ve always done, slightly faster or cheaper. They’re hoping for a step change—a transformation. The tricky part is that transformation, by definition, often lacks clear baselines. This ambiguity makes it difficult to set clear expectations when engaging with the rest of the organization or with senior management. When you discuss what 'success' should look like for a new AI initiative, you may struggle to provide concrete benchmarks.

One way to make sense of this is through a 2x2 matrix based on what you’re doing (a Known vs. Emerging Objective) and how you’re doing it (an Established vs. Innovative Approach). Each quadrant reflects not just a combination of task and technique, but a mindset—how teams define success, navigate ambiguity, and engage with risk.

image.png

The Preservation Mindset

This mindset focuses on using established approaches to maintain stability and reliability for a known objective. AI initiatives here are not about disruption but about reinforcement. A prime example is the use of AI in critical infrastructure monitoring. An electrical utility might deploy an AI that analyzes sensor data from its power grid to predict equipment failures before they happen. The objective is known and constant: "maintain an uptime of 99.999%." The approach of monitoring is established. The AI simply ensures the stability of the existing system, preserving the status quo by preventing outages. Success is measured by the absence of failure.

The Optimization Mindset

The optimization mindset is about applying an innovative approach to a known objective to achieve measurable gains in efficiency. This is where most enterprise AI initiatives currently live. For instance, Amazon uses a sophisticated AI system to optimize the workflow of its warehouse robotics. The objective is known: pick, pack, and ship products as quickly as possible. The AI provides the innovative approach by dynamically routing robots, organizing inventory based on predictive demand, and minimizing human travel time within the facility. Success is clearly defined and measured by metrics like "order fulfillment time" and "cost per package." Similarly, John Deere’s See & Spray technology uses an innovative computer vision model to achieve the known goal of targeting weeds, thus optimizing herbicide usage.

The Experimentation Mindset

This mindset involves applying an established approach to an emerging, less-defined objective to explore new possibilities in a low-risk way. Spotify's "Discover Weekly" is a classic example. The company used an established AI approach—collaborative filtering—that was already being used for recommendations. However, they applied it to an emerging objective: "Could we create a personalized, weekly mixtape that becomes an essential part of our users' lives?" The goal wasn't to optimize an existing process but to experiment with a new product feature to see if it could deepen user engagement. The success of this experiment was measured by adoption rates and its impact on user retention, validating a new way to deliver value.

The Transformation Mindset

The transformation mindset combines an innovative approach with an emerging objective to fundamentally reinvent a product, market, or industry. This is the realm of true disruption. Waymo, Google's self-driving car project, epitomizes this mindset. It paired a highly innovative approach (deep learning systems for real-world navigation) with a visionary emerging objective (creating fully autonomous "Level 5" vehicles). The goal was not to make a better cruise control but to transform the nature of transportation, logistics, and even urban planning. Another key example is OpenAI’s ChatGPT, which used a novel large-scale transformer model to pursue the emerging goal of a universal conversational interface, forever changing how we interact with information and technology. Success in this quadrant is often defined by the creation of entirely new markets and capabilities.

Summary of AI Initiative Mindsets