BLOG · 01

Why most businesses fail to get a real return from AI and how to fix it.

A client had given every employee a premium Claude Business account, plus another $1,000 in token allowances. The intention was good. They wanted people to have access and enough room to experiment.

What they built was a mess.

People were vibe coding dashboards that showed the same information already available in a saved ClickUp view. Others had agents watching outstanding tasks so they could send yet another reminder into Slack. Some agents ran for two hours at a time, digging through months of closed tasks to produce summaries nobody read.

Creative staff were building QA skills for things like safe-zone checks and audio checks. Claude told them these checks were possible, but the skills had no reliable tools for inspecting the actual files. Millions of tokens were burned, and the team still had to perform the same human QA afterwards.

The business employed roughly 100 people and was spending upwards of $30,000 per month on inference. That figure did not include the time people spent building and maintaining tools they or somebody else had created. API keys were flying around. Context was scattered across personal agents, and several people had built different versions of the same thing.

They were paying a fortune for AI and could not point to a single number it had improved.

Subscriptions have their place. I think every employee should have access to a capable model and room to play with it. That is how people learn what it can actually do. Experiments still need a boundary. A dashboard somebody built on a Friday afternoon should not end up running a business process just because it looked good in a demo.

One thing I have noticed in companies that get this right is that chatbot use often goes down. Staff stop prompting a personal agent for every little task because the useful parts have been built into the systems they already work in. The AI does its work in the background and people get on with their jobs.

AI won't fix a bad process

A bad process is a bad process no matter how much AI you throw at it. In many cases, throwing AI over the existing pile of crap makes it more complicated and more expensive.

This was happening long before generative AI. A medical-products manufacturer, anonymised as MacroMed in the research, bought computer-integrated manufacturing equipment because it wanted a more flexible factory. The workers carried on following the old rule of avoiding product changeovers. The new machines were flexible enough that the team managed to run them much like the old ones, and the factory kept building large amounts of work-in-progress inventory. [1]

MacroMed had bought flexible machinery and then used it under rules designed for the old factory. It only began getting the return it wanted after changing how the factory operated around the machinery.

Sometimes the existing process was perfectly reasonable before automation entered the picture. Automation changes the cost and limitations of individual steps, so the same sequence may no longer make sense.

The physical work is also often a tiny part of the total time it takes to finish a process. A 2018 study followed requests for medical equipment at a children's healthcare centre. The median request took 50 days from beginning to end, while the actual processing time was 14 minutes. Approximately 99% of the elapsed time was spent waiting for the next process to begin. Staff changed the workflow and clarified ownership. The median lead time fell to three days. [2]

Making those 14 minutes twice as fast would have achieved almost nothing. Almost all the delay was in the waiting between steps.

Bad culture and poor leadership

If the leadership and culture within your business suck, no amount of AI is going to fix it. Teams need to share an outcome, and incentives need to reflect the performance of the business as a whole. A culture that rewards departments for protecting their territory makes cross-functional automation painful because a useful process rarely stays inside one team.

Leadership needs enough respect and political will to change the way work happens. Teams may have to be restructured and processes that have existed for years may disappear. Responsibility will move with them. Whoever is in charge of the transition needs the authority to make those decisions and the competence to know when they are required.

Employees need to know that they have a place in the business you are building. If the declared goal is to remove their jobs, they have every reason to protect information and avoid helping. A better objective is to grow the business and increase the amount of work it can handle without increasing headcount at the same rate.

People are far more likely to contribute useful ideas when they can see how they fit into and benefit from the future of the business.

Process discovery happens from the bottom up

The people doing the work understand what goes into getting it over the line.

A project manager knows how the job boards work and which exceptions require somebody to step in. An accounts person knows how payments arrive, how references get mixed up and which customers tend to pay late.

Much of this information never makes it into the documented procedure. Organizational research draws a useful distinction between the intended routine and the routine people actually perform. [3] Standard operating procedures may describe the intended steps but they should not be confused with what actually gets done.

Engineers need time with the people doing the work. They need to watch the process happen and understand the workarounds people have created. A workaround may reveal poor practice, or it may exist because the official process fails under real conditions.

Operators should be involved in discovery and testing, while production ownership belongs with people who can maintain the system safely. Leadership sets the outcome and gives those owners enough authority to deliver it.

NUMMI is a useful parallel. The factory inherited workers from GM's Fremont plant, where absenteeism had regularly exceeded 20% and quality was among GM's worst. Under a different production and management system, absenteeism reportedly fell to 2%, and the plant went from GM's worst quality to its best within a year. The workforce was largely the same. Workers had been trained to expose problems, and management had created a system that responded when they did. [4]

Senior leadership has to back the work or it gets blocked. The people doing the job need to be part of the design or you build the wrong thing.

Make your tokens work for you

Automation and forward-deployed engineering existed long before agentic AI arrived. Organisations were already building internal software and automations. AI has made bespoke internal tools much quicker and cheaper to build.

That change in economics makes the work viable for smaller businesses. Most of the finished process will still be plain software.

Many processes can be automated with a few scripts triggered by known events. When a clear rule will always produce the expected outcome, ordinary code is enough. The engineer can use AI to build and test the software. The finished code then runs without paying a model to make the same decision every time. There are still hosting and maintenance costs, but you remove repeated inference cost and model variability.

A task status changing does not require an agent to keep checking the board and deciding what to do. A webhook can react to the change. Code can enforce a fixed approval threshold and validate known fields against a schema.

Inference makes sense when the input contains ambiguity. A model might extract key details from a customer's email or match an invoice against an unclear payment reference. A single model call inside a controlled workflow may still be enough. An agent becomes useful when the path needs to change according to what it discovers.

OpenAI's own guide points teams towards agents when the work involves contextual decisions or unstructured information. For clearer use cases, it recommends a deterministic solution. [5]

The exact task matters

AI performance varies wildly between tasks that look similar to us. You have to test the exact job you plan to give it.

A study of 5,179 customer-support agents found that an AI assistant increased issues resolved per hour by 14% on average. Less experienced workers improved by 34%. The tool had been built into a specific support workflow and helped newer staff apply patterns used by stronger agents. [6]

A study of experienced open-source developers working on codebases they already knew well found a very different result. Across 246 real issues, they took 19% longer when AI tools were allowed. Before the study, they expected AI to make them 24% faster. Even after completing the work, they still believed it had made them 20% faster. [7]

The study covered a narrow group: experienced maintainers working in large codebases they already knew well. What stuck with me was the gap between their confidence and what the stopwatch showed.

That is what happened with the creative QA skills. The model could describe the checks and produce plausible instructions. It confidently said the workflow was possible without having a reliable mechanism for performing the inspection. It sounded convincing, but it still could not do the check.

Every production system needs to be tested against real cases. Demos are easy to make look good. Run it on the ugly cases that happen at month end or when a customer sends something nobody expected.

Measure what changed in the business

Usage metrics such as prompt counts and active seats tell you whether people are trying the tools. Token spend tells you what that activity costs. You need separate measures to see whether the business improved.

A Danish study linked surveys from 25,000 workers across 7,000 workplaces to administrative records. AI chatbot adoption was widespread, and workers reported saving time. The researchers found no measurable effect on recorded earnings or working hours in the first two years after ChatGPT launched. Their estimates ruled out average effects larger than 2%. [8]

Earnings and hours are rough measures, and businesses can take time to turn saved effort into money. Fine. I would still be suspicious of usage and self-reported time savings.

Saving somebody ten minutes does not automatically create ten minutes of business value. Another queue may swallow the saving, or the same person may spend it checking the output.

Before building anything, establish the number that should change. It could be cycle time or error rate. Another process may need throughput measured. A commercial team might care about gross margin or revenue per employee. Pick the number the business already cares about.

How we approach it at Derra

This is the process we follow with a new client. If the client does not have the appetite or time horizon for it, it is usually better not to start.

These are very rough timelines and can vary greatly based on company size, competence and willingness to engage by staff, quality of existing data and processes among others.

1. Audit: 1-2 months

We map the difference between the documented procedure and the reality of how the work gets done.

We spend time with each department involved in the process and document the normal path, along with the exceptions that pull people away from it. We find the bottlenecks and measure where work waits. Then we identify the decisions that require human judgement, what is deterministic and the stages where AI inference may help.

We also examine the existing infrastructure and its security, legal and other requirements. Data limitations are documented before we design around information that is unavailable or unsafe to use. Once the process is understood, we establish its current performance and agree on the target KPIs.

You cannot skip this bottom-up work. Building from the SOP alone will give you a system that works until it meets the real business.

2. Architecture and proof of concept: 1-2 months

We evaluate what we found during the audit and decide how the process should work in future.

We decide which steps belong in code and which need a model. Some stay with a person. Where a model is involved, we define what information it needs and how its output will be evaluated. We identify high-risk actions that require approval and make sure permissions are enforced outside the model.

We decide on the correct tech stack, infrastructure, models, map out the workflow execution and design evals with the assistance of operators to measure results.

3. Build: 1-2 months

We build the system around the real tools and data used by the business, with monitoring and audit logs included from the beginning. Operators test it against normal cases and exceptions that usually appear at specific times of the work cycle or when a customer does something unexpected.

We build systems to be provider agnostic, as new models ship we are able to plug new capabilities into the system without big redesigns.

We thoroughly test each automation, running evals and optimizing for efficiency and edge cases.

We will run workflows initially parallel as a shadow of the existing process, reevaluate it multiple times and work with operators until it's perfect. This is an important step, consistency and correctness need to be achieved before a new process can be moved into production.

4. Production cutover: roughly 1 month

The system goes live gradually as we are confident that processes are polished enough, evals still continue, the technology moves fast which still presents opportunity for adjustments and optimisations.

This should be the most fruitful phase of the process where we really start to see ROI.

The KPI agreed during the audit is measured through the cutover. If that number stays where it was, the deployment failed. The agent can run perfectly and still be pointless. Should we uncover parts of the implementation that bear this result we cut them or adjust.

5. Evaluation and maintenance: ongoing

This is the second most fruitful phase, once systems are running full speed in production we can run consistent evals, find shortfalls and improve processes enabling us to optimize for cheaper models and lower overall token consumption.

We want to get the same job done for as little money as possible.

Somebody needs to own the system after launch. Models get updated and integrations break. Evaluations make those changes visible before they damage the process.

What AI-native should mean

The point of becoming AI-native is to do more with the resources you already have and then to scale into that model. The business should be able to serve more of its addressable market without hiring at the same rate. The work should improve customer value or increase sales.

Measure how much the work contributes to the bottom line. If this is not what you're measuring against, then you are already wasting your time and money.

Sources

1. Erik Brynjolfsson and Lorin Hitt, “Beyond Computation: Information Technology, Organizational Transformation and Business Performance”: https://www.aeaweb.org/articles?id=10.1257/jep.14.4.23

2. Fields et al., “Using Lean methodologies to streamline processing of requests for durable medical equipment and supplies for children with complex conditions” (2018): https://pubmed.ncbi.nlm.nih.gov/29246539

3. Greenhalgh et al., “Ethnographic study of ICT-supported collaborative work routines in general practice”: https://ncbi.nlm.nih.gov/pmc/articles/PMC3224237

4. John Shook, “How to Change a Culture: Lessons From NUMMI”: https://www.lean.org/downloads/35.pdf

5. OpenAI, “A practical guide to building agents”: https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents

6. Brynjolfsson, Li and Raymond, “Generative AI at Work”: https://www.nber.org/papers/w31161

7. METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

8. Humlum and Vestergaard, “Still Waters, Rapid Currents”: https://www.nber.org/papers/w33777

Want to see where this applies in your business?