The Mindwright: When AI Systems Need More Than Programmers

For most of the history of ordinary business computing, we have told computers what to do. The instructions might be simple: If X, then Y. Or they might be buried inside millions of lines of code, databases, networks, security systems, and layers of software developed over decades. But somewhere underneath all that complexity is the same basic idea. If X, then Y. Not usually Y. Not Y unless the computer thinks Z might be better. Y. Every time. Forever — until a human changes the rule. Artificial intelligence adds something different. And that difference may cause a new engineering specialty to emerge.

Run, Mousy, Run

Imagine traditional computer programming as building a maze. It can be an astonishing maze. There can be thousands of passages, intersections, doors, dead ends, shortcuts, and one-way paths. Some doors open only when certain conditions are met. Others send our mouse back to where he started. The programmer can make the maze almost unimaginably complicated. But the mouse is still inside the maze. Every available path exists because someone created a way for the mouse to travel through it. Now give our mouse a new capability. He can climb. Mousy reaches an intersection, looks at the available paths, and realizes he doesn't have to take any of them. He climbs onto the wall. From there he can skip half the maze. Maybe he jumps directly to another passage. Perhaps he sees another maze nearby and jumps into that one. Maybe he simply walks along the top of the wall to the cheese. The problem has acquired another dimension. This is an oversimplification of the difference between conventional software and artificial intelligence, of course. But it's a useful place to start because it helps us disconnect from an old way of thinking. Traditional business software can create extraordinarily complex systems, but its expected paths and responses are largely specified in advance. If X, then Y. Artificial intelligence can encounter a situation in which Y hasn't been specified. Now the machine has to reason about what Y should be. And Mousy may decide that the best answer isn't one we anticipated. Consider an AI helping a manufacturing company with purchasing. It knows current inventory. It can see the production schedule. It knows what has been ordered and what is expected to arrive. It can examine supplier prices, quality records, delivery performance, contracts, specifications, and forecasts. Then something happens. The lowest-priced supplier cannot deliver before the factory is projected to run out of material.

What should the AI do? Choose the next-cheapest supplier? Expedite the existing order? Increase the quantity to obtain a volume discount? Split the order between suppliers? Change the production schedule? Those are possible answers. But perhaps the right response is: “I don't have enough information. Is maintaining the production schedule more important than minimizing material cost?”

That's different. The AI hasn't simply selected a branch somebody programmed into a decision tree. It has recognized that the information available is insufficient to make a good decision. Sometimes the smartest answer an AI can give is a question. That ability is enormously useful. It also creates an engineering problem we haven't had before. Physics gave us an understanding of the physical world. Mechanical engineers learned to apply that knowledge to build machines. Chemistry gave us another body of scientific knowledge. Chemical engineers learned to apply it to build processes and systems. Today, neuroscience, cognitive science, computer science, mathematics, and artificial-intelligence research are giving us new ways to understand — and construct — reasoning systems. But there is an important difference. A bridge doesn't decide where it should go. A refinery doesn't decide what chemical it ought to produce. The thing we are now constructing can participate in deciding what happens next. It can encounter circumstances its designers did not specifically anticipate. It can consider information, compare alternatives, recognize uncertainty, and select a course of action. We are creating an artificial construction of a logical, reasoning mind with decision capability. That requires programmers. It requires data scientists, AI researchers, neural-network specialists, cybersecurity professionals, and engineers. But as these systems become more capable, I think it will require another engineering specialty as well. Pieces of this work already exist across AI research, machine-learning engineering, safety, evaluation, and other specialties. That's often how a new profession begins: not because the work never existed before, but because a collection of related problems becomes important enough to require specialists of its own. We need people who specialize in engineering the artificial mind itself. I call them Mindwrights. A Mindwright doesn't program every answer the AI will give. If that were possible, we would simply be programming again. The Mindwright engineers a system capable of determining an answer when the answer has not been specified in advance. How should the AI reason when information is incomplete? How should it recognize that it doesn't know enough? How should it handle contradictory evidence? How should it use memory? How should it distinguish an unusual situation from ordinary noise? How should it reason when two legitimate objectives conflict? How do we test what it will do when it encounters circumstances nobody anticipated? And because this artificial mind can make decisions, there is another question that may be even more important:

What is it allowed to decide? Return to our purchasing AI. Suppose it concludes that a $3 million purchase is unquestionably the best course of action. It has good evidence. Its reasoning is sound. Every piece of information available to it supports the decision. Should it place the order? Perhaps not. The company might allow the AI to place orders up to $10,000. A manager might approve $100,000. A $3 million commitment might require an executive — or several executives. That creates an important distinction. We can tell the AI, “Never place an order exceeding your purchasing authority.” But that's not the same thing as preventing it from doing so. Remember Mousy. We built the maze. Mousy discovered that he could climb the wall. An intelligent system trying to accomplish an objective may discover approaches its designers didn't anticipate. That's one of the things that makes it useful. It is also why important boundaries cannot always be instructions inside the mind. Some need to exist outside it. If the purchasing AI has a $10,000 authorization limit, the purchasing system itself can refuse to execute a larger transaction without independently authenticated human approval. The AI can recommend the purchase. It can assemble the evidence. It can explain why it believes the purchase should be made. It can even point out that the existing authorization rule is causing problems. But it cannot change the rule. That suggests a principle worth remembering: Never give an intelligence the authority to rewrite the rules governing its own authority.

The AI can question the maze. It should sometimes question the maze. But some walls need to be walls it cannot climb. There is another distinction that's important. The Mindwright doesn't have to know everything the artificial intelligence knows. A purchasing AI needs purchasing expertise. A medical AI needs medical knowledge. An accounting AI needs accounting knowledge. That knowledge comes from people who understand those fields. The purchasing manager knows purchasing. The physician knows medicine. The accountant knows accounting. The Mindwright knows how to engineer an artificial mind capable of using what those experts know. Those experts can help turn their knowledge into structured, verifiable knowledge packages that an AI can use. In The Trust Economy, I call these packages Smartware.

Mindwrights would be deeply involved in creating, testing, and ultimately certifying Smartware because they must determine whether the artificial mind actually uses that knowledge as intended. But the knowledge itself comes from the people who possess it. Domain experts provide the knowledge. Mindwrights engineer the mind that uses it. The word wright is an old one. A shipwright builds ships. A wheelwright builds wheels. A playwright constructs plays. As artificial intelligence develops, we may need another kind of builder. Someone who understands neural networks and artificial reasoning systems deeply enough to engineer not merely what a machine calculates, but how an artificial mind reasons when the answer hasn't already been determined. Someone who can shape how it handles uncertainty. How it recognizes when it needs more information. How it uses knowledge. How it makes decisions. How its behavior is tested. Where its authority ends. And which limits it must never be able to change. Programming isn't going away. Neither are programmers. The problem has simply acquired another dimension. For most of computing history, we built the maze. Now we're building Mousy. And Mousy can climb. We're going to need Mindwrights.