So what was my point over the last three days? Basically, that the people who will be paying for software still don’t have a very effective way of understanding what they’re asking for and how much it will cost and how long it will take.

If you’re an agile developer, you probably read what I wrote as though I were more critical of agile developers than traditional developers. Conversely, if you’re a traditional developer, you probably read what I wrote as though I were more critical of traditional developers than agile developers. I hope that’s what happened, anyway.

The truth is that I think both ways have a fundamental failure at solving the basic reason many software projects are still over budget or late. We all that that projects, by definition, involve imposing a change in the status quo, and therefore carry risks. But every project carries this risk. And yet many projects are successful, while many others are not as successful, and still others fail outright.

In general, when money is committed to constructing something, there is a plan to recover the cost, plus some profit. Perhaps by selling the thing, or by leasing it. Buildings, cars, watches, beds, traffic lights, whatever. The important part is to make sure that, on average, the costs are kept below the amount of money you’ll demand in exchange for the product. It’s usually not all that difficult to determine what someone would pay for a building, a car, a watch, a bed, or a traffic light. All of those things are commodities. And while creating a valuable label, or brand, for those products helps you charge more (thus allowing you to either profit more or to increase the quality of the product), they rarely allow you to be capricious when setting a price. For example, while The Coca-Cola Company certainly charges far more per can of cola than the pennies it costs to manufacture it, they will still be bound to prices similar to what PepsiCo charge, and to a lesser extent what other beverages manufacturers charge.

Customers know what to expect when they open a can of soda, regardless of the brand. They know what buildings should be able to do, what cars are capable of, the core features of a watch, their requirements for beds, and what constitutes a minimally-functional traffic light.

But software is another matter altogether. You and I, as software developers, know that these beasties can do anything. Anything!! Even the most ambitious types of software developments to date, around artificial intelligence, are not constrained so much by the limits of software as they are by the hardware. Only so much memory can be made available, the CPU requires so much time to process its instructions, the bus can only transfer so much data at a time…. Imagine how much closer we would be to artificial intelligence if we had networks of computers that performed 1 trillion teraFLOPS (amusingly called a yottaFLOPS), and if they had instantaneous access to 1 trillion tebibytes (called a yobibyte)? With that kind of power, it may even be possible to mimic intelligence using a program written in COBOL, although it would undoubtedly take many years to write such a program.

So, the challenge really comes down to this. Every new software creation is a new invention as yet unimagined by the people who want it, of complexity and sophistication which is incomprehensible to most people.

I wager that most people could think of a hundred ways that they would like to improve the tools they use, but only if they were free from the constraints of physics. Imagine a box that held more on the inside than it consumed on the outside. Imagine a battery that lasted for a thousand years. A knife which could cut through a steak but not a even scratch your finger. Now, imagine being asked to estimate the cost and schedule for a project to create something like that. You’d have no idea!

Every software project is a little bit like trying to create what seems like magic. All of software is, in a sense, imaginary. You can’t actually touch software. You can’t weigh it. It can’t wear out with overuse. And it costs nearly the same to make billions of copies of it as it does to make just one copy (if you’re not transmitting or storing those copies inside other things that cost more money per copy). And even when you’re creating software by assembling other software creations, we’re still largely working with tools whose existence is almost entirely within our imaginations, whose capabilities are limited only by our imagination—which is to say, not really limited at all.

So how do we get clear about our imaginations? How do we clearly share our expectations of some imagined solution that itself is created from other solutions which were created from imaginations? And, more importantly, how do we balance our imaginary creations with a very real limit on the ability of developers to fabricate the creation?

I think I’ll spend the next few days on this blog addressing this. I know there are many techniques for getting requirements, but maybe not so many on how to get the right balance of limits on the costs with what is possible in our imaginations and what is possible with the skills of the developers. I don’t know whether I have an answer, but I am very sure it’s a question that must be answered eventually.