Sorry to tell you this but your epistemology is showing

We're still a young industry. We've been building bridges for a few thousand years and we've only been building software products for a few decades. We've discovered some hard and fast rules for building better software products but much of the wisdom we've acquired now gets bandied about in the form of maxims, many with a kernel of deep truth, but passed around without context.

Engineers throw around phrases like YAGNI ("you ain't gonna need it") and "premature abstraction is the root of all evil" in well-intentioned attempts to prevent their colleagues from indulging in what we've commonly branded "over-engineering." Product managers champion "ruthless prioritization," justifying a compelling minimalism, forever under threat from the siren song of "more."

These ideas are important and stem from deep, fundamental truths about the nature of the products we build.

But, as with everything in our industry, context matters. A command--any command--, divorced from its purpose, is a dangerous thing. That's why so much of a product manager's time is spent understanding and then building empathy for the real problems users face and the actual jobs they seek to accomplish with the product the team is building. That's why--often--the majority of an engineering manager's job is spent reinforcing the purpose of the team's work, connecting the code to the outcome.

"Why" matters.

So, let's examine the why behind these minimalist maxims. What truth about our products underlies them and what determines their applicability? When and why aren't you gonna need it and when might you? When and why should you abstract and when should you solve only the immediate problem? When and why should you "ruthlessly" prioritize and when should you pursue the mythical p3, only so named because we had so many p1s we decided to make a new p0 category?

The answer lies in epistemology.

But first, let's agree on what our job is as a member of a product team. Our first instinct might be to say that we are solving user problems and enabling users to more efficiently accomplish the jobs they have to do. Unfortunately, the most direct way to accomplish that is not through software; it's by sitting down with them as their assistant. Our actual job then involves solving those user problems at scale, more efficiently than can be done by assigning more and more smarter and smarter humans to each customer. How then are we to accomplish that? By building in malleable software a model of the domain that our users work in, allowing them to easily map domain-level concepts to product-level concepts and domain level operations to product-level operations.

That's it. That's the primary job of every member of a product team: product manager, engineer, designer, copywriter, etc. Build a coherent model of the domain your customer is working in and allow them to easily map the things they want to do in the real world onto your representation of it, to manipulate that representation of the thing they actually care about, and then to map back from your representation into the real world.

If we do that well, our users don't even realize they're doing this mapping. They see a file and they drag it into a folder because that perfectly matches their mental model of what they want to do (for some version of a 1980s-era customer who still knew what a file folder was and once handled physical pieces of trees with writing on them).

That may sound easy but it is likely the hardest thing humans have attempted to do.

We've been trying to categorize things and create ontologies for our world for about as long as we've been using language.

And exactly what rules govern that process and how best to approach it is still an unresolved question but the basis for any answer lies in how we understand reality, how we understand what’s true and what isn’t and why.

Epistemology is our attempt at answering the question: what's true and how do you know it's true?

That’s our route to improving the way we build and evaluate our models of the world.

So let's get back to our favorite product-building aphorisms.

Historically, everything was built in a full waterfall model with a big design up front and implementation at the end. This reflects a perspective of epistemological certainty, a belief that everything is knowable and prestatable.

The maxims we mentioned at the beginning (ruthless prioritization, YAGNI, premature abstraction) all embody a reaction against the obvious absurdity of total epistemological certainty; they reflect a complete embrace of epistemic uncertainty, of the idea that the world is inherently or effectively un-modelable, un-prestatable, and--to a large extent--un-knowable.

Unfortunately, both worldviews are wrong in their absolutism. The obvious reality for anyone who has spent much time observing the world is that some things are knowable and prestatable and some things are not. And some things are knowable in principle but we just don’t know them yet.

Now, this does not imply that each category of thing is partially knowable. The truth rarely lies in the middle. It means just what we said: some things are knowable and some things are not knowable. In some areas, you can design up front and get it right and in other areas, you just have to iteratively unfold your product, letting it co-evolve with your user base.

The correct epistemic stance--and, therefore, the correct product development methodology--depends on the domain you're modeling.

Many things in the domains we care about have natural algebras, a set of objects and expected relationships between them. A collection of things, for example, can be set-like or list-like and we know quite well what types of things people do with set-like or list-like objects. To pretend that you know you want an add operation for a set-like thing (like user permissions) but that we can't possibly tell in advance if a delete operation is necessary or what its priority is doesn’t quite pass the smell test after millennia of human experience working with sets of things. There is regularity in nature in the way that objects relate to each other and that regularity underlies the domains we care about. It's our responsibility as product builders to recognize the true nature of the objects we're representing and to expose it to our customers. In these domains, missing entities, relations, or operations just confuses and frustrates users and inevitably slows down our ability to improve our product. Working around these missing pieces even distorts the trajectory that our product takes over time.

But not everything has an obvious, prestatable algebra that relates it to the other entities in our products. In fact, our products as a whole relate in un-prestatable ways to other products, our customers' environments, and the rest of the world. They are part of a co-evolving, complex adaptive system, where interactions between parts constantly create novel niches and affordances that other parts can inhabit and make use of. It is not just that we don’t currently know how our product will need to evolve in this dynamic environment but it is likely even in principle impossible to know the specifics of how our product will need to adapt over time. Here, there is no absolute knowledge. The exact details of the way that these systems unfold are unknowable beforehand; they are computationally irreducible so the only way to know where you will end up is to play every step of the game between now and then.

But even for these aspects of our domains, even when the details of how our product needs to work in the future cannot be known, we need not completely give up design and abstraction.

We know more than we might think.

We know that our product will need to evolve and we know that interactions between concepts and parts of our systems create constraints that make it more difficult to evolve the system. We know that the more closely the relations between the components of our system match the relations between the concepts we know about in our domain, the more easily we will be able to model new domain concepts in our product.

Evolvability does not arise as a natural consequence of taking small steps or building sprint by sprint. Evolvability is not a property that derives from how something is built. Evolvability is a property of the built thing; it derives from the structure of the relations between the parts of our product and it can be explicitly designed into our products.

This is why understanding the deeper reasoning behind the agile maxims is important. If you happen to be working on a part of a product where the epistemic premises for working in an agile fashion are met, you are also working on a part of a product where the most important trait of your system is evolvability.

Many take these maxims to mean that we should just directly build concrete solutions for individual user stories, one after the other. Instead, we think the answer is to build an evolvable product, where the concepts you know about are properly represented (reified), with hard interfaces between components, and with abstractions introduced to keep interactions between components to a minimum. Abstraction and structure become more important when building evolvable systems, not less. If you leave a domain concept that you've uncovered un-represented in the product, it becomes harder to operate with agility, not easier.

Ruthlessly prioritize to maximize your rate of learning, not as an end in and of itself. Abstract as you learn and as the true nature of things is uncovered but not before. Consider whether there's a natural relation between the thing you're told you ain't gonna need and the rest of your product before you throw it away.

Why is this on our mind at Simply Stated these days?

Because every product we know of is composed of user flows. The user flow is the abstraction that ties together all of the parts of the product from the customer perspective. The user flow is one of the primary things that product teams work on and that customers experience.

And there are almost no product teams that can point to their user flow. There's almost never a place in the system that encodes the user flow. And that's a problem.

That's a problem that we're aiming to help fix.

Because the user flow is where the action is. That’s the part of your system whose details will evolve most rapidly, where agility commands the greatest premium.

We have no idea whether the specifics of particular steps of your user flow are in the knowable or unknowable category but we’re fairly certain that the coarse grain relationships between those steps is eminently knowable.

And we think that making user flows real and tangible will help all of us build more evolvable, adaptable products.

Stay tuned for more.

The storyteller makes no choice
Soon you will not hear his voice
His job is to shed light
And not to master
- Terrapin Station, Grateful Dead