When kicking off a new project, whether working on an existing code base or a new one, should one rapidly prototype the solution (at the risk of writing throwaway code), or should one apply as many best practices and sound architectural approaches as one knows (at the risk of over-engineering the solution)? Towards which of the two ends of this spectrum, or where between the two ends, does one aim?
This is a question with which I've personally struggled over the years, and I've found that the answer only seems to become clearer when you attempt to answer a deeper question: how much uncertainty is there in your project?
In this article, I'll first elaborate on what I mean by the "rapid prototyping" and "best practices" approaches, why I see them being at two opposing ends of a spectrum of possible ways of solving a problem using software, and then I'll try to unpack that rather abstract question regarding uncertainty a little more so I can hopefully make it a little more practical.
Ultimately, the point of building software is to solve some kind of problem: usually different problems at varying degrees of abstraction. For example, you might be building a game: at the lowest levels, you're solving problems of CPU and memory management and performing tricky mathematical calculations to fool the player into thinking they're looking at a super mutant chasing them through a junk yard, while at a higher level you're attempting to help someone solve the problem of what to do with their free time such that they enjoy themselves the most. All of this in such a way that their enjoyment allows you to put food on your table.
When we get this mix just right at the right levels of abstraction, we say that we're delivering value.
This is somewhat analogous to the process of bridge building - one can think of software development as a process of building a bridge from a situation of little value, to a situation of greater value. When building a bridge, you're also attempting to solve many different kinds of problems at different levels. Where do I put the bridge? How sturdy do I make it? What materials do I use? How do I build it within the budget and time constraints I have?
The approach you take to building a bridge must necessarily be context-dependent, taking into account the local geography and geology, weather, labour force, etc. Ultimately, it must, at a higher level of abstraction, take into account the overall purpose of the bridge: why connect points A and B at all? Why not just go around the canyon, wade through the river, or fly over it? Building a multi-lane tarred bridge with steel and concrete reinforcement wouldn't make sense if one was just building a bridge for preschool children over a tiny puddle of a pond on their playground. Similarly, one wouldn't build a bamboo and rope bridge for long-haul trucks to traverse a canyon.
Assuming you have a clear overall problem you're trying to solve by building this bridge, you may encounter all kinds of uncertainty:
If the situation allowed for it, you may start out by building a prototype bridge. Perhaps you may build a scale model, where the analogy in the software world would be anything ranging from paper-based to clickable wireframes. Perhaps, if you still had too many critical unanswered questions, you may go all the way through to building a rickety version of the final bridge, perhaps in a different location, at a smaller scale, and only accessible by a select few people.
In certain fortunate cases, as I've often found in the software world, building this rickety version (replete with copious quantities of warning signs and disclaimers) can save you a tremendous amount of pain and suffering in the future. It can answer many of your unanswered questions and reduce the risk of failure of the final bridge's construction. If you're extraordinarily lucky, which is more often the case in software development than in bridge building, your prototype may be good enough for your client to use as-is.
The key defining characteristic of a prototype is its brittleness. The most bare definition of a prototype that I can come up with would be something that can or does solve all of the relevant problems, at all of the most critical levels of abstraction, under very strict conditions. It delivers value, but only within those strict conditions. The moment that one attempts to operate it outside of those conditions, it falls apart, and someone is likely to get hurt.
I would argue that, therefore, a prototype could be classified as an extreme point on the one end, as it just barely solves the most important problems with which we're currently concerned. There is, of course, another end of the spectrum.
When building a large bridge requiring a work force of hundreds, intended to carry millions of people for many decades, however, one should take a different approach to building a prototype from the outset. If you know you've got a decent amount of time, budget, a good labour force, great materials, a clear vision and paying customers, it really does often make sense to make use of every best practice you can. The point here is to make sure your final product is as robust as possible in solving all of the important problems at all the relevant levels of abstraction.
This kind of solution is also a valid solution to the relevant set of problems, but its key distinguishing property from a prototype is its robustness. This is the extent to which it can be stressed, at various levels of its solution, and still solve all of the most important problems.
Just as in bridge building, many books have been written on the topic of software architecture that focus on developing the right kinds software abstractions to deal with these various levels of problems, because over the years, many smart people have discovered that there are patterns that tend to emerge when attempting to solve certain kinds of software problems. Just do a quick search on Amazon for books on the topic of "software design patterns" or "software architecture" and you'll see.
When this approach is incorrectly applied, however, one ends up in a situation where the final product is over-engineered: it's built to withstand earthquakes one hundred times stronger than anything the country's ever seen in its recorded history, or built such that it won't need maintenance for a decade, or even built to allow aeroplanes to take off from it. You've wasted millions (or even billions), wasted years, killed your labour force's morale and nobody cares about your bridge any more because Elon Musk's built a Hyperloop right under it that makes it obsolete.
The incorrect over-application of "best practices" can often be seen as premature optimisation, and is seen as one of the cardinal sins of the software development world.
So where do you aim for your software project? I still often struggle with this question, because answering it requires wisdom and careful attention, which can be both challenging and draining. Do you start with a brittle prototype (maybe in the form of a handful of Python scripts), or do engineer the system that can withstand multiple simultaneous AWS data centre outages (i.e. withstanding the end of the world)? As you can imagine, the answer is usually somewhere in between. To know roughly where though, you have to answer the right questions that pertain to the uncertainty you currently face in your project's development.
The following questions are by no means exhaustive, but they tend to give me a good head start.
The first question I personally like to ask is: do you have any real-world end-of-the-value-chain customers that either are already, or have committed to, paying for and using what you're building? If you've already got a guarantee of payment for what you're building, then you should generally be aiming more towards using best practices. If you're working for a company and being paid to implement a new internal system, are you guaranteed that people are going to be using this new system once it's built and working well?
If you don't have a paying customer yet, or your customer doesn't have a paying customer, or there are no end users for your system yet, you have a high level of uncertainty in your project. I've found it's really useful to rather iteratively prototype together with the potential client/user in these sorts of situations. This requires an explicit up-front understanding that what you're building is a prototype (call it an "alpha" or "beta" version if your client is more amenable to that phrasing). At this point, your focus is most likely on "making the sale", or demonstrating real value to the client, end customer or end user.
The times when I've failed to apply this insight, the project inevitably fizzled out and died. One big lesson I've learned through such experiences is that, especially when you're taking a gamble on a new product or system, it's usually wise to choose the route that requires relatively little investment with the possibility of significant pay-off - and if it doesn't work out, your losses should be capped and affordable. This, by the way, is one of the central lessons I took from reading Nassim Taleb's book: Anti-Fragile.
This first question leads us naturally into the second.
Do you know what the end product of your labour will actually look like, practically? Perhaps you have to integrate with data sources where the quality is questionable, but you don't have enough of it to determine just how questionable it is. Perhaps you may be building a system where the client vacillates so often on the front-end design and/or basic features that you need to take prescription medication. Or, perhaps, you don't even have a customer yet, but your product owner is adamant that his/her vision of the product is what the client/customer will definitely want when they see it.
In these sorts of cases, you've got a high level of uncertainty, and you should probably be aiming towards the prototype end of the spectrum. Specifically with the data quality example, I've personally seen how building a prototype here can provide a very effective stop-gap solution while the robust version is being built, increasing understanding of the data's real quality and reducing waste when building the robust version.
What you really want to achieve in this sort of scenario is to allow yourself room to iterate with your relatively brittle solution until you've reached a clearer understanding of what the end product needs to be. More robust solutions, usually having more tests, more layers of abstraction and greater levels of compensation for errors, also require more work, and are therefore more costly, to change.
The third question has to do with uncertainty around the longer-term maintenance of the project: who will be looking after it over time? Perhaps you're only building something that's supposed to last for a few months or a year, like a short-term digital marketing campaign. Or, perhaps you're building a system for a bank, where you know that it's probably going to be running in production for at least a decade.
The brittleness of a prototype, just like a fragile wine glass, requires that one have the right knowledge and expertise to tend to it throughout its lifecycle. This usually means that whoever wrote the code would need to maintain it, as there would usually be hidden fragilities of which others couldn't possibly be aware. Learning about all of these hidden fragilities and the tolerances of these fragilities can be an expensive and painful process.
The more robust the system, however, and the more good architectural practices applied, the easier it will be for another trained developer to pick up development on the system.
I've had several young developers ask me the question: which programming language, or languages, do I learn? This isn't an easy one to answer, because new languages pop up at such an incredible rate these days. Follow Hacker News and you'll see: a new language seems to be born every week.
The reasons for this are as follows.
When looking at the range of possible software solutions to problems, it appears as though a spectrum emerges when measuring those solutions against their robustness. Brittle solutions usually take the form of prototypes, whereas the application of better architectural principles and design patterns tends to result in more robust solutions. Choosing where to aim when building your solution with your particular problem set and context in mind can be challenging, but can be made a bit easier when considering a few questions that attempt to uncover how much uncertainty there is in your project.
When you don't have a clear customer and/or vision for your project, aim more towards a prototype - this will allow you to iterate more rapidly on possible solutions until you reach the most appropriate one. When you know your solution will need to work for some time, and you don't know who will look after it, it would be better to invest in developing a more robust architecture, as looking after a brittle solution usually requires specialised knowledge, expertise and care.