Big, Bigger, Biggest Design Up Front

This was originally posted at Ternary Software. It is mirrored here because I wrote it and I want it on my personal blog as well as my work blog.

---

Background

Extreme Programming encourages its practitioners to do away with a technique known as Big Design Up Front. BDUF is a software development process that has existed for nearly as long as software development. The general idea is to design a software system completely on paper before writing a line of code (experiments and proof-of-concepts aside). XP claims that this is a bad idea. In my recollection, the XP book claims that you wouldn’t get the design right anyway. You really can’t get the design right until you have finished writing the code.

Reality

In theory, this doesn’t sound too bad. I mean, the best way to understand a system is to try to formalize it in code. You’ll spot all of the inconsistencies and problems pretty quickly. (In fact, I think that’s the real skill that a software developer brings to the table - the ability to analyze a system and find the problems). But still, to do no design up front? Clients don’t come to you with no design. In fact, if one did, would you really take their business?

Prospective Client: Boy, do I have a project for you!
Development Team: Great, what do you have in mind?
Prospective Client: Well, see, it’s a thing. It does stuff. It’s a lot like That Web Site, only different. By different, I mean really different. But I’m not quite sure how. Also, it transforms into a boat.
Development Team: What? That makes no sense!
No Longer Prospective Client: Well, I thought I’d design it as I go.
Development Team: Well, we’re really booked right now, but we’ll call you as soon as we have some free time.

So I suspect that developers, too, need to do some design up front. I can think cases where NOT designing up front has cost us a lot of effort in re-design. Developing a 2-tier system when we knew that the client really wanted a 3-tier system is but one example. (We fixed that problem within a month, but it was still a hassle that could have been avoided). But how much design is too much? The XP book says something about doing “enough” design, but that doesn’t really provide a lot of guidance.

Disclaimer

Now we get into the part that I don’t really understand well enough. These are my thoughts, but they’re surely wrong. My thoughts will probably change with time, and I would appreciate any feedback that I can get.

3 Kinds

Here’s an over-generalization: systems that you design can fall into 3 categories:

1. Systems that perfectly implement the requirements of today with no thought to the future. Any attempt to use them in a way that is different from the original intent will require massive amounts of redesign. I’ve seen a lot of Javascript code that seems to fall into this category. For that matter, a lot of code written on consulting projects seems to fall into this category.
2. Systems that implement any conceivable requirement for the next 5 years. Sometimes, they do a good job of this; sometimes, they fail miserably. Yes, ZZZManager, I’m talking about you! A lot of libraries tend to lean in this direction.
3. Systems that, while implementing the requirements of today, also have a lot of hooks for the requirements of tomorrow. The assumptions that they make (and the hooks that they provide) aren’t necessarily perfect, but it’s not TOO hard to change them to support new requirements. Some really good libraries and really good frameworks do this. When I saw how simple it was to write a Java Servlet, I remember thinking that Servlets strike an excellent balance of providing functionality to the programmer without making too many assumptions.

I would furthermore attempt to characterize those three kinds of designs as:

1. Not Enough Design
2. Too Much Design
3. A design that’s juuuuust right.

The Source of All Design that is Good

Let’s assume that I’m correct in my characterization. How do you get this kind of design? Well, I think it’s a little like Chess. For chess masters, the match is won by the person who can “see” further into the future. I think that good software design comes from people who can (correctly) guess the future. There are a number of contributing factors:

• You yourself. I think some people just naturally think more about the future than others. I don’t think that people who think about the future are better at designing (see Too Much Design). But I do think that it’s important to be thinking about both the present and the likely future.
• Your experience. If you’ve seen a problem before, there’s a chance that you will try to avoid it in the future. I think that experience can help you avoid problems without even realizing that you are avoiding them - you just make really good choices.
• The pressures on your team. I think teams that have some healthy amount of slack time will be more likely to think about things that are a little broader than their current task.
• Your level of attunement. The more in-tune you are with your client and their needs, the more likely you are to guess correctly about their future requirement needs. You need to understand not only their product, but to a certain extent, their business model and their users.

Tips

Those factors are often hard to change. Are there more immediate guidelines for avoiding problems? I certainly don’t know how to avoid all problems, but here are some general thoughts that have helped me so far:

• If, while implementing a feature, you think you will be ripping out the code in a month, you probably need to design a little more. Changing code in a month is OK, but replacing it is scary. Replacing it in 3 years is probably OK.
• If you can’t explain to the client what you are working on and why they need it, you’ve probably designed the wrong thing (i.e. over-design)
• Don’t take shortcuts. If there’s an easy way and a hard way to design a system, make sure that the easy way isn’t really the “really hard later” way. It might be perfectly fine to take the easy approach. In fact, you probably should. Just make sure that you’re not going to get screwed in the near future.
• If you are writing very similar (but also slightly different) code in several places, you probably need more design. For example, you use an Excel writing library in 20 places in the code base, something is screaming to be extracted into its own class (or set of classes).
• If your code has no layers, you probably need to design some more. Just as classes need to be small and simple, so too must groups of classes. If you have a class that is referenced by code in several namespaces, you should wonder why everybody needs to know about him. (Perhaps everyone really does need to know, but you should still think about it).
• If you end up writing a lot of static methods, you need to extract some classes. A few static methods (especially named constructors) are OK, but if you have too many, you’re just writing procedural code.
• If you end up writing a lot of private methods, you need to extract some classes. A few private methods are OK, but if you have too many, you’re just writing procedural code.
• If you know that some of the code that you are writing is more general than your specific task, break it into 2 pieces. Write some of the code in a general way (using Template Method, aggregation, or something cooler), and write your specific code in terms of the general pattern. LISPers (and other functional programming dudes) know how to take this to the extreme. (Every functional language that I’ve encountered comes with a built-in function that can apply a function to every element of a list. Finding the sum of a set of numbers is often as simple as applying “+” to every element of a list). Be very wary of this, however, as this can lead to some scary over-design.
• If you don’t know what you are going to be doing later this week, you can’t anticipate the future. Have a plan.
• If your plan describes what you will be doing for 6 months, you will be using a different plan in 6 months. You’ll probably write code that won’t make sense in that new plan. Spend less time designing, and focus on short, iterative releases.
• If you write code that everybody uses, and which affects code throughout the system, and your code needs to be changed whenever some other requirement changes, you have too much centralization. You’ve designed a fancy system, but it’s going to cause you problems.

Conclusion

Those are all of my ideas for now. Actually, I could keep going, but I would just be rambling more. I want to again stress that this area is very unclear to me, and what I have typed here are only my current thoughts on the subject. I would love to hear other people’s opinions!