Time spent planning does not always save time and money in all cases. We have all seen people who love to build prototypes deliver something working while the guy who insists on planning still hasn't delivered anything. And the guy who built the prototype comes with a list of valuable questions discovered only through the act of building. You wouldn't build a skyscraper in the middle of a busy city without loads of planning, and you wouldn't do loads of planning to build a new product feature for a web application when you are told ahead of time that requirements will change during the build. Knowing which is which doesn't come from reading. It comes from experience. As much I do loving reading about software engineering, like everything else in life you've got to get your head out of the books and ride that horse eventually.
"ride that horse eventually" - I'm pretty well on with the horse and riding part, and I make plans I'm held to all the time, from investors, CEOs, and customers. I would have been fired long ago (and have been fired in the past) if I didn't make plans, committed dates, and contracts with customers that managed scope, schedule & budget (see: all of professional program management).
Please look up the papers on planning & quality & cost & schedule from Capers Jones. The textbook is literally titled "The Economics of Software Quality". Tom DeMarco's classic is "Controlling Software Projects".
I'm trying to give you a starting point to see that it's not even my anecdotes, but repeatable studies that show this.
Your examples are that requirements change, and it's hard. Of the few buildings I have built, the requirements changed drastically there in all cases, so I can only say that software engineering is not magically different. After decades of practice with software engineering my experience of planning lines up with the textbooks you can find on the topic, and not this blog post.
And after decades of experience myself I've seen the vast majority of plans become useless the further out they try to predict the future. I guess we'll have to agree to disagree about the value of learning by actually starting to build the new thing that no one has ever built before. Plan away if that suits you. I'll be building and learning about the domain in the meantime.
Planning upfront takes less time than not planning up front, in all cases. You can plan to do a prototype, you can change your plan later, the activity of planning itself produces clarity.
Reducing the argument to "planning away" is the same as me saying you're "flailing around like an amateur". It's not what you're saying though, so I wish you'd see this thread as a discussion rather than something to win with gotchas.
Exactly. You could learn about when and how to be more efficient when planning isn't called for.
> Planning upfront takes less time than not planning up front, in all cases.
It does not.
> you can change your plan later,
That costs time and money. Maintaining and changing a plan does not come for free. The more complex and further out your plan reaches, the more you'll have to change it.
> the activity of planning itself produces clarity.
Yes it does. And sometimes the activity of building a prototype produces more clarity than a plan.
I like Capers Jones too. His recommendations are not the silver bullet you are treating them as. He sells software used for estimating, measuring, and planning... so of course he's not entirely unbiased. A lot of what he does applies very well to large organizations where large and frequent pivots are nowhere near as frequent as when working for a startup while exploring the features of something that is not evolutionary, but revolutionary. Check out this article where Capers Jones shows that Agile is one of the quickest and cheapest methodologies out there.
And where he concludes "Overall the Agile family and the methods that emphasize speed have achieved their goal, and they are fairly quick. The methods that emphasize quality such as TSP, RUP, and CMMI 5 have also achieved their goals, and deliver very few defects. No single method appears to be a universal panacea that can be successful on every size and kind of software application."
Huh.
> I wish you'd see this thread as a discussion
You're not even willing to consider that planning too far ahead can waste time and that planning does not optimize for speed. You are giving absolutes like "in all cases". If you want a discussion then you have to be willing to learn yourself. In my experience there are times when planning is essential. And times when planning, especially out more than a few weeks, is a waste of time. If you want to trade tips, great. If you want to continue to insist planning is an absolute, no thanks.
Great response, thank you! It seems I'm rubbing you the wrong way, or not understanding you well enough to respond in a way that encourages conversation, and that's crap. I'm sorry.
> And where he concludes "Overall the Agile family and the methods that emphasize speed have achieved their goal, and they are fairly quick. The methods that emphasize quality such as TSP, RUP, and CMMI 5 have also achieved their goals, and deliver very few defects. No single method appears to be a universal panacea that can be successful on every size and kind of software application."
This article must be taken in the context of his research in general. If you're optimizing for speed of initial delivery, rather than TCO, then any method with less planning delivers faster and costs less. However, if you assume your software lasts longer than ~year (as pointed out in my initial reference, and his much larger work "The Economics of Software Quality") then they perform much worse. The actual payoffs in most of his research is around 8 months of development.
Being "quick" is basically a jab from Jones about agile project quality.
> You're not even willing to consider that planning too far ahead can waste time and that planning does not optimize for speed.
I'm willing to consider it, I think. My experience has shown the opposite for any project involving more than 1 person that a customer uses for more than a year. The research I can find shows me the same. Most projects I embark on tend to live in the market for longer than a year, and thus quality, TCO, and delivering on time matter.
I guess I don't see how you manage a team, a project, a customer, or investors where you need to hire people based on work you need to do, if you're only planning out a few weeks at a time. I don't see any case where you're going to get VC funding, exec sponsorship, etc (i.e. manage the risk and expectations of capital) by saying "we'll see where we are in two weeks".
This is where I start to fall into saying things about professionalism, as to me the difference between a professional and an amateur is exactly this capital and risk management.
> If you're optimizing for speed of initial delivery, rather than TCO, then any method with less planning delivers faster and costs less.
I feel like we are getting somewhere now. For some projects speed of initial delivery is by far the most important factor. Which is why it's so important to qualify your claims instead of using absolutes like "planning always results in..."
> However, if you assume your software lasts longer than ~year (as pointed out in my initial reference, and his much larger work "The Economics of Software Quality") then they perform much worse. The actual payoffs in most of his research is around 8 months of development.
Why would you assume that? Lots of software these days lasts far less than a year, and that's intentional. This is not the 1970s when building software was an expensive and rare thing. There are literally hundreds of millions of applications out there. And billions of dollars being pumped into startups, the vast majority of which will throw out their initial idea and eventually pivot when they learn what their customers really need. Learned by building quickly and delivering something that works well enough to get it in front of people to see how they actually use it and how different that is from what you had envisioned (or planned). What comes to mind is the cardboard box that the kids play with while the new toy it came in sits ignored.
> Being "quick" is basically a jab from Jones about agile project quality.
Why would it be? Unless you are heavily biased towards software always being high quality. Which shows a lack of flexibility and a lack of professionalism about optimizing for what is important to your client or employer. If your employer needs a project to optimize for initial speed, then focusing on quality and planning is unprofessional and irresponsible and putting your own personal preferences ahead of what the company has decided it should optimize for.
> My experience has shown the opposite for any project involving more than 1 person that a customer uses for more than a year.
I'd chalk some of this up to confirmation bias. But at least this far better than "always". At least we are starting to add qualifiers like "uses for more than a year"
> I guess I don't see how you manage a team, a project, a customer, or investors where you need to hire people based on work you need to do, if you're only planning out a few weeks at a time
You manage the team by focusing on building and learning from what you build. Let's say you plan for an easy build over a few months and hire 1 person. They start building and as a result of building a new market opportunity is discovered so new requirements come to the surface. Some of these requirements are difficult to implement and will require 10 people over a few months to build. But that is where the company now wants to pivot. How did your plan for hiring 1 person help you here? I'd say it gave you a sense of false confidence. What would be more professional and frankly more honest is to say "we don't know yet". That can't go on forever of course. The entire point of favoring building over planning is that in the course of building you should be reporting back on new risks discovered. And new opportunities. It's not all downside. When you have a plan to take six months to build something and it only needs a month, well Parkinson's law is nothing new so we all know it will take six months.
Lacking a plan, and with a focus on urgency and speed, you are forced to ruthlessly eliminate requirements. Since you will get something built in a few weeks the chances of you getting the thing you needed in a month are much higher than the six month plan. The exercise of delivering working software every couple of weeks gives you laser focus on what your core offering is.
> This is where I start to fall into saying things about professionalism, as to me the difference between a professional and an amateur is exactly this capital and risk management.
If your only tool for managing risk is extensive and detailed plans that must be constantly maintained and that do a poor job of optimizing speed over quality, then you are not being as professional as you could possibly be. There are more tools available for managing capital and risk than "let's make a detailed plan".
If you are employed by a company where the culture is all about using plans to manage risk, then you've found a great fit for yourself. But you shouldn't look down your nose at alternative ways of managing risk and call them amateur. There are different types of risk and there is no silver bullet in managing them.
One alternative for managing risk is admitting that you don't know and just building the damn thing and learning from it. This method of managing risk is especially important for really new things that have never been done before. You want to fail early and often and that is the most professional thing you could possibly do on those types of projects. At least until someone comes up with something better.
I'll just end by saying thank you for making the effort to steer this to a higher quality and more nuanced conversation. I enjoyed talking about this with you, and now I have to move onto other things.
> Unless you are heavily biased towards software always being high quality.
This is the main discussion of his research, to show that quality and defect removal actually lower cost and increase speed. The tradeoff between time & quality is one he regularly shows is not a tradeoff at all, even for very short timelines.
> I enjoyed talking about this with you, and now I have to move onto other things.
