This article falls prey to a very common pattern of defining waterfall as "anything that doesn't meet my narrow definition of agile". Since everyone knows that waterfall is a fiction that doesn't work, my opinion is correct.
Developing a plan is not waterfall. Designing an Architecture is not waterfall. Agile is about adapting to change as it occurs - how do you know if there is any change if you didn't start with a plan?
Good plans and designs account for risks and unknowns and anticipate (certain types) of changes and delay locking into assumptions until necessary. But not having a plan or design is not agile - it's failure.
Agile was created for assembly line work, not big development projects. Hence it does not concern itself with overall plan and architecture, only with the day to day, week to week, work of implementing it.
Understanding this, agile works fine. But zealots without practical experience want everything to fit the agile religion. And so you get the confusion about how to plan an architecture, how to develop overall system requirements, when everything must be incremental and in sprints.
We had pretty good solutions to many of those things twenty years ago, but that is "waterfall" now, the tarnish used to denigrate everything agile acolytes don't understand.
Agile is a product now, sold by ignorant business consultants. I've never seen a more ridiculous mess than watching a non-software organization trying to transition to 'agile' in the whole organization. Sales, marketing, business process management, doing sprints, with scrums and epics and story point estimation, etc. Yes, that is what the agile industry is selling now, agile for the whole organization, in any industry. The horrors this will produce we have yet to fully enjoy.
> Agile was created for assembly line work, not big development projects.
This reminds me: I saw SAFe everywhere a few years ago and now I don't see it in the wild as often. It seemed kind of snake-oily to me. Does anybody have any idea what happened when I wasn't looking?
> Developing a plan is not waterfall. Designing an Architecture is not waterfall.
Yes it is. Fundamentally, any project stage that you can finish without delivering a working system to the end user is waterfall.
> Agile is about adapting to change as it occurs - how do you know if there is any change if you didn't start with a plan?
Why do you care whether there is change? The whole point of agile is that it doesn't matter if the customer's requirements change, you'll deliver value to them just as effectively whether they do or don't.
> how do you know if there is any change if you didn't start with a plan?
Easy. The designs change. The requirements change. A plan is not a design. A plan is not the requirements. A plan is a step by step description of how you are going to implement the designs and requirements, and in what order. And if the designs or requirements change enough, you now have to throw out that part of your plan.
Could you give some sources where plan means something different in the context of software development methodologies (the topic in question) and that most people in software engineering use that different meaning?
More importantly, is there a definition of "plan" in the context of software development where my claim fails? The claim that if requirements or designs change enough, some parts of your plan have to be thrown out?
The meaning I used is also a primary definition and often enough the first definition you'll find in dictionaries. [1] [2] [3]
Your first link seems to contradict you. "A proposed or intended course of action" covers everything from the roughest outlines to the most detailed sequences of steps.
Regardless, I'm not particularly interested in the dictionary definition. This is English, and what matters in English is how people actually use the word. You are literally the first person I've ever heard of who believes that "plan" is restricted to high levels of detail.
It is useful and efficient to defer making decisions until you have to.
But it's critical to avoid waiting too long to make decisions. Because that means the decision was to make no decision, and that has only a random chance to be the right one for you.
Planning is a big part of how you have the information you need to know when to make a decision and when to defer it.
Plenty of work be organized to fit into ~two-week, focused sprints, and can be done efficiently that way. Use your planning to figure out what should and should not be tackled that way.
There's also work where it makes no sense at all to do while looking ahead at most only two weeks.
For instance, if you have a good experienced architect, a small amount of time spent on a high-level arcitecture will pay you back on even a slightly successful project. Not to mention it will increase your agility. E.g., don't design a sophisticated privilege mechanism until you know you need it. But if there's a reasonable chance a privilege mechanism will be important in the future, know where one would fit in in a way that wouldn't require a substantial rewrite. Now you'll be able to slot in a simple privilege mechanism, if/when you need one, and later replace/evolve that with a sophisticated one, if needed, all quickly and without substantial rewrites.
I worked on two very large software projects (AAA games count, right?) and failure to make decisions in a timely manner was one of the largest factors in how much the project suffered, both staff and the product.
What I mean by that is, the project which did not decide things on time and tried to save the decision until they knew everything suffered much more than the project which decided in a timely manner and re-evaluated the decision in the rare instance that they were wrong.
It was greatly inefficient to have people sitting around not knowing if what they’re working on will even be used because they’re touching something that hadn’t been fully decided on yet, but was required for them to deliver on something else which _had_ been decided.
It is sometimes better to decide, iterate and reevaluate. Instead of attempting to defer a decision.
Analysis paralysis (refusing to make a decision until more research is done, to the detriment of a project) is a different error than delaying a decision until beyond the last viable moment.
> There's also work where it makes no sense at all to do while looking ahead at most only two weeks.
> For instance, if you have a good experienced architect, a small amount of time spent on a high-level arcitecture will pay you back on even a slightly successful project. Not to mention it will increase your agility. E.g., don't design a sophisticated privilege mechanism until you know you need it. But if there's a reasonable chance a privilege mechanism will be important in the future, know where one would fit in in a way that wouldn't require a substantial rewrite. Now you'll be able to slot in a simple privilege mechanism, if/when you need one, and later replace/evolve that with a sophisticated one, if needed, all quickly and without substantial rewrites.
No. This is the most seductive myth in software development, and it's exactly what Agile exists to avoid.
I've worked with experienced architects. I've worked with very intelligent architects. I've worked with extremely highly paid architects. None of them ever managed to add value to the project rather than subtracting it.
Write the simplest thing that might possibly work. Write plain code that's easy to refactor. Ensure that your test cases (or better still your type system) capture your business requirements so that you can refactor fearlessly and mercilessly. And as if by magic, retrofitting that sophisticated privilege mechanism will turn out to be easy, because you don't have a bunch of architecture getting in your way.
Create types that express the requirements that you have, and use them to ensure that requirements are always met. For example, if some fields should only be accessed by administrators, have an AdministratorsOnly type; if some method can only be invoked in a transaction, have a MustHappenInTransaction type. I find that when you have cross-cutting requirements, types are often the most lightweight way to express them.
Planning out a type system to match business requirements sounds like architecture level work to me (especially if you use a diagram to plan out those type relationships/hierarchy).
Of course my perspective might be skewed, I have only really worked in small teams.
>How would you build a rocket if you only thought about the next 2-4 weeks?
We're not building rockets. We're building business software. And the requirements for that are not a hardline set of rules laid out by the laws of physics. They are soft and fungible. Beholden to the whims of a product manager, designer, or client. Generally almost nothing is truly understood about the shape and function of the resulting software at the beginning of a development process. And sticking to a "plan" that was laid out before these things have actually been discovered is a recipe for disaster. Better to build in tiny, discrete chunks of usable MVP that can be used as a guidestone to the next desired level of functionality.
Business software projects can have very clear requirements known months in advance. Eg a project to ensure we comply with newly legislated tax regime. Yes, details will be discovered as the regulations are released, but to embark on that project with a 2-4 week planning horizon would be malpractice.
"plans are useless, but planning is indispensible".
If you’re building business software then yes mostly. Change is cheap.
If you’re actually building a rocket or any other physical thing with real world stuff (like some of us here) you need a real plan. The fits software engineers get into about agile is odd at times. I don’t really get it.
Setting aside the specifics of particular agile methodologies like scrum, etc I think the core insight of the Agile manifesto is that building software products is fundamentally different than building things like rockets. When you are building a rocket you more or less know exactly what you want to build from the beginning so doing a detailed up-front design is definitely the right way to go. When building a software product (or really most software products, there are of course pieces of software that are well specified in advance) you don't know what the end state is or even if there is an end state. The information flow is bi-directional in the sense that observing how the software is used in the wild feeds back into how the software should be designed.
Ughhh, you do realize that there are software systems out there probably just as complex as rockets, as they are, well, used in rockets themselves to control the physical parameters.
You do know what the end-state is. It's called the minimum viable product, MVP. The most successful product owners are ones who are technical and can bridge the gap between the business requirements and technical implementation. For example, there would be no SpaceX and vertical landing of rockets if it wasn't for Elon Musk, who is both an engineer, a businessman, and an entrepreneur. He gets the full picture and can talk to all divisions of the company with incredible competency.
If you have a product owner who knows nothing about building software, in charge of a software product, you'll have a much smaller chance of succeeding, than if you had someone who was both business and technically savvy.
For what it's worth, I agree that the advice in the article is badly wrong. I'm just saying that building a rocket is fundamentally different than building a software product because a rocket is well-specified in advance. To you point, the MVP is different from the product as it will exist once fully realized. And the reason is because you don't know in advance how users will respond. You're not trying to satisfy the physics of rocket flight, but trying to please actual human beings who have weird and idiosyncratic preferences. So you build something (an MVP), let them use it and then incorporate their feedback into the next iteration. I'm with you in the sense that once you have a concrete MVP that you want to build, then you should approach building it as you would approach building a rocket. It's just that you may find that user didn't actually want a (proverbial) rocket, they wanted a helicopter.
How would you build a rocket if you only thought about the next 2-4 weeks?
Okay, I’ll bite...
I’ll grant you the 2-4 weeks part isn’t quite right for building a rocket; but the key insight of Agile, of developing in small steps and staying flexible, definitely applies.
You do not start by designing the entire rocket before building anything. Your rocket won’t work. You start, you must start, by building a series of prototypes and exploring the design space. As you do that, you gradually refine your ideas of what’s possible and how to do it. As far as I know, every successful rocket design has followed that path.
Sure, at some point along the path, you put together the blueprints for the final design. So there’s an element of waterfall in there. But staying agile is still critical. Unexpected stuff is sure to come up and you have to retain the flexibility to deal with it.
As others have noted, software is somewhat special in that it’s cheaper and faster to build than physical stuff, can be copied and refactored much more easily, etc. And that shifts the agile-to-waterfall balance where you can realistically talk about working entirely from short term requirements.
Yes and that’s what almost everyone does. Learning loops with prototypes til you have a feasible product. But then you need a big development plan for how to scale it to a polished, sellable product. Because there’s a lot more to the product then the artifact.
> But then you need a big development plan for how to scale it to a polished, sellable product. Because there’s a lot more to the product then the artifact.
That's a total non sequitur. Agile is all about building the sellable product. But doing it iteratively works a lot better than doing it via a big development plan.
You’re getting downvoted because you may have missed my point.
To clarify for
your sake, once you have hardware in the loop (eg mechanical engineers designing things, manufacturing scaling for production) and a large commercial organization who needs to be ready for a product launch, you need a waterfall type plan. Agile principles (customer focus, respond to change) can apply, and you can break up the work into 2 weeks, but you do need to look past the next 2-4 weeks and know where things are planned. Commitments matter in the $millions.
Software-only SAAS has the luxury of avoiding that complexity.
Would you use an 'agile' methodology to build a rocket at all? There some projects not suitable for an agile approach, especially if we're talking "projects" generally beyond software engineering, perhaps MOST aren't. "Agile" is software engineering advice. OP is advance for projects that an agile approach is suitable for. Ie, not building a rocket.
Agile isn't, strictly, about software though it certainly is the predominant application. So first, let's separate two things:
- building
- designing
Some parts of agile do apply to building. See Lean Manufacturing, parts of it are responses to maintain quality (Andons used to stop production) and parts are to change over the manufacturing floor to build something else (same plant can make many kinds of cars without a fixed production line for each, letting them meet demand).
Other parts of agile, though, are aimed more at the design portion. Sprints, for instance, are meant to give you short iterations that let you frequently evaluate both your process and your product's success/failure.
You can apply agile concepts to developing almost any kind of thing. Actual rocket scientists create simulations, scale models, full-scale models, prototypes and other things on their way to the final system. So they're already applying the same iterative approach that most agile proponents endorse. The question is, is the iteration deliberate or accidental? Or do they try to deny the iterations even exist? Once they get to building the rocket, that might be less agile depending on circumstances (is it the first one they've built? are they setting up a production line? is it a one off?).
Note that when building cars flexibility is a trade off. You can switch car type, but the jigs for the car you are not currently making is still a cost, separate lines at near full capacity are a better use of resources. Only if you can forcast demand years in advance to build the factory to the right size. This is a hard problem.
Note that flexible factories have other limits. Honda can't make a large truck in their factory as there isn't room. I don't know if Honda regrets this cost or not.
> Fisher's Fundamental Theorem: The better adapted a system is to a particular environment, the less adaptable it is to new environments. -- Gerald Weinberg,"The Psychology of Computer Programming"
There are always trade offs. The more flexible you make a system, the less efficient it is (most likely) in achieving its (now increased) goals. But the more efficient you make it, the harder it is to adapt it to fit additional or changed goals. Which trade offs you accept depend on your objectives, and sometimes your ability to spin up a whole new factory.
We can probably force this to be true, to some degree, by defining "adapted" differentially... but otherwise this smells more like a "possibility frontier" situation than a strict spectrum.
I think it's useful to separate building and designing like that. What it makes me think of is: One thing fairly unusual (if not unique) about software is that the designing and the building generally happen "simultaneously", or in tandem, or in very tight iterations. We design software as we build it, and we build it as we design it.
At least often/commonly. If we don't always do that with software, I think we always do it at least to some extent (in an unusual way compared to physical things), and software is (often if not always suitable to designing and building.
I think "agile" as we know it (which is really short for "agile software development", that's what the agile manifesto hasin it's title!) is particular to something you are (or can profitably) design as you build.
While there are analagous things in physical products, and there is (or profitably can be) some element of design/build cycle in physical products, I think rarely to the extent it can and does happen when making software. Rockets are probably at the far end of separating design and build.
But the tldr version is probably just: The Agile Manifesto's full name is literally "Manifesto for Agile Software Development". While some of the things may happen to be useful or applicable in other fields of endeavor, others may not be. Saying that something from the Manifesto for Agile Software Development and ensuing methodology doesn't apply to a rocket is not a relevant critique of the methodology as applied to software, as intended.
> One thing fairly unusual (if not unique) about software is that the designing and the building generally happen "simultaneously", or in tandem, or in very tight iterations. We design software as we build it, and we build it as we design it.
I actually had something about that but cut it for some reason.
I'd like to point out that while the manifesto is about software, a lot of the ideas (beyond the manifesto itself and influencing the manifesto) come from other disciplines. Mary and Tom Poppendieck, for example, are major proponents of Lean Software Development, whose ideas are a translation from the world of physical manufacturing. Many of the DevOps ideas (not necessarily how it's played out) come from Eli Goldratt and his Theory of Constraints.
An interesting thing about both Lean and ToC is that neither is terribly prescriptive. They're actually generalized concepts that can apply to almost any kind of system and (mostly) lack prescriptive text saying, "Thou shalt do...". Compare that to much of the Big-A Agile movement which is inflexible, ironically, and often very prescriptive. Perhaps not down to the specific tool that should be used, but often to the specific methods that should be used (Scrum: Thou shalt have 1-4 weeks Sprints; Thou shalt have time boxed meetings of this nature; SAFe: Thou shalt have a massive meeting every 6 months in the format we describe in this chapter).
Ok, that's a bit unfair to Scrum, it's actually more flexible than that. But I'll happily burn my books on SAFe (picked up for a previous job).
Many of the other ideas for Agile are born out of Systems Engineering which had already dealt with Waterfall. You had the V model (an improvement, but still many of the flaws if taken too literally, over Waterfall) which makes V&V run concurrent to design and development and encourages feedback loops. In many ways this is akin to TDD. Incremental and iterative development processes, which are fundamentally no different from Scrum except Scrum pushes you to much shorter times and has a stricter structure. Evolutionary development which is really close to what many, if not most, agile proponents endorse (MVP, then add on features and modify the base as needed). None of that came from software (specifically), it came from developing physical and (in their parlance) cyber-physical systems.
I would use ideas from agile to build a rocket. Automative testing, keeping everything green. Just because there are millions of lines of code left to write before the rocket can take the first test fire doesn't mean I'm not keeping the code in shape that the current features could ship.
But my rocket needs to be ready when the earth-venus system is in the right stage or I delay a few years. Agile cannot handle this. Agile can handle getting rid of some features to meet the deadline, but waterfall can do priorities as well. What is needed is watching the end date and assurance that you will make it, or warning to add people before the cost of adding people is greater than their future contributions.
I feel like I write about Waterfall and its problems a lot on this forum, but I'm going to do it again:
> Agile can handle getting rid of some features to meet the deadline, but waterfall can do priorities as well.
Waterfall is not particularly good at handling priorities. It's actually worse, in my experience, than anything else. Waterfall is about the order of doing things (it is fundamentally linear), committing to a plan (lack of flexibility), and has minimal feedback loops (lack of responsiveness).
If you have a 2 year deadline (true deadline like in your example), Waterfall only helps you if you assume everything will go right or put in drop dead dates for lower priority things. But even then, you'll have wasted so much time in your planning and designing you probably won't be able to keep it on track within a reasonable budget extension.
What you really want to handle priorities is an incremental and iterative development model. You know your target, but you break it down into smaller increments and build it out over time. After each iteration you re-evaluate your ability to hit the target. Within each iteration you may use a Waterfall-styled approach (particularly if it's a portion that's well understood in your organization or sufficiently small) or you may extend the iterative approach by adopting an Agile approach (think 3-6 months of 1-2 week sprints).
Using your Earth-Venus example. You need X, Y, Z. You want A, B, C. You only have 2 years to deliver. In Waterfall you'd spend 3-6 months designing everything before building it. If you're smart you design it with a way to drop A, B, and C. But that's a lot of time spent up front on contingency planning and pushes you closer to 6 months. Now you're 18 months from the deadline before production even starts. Oh, and testing doesn't happen until pretty late in this process. So your production is constrained to no more than 12 months in order to allow sufficient testing time (6 months).
If you're actually smart, you create checkpoints/milestones and do it incrementally. You start work on X, Y, Z and as much of A, B, C as you can afford. After 3 months you see that X is progressing fine, Y is possibly going to miss the target, and Z is definitely going to. So you check and A is going to be done in a month, B turned out to be a cluster, and C may complete on time. Cut your losses, B is gone, the money and manpower are moved to the core mission.
Next iteration you reexamine, and find out that you C may still complete but Z is barely back on track. So you take some or all the funding and people from C and dedicate them to getting Z back on track. In the best case C can still be done, but it probably won't. We accept our losses and continue this process.
That is a fundamentally agile approach. It's about responding to the changes that come up. Waterfall is not, in itself, capable of this.
Your point is well taken, but one thing waterfall does is the required up front planning to get a good architecture in place. The agile approach tends to ignore such things even though they are critical on something as complex as a rocket.
J. J. Sutherland claims that Saab designs fighter jets using scrum. He claims not only have they been successful from a cost perspective, they also have an easier time staying up to date in the arms race. Pun intended.
He also sites case examples of restaurants and house remodellers using scrum successfully.
maybe this person is thinking of a scope similar to an elaborate to-do app?
Many enterprise scale business centric applications have tremendously complex business rules and overreach. One can not simply make a casual decision last minute, and expect perfect execution, towards an end result of increased business value.
EDIT: I see he is a Drupal consultant. There is a very explicit ceiling on the complexities of primarily CMS driven sites.
Yep - the hilarious thing is even the most Agile/Sprint heavy consulting companies usually have to succumb to some Waterfall concept in order to provide an estimate or meet a fixed bid.
Why do people ask for estimates or fixed bids? It's to manage risk. And how often do projects not go over estimated budget? Even fixed bids are poor at managing risk. Lots of consulting companies will walk away from a fixed bid contract after months of work at the point they realize they will lose money on it. And honestly, what's the quality of work once the contract is no longer profitable? We've all seen it. Everything is a quick hack at that point in a desperate effort to achieve the minimum that could be defensible in court. A fixed bid project means the consulting company and the client have opposing interests: the consulting company wants to get it done as cheaply as possible and will make every shortcut they can get away with. And the client wants every last bit of quality they can squeeze out of the contract for that price. No wonder it so often ends in dissatisfaction for both sides.
What are the options for the client when the consultant says "sorry we didn't actually finish and we are out of money"? They either pay more or have to be willing to sue the consulting company. Most companies suck it up and pay more. This is so endemic to the industry that most companies put extra in the budget for software project overruns no matter how tight the fixed bid contract is.
So one way to sell agile is to help clients understand how it can help manage risk better than a fixed bid contract or always inaccurate estimates. And how it can be used so that the interests of the consulting company and the client are better aligned.
I talked to a consulting company once that offered me a fixed price per sprint, with no commitments on how many sprints the project would take. I did not feel like our interests were any better aligned than in a fixed bid contract.
So one data point. Sounds like they didn't differentiate themselves well enough from the fixed bid contract. The great thing about fixed price per sprint is they have to impress you with progress every sprint or they risk losing the job. That's great at motivating them to deliver value to you every sprint. Another way your interests are aligned: you know how many sprints you want to pay for. That motivates you to prioritize the most important features so they are completed before you run out of budget and sprints. What a great way to manage risk.
> But lack of alignment is known to be an issue with both flat AND time-based fees.
Agreed. Agile only puts you in a more flexible relationship. It does not fix every alignment issue. For that you need to be more creative with the contract. However it does spread the risk more evenly. And by you as the client taking on more of the risk instead of pretending an estimate or contract erased it, you will do a better job of managing the risk. Every sprint you'll decide what must be built next. Every sprint you get the opportunity to change requirements. Both of those are far better ways of managing risk than fixed price contracts or estimates.
> Charging per sprint is only superficially different from charging hourly/per day/per week/biweekly.
Not if you are delivering a working product at the end of each sprint. And yes, milestones are somewhat similar to sprints. Except as a client you've now locked yourself into what gets built and if requirements change you've got to renegotiate the contract.
> You can prioritize features? But why wouldn't you do that without sprints?
Because your fixed price contract legally "guarantees" all the listed features will be included.
> Clients will want it to be over as fast as possible.
And you can have that if each sprint ends with a working product. Up to you when the product has enough features included that you are happy with it. Or that you are out of budget. And you've managed risk aligned with the consultant instead of getting an unrealistic estimate they won't meet, or having a to head to court with a broken contract.
Waterfall is shit, don't think waterfall. Better is to think about things like work breakdown structures paired with an iterative schedule (aka, sprints). Your WBS establishes the things that you know now, the things that can be done in parallel, and the things that depend on other actions. Your WBS also gives you a rough schedule (but do not take it as gospel because you aren't a prophet). Based on this, you can start prioritizing the work that goes into the iterations. Periodically (with each iteration or after several) you revisit the WBS and determine if what you're doing is keeping you on track, or if it's even still the right structure.
Agile is the wrong model for building something that must fly or you die. The opposite of much of the software we build where a quick prototype in front of real users gives you far better information than any amount of internal brainstorming.
This is a common sentiment but I don't think it's actually true. Users like software the works and is performant. Putting a quick prototype in front of users doesn't tell you anything beyond what the users think of the quick prototype. Did they have a bad impression because they ran into one of the many edge cases you didn't account for in the prototype? Or is it because the software is ill-conceived and not something they would like even if executed flawlessly. It's hard to tell. I think you run the risk of putting together a crappy prototype and then iterating your way into a local optimum that is still far inferior to what a properly engineered piece of software could be. Or it never gets that far because users hate the initial iteration because it doesn't work very well.
The vast majority of invented products - software or not - are complete market failures. No one wants to use them no matter how well engineered they are.
When we talk about building an MVP (minimum viable product), the V part is very important. It does have to be useful. It does have to look great and work great. It does have to be performant. Edge cases should not get past your QA department.
So with those constraints, about the only way you can get out a prototype in a sprint or few is by ruthlessly reducing scope until you get to the core part of your product that is adding value. That process in itself results in far better products.
My goal (that I sometimes don't achieve) is that each sprint results in a product we can ship that is better than the last iteration. Run out of budget? Company priorities change? No problem. We still have a useful feature we can ship.
Per my previous comment, this is the wrong process for plenty of products. You have to know what you are building and what the risks are, what the constraints are that you are optimizing for, and the pros and cons of different types of processes before deciding which process is the best fit. If you have a strong idea of what you are building ahead of time and you have experience building those types of things and you know there is already a market demand for those types of things, then by all means come up with a solid plan ahead of time and then execute it.
I agree, but I think the constraints are doing all the work in your statement. I've just been in the position many times in my own career where product managers are pushing to get something, anything out there so they can get "real user feedback" And the V in MVP goes right out the window.
I was responding in particular to "a quick prototype in front of real users gives you far better information than any amount of internal brainstorming." Maybe it's just a semantic difference but to me "quick prototype" doesn't imply look great, work great and handle all edge cases. Perhaps you meant quick prototype that looks great, works great and handles all edge cases but I think that implication is often lost on product people. They want something deployed and don't care if all of the engineering team's fussy edge cases are handled.
> I agree, but I think the constraints are doing all the work in your statement
Yep.
> I've just been in the position many times in my own career where product managers are pushing to get something, anything out there so they can get "real user feedback"
Me too. There are a lot of bad product managers and even companies out there. But I have worked with great ones occasionally, so it's worth describing the ideal so we know what we are aiming for.
> I was responding in particular to "a quick prototype...
You're right that I need to qualify that because it more often means what you rightly assumed based on your experience. I guess I'm just so in the habit of demanding that we reduce scope that I too easily forget that "quick" more often means the compromise was quality.
That's a good point. I think we know enough about building things that fly now and we can do it quickly and cheaply enough that we can build prototypes that fail, and iterate until we get it right.
I could be wrong, but I think that was less of the case in the past. Or perhaps we just didn't know better. Building big expensive things that can kill lots of people - I'm thinking infrastructure, commercial planes, etc - comes along with the idea that we should know ahead of time how we will make it safe without it failing. We can't build a prototype of a skyscraper in a city and have it fall on people. But perhaps the way we engineer those types of things could borrow a lot more from agile than I was initially claiming.
Rockets were an iterative process of gradual and revolutionary improvements that made engines you would actually be willing to put a person or nuclear warhead on top of.
Remember that the “build” phase of software is relatively instantaneous. The design phase is the only part anyone sees anymore. (The heroes who wove the Apollo ROM were examples of actual program builders).
Making the Saturn V static test rocket was heavily design work slowed by manufacturing. Making Apollo 11 was a process of knowing mostly what was needed and building it from start to finish.
Could not disagree more. It's a disingenuous rhetorical device to say "anything I disagree with is waterfall and therefore wrong".
Planning is necessary and useful. The detail of that plan depends on how far out you're going. The next sprint's plan should be very detailed. The next year's plan should be a rough sketch. The next quarter's plan should be somewhere in between. And regardless of the level of detail, those plans can still change if they need to.
I've run into enough garbage code and garbage systems to see what happens when people don't plan enough up front. No thanks.
It's odd that this bit from the Agile Manifesto is used in part as justification:
> Responding to change over following a plan.
I don't read that as "don't plan"; I read that as "plan, but expect your plan to change over time".
> Don’t create tickets for things you won’t work on for months, if ever.
Yikes. Dropping tickets in the backlog for ideas that pop into my head is a great way to make sure they're not forgotten! The team may decide later that it can be closed, but it opens the door for a conversation and helps collect ideas.
Months down the line your ideas might not even be relevant anymore. On my team, if you put stuff in the backlog, you're responsible for maintaining it. If you're not prepared to accept that responsibility, it belongs in a notebook or a word doc, not the backlog.
Yeah this suggestion seems like addressing a symptom - a backlog full of useless items - instead of addressing the disease - not having regular backlog grooming sessions where outdated/irrelevant items are either updated/fixed or removed.
The alternative to this that I use is "don't precisely estimate tickets more than a month out" because if something has 6 other tickets between now and it, things will probably change by then, so a precise fully-fleshed-out estimate will probably just wait time and not be any more valid than the "about a week" one anyway.
This is 100% anecdote, but IME agile doesn't work well enough to warrant "waterfall shaming" like this.
So far, in 25 years of software development, the best track record I've personally seen for on time and on budget was a big defense contractor that had a bunch of senior engineers and a relatively old fashioned waterfall process with some "extreme programming" and agile practices thrown in. It was documentation heavy and a hassle to think things through up front, but it reduced surprises later on, and the rigid documentation and process made it easy for different people to take over work at different stages of the workflow (requirements -> design -> coding -> testing) for each feature.
It wasn't perfect, but it was predictable, and I never saw them toss features out at the last minute or deliver a half-assed release with a bunch of minor changes.
And I'm sure the response to this will be a dozen people telling me all the companies I've worked for have done agile wrong, and then everybody will have a unique idea on how they should've done it instead...
Agile might work great for CRUD websites, but it doesn't seem to work very well for other types of projects, and choosing something else seems reasonable.
I guess my point is, even if these projects aren't "real" agile, I don't know if it's a real problem. The goal isn't to be agile, it's to get software developed.
I don't think it's good to abandon up-front planning entirely. It's important to know what you're aiming for and to have a plan for getting there. The key thing about an agile approach is to regularly review the plan and be open to changing it as and when new information comes in.
Totally correct. This article is on the "totally idealistic" side, and I think it's important to know what is ideal as a benchmark when working in non-ideal situations.
Executives and clients like to see timelines and total project estimates. I wrote an article* about this. The idea is that you do some rough planning, applying the principles of Agile story point estimation, and extrapolating a rough/relative timeline.
Relative timelines with non-specific dates are useful for getting an idea of the scope and possible time of a project. Helps determine what should be built first, prerequisites, and what other teams might need to be involved (and at what stage). Also useful to get the product managers or executives thinking a bit more deeply about the project and perhaps reconsidering it, or reducing scope if it looks like it will be larger or take longer than expected.
Once a project is kicked off, after a few sprints the initial plan can be updated based on the actual velocity of the team.
They need them because they all have their dependencies that they need to manage separate from this project. A client may need to plan training, transitions, budgets, deprecation of old systems, etc.
Others above me have mentioned the reasons why they need them. But I'll reiterate my emphasis on rough/relative timelines. Exact dates are impossible and the farther away a milestone is, the hazier the date is. As an example, a milestone a month away could be +/- a week, but a milestone a year away could be +/- three months.
Yes. They need to not sell the next version too soon, but start selling soon enough to develop hype. They need to decide if they should invest in updating the previous solution one more time while waiting... They need to decide if the whole is worth the cost or should they develop a different project instead.
It depends on the level of detail in the plan. The article says to delay the plan, not to skip it. The goal of that is to avoid fleshing out the details of parts that are not relevant to the current focus (with "focus" being defined as the scope of the current sprint).
Of course, you need to know the direction of the project. But spending time on details that you cannot directly validate with implementation, risks both wasting time because of invalid assumptions, and risks wasting time because the requirements may change when you get around to implementing it.
Remember that "planning a sprint" gets very deep very quickly. It's not just about setting direction, it's about deciding display names, storage types and UX interactions. The point of the article is that that level of planning should not be done months in advance.
Your sprint planning is deeper than sprint planning some other places. Some places that at least claim to be doing Scrum will groom the backlog in one meeting, then choose which stories go on the sprint in another. Anything that needs agreement among developers is then tackled in pair programming or fairly ad-hoc "hack sessions". Implementation details, including details of code interfaces, user interfaces, and system interfaces, are left up to the contributor unless there's a need to align effort before that story is done.
The agile manifesto is so terse and carefully written -- every word is there for a reason -- and so many people ignore the words they find inconvenient.
"Responding to change over following a plan" specifically says "over", not "instead of!"
They even say it twice, to avoid misunderstanding: "That is, while there is value in the items on the right, we value the items on the left more." In a document that's only a few sentences long, to repeat a meaning is a little like shouting.
Don't value your plan more than your ability to respond to change. But do still have a plan! How will you know if you're responding to change without one?
This is how you end up with poorly architected code that will be so difficult to change that development will eventually slow down, stop, or begin going backwards.
Not in my experience. If you are agile, you can fix the architecture as needed. You end up with poorly architected code by deciding to continue building on top of something when it is no longer suited to purpose. That can also happen as a result of big investments in planning. That investment makes it very hard to rationalize architecture changes when the requirements change mid project. Which they very often do. No amount of planning protects you from the unpredictable.
This seems like it's really narrowly focused on the software developers' part of an agile project's lifecycle. And I agree, within that scope, you should absolutely shut up and plan the sprint.
Product strategists, though, can't afford to be so shortsighted. And most companies don't (and maybe shouldn't) have such a clear separation of roles.
I've found that, when your job involves wearing multiple hats, the trick is to pay attention to what hat you're wearing at any given moment. Don't let them all get muddled up. When it all blends together, then the smallest unit available to you to boil is the whole ocean.
Awful advice. This is how you build terrible systems and avoid having to ever look at yourself for building useless crap. You're never looking anywhere but your feet expecting not to run into a wall. Yes, you probably shouldn't build something massive without ever getting feedback, that has nothing to do with sprint vs project planning.
People in this industry need to let the worlds "lean" and "agile" just die the death they deserve because they mean literally nothing at this point. It's nothing but lip service for SEO and junior managers.
A plan is not a decision. Creating a plan is not tantamount to deciding too early, it's just a good idea.
Committing to the plan (especially a long range one) with no intent to deviate is just stupid, and is the critical flaw of Waterfall.
Agile is about responsiveness. In order to be responsive you need tight feedback loops (a reason the idea of sprints and scrum are ok). But are you responding if you don't have a plan? You aren't responding, you're reacting. Make the plan, but don't make a commitment too far out (commitments are kinds of decisions). Respond to changing circumstances and don't let yourself get caught in an action-reaction loop, because you can't react fast enough. At some point you have to think and respond.
Also, "decide as late as possible" is better read as "decide at the last responsible moment". If I know that I can get someone hired in two weeks, I can, in the first phrasing, wait until 2 weeks before someone retires to start that process. But it's not responsible. The last responsible moment is probably 2-4 months (at least) before the retirement so people can get trained up and moved around.
I’ve seen a few articles posted recently about people bucking at jira, thorough upfront planning, etc. and saying we’re all doing it wrong. The projects I haven’t been rigorous with designs and project execution are the projects that went wrong.
I always find myself curious about the author and how they arrived there. Are they junior? Have they worked on multiple teams and/or large projects before? Are their managers terrible?
What you actually need is pragmatism and not dogma.
I have a hard deadline external to the company by which time the project, and it is a project, needs to be done of else the company can't sign up new customers.
It would be pretty irresponsible of me to only focus on the next two weeks until it was done as I'd have no way of really knowing how close or far it is to completion, how we're tracking and if we need to make adjustments.
We did a rough plan. We worked to it. We evaluated how we were tracking and we shifted things in sprints around accordingly.
The benefit of agile is introducing short feedback loops that provide information you should be acting on.
Good project management also provides benefits.
Take the benefits from both, or leave one set of benefits on the table.
I agree with the author that ideally, you plan only the next sprint.
Unfortunately, I'm afraid this doesn't survive the real world, unless you're a small consultancy w/ a great relationship w/ your client, or you're doing work on your own.
A company that is any larger cannot afford to rely only on collaboration, it's impossible to have entire teams collaborate w/ each other since the communication overhead explodes [1]. I don't think you can make everybody converge down the line without some planning ahead and definition of some contracts.
Another situation is if you have any kind of external deadline (e.g. regulated companies). You can't answer with "we'll figure it out, trust me", you need a plan - even if you happen to have to change it.
The point of the Agile Manifesto was responding to change OVER following a plan - not that you shouldn't have a plan in the first place.
This might be good guidance to avoid getting too deep in the weeds and avoid starting work. Might even avoid the whole "waterfall with sprint" trap, but I'd argue it's quite important for the entire team to have an idea of where they are going, generally before you start work. Agile Roadmapping is a thing that works. Feature prioritization, generally, helps to get the highest value things out the door first, etc.
The team doesn't necessarily require those waterfally milestones - it could be as simple as a value delivery statement ("Fastest shopping cart west of the Mississippi" or something) - it doesn't have to be a whole big deal with specifics, but without it i'm not sure how you determine the sprint goals, let alone what the specific technical tasks or stories are going to be without that shared context.
Lastly, I'd definitely recommend some sort of longer-term roadmapping exercise up front to determine what the high risk things are so you can solve them first, and test the big hypotheses quickly. That would be a nice lean thing to do.
Plan the sprint, sure. Be militant about it if you want. "Shut up and plan the sprint!" But the sprint is not the goal. Delivering increments of value is the goal. The product owner looking at the sprint goal needs to know what they're trying build, and a longer vision can help.
I'm not sure the author meant to sound so short-sighted and maybe was expressing frustration around customers getting so wrapped up in the "plan" that they end up waterfalling / micro-managing to death, so I get the point, but there's a way to do this that doesn't sacrifice the plan, too. They probably know that.
if you are only planning 2 sprints ahead, what if you need to build a feature which requires a lot of requirements gather, user research, even before materializing into tangible stories that developers can execute?
If you fail to look beyond that myopic vision, then when a tru need materializes, it may be too late to react.
I was wondering this as well. This all seems like the complete exact inverse of shaping in some ways and reenforces some others. Generally for small, extremely skilled teams, the Shape Up methods are hyper efficient and can focus on a product/engineering goal, notably it forces management to define deliverables in clear and specific terms that outline a developer's jobs to do and semi-long term goals. This allows the dev to work on building something in whole rather than parts but with defined time and outcome goals. (~6 Weeks in the Shape Up Guidelines). An interesting Shape Up Guideline, (also seen here) is the idea to avoid excessive long term planning or backlogs.
Agile is shown as God in this post but..why is Agile so good according to the author? The main advantages is the ability to course correct fast and be reactive but if you are making good work, Agile blinds you from seeing the whole product and causes bad, segmented, low-focus mistakes. If you are TOO agile you may build a DB/API/Logic/Data structure that is too narrow and limiting of the eventual goal.
Personally the real danger of this thought process is that it limits developers from understanding the whole of what they may be building. That's downright draining, makes every sprint seem like going to a run to nowhere. This method and plan seems just like an excuse to let management do less and define and shape work to be done less so that you can avoid the dangers of 'Waterfall'.
" Agile blinds you from seeing the whole product and causes bad, segmented, low-focus mistakes. If you are TOO agile you may build a DB/API/Logic/Data structure that is too narrow and limiting of the eventual goal."
I disagree, your goal should be agile too!!!
It has to always change !
ShapeUp TLDR;
>> Backlogs are big time wasters too. The time spent constantly reviewing, grooming andorganizing old ideas prevents everyone from moving forward on the timely projects that really matter right now. <<
this may work for internal teams building internal tools, but there's very little chance to find an enlightened client that will sign a scopeless development contract.
and it sounds awfully close to being body rental, with all that it entails.
Over 60% of software projects go over budget, and often by a lot. What's unprofessional is giving estimates for anything that is not small enough to get done in a couple of weeks. You should also understand why someone asks for estimates. They are trying to manage risk. There are far better ways of managing risk than "Brad said it would take three months". Start out by calling it what it really is. It's not estimating. The end goal is risk management.
> Over 60% of software projects go over budget, and often by a lot.
That's so few!
Most companies fold, most construction projects go over budget too. I don't see anyone saying we shouldn't try to get good at budgeting construction, or estimating & planning a project before starting.
We're not just managing the risk of failure, we're managing days & dollars! I don't understand why software engineering is the one form of engineering we don't think we need to be able to plan.
If you are building something you have built before: a bridge for example, then by all means use your prior experience to budget and plan.
The majority of what software engineers build is new territory. Estimating and planning are an attempt to manage risk. And a really bad way of managing risk when you've never built that thing before. Actually starting to build the new thing in question will give you far more useful information about possible time and budget than any amount of planning. And you still won't get it very right. We need to stop pretending that building completly new things is actually possible to estimate. We cannot predict the future and trying to do so is unprofessional.
> The majority of what software engineers build is new territory.
I think this is where we just fundamentally disagree.
I can send you research about zero defect software, or the economics of software quality, or even research of across many disciplines how delivering value earlier is not in conflict with planning. Time spent planning in all cases saves time and money. However, if you think it's all going to be invalid because of your perception of new territory, then our discussion probably won't go very far.
I encourage you to read Capers Jones and Tom DeMarco, but they're not new.
Time spent planning does not always save time and money in all cases. We have all seen people who love to build prototypes deliver something working while the guy who insists on planning still hasn't delivered anything. And the guy who built the prototype comes with a list of valuable questions discovered only through the act of building. You wouldn't build a skyscraper in the middle of a busy city without loads of planning, and you wouldn't do loads of planning to build a new product feature for a web application when you are told ahead of time that requirements will change during the build. Knowing which is which doesn't come from reading. It comes from experience. As much I do loving reading about software engineering, like everything else in life you've got to get your head out of the books and ride that horse eventually.
"ride that horse eventually" - I'm pretty well on with the horse and riding part, and I make plans I'm held to all the time, from investors, CEOs, and customers. I would have been fired long ago (and have been fired in the past) if I didn't make plans, committed dates, and contracts with customers that managed scope, schedule & budget (see: all of professional program management).
Please look up the papers on planning & quality & cost & schedule from Capers Jones. The textbook is literally titled "The Economics of Software Quality". Tom DeMarco's classic is "Controlling Software Projects".
I'm trying to give you a starting point to see that it's not even my anecdotes, but repeatable studies that show this.
Your examples are that requirements change, and it's hard. Of the few buildings I have built, the requirements changed drastically there in all cases, so I can only say that software engineering is not magically different. After decades of practice with software engineering my experience of planning lines up with the textbooks you can find on the topic, and not this blog post.
And after decades of experience myself I've seen the vast majority of plans become useless the further out they try to predict the future. I guess we'll have to agree to disagree about the value of learning by actually starting to build the new thing that no one has ever built before. Plan away if that suits you. I'll be building and learning about the domain in the meantime.
Planning upfront takes less time than not planning up front, in all cases. You can plan to do a prototype, you can change your plan later, the activity of planning itself produces clarity.
Reducing the argument to "planning away" is the same as me saying you're "flailing around like an amateur". It's not what you're saying though, so I wish you'd see this thread as a discussion rather than something to win with gotchas.
Exactly. You could learn about when and how to be more efficient when planning isn't called for.
> Planning upfront takes less time than not planning up front, in all cases.
It does not.
> you can change your plan later,
That costs time and money. Maintaining and changing a plan does not come for free. The more complex and further out your plan reaches, the more you'll have to change it.
> the activity of planning itself produces clarity.
Yes it does. And sometimes the activity of building a prototype produces more clarity than a plan.
I like Capers Jones too. His recommendations are not the silver bullet you are treating them as. He sells software used for estimating, measuring, and planning... so of course he's not entirely unbiased. A lot of what he does applies very well to large organizations where large and frequent pivots are nowhere near as frequent as when working for a startup while exploring the features of something that is not evolutionary, but revolutionary. Check out this article where Capers Jones shows that Agile is one of the quickest and cheapest methodologies out there.
And where he concludes "Overall the Agile family and the methods that emphasize speed have achieved their goal, and they are fairly quick. The methods that emphasize quality such as TSP, RUP, and CMMI 5 have also achieved their goals, and deliver very few defects. No single method appears to be a universal panacea that can be successful on every size and kind of software application."
Huh.
> I wish you'd see this thread as a discussion
You're not even willing to consider that planning too far ahead can waste time and that planning does not optimize for speed. You are giving absolutes like "in all cases". If you want a discussion then you have to be willing to learn yourself. In my experience there are times when planning is essential. And times when planning, especially out more than a few weeks, is a waste of time. If you want to trade tips, great. If you want to continue to insist planning is an absolute, no thanks.
Great response, thank you! It seems I'm rubbing you the wrong way, or not understanding you well enough to respond in a way that encourages conversation, and that's crap. I'm sorry.
> And where he concludes "Overall the Agile family and the methods that emphasize speed have achieved their goal, and they are fairly quick. The methods that emphasize quality such as TSP, RUP, and CMMI 5 have also achieved their goals, and deliver very few defects. No single method appears to be a universal panacea that can be successful on every size and kind of software application."
This article must be taken in the context of his research in general. If you're optimizing for speed of initial delivery, rather than TCO, then any method with less planning delivers faster and costs less. However, if you assume your software lasts longer than ~year (as pointed out in my initial reference, and his much larger work "The Economics of Software Quality") then they perform much worse. The actual payoffs in most of his research is around 8 months of development.
Being "quick" is basically a jab from Jones about agile project quality.
> You're not even willing to consider that planning too far ahead can waste time and that planning does not optimize for speed.
I'm willing to consider it, I think. My experience has shown the opposite for any project involving more than 1 person that a customer uses for more than a year. The research I can find shows me the same. Most projects I embark on tend to live in the market for longer than a year, and thus quality, TCO, and delivering on time matter.
I guess I don't see how you manage a team, a project, a customer, or investors where you need to hire people based on work you need to do, if you're only planning out a few weeks at a time. I don't see any case where you're going to get VC funding, exec sponsorship, etc (i.e. manage the risk and expectations of capital) by saying "we'll see where we are in two weeks".
This is where I start to fall into saying things about professionalism, as to me the difference between a professional and an amateur is exactly this capital and risk management.
> If you're optimizing for speed of initial delivery, rather than TCO, then any method with less planning delivers faster and costs less.
I feel like we are getting somewhere now. For some projects speed of initial delivery is by far the most important factor. Which is why it's so important to qualify your claims instead of using absolutes like "planning always results in..."
> However, if you assume your software lasts longer than ~year (as pointed out in my initial reference, and his much larger work "The Economics of Software Quality") then they perform much worse. The actual payoffs in most of his research is around 8 months of development.
Why would you assume that? Lots of software these days lasts far less than a year, and that's intentional. This is not the 1970s when building software was an expensive and rare thing. There are literally hundreds of millions of applications out there. And billions of dollars being pumped into startups, the vast majority of which will throw out their initial idea and eventually pivot when they learn what their customers really need. Learned by building quickly and delivering something that works well enough to get it in front of people to see how they actually use it and how different that is from what you had envisioned (or planned). What comes to mind is the cardboard box that the kids play with while the new toy it came in sits ignored.
> Being "quick" is basically a jab from Jones about agile project quality.
Why would it be? Unless you are heavily biased towards software always being high quality. Which shows a lack of flexibility and a lack of professionalism about optimizing for what is important to your client or employer. If your employer needs a project to optimize for initial speed, then focusing on quality and planning is unprofessional and irresponsible and putting your own personal preferences ahead of what the company has decided it should optimize for.
> My experience has shown the opposite for any project involving more than 1 person that a customer uses for more than a year.
I'd chalk some of this up to confirmation bias. But at least this far better than "always". At least we are starting to add qualifiers like "uses for more than a year"
> I guess I don't see how you manage a team, a project, a customer, or investors where you need to hire people based on work you need to do, if you're only planning out a few weeks at a time
You manage the team by focusing on building and learning from what you build. Let's say you plan for an easy build over a few months and hire 1 person. They start building and as a result of building a new market opportunity is discovered so new requirements come to the surface. Some of these requirements are difficult to implement and will require 10 people over a few months to build. But that is where the company now wants to pivot. How did your plan for hiring 1 person help you here? I'd say it gave you a sense of false confidence. What would be more professional and frankly more honest is to say "we don't know yet". That can't go on forever of course. The entire point of favoring building over planning is that in the course of building you should be reporting back on new risks discovered. And new opportunities. It's not all downside. When you have a plan to take six months to build something and it only needs a month, well Parkinson's law is nothing new so we all know it will take six months.
Lacking a plan, and with a focus on urgency and speed, you are forced to ruthlessly eliminate requirements. Since you will get something built in a few weeks the chances of you getting the thing you needed in a month are much higher than the six month plan. The exercise of delivering working software every couple of weeks gives you laser focus on what your core offering is.
> This is where I start to fall into saying things about professionalism, as to me the difference between a professional and an amateur is exactly this capital and risk management.
If your only tool for managing risk is extensive and detailed plans that must be constantly maintained and that do a poor job of optimizing speed over quality, then you are not being as professional as you could possibly be. There are more tools available for managing capital and risk than "let's make a detailed plan".
If you are employed by a company where the culture is all about using plans to manage risk, then you've found a great fit for yourself. But you shouldn't look down your nose at alternative ways of managing risk and call them amateur. There are different types of risk and there is no silver bullet in managing them.
One alternative for managing risk is admitting that you don't know and just building the damn thing and learning from it. This method of managing risk is especially important for really new things that have never been done before. You want to fail early and often and that is the most professional thing you could possibly do on those types of projects. At least until someone comes up with something better.
I'll just end by saying thank you for making the effort to steer this to a higher quality and more nuanced conversation. I enjoyed talking about this with you, and now I have to move onto other things.
> Unless you are heavily biased towards software always being high quality.
This is the main discussion of his research, to show that quality and defect removal actually lower cost and increase speed. The tradeoff between time & quality is one he regularly shows is not a tradeoff at all, even for very short timelines.
> I enjoyed talking about this with you, and now I have to move onto other things.
Developing a plan is not waterfall. Designing an Architecture is not waterfall. Agile is about adapting to change as it occurs - how do you know if there is any change if you didn't start with a plan?
Good plans and designs account for risks and unknowns and anticipate (certain types) of changes and delay locking into assumptions until necessary. But not having a plan or design is not agile - it's failure.