Agile talk by Bertrand Meyer

Bertrand Meyer ran a very good webinar on Agile development on Feb. 19, 2015.

(The recording of the webinar has been posted here: https://www.youtube.com/watch?v=ffkIQrq-m34)

The content of the webinar is based on his new book: Agile! The Good, the Hype and the Ugly. The book is a serious critique of agile development approaches, practices, methodologies, and culture.

Here is a link to the book on the publisher’s website --

http://www.springer.com/computer/swe/book/978-3-319-05154-3?otherVersion=978-3-319-05155-0

It is a critical book that reflects on the good and bad things in the ways that people use agile development. He agrees that the Agile community has come up with many good principles and practices, but they have unfortunately rejected many good principles and practices from classical software engineering.

I am including by raw notes below.

Bertrand’s basic conclusions:

Positive:

iterative development is great, and Agile has been instrumental in getting development organization to adopt *short* iterations
freezing changes within an iteration is extremely valuable -- one of the best contributions of Agile
it is a very good idea to put testing in the center of software development activities, so every piece of code has tests

Negative:

the Agile community’s total rejection of “up front requirements and design” is a mistake
user stories are not a good substitute for good requirements and specification practices -- user stories are too detailed and specific, they create unnecessary complexity in development, and they don’t support product evolution
some ideas in Agile are oversold (test driven development, self-organizing team, open-space working environment, building a minimal marketable product)

On the whole, Bertrand is a proponent of Agile development, and his company uses many agile practices. But he is also interested in making sure that we don’t throw away many valuable ideas from classical software engineering.

(There are more comments by Bertrand Meyer on his blog: https://bertrandmeyer.com/category/agile.)

Bertrand Meyer is the head of the company Eiffel (creator of the Eiffel programming language), and he is also the chair of the software engineering program at ETH in Zurich.

----

Notes from Bertrand Meyer’s presentation - Feb. 18, 2015

Many software development ideas can either be labeled as “good” (we should use it) or “bad” (it doesn’t work). But in the case of Agile - it is a *combination* of the best and the worst.

We have applied agile things at Eiffel, and many of them have worked well. But in my consulting work, I have seen some of the not so good things.
I became a (proud) certified Scrum Master to be able to talk about these things from the inside.

Agile is everywhere -- it is difficult to find companies that are not applying some agile methods.

I will discuss agile ideas and practices from four sources: XP, Lean, Crystal, and Scrum.

First method was XP - Kent Beck

(very successful early -- but most people use Scrum today - it has vanquished all the others)

Lean - Mary Poppendieck

(from manufacturing disciplines)

Crystal - Alistair Cockburn

(a family of methods, less dogmatic, less extreme - he has tried the most to reconcile agile with traditional software engineering ideas)

Scrum - Schwaber and Sutherland

(most people don’t use “pure Scrum” -- most people use a mix of techniques from Scrum and other methodologies, mostly XP)

The rest of the talk is in two parts:

Key agile concepts
Assessment

Bertrand discussed the 12 Agile principles from the Agile Manifesto (http://www.agilemanifesto.org/principles.html) - Bertrand’s contention is that these principles are not necessarily the best ways to explain agile .

some of the principles are actually “practices” (6. face-to-face conversation, 12. team reflects regularly)
some are assertions (7. working software is primary measure of progress, 9. attention to technical excellence and good design, 10. simplicity is essential)
principles should be relatively independent -- but 1, 3, and 7 are all connected -- they all reiterate the importance of frequent delivery.

What isn’t in the agile principles that should be there?

testing should be there (not mentioned at all)
some of the assertions are wrong (for example, simplicity and maximizing the amount of work not done are not the same thing - and you often have to do more work to make something simple)

Bertrand has created his own set of 5 agile values:

Agile values:

new reduced role for manager

(not to tell people what they should be doing, day after day, hour after hour)
(protecting the team and other things are good)

no “big upfront” steps

(big upfront anything is deprecated, instead, start coding and building a solution, check yourself and check with prospective users, refactor, etc.)

iterative development

(teams don’t work on their own for 6 months)
(short iterations of a few weeks to 1.5 months - adapt to changes)

limited scope

(don’t try to do everything)
(negotiate the scope with the customer)

focus on quality, using testing

(testing has a central place in the development process)

Bertrand also has two sets of agile principles (organizational principles and technical principles):

Organizational principles

put customer at the center - involve the customer (XP: embedded customer - doesn’t work too well; Scrum: Product Owner)
accept change: (not “welcome change” - no one really is glad about change), accept change as a normal part of the software process - but change is a technical item... Mary Poppendieck and others deny the value of existing techniques for dealing with change but don’t suggest others - some slander of OO techniques
maintain sustainable pace: look out for death marches, long hours, keep developers safe (in the Demarco/Lister sense)
produce minimal software

produce minimal functionality
produce only the product requested (don’t work for reusability)
develop only code and test (not diagrams and models that won’t be delivered to the customer)

Technical principles

develop iteratively

produce frequent working iterations (and we used short iterations at Eiffel - 1 month releases)
freeze requirements during iterations (“closed window rule” - came from Scrum)

treat tests as a key resource (comes from XP)

don’t start new development until all tests pass (regression tests)
test first (more open to criticism)

express requirements through scenarios (not complete requirements documents, try “user stories”)

What about some of the common agile practices that are good or not so good?

User stories - they have taken the industry by storm - and not for the better

The “As a ... I want ... so that ...” formula is common. You pile up a bunch of stories to drive your development
But it is a fairy tale. Fairy tales are nice, but developing from user stories doesn’t really produce good systems.

User stories may be useful for requirements elicitation, but they are not a fundamental requirement technique.

(not abstract enough, they are often too specific)
(describe current processes)
(don’t support evolution of the system)

As soon as your needs depart from the existing behavior, you are in trouble.

The idea of implementing more features runs into complexity problems:

We would like to have “Lasagna complexity” -- adding a layer at a time -- complexity is additive (easy)
We often get “Linguini complexity” - if you try to pull out any part of the design, the whole thing will fall in your lap -- every feature interacts and is intertwined.

One person in AT&T (Pamela Zave) discusses the possible complexity of feature interactions.

Each user story may make sense, but when you add them together, you find you need a better, more abstract view.

“Closed-window rule” - during a Sprint, there is a rule that says -- we don’t allow new work to be added to the Sprint

This works! It works because we have short iterations (1 month iteration, the worst is that you have to wait until the next iteration for your great functionality). And sometimes brilliant ideas don’t look so great when you are sober (so it is better to have a great idea wait -- kind of a natural selection of ideas).
Of course, there are times when some functionality is really really important -- so there is an escape mechanism... The escape mechanism is to cancel the sprint
This is a significant contribution of agile methods.

Agile tells you “no upfront design” -- this is not good, because it causes catastrophe. Sometimes just a few weeks of analysis and design would have saved you from disaster. Agile folks are right that we often spend too much time and effort in analysis -- and we sometimes analyze in circles, spend too much time in the abstract.

I like the idea of “dual development.”

Early on: build infrastructure -- as a set of layers, easier to manage.
Later: focus on a “production mode” (creating frequent releases)

Beck talks about the problems of a negotiated scope contract -- I can promise a deliver date or a set of functionality but not both. Classic software engineering disagrees - you can negotiate a small essential scope plus some add-ons that will cost more money or time.

Assessment...

Ugly - shown to harm projects

rejecting upfront tasks (pendulum has swung too far from classical)
user stories are not a good replacement for requirements (not complete enough)
tests are not a replacement for a specification - because they only describe one case
Scrum Master who doesn’t code is a bad idea - want someone who is part of the team
TDD - as a development method, it goes too far, too haphazard
most teams need a manager, still need some direction
still need plans and other non SW artifacts - it is good to focus on the code, but don’t reject everything else
Luddite approach (YAGNI) dismissal of the concern for reusability

Indifferent -

pair programming - has about the same effect on quality as code inspection
open-space working arrangement
sustainable pace is nice, but it is wishful thinking
hard to decide on minimal functionality
planning poker - not going to change the world

Good -

Acceptance of change - really important - agile people have said it more loudly, but we need to have more technical mechanisms
frequent iterations have revolutionized the software industry already
emphasis on working code is healthy (not to the exclusion of everything else)
constant test regression analysis
no branching - agile folks have this right
artifacts like burndown chart are very helpful
daily meeting (it is OK if it is weekly for distributed) - the three questions are simple but extremely healthy - can’t make empty promises

Brilliant-

short iterations - a few weeks.. really major
closed-window rule - can’t add work in the middle of an iteration
refactoring (but not a substitute for design) - always question and improve
always have a test with each piece of functionality - “test with” (not necessarily test first)

The Agile community has a bit of an “adolescent attitude” that our stuff is better than anything that came before... but I think of it as a step in the 50 year evolution of Software Engineering, and they are building on some of the best work that came before.

Question and answer session:

Q. What kind of negative comments have you received from the Agile community?

I spoke with many of the Agile pioneers in creating the book. I shared the PDF of the draft of the book with people, and many said that they welcomed criticism. Most of the post-publication feedback was silence, but I did get some supportive email from Kent Beck.

Q. No branching - what is it?

In some of the agile literature, they say your code changes need to be made to a single trunk in your code management system -- everyone checks in to the main trunk. Some people disagree -- the git community is fine with some branching. But most agile experts think that branching is probably evil, or at least risky. Developers shouldn’t create extra code branches for no good reason.