You may find the following useful if you are an architect in an agile team defining your role, or if it is unclear if what the team delivers conforms to the organisation’s architectural principles, or if you are being slowed down by an overload of communication relating to official approvals for delivery.
We got interested in this problem because a developer created some code that was inconsistent with a principle that the architect had incorrectly assumed the developer would follow. We were resource constrained so needed to find a more effective way to solve the problem than having formal meetings to approve the details. The analysis suggested a resolution to the long-running discussion about whether architects should be reviewing code, suggesting that in an agile context it is necessary to give sufficient protection against misunderstandings on architectural principles like we had experienced.
More generally we also wanted to make a recommendation on what an architect should do in an agile team. Agile is meant to give better business outcomes by replacing the two stage model of relating a business need to a detailed specification and then relating a detailed specification to delivery with a single stage model where the business needs are matched directly to delivery, facilitated by the product owner. Because it’s a single stage, it then reduces the management overhead and iteration becomes more practical. The product owner needs to understand the business needs at the one end, and inspect demos of the delivery outputs on the other end, and then specify what needs to happen next to progress. If the product owner doesn’t inspect the demos to make sure they’re meeting the business needs, then the process doesn’t work. There are techniques to improve this, such as getting real users involved, but the product owner needs to make it happen. This then exposes the product owner directly to the fundamental challenge of identifying the business needs.
Similarly if the architect does not do code reviews to makes sure the delivery conforms to the architectural principles then the process doesn’t work. Similar to how the product owner is responsible for improving the feedback information through demos by using techniques like getting real users involved, the architect can get inspections done by other people such as technical specialists, but the architect needs to make it happen. This then exposes the architect directly to the fundamental challenge of understanding the organisation’s architectural principles.
The problem was especially interesting because agile teams can tend to focus on the product owner’s fundamental challenge of relating business needs directly to demos of functionality, but neglect the architect’s fundamental challenge of relating architectural principles directly to validation that delivery meets them. The organisation can’t give the team the full freedom to be agile on the business side but maintain old fashioned controls on the architectural side, so the solution is valuable. The solution concepts below concentrate on the architect’s challenge, we don’t intend to give the impression that the architect only thinks about principles and compliance to them all day long, as he or she has other things to do that there isn’t space to discuss here, but without resolving this challenge, the other things are unlikely to be successful.
The solution concepts were
For improved agility the organisation must stop requiring central approval of individual solution designs
When central approval of individual solution designs is removed, to avoid chaos both the central organisation and the teams must compensate by improving governance at the level of principles, patterns, guidelines and standards
The architects in the teams are responsible for understanding the architectural principles, patterns, guidelines and standards that are relevant to the team’s work, and negotiating modifications to those with the central organisation
The architects in the teams are responsible for assuring that what the team delivers conforms to the relevant principles, patterns, guidelines and standards
The solution in practice
If a central body approves individual solution designs, we have the following disadvantages:
it is slow
it is ineffective as a control because if there is a useful level of detail in the solution design, it is unlikely to get sufficient attention from a central body to reliably identify issues
This could be especially challenging for identifying conflicts between solution designs
it is even more ineffective because the end delivery won’t match the approved solution design exactly and it is impractical to have constant reviews of changes to the design
it subverts attention to proper governance of delivery either by the team or by others, because as above the approved solution design will be out of date versus the reality of delivery, so any attempts to review delivery will be to compare to a solution design which is wrong. This causes a chain reaction of inefficient discussions and real problems could get lost in the noise.
it elevates the solution design to the level of a deliverable and consumes resources and optimises it for that purpose, rather than minimising resources and optimising it for its real purpose which is guiding better delivery and operations
it subverts proper attention to the architectural principles because solution designs are being approved as well.
Lack of attention to the principles makes it challenging to integrate a large organisation because it is at the level of principles that challenges that require cross departmental support can be solved. It’s difficult for one department to get other departments behind a large initiative justified only by the solution design for a specific project
Lack of attention to the principles in favour of solution designs also biases towards focusing on initial project delivery versus ongoing operations. It doesn’t matter that the solution design has a section for ongoing operations, because by the time it’s delivered, the design has changed from what was approved, and there isn’t enough time to solve it, and it’s out of reach of the controls. And once the project has completed and the system is in production, if there has not been good attention to principles then there will be a disconnect for integrating the output of the project into the organisation.
If we eliminate the requirement for central approval of solution designs, and instead we have a central body approving principles, patterns, guidelines and standards and the individual teams have the responsibility to deliver based on those, we eliminate the disadvantages above and have the following advantages:
the set of principles, patterns, guidelines and standards then become far more important, because they are the focus for the organisation to control the delivery
the team is free from friction to evolve the solution architecture as needed
there is no baselining and approval of the solution architecture itself
typically this is a confluence page and can be updated at any time
There is a common misconception that there is “less” documentation in an agile project versus the straw man waterfall, this is better reframed as follows:
We don’t deliver the solution design for approval, instead we design to deliver the solution for ongoing operation, so we eliminate the overhead of documentation boilerplate that relates to it being approved, because solution designs are not deliverables
The team is free from friction to refine the designs for internal discussion and for a substantial project it’s likely that by the end of the project there will be more useful valid design documentation available than for straw man waterfall
In a large organisation the importance explained above of the principles means that there will be high level of quality in the organisation-wide architectural principles documentation, and this will need to be understood and updated where relevant for each project. Interacting with this is not just “documentation” work, it is research, insight, vision, judgement, negotiation, compromise and justification applied both patiently and rapidly to material that is for the organisation as a whole, not for the individual project. That makes it far more challenging. This material is the basis of a large organisation being able to be agile, and the content is absolutely not trivial. It needs to specify clearly how the rule of law will apply fairly to independent projects and team. Framed this way we can see the benefits of the freedom enabled by preventing any unjustified restrictions being specified, but we can also see what goes wrong if we lack the rules that we need. Following this to its logical conclusion, a delivery approach with a central body approving specific solution designs looks analogous to a political system where a central government makes executive decisions about everything. The excessive centralisation in the political case prevents people having the freedom to choose the best actions and the country is poorer. When authoritarian countries make the transition to capitalist democracy, if the centralised executive decisions are replaced with nothing rather than with the rule of law, then there is chaos. Similarly, when making a transition to agile, if an organisation removes approval of solution designs but does not replace it with the rule of architectural principles, then there will be chaos. It is not trivial for a team to understand and interact appropriately with an organisation’s architectural principles and simultaneously assure that delivery is consistent with them, and so when people say things like “in agile there is no documentation ha ha ha” they should be cautious about neglecting their responsibilities to avoid chaos.
Straw Man Waterfall is where the team interacts with the organisation by delivering to an approved design created before delivery starts. To make the straw man extra flammable we set expectations that the solution can be delivered at a precise time and that there will be no changes, while simultaneously deliberately creating an inadequate and ambiguous specification in a volatile context. Then no-one checks anything until the budget runs out.
Reasonable Waterfall is where the team creates design change requests which are approved by the organisation as the team learns more or as circumstances change. If the team has good judgement for creating the initial design and the change requests and the organisation is focused enough to understand them, and the volatility at the solution design level is low enough that the overhead of approving changes isn’t too bad, then this can work fine.
Simple Agile delivery is where the team is apparently free to do whatever they like to deliver business value. All restrictions have been removed according to the agile training material. There are of course principles to follow, but in contexts where it’s easy to know what these are because they are just industry standards, then this could work because conforming to the principles will just look like doing the development work properly. In an organisation where it’s not straightforward to know what the principles are, challenging to get consensus to change them, and it’s essential to success to get other teams to do things, then this approach is comically ineffective. There’s no structure to interact with. Chaos ensues, then an old school delivery person who was good at Reasonable Waterfall suggests that we should carry on being agile but all the solution designs will be approved by a central team to make sure people do the right things. There will be JIRA ticket conventions for change requests and so on.
Principle Driven Agile delivery is where the team interacts with the organisation via the organisation’s architectural principles, and the teams have freedom within that. The freedom enables dealing well with changes at the solution design level and the problems of managing the interactions between teams and governance of design are solved by reshaping them into a problem of architectural principles interactions that can then be solved by a high skill architect using the principles of law. Volatility at the architectural principles level (a higher level than solution design volatility) is then addressed directly - it can be a project destroying force, but with this approach it does not hide behind solution design approvals. This approach allows teams to own their own designs fully, and address the challenges directly.
The architect is responsible for understanding the architectural principles, patterns, guidelines and standards that are relevant to the team’s work:
Where these need changing or augmenting to support the team’s work, the architect must build consensus in the broader organisation and get the updates approved
The architect is responsible for making sure the team’s delivery is consistent with the above and is as effective as possible
There are advantages in having an architect who can simultaneously cover the full range from negotiating changes to architectural principles through to reviewing code, because this reduces the risk of missing identification of code that didn’t conform to principles, or failing to correctly negotiate principles because the code wasn’t understood
The architect is NOT responsible for delivering a solution design, because the solution design is not a deliverable. However, because the architect is responsible for making sure the delivery is consistent with principles and is as effective as possible, there is usually design material that the team creates and if there is any doubt about ownership then the architect needs to do it or arrange it. There are plenty of times where it is more appropriate for a developer or SME to create specific design material.
Other Considerations and References…
Patterns are particularly useful to the organisation’s central architecture team to clarify what kind of solutions are acceptable. Approval of generic patterns gives some of the benefits of the approval of solution designs without the disadvantages. It also becomes important to confirm what is considered an anti-pattern to make it clear what designs are unacceptable. A central architecture team that wants more specific control could require approval of more specific patterns than one which finds it sufficient to rely on more generic patterns.
The architect’s job has both a high level of responsibility and requires a high level of skill, and Principle Driven Agile only succeeds if the architect is successful, so to avoid any gaps occurring:
the architect should peer review on specifics with multiple other architects, not just architects assigned to the same team
the architect should deliberately apply skepticism to any plans to avoid overloading the effort - this deliberate application may help counterbalance the bias some architects have towards creativity and away from task planning
There is significant benefit in the architect being able to review code. In a typical business intelligence team there will be SQL, bash and Python code and it is well worth the architect being competent to review these directly. If the code level detail is in a form not easily understood by the architect, then the architect needs to make sure someone else competent is reviewing it to make sure the delivery conforms to principles. It is essential that these reviews are prompt so that any disconnects from the architectural principles are discovered as early as possible.
The architect needs to drive backlog grooming sessions, and avoid problems where disjointed user stories cannot be related back to a coherent solution or to architectural principles. The architect will spend a significant amount of time working out how to minimise the number of story points that are needed to solve the problem. While the architect and product owner spend the most time on this, everyone could be involved to some degree. It doesn’t work for the backlog grooming preparation and related architect’s negotiation on principles with the organisation to be measured in story points, because these activities can’t reliably be estimated in story points. There are valid arguments either way about whether putting story points on this kind of preparation activity also gives a potentially misleading impression of progress, but they are moot because the activities upstream of backlog grooming can’t be estimated to the same degree that the activities downstream of backlog grooming can be, to the extent that prior to backlog grooming you don’t even know whether you have the correct user stories.
Design work can be done as part of a user story, and there’s a straightforward question which determines what can be designed as part of the story versus needs to be addressed separately. While acknowledging that we can’t know this completely reliably, the key question is whether the scope of this user story can be significantly changed by this design work. If it cannot, then it is best to do the design work as part of the user story, because then it is done at the last responsible moment. If the scope can be significantly changed, then the user story is insufficiently specified for inclusion in a sprint, and instead the scope must be better determined either with more solutions architecture and backlog grooming work, or by a spike solution, or by another related story. There is an art to determining what should be solved with more solutions architecture and backlog grooming versus what should be a spike solution or another related story, because focusing on solutions architecture and backlog grooming can lead to a more elegant solution but it might be based on incorrect assumptions, whereas accelerating work on a spike solution gives information about how it really works but might result in unnecessary work, and it can be very tempting to make excessive spike solutions because they are fun. If there’s a related story that is already confirmed as necessary that could be used to confirm the scope, that avoids the risk of overdoing spike solutions and optimises for the benefits of iterative development.
The architect can work on spike solutions or other user stories directly as long as this does not conflict with leading the team to work out the minimum effort necessary to solve the problem while understanding and negotiating architectural principles, patterns, guidelines and standards and assuring the team delivers to these.