Descaling Organizations for Scaling Agile – Top takeaways from Agile 2014 Conference

In the Agile 2014 conference we heard many speakers echo the same signal – “Lets descale the organization instead of scaling agile”. There’s an increasing realization that instituting new models like scrum of scrums to force fit agile processes to corporate hierarchies isn’t going to work. Models are rigid and linear whereas human systems are non-linear and need more flexibility. We need to take a hard look at restructuring our organizations to become less siloed and hierarchical.

Its not the frameworks, its the culture

Its not the frameworks, its the culture

Many frameworks are proposed. The fact that there are so many makes the magnitude of the challenge evident.

  • Scrum-of-Scrum (SoS)
  • Large Scale Scrum (LeSS, Larman/Vodde)
  • Scaled Agile Framework (SAFe, Leffingwell)
  • Disciplined Agile Delivery (DAD, Ambler/Lines)
  • Spotify “Model” (Kniberg)
  • Scrum at Scale (Sutherland, scruminc.com), meta framework

The Agile 2014 conference can be viewed as the watershed year where everyone agreed that its not about framework – Its about Culture.

Descale your organization

Change in culture needs enterprises to stop paying politics tax. Politics tax is defined as the time spent on CYA. We need to allow people to choose to unleash their potential and stop fearing failure. Fear results in lies and lies result in mistrust. We need to build fail safe relationships.

Olaf Lewitz in his passionate appeal proposed a very simple manifesto “We value people”

Rounabouts Vs Traffic Signals

In an earlier posting I had mentioned that we need distributed decision-making in more decentralized and flatter structures to deal with rising Business VUCA  . Bjarte Bogsnes echoed the same thoughts. We need Theory Y organizations to deal with business VUCA. We need to trust our people instead of controlling them with rules and policies. Rules and policies require monitoring and enforcement. Like traffic signals- a rule based system is often inefficient. Roundabouts are self regulating value based system. A roundabout trusts users and demands self-discipline and its scalable.

Budgets force decisions to be taken too high up and too early. Budgets often lead to people spending money even when there’s no need just because they have the budget. Some of them do so to avoid losing the budget next year if they don’t spend it this year. Expense allocation should be need based. We should trust people to spend only what is needed and not have them ask for a budget. Read more about Bjarte Bogsnes’ presentation here.

Squads,Tribes,Chapters and Guilds

Spotify offers a fascinating model to grow without losing the benefits of being small. Spotify has kept an agile mindset despite having scaled to over 30 teams across 3 cities. This model has gained a lot of popularity in the agile community.

The nomenclature “squads,tribes,chapters and guilds” indicates the resemblance of the proposed structure more with communities than corporations. We have already seen the advantages offered by communities over corporations in this earlier posting.

You can read more about the Spotify model here

 

Self Organizing Organizations

Success with Fedex Day got Trade Me a New Zealand company with 80 people in their engineering team thinking what if we allowed people to organize themselves and do what they wanted to do all the time? Further inspired by the Spotify model  they decided to let people choose what they wanted to do. They had product owners define 11 work streams for which 11 Squads were formed by allowing employees to choose the squad they wanted to join. They limited the size of each squad from 3 to 7 members . They also ensured that each squad was required to be self sufficient and co-located.

This unique social experiment inspired and empowered people to self-select. This resulted in stable, focused teams that delivered better. People were happy, as they got an opportunity to do what they wanted with whom they wanted. Read more about this experiment here.

T Shaped Skills

Decentralization and distributed decision-making demands cross functional skills. These changes also demand new skills as covered in this earlier posting. Augusto Evangelisti shared his experience building an extremely cross-functional team with each member acquiring skills in all functions. T shaped skills characterized by deep skills in one functional area like QA and breadth of knowledge in other functional areas like development, product management and operations. For cross skilling; communication, curiosity, respect and empathy are more important than technical skills. No one works alone in these teams. E.g. at least three people work on writing a user story. Shared activities made people more accountable and resulted in better quality -0 bugs ,3X efficiency, shared understanding of goals and mutual trust.

Mob Programming,

Woody Zuil shared his experience at Hunter Industries where he is running their engineering team using what he calls as the mob programming model for the last 3 years. The team members self selected as they were attracted to work together due to awesomeness of the vision. The whole team always worked on the same task whether it was a development , testing or deployment task. That way developers got to think like testers and testers got to think like IT operations. Overall interaction improved which brought in more kindness, consideration and respect. No one was blocked waiting for inputs as all the concerned members were in the same room. There was only one computer and two projectors. The person at the computer worked as the driver and all others worked as navigators. You can see a video of the team here.

The fact that Hunter Industries is supporting this model makes it evident that it must be working. Best requirements, architecture and design emerge from self organizing teams. There’s no duplication of code resulting from two developers independently working on similar code in their cubicles. No duplication means less code and low technical debt.

DevOps is a cultural change

In a session by Pete Cheslock the same thoughts were reiterated in a different context. DevOps often gets mistaken for a practice that can be implemented by appointing a few “DevOps Engineers” and buying a set of tools to do continuous integration and delivery. The bigger point gets missed out. DevOps is a cultural and professional movement. You can’t solve social and cultural issues with tools. It’s a journey. No need to hurry to get everyone there today. We need to get everyone’s buy in . It’s a slow process. We have to actively cultivate trust and learning. We have to allow teams to fail and learn. We need to march with a conviction that the direction is right.

Modern Corporate Structures and Skills Needed- Takeaways from Agile India 2014

Modern Organization Structures

Modern Organization Structures

In my last blog post we took a fresh look at hiring in the context of changing organizational structures. In this blog we will see how these changes are demanding new skills and strategies. Most of the content here is based on various presentations at Agile India 2014, held in Bangalore in February and March.

Organizations Powered by Cross-Functional Collaboration

Organizations are progressively moving towards less structured , more collaborative forms from Agile to Holacracy to Lattice to NoManager. Here are some of the thoughts on leadership presented by Tathagat Varma –

1)   Adult Supervision- Management is a service available on demand for employees if and when needed. Otherwise employees are work independently without any management oversight.

2) Horses for Courses – Use networks for innovation agility and speed. Use hierarchies for efficiency, predictability and scale.

3)   Distributed and Rotating Leadership – Shared decision-making across the board. No permanency or carry over of roles. 37 Signals and Valve use rotating leadership model.

In these flat organizations; individuals don’t rise up but move laterally in multifunctional teams. The skills required are T shaped with depth in one or two areas accompanied by breadth in many areas. Open allocation is a concept where employees choose what they want to do.  Valve, GitHub and TreeHouse are some of the well-known companies using open allocation. Holacracy focuses on structures that establish lines of communication to get work done. Holacracy does not focus on roles and designations of individuals. In 1855 an organization chart for NY Railroad was drawn on the basis of how work happened. Not surprisingly it resembled the rail network and traffic patterns and not the hierarchy.

Innovative Culture Nourishes Learning Minds

Social exchanges and idea flow facilitated by an open organizational culture help individuals to innovate as per Alex Pentland. We are better off creating the right culture than importing innovation by hiring innovative individuals. Idea flow between individuals is more important than individual creativity.  True innovation happens when people are allowed to experiment and tinker without the fear of failure. In Etsy the developer who breaks the build is rewarded a sweater with 3 arms – thus providing feedback in a sportive manner.

Quick successive experiments is a continuous and ongoing process in such organizations. No amount of classroom training or attempts to implement a tool /process to innovate can replace the iterative , trial and error nature of innovation. Growth often hinders innovation as processes are brought in to lay down rules to bring discipline and make things more predictable.  The sheer complexity of a growing organization impedes the performance of high performing innovators who relish the messiness of creativity and dislike processes.  Complexity often leads to unknown consequences- taking the same action in the same situation doesn’t yield the same results. HP Laserjet business is an example of an organization where the average time spent on innovation increased from 5% to 40% in 3 years. This was a result of continuous improvement effected by daily use of practices like “Improvement Kata”. Change has to be continuous ; hence we need to establish a sense of urgency.

To give latitude and independence of implementation; define the end state but not the implementation. Let the team decide how it wants to reach the desired end state. For software development the project paradigm needs to change.  Building software is quite different from building a bridge where the end state is known. Enormous amount of time and money is spent on defining end state or scope of a software which is essentially undefinable. Jez Humble says that 50% of efforts are spent on this fuzzy front end.  The “project” simply doesn’t start till we have some semblance of scope defined. Also the software is not really used till its “ready” for release. This leads to what Jez calls as the Water-Scrum-Fall model. Unlike bridges or dams; software can start yielding value much before its completely “ready”.

Skills 2.0 – New Age Skills Common Goals – Independent Strategies

Self organizing teams not only have the freedom to implement their chosen tactic for a given end state but also have the latitude to decide the strategies to achieve stated long term mission. Many collaborating teams work on a common mission that guides the whole organization but have their own strategy. Phil Abernathy presented how agile method can be applied to the process of strategy formulation and execution. Strategies of other collaborating teams decide its goals. Each team in turn formulates its own strategy to meet its goals. This exercise of goal setting and strategy formulation is done on a what is popularly called “Fedex Day”- strategies are shipped on the same day. Teams work to get a buy-in by all their members thus making their strategies inclusive and based on shared understanding. While executing the strategy no command and control is required as all members work in unison. Frequent feedback is taken by comparing progress against goals and timely course correction is applied.

Tacit Knowledge

Crossfunctional, self organizing teams depend a lot on idea exchanges that happen in close knit teams that are socially and functionally integrated. These idea exchanges happen over coffee or at the water cooler or in a chat session. The knowledge or social intelligence of the group gets build over a long period of time and this “Tacit Knowledge” is what differentiates and experienced team member from a newbie.  Dave Thomas says that only usable knowledge is “Tacit Knowledge”.  Pilots don’t refer to manuals or guides while flying. Most of their actions are reflex actions directed by “Tacit Knowledge” . Any number or detailed manuals or video demonstrations can’t capture all that a pilot knows. Human intuition is far more deep seated than big number crunching and analysis. Experts often can tell the outcome of a situation based on their intuition. They already know the answer which they rationalize later. The “Fast Thinking” powered by “Tacit knowledge” is hard to explain . Michael Polyani in his book the “Tacit Dimension” says we know more than we can tell. This is the unknown known says Dave Thomas . Most stimuli like gravity are subconscious. Its hard to explain the visual signals that help you accurately recognize someone from distance. Explicit knowledge that is shared in classrooms and conferences only becomes useful when one fully internalizes it by experience.  Ideal mentors don’t teach what they were taught but inspire people with what they have learned.

Versatile Experts

One of the requirements of a flat , crossfunctional , self organizing team is the ability of team members to rotate roles they play. This results in innovation when some one takes a fresh look at a problem.  This mandates the  team members to have “T” shaped skills – where they have breadth covering many subjects and in depth knowledge of one subject. Polyglot programmers. are language agnostic and so versatile that they can program in any new language with minimum learning curve. Polyglot programmers believe that just like an artisan they need a different tool for every job. They master many languages including JavaScript, Scala, Groovy, JRuby, Jython and Clojure. All these are JVM based languages. Polyglot teams map each sub domain of a problem to a tool and decide on a set of tools that they need to use. Many of these tools are new to the team. In their search for tools they underrate their current skills. The right tools are selected keeping in mind that tools are good only in certain contexts. The rotation of developers in the team helps one overcome resistance in the team – this balances the comfort zone and new learning equally in all members.

Take Aways from Agile 2013 – Part 4 of 4 Data Migration Issues Addressed by NoSQL Databases

This session by Rebecca Parsons was very insightful. Data models change as products evolve through iterations. Scott Ambler says that relational databases evolve in agile manner by refactoring/migration pair in small steps. Like everything else; data changes. Our understanding and access patterns change- requiring database migration.

Code changes can be easily managed in version control repositories. Data is not version controlled – but data models are. Developers need to meticulously create and maintain data migrations so that data can be rolled forward or backward in sync with the code. Developers also need to provide default values for columns in records created by older versions where those columns didn’t exist. Over all data migration is a hairy problem.

One would tend to think that data migrations for Big Data would be a bigger problem. NoSQL databases are characterized as non relational , schemaless, cluster friendly, open source, 21st century web. You have Raven, Couch and Mongo – which work with Documents, HBase and Cassandra which work with columnar data, Riak and Redis which work with Key-Value and Neo4j which works with graph. Each of these have different ways of dealing with migrations.

NoSQL databases like MongoDB provide a clean way to address this problem. The loose structure of MongoDB allows data from multiple versions to co-exist in the same database. All data doesn’t have to look the same. This makes evolution easier- e.g. no change in data is needed to add a field. Thus there is no need to run large scale migration to roll a version forward or backward.

This does not mean that migrations are never needed. You do require migration to add a non sparse index in MongoDB. Migrations in graph databases like neo4j are a bit more complex. However we can conclude that NoSQL databases can be modeled for easier migration.

Tips on Emergent Design- Takeaways From Agile 2013 – Part 3 of 4

I always attend some technically heavy sessions. Often some of what I hear goes over my head – but whatever I learn from rest of it is worth the time. Neil Ford’s session at this conference was one of these.

I am going to highlight some of the learning in the form of tips – listed below in no particular order

Emergent Design– waiting till the last responsible moment to take design decisions. Often trying to predict leads to over-engineering. Prefer proactive to predictive or reactive. Over time a component starts taking more responsibility – that is when you should start tracking its last responsible moment. Another case or over engineering happens when objects start looking too much like real life objects- Neil calls them anti-objects.  Emergent design also needs developers to find abstractions defined by idiomatic patterns – both domain and technical. Developers should realize that up-front design can address knowns and known unknowns. It can’t address unknown unknowns. Waiting till the unknowns become known is the only solution. Emergent design also requires documentation to remain in sync with the code. Neil says the best way is to write code that reads like design document.

Importance of Ruby– Its easy to write Java unit tests in Ruby. Prefer language based build tools like Buildr than XML based build tools like Maven. Generally Ruby is a good language for build and test engineers. Ruby has much simpler monkey patching which works exactly like point-cuts in AOP ; allowing testers a sneak peek at what is going on at entry and exit points of important methods/ code blocks.

Cyclomatic Complexity (CC) and Afferent Coupling(AC)- CC and AC are good metrics to sniff out refactoring opportunities. Once you see these opportunities ; you can harvest them by abstracting out dependencies and exposing APIs for the same. CKJM tool measures and reports CC and AC at class level. Events such as merging code bases or reduction in test coverage introduce CC. CC per line peaks near release time as everyone rushes to ship it .

Importance of TDD – Need to write tests for all code including private methods. TDD helps you zero down your debugging efforts at brick level instead of building level. It has been observed that code written using TDD has Cyclomatic Complexity of  2 vs 10 for code without TDD.

Technical Debt- The longer the time taken to put to use a feature that is ready more the Technical Debt incurred. Developers often try to introduce genericness in their code-which increases software entropy leading to accidental complexity. Sonar is a build and automation product- which has a technical debt calculator. Estimating technical debt helps in clearing it. But the real trick it does it that the team starts negotiating repayment with customer or management. Runaway technical debt and management’s insistence on sticking to a particular technology hinders emergent design.

Empathy Improves Efficiency in Agile Teams. Take Aways from Agile 2013 – Part 2 of 4

Last month I blogged how agile is finding relevance outside the process of software development. Now its also becoming apparent that other disciplines – particularly behavioral science is becoming more relevant in Agile thinking. Linda Rising in two of her insightful sessions –on the topics of estimation and retrospectives showed how being empathetic with Agile Team members will go a long way in improving efficiency in agile teams.

Estimates become goals – and finally decide the efficiency of any Agile Team. Asking a developer to estimate based on limited information puts psychological burden. Developers often have to make tacit assumptions as there is no time or information available to explicitly validate them. According to Linda this amounts to lying. Smarter people are better at justifying their lies. They offer logical reasons and have others believe them. We need to admit that we lie while estimating. In fact it’s the worst kind of lie because we are lying to ourselves.  Like all lies our estimates come back to bite us. When it does happen a team that has admitted that an estimate is after all a lie won’t produce more lies to defend it. Such a team will also avoid multiplying the risk by using these “lies” in mathematical calculations. Linda advised Agile Teams to re-estimate instead of justifying an estimate or blaming others. Once every one understands that estimates are guesstimates- the psychological burden on individuals reduces and they can work better.

Agile Team members also carry the baggage of pent up feelings if they aren’t allowed to  air their views. Often chronic personal conflicts and team imbalance drag the team down. Linda wants us to understand that people feel good – a lot lighter if they simply express their feelings. She demonstrated a tool called as “The Time Line”. It’s to be used when Agile Team meets for end-of-sprint retrospective.  The team identifies certain significant events that happened during the sprint and mark them on the timeline. Each individual is then allowed to put anonymous feedback about how he felt when an event happened by putting sticky notes under relevant events on the timeline.  Its important for all to believe firmly that each team member did his or her level best under the given conditions. There should be no finger pointing- only understanding and empathy. The team is thus ready to move on to the next sprint with a fresh mind.