In recent years, the software development community has converged around a process called Scrum, which is defined on Wikipedia as an “iterative and incremental agile software development framework”. I’ll get back to parsing that mouthful a bit later. I don’t have any hard numbers on the adoption rate of Scrum, but based on the dozens of engineers who work for me at AutoLoop, and those that I interview, I would estimate that roughly four out of five of them have used it in the past, and at least half were actively using it at their most recent position.
Scrum seems to have been accepted as the best way to organize software development projects (although there are popular alternatives such as Kanban), but I’m going to argue a few points:
First of all, that one of the primary values of choosing a methodology is simply the adoption of a shared vocabulary. There are things that need to happen as a part of delivering quality software, no matter what process you choose. Scrum gives names to these elements so that everyone is speaking the same language.
Second, that the real value of Scrum is in the creation of fully functional, independent teams.
And third, that Scrum represents a sub-optimal compromise, rooted in the dysfunctional way that most software companies have evolved their reporting structures.
What Is Scrum?
Scrum is a process methodology that defines team structure, roles, and timeline phases for software product delivery. It was conceptualized in the 80’s, and then became popular in the 90’s and 2000’s as a counterpoint to traditional techniques which are often categorized as “Waterfall” methods. Waterfall represents the traditional, long term product planning over months or years, with a gnarly Gannt chart to map dependencies and timelines. A development team would basically go dark for the duration of the project, and then at the magical release date, a complete software system was ready for deployment. Of course in practice, these release dates were rarely met with any degree of accuracy. Agile processes, of which Scrum is the most prominent, admitted that a better way to develop software was in small iterations, with lots of end-user involvement, and no long term timeline predictions or commitment. Getting back to the definition of Scrum, iterative means that we follow a short, repeatable pattern of releasing small bits of working software. These repeating patterns are called Sprints, and they usually last from 2 to 4 weeks. We’re not trying to release the entire finished product at once, but rather we release it in phases, which is why we call it incremental.
Of course, some products would be useless without being completely finished, so we might be releasing these partial products behind the scenes, or to a small group of beta testers, but the point is that were constantly releasing, getting feedback, and making small course corrections as we go.
Scrum Purity and Necessity
The truth is that nobody follows Scrum to the letter of the definition. There are countless books published about Scrum, and after several decades, there are alternate takes on how things should be done. After asking “do you use Scrum at your company?”, the next question is “how closely do you follow it?”. The answer is usually “about 80%”. Every organization is different, and so everyone picks and chooses those elements of the canon that suit them best.
So why would an organization adopt a process like Scrum and then not actually follow it to the letter? That 80% I mentioned above is optimistic. The reality is probably more like 50%. The elephant in the room with every software process is that it’s not actually necessary. That’s a horrifying statement to people who love process, but it’s true. Software is a very bizarre engineering discipline, in that it doesn’t actually require any discipline at all! Personally, I categorize software as art, not science, but that’s a separate discussion. What about all the fundamental elements of a software development methodology? Requirements, design, planning meetings, source control, build servers, QA? Throw it all out. All we really need is code. I can fire up an editor, write a new page for my app, upload it to production, and I’m done. I just skipped the whole process, and the page works. When it’s so easy to skip the process, it can be a challenge to get people to actually follow it.
One of my pet peeves is an insidious little phrase that works its way into too many software companies: “the business”. A developer might say something like: “we’ll have to wait for the business to figure out requirements for the new product”. Or a sales and marketing exec might say in response to a developer proposing an idea: “just let the business figure out what products we need to build, you focus on implementing what we ask for”. This is a toxic separation of concerns in a software company! A company that sells software is selling what comes from the fingertips of the engineers, so why would you relegate them to a status of mindless robots who should never think about business concerns?
One of the core principals of Scrum is that you create fully functional product development teams that are capable of conceptualizing, designing, building, testing, and delivering products on their own. They don’t have to go consult with “the business” before making decisions. You put the right people on those teams, get them working closely together, and let them create.
This is the main reason I chose Scrum for my company. We were devolving into a “throw it over the fence” mentality between product management and software development, and I had to find a way to fix it. But it also represents a broken compromise, which gets me back to the title of this article. We see the problem when we start to look at reporting structures.
The Reporting Structure
Reporting structures get complicated with Scrum. Let’s start with the makeup of a typical Scrum team.
This is pretty straight-forward, and clearly shows the primary strength of scrum. Everyone required to create the product is right there on the same team.
One role not represented above is the Project Manager. Where do they come in? Let’s say you have Team A, which is responsible for the company’s web site, Team B that writes the company’s mobile app, and Team C that manages the company’s back-end software product. A project manager might need to coordinate a company-wide effort that spans all three teams.
Project Managers have a tough job: get things done without anyone actually reporting to you, and without belonging to an actual development team.
The Org Chart is where things get weird. This represents managerial reporting structures – these are the people who hire you, monitor your daily activities, do your reviews, and (hopefully not) fire you. If you ask the engineer “who is your boss?” the answer might be “the VP of engineering”. The designer might answer “the chief marketing officer”. A project manager might report up through sales, or directly to the CEO.
But it’s not a cross-reporting organization, even though it looks like it might be, because on a Scrum Team, nobody is in charge.
This org chart shows the actual reporting structure of our hypothetical software company.
Personalities have to be taken into consideration when putting teams together. Engineers tend to be a bit more introverted; for the most part, they want to be given a hard problem to solve, and then be left alone to solve it in peace. Combine this with a general aversion to meetings, and what you’re left with at the daily standup meeting is often silence. To get things moving, you need a strong voice, and it’s generally up to the Scrum Master to make sure important things are being discussed. But since scrum combines people from different reporting structures onto teams, no single person is in charge. The scrum master doesn’t actually have the authority to tell anyone what to do. This, I think, is a major failing of scrum. Without a designated leader, it’s hard to make meaningful progress.
The flip side of a scrum meeting where you can hear crickets chirping is one in which the team can’t come to consensus. Let’s say that the product owner has come to the meeting on a mission to get a new widget added to the main landing page of the company’s web portal. But the designer is under orders from the marketing department to keep the landing page as lean and simple as possible, and she is against the idea. The engineers are neutral, since the widget would be easy to implement and wouldn’t incur much performance penalty. How does the team move forward? In an ideal world, the team would discuss the pros and cons and come to a mutual, agreeable decision. But the real world doesn’t always work like that. A good leader would gather information from both sides, encourage discussion, and hope for consensus. But when that consensus is not forthcoming, the leader makes the call, picks a direction, and moves on to the next topic of discussion. This isn’t possible with scrum. Decisions have to be made elsewhere, and this slows everything down.
Command and Control
At the top of my org chart above is “The Big Boss”. This might be the CEO or the VP of a large and mostly independent business unit. This is the person who is ultimately responsible for the success of the software product. With scrum, it’s actually very difficult for this person to give orders and make things happen.
Let’s say the Big Boss is giving a demo to an important potential new client, and the application’s billing module refuses to load. He’s greeted with embarrassing error screens, and after the meeting he storms in to the Vice President of Engineering’s office demanding explanations and immediate fixes. The VP then calls a meeting of his engineering leads and passes on the urgency from the Big Boss. Later that day, at the standup meeting for the team responsible for the billing module, the lead developer points out his direction from the VP. But the designer points out pre-existing urgent tasks that need to be coordinated with several other teams; if those tasks aren’t completed right away, dozens of engineers on those other teams will be blocked. The team doesn’t have the time to compete those tasks and fix the billing module, so they are faced with competing top priorities. Which one is more important? Who makes the call? We need to get the Big Boss involved to move forward, and this ruins the idea that scrum teams are self sufficient.
The Chain of Command
I have a military background, and this undoubtedly introduces a bias to my thinking about how teams should be structured. I don’t like anything that smacks of cross-reporting. Every employee should be a member of a single team, and that team should have a single boss. In a software company, that boss should be an engineer, someone who came up through the ranks doing the same job as his or her subordinates. With a clear chain of command, there is never a doubt as to the team’s priorities.
The trick is, how do we design our org chart to make it work? How do we cure the dissonance between our reporting structures and the product teams that are actually getting things done?
Sadly, at an established company where these structures are institutionalized, it may not be possible. Scrum is probably your best bet. Even with my strong opinions on the matter and my influential position in the company, I’m not rushing to abandon Scrum. But let’s go over some hypothetical arrangements that might work, especially for a smaller company that is still actively tuning its organization.
As I stated in the introduction, Scrum has a very strong attribute: fully functional teams. So that’s the part we want to keep. The simplest fix is to simply change the reporting structure so that one person on that team is in charge. The team leader would obviously fill the role of Scrum Master, and would be a leader from both the product standpoint and as a traditional personnel manager.
An argument can be made for putting the Project Managers in charge of Scrum teams. PMs often represent major vertical initiatives in the company, and a cohesive chain of command would go a long way to improving the results of those efforts. Difficulties arise when we have several teams all touching one piece of the application. If you have three teams that all want to make changes to the application’s landing page, who coordinates the efforts? Are we just playing a shell game with our team structure and introducing the same problems we had with Scrum?
I don’t think it’s possible to design any organization so that there is no crossover of responsibilities between groups. After all, it is one company and one product. But with careful consideration, we can minimize the crossover and reduce the chain of dependencies between teams. There is no patent solution I can offer, since every company is different.
For lack of a better term, I’ll call my proposed Scrum variant Military Scrum. Here are the core principles:
- Rapid Iterations. Release new features to users as often as possible.
- Fully Functional Teams. No “throwing it over the fence”.
- A Single Leader. Each product team has a boss who makes decisions about product direction, engineering design, and personnel. These leaders need to be super human, since it is very hard to find a person with all these skills wrapped into one body.
- A Clear Chain of Command. Each person in the company has exactly one supervisor.
We don’t want to throw out the baby with the bath water. Scrum has a lot going for it, and it’s very well established in the industry. What I have proposed here is a simple but significant change to the way Scrum teams are structured. If you made it this far into what is admittedly a very long post, feel free to offer me your feedback. I’d love to hear some alternate opinions on the subject.
Eric Z. Beard
December 28, 2014