This is a story about a software team. I feel like it applies to many teams, since I have adapted the principle to other startup concepts, beyond software startups to include church planting and the coffee industry. Nonetheless, it began for me with software teamwork. It has, over time and application to much broader domains, reshaped my idea of the critical role of teamwork in conceptualization and innovation. We are enamored with the idea of the lone inventor, yet when we dig deeper (for example, into the story of the Wright Brothers) we find a motivated team behind most of them. The mantra, or principle, of this story is that The Whole Team Owns The Whole Code (now more aptly, the “Whole Product” as it promulgates beyond software). Sounds simple, maybe even makes sense on the surface to any casual outsider or programmer; while it might be easy to grasp, however, it is very hard to actually put into practice. This story taught me that lesson in spades.

A little background on the project would be helpful. The specific details are unimportant, as the principle of “teamwork” defines how many groups function under normal circumstances. I mean, when a defect comes in, there might be a person more familiar with the code who is out on vacation and someone else should be free to fix it, right? Then there is normal turnover, the person who wrote that algorithm has left for a better job and someone else has to unravel it to tune-in some new addition. It clearly applies beyond software as teams help each other field sales accounts, treat patients, perform complex systems repairs, and manage large merger & acquisition transactions. Teamwork is a foundational concept. This project of mine was a different starting place, however, because I was asked by a friend to take on the job of repairing a broken team as well as the broken code they left behind.

Many of my roles have been with complex systems. Complexity is often a harbinger of a dysfunctional process or team dynamic, since good communication helps decompose complex systems into smaller, simpler, and more manageable modules. I was no stranger to coming in as a fresh set of eyes on a project with history. Yet day one on this job revealed that I had not realized what I was in for. There are teams with problems and then there are truly broken teams. In this case, a team broken so badly it disintegrated. The former team had undergone a recent and significant upheaval while still needing to support long-time subject matter experts (SMEs) and a series of looming critical deadlines. The original software lead developer had received a promotion and the company hired two new senior developers, brought in to fill his one legacy senior role. The duo were tasked with leading the team building version 2 of the legacy software he would leave behind for them. If you have ever seen the movie Space Cowboys, you know the scene where the young rocket scientists look upon the old technology with distain but cannot replicate its successful design – that is, until Clint Eastwood just fixes it. There is an almost unbelievable amount of truth in that. There were a couple of other more junior programmers and a few programmer-analysts that processed the data and wrote also some of the code, making the total software team around a dozen people. The whole group became split, as unfortunately often happens, over the maintenance of multiple versions of the product. [Pardon some geek-speak, non-programmers reading this may want to skip to the next paragraph!] The original software guy wrote the user interface in VB and several core algorithms in C++, the new guys wanted to rewrite everything in C#. The legacy code was ActiveX and MFC while the new crew were abandoning that, skipping the newly released .NET framework entirely and opting instead to build their own control engine in ATL.

These are not insurmountable challenges, per se. I discovered it was not a technology problem at all, it was a teamwork problem. What started as a slight difference in vision and the inability to mesh competing innovations became an all out war that ended with the two new guys fired and all but one of the junior programmers asking for new roles. It was a mess, to put it mildly. Sounds like your dream job, right? Being one to never shy from a challenge, when my friend asked for my help to fix this, I came in with a request – before we rebuild the team and hire new folks, give me 60 days alone with the code. You probably know from any good organizational transformation book about the importance of having a champion for key initiatives; my friend was the VP at this company and had worked with me on other projects. He trusted me to fix it and I trusted him to provide the necessary top cover. The legacy guru was still around for some consultation (albeit more than a tad salty about yet another new guy coming in to “fix” his legacy software) and the one junior programmer left was willing to work hard to find a solution, in spite of her nervous anticipation towards the possibility of another failure.

The project was unique in that it was a code base that needed to follow a rigid release cycle for internal use on a critical analysis project (how’s that for cryptic). Since we are talking about the principle of teamwork and product ownership and not that specific project, it does not matter much – what I want to emphasize is that this principle is all the more important if the product itself is critical. There is an inverse relationship between total product criticality / quality and the heroic effort of individual egos. (For more on that, look at “individual heroics” from CMMI Level 1). Imagine something along the lines of that Space Cowboys movie, processing all the data from a prior space launch in time to support critical analysis before the next launch 30 days latter (fixed schedule and high complexity) and you will have an idea of both the pace of the project as well as the magnitude of the request to pause – two whole cycles with zero bug fixes, zero updates, zero new features all with a delay in hiring new staff – no new team members – until we got our arms around the problem. It was a huge ask. I was granted my request, said my prayers, and settled down to dig into the code.

They say that the code is the documentation far more than anything written about the code. In this case, if you don’t actually know who the proverbial “they” are, then I refer you to the Agile Manifesto and wish you well as you continue reading. The rifts in the code were ghastly. What that revealed is that the rifts in the team were actually documented in the code itself. When programmers with different views of how things should be done and using two workable but incompatible views of “correct” go to war, you even see it in the snarky comments and disrespect in the way they name methods and properties. In a way, it was like reading the diaries of competing college frat teams trying to outdo one another. Dismantling this and removing the snark, bringing unity into the two code bases, and finding a way to heal the rift left over in the code was critical before inviting in new team members. There was no way, however, to rewrite it all, remove all the scars, and eliminate the history of this mini civil war. Worse still, the stress of knowing that the deadline would not go away and this code HAD to work when we were done was ever looming. It was not like this code was the kind of stuff you could look up for help online, either. The very fact that it was a unique problem set invited brilliant people to solve the problem – so when they intentionally stop communicating with one another, the results are multiple paths of ingenuity, each one only half as powerful and yet still equally complex.

One fun example, the code-war had evolved to the point that competing algorithms yielding different results were referenced by the two versions of the core framework and no real comments, documentation, or tribal-knowledge explanation was available for what these routines did. How, then, do you know which one is right in that case? There was a complex DLL called “gator.DLL” that was critical in several higher level methods, with a “gator2.DLL” being its competitor. No one was left to explain what these DLLs did. Alligators, really? Talk about dangers in the code swamp! Maybe named by a Florida grad? Anyway, we eventually discovered that was short for “propagator” – as in propagation of an object through space. Back to our Space Cowboys example, how do you launch a resupply rocket and have it meet up with a space station when both are moving in different orbits and moving so fast that by the time you get the report of their last position they have already moved hundreds of miles? You need to propagate their position forward in time and predict an intersection. But if you use different propagation algorithms you could miss by, oh let’s say a few feet. Miss is still miss and at incredible speeds there is no easy way to just tap the breaks and make a U-turn, thus no payload and the space station occupants starve. Should be a straight forward problem to solve, all you need is some non-linear math to propagate a point in space based on earth-centered, um, wait, which coordinate system and vectors from where? Without getting into framework nerd speak, what was occurring during the migration of tools on the framework as they were moved from Version 1 to Version to was a de-linking and re-linking of these dynamically linked libraries, got it? No? Never mind.

Full-Disclosure: I am not a mathematician. Not even close. In fact, it is a wonder I ever got this problem solved because you would never let me work on critical math problems – especially not the kind that the rocket scientists in Huntsville discuss casually over beers! I love living in a town where whole breweries are named for our space and rocket legacy. The secret is that I am surrounded by the appropriate skillsets, and most of them have programming skills of some kind. Rocket Scientists are brilliant, they are just not framework architects (generally speaking, deference to a few individuals, you know you you are). Thus we reached our first critical juncture leading to my new mantra. We had to develop the framework so easy to use that a PhD mathematician could not mess it up (you think I am joking, I am not). On previous jobs I had the, eh, privilege of translating some cryptic code from late-80’s era Fortran into C++ after the duo that wrote the algorithm had retired a few years prior and the use of newer machines could no longer be held back by the onslaught of 32 to 64 bit upgrades. Seeing the cryptic math code from lone rocket scientists or small groups of people that speak math as a first language will scare you if you have ever done large complex software integrations. Thus we had to make a way to ensure that the ability to swap out and improve these kinds of algorithms into the framework did not require, well, any knowledge of the framework itself. Folks familiar with object oriented program understand encapsulation and abstraction academically until they have a project like this – then they understand it emotionally. I always had it in mind that the core development team needed to clean out the after-effects of the code-war, but I had not realized that the “whole team” included not just the core developers and the equally sized QA/Analysis team (all of whom could code and write apps on the framework), but also the team of 40-60 specialized PhD subject matter experts who developed unique tools to process data for special challenges that only a handful of people in the world would understand. The Whole Team had to own the Whole Code.

Hiring the new team was both a huge relief and a huge risk. Should we tell them about the prior wars, I mean, who really wants to join a team that once imploded? However, after taking the pause and resetting we were able to start each new team member on the right track to build on momentum we were developing. We were not simply building a product, we were building confidence into the new team that they could hit the ground running and help shape the direction. Because we were clear about remembering the code-wars and ensuring they would never occur again, we could confidently instill agile principles and continuous quality into the process. This was an institutionalization of agile from the “new” day 1. We added just the one thing to well understood practices like Onsite Customer and Daily Scrum meetings – the Whole Team Owns the Whole Code. No sacred cows, no personal libraries, and no pride of heroic personal skill above pride of the team’s accomplishment. We would see success or failure as a team.

Everyone knew that from the day they interviewed. What they received in return was palpable confidence in the success of the team they were joining. In some of our early meetings as the team began to grow, I would say (then with the title of Software Development Manager): “My job is to remove roadblocks; I am only a 1/2 time programmer since the other half of my job is to keep the battle outside of our team. All of you will eclipse my skills, so don’t expect me to be the answer man. Lean on each other. Make this the best team you have ever worked with.” One of the challenges in that statement is that the SME users outnumbered us almost 10:1, which means that a SME with a PhD will want to latch on to their favorite programmer and come in behind the lines to get favors written into their pet projects. Since that violates the principle we agreed upon, everyone knew features could only come in through our established story backlog process – no favors and no favorites, instead put the team first.

It is worth mentioning this was over a decade ago, yet many of the days in the swamp I remember quite vividly. I have since moved on to large scale System-of-Systems projects using Model Based Systems Engineering (MBSE) tools – which keeps me a step even further from the code. Applying The Whole Team Owns The Whole Model principle does not have the same ring to it, but is equally true. It does not matter if it is a code base, a system model, or the workflow behind the espresso bar – in order for it to really work the team has to own it. We continue to witness the positive results not only on the product itself, but on the level of collaboration across teams-of-teams building systems-of-systems because people understand that the product is owned by the whole team. People are both bolder to recommend innovations and more humble to hear refinements of their own ideas. In the often quoted, misquoted, and multi-attributed words of a famous dead guy, “it is amazing what you can accomplish when you don’t care who gets the credit.” (I have, ironically, seen a similar citation given to Einstein, Mark Twain, and Harry Truman, with the later being most common – I guess the real author was some poor dude in a bar after a team meeting that did not actually get the credit).

One of the ways that I have had the pleasure of contributing to such a common collective concept is in the Future Airborne Capabilities Environment (FACE) Consortium, where the whole team now owns the whole technical standard. There can be heated exchanges as alternative paths to complex avionics challenges are worked out. The teamwork came to include a small bit of code that I had the joy of contributing some towards called the Basic Avionics Lightweight Source Archetype (BALSA). For those that don’t get the pun, balsa wood is a light material (also rather weak) that many model airplanes are made out of, and the name BALSA was not mine but the crafty wit of one of our team members. It was supposed to be an example to train with, not a robust prescriptive solution, forcing the team to focus on the problem and use case of on boarding new developers. Now this whole project lives on in the collective ecosystem and, though I have not touched the code in a few years, I still feel the joy when people get excited about this simple introductory code example to introduce the very complex world of modular avionics.

I suppose that the application of the principle to complex systems models and technical standards did not really surprise me. Many agile software principles cleanly translate into the world of Systems Engineering, just as many principles from the architecture of buildings have translated into the architecture of software. What did surprise me, however, was the application of this concept in the entrepreneurial and management space as the “code” of a good organization is the product of much collaboration. The secret sauce, as any good venture or angel investor knows, is in the team more than the idea. Lots of people have good ideas, surprisingly few can assemble a team that can execute collaboratively on those ideas.

Through the lens of this principle I have looked back and learned that Jesus actually invited his messy and imperfect disciples to help develop the “code” for planting churches; as one of my professors at Princeton Theological Seminary put it, Christianity has the most dynamic enculturation of any concept in history. Messy as that has been in many places, the people of the church continue to reshape what that means as they live it out in their present ethos with their present teams, over and over again all around the world. The core principles have been well established and have been shown to persist, encapsulated and abstracted from the cultural adaptations. Love, it would seem, is a very powerful product, lending well to continuous integration. The concept also applies in starting new businesses and new ideas in existing businesses. We have seen this in our coffee startups as well as in my tech startup ventures – our CRAFT values are a way of instilling ownership to the whole team, like the source code of the business itself. It seems that Respect and Fidelity are also highly pliable and powerful products that equally lend well to agile principles.

I have been keeping most of my writing to myself. Evernote is bursting with stories like this. Thinking about the application of this principle to writing, have I really invited coauthors this way? For that matter, what about just other folks who could benefit from the conversation? I could really stand to learn my own lessons on teamwork in other areas of my life. Research projects? Prayers with my brothers? There is certainly a need for leadership in vision and direction – I had to provide that when we set the basis for the code we built upon – but then I had to release it to the team so that the whole team could buy in and own it. I had to leave my code as no longer mine. Should my whole life not also in some ways be this way? We tend to think of our lives as our own, with our purposes shaping our destiny as captain of our own souls. Yet we are increasingly learning that we are shaped by community, forged by social relationships and pliable to be rewired by events around us. It is as if our lives are being built by the team we journey with. I still have much to learn about this subject but I am certain now that it is a powerful principle that requires far more courage and much stronger leadership than may appear on the surface. Keeping a team together and headed in the same direction while turning over the keys to the most critical aspects of the project requires a subtle kind of humble strength that is willing to take a few blows to the old ego. What more important code, product, or purpose than each other? It is worth it.