It’s only a good practice if you have the cultural discipline to make it work.
Stopping the factory used to be sacrilege - companies figured they were only making money if the line was moving - so when Toyota adopted this practice on their manufacturing line is was a huge shift. It was taken so they could resolve quality problems immediately, which saved them time and money because otherwise the problems occurred in many cars and had to be fixed later. It was better to pay the price of stopping the line now than the price of redoing the work later.
This was a huge cultural shift throughout the organization. Management had to recognize the cost of fixing quality later, and change their short-termer mindset around the moving line. The workers on the line had to learn not to let problems slide by, to ask for help immediately and not rest until the problem was truly resolved. Everyone had to learn to diagnose problems in the line without blaming individuals, otherwise trust was impossible and the real problems could not be diagnosed. (This is also where Toyota developed their ‘Five whys’ practice for getting to the bottom of problems.)
While we’re obsessed with individual techniques like Kanban, the team at Toyota knows that it’s culture that allows them to win, not the tools they use. That difference is also exactly why it took decades for the rest of the industry to catch up: You can adopt technology quickly, but culture? Not so much.
We experienced the same problem when we tried to adopt this practice in our software teams. People loved the ability to stop the factory, and they used it all the time. Often, it did actually increase the quality of the work we did. But overall, it made things worse for us, not better. It was causing arguments, bad relations, and most importantly, was reducing our long-term efficiency, not increasing it. It became nearly impossible to ship software.
What went wrong?
We had adopted this practice without developing the cultural discipline to make it work. People weren’t stopping their own lines to resolve systemic quality issues; instead, they were stopping the work of other people to question their decisions. A technique for improving our work became a technique for attacking our strategy. Disagree with the product that team is building? Stop the factory! Don’t like the database they chose? Pull the cord!
Yuck. That’s not how that’s supposed to work. It didn’t help that I’ve always pushed hard for people at the front line to be opinionated and educated about as much of the product as possible. Suddenly they’re applying those opinions to the work of others, and we’re not building higher quality products, we’re just arguing non-stop about what we should be building and how.
This required that we develop a couple of new cultural skills. When you read about the Toyota system, it’s striking how much of what they accomplished was essentially cultural technology - that is, tools for developing and sustaining culture - rather than manufacturing tech.
Our failed experiment at letting anyone stop the factory showed that first everyone needed to understand the value in stopping the line. We started to point out that they should be focused on their own work, not others’, and thus should only be able to stop their own line. Most importantly, remember back to the point of pulling the cord: You’ve discovered that your own work is compromised because of a quality problem in the system, and you need time and help to fix it. That had to be why we stopped the line, and nothing else.
That being said, people’s concerns about decision making in other parts of the organization needed to be heard. Expert teams should being given the power to build what they think is best. But even the best teams need outside reviewers, need other experts to question their assumptions and conclusions.
Our work cycle needed to include productive, constructive feedback opportunities. The best process like this I’ve ever seen is in Pixar founder Ed Catmull’s Creativity, Inc, where he describes their “Brain Trust” system for helping directors make great movies. This is such a great system that it’s worth reading yourself, but the aspect that stands out is the responsibility of the team giving feedback to the director:
Your job is to help them see the flaws in their proposal. You can not and must not propose solutions, and you can not demand that they agree with your complaint. Even if you’re the CEO. You are a collaborator helping someone do their best work, not a competing director trying to get your ideas into the movie.
See the parallels to the value in stopping the line? It’s about improving the quality of work, not making a power play, or being right about something.
Pixar has a critical second layer to this process: Management decides whether to ship the movie. This allows them to give the director full responsibility for the movie. If directors don’t agree with the team’s list of flaws, that’s fine, but in the end, you have to convince the management team that the movie works.
The thing I love about this is it gives you concentric circles of iteration and control. The movie team makes decisions constantly about what to do and why, then they loop out to the Brain Trust to get feedback on what they’ve done, and finally there’s a business cycle that determines whether the movie should ship. Everyone gets to contribute, but your business concerns are isolated from the work cycle, giving teams the freedom they need to build something great.
I was never able to figure out how to adapt their process to software, but I hold out great hope that there is a way, it just requires a lot of evolution. Pixar gets to take five years to make a great movie, but you never get that kind of window to make software. They also kept all of the iteration inside the building, where modern software should be built in layers, in collaboration with your users and the market.
Even without being able to directly adopt a better practice, we recognized the need for two layers of process. We thinned down people’s ability to stop the work of others, and at the same time built feedback loops at larger intervals and in different forums for strategic concerns to be heard. We also began building cross links between teams that allowed those with concerns to collaborate with those doing the work, rather than leaping in and shouting ‘stop!’.
There are many great inspirations for how to build better software, faster, but all require some level of adaptation. I am absolutely convinced we will see a software production system as coherent and clear as Lean Manufacturing, but I also think it’s going to take a long time, a lot of experimentation and failure to get from here to there.
Hopefully our experience, our successes and failures, can help you get there faster.
I was asked by a CEO recently, “How do I know if my product is ready to scale?” He thought he was ready to raise money and start growing the company, but he didn’t really know how to be sure.
Venture capital exists to invest in enterprises that have a huge opportunity but are also a big risk. Yet VC firms tend to actually be pretty conservative, relying on pattern recognition and social networks to make their decisions. This turns investor pitches into bizarre encounters where you have to help them see the huge risks in your business and how this risks can generate massive returns, but you also must convince them that you have all of that risk managed, so there’s no reason to be afraid. I’ve been turned down so many times by investors “until we’d taken the risk out of the business,” and every time, I’d sit in the car outside their office park thinking, “But don’t you exist specifically to make big bets?”
I can’t tell you how to raise money, because no matter what it’s hard, and there’s no special trick to it. Well, there’s one trick: Sell a company for a billion dollars. Do that, and you’ll never have trouble raising money.
This article is only for those who are not quite so lucky.
What I can help with is framing your company so that you can more coldly see and explain where the risk is, and with that hopefully you can help others get excited about the vision in your head, the beautiful thing you’re trying to build.
Step one is understanding your market risk. If you’re raising money, you have to convince your prospective investors (if you’re talking to typical valley VCs, anyway) that you can build a company that can do $100m in revenue at some point. So, convince me your market can support a company that can hit that scale and still grow. That means the market needs to be hundreds of millions, and probably billions. With Puppet, we knew the overall management market was between $6B and $25B a year, depending on your definition, which made this part pretty easy for us, but lots of companies are creating a market, so they have to spend a lot of time on this topic.
The next big question is why you, particularly, will win. Sometimes this is called technology risk, but companies win for many reasons other than technology, especially these days. Puppet didn’t necessarily have the best tech in the world, but we had a focus on the user that no major player in our space had. Everyone else was worried about the buyer, and we focused on helping the individual users succeed. Other companies have a cultural edge, or even just a mindset that others can’t copy. Whatever it is, though, there needs to be some reason why you’re going to take this big, awesome market.
I call this strategy risk. You’re planning to take the market in a particular way, and you think it will work but you don’t really know until you play.
A high-growth company usually trades off strategy and market risks. If everyone else agrees this is a great market, then you’ll have lots of competition so you need a unique strategy that sets you apart. If it’s a mature market with clear revenues and margins, you need a great strategy to fight against big incumbents, who won’t go quietly. If, on the other hand, no one else agrees this is a good market, and you get it all to yourself, then the strategy doesn’t need to be differentiated, so you can run the least risky strategy you can think of.
You can’t walk into a VC’s office and say there’s neither market nor strategy risk, because that business is just too obvious. It’s either so obvious it’s actually stupid, or so obvious that you’re one of a hundred players running the same game - both of these are bad bets. You’re not getting that funding round.
The last piece is execution risk. You’ve got a good market, you’ve got a solid strategy. Now convince me that you, particularly, can build a team and make this happen, within the window you have available. This is 99% about the team. Can you hire people? A lot of them? Get them working together? Do you know how to ship software, sell software, market it, etc? Some of this you can do through force of will, some you can do by just being smarter or faster, and some requires that you really nail a couple of pieces, but most of it is about building and scaling a company that can execute your strategy. This is the grit, the work, the hard part. The best strategy in the best market is worthless if you can’t do the work.
There’s one more critical point about risk. It should be obvious, but somehow it’s not.
You want to minimize the risks you take. There’s something about what you’re doing that’s unlikely, probably scary, and quite possibly stupid. That thing has to exist, because it’s what makes your business valuable. Without that huge risk, you can’t possibly achieve the massive payoff you’re looking for. Take that risk, put it on a pedestal, love it.
And then do everything you can to not let other types of risk into the rest of your business. If you know your major risk is strategy, why introduce a massive execution risk at the same time? No. Your work is scary enough that you should be conservative everywhere possible. Pick something you know works when you have that luxury, because you won’t get it very often.
This frame should at least get you the first three slides of your presentation. There’s plenty more to know before you can convince someone you can go out there and win, but being able to clearly talk through the risks in your business and how you’re managing should at least help you get a seat at the table.
You’re doing productivity wrong. Actually, it’s worse than that. You have no idea what it is. And no, this isn’t some sort of technical, jargon-y sense where I’m arguing over whether to measure it in terms of lines of code, hours spent typing code, or something else. Your core understanding of productivity is entirely wrong, and it’s making your company worse at everything it does.
Productivity in the software world is discussed in terms of the individual, and usually phrased as getting more of a person’s time on the thing they were hired for. “Hey, I didn’t hire you to talk to users or attend training! I hired you to write code!” Even the developers join in: “Every minute I spend in meetings instead of writing code is wasting company money.”
This is stupid and wrong.
I had a conversation with a CFO recently who asked me why his company should use Puppet. I asked him in turn, what does it cost you to ship software right now, and what’s the value of lowering that cost? He said, not surprisingly, that he could not answer that. This isn’t to pick on him - most software CFOs would say the same things, since their job rarely delves into software operations.
Can you imagine having the equivalent job description with a manufacturing company? Picture the CFO of GM saying their job doesn’t require they understand the financial aspects of building cars. They’d be shipped out in disgrace. Yet somehow the software world tolerates ignorance of the financial implications of their core work product, software.
Sure, you might say, our finance teams can’t help us build more, better software for less money, but we don’t need their help because we intuitively know that more time spent writing code equals more productive development teams. Right?
No, you don’t. Again, you’re wrong. This is exactly, specifically how you’re wrong.
Toyota showed decades ago ensuring everyone is busy all the team is a great way to lower productivity. Yeah, you read that right. Lower. If you can’t shut the factory down (thus making everyone idle), you can’t fix problems at their source, which means every car you make is broken in the same way.
As another example, teams can almost never be perfectly balanced in output. Maybe you make doors much faster than you make engines, so if your door team is always busy, you end up with too many doors. Just like in development: The interface team might work a bit faster than the backend team, so they build an interface on a myth and then you can’t ship their work because there isn’t coverage on the backend. Oops. But hey, at least that interface developer didn’t spend any time in training, testing their prototypes with users, or bringing fancy coffee to the backend team.
The stupid thing is modern industry figured this out with its very first business consultant, Frederick Taylor, who developed a thing he called ‘Scientific Management.’ His ideas sounded good: I’ll hold a stopwatch and find ways to encourage people to get more done in a given amount of time. Seems reasonable, right? Probably more advanced than how most software companies do it today. In fact, he was really successful: He spawned our nation’s first management consultants, and they’ve continued to make tons of money on the fraud that he perpetrated more than a century ago.
Because that’s exactly what he was: A fraud. He was fired from his first gig because it ruined actual productivity, yet managed to convince corporate America that it was such a good idea they should ignore the little problem of it not working. Here we are a century later and the Harvard Business Review is still giving advice using Taylor’s own language.
But, you ask, what do I focus on if I’m not allowed to just track what the developer works on?
Now, Grasshopper, you set foot on the path to wisdom.
These are examples of waste in the form of rework (fixing quality problems later) and inventory (the built but unshipped interface work), but there are many other kinds that pervade modern software development. Lean Manufacturing is our most complete and coherent philosophy of productivity, and its name isn’t just decorative. It reframes the problem of optimizing productivity into the practice of reducing seven (or sometimes eight) kinds of wastes, including those above.
The funny thing, though, is that none of these wastes are visible if you just look at the individuals involved, so measuring what the people do isn’t just worthless, it’s actively counter-productive. See what I mean about all the wrongness?
Transport is a great example. You might have a person at your company specifically tasked with this (the most famous example from this is certainly from Office Space, but there are rational reasons for such roles in modern software teams). Intuitively you’d want this person to be 100% busy, yet optimizing the system to need more transport hurts the system even while it makes your individual stats look better.
Our world of software is fundamentally confused because it can’t move past thinking about the people to seeing the system. The first page of results on software productivity pretty much exclusively focus on the people, and even developers will say they should be spending more time writing code and less time doing other stuff (like in meetings with product managers or customers).
This is the challenge of our industry over the next few decades. It took around two centuries to get from the first industrial mills in the 1700s to the development of lean in the late 1900s. There are times where I think we can do better, learn faster, but then I realize how much more complicated our problem domain is, and I start to wonder. At least in factories you can physically measure inputs and outputs, and you’re consuming and producing discrete, easily measurable work. With software - especially SaaS built on the public cloud - discreteness is impossible, and even measuring the cost of inputs often approaches impossibility. It’s much harder to optimize systems that extend far beyond our own data-centers (if we even have them).
Nonetheless, this is our challenge. The first step is to move away from this focus on people, from this view that the developer’s precious time at keyboard is our main concern. Only then can we start to truly get better at building better software, faster, and for less money.
Early in my tenure at Puppet, I had one of my employees, José, make laptop stickers. He said we already had some, and I said, yeah, but they suck. So he designed something different. I said nope, still not right. Finally, in frustration, he said, “You clearly know what you want, can you just tell me what it is?”
I expect many of you will recognize this problem. It is sometimes called “Bring Me a Rock”, which pretty well captures its absurdity. Becoming a better leader required I find a way past it.
To my chagrin, I am often described as having “vision.” I hate this. It implies I know what I want, that I have some kind of clarity. I don’t. If anything, I pride myself on my uncertainty. I am a deep believer in open-minded, unsentimental experimentation and I think one of my true skills is knowing where to focus that research, rather than knowing what the research will find. This is pretty much the exact opposite of having a vision. So why did so many people think I had one? Why did everyone around me look to me for answers, instead of doing their own exploration?
I was tempted, but Kanies’s Razor kept me from believing it was just because I was so smart. Finally, someone partnered fantastically with me and helped me see that these are the same problem, solvable through better communication on my part.
In 2014, Puppet moved into a large, new office in Portland, OR, and we had the opportunity to strip the building down to its bones and build out the space we wanted. We were lucky enough to work with Corey Martin of Hacker. He and his team were great at really listening to me, and pulling out what I truly cared about, never getting frustrated or giving up when I was equal parts opinionated and incoherent.
I don’t always know what I’m talking about, but I’m pretty much always sure that I’m right. - Mojo Nixon
Corey quickly realized that I didn’t know what I wanted, but there was something worth extracting from all of the beliefs I was expressing. And, of course, I was paying him, so he did kind of have to actually listen to me. We spent the first term of our partnership working out the direction: What do you want the space to feel like overall? What impression do you want to leave? There are a few different ways to arrange this kind of office, which one appeals more and why?
Once we had that theme, now we could create an actual design. This is where we started pulling little ideas off the shelf - some mine, many theirs - but a design needs rules, some rubric to tell the good ideas from bad. For instance, one of the awesome things about our space is all the great windows and the views they provide. It was important to me that every person be able to get natural light (which is in short supply in Portland) and to have part of that view. This is one area I wouldn’t compromise, but there were many other hard constraints, for reasons of style, cost, or good old Newtonian physics. We captured all of these, and used them to quickly reject ideas that didn’t fit, no matter how great they might have seemed.
We still didn’t have a design, but we had a direction, and a bunch of constraints. Crucially, the whole team had them, they weren’t just some black box between my ears.
It turned out that this was enough of a platform that the team could quickly generate, tune, and reject new designs, so their creativity was opened wide, and my feedback on their ideas made sense. Once the framework was explicit, it became easily extensible. Suddenly discover a constraint you care about but haven’t shared? Add it to the list. Reevaluate your design with the new rules. Done.
They built a beautiful office. Yes, there are some details I lay claim to, and there are a couple of areas where I put my foot down on high-level design components, like open stairways between the floors, but this is their design, their awesome work space.
As I reflected back on how successful this partnership was, I knew it was a teaching experience. You learn far more from successes than from failures, and most of my attempts at this kind of partnership had failed.
Obviously, a big part of it was just how well we were able to partner. You can’t force this. Corey and his team trusted me and knew they had to deliver something I loved, and I trusted them, knowing that they were great at both the creative and implementation aspects of their jobs.
The framework that grew from this partnership enabled the team to do great work, work that I loved. I think I’m most proud of just realizing how special the experience was, ensuring I took the time to understand why. That investment provided a tool I’ve been able to use in almost all of my subsequent work as a leader.
My single most important role as a leader is to describe the direction I want to head, and if at all possible, I need to explain why. A lot of leaders can’t or won’t cover that, and frankly, lots of people have done great work without ever knowing why they were doing it. I agree with Simon Sinek, you’re a better leader if you Start With Why, but you don’t meet the basic qualifications of a leader if you can’t at least point the team in a direction.
That’s not enough, though. The gap was why I often found myself saying the equivalent of, “No, a different rock.” Working with Corey and his team taught me the importance of expressing the constraints, not just the direction. These allowed the team to take ownership of the design, which was absolutely critical to unleash their best work.
This framework of communicating direction and constraints provided necessary separation. I could step away, stop being in people’s faces and checking their work, and they could go off to their lab benches and iterate in peace, knowing both that they could quickly see if they were close, and also that if we had some sort of problem, we had something we could tune and evolve.
In the other big projects I’ve had since then, such as rebranding Puppet, designing new products, and hiring executives, this framework has helped immensely. With it, I am a better leader, and a more pleasant leader, and the teams I worked with do better work, faster. And we all enjoy the process much more.
Every founder understands the importance of culture in the long-term success of their company. Yet rather than valuing and supporting all employees, organizations like Uber still manage to build a toxic, poisonous one. How does this happen, and what can you as a founder do to prevent it?
We’re still in the early days of understanding how to build great culture. However, the barriers to doing so are surprisingly similar to another area where we’ve come leaps and bounds in the last decade: Creating great products. The similarities in their problems mean the tactics for solving them should be portable, too. In both cases, the barrier is essentially:
You’re creating for someone else, and they get to decide whether you succeeded.
Your opinion matters, but only as part of the feedback loop between your work and the outside world. Eric Reis and Steve Blank have done a great job of teaching us that facts live outside the building, and the real test is to go find them. You can’t rely on your own perspective, because you’re not the ones deciding whether to buy.
A product solves a specific problem for a specific customer, whose perfect solution might fail the very same problem on another person. When it comes to testing your product, the only experience that matters is that of your target customer. Failing them is failing full stop.
Your culture has the same need to dial into your audience. As the founder, it doesn’t matter what you think of the company’s culture. It’s not for you. But for better or worse, you do get to decide who your culture is for. A lot of these failures of culture are actually the CEO making bad decisions about who the culture is for.
For example, we all know stories of toxic employees driving other people out of the company. The excuse we hear is that this person is “too important to fire.” It’s presented as a choice between firing this person or making some people uncomfortable, in which case it’s easy to keep this high performer around. But that’s not actually the choice you’re making.
When you choose not to fire someone who harasses women, what you’re really saying is, “This person is more important to my company than women are.”
You’d rather your organization and leadership discriminate against women, have them leave your company, and have your people and teams suffer all the psychological damage that results from choosing to protect one person over a whole class of people.
It’s the same for an employee who discriminates against minorities, veterans, disabled people, or any other group. You can keep that one person, or keep the ability to hire from that entire labor pool.
Of course, that toxic person won’t actually prevent you from hiring from a whole group. That would actually be better in the long run. Instead, you’ll hire a few, but they’ll only stick around long enough to be almost valuable to you. People who stick around for one to two years are incredibly expensive - but your toxic employee will cause them to leave. You get to pay to recruit and replace them, to train them up, to watch teams form, and then to watch them fall apart again. And again. All because you couldn’t make a hard call on one person.
Conveniently, you’ll then be able to look back after a couple of years and say that these people who left just didn’t fit in, as though it’s somehow their fault that you’ve designed an organization that excludes them as target users.
I think there’s a strong moral case to made for inclusion, but you don’t need it. All you need is basic economics. Great people are scarce, and every group you exclude reduces that pool even further. Your investors want you to have the biggest possible user base to increase your addressable market; don’t you also want the largest possible employee base? Women are 51% of the population, and in the US, whites make up 72% of the population. If you only hire white men, with maybe trifling exceptions, then your target market is only 36% of the country. Reduce to just the working age population, and you’re at 22% of the pool. And that’s before you slice for experience, location, etc., much less silly requirements like a degree from an elite school and experience working at Google or Facebook. Every demographic you exclude reduces your possible employee base, and with it your chance for building a great business.
Let’s say I’ve convinced you. You have to design your organization to include other groups. Now what do you do to ensure it’s working for them?
Again, the facts live outside the building. Even worse, as the founder you don’t get the truth. Most people do not speak truth to power because they either feel unsafe, or actually are. Even worse, most powerful people don’t want to hear it. This attitude helps explain why a lot of cultures are so broken. You have to fight your instincts, your comfort, and often, your happiness to get the truth from those who own it.
In the fall of 2014, I had dinner with eight women who worked at my company. This was my most difficult experience as CEO. They were pissed off. Their experience was very different from what I thought was happening, and hearing it directly from them hurt.
I realized in hindsight that I’d made the tactical mistake of collecting the most upset women in the company - getting together with everyone who was having the worst experience- but that didn’t make their experiences any less valid or their feedback any less important.
Thankfully, I was aware of some aspects of what they were struggling with, and I was able to point to three separate projects I was working on, all with defined goals, owners, and deadlines, that would target what I thought were the biggest issues. But I had not realized how bad it was for them, and I was far from achieving my own goals in this area.
My behavior as a result of that dinner would make an explicit statement about who my company was for. I could have taken that deep pain, that searing discomfort, and run from it. I mean, I didn’t experience any of that, and all I had was these whiny women who had this problem. Right?
Would you accept that with your product? Would you dismiss the feedback of eight customers who were on the verge of dumping your product? Would you look at their feedback and say, well, it’s just eight whiny users, I can ignore them?
Many founders choose this path in culture. They keep around the bad managers, the toxic star player, or the oppressive HR policy. But like with your modern software, you have the opportunity - and I say the obligation - to do better.
So go, build your best culture, set high standards for your team. But be very clear about whose feedback you rely on to test whether you’re meeting those standards. If your goal is inclusion and diversity, which it absolutely should be, then you need to listen, really listen to diverse voices, and to those who are systematically excluded from most organizations.
If you can do this, you can emulate the Dodgers hiring Jackie Robinson and the Sorbonne University promoting Marie Curie as the first woman professor of general physics. If not, you will find yourself competing with them instead.
I’m a tech founder and recovering SysAdmin. I helped found DevOps and grew up in open source. These days I am convalescing from a decade of physical and mental neglect while running Puppet.
Read more
