The rituals of great teams | Shishir Mehrotra of Coda, YouTube, Microsoft

[00:00:00] Shishir Mehrotra: I genuinely value the reference check over interview signals. You know? If I had to stack rank in interviews, what is the best signal, the uh, reference check is the top of the list. Those, those people, you know, they work with sports crew, you know, sometimes for years. Like, their knowledge, what you're going to get out of 30 minutes of artificial scenarios is just like, never going to compare what, what a good reference check will give you.

[00:00:22] Lenny: Shishir Mehrotra is the co-founder and CEO of CODA. Before starting CODA, Shishir led the YouTube product engineering and design teams at Google where he spent over six years. Before that he spent six years at Microsoft, he's also on the board of Spotify. As you'll hear in this episode, Shishir is an incredibly deep and very first principles anchored on all kinds of topics. And in this episode we cover growth strategy, specifically a framework that he calls blue loops and black loops. We talk about the rituals of great teams, something that Shishir's been passionate about has been collecting from all the best leaders in tech for the last two years and which will soon turn into a book. We talk about I IN questions, which is not a German game show! He shares how he evaluates product talent and gives some really great advice on doing reference checks. We go into so many other topics, this is the longest episode that I've recorded yet, and you'll see why. Shishir is so full of wisdom and we could have kept going for at least another hour. And so, with that, I bring you, Shishir Mehrotra.

Hey Casey Winters, what do you love about CODA?

[00:01:28] Casey Winters: CODA is a company that's actually near and dear to my heart, because I got to work on their launch when I was at GreyLock. But in terms of what I love about them, you know I love loops, and CODA has some of the coolest, most useful content loops I've seen. How the loop works is someone can create a CODA and share it publicly for the world. This can be how you create OKRs, running annual planning, build your own map, whatever. Every one of those CODAs can then be easily copied and adapted to your organization without knowing who even originally wrote it. So they're embedding the sharing of best practices of scaling companies in to their core product and growth which is something I'm personally passionate about.

[00:02:05] Lenny: I actually use CODA myself, every day, it's kind of the center of my writing and podcast operation. I use it for first drafts, to organize my content calendar, to plan each podcast episode and so many more things. CODAs getting listeners this podcast $1000 in free credit off their first statement. Just go to coda.io/lend. That's coda.io/lend.

Hey Ashley, head of marketing and FlatFile, how many B2B SAAS companies would you estimate need to import CSV files from their customers?

[00:02:38] Ashley: At least 40%.

[00:02:39] Lenny: And how many of them screw that up and what happens when they do?

[00:02:42] Ashley: Well, based on their data, about a third of people will consider switching to another company after just one bad experience during onboarding. So if your CVC importer doesn't work right, which is super common considering customer files are chock full of unexpected data and formatting, they'll leave.

[00:03:02] Lenny: I am zero percent surprised to hear that, I've consistently seen that improving onboarding is one of the highest leverage opportunities for sign up conversion and increasing long-term retention. Getting people to, to their ah-ha moment, or quickly and reliably is so incredibly important.

[00:03:16] Ashley: Totally, it's incredible to see how our customers like Square, Spotify and Zoro are able to grow their businesses on top of FlatFile. It's because flawless data onboarding acts like a catalyst to get them and their customers where they need to go faster.

[00:03:33] Lenny: If you'd like to know more or get started check out FlatFile at flatfile.com/lend. Shishir, welcome to the podcast!

[00:03:47] Shishir Mehrotra: Thank you for having me.

[00:03:49] Lenny: I don't think I've actually shared this with you, but you're actually the very first CEO that I've had on this podcast. I actually have a rule of no founders or COs on the podcast at least at this point, and you're the first person that I've let break this rule, and so how does that feel?

[00:04:03] Shishir Mehrotra: [laugh] I've always been a rule breaker.

[00:04:06] Lenny: [laughs] perfect. Uh, so bio for listeners, just briefly, so you're the founder and CEO of CODA, you spent six years at Google where you were VP of product and engineering YouTube, basically leading the YouTube product team, spent six years at Microsoft, you're on Spotify's board of directors. You're also a prolific online writer, and that leads to my first question, which is, I hear that at CODA there's a contest internally who people who... actually maybe a little context, you encourage people to write a lot of stuff externally within CODA. You want people to be writing, and you have this contest of who gets the most views and likes out of people internally. So, is that true and who's winning?

[00:04:47] Shishir Mehrotra: By the way, yes, it's, it's sort of true, we, we did a similar thing at, uh, at YouTube, and YouTube creators, I mean, obviously kicked our butts. But it was a good way to make sure we understood our tools and learned how it worked. And, you know, I think for a while at YouTube I had one of the top videos. There's a really cute video of my daughter taking everybody's orders when she was like three or four years old.

[00:05:10] Lenny: That's not fair.

[00:05:12] Shishir Mehrotra: Yeah, super cute kid is a, is an easy trick for YouTube. Uh, but I got to learn all the tools and thought, you know, here the equivalent at, at CODA is you can publish CODA docs, they show up in our, in the gallery, go to, you know, CODA gallery, you can see lots of them, and um, you know, at this point thousands of docs have been published from lots of different people, it gets millions of views, and like a YouTube, the most popular ones are not ours, but it is sometimes helpful for us to make sure that we understand how the whole system works in order to do it. So, I think that the current winning doc at CODA is, um, is one in a pretty deep niche. It's from a guy named Kenny Wong on our data team. And, it's an orange theory workout doc.

So, it turns out that orange theory has this like deep subculture that they all hang out together on Reddit and so on, and this doc just took off as a... I don't really understand the doc, because I'm not in that subculture, but again it's sort of similar to YouTube in someways, it's like the, the, the niches are actually much bigger than, than people think. You know, of the ones that are, right, more work related, I think [inaudible 00:06:20] still outranks all of mine, but uh, I have a couple good ones on there too.

I would say this competition exists in my family too, and I don't usually even win within my own family. My older daughter Anika, who also had that great video on YouTube has a, has two docs that do really well. One is family quarantine Olympics which is a thing she put together at the beginning of, uh, COVID, like fun games for families to play. And the other one is a score tracker for a game called Ticket to Ride. Which have you ever played the game?

[00:06:49] Lenny: Mm-mm. But I've heard of it.

[00:06:50] Shishir Mehrotra: It's the most complicated scoring system on the planet. Because, it's like you play the whole game, and then you spend 10 minutes scoring it, and she built this little scoring calculator, and that turns out to be super popular too. So anyways, but, my, my docs do okay, but yeah, some, the kind of interesting variety of what people end up doing.

[00:07:06] Lenny: I love that. Which of your docs has been the most successful? Hopefully we end up talking about it.

[00:07:11] Shishir Mehrotra: I, I think it goes back and forth between two docs I've written. One called [inaudible 00:07:17] questions, which I, I think you intend to talk about, so we'll get to that.

[00:07:20] Lenny: Yeah. Absolutely.

[00:07:20] Shishir Mehrotra: And the other one is one I wrote a while back called four myths of bundling. Um, that's all about how subscriptions work, and it's how I kind of ended up on the board of Spotify was Daniel and I geeking out on bundling theory. Which is like a super weird hobby, but you know, I have normal hobbies too, but I like to, I like to explore bundling as a, as a kind of fun hobby and people enjoy that doc as well.

[00:07:41] Lenny: Yeah, you've shared a lot of good thoughts on that and we're not going to cover that, because you've covered that in depth in many places. But just to clarify, did you write that doc and that led you to being board, on the board of Spotify, or is that after the fact?

[00:07:47] Shishir Mehrotra: The conversation led to... so it was, it was a napkin sketch at YouTube that turned into a really fun lunch I had with Daniel, Daniel Ek at Spotify, uh, founder and CEO and then he encouraged me to write it down. And for me, I mean, I don't... you write prolifically and I, I have a, I have... writing for me is actually surprisingly hard. I, I feel like I have to think about it.

[00:08:08] Lenny: Same.

[00:08:08] Shishir Mehrotra: [laugh] you make it seem really easy. The uh, the, for me, like, he's like can you write that down? It's like, great, now I'm going to like take a year to write this thing down. Because, you like, you like think through each part of it, and you kind of come up with the right framing. I have a whole review process I use for my docs that, that allows other people to kind of help me make it better, which is, which is always really helpful. But yeah, that's, that's how that started. So it kind of got written down after. Yeah.

[00:08:47] Lenny: So, we're talking about writing and content and things like that and that's a really good segue to the first thing I wanted to talk about which is black loops and blue loops. Which, to folks who haven't heard this before, it might sound like some kind of ultimate fighting nightmare scenario. But, it's something that is really important to you and CODA and so to set it up, can you just talk about what black loops and blue loops are.

[00:09:08] Shishir Mehrotra: Yeah, and it's, it's probably worth mentioning, I think a lot of businesses have a diagram that describes their ecosystem and how it works. Sometimes it happens a little later in a, in a companies journey. For us, we were probably three or four years in before, uh, Matt Hudson's uh, um, runs our data finance teams here, he came up with this, this diagram, and it really stuck for people. But, I highly encourage drawing a diagram like this for your business. I'll, I'll flash it up on screen for a second, and I'll describe it, but this was, this is what the diagram looks like. Black loop, blue loop, and it's basically the two different ways that our product spreads.

The black loop is someone comes in, they make a doc, they share it with a group of people, subset of those people turn around and make another doc, and then the process repeats itself over and over again. The blue loop is, someone comes in, makes a doc and instead of sharing it with, with a team or, or with the collaborators, they publish it to the world, and in that process expose it to, they can choose how it should appear, what publishing, and CODA is a lot like building a website, you pick a URL, you tell us whether or not Google should be able to find it, should it show up in the gallery and so on, and what, what ends up happening from that is they turn into broad promotion of CODA, but really it's about that person, what they're trying, trying to get done.

I'll stop sharing so I talk a little about the dynamics. So, they, they... I sometimes refer to them as the, the Microsoft loop and the YouTube loop, because there's two inspirations for it. So the black loop feels a lot like how documents naturally spread. Like, the viral actions of a, of a document platform are share create, share create, it happens over and over again. The best way you learn about Office or GoogleDocs or so on is somebody shares one with you, and you're like, oh that's pretty cool, I bet I could create one, and that loop can happen very, very quickly, and it really drives, for us, a lot of how we think about how we work mostly within companies and teams but sometimes across them as well.

And so, for example, it led to our pricing model. So, our pricing model is, um, a little differ than most, most companies in this space, that we do what they call make them, bill them. So basically, all, all document products, for all documents of the metaphor have, have three personas. People who can see things, people who can change things and people who can create new things, basically everybody charges for the top two. They charge for editors and makers. If you can make changes, then you have to pay, and that's like every document product you can think of including ones, you know, drawing, they all, all kind of do the same thing.

We decided we're only going to charge for people when they make a document. So the, the, you think about it, you get a CODA doc, only, you only need one, you know, if you're using any of our paid features you only need one paid license uh, for doing it. And the reason we do that in terms of that diagram is that I wanted no friction of the share edge. I mean, the share edge for us is like, that's the moment of, hey look I, I'm doing this thing, it's so cool, and, and that's the moment where the, the line I gave to the team is I want no dollar signs in the share dial. So that's, that's your... every product has it's moment of how you, how you spread growth, and going back to YouTube, imagine you had to pay for people you shared with.

Like, nobody would ever share anything. But, that's how basically every productivity product works is the moment they charge you is when you share with somebody. The blue loop, I often call the YouTube loop, because the emotions of publishing a doc are, are incredibly close to that of publishing a YouTube video. And people have all sorts of reasons why they do it. I mean, sometimes people do it, um, there are people who do it for money, but a lot of people do it for exposure, for brand building, they just want to get an idea out in the world. They want to get feedback, some people do it for fun, some people do it as a, as a charitable contribution. There's lots of reasons why people do it, but the net effect of what happens is, you know, for YouTube, the vast majority of how people found out about YouTube was through a video that was shared with them. I mean, that's sort of the, uh, the, uh impact.

But when it changes the dynamic that allows everyone who publishes a CODA doc, now it's a very natural incentive to go share it with, with the right population. You know? If you're, if you're in orange theory, you share it with the orange theory population. If you're into playing Ticket to Ride, you share it with the Ticket to Ride population. But, they, um, if you're into bundling you try to find the, a small group of people that care about bundling. I don't know about that, but the, um, and, and what happens for us is, that then becomes a loop that, um, means that most people's exposure.

And almost a third of our users come through this, this loop, they're not actually exposed to CODA. They're exposed to a great idea for, you know, how to run an offsite, or how to, how to, you know, how to win Ticket to Ride, or whatever it might be. And in that process they learn about the product. And so then they come in, uh, through this vehicle, and you know, one, one reason is very important is because for components like ours that are very horizontal, you get, you get um, you get different types of users. There are some types of users that I call the building block thinkers, they like to kind of build up from scratch and the blank surface CODA is really amazing for them, but for most people, that's intimidating. I don't, don't really know what to do.

Most people in the world are, are, are problem solvers. And so they start, not by do I need a new document, they start with, I've got a problem. You know? We don't make decisions about stuff in our company, or you know, my family can't figure out what to do on the weekend, or whatever it might be. And then when they find a solution to that problem, they then pick the right tool, and so the blue loop allows us to uh, to go after that. It kind of changes how the motion of the ecosystem works as well. But that's what blue loop and black loop are.

[00:14:31] Lenny: Awesome. I have a bunch of questions I want to ask. The first is, how did you... for a founder listening to this, and they're like, "Oh man, what are my, what are my loops? What's my fly wheel, how do I think about my business?" Can you talk a bit about how you kind up on this way of thinking about the company? And then also, how do you structure teams to kind of work in this way, if this is the way you're thinking about growth?

[00:14:50] Shishir Mehrotra: Maybe on the first part, uh, I think you just hosted Casey Winters who's like pretty, pretty famous for talking about loops, not funnels. And I do think there's a, a very natural thing when you're building a product, or building a business to think about your funnel. And you think about things as being linear. That somebody comes in, they go up to your signup process, and then they see your onboarding, and then they get exposed to the first magic moment of your product and the second magic moment and so on.

But the truth is, it doesn't really work that way. I mean, almost all products have some form of loop. I mean, that person turns around maybe sharing is built right into your product, or maybe it's not, or maybe there's a way that it happens through advocacy and so on. But, understanding the, the way products actually grow and spread happen through some type of loop not funnel, is, I think, pretty fundamental. So your first piece of advice I would give is, you probably do have a loop. I mean, whatever the product is, there's probably something about it that causes that loop, and understanding how that, that works is really important. I mean, in terms of like, what, what it is, you know, in our case I'd say if you take black loop and blue loop, you know, the black loop is every product in our category has that loop.

I mean, it's actually, we didn't invent it, and you, you, you build a document, you put a share button on it, you know, every product has that. Though sometimes it's just recognizing what's there. I mean, it's not, it's not that, it's not that interesting. The blue loop on the other hand is not something that every product in our category has, and it's not really a thing that you expect to do with GoogleDocs or Office or so on. It's a, it's kind of our unique take on hey we're going to build a publishing platform that isn't just for sharing ideas, and building things with your team, therefore putting things out in the world. I mean, one of the best, best compliments we hear about the CODA gallery is I had this user tell me this line I really love. He said that the CODA gallery feels halfway between media and an app store.

And, you can come and you can read about anything interesting in the world. And, you can go shopping and say, "I need one of those, I need one of those and I need one of those." And it's my view that this category, we call it the all in one doc category, I think this is going to be critical. I, I don't think that there's enough people out there that are looking for, like a horizontal new blinking cursor.

I mean, they exist and you could get through your first million users that way, but I think to get to the like, the level of, of impact we want to have, you've got to find this [inaudible 00:17:00]. So you probably have a loop not a funnel. And it might be hiding in plain sight, or it might require invention. That's, that's the bound or dance and the fun of it is, but, but um, finding it, writing it down I think is really helpful. Um, uh the second part of your... oh how do we organize our team.

[00:17:18] Lenny: Yeah, but let me ask one quick question.

[00:17:19] Shishir Mehrotra: Yeah.

[00:17:19] Lenny: So you said this, I think it was the data scientist that kind of first imagine they kind of diagram this out, because I'm, I'm curious as a founder, they're listening they're like, "Oh, how do I, how do I find something like this?" I imagine part of this was, oh this person brought you this interesting way of thinking and there's find of this process of oh wow this is cool, let's think about this. What was that like kind of just, or high level, that process.

[00:17:40] Shishir Mehrotra: He currently runs data and finance for us, he's actually one of the early founding members of the team, Matt Hudson, so he's kind of in every job [inaudible 00:17:46]. He's very insightful, but honestly the idea didn't come from anywhere. I mean, there's a really famous loop diagram or for Uber, then I think, I think one of the core members drew it, or Travis drew it. I don't-

[00:17:59] Lenny: I think Bill-

[00:18:00] Shishir Mehrotra: Yeah, right, the napkin. The napkin sketch and who knows how true that is. The, the... probably lots of dispute on that. Uh, I think Airbnb had a similar diagram, I'm trying to remember.

[00:18:12] Lenny: We, we tried. You know?

[00:18:13] Shishir Mehrotra: Yeah.

[00:18:13] Lenny: We had some sketches.

[00:18:15] Shishir Mehrotra: Right, right. So the um, it can kind of come from anywhere. I do find that the most natural place to, to see it is just when you're talking about your business to someone you're pitching. A customer or a candidate. I actually think candidates, you know, I think we're going to talk a little bit about it in the interview, but I think talking to candidates is one of the best ways to hone what your business is about. Because those, those people are in some ways even more critical than investors. I mean they're investing their, they're investing their time, not just their money. And, and so your ability to get across to them why this thing is going to be interesting and how it'll grow and the kind of the most discerning investors out there in a lot of ways.

They're actually not that easily confused by metrics and so on that could be temporary and you know, they're, they sort of put themselves in that picture like can I see that happening? And so for us, like the black loop part was pretty obvious, the blue loop part, you had to like squint a little bit to think, will people really do that? Will people come and you know, publish these documents like, you know, some hybrid between websites and blog posts and templates. Like, what, what, what are they going to, what are they going to do, and why and so it required a little bit of creativity, which forced me to get better and better at pitching why that's going to happen. And what that role is going to feel like, and you know, this analogy of half way between a medium and an app store is like, it helps people crystallize what that, what that promise has to feel like.

So I think that, the, that this idea didn't come from anywhere, but if you want to mine for your own loops, go look at what you told the last few candidates you talked to.

[00:19:48] Lenny: I like that, it just takes some, takes some attempts at drawing some kind of diagrams, that's kind of how I thought about that when I was thinking about it.

[00:19:53] Shishir Mehrotra: Right.

[00:19:53] Lenny: Do you find that the quality of user is different amongst the loops? I imagine one is like 80% of the growth but maybe the other's a different type of user, maybe higher quality? I think about a little bit with Airbnb referrals drove higher quality hosts. Even those it was still a small portion of all hosts and so it was a really lucrative and interesting channel. Do you find anything like that?

[00:20:14] Shishir Mehrotra: So, I mean, quality and activation are a little bit different. I mean, the, the, the blue loop definitely, the, so there's actually kind of three entry points on that diagram. So people come through the blue loop, those people get shared through the black loop, and then people come through the top of the funnel. They, they, it has to be like, your, your seed population. Right? They, they have to... somebody starts with blinking cursor and nobody shared anything with them, either blue loop a templated document or, or black loop in a way the team is running. You know? If you look at activation retention and so on, certainly the, the highest is the black share loop. I mean, that, that, you kind of... if somebody shares a document and says, "Hey this is how we're running the staff meeting." You're just going to use it.

So, the uh, the job or retention there is um, is a little different. And actually one, one thing that is... actually let me come back to [inaudible 00:21:01]. Second, second best is through the blue loop and then the third, you know, the worst, the hardest is activating through the very top. You know, from, from there, you know, roughly one in five people make it what we think of as our activation moment. Which is hard, right? It's like you're going to hand somebody a new product that they didn't start with the problem on, and nobody handed them a document and said just work in. That's, that's hard.

Now, now all our nexplos are really important and so on. But if you think about these three different dynamics from how you, how you asked about how you construct your team, they're incredibly different mindsets, right? Because coming from the top of that diagram, we get to own the conversation. We, we tell you, we, we have our opportunity to tell you what the products about, what you should do with it, here's the, there's the minimal set of things you need to know in order to be productive, and gradually reveal the other things that you might need to know. Meaning the order of those things is wrong, real trouble and so on. But it's actually a minority of how people get exposed to CODA.

Because in the black loop, you know, the person who owns that conversation is the person sharing the document with you. If that person does it and mispositions it, or doesn't just like, just makes a crappy document, or something, we can't... we have to like, help them on board their users, would become our users. Same thing in the blue loop, you know that conversations now owned by the publisher who's... they, they're really not that interested in teaching anybody about CODA. They're mostly interested in here's this like, really cool way to do orange theory, or here's this, here's this interesting way to, you know, run a meeting. And, and so one of the interesting things about building platforms, which I think is a little bit different than, than products that they get to be sort of direct.

I mean, I have this, for better or for worse most of the products that I've gotten to work on are platform products, and I, I find that there's few very different kinds of people that like that challenge, so some people, I think Steve Jobs was the quintessential example of he didn't really like being a platform. You know? The iPhone shipped without and app store, you know, they locked down the screws on the back of... on, on all of the devices so nobody could open them, and his viewpoint was I'm going to control every element of what my users see, and on the other-

I'm gonna control every element of what my users see. And on the other hand, you know, platform thinkers, you, you sort of assume that my connection to my eventual user is through someone else. Like YouTube, you know, uh regularly they come to work at YouTube, and somebody would say, "Well, here's what happened last night," you know, and- and uh, sometimes it's heartwarming. Like, "Oh my gosh, this person ... you know, this kid bit this other kid's finger, and it took off like crazy, and you know, this- this Korean pop store just like broke through the billion view mark before everybody else did."

But, and sometimes, it was not heartwarming. [Laughs]. And they're like ... and you don't get to control that, because that's the uh, part of-part of being a platform. And so it does change, you know, how you think about the- the way you run the team. Because, if you have a loop where your community, ecosystem, user, so on control that narrative, then you have to- you have to in- incorporate that.

I mean, like another close analogy to think is like, you know, Airbnb and the famous story of them, you know, taking pictures of people's apartments. It's like, you know, they had to... they had to reach out and try to control that. And- and eventually you can't do that. And have to ... you had to sit back and let people, you know, market their... market their hotels. And uh, you know, thankfully they just got good at it. But it kind of had a similar dynamic, I think.

[00:24:14] Lenny: Absolutely. And with Airbnb, pricing is even more of a challenge. A lot of hosts don't really know what to price. They think their place is worth a lot more, and we can't tell them the price. There's laws around that and so it's like, "Maybe you want to price it at this rate. You know what's good for you." So it's a lot of encouraging. So yeah, I've seen that in action.

You talked about Teams and how you think about structuring them a bit and that's a good segue to our second topic, which is around a book that I hear you're writing, called "The Rituals of Great Teams." And, I think what you're doing there is exploring rituals that have emerged at some of the more successful companies. And so, just a question there. One, how'd you get interested in rituals, so much so that you decided to write a book, which is such a trudge and endless amount of work? And, just like, yeah, how- how ... where's it at? How's it going? And then I'll ask a few more questions there.

[00:24:59] Shishir Mehrotra: You know, the, the writing a book, so I'm writing this book. It's called "The Rituals of Great Teams." And the, uh, when I signed up to do it, I thought I was going to take six months. I'm now almost two years in. Uh, I [inaudible 00:25:09].

[00:25:09] Lenny: Approximate.

[00:25:09] Shishir Mehrotra: [laughs] I can hold my manuscript in four months and I am probably half done, so there's a lot of work, uh, ahead of us in, um, but it's one of the most fun projects I've ever done. So, so the history behind this was as, as in many cases, is a lot of, you know, odd luck and, and happenstance. I got hosted, right at the start of the pandemic, I was interviewed for a different podcast called Masters of Scale by Reid Hoffman. And the way Reid records, which is not, I'm not sure I would recommend this, but he does it a little bit differently than, than, than you do. Sit down with, you know, no idea what you're going to talk about and you talk for like three hours. And then he has a group of editors, the same group that actually edits TED. And they come in and they pick 20 minutes of it, and they turn it into an episode. And you have no idea what it's going to be. You talk and talk and talk and gets the 20 minutes out. And they're pretty good at getting to a nugget.

So they picked, out of this whole discussion, this part that I thought was like really small. And it was, um, Reid had asked me for one of my favorite quotes. And I talked about this quote from a guy named Bing Gordon. If people don't know Bing, Bing was one of the founders of Electronic Arts. He's a famous investor around Amazon, Zynga, so on. Lots of great companies. And I happened to sit on a board with Bing, and he used this line. I think Bing's one of the best nonlinear thinkers in the valley. Uh, always learn something with uh, with Bing. And he- he used this line that really stuck with me. He said, uh, "Great companies has the very small list of golden rituals. And there are three rules of golden rituals. Uh, number one, they're named. Number two, every employee knows them by their first Friday. And number three, they're templated." And, and he has great examples. You know, Amazon has six pagers and Google has OKR's and Sales Force has v2mom and, you know, there's all these different rituals that, that people do.

And I ended up sharing on the podcast a little bit about what Coda's golden ritual is. If you were to ask a, a set of Coda employees on their first Friday, "What Coda's golden ritual is, they would almost certainly tell you about this thing we do called Dori Pulse. It's sort of the key of how we run meetings and do write ups and so on. It's a pretty simple idea is that in our write-ups, instead of, um, and in our meetings, instead of going around the room and hearing about what everybody thinks, we do this thing called pulse. Everybody writes down what they think and we hide everybody's else's until you're done writing. So you're kind of- You force yourself to- to be eloquent about your opinion on the record about it, and unbiased. Then then the second thing we do is called Dori which is instead of sort of randomly asking questions, we ask people to put the questions on the table and then we take a round of up voting and down voting them to actually figure out what we're going to talk about.

Dori's named after the fish who asks all the questions. It's- it's a tool we use a lot at Google that we kind of turned into this mini tool. If you were to ask a set of Coda employees on the first Friday about Coda's golden rituals with Bing's three tests, they'll almost certainly talk about Dori Pulse and it's not because they're meeting [inaudible 00:28:04]. It's because it's indicative of the culture of the team, and so the, uh, you know I'll regularly hear employees say things like, "I just joined Coda. It's been a week. It's amazing. The culture is so open that I got to ask a question at a meeting and it outvoted the CEOs." Um, or they'll say, "I got asked for my contribution to this, this really hard decision we're making and it was thoughtfully presented. I had space to be able to do it well without bias and it was actually read to be considered as part of the decision-making process."

The podcast on, um, Reid's podcast did pretty well and I got all these questions about rituals, and so I decided to do a, uh, a dinner which turned into a dinner series. And we basically every third Wednesday we would post a group of people to share their rituals with each other and I learned a bunch of stuff in this, in this process. I've now interviewed over a thousand people for this book and, and you know there's lots of really interesting rituals that come out of it. The first thing I learned is people love sharing their rituals. I mean- you know I've interviewed people from many companies everybody's heard of, you know Nike, Disney, New York Times, you know all the way down to many- many start ups that maybe people haven't heard of or companies in industries that people don't, uh, talk a lot about, book authors, pundits, lots of different people that have come to this process.

Everybody loves sharing their rituals. I mean everybody's little secret to how they run their business, but for some reason, the how we work part, you know, everybody's very willing to share. People also love hearing about them, and I was kind of, it's like is this going to be interesting for a dinner, like, you know, geeking out about how a team works? And it turns out not only the people like hearing about it, it's the, the littlest details that matter. It's like, "oh yeah, we kind of do that too, but we have this issue. How did you get past this?" And you start discovering that actually those little details are what make, what make or break a thing that, you know, you can't quite do it the exact same way. And then the other thing I realized, that's probably the most important point, is that rituals are, I like to say that they are mirror of culture. That one of the attendees of Dharmesh Shah, founder of HubSpot and very thoughtful, loud person, he talked a lot about this thing that originally was presented as something called [inaudible 00:30:13], which is a really cool example.

Dharmesh talked about how we, when we're building companies we actually build two products. We build one for our customers and we build another one for our employees. That's actually the how we work part of it, that's, the term he uses for that is culture. That's the product we build for our employees. I think it's a very, like, interesting way to define what, what culture is. Interestingly, when you ask people about their culture, "Hey what's the culture of Google or Airbnb or so on," the way they'll answer the question is through rituals. Say, you know, here's what we do and the way you know this is what we do is through this ritual that, that, that's in place. So I thought that was pretty interesting. Um, started off just building a little [inaudible 00:30:55] of like here's all the great rituals. And then I started realizing that I actually did a comparison between them that's kind of interesting. So I started sort of filling in the gaps between them of like when would you do X versus Y and what did I learn through that process.

Publisher asked if I would turn it into a book, and I, I, I, uh, agreed without really contemplating how hard it would be. It's just, it's become what most of my evenings end up being on this. I have a wonderful co-writer, Erin [Dame 00:31:20], who is incredibly gracious with her time and, uh, um, and her helping me sort through the best ideas or worst ideas, but it's a really fun project.

[00:31:29] Lenny: What are some of the more, uh, wacky or and/or impactful rituals that you've come across that you can share?

[00:31:35] Shishir Mehrotra: Oh boy. It- It's sort of down the list. I'll tell you some that are interesting and recognizable. One of the most fun ones is from Ariana Huffington, and she shared a ritual called reset. And there's a bit of background on Ariana, I mean, she's been well known for Huffington. She now runs Thrive. The um, she had an accident a few years back and ended up going through a period of doing a lot of research on how the brain works and ended up coming to this set of conclusions about how you can affect your own brain chemistry. And one of the things she does as sort of a personal ritual is a thing called a reset, which basically you make, the, the, the ritual is you make a one minute video that is, um, personal. And it's, they have a little template for doing it, but it's a breathing exercise.

So it's like you, you're supposed to like play it while you do this breathing exercise, but it's personal. So it's like, her video, you go search YouTube for Ariana's reset you'll find it. It has, you know, it has pictures of her kids. It has quotes she loves. It has, you know, videos of her hometown in Greece and, and so on. But the way she brought it to the team was they start meetings by randomly picking someone, they call it spin the wheel. They randomly pick someone and they play their reset. And the idea is you get this like, like two for one where you, uh, everybody gets, you know, a little bit, the brain chemistry rewiring of 60 minutes, 60 second breathing exercise. Everybody gets back into that sort of zen state a little bit, and you learned a little bit about each other. And the, she was saying that, you know, the pictures people pick and so on are interesting, but actually the music people picked is some of the most interesting.

Like a lot of people picked calm music. Some people will pick something they rock out to. Some people, but you know, everybody does their, their reset a little bit differently. That, that was a really fun one. Um, another really fun one that I was surprised by, Gusto does this thing in their hiring calls. You get an offer from Gusto and apparently when you get on the offer call of you know, "Congratulations. You got an offer," instead of just meeting their recruiter, which is what most companies do, they have the entire group of people that interviewed you join the call and they all say something about why you're amazing and you should join Gusto.

And, like I said, it's such an interesting ritual in so many ways. I mean, like, it's for the candidate obviously it's like what an amazing experience. Like, you know, the, to use an Airbnb term that's like a level eleven experience of, uh, of what that feels like, but also for the company. It's like, hey we're like, we're going to give, you know- One of the questions I asked, "What if I voted no?" What if I said this person is a no hire? They said, "Doesn't matter. You're on the call. You're going to work with this person. You're going to help them feel welcome and you're going to help them understand, you know, where they stand." I just, I think it's, obviously it takes a bunch of time, but you know it also is a signal to the company of how important hiring is and something that obviously we all, we all prioritize.

So those are maybe a couple of the different kind of, uh, maybe different ones that people might not have heard of before.

[00:34:31] Lenny: Those are amazing examples. Have you integrated any of these rituals from other companies doing this research into, into Coda?

[00:34:37] Shishir Mehrotra: Oh, all the time. And it's, it's the cheapest form of research. I get to like borrow all these great ideas from all these companies. We just added one to our decision making process. This is like, this is a good example of a little detail that really matters, is Coinbase has a ritual that's formed around, it's a decision making ritual they call- It's amazing how many companies have decision making ritual with a name, with a verb. So like that, uh, um, at Square [inaudible 00:35:06] called them spades. Like he kind of verbified it. It was a template. You know, you use Bing's three [inaudible 00:35:14], right? It's got to be named, every employee knows it by the first Friday and it's templated. Coinbase does a thing they call rapids. And rapid is, you know, a framing around what the, uh, the roles in it are, you know the responsible approver participating, informed and decider, but their technique of doing it was really interesting.

They have this like subtle nudge thing that we weren't doing that I've now incorporated at Coda. At Coda we have Dori and Pulse, a very common ritual we do. It spread through a lot of different companies that use Coda. You don't really have to use Coda to do it, but, you know, I think Coda's pretty good at it. But one of the things we were facing was that you would do this pulse. So you would have a meeting and it could, if you did it wrong it could feel like voting and it could feel like consensus building. So we would get this, you know, people would talk about it, and every ritual has its pro and con, but people would look at it and say very open culture, you're allowed to share whatever you want, but on the other side for the person that's trying to make a decision, it can feel like, "Oh my God. I now have 30 pieces of feedback. Am I supposed to wait for like all 30 of them to be yes? Am I supposed to wait for it to be a majority? How do I know what I'm supposed to do here?"

Coinbase has this really simple idea that we sort of smooshed together which was at the top of their rapids they named who all the people were. And then next to each one they put a little box that said what is the decision from that person. It just organized. It's very similar to pulse, but they kind of organized them and said like everybody that's in the informed bucket can like, they can comment. That's totally fine, but we really care about the approvers, the responsible and then, of course, the decider, the one that really matters. So we just added a column to our pulse, which is what is the role and we group the table by that.

And the other thing that Coinbase does, which sounds really subtle and small, but it really, a detail that really matters is the person running the meeting pre-fills that with what they want from that person. You are an approver. Maybe I have three approvers because I have a budget approver and I have a marketing approver and I have, you know, I have a sales approver, whatever it might be, and I need you to give me this answer. I don't need you to comment on everything I'm doing, but I need you to tell me do I have the budget or not, or I need you to tell me am I authorized to make this change in this part of the product that we generally don't change, or can I change the onboarding flow, or whatever, whatever it might be.

And we took a process that I think was like doing a pretty good job of getting, getting rid of group think which is really the heart of what we're doing with pulse, but had this danger of being overly leaning toward, uh, consensus building to a fault. I think consensus building is a good thing, but consensus building to a fault is not. And we sort of stole this one from Coinbase and we smooshed it in and it like got better. And I, I think that's a good, good example of a very recent one.

[00:38:04] Lenny: I feel like you have a clear best seller on your hands here, and I can't wait to read this. It must be like you have an unfair advantage right now having all these insights before you share them. Being able to execute so much more efficiently.

[00:38:17] Shishir Mehrotra: Well, yeah. It's, it's interesting. I don't think it's, um, I'm obviously not trying to keep any of it a secret, so with the whole point of publishing it is because I think other people will enjoy it, um, and can get, can get benefit out of it. But, you know, if people are interested, uh, one of the other choices I made in writing this book is I decided to do it somewhat in the open so there is a, what I call the rituals of great teams brain trust. So if you go, if you just search for me and rituals of great teams you'll find it and I'm sure we can add the link to the show notes, but, uh, you can sign up and basically as I finish a chapter, I put it out to the group.

There's now a few hundred people that are helping me co-edit this thing. Some of them just because they, like, some of them come in and they give me help on storytelling, grammar, and so on, but a lot of them are contributing. They show up and say, "Hey, you missed this one. We actually do that, but we do this other thing a little bit differently and you should really talk about that." I kind of view it as a, um, you know it started as a dinner series. It started with a, you know, everybody's going to give to each other and so I kind of wanted to bring that into the writing process. It's also a cheap way to get some editors, um, and pretty helpful.

[00:39:23] Lenny: I love that. That is really smart.

This episode is brought to you by Eppo. Eppo is a next generation AB testing platform build by Airbnb alums for modern growth teams. Companies like Netlify, [Contentful 00:39:37], and Cameo rely on Eppo to power their experiments. Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern growth team stack. This leads to wasted time building internal tools or trying to run your experiments through a clunky marketing tool.

When I was at Airbnb, one of the things that I loved about our experimentation platform was being able to easily slice results by device, by country, and by user stage. Eppo does all that and more delivering results more quickly, avoiding annoying prolonged analytic cycles, and helping you easily get to the root cause of any issue you discover. Eppo lets you go beyond basic click through metrics and instead use north star metrics like activation, retention, subscriptions and payments, and Eppo supports tests on the front end, the back end, email marketing, and even machine learning clients. Check out Eppo at geteppo.com. Get E-P-P-O.com and 10x your experiment velocity.

How have you found these rituals form for someone that's listening in or like, hmm, we need some of these. We need some rituals. We need to move more effectively. How do these come up? Are they just organically kind of organized? Do they come in from other companies and they evolve? Is it founder driven? What have you found so far? I know you're still working on the book, but curious.

[00:40:53] Shishir Mehrotra: There's a whole section in the book on, I get this question a lot. Like, you know, "How do I find the rituals we have?" "How do I adjust the rituals to match?" "Are they supposed to change?" There's lots, lots of different things you see because some rituals are like great when you're 100 people and they're terrible when you're 1,000. You should actively change those things. Some work great and what sometimes people call peace time versus war times. That you should do this during this time but you should like actively not do it when you're in this other time. And actually, that in itself is a ritual. I mean every company has some form of a war room ritual that is, I guess Facebook, I was learning, we did a dinner last night so I learned a little bit. Facebook apparently calls them lockdowns, which is a term I hadn't heard before, but apparently like it's well understood. When they say lockdown, everybody knows exactly what it means. It's like we no longer do the goal setting process. You know, you draw up this type of work and like everybody just knows this is what, this is what needs.

But I would say that, you know, when we talk to people about rituals, a set of rituals happen organically. I mean those are the easiest. Like Dori and pulse for us was one of the project managers running a meeting just though it was we're a, we're a distributed first culture, but like I didn't really adapted to it properly. This, this project manager just annoyed at waiting around on Zoom for everybody to go around and say their peace. It just took forever and by the end he was like everybody is like quiet. "Can we just pause? Can everybody just write down?" I mean he just happened to do it in this thoughtful way and it just took off. Some rituals grow organically and you've just got to like wait for them to, to do it.

But there are cases where companies actively form a ritual to, to drive a certain behavior. And, you know, the best advice I've given on this topic is, um, to read one of my other favorite books. This is, um, it's a book called "Switch." It's by Chip and Dan Heath. All their books are amazing, but they also wrote "Decisive" and "Made to Stick" and "Moments," but this one is my, if I could recommend five books, this would take two slots on the list. And the subtitle of the book is "How to Change Things When Change is Hard." The basic idea of the book is they use this analogy of an elephant, or a writer on an elephant on a pad. And when you're trying to change things, you have three options for what you can do. If you kind of map the Bing's analogy, so you can direct the writer, you can tell people what to do, you can motivate the elephants, you can kind of give this thing a kick in the butt and it's going to move. You don't know exactly where it's going to move, but it's going to move, and you can shape the path. Shape the path is I'm going to set this way up so that you can only do these things.

So, you know, the way I think about it is, you know, direct the writer, tell people what to do. That's why, you know, if you look at Bing's test, that's why you teach employees this before the first Friday because you tell people this is what we do. Some rituals, like how do you get this ritual to work? You put it in your new hire onboarding and you, you make it work that way. Motivate the elephant? A lot of that's about branding. So what do you do with rituals? Like there's an amazing number of rituals where I'll tell people, "That seemed like a great ritual. I would highly encourage you to give it a name. Give it something that lets people anchor ideas to." Names are very powerful things. If I, if I said, "Oh yeah, Coda, we do voting and sentiment writing" or something, I don't even know what you would say, it would sound boring. It wouldn't sound like something you could brag about, something you could form identity around. And so it's very important to give it a name.

And then finally, like, why do you shape the path or like set things up? You templatize. You make it easy. Make it as easy as possible to be able to follow this ritual and make it [inaudible 00:44:21] part of what we do enough that, you know, if you're at Gusto and you're like I don't really understand how this hiring call thing works, it's like guess what? You're going to be invited to one soon. Like you're going to see it and then you're going to have to do it, or, or if you're at Square, like you're going to see this [inaudible 00:44:34] template and you're going to, you're going to learn how to do it. So I think "Switch" is, I recommend this book for lots of purposes, but as you're thinking about rituals for your, for your teams, it's a particularly relevant frame.

[00:44:45] Lenny: Awesome. By the way, that's a little plug for the YouTube version of this podcast, which is now a thing we do. If you're like, "Hey, I don't see what you're talking about," just search for Lenny's podcast YouTube and I think you'll find it and it will probably

[00:44:58] Shishir Mehrotra: Yeah.

[00:44:58] Lenny: Link to the-

[00:44:59] Shishir Mehrotra: Yeah.

[00:44:59] Lenny: Add in the show notes. I'm also reminded of Airbnb's rituals, which you probably already know about, but they're kind of hilarious and weird. One is formal Friday where people dress up in suits and gowns on Fridays. Another is human tunnel for all new employees where every new employee has to run through a human tunnel and jump into like a beanbag or something.

[00:45:17] Shishir Mehrotra: Yeah.

[00:45:18] Lenny: Then there's a new hire tea time where new hires drink some tea with some veterans and chat about where they're from and things like that. There's a bunch more, but there's the ones that are my favorite.

[00:45:27] Shishir Mehrotra: Those are great. I also included some of the, the sort of Level Eleven thinking and it's also a good [inaudible 00:45:37] favorite. Brian also has another one that's in the book that's about how to rank your to do list by finding leverage. That's a, that's a really fun one as well. They're like don't rank your to do list, but a lot of people do like importants versus urgency or, or so on. I guess he sorts his by which of these most likely to create leverage of, like, getting rid of the rest of my list which I thought was like a very, like, I started doing that in my to do list and it's very interesting and [inaudible 00:46:01], so-

[00:46:01] Lenny: That's actually an incredible segue-

[00:46:02] Shishir Mehrotra: Listen, it's very interesting and impactful, so...

[00:46:02] Speaker 1: Yeah.

[00:46:02] Lenny: That's actually an incredible segue to our next topic, which are Eigenquestions.

[00:46:06] Shishir Mehrotra: Ah, yes.

[00:46:07] Lenny: And so Eigenquestions, one of your most classic posts, you mentioned this at the top, maybe your most liked post other than maybe bundling. It sounds like some kind of German game show, Eigenquestions.

[00:46:17] Shishir Mehrotra: [laughs]

[00:46:17] Lenny: Can you tell us what Eigenquestions are? And then I'm going to ask you a few more questions around that.

[00:46:22] Shishir Mehrotra: I hadn't thought about the German, uh, game question. Yeah, uh, [inaudible 00:46:30] Yeah. [laughs] We, we're going to, well, this would be a very interesting game, but we can, we can, we can try, try to create one.

Okay, I'll, I'll describe what Eigenquestions are, but maybe I'll start by telling you a little bit about where the concept came from. And, um, this is actually a YouTube-ism that, uh, maybe just to, to, to place ourselves in history. So 2008, I joined YouTube and, um, many people don't remember this, but YouTube at the time was seen as a mistake. It was seen as Google's first, first bad acquisition. Everything else had worked. But this thing kind of seemed like a disaster. I mean, it was, it was grainy videos. We were losing lots of money. We had billion, uh, billions of dollars in lawsuits. It was like, none of it really seemed that obvious. There's lots of discussion on how to reorient it, fix it, and so on.

And so everybody outside, that's what they, that's what they saw. If you stepped inside a YouTube staff meeting in 2008, actually one of the toughest questions we were answering was this question we called the Modern Family question. And, and, um, it may sound small but it actually was kind of, uh, very perplexing.

And the question was pretty simply. The question was, you know, people, if you looked at our search traffic at, at YouTube, one of the top five queries basically every week was for a show called Modern Family. And Modern Family was number one show on television at the time, you know, by far the most popular. And we were, you know, second, second biggest search engine in the world behind our parent company, Google. So search app was very important.

There was one big problem. People would come search for Modern Family. One big problem: we didn't have Modern Family on YouTube. And so we gave them some pretty crappy responses. And the question was, you know, oh, and, and, and, by the way, ABC.com had decided to post every episode of Modern Family live on their, on their, on their website, which nowadays is kind of typical, but in 2008 that was not typical. Those, it's a, kind of a big gamble they made and they're going to post all these, all these shows. It's like...

So the question was, you know, should we answer the qu- the query Modern Family by linking off to ABC.com? Do, do we link out or not? And the, the company basically divided into this, like, half the, half the company, mostly the product team, vantage team, so on, kind of aligned around a viewpoint that was that's what the user wants. Link them off to ABC.com. You know, we're now owned by Google. Google tries hard to prioritize, do right by the user. The rest will follow. Like, that seems like the right thing to do, so let's do that.

And then the other half of the company, most of the business functions, sales, marketing, especially content partnership, said please, please, please don't do that. If you do that and you start linking off to all these other places, nobody's ever going to put good content on YouTube and we're just going to get the stuff that doesn't deserve to live anywhere else. And that's, like, not a very good path to be.

And you can imagine that, like, those two mindsets, it's almost like it was, like, good versus evil debate, which is like, do right by the user or the business. We're talking, like, these are all, like, it, almost impossible to solve problems. And this would, like, happen meeting after meeting after meeting. What would, did we make a decision yet? Modern Family. What are we going to do?

So we had this offsite where we decided we're going to, like, spend the whole day and we're going to figure this problem. And we went up to this, this hotel at Half Moon Bay and the, the executive team all sat down. And I was asked to frame the discussion. Go collect everybody's opinions and collect all the data and just, like, ground it all in facts. And then we're going to have a discussion. We're going to reach a decision.

And so the night before, I'm sitting and thinking about, like, how, you know, how the hell am I going to have this discussion not just be, like, a shouting match of, you know, uh, of, of this good versus evil position? And I happened to read a, uh, analysis that was being done by a different team at Google, the Google Shopping Team where they were facing this interesting challenge of, um, they were in this deep fight with Amazon and they were getting their butts kicked and they were trying to figure out why. And it, the walking in theory was, you know, the Google Shopping Team's view was why would anybody ever pick going to Amazon? Like, you could come to, to Google and we had indexed all of Amazon and the entire internet. Why would you ever pick Amazon?

And the feedback that was coming back from users was they would, they'd say, "I picked Amazon because I value consistency over comprehensiveness." They would say things like, "You know, I, I really value that, you know, I go to Amazon.com and I understand how the reviews work and how the ratings work and I know that the returns work the way I want and I understand how shipping is going to work, and, and it just felt consistent. And I know it doesn't have everything, but it has enough and I value consistency over comprehensiveness."

And so the night before this, this YouTube meeting, I decided to reframe the question and say "Let's not have the discussion about linking out at all. We're going to start by having a theoretical discussion about in a decade from now, is the video market more likely, the online video market more likely to be about consistency or about comprehensiveness?" And that i- is a question that you can have a very reasonable debate over. Like, what are the, what are the reasons why a market evolves towards consistency over comprehensiveness? So...

And we basically have this discussion and we all came to the conclusion this market is going to value consistency over comprehensiveness. And by the way, I think, I mean, now, almost 15 years later, I think we were right. I mean, I think the, if you go look at the, the, I mean, video market obviously exploded with so many great video properties out there. None of them are comprehensive. There is no one-stop shop for, uh, uh, or a place where all the video exists. And so I think we were right.

But what, what happened was by answering that question, the link out question all of a sudden became super easy. Like, we value consistency over comprehensiveness. We definitely don't need to link out. In fact, we should make a whole bunch of other decisions as well. So we went and, you know, at the time we used to, this was the, the days of flash players and so on. We embedded other people's players on YouTube. We stopped doing that as well.

Probably the most famous decision we made was with the iPhone. As I mentioned earlier, the i- the iPhone when it first shipped had no app store, and so they built all the first, first few apps, including YouTube. Um, and so here we were a few years later, iPhone most popular phone on the planet and the YouTube app on the iPhone was built by a team at Apple, and they had not been able to keep up with what we were doing. Almost half the catalog didn't play back on, on the iPhone. They were missing a bunch of features and so on.

So I drove down to Cupertino and I sat down with, with, uh, Scott and Phil and said, "Hey, we're going to have to take back the, the YouTube app." And they, they said, "I don't understand. Like, why would you do that? Like, you're, you have default distribution on..." you know, as far as they're concerned, the most important operating system in the world. "Why would you do that? You're going to have to, like, rebuild all this from scratch and, you know, uh, it just seems like a really bad choice." And I said, "No, no, no, it's, like, actually quite an easy choice. We value consistency over comprehensiveness. We would much rather be on fewer phones with a more consistent experience than be on all of them with an inconsistent experience. So, look, this is the choice we're making." And it worked out fairly well.

So this decision, the, the Modern Family question ended up becoming named as the example for a term called Eigenquestion. So Eigenquestions, it's not a German game show [laughs] uh, and it's not, it is a made-up word and it's named after a math concept called Eigenvectors, and, um, you know, the math is not really necessary, but for people who are curious, you know, go back to linear algebra. Eigenvectors are in a multidimensional space, they're the most discriminating vectors of the, of the, uh, vectors in that space, the dimensions of that, of that space. It's a concept that gets used a lot in machine learning and, and, and so on. But actually the math doesn't really matter.

The, the Eigenquestion, the simplest definition of an Eigenquestion, it's the question that when answered also answers the most subsequent questions. And it's a very simple idea that when you sit down and you say, "Hey, here's all these questions we got to answer," how do we all usually rank them? Like, sometimes we rank them just by, you know, what order we came up with them. Sometimes we rank them by importance, like which is the most impactful decision we're going to make? But this methodology says don't rank them that way. Rank them, like you said [inaudible 00:53:52] about leverage. Same, same idea. Rank they by which ones would eliminate the most other questions on the list.

So if you take that list, you said, "Should we link out to Modern Family? Should we own the, the, the YouTube iPhone app? You know, should we do..." Like, those were all, like, really hard questions to answer. But it turns out if you answer just one question, you know, do we value consistency over comprehensiveness, you answer all the others. It's like really they all, all of a sudden become very simple.

And so this, this idea of Eigenquestions became part of our vocabulary, became a clear ritual for, uh, for YouTube that saying, "What is the Eigenquestion here?" And [inaudible 00:54:22] as well, and it sort of spread to other places. But that, that's the basic idea.

[00:54:26] Lenny: Amazing. What a, what a baller move with Apple and, uh, and [laughs]

[00:54:29] Speaker 1: It's pretty [inaudible 00:54:30] move. Yeah. [laughs]

[00:54:31] Shishir Mehrotra: Yeah.

[00:54:32] Lenny: But I was just going to say, I, I think YouTube's probably in the top five, 10 most downloaded apps, so it worked out.

[00:54:37] Shishir Mehrotra: You know, the, the, we'd go up for that meeting and I'd bring along our, uh, the, the, the product manager for the iPhone app, guy named Andrey Doronichev, and he's the one, like, having to explain what we're going to do and so on. And it's like a hard, contentious meeting. And as we're leaving the meeting, he says, "Hey, uh, can I get a selfie with Bill and Scott?" And [laughs] I was like, "Andrey, what are you doing? Like..." [laughs]

And, and I took this great picture of them, like, with us asking for, you know, take this back and he's, clearly he's very starstruck. I mean, it's, it's, it's, like, there's a lot of people and he was, like, "Apple would do a better job of building the YouTube app than us. Like, what, who are we to tell them to not do that?" Um, of course, you know, in retrospect, that was, that was the only way to think about it.

[00:55:17] Lenny: Wow. I would do the same thing. [laughs]

[00:55:19] Speaker 1: Yeah. [laughs]

[00:55:20] Lenny: And it's amazing. Uh, who's this person, because that's awesome, because I love that as a leader you bring the, the PM of the team working on it versus just, you know, the big, big shots at the top.

[00:55:29] Shishir Mehrotra: Yeah, Andrey, he's, I mean, he's now a founder. It's, uh, he started a company called Optic, which is a, basically building content ID for NFTs, which is a, you know, v- much needed thing in the Web3 world. But yeah, I mean, the, the team building it, I mean, it was, uh, that meeting, I brought him. I brought my partnerships lead and I brought the eng lead that was covering the area too.

And s- yeah, I think that some of it is... You know, if I had to be honest, some of it is, like, they really wanted to come and be with Apple. Some of it was, like, you know, for my own sake, like, I kind of wanted some backup. [laughs] That's the, I'm about to make this kind of bold-ass...

And it could have been, uh, I mean, to Apple's credit, I mean, they could have been, you know, pretty bad about it. I mean, they could have, like, not allowed us on the store and, and, and so on. And they, they said, "Okay, we'll, you know, we don't like it but we understand your choice. You have to know that you're going to start from zero. We're not giving you a single download, uh, uh, for free. You're going to have to start from zero and we will brick the current app on your agreed-upon date." And the negotiation was, "Can you please just tell those people that there's a new app?" And so that's what we negotiated out of it. And they, they eventually did that and that was, uh, uh, that was, that was fine.

And in the end, YouTube is now, you know, one of the top downloaded apps on, on iPhones. I think, I mean, y- it was, like, six months after launch we had, like, 80% share. Like, almost, like, it, everybody downloaded the app and said, and, and said y- it kind of ended up not being that much of a comprehensiveness, uh, [laughs] uh, uh, choice, but it was a, yeah, clearly hard decision made much easier by asking the right Eigenquestion first.

[00:56:55] Lenny: Wow. Speaking of Eigenquestions, are there other examples of Eigenquestions that come to mind to make this even more concrete in people's minds? I don't know if that's the right way to frame it, or is it more just when you have a list of questions, look for the one that'll answer the most? How do you, how do you kind of operationalize this concept?

[00:57:12] Shishir Mehrotra: You know, I, there's lots of them. I mean, for Coda, the Eigenquestion, um, the sort of most conceptual Eigenquestion for Coda was, you know, we use a line a lot for Coda that Coda allows anyone to make a doc as powerful as an app. You could reverse that statement and say allow anyone to make an app as easily as a doc. And those two sound similar, but they're not. They're, they're, like, actually quite different statements. And so our, our most commonly debated Eigenquestion is, you know, are we more committed to being a doc or being an app? And, you know, which way do we want people... If people are going to misunderstand Coda, would we rather them perceive it as a document or perceive it as an app?

And we decided on doc, which is a, actually, and the way I, the way I cemented that decision when we made it was I named the company that way. Coda is "A doc" backwards. I said, "Well, well, it's all, it's, uh, [laughs] it's a, this is, this is going to be a, we're definitely not revisiting this one. This is a, uh, Coda is a doc first." That's a good example.

I mean, another one, I, I, by the way, I would say Eigenquestions is a, is a term that, um, you know, it got good resonance itself, but it's a hard technique, or it's not, it's not always easy to know how to do it. And one of the things, you know, I get asked a lot is, like, is it a skill you can learn? I absolutely think it's a skill you can learn. It's a thing that, once you observe it, get better at it, you can learn it. But it's not, it's not easy to learn it.

And one of my observations about learning skills like these is you want to learn them in non-pressure-filled environment. Like, to use a, an analogy, you know, if you were trying to learn a sport or learn an instrument, so on, imagine if you never did practice. Like, every time you played basketball was in a real basketball game and every time you played the piano was in a recital. You know, you probably would never get better. And I think one of the troubles with the, a concept like Eigenquestions is we tend to only practice it in real world scenarios that are high stakes.

Uh, and so one of the, uh, one of the things I encourage people to do is to practice Eigenquestions in completely almost frivolous situations. So I, I had an interview, interview question I asked, which I think, and maybe we'll get to this a little bit la- uh, uh, later as well, but it's a very, it's a very simple question, but it's a, and, and it's a coded Eigenquestion test. And the question is, um, a group of scientists have invented a teleportation device. They've hired you, Lenny, to be their sort of business counterpart, bring this to market, product counterpart, whate- this question actually worked well for any role. Let's say you could be a product manager for this thing, bring it to market. Um, uh, what do you do? That's, that's the whole question, right?

And so, so y- people, usually people will start asking a bunch of questions and say, "Well, tell me more about this device. Like, what does it do? How does it work? And, you know, is it, uh, is it big? Is it small? Is it fast? Is it, like, does it dis- disintegrate things or not? Does it need a receiver and a sender? Does it, you know, is it safe?"

Like, all these different questions would come out, and at some point I'll just let those questions come out, and at some point I'll say, "Okay, nice, uh, nice job generating all the questions. Turns out the scientists, they kind of hate talking to people and they're kind of annoyed by all your questions, and so they've decided that they will answer only two of your questions. And after that, they expect a plan. What two questions do you ask?"

And interestingly, all of a sudden, like, the sharp, you know, product managers, engineers, so, and g- and basically every role, they very quickly find what are the two, one or two Eigenquestions on this topic. And there's no right answer, but I'll tell you, like, one of my favorite ones is, as their product manager, said, "Okay, if I had to ask, ask two questions, the two questions I'd ask, one is, is it safe enough for humans or not?" And I would say, like, a very, like, crisp way to get to, like, the safety. How reliable is it? It didn't ask how reliable it is, how many bits [inaudible 01:00:47]. It's like, like, "Just tell me, is it safe enough for humans or not?"

And the second one is, is it more expensive capex or opex? Is it more expensive to buy them or to run them? And then he took those two questions and he said, like, "Just with those two questions, I can form these quadrants." And you can say, "Oh, it's safe enough for humans and it's cheaper to, they're very cheap to buy, but expensive to run." Then you'd probably run them like human fax machines. If you put them everywhere you can and you say, "Hey, look, it's expensive to use, but, like, you all have the ability to teleport anywhere you want and this is, this is how we're going to run it."

On the other hand, they're very expensive to buy but very, um, but cheap to run, you probably have to place them very strategically, in which case what you'd probably do is replace airports. Because, like, airports are pretty strategically placed in places where people are trying to, trying to get, get around places.

If it's not safe enough for humans, then you've got a whole different class of use cases where you go value what goods are transported in very costly ways and, you know, people come up with, like, you know, do you do the most expensive things or do you do the, like, you know, the, is, is, teleporting, you know, people's uh, replacement hearts. Is that, like, a, like, a really demanding thing?

So, so but these two questions kind of, kind of get to the, the, the heart of it. The question's totally made up. Like, no, no teleportation device exists, at least not yet. And I find that people's ability to learn the method is significantly higher if it's low stakes. That question, by the way, if you ask a kid that question, the, you know, hey, new teleportation device, you know, you get, you get to ask two questions, a- almost every kid will, like, quickly get to two pretty good Eigenquestions. Again, kids are incredibly good at simplifying these things down.

It's actually a skill we, like, remove from ourselves. Like, I'll say, I'll hear candidates tell me things like, "Well, I guess I would ask them what, what size it is." And like, why would you ask them what size it is? What, what, what decision is that going to allow you to make, to, to know what size it is? And, and you know, sometimes they can explain it, but sometimes not and don't get hired.

But they [laughs] but then, actually, the, the thing I'd say about it is there are Eigenquestions kind of everywhere. I mean, it's a, you could, you could take any product out there... I'll, I'll do it with my kids a lot and they'll say, you know, uh, was just riding with, with my younger daughter and she said, you know, "How come there's three gas stations, like, in the same corner? Why, why, why do, why do people do that?" That's a, you know, uh, that's a really, that's a really insightful observation. What's the Eigenquestion? How do you place a gas station?

And there's, like, a, you know, there's, uh, and you can take, you can almost take anything and say, "What is the question that really drives, drives this answer?"

[01:03:07] Lenny: I love that. Do you, do you actually still ask this question? Because you're sharing it and all the answers.

[01:03:13] Shishir Mehrotra: No, I don't. And I have a new one that I can't share. But it's a, and we've written about it. In fact, one of the big debates about publishing the Eigenquestions thing is in, in order to bring this to life, I needed to answer your question, like how do I, how do I test this? How do I practice this? And, like, it's much easier, like, nobody can repeat the YouTube one because, like, they, yeah, nobody has that choice sitting in front of them so it's kind of a useless, it's, it's entertaining, but it's, as a teaching tool it's kind of useless. You can't, you can't really go reinvent history and decide [inaudible 01:03:40]

[01:03:39] Lenny: Yeah. You have to sacrifice one.

[01:03:42] Shishir Mehrotra: We sacrifice one, yeah.

[01:03:44] Lenny: So pull on that thread further and kind of dive a little deeper into, into evaluating talent and product talent. I hear this is one of your superpowers, and so I'd love to learn from you and what you've seen around how to evaluate talent. So you talked a little about interview questions you ask. Um, so maybe we could either go in that direction or just, like, what do you look for in people that you're hiring or interviewing that maybe other people aren't?

[01:04:05] Shishir Mehrotra: I have a technique for it. Let me show, I'll, I'll show a quick picture and...

[01:04:08] Lenny: Little YouTube plug.

[01:04:09] Shishir Mehrotra: Yeah [laughs] YouTube plug. You can, uh, I mentioned it before the call, you can, you can put video on Spotify now too. But they, they [inaudible 01:04:16]

[01:04:17] Lenny: Spotify plug.

[01:04:17] Shishir Mehrotra: Yeah. [laughs]

[01:04:17] Lenny: All your platforms that you've worked on [inaudible 01:04:21] videos.

[01:04:21] Shishir Mehrotra: That's right. Now, so, I'll talk through this diagram that has two axis, scope, this acronym, PSHE, and this line. I'll stop sharing and describe it and we can come back to it. But I'll tell a little bit of this technique sort of changed how I think about evaluating, not only product talent, but it actually turns out you can use the same set of rules for evaluating basically every role. But I'll tell it from how you asked it about product talent.

So in 2011, Larry Page took over at Google, and he made a bunch of changes to the company, mostly quite positive, and one of the ones he did was he, he moved us from being a functional organization to being a business unit organization. We call them product units, but roughly the same thing. And there were eight product units, uh, at Google, you know, YouTube and Search and Ads and Chrome and Maps and so on. And it was very positive, uh, it's, like, hard to believe that we were already, like, we were, like, 20,000 people and we're still functional. Like, all of Engineering, all of Products reported in to the CEO. Just seems, like, totally crazy with the breadth of products that we cover.

One of the downsides of it was, like in any functional to business unit switch, is you lose some of that, that what does the function mean. And the particular thing's, like, what is a good product manager was a question we were at risk of losing. So a group of, at the time I was running Product for, for YouTube. The group of the, the eight product leaders around the company got together and said, "Hey, we need to, like, keep some level of, um, consistency amongst how we think about what's a great product manager or it's all going to diverge and it's not going to mean anything anymore."

And, um, actually, as a fun aside, we did a ritual that I've repeated a few times but isn't done often enough, is, uh, we said, "So who's going to, like, drive this process?" And we did an election. I, I don't know why we do elections in, like, the public world but not in the private world, but it actually, like, quite effective. We used to do them in YouTube where we would, like, we would elect into certain roles and, like, you got a one-year term. You gave a little speech. You, like...

[01:06:07] Lenny: [laughs]

[01:06:07] Shishir Mehrotra: But yeah, yeah, so we, we, we, we, uh, uh, we did, we did an election. Anyway, so I, I got chosen to be, uh, the, uh, the first sort of leader of this, of this, this, this challenge, kee- keeping the, keeping the product manager function together at, at Google. And the, the most obvious job we had to do was come up with a speech for the calibration committee.

So Google, Google does, um, calibration a little bit different, or promotion a little bit different than most companies. Most companies your boss decides if you get promoted or not. At Google, there's a committee that decides, and they're supposed to be a committee that doesn't actually, you know, work directly with the person so they can be a little unbiased and, and so on. And it gets done, the ritual was to do it in a hotel near the airport here in San Francisco and everybody get in these different rooms and there was always a, a speech given at the beginning that used to be given by Jonathan Rosenberg, who ran product for Google for many years, and now had to be given by somebody. So, like, okay, I'm going to give this speech.

Now I got to figure out what, what the hell am I going to say? Like, what's a good, what's a good product manager? And so as I was going through this, I decided to run this little exercise. So this, this group of eight product leaders, we took the level guide, so we had a level guide for product managers. Every company does. And we took it, we printed it out on a sheet of paper, cut it into little slices, one per level, and we cut off the title and the number. And I handed them out and I said, "Can you reverse identify what level you get?" And turns out nobody could do it.

Then, you know, and, and it's, it's not easy to do. Like, they, these lev- the level guide had been, like, you know, kind of added to over time and nev- not really that refined. And so it's full of all sorts of things that were, you know, whatever the, at that time was the priority of the team, they stuck it in the level guide. And so it, it would say things like, "This person can manage a medium-sized project and they interview at least three people a week and they always send the notes out on time and, uh, their expense reports are always fine."

[01:07:55] Lenny: [laughs]

[01:07:55] Shishir Mehrotra: And it'd be like, "Oh, that's a director." And it's like [laughs] you know, and, and you know, uh, just whatever we were trying to incent was sort of stuck in this thing. So, but we noticed that if you took all these sheets of paper and laid them out side by side, you could order them by exactly one statement, which was, was one that corresponded to scope. So there's some word in there that was an escalating adjective that mostly correlated to how big is the thing you run.

And so we decided, "Okay, well, we're going to standardize. We're just going to focus on making that clear." And so we said, "We're going to define scope and so that we all kind of use it the same way," and we came up with some, some stopping points and ba- basically we said, "You own a feature. You own a group of features. You own a subarea of a product. You own multiple subareas of product. You own an entire product or you own multiple products. You own a product line. All right, and that's going to be how we think about scope. Go forth and, and evaluate your teams."

We had the next meeting a couple weeks later, and everybody comes back upset. It's like, "This didn't work at all." Like, "Why? What happened?" And said, "Well, the Search team is super mad at the Ads team because Search is, like, one product. Like, the entire thing is one product." The Ads team, they went and invented all these products, because it's like every little thing-

The ads team, they went and invented all these products. 'Cause there's, like... Every little thing you do in the ads platform has a skew, has a P&L, has a... Because, like, it... You, you can, you, you have lots of products.

So that doesn't work. The second issue was, they said the scope is actually an input, not an output. Like i- we w- we were, we were talking to my manager and said, "Well, this person should get promoted. They manage this huge scope." And then they would say, "But you gave them that scope. Like, that's c-... We should be judging you. Not the, not the person. How do we judge what they're doing with the scope?" Like that's actually it-, it's actually very different.

And the third issue was in almost every team, some of our best people were working on things with odd scope. That we didn't... They were risky projects. We didn't yet know, is this thing gonna work or not work? This might canceled. And w-, uh, w- if we put in place a system that only rewarded scope, we would heavily disincentive people from working on these riskier, more creative things.

So, um, we were kind of stuck. And then we ended up seven of them. We went through lots of frameworks. We ended up settling on this one called PSHE. And it comes from a old mentor of mine, a guy named Bud Clark. He was my boss at Microsoft for a number of years. And it stands for problem, solution, how, execution. PSHE. And I will say it's a, um... I've tried many times to come up with a better acronym, and I have not been able to come up with one. So it's PSHE. That's the word [laughs]. That's as good as I can get-

[01:10:20] Lenny: All right.

[01:10:20] Shishir Mehrotra: ... but here's how... It, it, it works good enough. Right? So PSHE. That's all you have to remember. The, um, uh... Doesn't roll off the tongue, but it's... M- maybe... I did better with [inaudible 01:10:28] questions, I think.

But the, [laughs] the, th- um, um... So here's how it works. So if you're sort of a junior product manager, what happens? You get handed a problem, you get handed a solution, you get handed the how. You know. Go talk to this person, write this document, run this meeting. So on. And all you have to do is execute. Run that playbook, and that's all, that's all we expect at it.

You can become a little more senior. We hand you a problem. We hand you a rough solution. You figure out the how. You figure out the, "How are we gonna organize this? What are the milestones? What, you know... How're we gonna get it to market? You know, how are we gonna do the meetings? What are the rituals?" Like, all those things show up, uh, in the H.

Some point, you become a little more senior. We hand you a problem and you come back with the solutions. And you're there, and you come up and we judge you on the creativity and the effectiveness of the solutions.

And at some point, you're senior enough that you tell us the problems, and you say, "Hey, I know you told me to go work on activation, but actually I think our issue is brand." Or, "I think our issue is quality." Or, "I think our issue is..." Whatever it might be. And that's sort of the, the, the, the pinnacle of, of, of this way of thinking about it.

Now, just back to this picture for a moment, one of the interesting things that happened was that the teams went and they evaluated their teams on these two axes. And there's... They ended up with this sort of curved line between them. It's not linear as you, as you work your way through. And what happens is early in people's career, they mostly sit at that E point. You get handed a problem and handed a solution, handed a how, and you just execute. And they gradually grow in scope.

Later in people's careers, similarly, you know, you're at that key level. You just do bigger and bigger products. And that's like, you know, the, the job of being an entrepreneur or CEO or an owner or so on. Is, just kind of do bigger and bigger projects.

But in the middle, the, the slope changes. And all of a sudden, it's not really about scope. It's about PSHE. And there's a circle drawn here for what I like to call the trough of dissolution. And the reason... I'll I'll start sharing so we can talk about it. But the, the, the... What happens in that phase...

And I was talking to the calibration companies about this. The reason we call it the [inaudible 01:12:23] solution [inaudible 01:12:24] is part of the... For the employee, for the person, this is a confusing time. Like, everything about leading up to this moment, from high school and college, has been about scope. And like, at this point, you're all of a sudden told, "We're not judging you on scope anymore. We're judging you on this PSHE thing," that's very confusing.

To the calibrator, or to the manager, it's also very confusing, because all of a sudden the difference... The way I would put it is, the difference between a level three and a level seven may not be scope. They may do the exact same job. It's how they do the job that matters. And here's some language for how they do the job.

And so PSHE became a very sticky way of thinking about it. It turns out that this way of evaluating people is actually not that specific to product management. It's really easy to see h- why you do the exact same thing for engineers and designers and, and, and so on. But to pick one that may not be as obvious, I'll pick salespeop-... Like, a, a very common thing people do with salespeople is they evaluate them based on quota attainment.

It's like... The easiest thing to do is take the salespeople and rank 'em by who hit their quota and who didn't. You go ask the sales team who's the best salesperson, and what you'll realize is they'll say, "Quota attainment is just a signal for how good you negotiate your quota and pick the right territor-..." And they say the, "Really, you wanna know who's the best salesperson?" They say, "Well, you know, so, and so. I mean, she can sell anything. And she can be in the region that's growing, or the region that's shrinking, or the new product, or the old product. Or..."

And if you think about that terminology, it's very similar to PSHE thinking. Like, this is the person who can come into a new space, identify the right problems and solve them. That's what makes a really great salesperson. So it's c- become my framework for evaluating talent in internal, all sorts of ways.

And you might recognize the pattern of, like, being a great P thinker is very correlated with being good with [inaudible 01:14:05] questions. Like, "Can you spot the right problems?" is very similar to, "Can you spot the right questions? Can you s-, can you decide what's important?" And so that's been my, my main framework for value.

[01:14:15] Lenny: Wow. There's so much there. A couple quick questions. Is this basically your calibration ladder framework for PMs at this point? And then this is also just, like, your interview guide? Just, like, interview at each of these pieces to level the person?

[01:14:29] Shishir Mehrotra: Yeah. So it's, uh, definitely how we do our version of leveling is, you know, PSHE and the, the, the, um... We use a, uh, a set of Radford levels.

Rad- Radford has this, like, really interesting way of describing that as you grow in a profession... He uses the analogy of, uh, of someone, um... Of, like, a sailor, and that a, a junior sailor is, like, learning to tie knots. And then you gradually can tie all the standard knots, and then you can tie the advanced knots and so on. And somehow you kind of work your way up. And at some point, the way you're judged is you invented nylon.

And it's like this, like... L- it's like list between there, and it's, like, a way to evaluate every role. It's very similar to PSHE. And so the... We sort of look at a, a kind of similar list. But yeah, basically all of our... All of our roles are evaluated on something that corresponds with PSHE. And in terms of interviews, yeah. You look for the same thing. And so you, uh... And in- uh...

And w- by the way, I should say. Interviewing is one part of this, and you talk about interviewing. You talk about calibration. There's one other really important one, which is reference checking. And I think the best way to assess PSHE is actually through references. And so the, the most important guide we write isn't the interview guide. It's that when we call this person's references, what do you ask to actually get at these questions? 'Cause I...

You know, people can often confuse them. Like, I ju- just to pick product managers as an example. We all know of some really amazing H-level product managers. And one of the reasons, one of the hallmarks, of an H-level product manager is that their counterparts usually love them. And they'll say things like, "Oh my gosh. The person runs such efficient meetings, and, like, everybody... All the communication's always clear. [inaudible 01:16:05]'s always, always buttoned up. You know, execs know what we're doing. The market knows what we're doing. Sales team know what we're doing. Like, the, the... That's, like, this great H."

And then you'll ask 'em a question. "Oh, okay. So when you're deciding which problem to solve, who is the leader of that, and are they picking the right problems?" Or, "When there's a hard problem, who is regularly coming up with the best solutions for them? Who do you turn to? Who is the most obvious person to turn to, to say, like, this is a really hard problem. What's the right solution?"

An amazing number of people will tell you, "No, they run a great meeting, but, like... Actually solving the problem, the designer does that." Or, like, the... Or the, "What are the problems?" "No, the CEO kind of tells us what to do." That's and [inaudible 01:16:40] not what this person's really good at. Yeah. "I'm not sure we're solving the right problem, but boy, we run a great meeting."

And [laughs] I'd say... And it's not meant to diminish any of that. I mean, we spent a while talking about rituals, which mostly happen at that H-level. So I don't think it's unimportant. But it's, it's actually quite hard to assess in interview, but incredibly easy to assess in a reference check. And I think getting good at that is really important.

[01:17:01] Lenny: You may have mentioned this, but what are... What's kind of the question there? Is there a question that you could recommend?

[01:17:05] Shishir Mehrotra: The absolute ideal case is, you get to the person that you're doing the reference check with, and you don't even tell them who you're asking about. Uh, and you just say, "When you think of your teams, who is best at, at..." And th- this is, uh, all of this, it only works in cases where you have a pretty deep relationship with the person you're getting a reference from, and, and so on.

But [inaudible 01:17:25]. You, you can't always, uh... You can't always do that. But ideally you wanna kind of mimic that behavior.

You know, br-... One of the things I think, uh, that's hard about reference checks is, people have... Uh, you know, uh, they have sort of perverse incentives in a reference check. I mean, they're... They're not really... Nobody, uh... Some of it is, is, uh, for good reason. Some of it is for bad reason. You know, people, you know, generally don't like, you know, criticizing people. And they also, like, feel judged themselves. And they're, you know, they're, they're, they don't want... There's legal reasons that things can blow back, and, and so on.

And so what I try really hard to do is to draw contrasts. So you, you try to say things... So, like, there's a couple techniques I've used for this. Is, uh... In the best case you say, "I know the per-, I do-, I'm not gonna even tell you who I'm asking about, but when you think about this team, who regularly identifies what problems they should focus on? Who is most reliable at coming up with the solutions to the hardest problems?" And you kind of work your way through it.

The other way I like to do it is to provide contrasts to give the person an out, to not make it obvious to them that I have this ranking. And so I'll say things like, "When you think about this person..." And, and I, I'll give you, I'll give you four different personas. You know? Someone who's, like, regularly coming up with the problems that the team should be focused on. Someone who, like, you know, given a set of problems is constantly, like, solving them in this, like, really creative way. The person that is just r- really good at getting a team moving and, like, and... Or the person who can take a playbook and execute it with high precision and high quality and such.

And I won't tell them that I have a qualitative judgment, that one is better than the other, but you want them... Because you just want your reference check to talk, and you want them to s- say what their, what's on their mind of, you know... [inaudible 01:19:00] You wanna give them an opening to not feel like they're judging.

I mean, obviously, like, another question I always ask people is, um, "would you hire this person again?" Or "How excitedly would you hire this person again?" I always ask them, "What question should I have asked that I didn't?" Another, another key technique for reference checks is, you know, you just need people to... Once they start talking, they'll reveal what they really feel. And often, like, the little things will come out.

But for this particular thing, if I wanna know where they stand on this axis, don't tell people what you value. And by the way, I will say, I value P over S over H over E. I've seen many companies that would've reversed that scale. And there are... By the way, there's industries where it's very required. Like, it... You don't want a bunch of people running around and constantly telling you to solve some different problem. Like, I just need people that can do this job that we give them super, super well. And it, you know... So depends a bit, like, what your personality is and what your company's culture is and so on.

So it's not actually that unbelievable that I might value the E over the P. And, you know, anybody who knows me, that's probably not true, but you can... For a reference check, you can probably not give that away.

[01:20:04] Lenny: Wow. That was, uh, gold. I hadn't heard these reference check insights, and so I'm really happy-

[01:20:10] Shishir Mehrotra: [inaudible 01:20:10].

[01:20:10] Lenny: ... we got to that. Maybe one last question on reference checks. How often do you find that a reference check leads to you not hiring someone? Just, like, ballpark? Or, or is that hard to say?

[01:20:18] Shishir Mehrotra: All, all... All the time. And I, and I will say, um, I try to do them as early in the process as, uh, practical. If it's, if it's poss-... I mean, 'cause I think it's actually the worst feeling for an interviewee to go through a process and then get dinged at the reference check stage. Just, like, it's a really crappy experience for the candidate.

And obviously you have to be careful about that. Ruining a- anything for them. It's... You can't always do the reference check as, as early as you can. When you're inside a company, when you're inside Google or Facebook or so on, you can generally do 'em even before you inter-... It's sort of an expectation, of like, "We're gonna hear a little bit about people, uh, in that process."

And I generally value the reference check over interview signals. You know, if I had the stack rank in interviews, what is the best signal, the, uh, reference check is the top of the list. Th- th- th- those people, you know, they've worked with this person for, you know, sometimes for years. Like, their knowledge... What you're gonna get out of 30 minutes of artificial scenarios is just, like, never gonna compare what, what a good reference check will give you.

And then the second best thing I value is anything that feels like a real work exercise. And that, and, and... Which is also, like... And that, even that is hard, because some people, like... Their skillset doesn't naturally lead to a compressed time work exercise. But, you know, we do a thing for basically all our roles where at the start of the interview loop, the candidate presents to everyone on the loop, and we invite some other people, uh, in the company to attend too. For a while, it was open to the whole company. Now we're big enough where that's not practical. But the, the original was very simple. At the beginning of the loop, the person presents.

Um, and generally for most roles, there's a, there's a exercise that they do. But about half the time is spent on them presenting whatever they want. I mean, they, they can, they can talk about themselves. They can teach us something. They can... And then another half is, like, we've giv- we've given them a prompt. Something that we, we wanna direct.

One of the things that I think, um... When you're doing interviewing, one way to think about it, I call it home court, away court, neutral court. You want interviews to balance in all those different spheres.

So like, you know, um... Common mistake people make is they [inaudible 01:22:11] all questions in home court. "Hey, you know, if you're joining Airbnb, what would you do about X?" And the, you know... What is the candidate gonna say? Uh, they can't, clearly can't be as thoughtful as you. He hasn't thought... Uh, they haven't thought about it nearly as much as, as you have. So really what you're testing is, did they come to the same answer that we did? Which is, like, a pretty crappy, like, way to judge someone.

That's, uh, like, tend... Home court questions tend to be tough. You have to be very careful about how you do that.

Away court questions can also be tough. Like, this person... You say, "How did you solve this problem?" And they say, "Well, we did this thing." You don't really know. Was that, like... Was that problem actually hard? Was it somebody else was telling them what to do? Um, or they're just, like, not telling you the whole story of what happened? Uh, and so on.

So you try really hard to make... Most of our interviews are done neutral court. So the vast majority [inaudible 01:22:54]... my teleporter question is a very neutral question. Like, it has nothing... You don't know anything about Coda. I don't have to know anything about Airbnb. Or wherever, wherever you might... I, I can just ask you this question. But this one thing, the presentation, is the away court question. It's the, "You now have an opportunity to talk about yourself in this, in this, uh, new way."

And it's super interesting what people do. I mean, I've seen people use that time... I of-, like, I often tell people it's the brag session. So this is like... And you know, for product managers or s-... Uh, we do it for everyone, but it's, like, recruiters, sales people, marketers. We tell them, you know. "We're gonna go through this whole interview process, and at most companies, you talk to six different people in six little s- uh, uh, segments. And at the end of the day, you say, "Gosh, I really wish they had learned this about me."

And I s- I tell them, "Don't leave this with that feeling. What do, what do you want us to know about you? Like, all our questions are, are, are in some senses, like, we're trying to lower bound you. Like, how bad could this be? I want you to upper bound us. I want you to tell us what's, what's, what's really amazing here."

And so we'll, we'll have the presenter go through that process. And what they choose to talk about is very important. How they choose to present it is very important. But I do it as upper bounding. But if I had to stack rank in, in interviewing, what do I look for? Reference check at the top. You know, work product and presentation next. And then all the interviews. If th- when they, when they disagree, that's the order we judge in.

[01:24:11] Lenny: Su- sure. I feel like you have five books in you that you need to write on-

[01:24:15] Shishir Mehrotra: [laughs]

[01:24:16] Lenny: ... s- so many of these topics. This is such good stuff. I wish I could keep going. I know you have to run. So we have this final lightning round. So I'm gonna ask you five questions. There's one at the end I didn't tell you about ahead of time.

[01:24:27] Shishir Mehrotra: Okay.

[01:24:27] Lenny: Uh, so it's gonna be a surprise. And so-

[01:24:29] Shishir Mehrotra: [inaudible 01:24:30].

[01:24:29] Lenny: ... I'm just gonna ask these quick questions. Let me know what comes to mind. Okay.

[01:24:32] Shishir Mehrotra: [inaudible 01:24:33].

[01:24:33] Lenny: First question. Just what are, what are two, three books that you find that you recommend most to other people? Oh, pulling out the bookshelf.

[01:24:39] Shishir Mehrotra: [inaudible 01:24:39]. Pull out bookshelf.

[01:24:39] Lenny: Or [inaudible 01:24:41].

[01:24:40] Shishir Mehrotra: One, one, one I already said, and we talked about. So this, this would take two of the slots. The next one on my list is, is Switch by, um, uh, Chip and Dan Heath. How to Change Things When Change is Hard. The other one, this is, uh, probably a surprising one. It's called Understanding Comics by Scott McCloud.

[01:24:53] Lenny: Oh, wow.

[01:24:54] Shishir Mehrotra: Super fun book. It's a comic book about comic books. And you don't-

[01:24:57] Lenny: [laughs]

[01:24:57] Shishir Mehrotra: ... have to like comic books at all to, to love this book. It's basically... The, the, the sort of starting point is, over time, communication has drifted at two extremes. So one is, we've gotten very good at written form. And the other is we've gotten very good at art. Single pieces of art. And comics are the hybrid, right? They are drawing mixed with writing. Sometimes I think he calls it sic-... Um, oh god, he had a good term for that I'm now forgetting.

But th- [inaudible 01:25:23]. He describes comics really well, and he goes through the, the, the actual reasoning of what, why comics are structured the way they are, and so on. And one of the reasons it's so important to me is... And you can probably tell from, like, how we talked about different things, there's, uh, often a diagram that crystallizes something. The art of, of storytelling, diagramming, so on, I think is so critical for basically any part of life. And this book is, it just... It's so thought-provoking h- on how to do it. Understanding Comics by Scott-

[01:25:47] Lenny: Understanding Comics. Wow. Good choice. Okay. Favorite recent movie or TV show.

[01:25:52] Shishir Mehrotra: Okay. Couple that come to mind. Only Murders in the Building is, uh, is a really-

[01:25:55] Lenny: Mm-hmm.

[01:25:55] Shishir Mehrotra: ... fun one.

[01:25:56] Lenny: Yep.

[01:25:56] Shishir Mehrotra: And, uh, uh, my family got really into the, the Marvel series during the pandemic. And so Wandavision, if anybody hasn't seen that. If you're not into superhero stuff and so on, which I'm not really. That k- like, I'm actually more of a DC Comics person. I like Superman. But Wandavision is, like, one of the best pieces of art that I've seen done in a very long time. Ver- very well done show.

[01:26:17] Lenny: I feel like with that show, I had to, like... I gave up initially, I'm like, "What the hell is this? G- what is [inaudible 01:26:22]-"

[01:26:21] Shishir Mehrotra: I know.

[01:26:22] Lenny: And then it gets... It gets good.

[01:26:24] Shishir Mehrotra: It gets good. Yeah.

[01:26:25] Lenny: And on the flip side, Only Murders in the Building. I don't know if you've seen the second season yet, but I'm just, like... I'm done with it. I don't... I'm just tired of it now. I don't know what they're doing.

[01:26:33] Shishir Mehrotra: Oh yeah? We're, we're only one episode into the second season, so we'll see.

[01:26:36] Lenny: I'm just... Okay.

[01:26:37] Shishir Mehrotra: Maybe we'll give up.

[01:26:38] Lenny: [inaudible 01:26:38] Talk. [laughs]

[01:26:38] Shishir Mehrotra: Okay.

[01:26:38] Lenny: The first season was so good.

[01:26:38] Shishir Mehrotra: [inaudible 01:26:41].

[01:26:38] Lenny: So good.

[01:26:38] Shishir Mehrotra: [inaudible 01:26:42].

[01:26:38] Lenny: And it's all about podcasters [laughs]. It's, like, so meta around podcasters. Oh man. Okay. Good, good segue. Okay. Favorite interview question. You may have already answered this.

[01:26:52] Shishir Mehrotra: I... The teleporter one is definitely my favorite. My second favorite one... Well, I was gonna give you a different one. Is, I have a series of questions around, um... Uh, let me, let me pull it here so I, I give you the, what the real question is. Um, but it's basically around a dashboard prompt. And it's, uh... The, the starting point of the question is to, uh, pick a product, and I [inaudible 01:27:14] say your favorite technical product. And the constraint is, it can't be something that you built, you know, worked on or competed with. It's gotta be in a space that, like, you're not an expert in.

And I generally ask people why, which is actually a really interesting passion test. And then I'll ask people, design the one-page dashboard for that product. So you're the CEO, general manager, whatever. You run that product. What's on the dashboard. Why? [inaudible 01:27:37]. It's an interesting [inaudible 01:27:39] questiony type question of, like, can you tell what's important for this product or not?

And then I ask them, uh, to basically redesign the product. And the way I do it is, you've been hired by a competitor to design a me too version of the product. I'm gonna leave aside for a moment that why you would wanna build a me too proje- uh, version. What is the bare minimum of what you need to build? And then I tell them, "You ran out of resources [inaudible 01:28:03] a quarter of the scope, and a quarter of the, uh... Uh, you [inaudible 01:28:06] a quarter of the time and a quarter of the, the, uh, the team. What do you actually bill?

And then I get down to the other side of, like, you've decided you can differentiate in only one place. What do you do to differentiate?

So there are a, a series of questions that are basically a form of PSHE, but just formed around a new problem space that gives, lets people wander a little bit. And I've seen some really amazing answers to that.

[01:28:26] Lenny: There's also s- a lot of biting question elements to this sequence.

[01:28:29] Speaker 2: Yes.

[01:28:30] Shishir Mehrotra: Yes.

[01:28:31] Lenny: Excellent. Okay.

[01:28:32] Shishir Mehrotra: Uh, hmm.

[01:28:33] Lenny: Who else in the industry do you respect as a thought leader? Who comes to mind?

[01:28:37] Shishir Mehrotra: Other than you? Um, I have to, have to [inaudible 01:28:39]. Uh, present company excluded?

[01:28:39] Lenny: Yes. That's [inaudible 01:28:41].

[01:28:41] Shishir Mehrotra: [laughs] Uh, I did, b- by the way, your, your newsletter is one of my top reads. I, I think that-

[01:28:45] Lenny: Oh, wow.

[01:28:45] Shishir Mehrotra: ... yours and, uh, and Ben Thompson from Stratechery-

[01:28:48] Lenny: Oh my God.

[01:28:48] Shishir Mehrotra: ... are t- two of the, two of the ones that... I, I think you both have a really, like, natural instinct for, for writing and synthesizing things that people are feeling without, um... You know, with, with a, with a clarity that's really helpful. So I, I, I, and I really appreciate that. I mean, I think-

[01:29:05] Lenny: Appreciate that.

[01:29:05] Shishir Mehrotra: I, I think the rituals process has exposed me to some really amazing leaders. We talked about Ariana. I always learn a lot when I, when I talk to Ariana. [inaudible 01:29:13] Doshi has contributed a bunch. I think he's, uh, really helpful, and is, like, is... Uh, uh, I can't believe he basically had no Twitter followers at the beginning of the pandemic. And, like, his tweet [inaudible 01:29:24] are just total gold. And this is so-

[01:29:25] Lenny: Mm-hmm.

[01:29:25] Shishir Mehrotra: ... so insightful and well put together. Uh, Fidji Simo is someone that I, I, I learned a lot from. She now runs Instacart. Daniel from Spotify, Daniel Ek, I think is, is, uh... He's got this really unique way of thinking about the world. And he's also one of the few people that can, like, hold a very long-term view and a very short-term view at the same time. Fantastic ethics.

I'd also say my entire board. Reid, uh, Reid Hoffman, Ammon Thonaja, Mamoon Hamid, Quentin Clark, Sarah Guo. Like, I think they're... I, I think they're all amazing and I'm super lucky to ha- have, you know, a group of people I can call with questions.

[01:29:59] Lenny: Awesome. Someone's gonna have a lot of work in these show notes.

[01:30:02] Shishir Mehrotra: [inaudible 01:30:02].

[01:30:02] Lenny: [inaudible 01:30:02] That list. Okay. Final, final question. What's your go-to karaoke song or dance move at a wedding?

[01:30:08] Shishir Mehrotra: Uh, karaoke song is If I Had a Million Dollars by the Barenaked Ladies, and, uh, it's a, uh... If people don't remember the song, part of the reason it's my favorite is I'm, I'm a very mediocre singer, and you don't have to be that kind of singer, and everybody can sing along. So you can kind of bri- [laughs] bring everybody into it. And it's just such a fun song. The, If I Had A Million Dollars.

[01:30:28] Lenny: You're as thoughtful about your karaoke songs as you are about everything else you're, you're doing.

[01:30:33] Shishir Mehrotra: [laughs]

[01:30:33] Lenny: Shishir, thank you so much for being on this podcast. You've set a really high bar for CEO guests, so we'll see [laughs]-

[01:30:37] Shishir Mehrotra: [laughs]

[01:30:37] Lenny: ... who comes up next. Two final questions. Where can folks find you online if they wanna reach out, or learn more, and how can listeners be useful to you?

[01:30:44] Shishir Mehrotra: Okay, I'll give the same answer to both. Well, you can [inaudible 01:30:47]... I'm easy to find. One of the benefits of having a not very common name is that [laughs]-

[01:30:50] Lenny: [inaudible 01:30:51].

[01:30:50] Shishir Mehrotra: ... it's uh, easy to find me on basically every platform. So you can find me on Twitter. It's easy to DM me. Shishir at coda.io will get right to me. But in terms of being useful to me and also finding me, I would highly recommend joining the rituals of great teams brain trust. And I think it's a pretty fun experience to get, uh, a chance to contribute to a book like that. Uh, and hopefully you've made it th- that this far in the episode that you probably are interested. Uh, and so I think you'll find it interesting, and I'm having a lot of fun with the, with the people in that brain trust.

[01:31:18] Lenny: Amazing. I'm definitely gonna join. [inaudible 01:31:20]

[01:31:20] Shishir Mehrotra: All right. Thank you so much, Lenny. This was, that was really fun.

[01:31:24] Lenny: Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at LennysPodcast.com. See you in the next episode.