Career frameworks, A/B testing mistakes, counterintuitive onboarding tips, selling to developers | Laura Schaffer (VP of Growth at Amplitude)

Laura Schaffer (00:00:00):
... Like the dead of the night. And by that, I mean 7:00 PM or something on. I'm pretty sure it was a Friday. We just asked for forgiveness and put these questions into the silent flow and ran as Navy test with a small group. And I'm fully expecting, "Okay, this is going to hurt our numbers, but maybe it won't be so bad and I'm going to be prepared to advocate the power of this data that we're getting." And I was totally geared up thinking about written, started to write the framework for how I wanted to surface this. And we start to get the data for this thing. I'm not kidding, an improved conversion. There's no personalization, nothing past it, just the questions. An improved conversion by like 5%, just improved signups. And it was one of those like, "What? Okay, what is going on here?"

Lenny (00:00:50):
Welcome to Lenny's Podcast where I interview world-class product leaders and growth experts to learn from their hard one experiences building and growing today's most successful products. Today my guest is Laura Schaffer. The week we recorded this chat turned out to be Laura's first week in a new gig as head of growth for Amplitude taking over for a previous legendary guest, Elena Verna. Prior to Amplitude, Laura was VP of product and growth at a company called Rapid. Before that, she spent over seven years at Twilio as Head of Growth and PM lead of the growth platform and experimentation platform at Twilio. In our conversation, we dig into Laura's career growth framework and the importance of carving your own path versus waiting for one to be carved for you. We also get into a bunch of tactical and surprising advice around running experiments, making decisions on gut versus data, developing your growth strategy and how to sell your product to developers. Laura has a wealth of wisdom and I learned a lot from our conversation. With that, I bring you Laura Schaffer after a short word from our wonderful sponsors.

(00:01:55):
This episode is brought to you by public.com who want to tell you about their new treasury accounts, which earn a 4.8% yield on your cash. That is higher than a high yield savings account, while still being backed by the full faith and credit of the US government. Treasure yields are at a 15-year-high. But buying US treasuries is super complicated. If you go to a bank or navigate an ancient government website, or at least that was the case. Now you can move your cash and the US treasuries with the flexibility of a bank account. You can access your cash whenever you want, even before your treasury bills hit maturity. There are no hold periods, no settlement days. Just a safe place to park your cash and earn a reliable yield. Public will automatically reinvest your treasury bills at maturity so you don't have to do anything to continue growing your yield. And you can manage your treasuries alongside stocks, ETFs, crypto, and any alternative assets. Do all your investing in one place and earn 4.8% a higher yield than a high yield savings account only with a treasury count at public.com/lenny.

(00:03:02):
This episode is brought to you by Eppo. Eppo is a next generation AB testing platform built by Airbnb alums for modern growth teams. Companies like Netlify, Contentful, and Cameo rely on Eppo to power their experiments. Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern grow team stack. This leads to waste of time building internal tools. We're trying to run your experiments through a clunky marketing tool. When I was at Airbnb, one of the things that I loved about our experimentation platform was being able to easily slice results by device, by country, and by user stage. Eppo does all that and more delivering results quickly, avoiding annoying prolonged analytics cycles and helping you easily get to the root cause of any issue You discover. Eppo lets you go beyond basic click through metrics and instead you turn north star metrics like activation, retention, subscriptions and payments. An Eppo supports test on the front end, the back end e-mail marketing and even machine learning clients. Check out Eppo at geteppo.com, get E-P-P-O.com and 10 x your experiment velocity.

(00:04:15):
Laura, welcome to the podcast.

Laura Schaffer (00:04:18):
Thanks, Lenny, it is so great to be here. Thanks for having me.

Lenny (00:04:21):
It's great to have you. So I asked Elena Elena, Elena, I'm not even sure how to pronounce her name, maybe. What is it?

Laura Schaffer (00:04:21):
Elena. You got it.

Lenny (00:04:27):
Elena. Okay. Okay. I think I've said it wrong all the time. All this time. Okay, Elena. So I asked Elena Ferna, who's a popular guest on this podcast who I should have on this podcast and you are the first person that immediately came to mind. And so I'm really excited that we're doing this and that you agreed to be on.

Laura Schaffer (00:04:42):
Well, she's the best and I'm really happy that she referred me because I'm just stoked to be here. So thanks for listening to her guidance.

Lenny (00:04:50):
Absolutely. And it's a cool time to be chatting. You're the newly minted head of growth at Amplitude and so congrats, first of all.

Laura Schaffer (00:04:57):
Thank you. Appreciate that. Yeah, this is my day two and a half here. So very [inaudible 00:05:04].

Lenny (00:05:06):
Wow, you're a veteran.

Laura Schaffer (00:05:06):
Yeah, right.

Lenny (00:05:06):
I love it. Some companies, there's a little percentage of that shows you how many people have joined before you and I wonder what that percentage already is [inaudible 00:05:12].

Laura Schaffer (00:05:12):
We had that in Twilio and I got pretty, pretty high up there after a while. We had a stack rank and a spreadsheet. Yeah, but it is funny. So wherever that thing exists in Amplitude, I am right fresh there at the very bottom.

Lenny (00:05:25):
So what was the number you got to Twilio? Any, do you remember?

Laura Schaffer (00:05:28):
Yeah, no, I was very proud to crack the top 50. That was my claim the same because as people left, you move up. Right?

Lenny (00:05:36):
Right. Yeah, it's bittersweet.

Laura Schaffer (00:05:39):
Well, yeah, right. On one hand it's like, "Ooh, very cool." And one of the OGs on the other hand, it's like, "Oh my gosh, this person's [inaudible 00:05:47]. Bummer." It's a shift, but I'm excited about it for sure.

Lenny (00:05:51):
So you have this new exciting role and I thought it'd be fun to start to chat about career growth and just how you think about career growth. I know you have a framework of how you think about your own career growth and clearly it's worked out, so I'm curious to hear about it and see how it could be helpful to folks that are listening. So yeah, can you just tell us about how you think about career growth?

Laura Schaffer (00:06:11):
Career growth is definitely not a straight lineup, but there's definitely some frameworks and methods that have worked really well for me. And I think to dive into it, it's first good to just talk about the one that I most typically see people use to try to grow their career and why that can be a little problematic, which is that I see most people try to work really hard the job that they have within the role that they have at a company. Do whatever you can to grow there, show your manager all these things. I see people keep spreadsheets, it wins. So it can come up with performance reviews. Maybe you try to get better advocating for yourself, maybe try to get peers to notice or your manager's peers. And that's all good. It's all stuff.

(00:06:52):
But the problem with it is that you're limited to what your manager's ability is to advocate for you, to promote you. And you're also limited by the explicit trajectory of your role at that company and where the there's room for that or not at the company. And then often that perception can sometimes be a little bit in contrast to what your perception is. And also other things that happen, your manager leaves and then you have to restart with someone else. So the method that I use tries to take that power back a little bit. And something that I learned really early on in my career, I was very lucky to learn by accident, was at a company called Bandwidth, which is my first real job. And Bandwidth is now a public company and they've done all kinds of crazy amazing things.

(00:07:38):
But I joined when it was just 50 people and I actually joined in sales and I was just hungry to make it succeed and grow and bright eyes and everything, first real job. But I realized after a few months of being in sales that I was often repeating the same thing over again, using the same thing to sell over and over again. And it's like, gosh, this isn't ideal for the customer because [inaudible 00:08:04] call me and ask these questions and get these answers and all this stuff. And it's not ideal for the company because they're paying commission on this every time. That's not going to be efficient for our growth. And because we were small, I was able to catch our GM and I was just like, "Hey, I've noticed this pattern where I'm repeating things over and over again and they're asking the same thing.

(00:08:29):
I think we should put that online. I think we should make that available so they can just see it and then buy it," because we had an online checkout process. And I was expecting him to be like, "Oh, well, I know it's important but for this [inaudible 00:08:46] another, we need to do it this way, and obviously you've thought all about it." And thinking, "Oh, I'm going to come in this new person, he's just going to help me understand what I'm missing here." There's a little bit of that that I was expecting. And he goes, "Wait a minute, tell me more about that. What do you mean?" And by the end of the conversation he was like, "Hey, why don't you go do that? Why don't you go build that experience? Why don't you put that stuff in a self-serve flow?"

(00:09:06):
And we called it e-commerce manager and it was like got a growth before this growth, this is like 2010. And that moved me into a totally new position. And the main learning that I had from that was, which really took life at Twilio and absolutely worked for me there and I'm happy to talk about that too. But the core that learning was, your executive team and executive teams at companies are often very sharp, but the nature of their day-to-day just does not link them with customers.

(00:09:34):
And that means that over time, especially as a company grows, they often lose access to some of the best insights and in the heartbeat of the people who they're providing value to in contrast to folks that are closer to the problem. And so that means that your superpower is in really pulling those insights in and bringing them to life, staying close to the customer. There's not a single leader or executive that isn't going to be stoked to hear about valuable customer insights that highlight problems they might not be seeing. And there's a lot of those. So especially when they align to North Star metrics, those ones are the powerful ones. That was the way that I grew my career at Phil and I'm happy to share that journey too.

Lenny (00:10:18):
Yeah, it'd actually be cool to hear maybe another example of that. But I think an interesting thing that comes up for me here is sometimes you may have an awesome idea and it may not immediately happen. It may not be like, yes or let's move on, that's right, immediately. And I think it's important to just recognize they're not going to follow all your ideas, but they're always looking for better ideas. And to your point, they may not have the information that will lead to an idea that you will have because you're on the ground dealing with real problems, day-to-day. So I think it's important to recognize you're not going to always get your way and that's normal.

Laura Schaffer (00:10:52):
Yeah, totally. And it's almost like building up your individual brand a little bit. And I think one of the most powerful and accessible ways is learning about your customers. There's always those people at companies who's like, "Oh, well, she just knows our customers or he just knows our customers, they just know our customers. They just know." And it's like, "Well, how?" "They just know. Let's ask that person. Let's get their feedback." And those people often have a good amount of brand recognition of powers within the company and they're often thought of when the company needs to do something new or different or if someone is hiring, maybe they're thinking about that person for a cross team thing.

(00:11:31):
So it's one of the ways that you can build that brand. And again, I think it's a sweet spot because it's something that is very valuable to everyone, all the way up to the most senior leaders, which we can talk about here in a minute. And so it's going to be valuable for you in a valuable tool no matter where you're at in your career. And that's not always an immediate payoff, but it often does give you a trajectory outside of just your role and just your manager. It gives you something a little bit broader.

Lenny (00:12:05):
So maybe a simple way of describing to mirror back what you're saying is carve your own path. Don't necessarily assume your managers will give you the path that makes most sense for you or even give you the biggest opportunity. Just propose, "Hey, I think this might be a better opportunity and I'd love to pursue it." I'd love to hear the Twilio example if that's generally-

Laura Schaffer (00:12:25):
Yeah. So when I joined Twilio, there was no growth team at all, not even a breath of it. I joined in product marketing and I was leading our product marketing for our messaging lines, but I followed the same guy that I just mentioned. I made it my own personal policy to like, hey, I'm going to do my job. And I'm going to do well, I'm going to keep notes of things I'm doing well and all that stuff because it's good, but I'm also going to get to know our customers. And I'm going to get to know our customers really well and I'm going to pay attention when I'm connecting with them, not just about the space I'm in, but just broadly what are some of the pain points and things they're articulating that are relevant to the business and what we're trying to get done.

(00:13:00):
And one of the things that came up was that users were struggling and folks were struggling to get started and use Twilio. And that contrasted so deeply to some of the things that our executive team was saying and had high conviction in our company had high conviction, which is that Twilio was so easy to use. In fact, it was top three things about Twilio that we were really trying to get out of their brand. Were so easy. Developers love us, they say we're so easy. And there were tweets coming all the time, developers saying like, "Oh my gosh, they got started in a couple of minutes." So there's all these things that made that compound and made that conviction stick. But as I was talking to customers, I was hearing a very different story and it made sense as we were penetrating new markets, adding more products, we were adding complexity and we were pulling in folks who were a little bit less motivated and those things contributed to people saying this is difficult.

(00:13:55):
And so at the time, this wasn't a 50 person in where I could just go to the floor and go to someone and be like, "Hey, there's this thing I heard about, I think we should do something." But there was another tactic that I could take, and I just started sharing a voice of the customer report. I started sharing my insights, started writing down and just sharing them. And it became with digest and eventually people were like, "Hey, can you share it with me? Can you share with me? Can you get on your list? Can share with me?" And this was in a few months of me joining, I was doing this. And then that turned into, "Hey, you should host a quarterly voice to the customer session or for all of product." And this was a request that was coming from some of the senior leaders at the company.

(00:14:33):
And when our Jeff Lawson is our CEO at the time heard about, he started attending too. So now in the session I started pulling in other people's insights too, because now they had a forum for this. I could do that and have people send that to me and I could compile it and all of these things. So then this established me as that person who knows about the customer even after short tenure. And then when came time to do annual planning that year, and I joined in 2014 at the end, so this is 2015, I pitched this idea, "Hey, we think that it's easy. It is not. Here's data that I have, the information that I have and I think that we need to start a growth team here and that needs to be a core focus." And I was able to bring in a really critical partner to that and other folks who could support that because I had built up some of that trust.

(00:15:28):
So by the time I was making that pitch, I had someone on Andre Crow who was the seventh hire at Twilio and got to number three on that spreadsheet or whatever who was really close to the CEO being like, "Yeah, we desperately need this." I'm seeing this. He led a website, he basically created the Twilio brand and he led all the website stuff and he is like, "Yeah, we definitely need this." So not only did I have that little bit of trust from the executive team, but I also had folks who were just trusted on their own advocating and supporting this that I was doing. And so it was approved just almost very easily. I put stuff together for it, but it was the meeting before the meeting had already been done by this other thing. So it helped me create the growth engineering, growth product team at Twilio.

Lenny (00:16:14):
I love just how proactive your advice is here. There's a lot of people that don't do well and then just like, "Oh, I never had the opportunity or I kept got looked over all this time." And I love that there's all this just like, here's things you can be doing to get in front of people to provide value, to just create opportunity for yourself. Any other advice along the lines of just like, here's the things you could do for yourself versus waiting for someone to come and give you opportunity?

Laura Schaffer (00:16:42):
Yeah, I think that's the most easily actual because to do all of our jobs, we need to know customers, we need to know about customer insights, product, we need to know. But then also customer facing teams, social, those want to crack into product. Your insights are extremely valuable. You're talking to customers every day more about their problems and their pain than a lot of other people do. And so that is by far and away to me the most powerful and accessible by anyone in any role in any space. But I'll also say that that broader concept of just, hey, there's things that, and things of value that you know that others can benefit from at your company and building your brand as someone that is supportive, smart, creative, able to solve problems, make sure that you're sharing that. And so maybe you're really freaking good at communicating with brevity.

(00:17:35):
I suck at that by the way. So more powered anyone that can do that, I'm actively working on it. So share that. Go to your general Slack channel or whatever and just be like, "Hey, just wrote some tips for how to do it, some ways that I am good at this." And those kinds of things can really go a long way towards people starting to view you as an SME and not just the space that you're in, but in broader areas. And that can always present open doors for you and other people are looking up to you and seeing you, someone who's strong in ways outside of just the role that you're in.

Lenny (00:18:07):
SME is a Subject Matter Expert, is that right?

Laura Schaffer (00:18:10):
Yes. Thank you for unpacking my acronyms. That's another thing that I am actively working on.

Lenny (00:18:16):
I got you. I'll be on the lookout.

Laura Schaffer (00:18:16):
Yes.

Lenny (00:18:19):
Maybe one last question along these lines is do you have any advice for framing the proposal, framing an opportunity to your manager, higher ups that you see has worked best broadly?

Laura Schaffer (00:18:34):
Yeah. Yeah. And one thing I want to say too is with this stuff, I don't think that it necessarily does counter to what your manager's doing. It's more supporting them. I've done this stuff and then it's helped my manager promote me. So it's not necessarily, "Oh, we'll do this if your manager's failing you." Or they are not the boarding you or they can't support you. It's more like do this because this is going to be an accelerator for yourself irrespective of your manager.

(00:18:55):
But then also it'll be an accelerator for your manager in supporting you because one of the things that comes into play a lot when managers figure out promotions and doing those things is they'll sit in a room, often calibrations and with a bunch of people, and it makes it a lot easier when those people have had some access or exposure or whatever to you in a positive light. So these things can all run with your manager and not against, but it's just another way of you taking back the ability to build that momentum instead of relying on all of that going through one single other person.

Lenny (00:19:33):
And what I like about your second example is you just did it. You just started doing that tenant report for the company. It wasn't like, "Hey, I have a proposal, here's what I think you should do. Should we do it?"

Laura Schaffer (00:19:42):
Exactly.

Lenny (00:19:42):
It's just like, yeah, just do it.

Laura Schaffer (00:19:44):
Yeah, ungate your knowledge I think is the buzzword that I'm hearing.

Lenny (00:19:52):
Mm. Never heard that.

Laura Schaffer (00:19:52):
I think that's an Elena, see how many times we can bring her up. But you can do that within your own company. Everybody is skilled at things that they aren't explicit to their role or their space. And I think that ungating that opens opportunities. And if you're not sure, then go to my favorite go-to, which is talk to customers, get insights. Those are incredibly valuable. So rarely do people share those when they find them. So be the person that does that.

Lenny (00:20:21):
Another area I want to chat about is experimentation and growth and data, which makes sense if strong perspectives is on being the new head of growth amplitude. So maybe we start with experimentation. You mentioned that there's a really interesting surprising result in an experiment you ran at Twilio that maybe changed your perspective on experimentation and what you think might work and not work.

Laura Schaffer (00:20:45):
A hundred percent. Yeah, I'm through a fortune of two mind-blowing experiments that really shifted the way that I think about growth. So one of them, one of my favorite ones happened very early on at Twilio. So after I created this growth team, one of the things that I saw as to me an issue was that under signup flow, we just asked people for a username and an e-mail, like a password, and that was it. And that's actually relatively common at the time. This is a while ago now, everybody is [inaudible 00:21:16]. But we didn't, and actually there was a lot of existing conviction around that. I was like, "Hey, we retarding developers. Developers, they just want to do, they just want to get their hands on things. Don't put anything in their way, it's going to be disastrous. We don't want any shenanigans here with these folks, let's just let them in the gates."

(00:21:33):
But to me this was a really big assumption to make and a very costly one. It's like, okay, if that's the case, we're not going to know anything about anyone. And we didn't know who was signing up, we didn't know what they wanted to do. And that hurt our ability to understand how people were performing from a quantitative perspective. We were a little bit lost with prioritization. There's a number of implications here, but it's obviously a very contentious space. So this is the very first thing that I did and the first experiment that I ran. I did some research to understand say, okay, what are the most important questions to answer? What would I do really, really need to know? And it was stuff like what language are you coding in, what's your use case? What product you want to use?

(00:22:14):
And then there's one around, are you developer at all or were you use something else? Because there is rumors that we're having, not just developers sign up, which is this whole other interesting story. And I think of these questions that would also potentially be things that our developers signing up would understand why we're asking that it would feel natural. But anyway, again, adding anything to the signup was very contentious, but I really just wanted to get a little bit of data on it. So I wanted to run a test. I didn't have a team, I didn't have an engineering team yet and none of that stuff had built out. It was just me, myself and I. But like I said, I had started to build a little bit of trust and [inaudible 00:22:50], Andre, who I mentioned earlier who, because he was early employee and he had access to everything, one of those people, and he also was supportive of this and had similar haunches.

(00:23:03):
And so like the dead of the night, and by that, I mean 7:00 PM or something, I'm pretty sure it was a Friday. We just asked for forgiveness and put these questions into the signup flow and ran as Navy test with a small group. And I'm fully expecting, "Okay, this is going to hurt our numbers, but maybe it won't be so bad and I'm going to be prepared to advocate the power of this data that we're getting." I was totally thinking with written... Started to write the framework for how I wanted to surface this. And we start to get the data for this thing. I'm not kidding, an improved conversion. There's no personalization, nothing past it, just the questions. It improved conversion by 5%, just improved signups. And it was one of those like, "What? Okay, what is going on here?" And I actually dug into it, and what I found from just talking to a few customers once through the flow, I'm just learning about [inaudible 00:24:00] about it.

(00:24:01):
It was actually for folks, it was comforting. When you think about it, when users are signing up for your product for the very first time, it's new. This is new, that means it's scary. They're expecting it to be difficult. They're anticipating that there's going to be friction and challenges and that they're not going to figure it out. Almost like looking for the bogeyman. And that's the headspace. It's often the headspace that any of us are in when we're doing something new for the first time like ooh, this could be very challenging. And so by putting in these questions it's like, what's your language? It's like, "Oh, I do. I code a JavaScript and I can select that." Well, that's something I'm uncomfortable with. That would make my journey easier. Like "Yeah, bingo. That's my use case. Okay, I'm in the right place here."

(00:24:51):
It was actually giving folks something comforting and challenging the notion that this was going to be difficult, just the questions because it was aligning to some of the things that they were organically thinking about, which is what if they don't support my language? Or can I even do this use case I want to do? And so it was just a really interesting, the takeaway for me for this, the really interest takeaway was the psyche of the user is so, so critical. That's just as important as understanding your product and the broader market you're applying to and all those things. Just the psyche of users, new people doing things for the first time in your user flow, understanding that is powerful. And the simple catchy thing I say is that ultimately the learning here is, bad friction is bad, and good friction is good. There's no such thing as it being simple. It's just all friction is bad, which is what I had assumed going into this.

Lenny (00:25:50):
I love that you were new to Twilio and you just year-load an experiment to production.

Laura Schaffer (00:25:55):
Year-load. Yeah.

Lenny (00:25:56):
That's a big move.

Laura Schaffer (00:25:58):
It ended up being very helpful for everyone. I shared the insights from it and all these things. I've shared the [inaudible 00:26:05].

Lenny (00:26:05):
And [inaudible 00:26:05] conversion.

Laura Schaffer (00:26:08):
But for sure, use such processes with caution for sure. Yeah.

Lenny (00:26:08):
I love it. It's amazing.

Laura Schaffer (00:26:15):
That the right way to do [inaudible 00:26:15] advocating for the engineers here is the right way to make any changes in production is through or with the approval of engineering, but it was the right move overall and definitely helped business, so yeah.

Lenny (00:26:33):
Yeah, I love it. No, that's great. I like that move. I think we need more of that probably. I want to dig into what you actually... So what is it you changed, you added how many questions and then what were the questions?

Laura Schaffer (00:26:37):
There was a question around what language are you coding in? And then as an option to that, it was like, "Oh, I'm actually not coding, I'm not a developer." So for us, it actually gave us two really, really interesting data points. One was how many developers versus people who are not coding or in our flow? And then what language of the coding, which was massively helpful not just for growth and onboarding, but our documentation team, dos team. Would that end up being a critical way for us to gauge trends over time and catch things before whatever reports would come out at the end of the year, what people are doing, you start to see it. And then also product. What product are you interested in using? That was very critical for knowing the basics of how to organize someone's onboarding. Are you doing SMS? You're doing voice? To 281 or whatever. And then use case in use cases, you're doing a appointment reminders or are you doing a autoresponder or are you doing anonymous communications for a dating app or something. Right? So those were the very first questions.

Lenny (00:27:39):
Wow, okay. So it was four dropdown questions and that increased conversion. I love these examples where friction and increased conversion... There's so few of them. You hear about this could work and it's rare. And so what did you take away? What's the pattern you took from this? There's the idea, it's good friction, but is there something that you're like, what is a sign of this is going to be good friction?

Laura Schaffer (00:28:02):
There's still alleviated a problem. They alleviated the problem they had where they're coming in and worried that it was going to be difficult or that they weren't going to be able to figure it out, they weren't going to be able to get their footing. And I'd say that that's not unique to Twilio. That's something that I think users experience at any front door, at any company, any signup beginning the signup loads, it's like, "Here we go, buckle up." Especially when it's in a work context and there might be extra pressure on you to succeed or for you to make as an accurate assessment. So I think that psyche of, "Okay, am I in the right place? Is it's going to do what I need it to do? Can I figure it out? Am I capable?" These are extremely common things for people to feel when they're signing up.

(00:28:48):
And so certainly, that I think can carry out to any place. I'd encourage absolutely everybody to be putting those experiences within their early onboarding, not just for you selfishly, so you can learn and segment them appropriately, but also so the user can feel more confident as they get going and like, "Hey, I'm in the right place. This is going to do what I needed to do." But I think that the carry over there is just the psyche of the user and just being so aware that it's not so cookie cutter as, "What is the problem my market experience is and what can my product do to help them?" There's also this other thing in the room which is so important to people's success, their ability to succeed with your products and your self-serve experiences, which is, what is the mentality and the psyche of the person at the various stages in your journey?

(00:29:37):
And if you're not incorporating that or addressing that, you will absolutely miss things or things will fail and you'll be very confused as to why. We had a great experiment that I'm happy to talk about where same concept, a totally different situation, which is later in onboarding. One of the things that we tried to do over time to make Twilio less complex was to offer steps like, [inaudible 00:30:08] onboarding, welcome step one, here's what you want to build. Great, we all know that now, "Okay, step one, go do this thing. Step two, go to this thing. Three, this thing. Four, this thing. Five, bam, you're live, congratulations, aha," all these things. So we shipped that, got that out there and I was like, yeah, it was improved conversion. It wasn't like that great. It's like, man. We went from there being absolutely nothing, "Choose you're adventure, figure it out, go figure it out. Good luck." To this prescriptive thing. And it wasn't converting [inaudible 00:30:37].

(00:30:38):
So to talk to some users and there wasn't anything particularly obvious that was coming out as to what the issue was. It was like, "Oh yeah, Let's go to step one." And we did mock the people. "Okay, now I know, I do step two." But there was one thing that I was hearing that was coming out that feels like something, and that was the telephone number, the telecom part. Developers when they were coming into Twilio, it was things that were familiar to them. APIs, the language they're coding in, code samples, documentation, things like the bogeyman, the things that would psychologically trip them up, telecom, phone numbers. These things that just were completely out of the zone of anything that they'd ever worked with before, especially earlier on in Twilio's journey.

(00:31:28):
But even now, right? Telecom's very different beast for most developers. And guess what was step one? Get a phone number because that's step one. Anytime that anyone's trying to teach one to use Twilio one-on-one, always going to sit down, ask and be like, "Okay, here we're going to go get a phone number and configure it." And that's what anyone every time will do. However, in a self-serve experience, when you don't have that safe person sitting next to you being like, "Don't worry, it's going to be okay, I'm going to take you through this crazy telecom journey." They're on their own, but that's psyche telling is them like, "Oh my god, telecom. Well, I can't do that. That sounds scary. We're getting a phone number configured. Whoa, I'm out of my debt."

(00:32:01):
And so what did we do to test this out? Test out whether that was the issue? Actually, and it's first we're in the MVP. They kicked them out of the portal entirely and put them into a docs page where we could manufacture an experience where the first thing they saw was code and they're in the docs safe place, the language that they're coding in and then snuck in there.

(00:32:26):
It was like, "Oh, get a phone number, let's go configure it." Not as step one, not as the leading thing, but embedded. And the analogy I have for this is pilling a hot dog. So if anyone's got a dog or an animal you have to feed a pill to, it's like you can't just feed the pill to the animal, it's never going to happen. But if you shove it inside of a hot dog, which looks good and that's exciting, then you can get them to consume it more easily. And so this was-

Lenny (00:32:53):
Yeah. We do peanut butter, that's [inaudible 00:32:55].

Laura Schaffer (00:32:55):
Yeah, exactly, right? Yeah, hot hotdog, peanut butter, all that. You bury it. You embed the scary unpleasant thing. And so that's what you said with the phone number stuff, that telecom stuff. And guess what? Even though we're a good amount of the console and they're going off and we had no easy return button, it converted better because we were addressing the big problem that was there at the time, which is their psyche. They were not ready to come in and immediately thrown into a phone number experience. That was letting the bogeyman and out to party and that's not what was going to work. We needed to put that bogeyman pill and the hotdog.

(00:33:31):
And so then once that validated, then we can actually go through the business of putting that into the onboarding float correctly and then that could be even better. But so again, the psyche of your user is such a critical thing to be thinking about. And if something very logical isn't converting well sometimes, it means that you're battling against the psyche of a user and you want to take a step back and think about and learn about where someone is psychologically in your space.

Lenny (00:33:59):
Feels like you had this experiment that was a complete redesign of the onboarding flow and that didn't work. And then your second attempt was a different approach that's like a full onboarding flow. And I'm curious, do you have a take on just when you run experiments? And it's something we dealt with a lot at Airbnb in other places is like, do you just redesign the whole thing or is it better to iteratively work from where you're today and just experiment piece by piece towards some future much better experience?

Laura Schaffer (00:34:26):
Here's what I would say to this is that from a high level, it's always going to be better to be iterative. And the reason that it's better is that roughly 80% of the times, ORs in the time are hypotheses and the things that we believe will be true [inaudible 00:34:42]. And this is amazing. There's an amazing article out there I'm happy to share with you so you can put in the show notes.

Lenny (00:34:48):
Yeah, absolutely.

Laura Schaffer (00:34:49):
That really takes a scientific approach to proving that out. Companies like Netflix and Microsoft, there's over and over again 80 plus percent. Some companies say 90% of things fail. And so the closer you get to something that you go bear your head in the sand or go into an attic and build something for six months and ship it, the more likely it is that you are going to ship the 80% wrong stuff.

(00:35:17):
Whereas the more iterative you are, the more likely that you're going to catch it sooner. And failure doesn't have to be a wall, it can be a compass, it can be the thing that leads you to the right thing. And so you always want to as best you can, get stuff in front of customers so that you can get that compass and get that compass activated, know where to go.

(00:35:40):
So that means doing ugly things. I tell my teams all the time, if it's not embarrassing, you've gone too far. Got to be embarrassing. The first thing, that was embarrassing, kicking people out, onboarding, spend all this money and whatever to get them into your center flow, and then the first thing we do, get out of here. That's nuts. But if it hadn't validated, that would've been a very cheap but very valuable learning. Instead, it was a very powerful cheap learning in the other direction. Okay, now we know we can invest in it. We know that's the right thing to do. So always better to be iterative so that you are letting failure work for you instead of having it be a trap that you fall into.

Lenny (00:36:22):
I know that you just shared as per experiment, you're probably wrong 80% of the time. In my experience, launching a halt redesign is as negative a hundred percent of the time. I've grown weary to avoid that as much as possible, which is like, you know that, you're taught that as you go into growth and product, but you're just like, "Nah, come on, let's just make it awesome. Just redesign this whole thing." Especially your designers is always like, "No, let's start again. Let's make it amazing." But it always ends up being negative and you're like, "Okay, well, it's too late now we got to launch this thing, we don't have time to start again."

Laura Schaffer (00:36:52):
Well, it's funny, in the articles, and you'll see it was written by somebody from Microsoft who built Implementation platform and did all these cool things, as he went into actually trying to apply a scientific method of figuring out how often people are wrong about their hypotheses and what they're planning to do. He's like, "I wonder if that applies to us here at Microsoft." Even for him, that question of [inaudible 00:37:16]. And I think it's challenging when there's a lot of smart people in this space doing things and it's very difficult to think, "Gosh, am I really wrong? 80 plus, 90% of the time?" But when you think about it, makes total sense because what has to happen for something to be successful? You have to understand the problem perfectly. You have to then understand who's having the problem perfectly, the customer. At what time they're having the problem.

(00:37:43):
Then you've got to put the right solution in front of them to solve that problem. Maybe you've got the problem, all that stuff, but your solution something off. Or maybe your solution is right, but maybe it's just not presented, it communicated in the right way. You could have any one of those things off and it's not going to succeed. It's not going to have the metric impact you're expecting it to have. So in that context, it's almost like incredible. We do succeed 20 to 10% of the time given everything that has to line up. And so I think it's one of those things where you really want to go into it embracing that, "Okay, this isn't about how smart I am or how good my team is or any of that stuff. It's just the logic of this is challenging to get it right and let's embrace that and let's lean into that knowledge and make it a part of our strategy," instead of finding against it.

Lenny (00:38:39):
Have you found anything that helps you increase those odds or is just, this is the way of the world and you probably can't significantly increase the chances your experiment works out?

Laura Schaffer (00:38:51):
So here's the thing, I think there's very little that we can do to make that space easier. All those things have to be figured out. And so I definitely think that everybody is going to be in a space where their original ideas, untested ideas are going to be around that hit rate. However, the way that you go about validating those can be totally different and you can be very fast about validating those ideas and that's the key. And AB testing is one of the most expensive kinds of ways to validate an experiment. It often requires design and engineering and the PM or growth person or marketing person who's crafting it. All these things are investments that take a lot of time even for simple thing. And then you have the time factor, how long's the thing I have to run to have an impact?

(00:39:47):
So all of that is extremely expensive. And so I think the key is to just think through, "Okay, what are the things I can do to quickly validate what these ideas are that we [inaudible 00:39:59]?" And you can do that with painted doors, which is where you test rate the concept and the idea before it exists versus the actual experience. You can do mocks. If you've got a designer, create those mocks for that experience, put it in front of people, see how they engage with it. That can be so powerful.

(00:40:16):
You could invalidate tons of hypotheses at that state. The only things you want to get to that deep AB testing environment or ones that have been vetted along the way. And that way, you reduce your fail rate, because you're failing faster by using other methods. So I think I more advocate for that side. Let's fail fast by using those tools rather than figuring out a way that you can rise above where everyone else is operating and figure out ways to solve all that complex stuff better because that's going to be challenging, but you can always get better at experimenting and validating things faster.

Lenny (00:40:59):
This episode is brought to you by Writer. How much hype have you been hearing about generative AI? So much. But how do you take it from a shiny toy to an actual business tool that helps you do your actual job? Writer is an enterprise grade generative AI platform built specifically for the needs of businesses and already widely deployed at world-class brands like Uber, Spotify, HubSpot, and UiPath. With writer, you can break through content bottlenecks across your organization from marketing webpages to sales e-mails in product messages to creating high quality on-brand content at scale.

(00:41:34):
And unlike other AI applications, writers' training happens securely on your data and your style and brand guidelines that you provide specific to your organization. The result is that you get consistent content in your brand voice at scale. Get AI that your people will love for a limited time. Listeners to Lenny's podcast and get 20% off if they go to writer.com/lenny, that's writer.com/lenny.

(00:41:59):
Where do you find the best ideas come from for driving meaningful lift? Is it gut instinct type and experience bucket or is it data telling you like, "Hey, or here's a huge opportunity," in your experience?

Laura Schaffer (00:42:15):
I'm a very data-driven person. I self-describe and think of myself that way. In large part because of that, I feel you have to be constantly checking yourself and data is a really great way to do that. But I definitely think that I would be described as someone who's going more by their gut when looking at date end results just because of the way that I approach it, which is I'm very comfortable and very common in using qualitative responses and things like that and supplement to quantitative data to make a decision and that puts less of a burden on the quantitative to really make an assessment of whether something was working or not.

(00:42:51):
One of the things I see, I think sometimes goes against what other folks do, although I'm seeing things shift a little, is that 95% confidence rate. My background in college, I was in a lab running experiments or really publishing two journal and stuff and we had to have that 95% confidence rate, had to because the things that were coming out of the lab and being published were influencing things like how we do education and how we understand how bias works and when it shows up and therefore how we can combat it.

(00:43:23):
Things were wrong. And sending a bunch of bologna, that can cause some significantly bad things like false positive, false negatives in that context can be very dangerous, for lack of a better word. And you think of other pharmaceuticals, the 95% confidence rate belongs in some companies and some industries because the risk of failure on the impact of a false success is very high.

(00:43:47):
But those of us converting users and trying to upsell folks, we are very fortunate to not have that level of burden on us and we can take advantage of that. And so there are definitely times where I will advocate for and I will push for and I will myself use lower confidence intervals and 95%, especially if that doubles amount of experiments that you can run in a year. End of the day, these are all methods that we use to try to validate the hypotheses that we have. And if you're doing a 95% confidence in a role, you're still accepting a 5%, some amount of false success, do that a little bit more, challenge you to do that a little bit more. And then run way more experience. If you look at the net of what your team is doing over the course of year, what you're doing over the course of a year, you will be positive.

Lenny (00:44:42):
Wow, that is a big idea. Idea of releasing the P-value confidence interval for experimentation and data teams. Everyone would be excited about this. Probably maybe not some data scientists on teams. Do you do that? How do you act? Is that how you operate on your teams? Just like we don't need 95% competence?

Laura Schaffer (00:45:03):
So I'll say this, this is actually very critically important. You must have this game plan set before you run something. Failure mode that I see so many teams fall into is they'll run the experiment or whatever it is and then they'll make the data fit the hypothesis. Or sometimes they'll go without a hypothesis and just be like, "This is going to do better things for our metrics," but not a core reason as to why or what exactly are we testing here. And so this is another area we could absolutely fall into that trap. "Let's [inaudible 00:45:36] on good 80. I think it's good. That Laura person said it was cool. So I think that that's fine." That will always be a trap. So it needs to be very deliberately thought of in advance as a way of like, "Hey, here's how we're going to validate this." And always, always, always, if you're going to accept more risk of a false success or false positive, false negative, you want to then be really thinking about how you're going to harden your validation of a hypothesis.

(00:46:03):
For example, let's take that when we talked about with Twilio where we are kicking people out and we're sending them to the pilling hotdog experiment, and we're sending people to that experience to hide the phone number. Now in that case, let's say that we were going to accept a lower confidence interval. I would very much want to see qualitative feedback to confirm that that hypothesis was true. I want to be looking at the qualitative data from the ones where people were thrown into the existing flow and one's put into the dogs that one of them felt more confident and more like this was really easy to get through and they felt out of a territory and things like that. And I'd be wanting to hear from the ones who were in the other one, things like, "Oh, I got stuck on that [inaudible 00:46:48]." Like, "Figure this out, but it feels like it's that amount of my depth."

(00:46:52):
I would want to be looking for other things to corroborate the hard data that I'm seeing. And yes, it opens the door to whenever you open the door to more risk acceptance, you are going to have some false successes there. But all of these things together can overall make it more likely that you're shipping more things that are going to positively influence the customer. And again, I can't say it enough. It is a huge risk in and of itself to not ship as much as you possibly could in a year. That is a huge risk given that very high fail rate. So to those data scientists, and I've chatted with a few of my time, what I try to explain is that that article, that data that the 80%, that's hard data about what a detriment it can be if you don't run an enough experiments.

(00:47:51):
If you just run 10 in a year on there, maybe two around impact, two of a course of an entire year if you take that approach. So data scientists can understand, "Hey, if we do this, if we run this down, we can double or triple whatever it is, the number of experiments where we can run and overall net that's going to result in more successes that will overall net us to a positive place." You can still tell a data story to the data scientist about why you're doing this. Again, this is why when you asked that question identify, as a very data-driven person. But I think some of the methods that I use can sound at the service level as more like, "Oh, I'm going by my gut." But again, very data driven is just embracing the reality of some of the hard data that I don't think we all embrace or are even aware of sometimes about that fail rate.

Lenny (00:48:42):
This is awesome. This is a big idea. Have you written about this anywhere for folks that maybe want to try this approach at their company? And if not, you should.

Laura Schaffer (00:48:50):
I appreciate that. It's funny, it's like all my general life to do is just start writing some of this down. I have three children, one of whom is five months old, and then I have two and four. And so sometimes I'll start to write and then one of them will crawl across the keyboard. And by one of them, I mean all of multiple times. But eventually, yeah, I'll be very happy to do that if folks would be interested. I'm always happy to do whatever I can to help folks, help empower folks with knowledge to do better because none of this is secret sauce really. It's just learn from experience and it's always better to learn from others' experience than your own. It's faster. So yeah, I would definitely [inaudible 00:49:35] is that, I think that's the best that I can say, but eventually my kids will get older. I hear this and maybe I can do so.

Lenny (00:49:42):
Hopefully. Cool. So maybe if you're watching us on YouTube, leave a comment and if you want Laura to write in depth about this idea and spread it to your company. Okay, I want to talk about growth, but I have one last question just along the lines of experimentation. Is there any other just, I don't know, big lessons or takeaways of running experiments that would be interesting to share?

Laura Schaffer (00:50:02):
I think we got into this one a little bit, but I just really want to exclamation point, underline it, which is that notion of making the data wrap to fit a concept. I think a lot of teams feel and are under a lot of pressure to show progress and, "What did you do this month? Where did the metrics move?" And it can cause folks to feel like they have to do that, where it's like, "Oh gosh, this experiment." Everyone's got the experience where you run an experiment and you're like looking at the data, refresh, refresh, refresh, oh my gosh, and actually perform worse. Or it's not the same and, "Gosh, we got everyone really excited about this thing that we all worked on really hard. Like, oh my god, what are we going to say in the QBR or the monthly report?" Whatever it is that the results come to light.

(00:50:55):
And to this, I'd say this, that it's incredibly important for growth teams to educate out and for folks outside of growth and leading growth, especially to understand that the best way for a growth team to succeed, the only way really for them to succeed is to embrace the fact that they're there to validate, to understand what the biggest opportunities are and to go after them.

(00:51:24):
And that is not something that can be done on a weekly timeline, sometimes even a monthly, depending on the space you're in and what's known and unknown. And so any growth team that's beholden to short timeline wins and improvement is always going to be dangerous. That's an environment that's conducive to vanity Metric usage and massaging the data [inaudible 00:51:47]. And ones that are more successful are ones that are reporting over longer periods of time. Because I think growth team, given enough time to fail, enough time to learn the right thing to do is absolutely going to show success, real success. Not that, "Okay, we're going to make this data fit." But real moving the metrics success.

(00:52:03):
And so definitely educating out. If you find yourself in a position where you are beholden to that, share that 80% fail rate. Just math, statistics, data. You cannot be successful in an environment, but over time you can be. And so that's one thing I definitely would draw on. I end up spending a decent amount of my pie chart at Twilio and then also at Rapid where I was after that and I'm sure I'll spend some time at Amplitude as well. Just helping folks understand what is the healthiest ecosystem, most powerful ecosystem for a growth team to operate in. And time and expectations over time is a big part of that.

Lenny (00:52:46):
When you say pie chart, it's like the pie chart of your time like a big chunk of your time goes to this?

Laura Schaffer (00:52:46):
Yeah.

Lenny (00:52:50):
That's awesome. I like that. I use white charts a lot as to describe that same idea. Just to be a little more concrete there, what is the timeframe you think is the minimum for a growth team to be thinking across?

Laura Schaffer (00:53:01):
I think it's good, especially for newer teams, but even teams in general. Commit to something that you can do over the course of a year and low, medium, high is always helpful in that space. A lot times-

Lenny (00:53:01):
What do you mean by that, by low, medium, high?

Laura Schaffer (00:53:17):
Low, medium high, more like, "Hey we've got a few bets that we have or few core hypotheses." And if they take off that's going to be our high bucket like, wow [inaudible 00:53:26]. We think these things could become lightening on a bottle here, but they could also be a bunch of [inaudible 00:53:31] missed. But until we run in, we're not going to know. And if those bear out though, then yeah, that's our high. And hey, we've got a few things that we think are safer. Maybe it was validated a bit in the previous year, what have you. And these looking really the metrics this amount.

(00:53:45):
So it's helpful to give people though that construct, it deviates from it very hard deviates from this notion of like, here's the single number that we're going to hit. Just things that help people understand that space a little bit better and what to expect. And because of that it can be a little bit lumpy. There were some things that you released. Truly for the most number of years, can be easiest to talk about in this construct here, but there's one thing that we did that generated tens of millions of dollars in the pipeline, was really, really powerful and took, sometimes navigate and validate. Other times we did that onboarding stuff that I was talking about catching those things. That could happen on a little bit of a faster clip but still took some time to validate and understand. But yeah, over the course of a year you should generally be able to commit to movement. But help people understand the methods there so that they're not coming at you on a weekly basis being like, "And what did you do these past couple days?"

Lenny (00:54:44):
Okay, I got to follow up on a couple of these things. What was that big change of Twilio that lead to tens of millions of dollars?

Laura Schaffer (00:54:50):
This is part of the course that I teach at Reforge.

Lenny (00:54:55):
Oh, amazing. You get to work for Reforge with [inaudible 00:54:57].

Laura Schaffer (00:55:04):
I'm actually interested in retention, I think is my part. Right now, I'll give you links so we can [inaudible 00:55:05] in the show notes. But yeah, the high level version, this was deeper into my journey at Twilio. This is fast-forward a few years, build up this team and some cool things going on. But I was really looking for what's the next big thing for us to do? What could that be? And I noticed, remember that question very back of the day when I asked about the developer versus not developer folks [inaudible 00:55:34]?

Lenny (00:55:33):
Mm. Yeah.

Laura Schaffer (00:55:35):
We saw that little non-developer little dude ad was growing. We were actually the number of people in the ecosystem who were identifying themselves as not a developer were in the space.

(00:55:46):
But very interestingly they were, as we got more refined in our understanding of those folks, a lot of them wanted to build with Twilio. There was a hypothesis of like, "Oh well maybe they're lost, maybe this want pricing, maybe they [inaudible 00:56:00] mistake." And I was like, "Nope, they're here to build, they won't build." And they struggled through the developer on onboarding and some of them would succeed and some would... But anyway, it was all about identifying what did they need to succeed. If we were made them successful, could it contribute to dollars? One of the core learnings I'd heard from sales at the time was, "Hey, it's very challenging for us to get the folks when a developer's not involved yet to go from zero to one to get something off the ground. But man, if we can get them to do that, if I can get them in $1 and spend, I can get them to five. If I get them to five, and get them to 50, like 10,000, then I can get them to hundred thousand."

(00:56:39):
This whole long journey like, "Hey, Laura, if your team could just get them off the ground, man, we can do so much." So yeah, the journey is all about, okay, what were the things that were missing in the experience we were offering and ultimately was they couldn't write code from scratch. That was really difficult. And also we're going to stand up a server. That was difficult. But we ended up iteratively experiencing a way to validate those hypotheses and what's the right way to do this and yeah, that was great. It's called quick deploy on code exchange. Anyone can go there and deploy an app without having to write code and get an aha moment there with Twilio.

Lenny (00:57:16):
That is awesome. So basically it's like a low code Twilio app?

Laura Schaffer (00:57:20):
Yeah, it ended up being, we had a lot of pet names, nicknames for it. I think probably the one that most succinctly describes it as just, it ended up being a create your own demo experience which made you talk about the psyche of people. We talked about how developers telecom until can be intimidating. We'll talk about the non-development, sometimes the buyers or the people who are instantly buying decisions, for them, it was not only was it telco, but it was the developer stuff was inaccessible but they still wanted to jump in and they wanted to have that experience. And so this was a way for us to give them momentum, give them comfortable. Geez, I can get this running my development team, I'd definitely do it. And so it was a very powerful moment where we could really address the psyche of those users, get them excited about Twilio, and then give sales the ability to give something powerful to those non-engineering buyers and folks they're talking to.

Lenny (00:58:17):
So genius, looking back seems like an obvious win. One of my readers suggested that I start a series of the story of a feature and walk through the discovery ideation development iteration and this feels like a really interesting example of that. But anyway. I got just a couple more questions. I know we've been going for an hour now. But I have questions, I don't want to let you go just yet, and they're on growth. So one question is just you worked at Twilio, which is very product led growth. You're now Amplitude, which is more sales driven and I know you're trying to go more product led. I know Elena talks a lot about this, how every company needs to have product led motions, otherwise they're going to be disrupted by someone that comes product led. And I don't know what hires, which blanket would they fall into?

Laura Schaffer (00:59:05):
Between the AI like SLG and PLG. Yeah, for me they're two sides of the same coin. Product growth and sales. It's all to me, very thematically the same stuff. The difference is that with growth, you are selling with your product, and with sales you're selling with person like one-to-one. And so companies need to be employing both of those forces to optimally convert their audience. We're in a world where people are expecting both. They're expecting to be sold by your product and sold at the enterprise level. And large companies buy by human beings. It's going to listen to their specific needs and really break it off for them. And if you only have one, you're going to miss stuff. So absolutely, I think you want those two forces together working well. And obviously there's different stages, things work differently in different spaces, but I think when it comes to Amplitude, I think there's a huge opportunity here.

(01:00:06):
I think the key is, and that the challenge for companies that have done the sales thing and are trying to crack into the PLG thing really comes down to how you fundamentally are approaching that space. And again, your users and where they're at and the psyche of where they're at, I think a lot of companies will say, "Well okay, hey, we're going to do this PLG stuff. Let's take that sales enterprise whatever offering that we have and let's chop it up a bit and cut access here and cut out this feature here and we're going to slap this plan out and we're going to put a price on it and we'll maybe have hours of debates over whether it's like 10 99 or $104 or 75, and eventually someone will win that battle and slap it on and then [inaudible 01:00:54]." And anyway, the discussion of the focus is a lot around the product.

(01:00:58):
What are we going to do with this product? How are we going to crack it open and shift shift it in and then give it to these people, these users, these visitors? And what it's missing, I think is, and a lot of times it's easy to miss, is that when we're doing PLG and we're shifting from sales to PLG, we need to reset. We need to recognize that, again, this is sales, sales via the product. What does a good sales rep do when they're engaging? They understand what the problem is of the person in the space they're talking to. So we need the same thing here. What are the unique problems of people who are coming into our self-serve space? And I think when it comes to a company like Amplitude, a lot of the folks that will be looking to address via the PLG motion, there's a number of things we want to achieve there, but one of the primary things is to tap into the SMV market market and really give them a really startups and give them a space to land and to grow.

(01:01:57):
And again, you have to think what are the challenges and unique problems that they have because we're going to be using our product to settle them. We need to meet them where they're at with the problems that they've got. And I think one of the things that I've observed from being in all these startups and advising some startups is I very rarely... I don't think I've ever come across a startup where they have the right number of analysts for their needs. In fact, a lot of them don't have any. And so what that means is that the CEO is being an analyst to create their dashboards for the board and the product manager is being an analyst to figure out what the heck's going on and creating their boards for their product.

(01:02:34):
And that's happening all over the place is that people are in their roles and they have to be an analysts too. And I think that that's a problem that especially younger companies and early stage companies have. And so when they connect their psyche, what are they caring about? What are they thinking about when they're sending it for product analytic product or something? They're looking for something that's going to help them feel reassured they're going to be able to actually get to the bottom of the right metrics, create the reports that show things the right way. What's the best way to show churn? There's got to be a best way so many people are doing it. Guess what? Yes, there are some really good ways to do it and there are some really successful ways to set up dashboards for the board. People have done that too.

(01:03:16):
There's a lot of that knowledge that exists in one of those frameworks that exist, benchmarking. Are these numbers even good? And so, one of the hypotheses that I have is that if we take that perspective and we understand that that is the problem, that there's a number of things that we can do to really change the way that self-serve experience works to help convert people and show them how Amplitude can make them that powerful. But the thing that I think sticks across all companies, not just amplitude making the shift, is just that, that when you're doing this, do not think this is a copy paste, but chop it for parts thing. Don't start with your product when you're building at your strategy, start with your customers, your users, your prospects, the people who are going to be coming into yourself or flow. Make sure you are understanding how their problems differ because they do from the people that you're addressing at the sales led side. And then make sure that you're orienting your experience of product around those people.

Lenny (01:04:12):
It's interesting that you almost have to start again as a product company, as a product because you may need to solve completely different problems that eventually lead to the same place. But it's interesting what you're saying that you may end up targeting analysts or PMs. I know Amplitude or has always focused on PMs, but-

Laura Schaffer (01:04:32):
Yeah, it's right. And there's always nice thing about it is it's in some ways, it does feel like you're starting fresh because you do need to start with the customer again and what's their problem. But in a lot of ways, you can carry over a lot of the same knowledge. At that point you know what's working well. Amplitude for example, does have a ton of knowledge around what some of the best ways artists set up reports. There's a lot of things that they have the momentum going, like where do you choose that momentum and how do you put that and curate that in front of users and make sure that they're getting the right things. There's a ton of momentum already there. It's just a little bit about harnessing it and understanding like, yeah, where are the gaps because there are going to be gaps.

(01:05:08):
But anchoring in a customer problem as I think the way that you start any new product, any new thing that you're releasing, should always think about the customer and the pain point. So no different than when you're doing PLG for the first time or cracking into it. You need to be thinking again, starting again with the problem, the problems they have, the psyche that they have coming to your space so that you can build something that is going to effectively make them feel like, "Oh you can solve my problem, you get me." And show them how your product's going to do that.

Lenny (01:05:42):
Final question, and this is around developers. You worked at Twilio, obviously Twilio sold developers. I think Rapid where you work right before Amplitude also sold developers. Selling to developers feels like such a hot space right now. There's so many startups that are just building developer tools, such a huge market. Used to be not. Used to be like there's not a market in developers. They're not going to spend money, there's not enough of them. And now it is a big popular spot. And so I'm curious, what have you learned about building a startup and a product that sells to developers? I imagine a lot of founders building search tools would be really curious.

Laura Schaffer (01:06:16):
The first is that developers are just a very different audience from any others. I've seen so many people who have come in strong on growth really well or product really well with other audiences and like, "Oh, I'm going to take all those learnings, I'll pivot into serving developers." And as it being a very steep climb because developers are so different. And let me give you a couple just fun facts that make them really different. And some of these have some interesting stories. One is developers, almost two, one, do not look at your marketing website at all. They go straight to your signup flow. So what that means is all that beautiful context that you're setting and the product aid pricing, all that stuff, very often they're skipping all of it context free and going straight to your signup. And so anytime you make an assumption like, "Oh, well, they probably know this coming in to signup." Or like, "Well, we don't need to include, that's on the marketing website." None of that's going to apply to this group of people.

(01:07:15):
They're there. The analogy I have for this group is they're the IKEA buyers who when IKEA package comes, they're not opening up the instruction manual and reading in and then starting to go through, they're in there tearing open the bags and starting to pull the pieces together and trying to build it. They'll come up for context and steps and such when they get stuck if they're motivated. So that's one thing. And then another one is just the aversion to talking to sales. And I think hearing that, some they're like, "Oh yeah, well, I hate sales too." When I'm sent out and get bombarded by sales, that's the worst. I totally get that. But developers are on this whole other level. There was a fang company sign up for Twilio, built a POC, launched to production, all this, and operated in that space for months without engaging once with sales.

(01:08:12):
I was trying to reach them and I ended up being the one that talked to them first because they reached out to support because there was something about their delivery that was off ,there missing a feature and they did not want to talk to sales. They ended up talking to me when I was in product marketing. And that was my first exposure of like, these people not want to talk to sales. And then there's another one where a giant retail company where the engineering team signed up with their personal e-mail addresses so they wouldn't get bombarded by sales. It was only later that we found out. Anyway. But the thing that's most important, these are fun facts, but the thing that I would say is the most important, the thing to leave with listeners here is what makes them so different?

(01:08:54):
Why? What's the deal here? And it stems from their charter and their responsibility. So if we put ourselves in developer's shoes for a minute, a developer, if a developer is required to use your product, especially if they're the primary user, the primary builder, it's really important to recognize that they're responsible for that. If your service goes down, that's their responsibility. Not just for themselves but their team. If the pager wakes up someone because the service they bought from you goes down, that's on them. If, oh, it turns out that doesn't work with the systems that they said it was, well, that's on them. Doesn't integrate with the data the way that everyone wanted it to? That's on them. Everyone lives to developer when it's not working right and it cannot work right. In so many ways, that's their failure, it can cost them their job, it could cost them the trust to their team.

(01:09:50):
It cost them their reputation. And that means that the stakes are very high for them every time that they're adopting something new. So they can't afford to take someone's word for it. Especially a sales rep who might have some other motivations from their perspective, they can't afford to trust your content or someone's word. They must do it. They must prove it themselves. And so that's why, for developers to be bought in, they need to do something, build something, a proof of concept at the very least, if not moving further than that. And so that means they're going to be pretty darn deep in their self-serve experience with you before they're ready to commit. And so if you are a company that is providing, that requires developers to build, you must invest in self-serve experiences in order to effectively convert your audience. And you should be thinking of them. Something akin your self-serve function and growth folks, someone akin to Salesforce because your developers are not going to accept sales coming in and trying to convert them at that stage.

Lenny (01:11:00):
I love that you always come back to the psyche of the user and how in this case, developers like, here's why they're responsible for this thing. Salespeople are going to convince them this is going to work. And it's not. That's a really interesting tool and that's a really cool takeaway.

(01:11:16):
Is there anything else that we didn't cover before we get to our very exciting lightning round?

Laura Schaffer (01:11:20):
Plenty. I think we covered it all, man.

Lenny (01:11:22):
You got all my questions and more. So with that, welcome to the very exciting lightning round. I've got six questions for you. Are you ready?

Laura Schaffer (01:11:30):
I am so ready.

Lenny (01:11:31):
What are two or three books that you recommend most to other people?

Laura Schaffer (01:11:35):
I'm a big believer in happiness. Not just being crunchy or we should all be happy, but also because it helps us do our best work and we're more creative and all this. So one is Simple Path to Wealth by JL Collins. I don't ignore the data, that money is something that often gets in the way of our happiness. I know so many smart people that just have not figured out the whole managing their finances thing. And this book will cover all of your basics. It's very easy to read. He's got an audiobook that he narrates himself. Simple Path to Wealth by JL Collins, he's fantastic.

Lenny (01:12:12):
What's a recent movie... Oh wait, wait, there's more?

Laura Schaffer (01:12:12):
Oh, there's one more.

Lenny (01:12:18):
Oh, let's do it. Let's do it.

Laura Schaffer (01:12:18):
[inaudible 01:12:18] Happiness, which is Atomic Habits by James Clear. If you ever want to change something about yourself or something's not quite working for you, this guy will give you a framework to change it. Guaranteed.

Lenny (01:12:30):
I really enjoyed that book. That guy's killing it. He was on Tim Ferris, he had a great interview. Folks that don't want to read it, they could listen to that. There's a lot of cool tips there. Favorite recent movie or TV show?

Laura Schaffer (01:12:39):
Unabashedly, the Great British Baking Show. I love that show. I love that show for all the reasons everyone loves that show. It's heartwarming and makes you feel good and uplifts you. But also because it is a competitive show. They're trying to be the best baker and they're out there helping each other. They're like a big family. Most reality competitive TV shows that I see, all of them are like cutthroat, they're sabotaging. So I'm just endlessly fascinated also by the psychology of what's happening here. I want somebody to do a research paper on it, get to the bottom of why they're all helping each other. It's wonderful though. Wonderful to watch.

Lenny (01:13:18):
Interesting. I always comes back to psychology with you.

Laura Schaffer (01:13:24):
I know. I know. I feel like I'm really, really sinking deep in there. And it's true though. It's very interesting to me, and I love that show.

Lenny (01:13:30):
What's a favorite interview question that you like to ask in interviews?

Laura Schaffer (01:13:34):
I love asking about a ship or release that is not cherry-picked by the person you're talking to. You can get it a lot of different ways. The thing is, everyone has a big success story. Everyone does. It really doesn't actually tell you very much to ask someone like, "What's a great thing you released?" Because everyone could tell that. Instead, take that away. What's the most recent ship is a really easy one because recency, time. But there's other things you can do to take that out. Just give them specific parameters for a ship that they've shared or whatever, and that will allow you to listen more and learn more about their frameworks versus the outcomes. Because if you're picking a random ship, odds are it probably wasn't fantastic. So they're going to want to talk more about how they approach getting there and that's what you want to know about to know if they're going to succeed, what their frameworks to how they approach things.

Lenny (01:14:24):
That is cool. I've never heard that one. That is a really clever idea. What are five SaaS products that you use in your day-to-day work? Can't say Amplitude.

Laura Schaffer (01:14:33):
I know, right? Still learning which ones we have here. But yeah, I'll just share the ones that I like a lot that I've used elsewhere. So one is Hotjar. Hot Jar [inaudible 01:14:44] also works. Just anything that allows you to put some quick little thing in front of customers, get that qualitative feedback we talked about. It's a critical, critical supplement to quantitative data to understand what's really causing the change or not causing the change [inaudible 01:14:58]. So that's important. I will say Amplitude is a fantastic tool that I have used and I would've said that if I hadn't just joined Amplitude so I got to use it... I know. I got to use it for the first time at Amplitude and it was awesome. So again, like asterisk, because I'm like working there now, but I do actually like it. And Slack, it's boring, everyone says Slack, but I just have to hand it to them.

(01:15:26):
It makes life so much easier and just nod their way. And then Builder, which I'll also put in asterisk on that one, but I really want to serve this. A lot of people don't know about it and it's really helpful. I do advise them so I'm in their corner. But this is another one I also say would be a powerful one. I think a lot of team gets stuck. They're relying on too much in their engineers to make changes. Again, we talk about rapid experimentation, getting these out, out, out. And Builder makes it really easy for folks to do that. Also, a headless CMS, you can drag and drop headless CMS so they do make it easy for non-engineers to make changes. So especially if you're trying to figure out how to get around that 80% [inaudible 01:16:11] that I mentioned, this builder would be a good way.

(01:16:14):
And then yeah, if you want one more, I'll give you Chat GPT, which is really boring and everyone's saying that, but I think I'll just say I don't have any crazy things to say about it except that I do think we all need to figure out how we pull that in to [inaudible 01:16:28] put people who don't do that are probably going to lose out or smart AI whatever bots. But that would be it for you, Lenny. But if you ask me in a few months after I've actually been an Amplitude for a bit, I'm sure I'd give you a different answer.

Lenny (01:16:42):
That's a good time to plug lennybot.com. And I wrote a newsletter or Dan Shipper who created the bot wrote a newsletter post about how he built this thing. And so you could go ask me questions using the content of my newsletter as answers. And it's very cool. Lennybot.com or lennysbot.com.

Laura Schaffer (01:17:00):
Amazing.

Lenny (01:17:01):
There we go.

Laura Schaffer (01:17:03):
I didn't know about that. Well, there you go. I've changed my answer. It's that.

Lenny (01:17:05):
Yeah, there we go. That's all I need.

(01:17:09):
Two more questions. What is something relatively minor you've changed in your product development process that has had a lot of impact on your team's ability to execute?

Laura Schaffer (01:17:17):
Yeah, the be embarrassed thing, like I mentioned earlier, be Embarrassed by the first iteration. If you are not embarrassed, you've gone too far. That really speeds up ships and helps people celebrate the unpolished as opposed to feel embarrassed about it. So just embracing that.

Lenny (01:17:35):
Awesome. And final question. I know you just started Amplitude, but do you have a favorite pro-tip for how to use Amplitude or maybe Hidden Feature people may not know about?

Laura Schaffer (01:17:44):
You tell me [inaudible 01:17:45], but I'll say one thing that was super cool actually that someone put together on my team at Rapid, literally before I left, he put together a video of how powerful Amplitude could be when linked up and integrated with other things like in this case Hotjar and Segment. There was a Amplitude report that someone had created and there was something that was an anomaly happening there. Users were using something in a way we didn't expect and Amplitude one of the reports surface it, but Al Kirsten, we want to know why is that happening. And so we could find out what the event is and then using by Segment, find out what that name was and look at Hotjar and actually go in and get screencasts of people doing that exact event.

(01:18:30):
And from that, we were able to form some really concrete hypotheses about what actually was causing it. And so obviously talking to customers is very powerful, but in this case, just that simple use of connecting and threading those technologies together could really get a good picture of that without needing to engage customers. So the tip would be how you can really amplify when you get an amplitude when you use it with.

Lenny (01:18:59):
Amazing. Laura, we covered a lot of ground, career experimentation, growth, embarrassment, psychology. Thank you so much for being here. Two final questions, where can folks finding online if they want to reach out, learn more, maybe send you an appreciation or two? And two, how can listeners be useful to you?

Laura Schaffer (01:19:17):
Yeah, find me on LinkedIn. I don't post a lot. Yeah, I'll blame my three children. Eventually, I promise that I will. But I'm pretty good about responding to messages, so definitely link with me there.

(01:19:29):
And then what listeners can do, I think I'm always happy to hear feedback, suggestion, all that, but I'll just say I also know that it's a little bit crazy out there right now, especially folks working in tech. So I'm also cognizant what I might be able to do to help all of you. I know there's a few places I advise and rapids hiring. I know of a few folks that are hiring growth, strong growth people and product folks. So if you are interested in learning more about that, don't hesitate to hit me up. I want to make sure that I help as many people as I can in that respect because it's trying times and I'm sure you've heard it and read it, but if you're laid off, this is not about you, it's not your fault. It's this crazy world we're in. Things will get better. And I would be feel very lucky if I could help even one person land. So feel free to hit me up about that too.

Lenny (01:20:21):
Awesome. And maybe if you share some links, we could include links to open roles in the show notes.

Laura Schaffer (01:20:26):
Yes. I know there's a few that don't have JDs open yet. They're that hot off the press, but I'm happy to surface a few things there for sure because I know that makes it easier for people to know.

Lenny (01:20:37):
Awesome. We will do our best with the show notes then. Laura, thank you again for being here.

Laura Schaffer (01:20:43):
Yeah, thanks so much for having me. This was awesome and so much fun.

Lenny (01:20:47):
Bye everyone.

(01:20:49):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcast, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.