Timestamps:
Summary:
In this episode, Amith & Mallory sit down with Neil Hoyne, the Chief Measurement Strategist at Google. As Google's top measurement strategist, Neil has led over 2,500 engagements with the world's largest advertisers, helping them acquire millions of customers, improve conversion rates by over 400%, and generate billions in incremental revenue. His first book, "Converted: The Data-Driven Way to Win Customers' Hearts," was published in 2022 by Penguin Random House.
In this discussion, Neil shares key insights from his book, including the importance of focusing on customer lifetime value (CLV) rather than short-term metrics. He explains how organizations can apply the concept of member or donor lifetime value to better understand and nurture their most valuable constituents.
📣 Follow Neil on LinkedIn:
https://www.linkedin.com/in/neilhoyne/
🌐 Visit his website:
https://neilhoyne.com
Let us know what you think about the podcast! Drop your questions or comments in the Sidecar community.
This episode is brought to you by Sidecar's AI Learning Hub. The AI Learning Hub blends self-paced learning with live expert interaction. It's designed for the busy association or nonprofit professional.
Follow Sidecar on LinkedIn
Other Resources from Sidecar:
Tools mentioned:
Other Resources Mentioned:
More about Your Hosts:
Amith Nagarajan is the Chairman of Blue Cypress (BlueCypress.io), a family of purpose-driven companies and proud practitioners of Conscious Capitalism. The Blue Cypress companies focus on helping associations, non-profits, and other purpose-driven organizations achieve long-term success. Amith is also an active early-stage investor in B2B SaaS companies. He’s had the good fortune of nearly three decades of success as an entrepreneur and enjoys helping others in their journey.
Follow Amith on LinkedIn.
Mallory Mejias is the Manager at Sidecar, and she's passionate about creating opportunities for association professionals to learn, grow, and better serve their members using artificial intelligence. She enjoys blending creativity and innovation to produce fresh, meaningful content for the association space. Follow Mallory on Linkedin.
Disclaimer: This transcript was generated by artificial intelligence using Descript. It may contain errors or inaccuracies.
Neil Hoyne: [00:00:00] All right.
Mallory Mejias: Neil, thank you so much for joining us today. I have been really excited for our conversation with you and for our listeners to learn all about what you do and the insights that you have to share. But first and foremost, I want to hear a little bit about your background and kind of what brings you to this moment.
Neil Hoyne: Of course. And first, the pleasure is all mine. Uh, it's great to be able to do this, to be able to share these, these nuggets and make them live and interactive more than a white paper that nobody reads. But let me go beyond the bio because the bio is pretty boring. And for those that made it through the bio, be like, ah, let's see if he actually has anything interesting to say.
I generally explain it that I do two things. On one side, I spend a lot of time with models, with machine learning, with AI, all the nuts and bolts and the greasy technical mechanics. Of how to figure out what's going to happen next in the world. The other side of it is trying to figure out how companies actually use that to make a difference.
And so there's a lot of technology that is underutilized. There's a lot of [00:01:00] technology that's intimidating. And there's even just that one statistic that came out from Harvard business school years ago that found that 95 percent of companies that implemented something as simple as a CRM solution to manage and understand their customers were unhappy with the results.
And so instead of simply saying we need a better designer, a new product is to say, what are the friction points that companies run into where they try to use technology to transform their business, transform their industry, interact in a different way with their customers? And what can we learn from that as practitioners, as leaders to make better decisions?
Mallory Mejias: That is really helpful. I don't think anyone but you would say that your bio is boring, Neil. So I'm just gonna, I'm gonna put
Neil Hoyne: that out I think it's boring. People always write these bios of like, I'll write it in the third person. I'll acknowledge that. Um, but, but I appreciate that as well. Usually I'll tell people, I say, I like to come in with modest expectations.
We needed to fill a slot. We found this guy at Starbucks. We brought him in and here he is.
Mallory Mejias: I know what happens
Neil Hoyne: on the BBC every now and then they, they bring [00:02:00] in somebody that's, It's not supposed to be on and they put them on, kind of the same thing here, but I appreciate the credibility there.
Mallory Mejias: For sure.
Well, first things first, Neil, you talk a lot about putting the customer first. You wrote a book called Converted that you released in early 2022 about a data driven approach to winning customers hearts, looking at the long term game instead of the short term, particularly sharing lessons from the past.
I'm wondering if you could, if it's possible, sum up your book into kind of one key insight, one key lesson. If someone reads your book, this is the thing you want them to walk away with.
Neil Hoyne: To have permission to do what they already know intuitively is right. Metrics, and this is why we have to train marketers so deliberately, is we have to train them on the metrics like return on ad spend that they're supposed to measure their business on and get them away from this.
Well, shouldn't we invest more money in our best customers and shouldn't we understand who's going to spend the most money with us? [00:03:00] And instead, a lot of marketers are traditionally trained to be fixated on short term metrics. No, no, no. Cast aside long term relationships and instead, if I'm going to give you a dollar, tell me how much money you made from that dollar today.
And really marketing itself, it's odd. Marketing is to blame for a lot of this. One of the historical problems with marketing is can we prove that it works? And when you go back to those days of radio and TV and print, it required a lot of time and a lot of math to figure out if that print ad did anything to drive your business outcomes.
And then digital marketing came along and we said, aha, look, somebody clicked on this picture and then they, they gave us money or they subscribed and we could see those immediate outcomes. And we became addicted to, we said, look how great this is. We overcame the biggest challenges facing marketers, except then we looked at it and we realized that customers are more complicated than that, that relationships are in fact, not built on that single click, but they're built on.
over time where customers may [00:04:00] or may not do something. Uh, we, we realize it's a lot easier to keep customers than it is to acquire them. And so slowly marketers are learning what they've already intuitively understood. They're starting to learn how to put it into practice. And so really, this is a long winded way of giving you the entire synopsis of the book is it's being able to go out to marketers and to say all this stuff that you know is right I'm going to give you not only permission to do it but a framework as to how you make it happen.
And so it's not that I have to teach you about a brand new way to think. I'm simply giving you the steps to say, here's how you start doing what you know is right in your organization. That's been beholden to short term metrics for far too long.
Mallory Mejias: That was a really great summary of the book. You mentioned kind of this framework that you want to teach to marketers about focusing on the longterm game, building and nurturing those relationships.
And it seems like you recommend doing this through CLV or customer lifetime value. Can you explain what that is briefly to listeners? Um, I'll ask [00:05:00] that first and then I'll ask my followup.
Neil Hoyne: Sure, here's the way I put it. Alright, I'll give a personal story because it's more fun. Um, my wife and I dated for about seven or eight years before I proposed.
Which is, the women will acknowledge at least here, it's a very long period of time. And, and she always pushes like, why did it take you so long? And I'm like, because I'm a guy and that's what we do. But I always ask her, I say, well how did you know that it was? And, and she had a, she had a secret. She had, uh, her grandmother, her grandmother.
And her grandmother in her 80s, she asked her, she's like, is this a good guy? And using that collective wisdom and intuition to look and be like, yes, he's a good guy. And that was enough for her. That was enough for her. And CLV is very much like, whether it's a grandmother, an aunt, a brother, a father, whoever that trusted person is in your life, looking at your customers and saying, there's something there.
They're going to be with you for a long time. They're worth your investment. They're worth listening to and getting to [00:06:00] learn and They're always right and that is CLV. CLV is just except instead of bothering your grandmother about like a thousand people or a million people You build these models that can effectively tell you with a great deal of confidence that these people are worth your time And for a lot of businesses, that's the first step And the reason why you do this even though it's a boring metric is really twofold One is To give yourself confidence that it's fairly easy to use these types of predictions.
That you can actually generate them yourselves. People think that they're, they're some strange magic. No, they're simplified to where almost any company organization can calculate this. And the second is even before you act, simply to know you have them. And just very much like in our own personal lives, how you can categorize.
You, you know instinctively, these are good friends. Or you meet somebody and you're like, we know, we're, we're gonna connect. This is gonna be fantastic. And other people you're like, I don't think I'm ever going to see you again. You're a great Uber driver. I appreciate you, but never again. And, and this is what CLV is doing for your business is [00:07:00] it's giving you the confidence that you can understand your customers.
And it's giving you that list to say, these are the people you want to pay attention to. These are the people you probably don't. And then it's challenging you to say, now that you know this, what are you going to do differently with your organization? Because of it.
Amith Nagarajan: That's, um, that's Super interesting. And I want to try to contextualize that a little bit for some of our listeners when they think about, um, Neil's commentary about customer lifetime value.
Translate that to member lifetime value or donor lifetime value or constituent lifetime value, which may be measured through an economic lens. In terms of dollars, it might be through volunteer hours and engagement. Um, the mechanism through which that you are projecting the V, the value part could vary depending on the organization and the way you're set up.
And I think that the idea that you're projecting or predicting what that lifetime value might be is a really powerful construct that does help you to think both in the longer term, obviously, but also to potentially differentiate where you spend your energy between people who are very [00:08:00] high on that lifetime value possibility and people who may not be.
Neil Hoyne: It's exactly that. It's exactly that. The customer part is as you define it. The value part, as is you define it. It's built around your business and your needs. It's just having that conversation to say, We can look more than what these people, these members, these subscribers, these donors do today. And let's look at what they're likely to do in the years to come.
Mallory Mejias: I love this analogy you gave of the grandmother looking at the man. In that case, you are looking at a list of people and saying, I have this feeling that These are the customers that have the highest lifetime value or members or donors, as Amith said. However, when we're talking about feeling right, we're actually talking about data and there is a way to calculate this.
Can you, without getting too far in the weeds, talk about what pieces of information you might need to find your customers with the highest lifetime value?
Neil Hoyne: It's really two groups of information to calculate the basic understanding of these relationships. We're going to look historically at what all of your donors, [00:09:00] members, customers did in the past to help inform these models.
And by that we mean An id so we know what their name is a first name a last name and id in your crm database if that's if that's what you use you want to have the date that something happened because we want to look at how frequently are people donating time or donating money and then we want to look at the assigned value of that moment ideally we want to look at if we're looking at monetary most companies will look at profit but if you're just looking at straight revenue a donation you can do that as well and we're going to generally look at that information for several years back and so Um, on a retail setting where transactions are happening every three or four, three or four times a year, that's going to be about two years worth of data.
If you're seeing people once a year, you may need four or five years worth of data. But you are able to draw it out, and that's the only information that you need. That's enough to inform the model. Uh, the recency of transactions, the frequency of transactions, the value of those transactions. RFM has been the cornerstone of [00:10:00] marketing for more than 40 years.
Those same metrics apply here. Now, once you have that Then always a question comes and you understand, here's a list of my high and low value customers and how long they're going to stay with me. The inevitable question comes up to say, well, why? What makes them special? What makes them different? We start as marketers, we build personas.
We want to know, how did these people come to me? Or what's most of interest? Or what connects with them? Or what activities do they partake in? What content do they look at on our website? And for all that content, Excuse me. For all that content, take it as it comes. Which means, you have your hypothesis, if you're curious about what emails they interact with, then the only data you would need is, well, what emails did they open or click on?
If you're looking at what sites did they come from, then the only data you would need is, how did you acquire these customers? And where I want to draw the distinction is a lot of people think I need to have all my data in order. I need to understand everything about my customers before I [00:11:00] begin. And really the takeaway here is one is to calculate the values, it's pretty straightforward.
It's the date of a transaction, the amount of a transaction, where was their value created. And then afterwards, it's a choose your own adventure based on what you think or what you're curious about as driving value. And so if you want to look at acquisition, do you know where they were acquired from? If you want to use it, look at the products they use or the creatives or who they spoke to over the phone.
It's just a question at that point. Do you have that data to understand how things are different based on those activities?
Mallory Mejias: Okay. And once you have these values calculated, let's say, I'm guessing it's twofold the strategy from there. So on the one hand, you're growing your business by continuing to engage with those high lifetime value customers, creating more opportunities for them to interact with your business.
And then on the other hand, you are trying to tap into a new audience, grow your audience by finding customers who are similar to those. with high lifetime value. Does that make sense?
Neil Hoyne: That's exactly it. [00:12:00] And I'd give also one opportunity. We also look, look, there's some people that love your organizations.
They don't need love. They don't need messaging. They're going to come back repeatedly year after year. And so part of it is understanding where your marketing efforts, where your outreach efforts are going to make the biggest impact. So if you have customers that are, if you have customers in, in any setting that are going to come back where you're, we all have these products and these brands that we just absolutely love, where you're like, it doesn't matter.
I'm a customer for life. They don't need to send you a coupon code for you to buy their products. You're going to keep buying. All they're doing is giving away margin. If you have to invite people to events, you don't want to have to invite people for events. If they're going to give you the same amount of money they would have given you otherwise.
And this is part of understanding where those limits are. And so what we do is. We put a lot of emphasis on acquiring great people because when you acquire these great people, one of the attributes is that they're a perfect fit for your organization and your mission. They will stay with [00:13:00] you, they will come back, and they will not be, so we may call it in relationships, high maintenance.
And then you have people on the other end that, no matter how much love you give them, they're not going to do anything different. And so, alright, so we can ignore both sides of those and where we spend all of our time is right in the middle to say, Hey, Who are those people that if we gave them a little bit more love, a little bit more attention, we'll see changes in their behavior result?
And that's really where we want to focus, those people. I always joke with it, airlines are a great example. I spend a lot of time flying. But on airlines, I fly, I fly United Airlines often. It's not uncommon that I'll get upgrades on them. But I'd have their question for their analyst to say, Is this really creating any value for me?
You're the only carrier that flies from San Jose, California to my hometown of Chicago, Illinois. If I want to fly direct, the easiest way is for me to go there, and I don't care what seat you give me, I don't care how much you charge, it's easier than flying into San Francisco. And so why give me an [00:14:00] upgrade?
It's not changing my behavior. You're giving me a bulk discount. You're giving me. Oh, well, Neil, you flew us ten times. Here's a free seat. No! Who you want to look for are those people that may just fly you one time from Delta or American and to say if you have such a great experience, if you're not in the very last row middle seat, then you'll fly with us a little bit more often than you would previously.
And so you win that share of their money and that's how you start building a relationship with them. And so the first thing is to get all the best relationships you can. And then to that second point you made is find those people where your outreach efforts, where your time and your investment are going to change their future behaviors, not simply recognize them for something they were going to do already.
Amith Nagarajan: So I want to try to apply this to a common business problem that associations in particular have. So these are membership organizations within various professions and trades and so forth. And a common problem that people tell me about is you have people who do join these associations because it's kind of the [00:15:00] expected behavior in a sector, whether that's law, you know, join the bar association, or if you're an accountant, join the CPA group, and if you're a doctor, et cetera, et cetera.
And at the same time, while it's fairly, uh, Low churn. They continue to get renewals and membership renewals from these people. Um, there's not really a lot of value being created in a way because they don't have those people engaged. They're not buying other products, attending meetings, taking courses, doing anything else.
So within the context of like the membership transactions, Since everyone's doing that within their existing ecosystem, um, would it be in, in the context of what you're describing to calculate CLV or member lifetime value to perhaps exclude the membership since that data could be potentially dilutive, uh, in the analysis in the model or would you still include membership data even if it's not necessarily the factor that relates to what we're talking about in terms of getting them to do other things with you?
Neil Hoyne: I'd still include them. You're just going to see, unsurprisingly, a very large portion of, of your base that doesn't contribute a lot of [00:16:00] value. They pay their fees, and the takeaway there is, we probably don't need to listen to them a lot. And so again, going back to the airline example, a good one is just to say, if you look at a typical airline, you're going to have, I think American Airlines, their outgoing CEO a couple years, acknowledged that 83 percent of the people on their plane are not going to fly with them again for the next 12 months.
All right, 10 percent of the revenue on the plane or, you know, 70 percent of the revenue on the plane is going to come from the 10 percent of people that sit up front. So if you're doing a survey, about what you need to do to build the future of the airline. Are you going to ask the people in the back of the plane who you'll never see?
Are you going to ask the people that are creating that value so you can get more of them? And again, we're just being a little bit more focused. We're not saying, Hey, you want to fly us? We're going to turn you down because you're not gonna fly with us a second time. No, we're happy to take your money.
We're happy to take your fees. But when we look at building the future of the organization, we want to know those people that are going to contribute. Another example is that you often may want to find early on [00:17:00] how to distinguish which category these people are going to fall into. Now, in those cases, we may certainly look to say, well, are they renewing?
Are they participating? One of the great case studies we had at Google was with our Google workspace product. Now, if you're not familiar with that name, that's our corporate version of Gmail. So if in your organization, you have your own version of Gmail just for your organization, that's Google workspace.
And a lot of people sign up because a lot of people want to play with the product. They want to explore the product, but not a lot of them actually buy licenses. And so we have that question, well, with our salespeople, who do you send the salespeople after? Is it anybody? Is it the people at large organizations versus small?
And so what we did was we were very deliberate to say, after somebody signs up for a product, after they just become that basic subscriber, let's look for activity in their account that says they may be slightly better than what we expect. And those are the people that we target. And so that might actually work in this case.
You have people that just [00:18:00] subscribe and say, look, we're going to let anybody join that wants to join this professional membership, but then we're going to look very closely at what they do in the first 60 or 90 days. Do they come to a meeting? Do they interact with our website? Are they opening our emails?
Give me something that says these are people we should lean in on because they look more engaged with our organization than people that just come around to check a box. And that again is what we're looking at that long term. These people are giving us signals that they want a better relationship with our organization, and we're going to identify those and we're going to respond in kind by trying to bring them a little bit closer.
Amith Nagarajan: You know, what you're describing is, um, first of all, I think really insightful for our audience and also, uh, in some ways it's, it relates in a way to work that associations do a lot around this idea of the first year in the journey where they analyze deeply, you know, what happens in the first 12 months of a member's journey with the association.
What are the key things that happen? What goes well, what doesn't. Uh, and sometimes there's some interesting [00:19:00] insights coming out of those types of studies when they're performed from time to time. And But what ends up happening in my experience is they then apply their findings universally across everyone.
So they don't like the idea of picking their favorite child, so to speak, and saying, Hey, this is where we're going to focus their energy. And as a result, they get mediocre results everywhere. Um, do you have any thoughts on, on that problem? And perhaps how people might think about it differently? Because they like to look at it from the viewpoint as a, Not for profit membership organization.
They are there to serve all their members. And they're seemingly disproportionately serving the ones that would grow the most or things like that. What are your thoughts on that piece?
Neil Hoyne: It's a well recognized problem. And the way that I present things in the book has to be delineated because it makes a lot of sense.
Good customers versus bad customers. Let me give you the practical reality as to how companies do this. They may spend 5 percent more time with their better customers and 5 percent less with their worse. In some cases, they will look and they will learn about their best customers, but they may not discreetly act on it.
There was [00:20:00] a gym I worked with one time where they said, look, we know who our best customers are. We know who our worst customers are, but our goal, our mission is to help everybody get in shape. And I said, that's fine. I understand that, but let's at least understand the different needs of your customer segments more than just if they signed up for a membership.
So if you can look at your customers and say, value is not created based on the number of dollars they spend with us in your organization, value is created maybe by the hours that they spend utilizing your gym. Interesting thing I learned, at least for Americans, that if you sign up for a gym membership in January.
Right? January. New Year's Resolution people. We know them. You will only use that membership six times in the next twelve months. Four of those times being in January. Now you might look at this from a value lens to say, great, they're giving me money for twelve months and they're not using any of my stuff.
Or you may identify to [00:21:00] say if value is created based on the number of times you're going to come back, and we're gonna say every time you come back, we look at that as a dollar of value created. You're gonna say these are people that I need to lean into because they're not gonna create any value for me.
They're not gonna use the gym after the first month. What programs do we build for them? Because that's consistent with our mission. And that's why we talk about the value. That's customer lifetime value is that that value has to be in alignment with what your goals are for publicly traded organizations.
Shareholder value for nonprofits. It may be mission value, purpose driven value, and that's how you define it. But again, that what we're looking at here is simply saying how much value are people going to create more than what they're doing today. And we have that perspective and how you adapt that into your mandate into your programs is entirely up to you.
There's no right or wrong way. I just want to bring awareness to say we can get an understanding as to what we think will happen next.
Amith Nagarajan: That's super helpful. Thank you.
Neil Hoyne: And by the way, I have to bring that up because whenever I bring up relationship things, you're going to get people bring up and be [00:22:00] like, so, so do you do a relationship value for all of your friends?
No! But you still are nice to everybody, but you still intuitively know when these people call, you will pick up, you will move mountains for them, and other people will be like, I'm happy to help you. And that differentiation is fine, you're still a good person, you're still a great organization. It just helps you to become, I would argue, more effective with your mission.
Because you know the people you can and cannot influence. And this is an important message to bring up. Sorry, I'm going to jump ahead of this. You may ask this, but if not, I'm going to volunteer it. We often find that a lot of people, whether we're talking non profits, B2B companies, retail companies, or travel companies, they have a certain behavior that's fixed on acquisition.
You just can't change it. Somebody's going to be a great fit or not. And where a lot of organizations actually go astray is they look at these people whose behavior inherently cannot be changed and say, we're going to invest a lot of time trying to change it. And again, personal relationships work best, and this is the example I [00:23:00] love to give, but if anybody ever comes in to you and says, I found the perfect person for me, they're dating somebody, I found the perfect match on, you know, I don't know, Tinder, whatever people use nowadays.
And then they follow it up by, But all I have to do is change them. You're gonna be like, No, no, no, no. Find good people outright. Don't try to change them. It's not to say you can't change them. It's just, it's expensive. It's time consuming. It leads to a lot of heartbreak. It's easier to find people that you can connect with outright that have what you need, that already have that affinity for your organization.
That's all we're saying. But no, it doesn't mean, if you want to try, yes. But just, we're giving you a prediction to say, we think we know where things are going to end up, despite your best intentions.
Mallory Mejias: That makes a ton of sense. Certainly my my alarm bells went off in my head. Even hearing you say that analogy, right?
We can't change people and we shouldn't go through life thinking that and we shouldn't go through business. Perhaps thinking that as well. Um, I kind of have one more question here, Neal, for [00:24:00] the our listeners who don't have millions of customers, millions of members. Um, Is there such a thing as leaning on customer lifetime value too heavily?
And in that sense, I mean, if you don't have that many customers or members to look at, and some of those might have potential, even though they don't display those typical markers, maybe of someone who is high lifetime value in your organization, but they have the potential to be just like someone might have the potential to grow and be changed by an individual.
Um, do you ever see companies or marketers ignoring those with potential or if you calculate it correctly? Should you not be losing those people with potential?
Neil Hoyne: Where customer lifetime value tends to go astray is, to your point, companies using it more aggressively than they should. They look at it to say, almost to borrow that airline example, if we're going to get 70 percent of our revenue from 10 percent of our flyers, we only need that 10 percent and then the rest of the seats on their plane are empty.
Or if we only want to service our best customers, then you're going to [00:25:00] have an empty ballroom the next time you have a conference. Really what we're looking at is to say anybody that wants to be part of your organization, that wants to share in your membership, you want to support them. You just want to do it in a way that's appropriate to the value that they're going to bring back.
And so really what we talk about is it's not only awareness, but exactly to your point, it's also offering that constraint to say we don't need to tear up all of our processes and our mission. We start with simply understanding that these behaviors exist, and then I really encourage organizations to say, your goal is not to transform your business tomorrow.
Your goal is simply to say, if we understand that these people may be better, or these people may be worse, what's one small thing we can change In service of those groups or in recognition that these behaviors are happening, and then slowly that organization becomes more member centric becomes more customer centric.
And that's really the change that we're looking to happen. It's not where it's so precise that you spend time say, Ah, you're not worth it. Get out. It's just to say, No, we [00:26:00] guide with the principles of the organization first and all these metrics and all this data and all these predictions are simply helping us better fulfill that mission.
And that's as far as we take it.
Amith Nagarajan: You know, Neil, your thought about getting too aggressive with CLV or any particular, um, objective metric, I think is a really good one. And it's interesting because I'd love to get your opinion on something. That's slightly off track here, but I think really relevant to the conversation for our audience, which is we think about, you know, optimization and pick any objective function for an AI algorithm.
And you can kind of, you know, get extreme with it. But you think about like social media and the way you're optimizing in a social media feed on any pick any of the social media platforms typically. Um, there tend to be optimizing on eyeballs on site, amount of time spent on site, the number of scrolls you have, therefore how many ads can be displayed, which is understandable given the business model, the nature of what those organizations are focused on versus necessarily providing the best [00:27:00] information or the most entertaining information or whatever the case may be, which may have different objective functions.
Some may be longer term in nature. Some may be shorter term in nature. We encountered this exact problem, actually, when we were building one of our AI startups called Rasa. io, which was, it's basically a personalization tool for email newsletters that delivers one to one emails. And, um, when we launched this company seven years ago, There was a lot of debate amongst the engineering team saying, Hey, what, what objective function should we essentially use?
Should we look for maximizing open rates, maximizing click rates? Uh, what's the most important critical number to focus in on? In a similar sense to what we're talking about within the context of CLV. And ultimately what they chose is to lean on their purpose statement, which is to best and better inform the world, which is a long term view.
And so they were looking at a 12 month open rate and a 12 month click rate rather than a more immediate thing, which is kind of hard for a lot of folks to necessarily grasp their heads around because it's a It's looking a little bit more long tail and so forth. But the theory was that if you could inform people quickly and get them out of the newsletter as fast as possible, [00:28:00] that actually better serve them.
And then they keep coming back and they get more value from it. Um, is there anything from that example comparing like social media feeds versus like the Rasa dot I o a I, you know, that approach like that applies to see, I'll be like the kind of mindset of how you calculate things near term versus mid term versus long term.
Neil Hoyne: You know, I think for any publicly traded company, as I mentioned to before, if you're driven by creating shareholder value, then your metrics will align to that profitability, profitability comes from ads. We need to show more ads, more interaction. If you're a nonprofit and you're focused more on a mission, then I would hope that the value that you define, whether it's customer member or lifetime value is reflective of your goals in that mission.
As it comes to specific metrics, and people often wander in circles to say, do we use click rates, open rates, new signups, app downloads? Do we look at lifetime value as the full lifetime value of someone, or five years out? The answer I often come back with is to [00:29:00] say, I want to know what the difference would be to the organization.
If you were to look at a three month open rate for email, or a 12 month open rate, or a 36 month open rate, what would be the difference? Which customers would you serve? Which messages would you put out more or less? And if there's that difference, that's where I want to have that debate. More times what we find out is the behaviors would be the same.
And so we argue about things that are existential to be like, well, we want to use this metric versus that. If the business outcome and the decisions are identical, then it's not as interesting to me to have that debate. If the lifetime value people, I work with some people who are very strict. It must be the full lifetime value of the person.
And you get a CFO that's like, I'll give you three years and they get frustrated about this and they have these organizational debates. We're not going to move forward until we resolve it. And then I look at it to say, you're going to be reaching nearly the exact same people in that period versus the other period.
If it's not changing your decisions. Let's not do it. The thing that I want to be [00:30:00] careful about is oftentimes in organizations, metrics fall by the wayside. You'll adopt a metric, you'll look at how it changes year over year, but people will forget why you ended up using those metrics. They just become historical.
You're like, well, that's how we've always looked at performance. And so I always encourage organizations to say, whatever metrics you choose, Just make sure you go back and re evaluate that time and time again to say, what happens if we use something different? Would our business look different from a different lens?
And this is a lesson that's very difficult for some organizations because it in fact creates controversy. Wait, if we looked at our business, businesses in data, everybody that you listen, you'll hear, in our world, in data, it's a single source of truth. And that's very bizarre to me. And I say, well, why are you going to cut out all the other voices that could provide you with an opinion as to how your organization is doing and saying, we're only listening to this person.
We're only listening to this metric. And so what I would do is I would look at other metrics as well. Use the metrics you have [00:31:00] today. But then tomorrow or six months or 24 months, go back and say, if we looked at our business in a different context, would we still be happy with the decisions we're making?
Would we be doing something different? And have that conversation amongst your stakeholders to say, which one do we believe is consistent with the mission of our group?
Mallory Mejias: Well, Amith kind of touched briefly on AI and Rasa. io. I think that's actually a record, Amith, for the longest we've gone on this podcast without talking about AI, which is pretty impressive.
So now it feels like a good opportune time to shift the conversation there. Neil, we've got to ask. What are you, how are you using AI currently with your own marketing? Um, and do you have a top few use cases that you could share with listeners?
Neil Hoyne: You know, I would, I would say AI in mind, AI has this model like a little left brain in the right brain, right?
You have a very quantitative side of your brain. You have a very qualitative and creative side. A lot of what's been happening in AI lately has been on that more [00:32:00] creative side, large language models, chat, GPT, Gemini, things that allow us to create stories, to create pros, to rewrite emails, to engage with customers and chatbots.
Those are interesting use cases because they just, they lead to almost a furthering of humanity and how we communicate. How do we communicate with our members more efficiently, more effectively? How do we, there's a circular argument internally to say, If, if an AI tool can build a better email, that's more persuasive and communicate, shouldn't I use that?
And then recipient saying, if I can use AI to read those emails and summarize it for me, shouldn't I use that? And they're like, do we create a cycle just where it's AI talking to each other? Be like, can we do this? And the AI responds back. Yes. I think it's a very fascinating experiment that as these tools see more and more adoption, we're going to see the implications of it, but everything is largely positive on that angle.
Now, on the other side, the more quantitative side of AI, that's around predicting. It's around being [00:33:00] able to look at MRIs or CAT scans and to say, what's the likelihood that disease will form that a radiologist may not be able to discern on their own? To say, we could take a radiologist who has 10 years of experience, but what happens if we use every x ray that we've seen over the past 20 years?
Could we make a better determination? Even everything we've been talking about, lifetime value, can we predict what customers, what members need next, what their sensitivity is going to be to different prices and options for memberships without having to roll those out. Those use cases are going to take a lot more time, but that's where I spend a lot of my time thinking about is, for any, for any owner of a business, if you could predict something about the future, what would you ask about?
What would change the way that you look at your business and that you look at your community? And that's where I think the really compelling part of AI is, is that you have all those immediate use cases around better conversations and more language and better [00:34:00] interactions, but then you have that harder, more quantitative side to say, if we could reasonably predict What's going to happen?
What would we do differently? And that's the part where it actually even becomes creative in and of itself to say, what would you want to know if you had a magic crystal ball that could predict something more than just the lifetime value of your members? But what if you could predict market conditions?
Or price sensitivity? Or different programs and options without having to build those? And what you get back to on both sides is that you see a very human component to AI. AI won't know the questions your organization should ask. It can simply accelerate your business by helping you make faster decisions or accelerate your business by being able to serve your members better or being able to accelerate how long it takes you to write that perfect email copy.
But it still comes back to now a lot of leaders saying that they actually fell intimidated by AI. AI is so technically savvy. How do I possibly get my hands around it? A lot of my time is trying to figure out how you get them to step back and to say, no, your [00:35:00] value is not figuring out what is my AI strategy.
Now, that's a ridiculous question. Your real thing is, how do I take my existing strategy and use AI to get it to go faster or to be better? And so it's actually for a lot of areas in AI, it's bringing business owners back to focus and to say, this is your role, this is how you focus on it, and this is what AI is going to do to help accelerate it.
And for them, that's a lot of calm, but it's also a large area of focus, which is to say, no, stop being distracted by everything AI, and let's go back to your business. And let's just talk about the things you want to do where AI can plug in to help you do it better or faster.
Amith Nagarajan: You know, you know, I think a good area of, um, exploration around what you're describing there, uh, for these listeners is around perhaps topics that people will be interested in in three months or six months or nine months where we're going to have an annual conference.
And at that email conference, we have to make decisions on the keynote speakers we're going to invite and the program content overall, what should we [00:36:00] focus on? And that is a very human decision. Uh, there's not a lot of predictive work being done in this field. Broadly speaking, some people are playing around with ideas, but I think that particular category, uh, potentially holds some promise in terms of being able to come up with, uh, programs and content that will be more appealing, more broadly, uh, marketable, et cetera.
Um, what are your thoughts on that? And what are your thoughts on, um, you know, applying some of the thought processes you've already shared into other use cases like the one I just described?
Neil Hoyne: Hey, if you want, if you don't want keynotes, reach out. I'll go do it for you. Uh, just as seriously with, with a lot of it, what I find organizations need is they need permission to decide a path forward.
When things advance as quickly as AI, there's a lot of uncertainty and there's a lot of fear that they're going to be left behind. And what I worry about is it causes distraction, it causes over investment. And when I look at these large groups, oftentimes when I see these people in the [00:37:00] audience, when I do speak to them, a lot of them are intimidated, they're confused, they're worried that here's a pivotal opportunity where they're going to miss the boat.
And a lot of them bring up, uh, here in the U. S. at least, they bring up the Blockbuster and Netflix example. Right? And they're like, Blockbuster is like, they had VHS and DVD rentals and then Netflix came around and the Blockbuster leadership team missed that opportunity. And then they say, we're never going to be Blockbuster.
Our organization will see what's coming next. Here's the thing, when you actually talk to the Blockbuster execs, they all saw that streaming was coming. What they couldn't do was affect the organizational change in their business to take advantage of it. Blockbuster had 800 million dollars annually in late fees.
And they organizationally could say, How do I transform my organization to use what's coming next? When everyone's like, To use what's coming next, streaming means no late fees. Shit, what do we do? And so, that's, that's what they need to learn. They need to know, [00:38:00] not just what AI does, but how, as leaders in an organization, do you start to pivot?
And the same thing with your members. If you're asking your members for content, how How do they know what's coming next and how do you navigate them through and that that's the value of those organizations is that it's not only that they're listening to people like us go back and forth about what's happening, but how do you get it down to every single one of your members who arguably in their own organizations are seeing the same type of change to say, Do we need to use a I are we not using enough a I should we hire people for a I What do we do?
And then again, messaging to them to say, Relax a bit. You're great as you are. Let's focus on the right things.
Amith Nagarajan: In many ways, part of what you're describing with Blockbuster and I would say existing successful franchises of any kind is this classic innovators dilemma where it's very difficult to get out of your own way when you have significant critical mass and a flywheel around it and all all of the usual things that lead to people [00:39:00] not wanting to undermine the revenue streams and the profitability from their current business.
And I think associations and non profits do have. Um, I'd love to hear to the extent you can share, um, Google's perspective on generative A. I. A. And the impact with respect to the traditional search business and how that's going to adapt. Clearly, Google's moving really rapidly right now from a tech perspective with Gemini and introduce of generative results in the search bar.
Um, really curious how you know the ad business fits into that long term and, You know, how you balance that, where you have, you know, a significant amount of revenue coming in to power an organization the size of Google and relative to the potential disruption that generative AI may have in that realm,
Neil Hoyne: you know, we spoke a lot about missions.
And one of the one of the defining missions of Google has always been to organize the world's information to make it accessible to make it useful. And generative AI is simply an extension of that mission. Uh, the mission's never been, and to, to maximize profitability. No. The goal is to make sure that people have the information that they need.[00:40:00]
Now, interestingly enough, a lot of people look at search, and surprising, even in this day and age, 25 years out, that nearly 20, 22 percent of the queries we see every day on Google are brand new. People expressing themselves and asking questions in a way that's unique to all of humanity and everything that we've measured.
And for the first time, and I think what we, as we start integrating generative AI, we're starting to see how people react to it, how people use it, the queries that are helpful, how they interpret and process that information, opinion versus absolute truth. And I say we're in the early days of that experiment.
I say what's fantastic about is that we have a long history of AI. As we're talking, even on the pre call, Google has a lot of the foundational technology of these large language models was developed at Google. And so we have this rich history by which we can build and develop these models. And now I think we're on that part to say, how do consumers, how do users recognize them?
How do they interpret trust? What's too much versus too little? What query are they going to ask [00:41:00] next? And so I like to think of it all as a grand experiment, one that's super exciting, but one that as we go and we look back every three or six months, what we learn and the surprises we see along the way, it's are really going to be enriching in terms of what it looks like next.
And I would argue in five years, it's going to look fundamentally different in terms of how people access and get information, but not in a way that anyone can predict today. I think we learned that based on how people use and how people respond to these queries.
Amith Nagarajan: Yeah, I appreciate you sharing that. And certainly, um, that makes a ton of sense.
And I think that You know, looking at the traditional ad business and how that revenue stream folds into generative results or whatever comes next is an interesting Uh, thing to constantly be balancing between how you're delivering the information that's mission centric versus ensuring that the revenue stream, uh, is not only maintained, but that there's growth opportunity around, which is, of course, important to any company, including a company like Google.
Um, you know, I think that, uh, the [00:42:00] people, people generally who are outside of AI circles don't realize what you just said about Google's history in this field. And I mean, Google from the very beginning is algorithmically in the AI camp and traces its roots all the way back to that. So it's a. Very interesting thing to watch.
Um, you know, ultimately, when you think about like customers in the world today, whoever they are in whatever field, they want to get to the thing that they care about faster and faster and their expectations are higher and higher and their tolerance for friction or any kind of, you know, inconvenience becomes lower.
And so I think That that's a key message. We try to share with this community that they may have a tremendously valuable product or offering. They might do amazing things in this world, but if they make it hard to deal with them, they make it such a high friction experience. If you will, that's very challenging, and I think that's where a lot of people could look at things like the Google business and say, Look, this is an incredibly complex product under the hood, but the surface of it is so So incredibly simple that everybody in the world can use it.
So [00:43:00] there's a lot of lessons to be learned for that, for this audience as well.
Neil Hoyne: The simpler, the better, right? It's that you don't need to know. You don't need to know how the sausage is made all the time. You just need to know that, that it works and that simple. And I think AI is just one where people start poking their head about to say, Oh, What's going on there?
And I'd also point out, this is just something, a little bit of a legacy thing when we talk about product simplicity. Uh, some of OpenAI's, uh, GPT models, in fact, GPT 3, if you go back to Wired Magazine in September of 2020, Wired had a glowing editorial about how this technology would transform everything from writing book reports to doing research to composing music.
And collectively the world didn't get on board. It's kind of like, ah, well that's kind of neat, it's coming. And part of the reason was that it was too technically complex for anybody to understand how it fits into their day or their work. You look at it and you go to their interface, and I used it at that time, and it was like, and you have to specify tokens.
And then you're sitting there and be like, what's a token? I need to learn this first. [00:44:00] And then this brilliance that came in, and I give, I give the OpenAI team credit for this, of putting it into a chatbot. So that way somebody could sit there and you say you type a query like you would send a message to a friend or something on slack and then you see it respond and then suddenly people understood what AI was doing for them.
And so I really look at it as almost the best example of product market fit. You could have had the best product in the world, but until people knew how to use it, how it integrates into their life and to not be intimidated by everything happening under the hood and how many tokens are being used, you're never going to recognize its full potential.
Amith Nagarajan: Makes a lot of sense. And that quote that you are that the statistic that you shared a moment ago about 20 percent if I heard you right, 20 percent of each day's Google queries are net that have never been seen by Google ever in the history of Google. That's a little bit more than 20%.
Neil Hoyne: If you could believe it almost, let's just say one in four queries have never been seen before.
Amith Nagarajan: That's amazing.
Mallory Mejias: A lot more
Neil Hoyne: questions that people have to [00:45:00] ask.
Mallory Mejias: People have
Neil Hoyne: different ways of expressing themselves and typing in queries. And it's, we think about ourselves and our own experience, but you know, there's different audiences that will expect different things from search. You know, my children actually grew up on voice search.
That's their method of interacting. I know other people and researchers that grew up using syntax in specific Boolean fields. If, not, include this, exclude this. That's their language. Even when you look at Google Translate. Google Translate was designed explicitly to work for, this is why we built it for Klingon, was to say, we don't just want to build it for known languages, but as new languages emerge, can we build an understanding of language that helps these models understand what people are saying, even if we haven't seen it before.
Based on what we learned, and so it's just as much as we try to simplify down humanity. It is the most complex thing we'll ever interact with, and I think what we see here with large language models is trying to give people a new medium by which they can express themselves, by which they can [00:46:00] ask these questions.
Mallory Mejias: I'm curious. I'm sure we have some listeners that are curious as well. Are you, Neil, chief measurement strategist at Google using large language models, generative AI every day in your own workflows?
Neil Hoyne: I, I try not to. I try not to. In fact, I, I tell people this, if you look at my cell phone, my cell phone has the fewest amount of apps possible.
If you look at the desk, I have as little technology as possible. I'll actually personally rely on this quote to say when you have an abundance of information, it leads to a scarcity of attention. And so sometimes I've learned over my career that having as much information come at me at a day to day basis is not the best strategy.
And if I assume my counterparts at other companies have the same situation, then there may be a competitive advantage for doing less and stepping away from the data from time to time. I don't ignore it forever. Which means there's certainly times where I can't move myself and I'll have [00:47:00] monitors and data all over the place.
But at least when, when I'm at home, it's just kind of a quiet thing to say, let's think about where things are going as opposed to just being in the middle of it all the time.
Mallory Mejias: Amith, what are your thoughts on that?
Amith Nagarajan: I love it. I think that's a practice that I could improve upon because I certainly have a flurry of information coming at me at almost all times.
But I think you're right. That attention is inversely correlated with the amount of information flowing at you. And you got to pick and choose your battles. You know what I try to do at the beginning of each day is to list very clearly what my priorities For that day and to focus on those before I allow myself to get engulfed by email and all your things on instant messaging and so forth.
But, um, creating focus is the key to achieving outcomes. And I think that's a, what you described is a, is a really good way of describing that. And in the world we live in today, that's, it's, it's acceleration of both information and, and, uh, and other things. So it's a, it's a challenge for sure.
Mallory Mejias: I'll kneel.
We've got to say thank you so much for joining [00:48:00] us today for everything you've shared I feel like we could write a book based on just this podcast episode. Can you let listeners know where to find you?
Neil Hoyne: Just, uh, you can find me on LinkedIn. You can go to neilhoyne. com. N E I L H O Y N E. com. Uh, if you're interested in more about the Lifetime Value, certainly my book is a great place to start.
In fact, uh, the profits from the book are donated to non profits. So, food pantries around the country, which is fantastic. Um, but I always encourage people on social media to reach out just because I love hearing from, from them, from you. about what you took away about the questions you had because that again guides my thinking more than anything I can see in structured data sets.
It's that human connection that's so important and often missing.
Mallory Mejias: We will link those in the show notes. Thanks everybody. Fantastic.
Neil Hoyne: Thank you so much.
Amith Nagarajan: Thanks so much.
We'll catch you in the next episode. Until then, keep learning, keep growing, and keep disrupting.