Andrej Karpathy's Eureka Labs, ChatGPT-4o Mini is Mighty, and Llama 3.1 Unveiled [Sidecar Sync Episode 40]

Timestamps:
00:00 - Introduction
04:23 - Mallory's Experience with AI Webinar for HVAC Companies
06:54 - Andrej Karpathy’s Eureka Labs
09:08 - The Importance of AI in Education for Associations
12:22 - Exploring Khan Academy's Khanmigo and Its Implications
17:39 - Potential Partnerships with Eureka Labs
19:11 - Overview of ChatGPT-4o Mini and Its Capabilities
22:10 - Impact of Cost Reduction in AI Models
24:43 - AI Generated Tagging and Its Applications
30:04 - Meta's Release of Llama 3.1 and Its Significance
34:06 - Capabilities for Synthetic Data Generation
40:24 - Multi-Agentic AI Models and Their Potential
42:55 - Synthetic Data Generation and Its Applications
45:12 - Open Source vs. Closed Source Debate in AI
48:26 - Closing

Summary:

In this week's episode of Sidecar Sync, Amith and Mallory dive into the latest innovations in artificial intelligence and their implications for the association sector. They explore Andrej Karpathy's new venture, Eureka Labs, and its revolutionary AI-native education platform. The discussion then shifts to the release of ChatGPT-4o Mini, a smaller yet powerful model from OpenAI, and Meta's groundbreaking Llama 3.1 models. Listen in as they unpack how these advancements can transform education and professional development within associations and beyond.

Let us know what you think about the podcast! Drop your questions or comments in the Sidecar community.

This episode is brought to you by Sidecar's AI Learning Hub. The AI Learning Hub blends self-paced learning with live expert interaction. It's designed for the busy association or nonprofit professional.

Follow Sidecar on LinkedIn

🛠 AI Tools and Resources Mentioned in This Episode:

• Eureka Labs ➡ https://www.eurekalabs.com 
• ChatGPT-4o Mini ➡ https://www.openai.com 
• Llama 3.1 ➡ https://www.metaai.com 
• Khanmigo ➡ https://www.khanmigo.ai/  
• Andrej Karpathy's YouTube channel ➡ www.youtube.com/@AndrejKarpathy

⚙️ Other Resources from Sidecar: 

• Ascend: Unlocking the Power of AI for Associations E-book 
• Sidecar Blog
• Sidecar Community
• digitalNow Conference
• Upcoming Webinars and Events
• Association AI Mastermind Group

More about Your Hosts:

Amith Nagarajan is the Chairman of Blue Cypress 🔗 https://BlueCypress.io, a family of purpose-driven companies and proud practitioners of Conscious Capitalism. The Blue Cypress companies focus on helping associations, non-profits, and other purpose-driven organizations achieve long-term success. Amith is also an active early-stage investor in B2B SaaS companies. He’s had the good fortune of nearly three decades of success as an entrepreneur and enjoys helping others in their journey. Follow Amith on LinkedIn.

Mallory Mejias is the Manager at Sidecar, and she's passionate about creating opportunities for association professionals to learn, grow, and better serve their members using artificial intelligence. She enjoys blending creativity and innovation to produce fresh, meaningful content for the association space. Follow Mallory on LinkedIn.

Read the Transcript

Amith Nagarajan: Greetings, everyone, and welcome back to another episode of the Sidecar Sync. We are excited to be back with you today with some crazy, interesting news all about artificial intelligence. And we're going to tell you all about why these particular topics matter so much to the association sector.

Amith Nagarajan: My name is Amith Nagarajan.

Mallory Mejias: and my name is Mallory Mejias.

Amith Nagarajan: And we are your hosts. Before we dive into all of the fun and crazy things that have been happening in the world of AI and how they apply to associations, we're going to take a moment for a quick word from our sponsor.

Mallory Mejias: Amme. We are on episode 40 of the Sidecar Sync. I can barely believe that you and I have been meeting every single week to talk about ai, at least for an hour, for 40 weeks. What do you think about that?

Amith Nagarajan: I think it's been a lot of fun. It's, it's, it's crazy how quickly it's gone by too. Cause it does seem like we just started it. So 40 episodes, it's a good start.

Mallory Mejias: It is a great start. We're also seeing a lot new, a lot of new listeners, or should I say, viewers joining us on YouTube. So if you are listening on your favorite podcasting platform, you should also subscribe to our YouTube channel and join us there.

Amith Nagarajan: Yeah. YouTube is a fun medium. And I think over time we will experiment with some additional things we can do for our YouTube viewers.

Mallory Mejias: For sure. It also means you and I have to look just a bit nicer. Remembering that we do have a YouTube audience as well. Amith yesterday, I told you a little bit about this. I got the chance to co host an intro to AI webinar for an association of HVAC companies. Um, that's something that we, we offer at Sidecar and it was a really great session, it was kind of a, A mock up of our regular Intro to AI webinar that we do for associations and non profits every month.

Mallory Mejias: But this time, all the examples and use cases were tailored to HVAC companies. So, it was definitely a test of my AI skills and knowledge. We mostly used the same tools. We showed ChatGPT, Claude, Gemini, and Midjourney. But, I will say the types of things we were doing with these tools were a little different.

Amith Nagarajan: Yeah, it makes sense. You know, that last mile problem, as they'd say in telecom or it's like that final bit of contextualization where you make it exactly what these people think about in their terminology with their examples. It makes such a big difference because you're taking away kind of that intellectual gap that someone has to translate a generalized concept or a similar concept that's in another space into their world.

Amith Nagarajan: So, you You know, if you're talking to accountants and using examples from lawyers, sure, the accountants can pretty much understand what you're talking about, but they have to translate it in their minds. So you've done that work, and I'm sure that it was well received that way. That's that's really cool.

Mallory Mejias: Yeah, we've gotten some good feedback so far. For me particularly, I, I focus on midjourney normally in that webinar, and typically with Sidecar, we are creating kind of cartoon images, if you all have seen our branding before, but HVAC companies are creating images of real people, so it was definitely a challenge for me to learn how to create more photorealistic images in mid journey, but happy to say I've added that skill to my toolkit.

Amith Nagarajan: Yeah, that's really cool. Well, and I'm excited about some of the content around video images and all these new AI tools that we're including in the forthcoming edition of Ascend, which, uh, the team here has been hard at work at for a number of months. And, uh, I think we're probably going to have it out there on Amazon in the next two or three weeks.

Amith Nagarajan: So can't wait to see that drop.

Mallory Mejias: Absolutely. Everybody stay tuned for Acsend second edition. Today we've got some exciting topics lined up. We're going to be talking about Andrej Karpathy's new startup, Eureka Labs. We're going to be talking about GPT 4. 0 Mini. And the release of another family of models, I should say, Lama 3. 1. So it has been a really exciting couple weeks in the world of AI, starting with Andrej If you don't know him, he's a Slovak Canadian computer scientist renowned for his contributions to artificial intelligence, particularly in deep learning and computer vision.

Mallory Mejias: He was a founding member of OpenAI, the company behind ChatGPT, and he became Senior Director of AI at Tesla. And now he's launched a new venture called EurekaLabs. So what is it? EurekaLabs is described as an AI native education platform that seeks to create a new kind of school. EurekaLabs envisions a learning environment where AI and human teachers collaborate seamlessly, allowing anyone to learn anything efficiently.

Mallory Mejias: The platform will use AI teaching assistants to support and scale the efforts of human teachers. These AI assistants will guide students through expertly designed course materials, making learning more interactive and personalized, and it's teacher plus AI symbiosis is expected to expand both the reach and depth of education, allowing more people to learn a wider array of subjects more effectively.

Mallory Mejias: Now, its inaugural offering is an AI course called LLM 101 N. It's an undergraduate level class designed to guide students through the process of training their own AI, similar to a scaled down version of the AI teaching assistant itself. The course is available online with plans to run both digital and physical cohorts and overall, Eureka Labs is targeting the education sector, particularly focusing on digital learning and AI education platform is expected to appeal to universities, online learning platforms, tech enthusiasts, eager to explore AI. And I'm thinking even associations, Amith. So what were your initial thoughts when you saw this release?

Amith Nagarajan: First of all, for those in the world of AI, Andrej Karpathy is one of these people that you just, you follow what he talks about and you listen very closely because he's a brilliant mind. And, uh, he's actually seems to be historically pretty generous with his time in regards to sharing ideas and, you know, kind of open.

Amith Nagarajan: Uh, communication with the community. In fact, he has a series of YouTube videos that are really great, um, that explain lower level concepts than most folks want to dig into. But if you want to dig deeper on some of the fundamentals of AI and deep learning, he has some great stuff on ideas like Backpropagation and other other things that are really interesting, um, that I'd encourage people who want to go deeper to check him.

Amith Nagarajan: I just Google him on YouTube and we'll include the links in the in the show notes. But, um, so anything he does, I think it's worth noting. And then specifically what caught my attention about this is it's in the bullseye of what I think associations need to pay attention to is -- how do you deliver professional education or education of any kind in a in a better way, right? How do you do it differently? How do you take that next leap forward? Um, and what they're talking about at Eureka Labs seems to be similar to what the Khan Academy has done with their Khanmigo. which we've talked about in prior episodes of this podcast.

Amith Nagarajan: Uh, and similar to the vision that we laid out for what associations should be doing from an education perspective in this forthcoming edition of Ascend, uh, where we talk about personalization, we talk about tutoring, we talk about the whole idea of guiding the learning journey based upon the individual.

Amith Nagarajan: Um, and so it seems to touch on similar themes. You know, because Karpathy is a, you know, fundamental research scientist, I'm thinking they're going to be creating some new innovations. That are different than just applying large language models, you know, like Khanmigo is really cool, but they've essentially just taken GPT 4 and tailored it to work in the context of what Khanmigo needs to do.

Amith Nagarajan: So I'm very curious as they share more, if there's some fundamental improvements in the models that they're making that make this a better, you know, better fundamental technology platform for education. I think people need to pay attention really, really closely to this because it's at the heart of what many associations really consider a really key part of their value prop is the way they deliver education.

Mallory Mejias: If you go to the Eureka Labs website right now, you see kind of a big paragraph of text and then a link to that course that I mentioned in the intro for this topic, but there's not a ton of other information just yet as I guess they're kind of building out their, their products and offerings. But, Something that stood out to me was the idea of the human plus the assistant.

Mallory Mejias: I feel like a lot of these tools that we're looking at these days tend to kind of lead with this can do it all for you, but just the fact that they're leading with this AI plus human symbiosis, I think that's really powerful.

Amith Nagarajan: Yeah, I agree. And I think it touches on the idea of how can the humans involved in education. spend more time on the human part of education as opposed to the, you know, things that are automated automation capable. And so that's exciting. I also think there's, there's not enough humans to teach. There's not enough humans to tutor.

Amith Nagarajan: And so if we can create platforms that democratize access to the best possible, you know, quality of education in any topic, in any domain, At any location at any time for anyone, essentially for free, which is, of course, that's the vision of the Khan Academy and many other institutions are pursuing a similar vision.

Amith Nagarajan: That's exciting because the eight billion people online have access to the best tutoring, the best education, and that's really what AI promises to bring is to be able to You know personalize and to deliver meaningful education if you think about your own experiences in life Um, this has certainly been my experience in my journey is that you know Sometimes you encounter a teacher whoever that may be it might be in a formal setting or an informal setting And that person really makes a big impact on your life because they inspire you they p erhaps get you interested in a subject that you may not have thought you were good at or may not have thought you were interested in And a lot of that isn't because of the actual content, but it's the way they connected with you the way they contextualized it Um the way they just you know related to you And so I think there's more opportunity for that with the humans involved But I also think that there's not enough humans who are good at that to do that at scale and have that kind of impact f or every person on the planet.

Amith Nagarajan: So if AI can play a role to approximate that or be a facsimile of that, that's exciting.

Mallory Mejias: I'm thinking in terms of prompt engineering and how we always say it's good to tell the AI to adopt a persona. So I'm going to tell you that, Amith, for this next question. So if you were the director of education at an association or even someone who worked at a company who was in charge of education, what would you be doing right now to prepare for a potential product like this?

Amith Nagarajan: Well if I was director of education at an association and I wasn't super familiar with what we were just talking about, I would do everything I could to learn about the fundamental capabilities. So the first thing I'd recommend is go to the Khan Academy. It's just Khan Academy dot org. We'll include that in the show notes.

Amith Nagarajan: There's free access to Khanmigo now available. It used to be like 10 bucks a month or something, but through a grant from Microsoft, uh, they made it available for free to every K through 12 educator in North America. And I think you can get free trial for anyone else as well, but if you have to pay 10 bucks, pay 10 bucks.

Amith Nagarajan: But like the idea is go in there and play with this thing. Uh, what they've done is, uh, I think a pretty good preview of what might be possible in your own organization. So Khanmigo is pretty interesting because. Uh, Khanmigo learns the student and Khanmigo is able to guide the student in kind of a Socratic way as opposed to just providing the answer.

Amith Nagarajan: You think about like, you know, your experience with Sonnet or with Chat GPT, you say, Hey, what's the answer to this question? And it gives you the answer. Well, but is that really the best way to educate someone? Are you really teaching them anything? Sure, it's solving your problem, but then you're moving on to the next thing.

Amith Nagarajan: But if I go to Khanmigo and I say, I don't know how to solve this problem, Khanmigo is not just going to give me the answer. It's going to try to guide me to the right answer by helping me learn how to solve the problem. Um, so that's the difference. That's a significantly different approach than what a generic large language model would do.

Amith Nagarajan: I suspect that has something to do with the way they're training models at Eureka labs as well as they're optimizing for this use case. Whereas Khanmigo is built on top of a generic language model. If Carpathian team are building a model that's good at this at the fundamental level, where that kind of.

Amith Nagarajan: Mindset of being a good educator is built into the model itself. That could be a new level of capability. I

Mallory Mejias: To take that preparation step a bit further. Do you think the director of education in this imaginary scenario should be taking a catalog of kind of all of their educational offerings and seeing which might be worth training a new AI model on? Is there anything else kind of more tangible that they could be doing in the next six months?

Amith Nagarajan: I think beyond getting that basic sense of what the A I technology currently can do, which is important. And I suspect most education directors and other similar roles and associations haven't really taken a deep dive. So beyond that, I do think taking an inventory of your current offerings and thinking about where you can apply this would be good.

Amith Nagarajan: Um, you know, rather than thinking about how to apply it to everything, it might make sense to say, well, could we potentially apply it to just one of our courses? Could we apply it to something really simple? That would be fairly easy on a comparative basis to the entirety of what we offer because many associations have quite comprehensive offerings in this whole world of education.

Amith Nagarajan: So I think obviously there's also different modalities. So you have live instruction, you have, you know, web based instruction that's synchronous, you have asynchronous courses in an LMS, you have a lot of different modalities and the way I can help will be potentially different in each of these scenarios.

Amith Nagarajan: So I think it is important to start thinking about that and, um. You know, maybe run a small experiment. See if you can maybe use one of these new tools. Um, and you know, the thing you could do is take some of the content from one of your courses, upload it into one of these AI tools. And again, you have to be thoughtful about privacy of your data and things like that, uh, making sure you're comfortable with the vendor that you're working with, but you could take, you know, your content to something like a Claude sonnet, you could take it to chat GPT, uh, Gemini, you And give it to the model and then ask questions, um, in the context of a, of a good prompt to your earlier point about prompt engineering and say, Hey, this is what you are doing. You are the tutor on this subject matter. And then see, see what the interactions are like. Um, that will give you a basic idea, you know, cause that, that AI will then have familiarity with your content.

Mallory Mejias: Um, That's a great idea. Anything else you want to cover here, Amith? Um, Um,

Amith Nagarajan: My main thought here is that this announcement from Eureka Labs is a really important signal that there is mainstream fundamental research interest in this topic. Not that this is the first time this has been stated that education is one of the obvious bullseye targets for AI to make a big difference. There's a lot of excitment in the field But if you're an association thinking that, you know, you're doing just fine and you're kind of, you know, delivering value to your professionals and all of that, you know, maybe you are, but. You know, you've got to look at this technology and understand that it fundamentally changes the expectations of the consumer.

Amith Nagarajan: You know, people are going to be receiving education through AI enabled platforms and really AI native modalities, uh, in the very near future. Some people already are. And if they come to your association, they're receiving a very static one size fits all type of experience. That's just, you know, the tip, the things that up until recently might've been considered.

Amith Nagarajan: State of the art with a really nice LMS on the web and on mobile device. It's not going to be good enough for long. So you have to look at it both in terms of that risk, which is that friction you're creating if you don't enable your content with this technology in the, in the near future, but also the opportunity is the flip side is because a lot of associations think about their you know, the center of their universe, the nucleus of their audience is where they look first. Well, let's say, Hey, we deal with this medical subspecialty or this other area, this very narrow vertical, uh, but their content may have applicability in other contexts as well. And so AI potentially could be helpful in broadening their market appeal.

Amith Nagarajan: So I think it's both a risk mitigation thing. But more importantly, like an opportunity to deliver, you know, much greater value to their current audience and potentially expand the audience. So I think that's the thing people have to look at and say, well, if, if major players in Silicon Valley like this are really putting their focus on education associations, hopefully we'll see that as a great opportunity.

Amith Nagarajan: I think maybe Eureka Labs would be a great partner for some associations. I don't know. I don't know what their business model is. We're certainly going to be investigating it because I think, um, whatever they're doing should be. You know, it's it's just gonna be interesting, but there will be many companies like this, right?

Amith Nagarajan: There are others out there doing this. It's time to take a look

Mallory Mejias: Moving on to topic two for today, GPT 4. 0 mini on July 18th, open AI, the company behind chat gpt released GPT 4. 0 mini, and we covered the release of GPT 4. 0 on the podcast earlier this year. And the new release is pretty much what it sounds like a small model that is significantly smarter, cheaper, and just as fast as GPT. 3. 5 turbo. So diving into a few key features of GPT 4.0 Mini, this model is over 60 percent cheaper than GPT 3. 5 turbo. Despite being a smaller model, GPT 4.0 Mini outperforms GPT 3. 5 turbo in various benchmarks, including textual intelligence and multimodal reasoning. It supports text and vision inputs and outputs, with plans to include audio and video in the future.

Mallory Mejias: And it can handle up to 128, 000 tokens, which is useful for processing large documents or maintaining long conversations. And just as a heads up, one token is about three fourths of a word. Now, Amith , we have revisited this idea over and over the idea of smaller, smarter models being released. Um, we've seen it across companies at this point in the AI space.

Mallory Mejias: Why do you think that is?

Amith Nagarajan: Well, I mean, fundamentally, accessibility, performance, you know, cost, it's, it's making it possible to build applications at scale that perform quickly, inexpensively, and are usable, you know, in, in so many other domains. So, uh, GPT 4. 0 mini, I think is one of these things where people are gonna look back at it and say, Yeah, that really opened up the doors to a lot of new applications.

Amith Nagarajan: So GPT three, five turbo was a workhorse for a long time. Uh, it's really old at this point, you know, it's like 18 months old, which I think in the world of AI means it's, you know, it's really ancient, but you know, it's, it's actually still a pretty good model, but compared to GPT four Oh mini, it doesn't even, it's not even comparable.

Amith Nagarajan: Um, so 4.0 mini. is almost as good as four. Oh, in a lot of testing and for lots of use cases. So, um, you now have a super inexpensive and quite quick model to weave into business applications. Of course, this is based on, you know, I think open A I would probably pursue this path generally because they realized the importance of it.

Amith Nagarajan: But also the competitive pressure is substantial. You know, have different levels of competition. You have different sizes of other models from other companies, you know, Claude being probably their main commercial competitor, um, with the Sonnet and the Haiku series of models, which are, I should say, flavors of the, um, Claude model are the kind of medium and small models, respectively, Sonnet and Haiku, that is, and that's just their branding, but the idea is that those smaller, faster models are quite performant and they're capable of doing a lot of really cool things.

Amith Nagarajan: You see the same thing with Mistral, uh, producing small models that are really performant and really inexpensive. So, it's gonna be a major area where people play. You know, the capabilities of current state models are so much greater than the way people are using them that even these small models are way more capable than what most people realize.

Amith Nagarajan: So, it's exciting because it basically means the stuff is close to free.

Mallory Mejias: I think some people probably hear us talk about this idea of, of things getting cheaper. And maybe they say, well, you know, chat, GPT is still 20 bucks a month for me to use. Maybe not thinking about the full context of what this means. And one, uh, someone in our family of companies, Dray McFarlane, posted this in our AI channel, which is where we post a lot of updates and new tools and things like that with one another.

Mallory Mejias: And this was a quote from OpenAI, the cost per token of GPT 4.0 mini has dropped by 99%. Since text DaVinci three, a less capable model introduced in 2022. And I think the point that Dray made was that he's one of the creators of Betty bot that they originally trained Betty bot on that text DaVinci three model.

Mallory Mejias: Um, and we're seeing a 99 percent reduction in cost in, what did you say? 18 months, or maybe it was a little longer than that. So can you talk a little bit about how you've seen, uh, this reduction in costs, even affect the products that we've spun up in the blue Cypress family? Okay.

Amith Nagarajan: Sure. Well, Betty's a great example. When we were first starting work on Betty in 2022, right around the time chat GPT, you know, had its moment in the public consciousness. Um, we were looking at the cost of inference, right? The AI model cost as a pretty major component of what, uh, We'd have to figure out what the economics were.

Amith Nagarajan: And the original model for Betty was to actually pass that along to the customer. We had a license fee for Betty, which would be for the Betty software and then customers to directly pay for their token usage. And we thought that would be, you know, a fairly significant impediment to adoption because There was a variable there that people really couldn't predict that well.

Amith Nagarajan: Uh, and in the first six months of Betty's adoption in early 23, it turned out to be a factor and then in kind of the, I think it was the March, April timeframe of 2023, if memory serves me, GPT four came out. And then GPT 3. 5 became much less expensive. And so between the two, we were able to, you know, significantly lower the cost, but it was still a meaningful component of a Betty investment and deployment.

Amith Nagarajan: And since then, what you described is accurate. So now there really isn't much incremental cost for inference on top of the base license. So that's really exciting in terms of adoption of tools like Betty. Um, you know, skip, which is our A. I. Uh, business analyst and A. I. Reporting analyst that that product also is the same thing.

Amith Nagarajan: And in fact, skip is a very, very heavy use of user of advanced A. I. Models in order to do the work that it does for code generation and for, you know, true, like N. B. A. Level data analytics. Um, So having lower cost models that are quite capable in a blend of models where you can, let's just say, for example, use something like, um, GPT 4.0, for some things, but then use 4.0 mini for other things, um, really makes those types of products more accessible. So, and that's applicable to all sorts of problems in the association domain. So for example, if you say, Hey, I have a million documents that I want to process in some way through a language model.

Amith Nagarajan: Uh, for example, I want to automatically tag every article we've ever published in our journal and our journal goes back to 1920 or something like that. And we have a million articles, um, doing that with GPT 4.0 when it First came out probably was actually already pretty reasonable, but GPT 4.0 mini makes that something we could say, yeah, I can do that for a few thousand dollars versus hundreds of thousands of dollars.

Amith Nagarajan: So the applications become more palatable when you have access to really high quality models at a really low cost. Plus of course the, the speed is great too.

Mallory Mejias: Okay, kind of a counterpoint here. Then back in 2022, if an association wanted to do AI generated tagging, do you think it would have been best case for them to wait at that point?

Amith Nagarajan: Not necessarily. I would say it would be more of this thing where it's still a somewhat scarce resource. So they might apply it to just new content that they're publishing. They might apply it to, you know, uh, particular types of content that are really the most important content elements. So you'd be kind of looking at it as a gated resource or a constrained resource.

Amith Nagarajan: Um, so you'd say, Hey, like I only have a budget of, you know, X dollars per month that I want to spend on this. So therefore you limit it to what you can, what you can do with that budget. And now the budget just goes way, way further. You know, 99 percent is kind of that inverse of going to infinity on capability.

Amith Nagarajan: Right? So, um, it's, it's a really exciting thing. And I know we're going to talk about Llama 3. 1 in a minute, so I won't go there, but just the idea of small models being super fast and super inexpensive. Um, has, it just makes it possible to do more and to be less concerned with cost. You know, we don't think about internet bandwidth and delivery of education or delivery of video anymore.

Amith Nagarajan: It's just kind of assume that there's high speed bandwidth available in most places and the incremental cost is pretty close to zero. You know, I make a point when I do executive briefings. For association leadership teams. And I do those fairly regularly for folks who ask for them. Um, I'll deliver an hour of education and I'll talk a lot about kind of this, this curve of what's happened, these doublings.

Amith Nagarajan: And part of what I talk about is how that's also reduced the cost of what was previously scarce, expensive, out of reach resources. And the point that I usually make, because it's usually over zoom video or teams video is like, we're having a high bandwidth, you know, really high quality video conference with 10, 20, 40, a hundred people, whatever it is, uh, and we didn't think about the cost.

Amith Nagarajan: No one thought about like, oh, should we do this AI executive briefing with Amith? Um, you know, because it's going to cost us a thousand dollars of video conferencing bills. And not that long ago, you would have been thinking very, very carefully about that before you used video conferencing. It was a scarce and expensive resource.

Amith Nagarajan: Now we use it for everything because it's effectively free. Same thing is happening here with AI.

Mallory Mejias: I know we talk about a lot on the pod that one day, maybe in the near future, we'll just use the AI. We won't necessarily know the model that's behind it. We won't know how big it is. It'll just be there for whatever purpose we need it. But at this point, if you go into chat GPT, you can actually toggle between GPT 4, which they're calling their legacy model, GPT 4 0 or Omni, and then GPT 4 mini.

Mallory Mejias: I'm wondering, you, Amith, are you ever, you know, Do you think you're opting for many at this point? I know it was just released, or do you find yourself just sticking with GPT 4.0?

Amith Nagarajan: I've been using 4.0. I've played been using 4.0 mini just for kicks see what it was like in the playground, but I haven't really done anything with it. Um, you know, for, for the most part, 4. 0 for me, you know, we have the, um, I forget what it's called. It's the, the next level up from the individual subscriptions that I think it's like the team's plan at Blue Cypress that we pay for.

Amith Nagarajan: So it's a little bit more expensive and we get really good performance out of that. So when I'm using it, it's great. I just haven't bothered. I'm almost at the point, honestly, where what you described, which is the future where people don't really care that much about the model. Like when I'm, when I have my consumer hat on and I'm, you know, working on business problems or I'm talking to the models about marketing, or I'm just like brainstorming ideas, a lot of times it's over voice too, and I'm just walking around on my phone and, you know, talking, talking to my phone.

Amith Nagarajan: Um, I don't really care which model it is. These models are all really good. Um, so if I'm doing something like really deep, like at a technical level, I might want to make sure I'm using the best and latest model. Um, but oftentimes I just really don't care that much cause all the models are, are so good at this point.

Mallory Mejias: Well, that is a good segue into our third topic for today, which is llama 3. 1. And I think Amith you just sent me this topic yesterday, a few days ago, right?

Amith Nagarajan: Came out yesterday morning.

Mallory Mejias: Exactly. So you can be assured listening to the Sidecar Sync podcast, you're getting the latest AI news. Meta recently announced the release of LLAMA 3. 1, the latest and most advanced version of its open source, large language model family. LLAMA 3. 1 is available in three sizes, eight billion or eight B. 70 billion or 70B and 405 billion or 405B parameters. The 405B model is the largest open source AI model to date, designed to rival top proprietary models like OpenAI's GPT 4 and Anthropic's Claude 3. 5 Sonnet. Llama 3. 1 introduces capabilities for synthetic data generation and model distillation, which I want to talk about in just a bit, enabling developers to use its outputs to improve other models, which is a significant step for open source AI. The meta website says Quote, until today, open source language models have mostly trailed behind their closed counterparts when it comes to capabilities and performance.

Mallory Mejias: Now we're ushering in a new era with open source leading the way. End quote. Listeners, you can try out Llama 3. 1 right now by going to meta. ai and you actually don't even have to log in. I didn't test it out much, but I dabbled with it just a bit before this call. Amith, we've seen lots of new model releases lately.

Mallory Mejias: Does anything stand out for you with the llama release?

Amith Nagarajan: So the Llama 3. 1 release is on the heels of the 3. 0 release that came out in, uh, the April timeframe, I believe, April, May timeframe. And 3. 0 was a really good release. And they had mentioned that they have a bigger model that they were still training and that they were going to update the three. Series models with 1, and even the two smaller versions, the really small one, the 8 billion and the medium sized one, that's 70 have both been updated and they both are better than the 3. 0 versions, which is really cool. Uh, and then the 405 billion parameter model. First of all, this model is considerably smaller than GPT 4. 0. It's smaller than what we believe the Claude 3.5 Opus is. Uh, so it's a little bit smaller size wise, but. Yeah. It's benchmarking shows that it's essentially at parity with GPT 4.

Amith Nagarajan: Oh, um, and with, uh, Claude 3.5 Sonnet, um, in pretty much all categories. So in fact, it's a little bit better in some, so it's kind of like saying, Hey, uh, do you have a Ferrari or a Lamborghini? You know, one goes 210 miles an hour. The other one goes 205 you know, they're both way faster than you're probably ever going to drive.

Amith Nagarajan: So you're good. Uh, and I think we're getting to that point for the current sets of use cases that we have. So what is notable about the bigger model is it is on the order of these proprietary closed models. Um, and it's totally open source and you can deploy it anywhere. You can deploy it on Azure and AWS.

Amith Nagarajan: You can deploy it in an environment you have complete control over. That means that if you have a highly sensitive application. We need to maintain 100 percent control over your data, never send it to any third party vendor, but you also need to have frontier level A. I. Capability from the smartest and best model.

Amith Nagarajan: You don't have an option. You have the ability to deploy Llama 3.1 405 B. In a private cloud type environment, or even on physical hardware, if you wanted to, you'd need some pretty beefy hardware to run a model that size. But you can do that. And that becomes very affordable for for enterprises that are interested in that kind of private deployment.

Amith Nagarajan: So if you're in the health care sector, or if you're in a particular field where, for whatever reason, your data sensitivity is so high, That you really need to focus on control. This is a whole new cap capability that didn't exist until now, because the earlier open source models, including llamas earlier and current small models are not at the level of four or five B.

Amith Nagarajan: So four or five B is a big deal because it gives you essentially the capabilities of the biggest, most powerful models, but in a totally free open source format that can be deployed anywhere. So to me, that's a significant shift. It's also just great competitively because what's going to happen here is, um, companies like Mistral and many others are going to fast follow with all sorts of derivative products that are new models based on the llama architecture, or in some cases are parallel universes to the llama architecture.

Amith Nagarajan: Like Mistral kind of is, um, and you're gonna see a lot of innovation and lots of innovation leads to lots of growth, lower costs, better capabilities. Um, and you don't get that quite as directly from the closed models coming out with new, you know, capabilities. The last thing I'll quickly mention is that I'm very excited about this product because GROQ, G R O Q, um, That company we've talked about here that has the language processing units.

Amith Nagarajan: The LPUs has 10 times the speed for runtime or inference for AI models than the GPU based approaches. They have just crazy fast inference and they are a launch partner with Meta. On the llama 3. 1 family of models, uh, or what they call the herd of models, actually, which I think is kind of cool. So, um, the llama 3. 1 herd all runs on GROQ. So if you're building an application, you can inference your app on GROQ. What that means is you're going to have really this state of the art AI capability that's way faster than either Claude or chat GPT. So. That's really exciting too. So for applications as sophisticated as something like skip that we were talking about earlier, um, that's a big deal, so you can be assured that like our teams are all over this stuff, experimenting with it, but I think there's, there's so many opportunities with this.

Mallory Mejias: We do have a previous episode on GROQ as well, and I will reiterate what Amith just said. It is fast, and that was just a few months ago. I don't know if it's any faster. In my mind, right when I see Claude or ChatGPT work, I'm like, Ah, it's quick enough. I don't really need that to be faster. But then you see the examples of a GROQ chip, and you realize it's, you know, near instant, your responses.

Mallory Mejias: So, recommend checking out that episode if that's of interest to you. Amith, I'm not sure how to phrase this next question, but I feel like some of our listeners might have it too. So I'm going to do my best, but because these models are open source, does that mean this is kind of the new foundation for AI models?

Mallory Mejias: In the sense that if someone else out there wanted to create a brand new AI model, could they use, uh, the Llama 3.1 family as kind of their starting foundation and build on top of that? Is that how that works?

Amith Nagarajan: Sure, yeah, there's a lot of ways you can do that. So the actual software code, the interesting thing about these AI models is if you look at the source code, it's very limited. It's like a thousand, 2000 lines of code, and it's not particularly interesting and they all work in, in similar ways. Um, it's the open weights, which is essentially, when we talk about these parameters, uh, which is the output of this pre training process that you do when you're taking massive amounts of data and spending months of time and millions or hundreds of millions of dollars, these Um, on, you know, GPU farms and clusters that are generating the models, that process results in these things called weights and those weights are open.

Amith Nagarajan: If you have the model itself, which is that small amount of source code and you have the weights, you can not only run it, but you can do what you're referring to, which is you can create, um, Various versions of it. You can further train it through fine tuning. You can actually quantize the model to shrink it.

Amith Nagarajan: Uh, you can do all sorts of things in order to build additional capabilities on top of it. Um, some of the fine tuning might be to create flavors of the model that are particularly good at certain things. So for example, in the prior iterations of llama, there was something called code llama that actually meta also released, which was particularly good at code generation.

Amith Nagarajan: I'm sure there'll be a code llama for 3. 1 as well. Um, but people created versions of llama that did all sorts of things. So you're gonna see the same thing happen. It's gonna be an explosion of innovation. Uh, it's one of the reasons Linux, you know, if you think about Linux is kind of the standard infrastructure for the web and for the internet that didn't happen overnight, it used to be that back in the day, you know, the closed source Unix systems were far better than Linux when Linux first came out.

Amith Nagarajan: But because of the community behind Linux, it's become so much more robust, more secure, more reliable, and, you know, more capable. Um, that's why it's become the standards. It's become the better operating system for enterprise scale, you know, everything basically, and that's really the strategy behind Llama, you know, Zuckerberg talked about that in his release, uh, comments yesterday.

Amith Nagarajan: Um, so, you know, it's, they're not, you know, This is not something that Meta is doing out of the goodness of their hearts. This is because it's good for them. Um, having open source means if Llama, if Llama is successful at becoming kind of the gold standard open source AI model, then there's going to be millions and millions of developers working on it, tons of companies investing in it, uh, and that helps them because they're, all of their, you know, products they make money off of are based on Lama as well, or will be going forward.

Mallory Mejias: So pretty much everyone on earth just got access to a frontier AI model to do with which what they please, right?

Amith Nagarajan: Yep. Yeah. And the small model you can run, you know, almost anywhere. The 8 billion, like out of the three, I think, you know, look at the, look at the performance stats, the benchmarks of the Lama three, 3. 1, 8 billion. Actually, I think on GROQ, they call it Lama, Lama 3. 1 instant is what they call it, which basically is, uh, and you, it's so small and fast and efficient that you can probably run it on a phone.

Amith Nagarajan: I'm guessing that they're going to package it into future versions of all sorts of devices. Um, so to me, that gets really, really exciting when you can have on device AI. We talked about that a little bit with Apple in the past. We've talked about that with, um, you know, Microsoft's fi models, which are also all really small, all of this stuff is going in the same direction, right?

Amith Nagarajan: It's basically to make it become invisible where the technology is just assumed. It's part of every application on every device everywhere. And we're going to be there in the next year or two. in terms of that basic capability, that basic assumption. Um, and then of course, you know, the question is, is like, well, what, where do you keep pushing in terms of new capabilities?

Amith Nagarajan: What can we not currently do? Um, you know, one thing I didn't mention earlier, we were talking about 4.0 mini and it's relevant to that topic and also the smaller llama 3.1 models is the smaller models are so cheap and so fast That you can actually use them in a way that you wouldn't have been able to use models up until recently, which is you can use them in a multi agentic style.

Amith Nagarajan: What that means basically is you can sometimes in parallel, go out to the model and ask it to do five things at the same time in parallel, bring back the results, compare them, analyze them, compress them, and then reprocess them. So, you know, there's this idea of zero shot, which is just go to the model with the prompts and hope you get something good.

Amith Nagarajan: And then there's these multi shot, five shot, there's a lot of the different benchmarks actually have. Criteria for whether it's zero shot, two shot, five shot, et cetera. Um, and what multi agent solutions do is they, they take the approach of end shots where they're basically going back to the model over and over.

Amith Nagarajan: So like, for example, something like skip, that's what its internal architecture is doing. Sometimes skip, we'll go to multiple models in parallel and get those models to do some piece of the work. Um, and then compare the results and pick the best version and then iterate from there. And that would have been both.

Amith Nagarajan: Way too expensive and way too slow up until recently, right? So it's, you know, when you have unlimited bandwidth, you start doing real time video and virtual reality. Whereas a few years ago, you know, you wouldn't have done that. You would have been real happy with just high quality phone calls with Like when Skype first came out, you know, 15 years ago or whatever.

Amith Nagarajan: So, um, it's the same thing with this. It gives you new applications and capabilities. So even until the frontier models are better reasoning, for example, you can get really good reasoning out of agents because the agents do what I described of chaining together prompts and doing multiple shots and stuff like that.

Amith Nagarajan: Um, so these innovations, my point is, is that they make those kinds of applications more possible and more affordable.

Mallory Mejias: I mentioned we would touch on this so I want to make sure that we do that LLAMA 3. 1 introduces the capability for synthetic data generation and honestly you brought this up a few times on the pod it might be worth having a topic around it one day but can you explain what this means?

Amith Nagarajan: Sure. Well, synthetic data generation, it actually is kind of what it sounds like, but it's basically using a language model to generate content for you. So let's say that I want it to create my own model and I needed a lot of really high quality training data to build a model. Um, let's say specific for my association.

Amith Nagarajan: I want it to build a model that was used for some purpose. It doesn't even matter what it is, but, um, the idea is, is I need a lot of data for that. And maybe I have some example data. You know, maybe I have ten thousand or twenty thousand pieces of data that are good starting points, but maybe I need 5 million pieces of content. And so I can use by license and by capability, and I'll explain what that means in a second, but by both license and by capability, I can use the llama 3. 1 family of models to generate this new content by prompting the language model essentially saying, Hey, here's several examples. I need you to produce more examples and then writing a program that essentially keeps prompting the model over and over and over and over again, asking it for more results.

Amith Nagarajan: And then I save those results and I have my synthetic data. Um, so by license and by capability, what I mean by that is up until recently, most of these large models said you're, you're not allowed to use them for synthetic data generation because what they're essentially trying to do is protect their moat.

Amith Nagarajan: So like. You know, open AI specifically, their terms of use do not allow you to use GPT four to generate training data, because if they did, then you could use a much lesser model, fine tune it, or even pre train a new model with stump something coming out of GPT four. So it's kind of a mode protection mechanism that the license hasn't allowed for that.

Amith Nagarajan: And then by capability is. You know, if, even if your license allows you to do it, if the output isn't great, um, then there's no value to it. But these models are so sophisticated that they're quite extraordinary actually developing synthetic data. Um, there's still a lot of questions to be answered about synthetic data in terms of the efficacy of training models that are based on them, but all of the initial indicators from the research that's been published and the models, uh, like the Microsoft 5.3 model, a lot of the work that Mistral is doing, um, uses data like high, high quality synthetic data.

Amith Nagarajan: Uh, it's going to be an explosion, right? Like this Cambrian explosion of models we're already seeing. You're going to see more of it because of this particular decision.

Mallory Mejias: Do you think, or do you know, if OpenAI and Anthropic are using, uh, synthetic data generation to train their models from their own models? And

Amith Nagarajan: I don't know if either of them have been particularly open about what their training data sets are. I'd be shocked if they're not because both of them have had frontier capabilities for a long time that have been synthetic data capable So, I would be fairly confident in saying that both of those companies have been heavily utilizing synthetic data, but I don’t really know the answer to that definitively It'd be hard to imagine that they wouldn't because it'd be an advantage that they'd be You know giving up for no reason I think

Mallory Mejias: then my last question for you, Amith. We had a whole, I think it was a really early episode that we did dedicated to kind of this open source. It's a closed source versus closed source debate. Um, Claude and OpenAI's GPT models are closed source. Google Gemini is closed source. But Google Gemma is actually open source.

Mallory Mejias: And then like we just talked about, Llama is open source. Um, can you kind of just give a high level, your take on what you think is most important with either, what you should keep an eye on, um, if you're a business leader listening to this podcast?

Amith Nagarajan: Well, I mean, there's there's a couple sides to it. So I'll talk about it from a business making a decision on which model to choose. And then I'll also talk about kind of the societal implications, the AI safety implications. And I'd love to hear your thoughts on this topic as well. Mallory. Um, first of all, on the business, making a decision on which model to use.

Amith Nagarajan: Always start with the basic rubric of saying what are the capabilities and what are the costs with any software vendor of any kind? You need to look at it and say, Hey, What will this product do for me? And how much will it cost me? And the how much might not just be financially, but it might also be deployment costs, complexity of integration, things like that.

Amith Nagarajan: Um, and so the capability side, what you might find is that at open source, you actually get capabilities that you don't get with closed source because you can use it with data that you consider too sensitive to provide to Claude or to open AI. So that's one thing that essentially creates a new capability.

Amith Nagarajan: It's not that the fundamental technology can do something Like the in llama can do something that open AI cannot, but you are willing to use it for this more sensitive application because you have it in a controlled environment. So that's one thing. And there probably are situations where llama is better or anthropic is better, but they're all getting so good that I think for most use cases, it's.

Amith Nagarajan: It's going to be fairly commoditized in terms of the fundamental capabilities. Um, you might want to think about who you're partnered with for deployment. So llama is going to be deployed across all the major cloud providers. It's going to be available on GROQ. It's going to be available from meta directly.

Amith Nagarajan: It's gonna be available in a lot of places. Um, that is an advantage because you have portability. Um, you can also do a lot of other things with, you know, fine tuning with it. Um, but the flip side is there's a little bit more complexity in managing an open source LLM like this. If, if you're using it like in an environment you're in control over, if you're deploying it, you know, through AWS or Azure or GROQ, it's just as simple.

Amith Nagarajan: It's just an API, just like using open AI, but there's pros and cons to that. So I think as a business decision maker, you're looking at it from a capabilities versus potential risks versus cost type of rubric that you would look at with with any software decision. Societally, I think there's an interesting conversation to be had about. Let's say for closed source frontier models or open source, you know, the people at open AI would probably, you know, continue to strenuously argue in front of policymakers in Congress and elsewhere, that open source is dangerous, that open source could lead to, you know, uh, state actors and others doing really bad things with the AI.

Amith Nagarajan: Um, and there's a valid point to be argued there that if you release the most cutting edge AI for anyone to use any way they want. What does that mean for the world? Right? Um, what does it mean for export controls? What does it mean for defense? What does it mean for a variety of things? The flip side of it is, um, there's a massive amount of ego that goes into that statement and also essentially regulatory capture mindset that an open AI would want people to say, Oh no, no, open a open source is really bad.

Amith Nagarajan: You've got to have closed source because that protects their business. So the open source folks would say, well, actually. Um, really the safest model is the open model because it's something you can look at, you can inspect it, you know how it was trained, you know where it's deployed, you have control over it.

Amith Nagarajan: And by the way, there's a lot of people who are going to be doing a lot of bad things, no matter what you do, and the more you can aggressively deploy good AI, the That's really the only possibility you have to protect against bad AI, whether it's bad AI, that's based on open source or bad AI, that's based on something else.

Amith Nagarajan: So I tend to lean in that direction. I think the debate is a very interesting one. And I think there's good points in both directions or on both sides of it. Um, you know, but I think it's one of these things where ultimately, Um, you know, it's a hard, it's a hard question to answer with a definitive yes or no, good or bad, open source or closed source, because it kind of depends.

Amith Nagarajan: So that's, that's where I, that's where my head is at. What are your thoughts based on everything, you know, you've been exposed to over the last year, year and a half with this stuff?

Mallory Mejias: It is a great question. I've learned a ton about this from you, Amith, honestly, um, and kind of your takes, I don't think I've made a ton of technology decisions, um, just in my career. So I feel like I can't really approach it from that angle, but I think as a human, as a person, I tend to be on the all ships rise kind of mindset.

Mallory Mejias: So I would lean open source in the sense that you can, you know, You can collaborate more, you can promote more innovation, um, more creativity, but I also understand the concern that, well, now everybody has access to 3. 1, um, and what are they going to do with that technology? I think in the end, I believe the more eyes we have on something, the more We'll be able to prepare for kind of the bad uses of it, like you said.

Mallory Mejias: So I would say leaning open source. However, I mean, I'm using Anthropics Claude every single day. So it's not to say that I'm only going to use open source AI models, but in theory, I think that's the, the path I align with more, but I'm sure I have much more to learn to kind of on the flip side. Now, if I were, uh, Sam Altman, I would be like proprietary all the way.

Mallory Mejias: I want to make as much money as I can off of this thing that we created. So I definitely understand the other side too. Okay. Okay.

Amith Nagarajan: And I do think there's a validity to saying, okay, well, Meta put 405 be out into the world. Um, you know, Kim Jong un and in North Korea can download it just like anyone else and use it for whatever. Uh, and it's a very powerful piece of technology. There's no export controls over that. You know, you can run it, anyone can take it and run it and do stuff with it.

Amith Nagarajan: So is that good or bad? Right. You know, what does that ultimately mean? So I think that there are a lot of reasons to have, um, a degree of Uh, thoughtfulness, at least around open source, uh, regulatory control ultimately is extremely unlikely to have any impact. You know, there's been conversation around this for a long time already.

Amith Nagarajan: Very little has actually been done, uh, to the extent that, you know, you see in the EU, there's a little bit more regulatory control or attempted regulatory control. And really what you're seeing there is there's just a stifling of innovation. A lot of companies are saying, Yeah, we're not gonna operate there.

Amith Nagarajan: We're not gonna offer our products there on. It's happening everywhere else. It's not really stopping anything because you have to have every country in the world agree to that at the same time, basically, to have that occur. So I don't know what the answer is. Uh, I think that we all have to be thoughtful about this.

Amith Nagarajan: We have to be willing to hear, you know, opposing viewpoints now more than ever. Um, truthfully, you know, about everything, of course, not just about technology and something as important as AI, but, um, you know, the more you know about this stuff, the more you should know that you really don't know a whole lot.

Amith Nagarajan: And that's how I feel every single day about this stuff. It's a bit overwhelming. It's a bit exciting. It's actually very exciting. Uh, but it also is humbling and it needs to teach us all that, you know, as we look to apply these technologies in the best possible ways for our organizations, we also need to keep our hat on or available to us.

Amith Nagarajan: That's our citizen hat and say like, what's the best use case and the best approach for deploying this thing, you know, throughout our world.

Mallory Mejias: Okay. That is a great point to end this episode on. Everyone, thanks for tuning in to today's episode. If you liked it, please drop us a review on your favorite podcasting platform, or if you're joining us on YouTube, give us a like, give us a subscribe.

Mallory Mejias: We so appreciate it, and we will see you next week.