When NOT to Use Large Language Models

Save Money: When Simpler AI Beats LLMs

Louis Bouchard

Jan 3, 2025 • 6 min read

Watch the video!

Are you feeling the pressure to adopt the latest AI trends but aren’t sure if it’s the right move for your business? With all the buzz around Large Language Models (LLMs) and the dozen videos I do on the new powerful techniques, it might leave some of your companies or future ones worried about being left behind. But are LLMs really the magical fix for all your problems? Does every company actually need one?

Of course, we all love LLMs, but sometimes they’re just way too overkill and use way too much compute and money when you could’ve used something much simpler…

In this video, we’ll dive into where LLMs truly shine and, more importantly, where they might fall short, along with the trade-offs you need to consider. This should give you a clear idea of whether or not LLMs are the right fit for your problem.

Where and Why You Might Need an LLM

Let’s start with where LLMs can really make a difference. Have you ever found yourself stuck on a creative task, like writing or brainstorming new ideas? LLMs are fantastic for sparking creativity. It’s just like a friend ready to help you draft an email, write a blog post, or even come up with a catchy marketing slogan. I used to spam my friends with ideas or questions, but now I can just spam ChatGPT even more. They’re the creative partner who’s always full of ideas — but, you might need to double-check their work because, let’s face it, LLMs can sometimes make stuff up. They are also quite repetitive and easy to spot. They’re great at generating ideas, but it’s always smart to give heavy editing and iterate with them . The one-off drafts they will produce will be super generic, and anyone who is used to working with LLMs will know it has been generated.

Take GitHub Copilot or Cursor, for example. Developers love it because it suggests helpful code snippets at the tap of a key, boosting productivity and making life easier. In customer support, LLM-powered chatbots have changed the game, making interactions more personalized and efficient. JPMorgan Chase, one of the biggest banks in the U.S., is a great case in point. They’ve integrated LLM-driven chatbots into their customer support system to provide better and faster answers while also helping with actual humans when needed.

In education, LLMs are being used to personalize learning experiences, just like Karpathy’s new venture. Think about a tutoring app that adapts to each student’s learning pace and style. By analyzing the student’s progress, the LLM can offer tailored exercises and explanations, making learning more engaging and effective. It’s like having a personal tutor available to everyone, anytime they need it.

Those are huge problems LLMs can tackle, often linked to creativity or adapting general content to specific individuals through the process of large amounts of data.

Where They Might Fall Short

Now, let’s talk about where LLMs don’t quite hit the mark. Despite their strengths, they’re not ready to make critical life decisions. You can’t solely rely on an LLM to diagnose a medical condition — that’s risky. IBM Watson for Oncology faced backlash for suggesting treatments that were either incorrect or unsafe because it was working with outdated or incomplete data. LLMs might give you advice that sounds good but could actually be harmful because they don’t truly understand the context or the consequences. You always need an expert to double-check what it generates, which ideally is yourself as the user of the LLM.

Specialized tasks, like legal work, can also trip up LLMs. In the legal industry, where precise language and nuanced interpretation are crucial, LLMs can sometimes miss the mark. There was a case where an LLM-based tool misinterpreted legal contract clauses, which could have led to serious legal risks. Even though there are law-focused LLMs being developed, it’s still wise to verify the facts before making big decisions. I believe lawyers will mostly use them to save time retrieving information with RAG-based systems versus regular people like us to get legal advice, at least in the short term.

Regulated industries like finance also pose challenges for LLMs. Some financial institutions have tried using LLMs for trading and risk management, but they’ve run into trouble because LLMs can be a bit of a black box. Without clear explanations, regulators were quick to push back, leading to compliance issues and potential penalties. However, there are success stories too. PayPal uses LLMs to detect fraud by analyzing transaction patterns, helping to spot suspicious activity before it causes damage. In these cases, the benefits of using LLMs outweigh the risks, but only because they’ve been implemented carefully with robust data and privacy practices.

When a Simple Model Will Do the Trick

This is all cool or scary, but not every language-related problem needs the power of an LLM. Sometimes, a smaller language model (SLM) is more than enough. These models are lighter, with fewer parameters, but they can still be incredibly effective, especially in specialized industries like healthcare, law, and finance.

For example, if you’re running a small business and need a chatbot for customer service, an LLM might be overkill. A well-trained small model like a distilled Llama 3 model can handle most customer inquiries effectively, saving you resources without sacrificing performance. It’s like using a scalpel instead of a sledgehammer — more precise, less costly, and just as effective for the task at hand.

Getting Into the Technical Details

When you’re thinking about bringing an LLM on board, there’s more to consider than just the technology. There’s a financial side too. LLMs can simplify a lot of processes, but they don’t come cheap. These models need a lot of computing power, which usually means investing in expensive GPUs. Whether you’re hosting your own LLM or using a managed service like OpenAI, you’re shifting the technical challenges into financial ones. It’s convenient, but it’s also a big investment.

There’s also something called technical debt. Traditional machine learning models often require extensive feature engineering, tuning, and maintenance. With LLMs, a lot of that complexity is offloaded to the model itself, which can be a huge relief, especially for smaller teams. But this simplification comes at the cost of an increased financial burden. You’re trading technical complexity for ease of use, but committing to the ongoing costs of running these massive models.

Latency and Task Nature

In user-facing applications, speed is key. For tasks other than language generation, traditional machine learning models are known for being fast, which makes them perfect for real-time tasks like financial trading or emergency response systems where every second counts.

But with LLMs, things are a bit different. Take a virtual assistant in customer support, for example. While speed is still important, the deep language understanding that LLMs offer can really improve the quality of interactions, even if there’s a slight delay. OpenAI recently released its real-time API, which is a big step forward there, but it’s quite hard or nearly impossible to replicate what you hosted locally. The same goes for content generation — sometimes, the richness of the output is worth waiting a little longer for or, in the ideal case, working a little harder for it.

So, the choice between traditional machine learning models and LLMs comes down to what you need: Is it speed, or is it depth and quality? Balancing these factors will help you decide which tool is right for the job.

Key Takeaways for Using LLMs

LLMs are nothing magic. They are powerful tools, but as with everything, they come with their own set of challenges. Knowing when and where to use them is crucial, especially if you are the person hired to manage and implement them. Whether it’s navigating costs, dealing with technical debt, or considering the specific needs of your industry, making informed decisions about LLM deployment will ensure you’re not just jumping on the AI bandwagon but actually getting the most out of these powerful tools. Remember that you always want to keep the most control over your solution. Always use the simplest solution possible, which will often be cheaper, faster, and easier to fix when something comes up.

And remember, the AI world is moving fast. What might seem like the best approach today could change as new models and techniques emerge. Likewise, what might seem impossible today might become a solved problem next week. Staying flexible and being willing to adapt will help you stay ahead of the curve.

Finally, don’t be afraid to experiment. Try different approaches, see what works for your specific context, and keep in mind that sometimes, the best solution isn’t the most obvious one. Whether you end up using an LLM, an SLM, or a combination of different models, the key is to find the right balance that meets your needs and helps you achieve your goals.

If you found this article useful (or at least interesting), learn more in our courses on the Towards AI Academy, both technical and non-technical!