The Problem with AI

My post the other day about this day in history is a great example of the problem of AI. Let me ‘splain.

I, like most people this week, am completely done with mainstream media, on both sides of the spectrum. But, I do still want to know what the hell is going on in the world. I’m in sales, so it doesn’t do for me to be completely out of touch. One of the soft skills in sales is the ability to build rapport with strangers. Well, guess what? To do that, you have to have something in common to talk about. And most people I talk to aren’t writers, so I cant’ talk to them about that. And let’s face it, most people aren’t readers anymore, so I can’t talk to them about that.

So I have to try to keep up to date so I know who won the world series, for instance. I’m not a fan of either team, and I won’t use their names here as to keep this writing from showing up in search results.

A Little Background

A few years ago, I found 1440. I’ll let them sell themselves to you, but they are the one unbiased news source I rely on nearly every day. They send a simple email most days that covers the day’s news without comment or bias. I also use Wikipedia’s current events page from time to time, but that one requires that I go look at it instead of just see it in my email. And because of this, I don’t look at it much, but it does have great info, and gets updated throughout the day.

So, being a geek at heart, I decided to try to create my own daily email using an LLM. I’ve been playing with large language models since ChatGPT was in limited beta, just about two years ago. And we use them in our product in my day job after I convinced our CTO that the LLMs could add some functionality that we struggle to get on our own.

I use these little projects to advance my knowledge – I’m task-oriented. So I was going to have to learn more about Python to get this all done. I didn’t know how to send emails in Python, and I had only written code to talk to an LLM in a very simple way, I hadn’t really parsed their response. This project had several aspects that I’d have to learn, and I enjoy that kind of challenge.

The Failure of AI

The long story here is that the LLMs can’t do what I want them to do. At first, I had high hopes that Twitter’s (sorry I can’t call them X, it just seems strange) LLM, Grok would be able to look at Twitter feeds and give me a summary of current events. It does not. Like all LLMs, it’s trained with static data, and it has a :”knowledge cutoff date” which it knows nothing about from that day forward. So far, the best I’ve found is still a year out of date.

So that’s not gonna help me stay up to date.

But I’m stubborn, oh so stubborn.

I know that one of the biggest uses of Python is to scrape web pages. But I didn’t go directly to Python for my answer. First, I looked at news APIs. There has to be services that you can sign up for that give you access to news feeds – that’s how most of these bozos get their information.. And it’s true, they do exist, but they are very expensive. I’m talking hundreds of dollars per month. The one free API I found has yet to return a response.

Strike 2 (no relation to the world series).

Python to the Rescue?

Back to Python I go.

I have also never used Python to scrape a website, so that’s where I turned my attention. Python uses something called libraries – basically, someone has already built something that makes it easier for me to do what I want to do – that’s a library in a nutshell.

I needed a library to scrape a website. I tried a few. One was a cool new AI thing you may have heard of – an “AI Agent”. It didn’t work. No matter what I did, I could not get the LLM to only refer to the data the “agent” was getting from the Wikipedia website. I was reminded of myself yelling “AGENT!!!!” into the phone trying to get past the automated voicemail systems we deal with. It was about as effective and definitely as irritating.

I found a library called “Beautifulsoup”. Python programmers, and programmers in general are a funny lot. Python itself is named after Monty Python, so there you go. I won’t go into the frustration I’ve had trying to navigate the innards of a scraped web site. Let’s just say I’m still learning.

Back to AI and LLMs

So now I have a way forward for my email (remember, this is all so I can get a stupid single email each day to keep in touch with the world instead of doing other things like watching TV or doing a quick Google search). I’m scraping the Wikipedia website, and then calling a BIG LLM (Meta’s Llama 3.1 405b) to give me historical information about what happened on the specific day throughout history.

I had to test about 10 different LLMs to find one that had enough data to make the email interesting to me. I tested, Grok (Elon Musk’s baby), GPT-4o, Mixtral, Anthropic, and a bunch of Llama models. I liked the results I got best from Meta’s biggest model that has been fine tuned by someone else – Perplexity’s Sonar 405b model. It gives me the most detail and longest history lesson.

The worst part of all of this was making sure I’m getting accurate data.

Let me repeat that. The worst part is knowing I’m getting lied to by a computer program.

How do I know? I looked up the information the LLMs were giving me. Only the big 405b model would give me accurate data based on the date I told it to. All of the others started to stray from the date given.

Caveat Emptor?

And this is the problem with AI. It’s like a great bullshitter. The information sounds correct. But if you don’t know any better, you’ll just take it at face value as true. And if you already knew the data was true, you wouldn’t be looking it up, right?

This is a huge problem right now. If you’re using LLMs to “look up” information, you need to be very careful about using the data that comes back. I’ve found (after two years of using them regularly) that they are good for first drafts of things. But I never rely on data they give me as being accurate.

And that’s a problem when we have easy global access to tech like this.

Spam emails are no longer going to be poorly written. They’ll be perfect. Just stop clicking on links from email right now.

A Little Doom and Gloom

AI-generated news articles are full of shit. Oh, sorry. They are inaccurate at best. With our current political climate, this is not a good thing.

They are fast., though.

I use this tech at work to create a story from a few pieces of data we lift from a handwritten entry form. It’s hysterical. It can compose a funny story, poem, song, or whatever from a few different components instantaneously. I can be as creative, but not in 10 seconds, and not repeat it over and over and over.

As of this morning my early edition email is now working. It’s not perfect, but it’s working.

But to get what I wanted, I had to go back to basically screen scraping technology – technology that’s more than 30 years old, albeit in a new shiny wrapper.

It took a combination of old an new technology to get what I wanted. And that’s really the crux of what I wanted to say here – AI alone is not going to solve our problems, but using AI in conjunction with already proven methods is very powerful.

You just gotta know what you’re doing or suffer the consequences of really bad data.

Thanks for reading. Please subscribe here or below – feel free to leave a note in the comments! Please?? C’mon, I asked nicely!