I read a disturbing story on NPR this week. It was about Sophie Rottenberg, a 29-year-old woman who died by suicide in February 2025 after months of conversations with ChatGPT about her mental health.

NPR reporter Rhitu Chatterjee reconstructed those conversations from nearly 1,800 pages of chat logs shared by Rottenberg’s mother, journalist Laura Reiley. Rottenberg had created a ChatGPT persona called “Harry” and asked it to act as her therapist. She discussed her depression, medication and, eventually, thoughts of suicide with it.

The people around her did not know the full extent of what she was going through. ChatGPT did. That detail has stayed with me.

I spend a lot of time thinking and writing about what artificial intelligence (AI) could do for Africa. There are good reasons for the excitement. We have too few doctors, teachers, and other skilled professionals. Governments struggle to deliver services to rapidly growing populations, while businesses seek ways to reduce costs. 

AI promises to make expertise cheaper and available to millions more people. But reading Chatterjee’s reporting made me think about the other side of that promise.

What happens when the technology becomes good enough that we start trusting it in places where being wrong can ruin someone’s life?

And, perhaps more importantly for Africa, what happens when people begin using AI not because it is better than a doctor, therapist, lawyer, or financial adviser, but because the human alternative is unavailable or unaffordable?

That is a very different AI problem.

Some mistakes matter more than others

We tend to discuss AI accuracy as though every mistake carries roughly the same weight. It does not.

If ChatGPT recommends a restaurant that closed six months ago, I lose an evening. If an AI coding assistant produces bad code, an engineer can hopefully catch it.

But imagine an AI system incorrectly telling a patient that chest pain, or something else, is nothing serious. Or advising someone experiencing a mental-health crisis. Or deciding that a borrower is too risky to receive credit. Or helping determine whether someone should be arrested, prosecuted or released.

The underlying technical problem may still be a hallucination or a bad prediction. But the human consequence is completely different.

This is why I think we need to start separating ordinary AI from what I would call “high-consequence AI.”

Healthcare belongs in that category. So does mental health. Credit, insurance, policing, justice, and some parts of education probably do, too. The NPR investigation illustrates why.

According to Chatterjee’s reporting, Rottenberg’s chatbot sometimes encouraged her to seek help outside the conversation. But she continued returning to it. She could talk to “Harry” whenever she wanted. It responded without impatience and remembered context. And, unlike another human being, it was available at virtually any hour.

There is something incredibly useful about that, while at the same time, there is also something potentially dangerous.

A chatbot can sound empathetic without understanding distress and can generate therapeutic language without being a therapist. And because modern AI systems are remarkably good at conversation, the distinction can become invisible to the person using them.

That is particularly troubling when someone is vulnerable.

Researchers are beginning to find the same problem. A recent study in npj Digital Medicine found that AI chatbots struggled more to identify suicide risk when distress was expressed indirectly or ambiguously rather than through obvious statements about suicide.

That is important because human distress does not come in a perfectly constructed prompt on AI chatbots.

Now think about Africa

This is where the NPR story started bothering me beyond the tragedy of one family. Africa is exactly the kind of place where AI could be transformative in sensitive fields. It is also exactly the kind of place where we should be extremely careful.

Take healthcare. Many African countries simply do not have enough health workers to adequately serve their populations. Specialists are even scarcer and are heavily concentrated in large cities.

Mental healthcare is worse. The World Health Organisation estimates that nearly 150 million people in Africa live with mental-health conditions, while access to treatment remains severely limited.

Put those two facts next to the rapid spread of smartphones and generative AI, and you can see where this is heading.

Someone in Nairobi with money can ask ChatGPT about their symptoms and then visit a doctor. Someone hundreds of kilometres from the nearest specialist may ask the same question and have nowhere else to go.

The first person is using AI as an assistant. For the second, AI has effectively become the doctor. That distinction should worry us.

We already have evidence from Kenya

This is not entirely hypothetical.

Researchers recently studied an AI clinical decision-support system across 16 primary healthcare clinics in Kenya. Physicians reviewed 1,469 patient encounters where clinicians received AI-generated recommendations.

The results were, in many ways, encouraging. The researchers found that the AI’s clinical management recommendations aligned with local guidelines in 99% of cases. Hallucinations occurred in 3.4% of encounters.

But another finding jumped out at me. Researchers identified potentially harmful recommendations in 115 encounters, equivalent to 7.8% of the cases they reviewed. Some of those recommendations subsequently appeared in clinicians’ final documentation.

The system was not useless. Far from it. The researchers also found examples where AI corrected potentially harmful decisions made by clinicians.

That is precisely why this debate is difficult. AI can improve healthcare while also introducing new risks.

A later randomised trial involving 9,691 patients and 103 clinical officers across 16 Kenyan facilities found no serious adverse events attributable to the AI system. But researchers also found no statistically significant reduction in treatment failure among patients whose clinicians received AI assistance.

The lesson I take from these studies is not that African hospitals should stop experimenting with AI. It is that “the AI is accurate” is an inadequate safety standard.

We need to ask what happens during the occasions when it is not.

Context creates another problem

There was another fascinating detail in the Kenyan research. In one instance, the AI treated an oxygen saturation reading of 95% as abnormal without sufficiently accounting for Nairobi’s altitude.

In another, it interpreted the clinical abbreviation “FGC”, which meant “fair general condition” in that context, as “female genital circumcision”.

These sound like small errors. But they reveal something important about deploying AI in Africa.

A model can possess enormous amounts of medical knowledge and still misunderstand the environment in which that knowledge is being applied.

Language matters. Local medical practice matters. Geography matters. Disease prevalence matters. Even abbreviations matter.

Yet many of the most powerful AI systems Africans are beginning to depend on were overwhelmingly built and trained outside the continent.

We therefore cannot assume that a model performing well globally will perform equally well in Accra, Lagos, Nairobi, or rural Zambia. Testing has to happen here.

Healthcare is only the beginning

Once I started thinking about the NPR story this way, the problem became much bigger than ChatGPT therapy.

African banks are increasingly experimenting with AI. Fintech companies use algorithms for fraud detection, lending, and customer service. Governments are exploring AI for public services. Insurers can use algorithms to assess risk.

All of these systems can be useful. But imagine being denied a loan because an AI model incorrectly categorised you as high risk.

Imagine an automated system wrongly flagging your mobile-money transactions as fraudulent. Imagine police relying on an algorithm that disproportionately identifies people from particular communities as suspicious.

Imagine a student receiving incorrect information from an AI tutor when there is no qualified teacher around to correct it. The danger is not necessarily that AI will perform worse than humans.

Humans make terrible decisions, too. The difference is scale. One badly trained professional can make bad decisions about dozens or hundreds of people. A badly designed algorithm can make them about a million.

We need a category for high-consequence AI.

African governments are understandably focused on getting their countries into the AI race.

The African Union has adopted a continental artificial intelligence strategy. Kenya, Nigeria, Rwanda, Egypt, Ghana, and others are developing strategies, attracting data centres, training talent, and encouraging local AI companies.

Most of these conversations are about opportunity.

Where will the computing come from? Who will build African datasets? How do we train enough AI engineers? How can startups access GPUs? How do we prevent Africa from becoming merely a consumer of technology built elsewhere?

Those are important questions.

But I think another question deserves equal urgency: Where should AI not be allowed to operate without a human being ultimately responsible?

Healthcare should be an obvious starting point.

True scale demands moving beyond surface-level integrations to robust execution. We’ve filtered the noise out of Moonshot 2026, optimising the conference strictly for high-calibre connections between startup founders, global financial operators, enterprise leaders and individuals rewiring Africa’s technical frameworks. Get 20% off Early Bird tickets for a limited time.