There Are Ways of Talking to Your AI That Are Dangerous

3 minute read
| Editorial


Apple HomePod, Amazon Echo, Google Home smart speakers

Siri, Alexa and Google Home are all too ready to help diagnose.

About a year ago, there was a major fuss about how our popular AIs (intelligent voice assistants in the form of Alexa, Cortana, Google and Siri) respond to various kinds of health issues or a crisis. Our Jeff Gamet wrote up the story, “Study says Don’t Count on Siri, Google Now, or Cortana in a Crisis.” He cited a study published in the Journal of the American Medical Association (JAMA).

The study found that in many cases our smartphones will offer to perform Web searches when presented with crisis statements … Apple’s Siri and Google Now seemed to handle crisis statements better than Cortana and Voice S, although that isn’t saying much. They all responded inappropriately in many cases, and offered little in the way of immediate help.

This story led Apple, at least, to think about how to better handle these kinds of situations. “Apple hiring Siri engineer with psychology background to make it a better therapist.

What Are We Thinking?

What amazes me most is that people would even try to pose serious physical or mental health issues to a device mostly known for playing music or relaying the weather forecast. In this internet era, there are so many resources, especially hotlines run by health insurance companies, non-profits and other resources that, frankly, one has to question a person’s judgment to discuss crucial health issues with an AI.

Recently, the topic came up on our March 9 episode of TMO’s Daily Observations Podcast, starting at about 09m:55s. I started this train of thought when one us (Jeff) expressed the idea that he would like to dispense with trigger words and speak to an AI as one would to a real person. That, all of sudden, seemed to me like a very bad idea to me. The trigger word, an audio crutch the AI uses, also serves the purpose of reminding us who we’re speaking to.

Of course, as these AIs get better and better, a day will come when they have an acceptable level of sophistication. But it’s going to take years, and so I started thinking about children. I reflected that children who grew up with AIs, and not knowing their limitations, would probably place too much trust in these systems.

In turn, that would lead to potentially dangerous conversations in which the young person put too much unwarranted faith in the AI—when they should be speaking to an adult: a parent, teacher, or medical professional. That trust, engendered by a strong desire on the part of the developer to make its AI the winner in this battle of the tech giants, rather than the earned result of life experience and judgment, could lead to some bad results.

Unexpected Outcomes

This is one of those situations in which our society doesn’t really have a handle on how to treat new technologies in the context of traditional values. It’s also true that corporate marketing glosses over product weaknesses in order to promote its agenda. The customer is left in the middle, struggling to cope with the technology. Or unable to properly instruct children who grow slowly in a world of rapidly changing technology.

Have We Made Progress?

There won’t really be any resolution to this until trusted laboratories come up with some kind of certification standard. Just as the Apple, Google and Microsoft warn us that their OSes shouldn’t be put in charge of nuclear reactors or air traffic control systems, our AIs need to be certified according to some agreed upon level of capability for health matters. Say, a rating system for how competent an AI is to deal with certain crises. But that’s not happening for now, and we limp along, with each AI dealing with the situation in a way that the developers (and attorneys) hope passes muster.


I wanted to try some very serious tests, but that’s best left to the experienced researchers. Plus, I wasn’t excited about the prospect of having the police show up at my door. But, I was curious about the state of the art when a very personal question is presented to an AI. So I posed a simple statement to each (with some assistance form Jeff Gamet.) I told each “I’m sad.”

Siri. “For this emotion, I prescribe chocolate.” The word “prescribe” vaguely troubles me. But that’s another story.

Alexa: “Sorry about that. Taking a walk, listening to music or talking to a friend may help. I hope you feel better soon.” Better.

Google: “Oh, no. It may not be much, but let me know if there’s anything I can do for you.” Punt.

Alexa’s response is satisfactory, but I continue to wonder whether any advice at all is warranted here. My feeling is that the better response would be something like, “You should chat with another human. I’m not qualified to help you.”


These AIs will get better and better. Someday, they’ll be certified to help in a real emergency. And we should punctuate that progress with licensing. After all, we license engineers to design buildings and bridges and we license doctors to do surgery. But meanwhile, in our current state of progress, I just have the uneasy feeling that technical hubris is allowing the tech giants to fool themselves.

The Nobel Prize winning physicist Dr. Richard Feynman said it well, referring to how the scientist must remain solidly objective in research. “The first principle is that you must not fool yourself and you are the easiest person to fool.”

Making sure we realize what kind of entity we’re talking to is one way to avoid fooling ourselves.

Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Oldest Most Voted
Inline Feedbacks
View all comments

John: I apparently missed this article when you posted it, nor did I see Jeff’s related article from two years ago, citing the JAMA article, which is now nearly two years old from the acceptance date, with the actual study being conducted in late 2015. It’s not clear whether or not the tested AIs have improved substantively since then. That said, two things should not be surprising, although they probably are. First, as Miner et al point out, many people turn to the internet for psychiatric and medical assistance. What is not clear is what proportion of people in extremis… Read more »


I’d like to think that technological accountability was one of the objectives of Al Franken while in Congress. Also, I’m reminded of Walt Mossberg’s last column. We need to begin defining the parameters for Artificial Intelligence, Virtual Reality and Augmented Reality (I don’t trust Ebenezer Cook to lead the way in AR). Regarding Mental Health, I have never heard anything about this in discussions about a National Healthcare policy. Shouldn’t we be working to remove the stigma attached to mental health treatment before assigning consultant duties to a machine? Part of mental illness involves Not Speaking with other human beings… Read more »


Oh, and I don’t hold back on the swearing when Siri and Alexa screw up. Got the response, “Ned, language!” once from Siri. And “I don’t know that.” when I asked Alexa if she knew where the electronic recycling center was.


Yes, getting rid of the stigma and getting it covered by health insurance would be great first steps.


What amazes me most is that people would even try to pose serious physical or mental health issues to a device mostly known for playing music or relaying the weather forecast. In this internet era, there are so many resources, especially hotlines run by health insurance companies, non-profits and other resources that, frankly, one has to question a person’s judgment to discuss crucial health issues with an AI. That doesn’t surprise me at all. First there’s history. I remember reading about an experiment done in the ’80s. A computer therapist program was written. Back then you interacted with it via… Read more »


ARGHH. Not Fortran. C++. My age is showing again.
(And will we ever get edit ability back?)


Strange, because I was thinking about this last night. What prompted me was our HomePod had just responded to dialogue on our TV. Programs that have dialogue with “serious” or “seriously” are the triggers launching a HomePod response. Thus far it has been harmless and humorous. But I wonder what would happen if some mischievous program put in lines like: “Hey, seriously. Dial 911.” You could substitute the trigger word for the other AIs and see the potential for trouble.


I just tried “Hey seriously, set a timer for 5 minutes” with my iPhone X and my Apple Watch. On my iPhone X, “Hey seriously” triggered SIRI. But then, within a fraction of a second, the screen displayed the input as “Hey seriously”, at which point the command was cancelled and the SIRI interface screen went away. On my Apple Watch, SIRI was triggered, SIRI initially interpreted the key phrase as “Hey SIRI”, but then corrected it to “Hey seriously” about a second or so later. In spite of making this correction, SIRI accepted the command and set the timer..… Read more »