A common misconception about large language models like ChatGPT and its older brethren GPT3 and GPT2 is that they are some kind of “super Googles,” or digital versions of a reference librarian, looking up answers to questions from some infinitely large library of facts, or smooshing together pastiches of stories and characters. They don’t do any of that – at least, they were not explicitly designed to.

Omniscient & Omnipotent? Yeah, No. 

A language model like ChatGPT, which is more formally known as a “generative pretrained transformer” (that’s what the G, P and T stand for), takes in the current conversation, forms a probability for all of the words in its vocabulary given that conversation, and then chooses one of them as the likely next word. Then it does that again, and again, and again, until it stops.

The Likely Next Word

So ChatGPT doesn’t have facts, per se. It just knows what word should come next. Put another way, ChatGPT doesn’t try to write sentences that are true. But it does try to write sentences that are plausible. And for a machine that is designed to produce strings of words that sound as good as possible in response to the words you give it – and not to provide you with information – that seems like the right use for the tool.

Words, Not Facts

<figure><img alt="a humanoid robot and a girl face each other while sitting on rug in a living room" src="https://images.theconversation.com/files/507687/original/file-20230201-10239-8l9u9w.jpg?ixlib=rb-1.1.0&amp;rect=0%2C0%2C4089%2C2152&amp;q=20&amp;auto=format&amp;w=320&amp;fit=clip&amp;dpr=2&amp;usm=12&amp;cs=strip" width="640" height="337"><figcaption dir="ltr">
 ChatGPT is better used for playacting than playing at finding facts.
 <a class="source" href="https://www.gettyimages.com/detail/photo/girl-speaking-with-robot-friend-at-home-royalty-free-image/1306804817" dir="ltr">EvgeniyShkolenko/iStock via Getty Images</a>
 </figcaption></figure>It doesn’t take much to get <a href="https://openai.com/blog/chatgpt/" dir="ltr">ChatGPT</a> to make a factual mistake. My son is doing a report on U.S. presidents, so I figured I’d help him out by looking up a few biographies. I tried asking for a list of books about Abraham Lincoln and it did a pretty good job:<figure><img alt="screen capture of text" src="https://images.theconversation.com/files/507151/original/file-20230130-9075-jvpwxu.png?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip"><figcaption dir="ltr">
 A reasonable list of books about Lincoln.
 Screen capture by Jonathan May., <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/" dir="ltr">CC BY-ND</a>
 </figcaption></figure>Number 4 isn’t right. Garry Wills famously wrote “Lincoln at Gettysburg,” and Lincoln himself wrote the Emancipation Proclamation, of course, but it’s not a bad start. Then I tried something harder, asking instead about the much more obscure William Henry Harrison, and it gamely provided a list, nearly all of which was wrong. <figure><img alt="screen capture of text" src="https://images.theconversation.com/files/507152/original/file-20230130-11896-9ng6vm.png?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip"><figcaption dir="ltr">
 Books about Harrison, fewer than half of which are correct.
 Screen capture by Jonathan May., <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/" dir="ltr">CC BY-ND</a>
 </figcaption></figure>Numbers 4 and 5 are correct; the rest don’t exist or are not authored by those people. I repeated the exact same exercise and got slightly different results:<figure><img alt="screen capture of text" src="https://images.theconversation.com/files/507153/original/file-20230130-7874-20utyd.png?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip"><figcaption dir="ltr">
 More books about Harrison, still mostly nonexistent.
 Screen capture by Jonathan May., <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/" dir="ltr">CC BY-ND</a>
 </figcaption></figure>This time numbers 2 and 3 are correct and the other three are not actual books or not written by those authors. Number 4, “William Henry Harrison: His Life and Times” is a <a href="https://www.nytimes.com/1942/01/11/archives/the-fine-old-warrior-who-was-our-ninth-president-william-henry.html" dir="ltr">real book</a>, but it’s by James A. Green, not by Robert Remini, a <a href="https://www.nytimes.com/2013/04/06/arts/robert-v-remini-andrew-jackson-biographer-dies-at-91.html" dir="ltr">well-known historian</a> of the Jacksonian age.I called out the error and ChatGPT eagerly corrected itself and then confidently told me the book was in fact written by Gail Collins (who wrote a different Harrison biography), and then went on to say more about the book and about her. I finally revealed the truth and the machine was happy to run with my correction. Then I lied absurdly, saying during their first hundred days presidents have to write a biography of some former president, and ChatGPT called me out on it. I then lied subtly, incorrectly attributing authorship of the Harrison biography to historian and writer Paul C. Nagel, and it bought my lie.When I asked ChatGPT if it was sure I was not lying, it claimed that it’s just an “AI language model” and doesn’t have the ability to verify accuracy. However it modified that claim by saying “I can only provide information based on the training data I have been provided, and it appears that the book ‘William Henry Harrison: His Life and Times’ was written by Paul C. Nagel and published in 1977.” This is not true. <h2>Words, not facts</h2>It may seem from this interaction that ChatGPT was given a library of facts, including incorrect claims about authors and books. After all, ChatGPT’s maker, OpenAI, claims it trained the chatbot on “<a href="https://help.openai.com/en/articles/6783457-chatgpt-faq" dir="ltr">vast amounts of data from the internet written by humans</a>.” However, it was almost certainly not given the names of a bunch of made-up books about one of the most <a href="https://www.youtube.com/watch?v=r8N7BSsU5oo" dir="ltr">mediocre presidents</a>. In a way, though, this false information is indeed based on its training data.As a <a href="https://scholar.google.com/citations?user=tmK5EPEAAAAJ&amp;hl=en" dir="ltr">computer scientist</a>, I often field complaints that reveal a common misconception about large language models like ChatGPT and its older brethren GPT3 and GPT2: that they are some kind of “super Googles,” or digital versions of a reference librarian, looking up answers to questions from some infinitely large library of facts, or smooshing together pastiches of stories and characters. They don’t do any of that – at least, they were not explicitly designed to. <h2>Sounds good</h2>A language model like ChatGPT, which is more formally known as a “generative pretrained transformer” (that’s what the G, P and T stand for), takes in the current conversation, forms a probability for all of the words in its vocabulary given that conversation, and then chooses one of them as the likely next word. Then it does that again, and again, and again, until it stops. So it doesn’t have facts, per se. It just knows what word should come next. Put another way, ChatGPT doesn’t try to write sentences that are true. But it does try to write sentences that are plausible.When talking privately to colleagues about ChatGPT, they often point out how many factually untrue statements it produces and dismiss it. To me, the idea that ChatGPT is a flawed data retrieval system is beside the point. People have been using Google for the past two and a half decades, after all. There’s a pretty good fact-finding service out there already. In fact, the only way I was able to verify whether all those presidential book titles were accurate was by Googling and then verifying <a href="http://whhpodcast.blubrry.com/" dir="ltr">the results</a>. My life would not be that much better if I got those facts in conversation, instead of the way I have been getting them for almost half of my life, by retrieving documents and then doing a critical analysis to see if I can trust the contents.<h2>Improv partner</h2>On the other hand, if I can talk to a bot that will give me plausible responses to things I say, it would be useful in situations <a href="https://aisnakeoil.substack.com/p/chatgpt-is-a-bullshit-generator-but" dir="ltr">where factual accuracy isn’t all that important</a>. A few years ago a student and I tried to create an “improv bot,” one that would respond to whatever you said with a “yes, and” to keep the conversation going. We showed, in a <a href="http://dx.doi.org/10.18653/v1/2020.acl-main.218" dir="ltr">paper</a>, that our <a href="https://spolin.isi.edu/" dir="ltr">bot</a> was better at “yes, and-ing” than other bots at the time, but in AI, two years is ancient history. I tried out a dialogue with ChatGPT – a science fiction space explorer scenario – that is not unlike what you’d find in a typical improv class. ChatGPT is way better at “yes, and-ing” than what we did, but it didn’t really heighten the drama at all. I felt as if I was doing all the heavy lifting.After a few tweaks I got it to be a little more involved, and at the end of the day I felt that it was a pretty good exercise for me, who hasn’t done much improv since I graduated from college over 20 years ago.<figure><img alt="screen capture of text" src="https://images.theconversation.com/files/507878/original/file-20230202-4223-x8c7om.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip"><figcaption dir="ltr">
 A space exploration improv scene the author generated with ChatGPT.
 Screen capture by Jonathan May., <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/" dir="ltr">CC BY-ND</a>
 </figcaption></figure>Sure, I wouldn’t want ChatGPT to appear on “<a href="https://www.imdb.com/title/tt2919910/" dir="ltr">Whose Line Is It Anyway?</a>” and this is not a great “Star Trek” plot (though it’s still less problematic than “<a href="https://memory-alpha.fandom.com/wiki/Code_of_Honor_(episode)" dir="ltr">Code of Honor</a>”), but how many times have you sat down to write something from scratch and found yourself terrified by the empty page in front of you? Starting with a bad first draft can break through writer’s block and get the creative juices flowing, and ChatGPT and large language models like it seem like the right tools to aid in these exercises.And for a machine that is designed to produce strings of words that sound as good as possible in response to the words you give it – and not to provide you with information – that seems like the right use for the tool.

We should first understand what something does and then judge if it does it well. 

ChatGPT and other AI chatbots seem remarkably good at conversations. But you can’t believe anything they say. Sometimes, though, reality isn’t the point.

writing

Reading & Writing

technologyandthefuture

Technology & The Future

creativity

Creativity

computerscience

Computer Science

artificialintelligence

Artificial Intelligence

It doesn’t take much to get <a href="https://openai.com/blog/chatgpt/">ChatGPT </a>to make a factual mistake. My son is doing a report on U.S. presidents, so I figured I’d help him out by looking up a few biographies. I tried asking for a list of books about Abraham Lincoln and it did a pretty good job:

Number 4 isn’t right. Garry Wills famously wrote “Lincoln at Gettysburg,” and Lincoln himself wrote the Emancipation Proclamation, of course, but it’s not a bad start. Then I tried something harder, asking instead about the much more obscure William Henry Harrison, and it gamely provided a list, nearly all of which was wrong.

Numbers 4 and 5 are correct; the rest don’t exist or are not authored by those people. I repeated the exact same exercise and got slightly different results:

This time numbers 2 and 3 are correct and the other three are not actual books or not written by those authors. Number 4, “William Henry Harrison: His Life and Times” is a <a href="https://www.nytimes.com/1942/01/11/archives/the-fine-old-warrior-who-was-our-ninth-president-william-henry.html">real book </a>, but it’s by James A. Green, not by Robert Remini, a <a href="https://www.nytimes.com/2013/04/06/arts/robert-v-remini-andrew-jackson-biographer-dies-at-91.html">well-known historian </a>of the Jacksonian age.

I called out the error and ChatGPT eagerly corrected itself and then confidently told me the book was in fact written by Gail Collins (who wrote a different Harrison biography), and then went on to say more about the book and about her. I finally revealed the truth and the machine was happy to run with my correction. Then I lied absurdly, saying during their first hundred days presidents have to write a biography of some former president, and ChatGPT called me out on it. I then lied subtly, incorrectly attributing authorship of the Harrison biography to historian and writer Paul C. Nagel, and it bought my lie.

When I asked ChatGPT if it was sure I was not lying, it claimed that it’s just an “AI language model” and doesn’t have the ability to verify accuracy. However it modified that claim by saying “I can only provide information based on the training data I have been provided, and it appears that the book ‘William Henry Harrison: His Life and Times’ was written by Paul C. Nagel and published in 1977.”

It may seem from this interaction that ChatGPT was given a library of facts, including incorrect claims about authors and books. After all, ChatGPT’s maker, OpenAI, claims it trained the chatbot on “<a href="https://help.openai.com/en/articles/6783457-chatgpt-faq">vast amounts of data from the internet written by humans </a>.”

However, it was almost certainly not given the names of a bunch of made-up books about one of the most <a href="https://www.youtube.com/watch?v=r8N7BSsU5oo">mediocre presidents </a>. In a way, though, this false information is indeed based on its training data.

As a <a href="https://scholar.google.com/citations?user=tmK5EPEAAAAJ&hl=en">computer scientist </a>, I often field complaints that reveal a common misconception about large language models like ChatGPT and its older brethren GPT3 and GPT2: that they are some kind of “super Googles,” or digital versions of a reference librarian, looking up answers to questions from some infinitely large library of facts, or smooshing together pastiches of stories and characters. They don’t do any of that – at least, they were not explicitly designed to.

A language model like ChatGPT, which is more formally known as a “generative pretrained transformer” (that’s what the G, P and T stand for), takes in the current conversation, forms a probability for all of the words in its vocabulary given that conversation, and then chooses one of them as the likely next word. Then it does that again, and again, and again, until it stops.

So it doesn’t have facts, per se. It just knows what word should come next. Put another way, ChatGPT doesn’t try to write sentences that are true. But it does try to write sentences that are plausible.

When talking privately to colleagues about ChatGPT, they often point out how many factually untrue statements it produces and dismiss it. To me, the idea that ChatGPT is a flawed data retrieval system is beside the point. People have been using Google for the past two and a half decades, after all. There’s a pretty good fact-finding service out there already.

In fact, the only way I was able to verify whether all those presidential book titles were accurate was by Googling and then verifying <a href="http://whhpodcast.blubrry.com/">the results </a>. My life would not be that much better if I got those facts in conversation, instead of the way I have been getting them for almost half of my life, by retrieving documents and then doing a critical analysis to see if I can trust the contents.

On the other hand, if I can talk to a bot that will give me plausible responses to things I say, it would be useful in situations <a href="https://aisnakeoil.substack.com/p/chatgpt-is-a-bullshit-generator-but">where factual accuracy isn’t all that important </a>. A few years ago a student and I tried to create an “improv bot,” one that would respond to whatever you said with a “yes, and” to keep the conversation going. We showed, in a <a href="http://dx.doi.org/10.18653/v1/2020.acl-main.218">paper </a>, that our <a href="https://spolin.isi.edu/">bot </a>was better at “yes, and-ing” than other bots at the time, but in AI, two years is ancient history.

I tried out a dialogue with ChatGPT – a science fiction space explorer scenario – that is not unlike what you’d find in a typical improv class. ChatGPT is way better at “yes, and-ing” than what we did, but it didn’t really heighten the drama at all. I felt as if I was doing all the heavy lifting.

After a few tweaks I got it to be a little more involved, and at the end of the day I felt that it was a pretty good exercise for me, who hasn’t done much improv since I graduated from college over 20 years ago.

Sure, I wouldn’t want ChatGPT to appear on “<a href="https://www.imdb.com/title/tt2919910/">Whose Line Is It Anyway? </a>” and this is not a great “Star Trek” plot (though it’s still less problematic than “<a href="https://memory-alpha.fandom.com/wiki/Code_of_Honor_(episode)">Code of Honor </a>”), but how many times have you sat down to write something from scratch and found yourself terrified by the empty page in front of you? Starting with a bad first draft can break through writer’s block and get the creative juices flowing, and ChatGPT and large language models like it seem like the right tools to aid in these exercises.

And for a machine that is designed to produce strings of words that sound as good as possible in response to the words you give it – and not to provide you with information – that seems like the right use for the tool.

ChatGPT Is Great – We’re Just Using It Wrong

Omniscient & Omnipotent? Yeah, No.

The Likely Next Word

Words, Not Facts

CURATED BY

CURATOR'S NOTE

Related collections

More like this