Botanical Accuracy: 2022

Monday, December 26, 2022

Scientific failure rate of up to 92% for ChatGPT in botanical essay on Symbolanthus (ring-gentians)

A lot has been written recently about the AI-written essays and how hard it is to tell them from ones written by humans. The construction and grammar of the texts can be elaborate, error-free, and include a variety of details that make it hard to discern if it was a human or AI bot that wrote it.

But, there is one thing that the AI bots seem really bad at, and that is evaluating if the information they include in their AI written essays are actually correct, especially if you ask it to write about a more obscure subject.

To make an evaluation of the factual content of a AI-generated essay, not just the writing style and grammar, you have to ask the ChatGPT (or other AI 'writer') to produce text about something you know well and can evaluate when it comes to content and facts. So I did.

About Symbolanthus: I am a botanist at a large research university, and I am the world expert on the ring gentians, Symbolanthus. There are 38 species, but not all of these are described yet, but at least 25 species are present online in various floras, databases, iNaturalist, and such. It is a member of the plant family Gentianaceae, and it found in the wet, tropical parts of South and Central America from Bolivia to Costa Rica along the Andean mountain chain, and also in the Guayana Highlands of Brazil, Venezuela, and Guyana, plus a few Caribbean islands. The flowers are gorgeous, large and pretty in pink, magenta, light green to white, often with stripes on the inside. They are shrubs, sometimes small in height.

image of Symbolanthus macranthus from Ecuador, large pink flower and glossy green leaves.

The pilot test: I wanted to test ChatGPT to check if a student writing an essay from my botany classes could use it successfully. In my classes I primarily grade based on factual content, not on perfect grammar. So, on Dec 9, 2022, I asked ChatGPT to write an essay about Symbolanthus, I did this ten times, by repeatedly requesting it to "Write an essay on Symbolanthus".

Fact analysis: I saved all ten short essays (see pdf here). Then I categorized the type of information ChatGPT had included in the essay into ten information categories in a table in Excel (available upon request), tabulated all information from the essays into the right category for each essay and calculated how many information categories it had gotten majorly wrong in each essay. Not all categories were mentioned in each analysis, and those were not included in the count.

Results (see table pdf here for details):

For the ten categories, ChatGPT produced erroneous, wrong facts in 67-92% of scientific information categories. Those are categories of information. The number of misstated facts in each essay amounted to up to 50 errors per essay.

ChatGPT never classified Symbolanthus into its right family, the Gentianaceae, instead it said Asparagaceae, Asteraceae (6 times), Acanthaceae, or Melastomataceae.
ChatGPT never correctly described any of Symbolanthus' morphological features, such as leaves, flowers, seeds, or fruits.
ChatGPT often said it is grown in gardens due to its beautiful flowers, used as a cut flower, used in flower arrangments, etc.. It is not used in horticulture.
ChatGPT included various ethnobotanical and herbal medicine uses. There is no record that I am aware of with Symbolanthus having any medicinal uses.
ChatGPT invented species it says belongs to Symbolanthus that do not exist (Symbolanthus tatei, for example)
ChatGPT said sunflower (Helianthus annuus) is a member of Symbolanthus.
ChatGPT invented a new ecological adaptation - raising air temperatures increases the color of the flower. Very interesting. But fake.

The result is that every essay on Symbolanthus written by ChatGPT is a blend of scientific facts from any plant in the world, and contains nearly nothing scientifically correct. The result becomes an absurd hodge-podge, 'blender facts'. These are not minor errors, in fact, these are an abundance, a tornado, of major errors.

You cannot reshuffle information as part of AI writing, the facts need to stay stable. Unless you don't care about reality, then facts don't matter.

If you don't know the topic, however, it all looks completely reasonable in these essays because the facts seems solid, but they are not.

Now, if you use ChatGPT to write an essay or gather information about any other subject, how much would you trust it?

(This post was written fast, and any grammar errors and typos are my own.)

What is Botanical Accuracy?

Would you care if someone called a cat 'a mouse' in the description of a medieval painting?

Would you care if someone served you horse meat, but said it was beef? I bet you would.

Would you care if the wrong chemical was listed in the ingredient list of your shampoo or cereal?

Would you care if you bought one plant, but got another?

Would you care if there were species or ingredient mistakes in advertising, menus, herbal pills, and such things?

Would you care if books on plants are illustrated with the wrong plants?

If so, here is the place for you to read about such problems in the world of plants and plant products, where unfortunately such mistakes, inaccuracies, and problems are not uncommon. This is usually due to lack of botanical knowledge or expertise, or sometimes because of plain ignorance.

Inaccuracies are common when it comes to plants, because it seems like we humans care to learn less about green things like trees, flowers, and herbs than we care to learn about animals, even when we eat plants, paint them, plant them, extract their chemicals, or use them in numerous other ways.

Without plants in the world you and I certainly would be dead. Some plants can also kill you with their toxins, so it is best to know which plant is what species. Time to learn some botany!

About mistakes, inaccuracies, and errors

Science is a process of gaining knowledge and understanding of the world around us. It is a never-ending process, and what we think are true facts today might change tomorrow. In science we are aiming for having the best understanding possible today based on what we and our predecessors have learned until now.

This means that what is botanically accurate from a scientific viewpoint might (and will) change. Other experts in the field of botany know a lot more about their particular research plants than I do. New scientific findings and conclusions are being published every day. This is just normal and part of the scientific process; we improve on our knowledge all the time.

The important thing is our willingness to continuously aim for botanical accuracy and the highest scientific standards in our use of names and facts. When things are wrong, let's correct them. Let us not perpetuate wrong botanical knowledge by accepting its incorrect use on commercial products, in everyday language, or in other parts of our contemporary cultures. Through scientific education and specific corrections we will improve botany and science for everybody, in supermarkets, restaurants, and garden centers.

It is the perpetuation of incorrect facts that are the problem, not the need for correction. Everybody makes mistakes, and everybody learns, throughout their lives. We need words to be able to communicate and talk about things, so let's use the right words and the right species names.

Undoubtedly there will be mistakes and errors on this blog, or things that need to be updated. If you want to get in touch, please e-mail me at botanicalaccuracy at gmail.com. If you see a mistake on this blog, e-mail me with the link to the post and an explanation what is incorrect and should be updated. If you represent a company and want to get in touch, please use the same e-mail address.