Missing The Point: The Iowa AI Book Banning Brouhaha

As AI pragmatists and freedom-to-read advocates, yes, of course we all saw the headlines last week: "Iowa School Uses AI to Ban Books." If you somehow missed the story, it was covered everywhere from local media in Des Moines, education news, and even a brief article in Smithsonian Magazine.

To summarize, Iowa passed a restrictive new law banning any book with any mention of "a sex act" from K-12 schools. School administrators were made responsible for scanning all of the books in their schools' libraries and classrooms to purge undesirable books from the shelves. One overwhelmed school administrator in Mason City, IA decided to use ChatGPT to help, by feeding it a list of controversial books and asking it whether each book contained a sex act that would require the book to be pulled.

Any librarian who's taken one of our workshops or read our articles about AI "hallucinations" can tell you why this is a terrible, horrible, awful idea: ChatGPT is NOT a factual database with an accurate transcript of each book, so it simply can't be trusted for this sort of decision-making.

From the Des Moines Register article:

Additionally, there are still concerns about the reliability of the information AI systems like ChatGPT generate, [American Association of School Librarians President Courtney] Pentland said. "Instead, I would recommend that school districts rely on school librarians, who are specifically trained experts, to provide insight into materials selection and review," she said.

From EducationWeek:

“We can’t rely on AI as a definitive tool, or maybe even a great tool, until it’s better developed,” she [Deborah Caldwell-Stone, director of the American Library Association’s Office for Intellectual Freedom] told Education Week.

From the Smithsonian article:

The staff of Popular Science repeated this process themselves to see if they could replicate the results. In their tests, ChatGPT gave contradictory answers about the 19 books the school district removed, suggesting the chatbot may not be the most accurate tool for the job.

You can read the Popular Science results yourself here.

But my heart really sank after seeing so many people - including the original administrator who made the decision, explaining her actions in a New York Times op-ed - completely miss the point! Yes, many of the surface-level news articles focused on the novel use of an AI-based tool for determining which books to ban. AI is a popular topic, book banning is in the news a lot (unfortunately), so the intersection of those two topics was irresistible bait to news organizations. But the point is that ChatGPT specifically is not an accurate factual database, so incorrectly using it as one to implement policy decisions is guaranteed to fail.

From her op-ed, where she defends her actions from accusations that she "used AI to ban books," she says:

I’d argue that we used A.I. to try to keep books. The technology allowed us to zero in only on those books that violated the precise letter of the law without collateral damage to other books, enabling us to keep on the shelf more than 30 titles we feared might be questionable.

But that's just it: the technology does not allow you to zero in on books because the ChatGPT model does not accurately represent the content of those books! It's "close enough" but also just makes stuff up, as evidenced by the Popular Science tests above.

To be clear, it is possible to use AI-based tools to analyze the content of books, but ordinary ChatGPT is not the tool to use. AI technologists use a technique called "retrieval-augmented generation" to "feed" books and other data into an AI model for analysis. You can read more about how this works with our "Anne of Green Gables" example in this blog post.

To be even MORE clear, I think it's reprehensible to ban books at all, regardless of the technology used to do it.