AI vs. Humans – SEO Experiment
AI vs. Humans – SEO Experiment
Artificial Intelligence (AI) is taking the world by storm. The release of ChatGPT, OpenAI’s Natural Language Processing (NLP) model, has created a massive shift in business, but with it comes a fair share of concerns.
Many worry about how this progressively improving technology fits into a world run and maintained by human content. That's understandable, especially considering how marketing techniques like Search Engine Optimization (SEO) play a significant role in online content that's readily viewable while remaining relatively unregulated.
Marketers have already taken the dive, with some content marketing experts finding that 82% of marketers claim AI content is "just as good as human content."
But where does Google stand in managing AI's exploding popularity and mass production of AI-generated content? How do search engines handle the ranking of human-generated content vs. AI-generated content?
Reboot, an online marketing company in the U.K. known for its industry-changing experiments, set its sights on answering these questions.
Before we dive into the experiment, let’s take a look at the components.
Search Engine Optimization (SEO)
Search Engine Optimization has become a key phrase in a growing empire built on keywords and phrases. Online businesses have focused on using the science behind optimizing content to push their websites, articles or blog posts, landing pages, and other content to the top of the SERPs (search engine results pages). How do they do this?
It's all about trending keywords, careful formatting, and metadata.
With AI becoming increasingly advanced, top-notch SEO content is becoming easier to produce. Marketers can provide prompts for AI programs to:
- Optimize an article
- Implement SEO practices into a pre-written article
- Write an entire SEO-focused piece of content
All this, within seconds, can significantly decrease the time spent on human-written content while minimizing overhead costs.
AI-Generated Content
Natural Language Processing (NLP) models have hit the ground running with advanced conversational chatbots like ChatGPT and Microsoft Bing AI, both capable of producing hundreds of lines of text in seconds and near-instantaneous revisions.
This software works by taking in a prompt from the user and feeding it into an AI program trained on large amounts of text gathered from internet sources, books, magazines, etc.
Next-word prediction is used via an internal ranking system. This seems simple, but the training on such a massive amount of information allows it to output complex, human-sounding text with incredibly high accuracy and precision — fast.
Google’s Stance on AI-generated Content
Google's position on AI-generated content states clearly that content is at the forefront of the company’s focus, whether human- or AI-generated. Content created by AI has been around for a long time — Grammarly is just one example — and can be used to increase quality and efficiency for readers and writers alike.
Google developers also state that AI content isn’t ranked in any particular way. It's rated with the same paradigm as any other type of content: “Useful, helpful, original, and satisfies aspects of E-E-A-T.” Whether human or AI-generated, the "winning" content will rank highest according to E-E-A-T.
But how truthful is this claim? That question was the basis for Reboot’s experiment. Through deep research, statistical testing, and articulate reporting and discussion, Reboot sought to answer that very question.
AI vs. Humans: The Reboot Experiment
The experiment began in the early months of 2023 with the hypothesis that AI- and human-generated content could be ranked differently on Google search engine results. The Reboot team wanted to create a controlled environment where all aspects of their content were the same, except how it was created: human or AI.
The researchers used ChatGPT to create eleven meaningless words with no search history or domain existence. They chose "flemparooni" as the keyword around which the content would be written.
Then, using the remaining ten words, they developed and deployed ten basic websites to host the content articles.
They began the writing process after outlining a 7-step guideline to promote replicability and equal optimization across all content.
Five of their company’s in-house content writers each wrote an article based on the keyword "flemparooni" and the guidelines provided. The researchers then used ChatGPT-4, the latest version of OpenAI’s ChatGPT, to feed the keyword, the guidelines, and the prompts needed to output five articles meeting the specified criteria.
Finally, the team analyzed data based on information collected through keyword position tracking. They also performed tests to determine the significance of the results.
AI vs. Humans: The Results
Interestingly, Reboot found that of the ten articles created (five by humans and five by ChatGPT-4), four of the top-five ranked articles were human-generated, with the only AI-generated article in that group ranking third.
Four of the bottom five ranked articles were AI-generated, with the only human-generated article in that group ranking number nine.
The researchers concluded that AI-generated content held an average ranking of 6.6, whereas human-written content held an average rank of 4.4, showing that content created by humans tended to outrank content created by AI. Data is only as good as its significance, however, and this is where the word "Science" in "Data Science" comes in.
Data Analysis comparing the rankings of websites hosting human-generated content vs. AI-generated content. (Reboot, 2023)
Reboot researchers tested the data using the Mann-Whitney U test.
Omitting the complex mathematical mumbo-jumbo, the Mann-Whitney tests the numerical data against a null hypothesis, in this case, that the probability of AI- and human-generated content are ranked equally is 100%. After some behind-the-scenes calculations, the test returns a number indicative of the difference in mediums (or middle numbers) between the two populations (AI-generated vs. human-generated content).
A higher result indicates websites hosting human-generated articles rank higher than their AI counterparts, and a lower result indicates that AI-generated articles rank higher.
After a total of 25 comparisons, the null hypothesis was rejected about 92% of the time, telling us that the results of the experiment were significant. This means that the results they found didn't happen by chance.
Furthermore, the results showed that ranking tended to favor human-generated content over AI content, assuming total SEO equality throughout all content.
Conclusion
The results of this experiment are interesting, especially when considering the implications.
SEO-based companies and marketers focusing on ChatGPT and other AI marketing tools to maximize content optimization may save on the overhead costs involved in human content creation. But this experiment shows their competitors may maintain an edge if they stick with human-generated content.
A theory is only as good as its replicability, of course, and research should be continued for more in-depth confirmation. However, the results are a good outlook for human content creators and should be a flag for all those online marketing decision-makers pushing for total content automation.
The answer lies somewhere between human and AI content generation.
But one should not supersede the other. Instead, each should be utilized in harmony to promote engaging, human-to-human knowledge and content while maximizing efficiency through AI ideation, outlining, and keyword generation, among other tasks.
Experiments like Reboot's show that we're not ready for complete automation, and companies that think we are will likely do themselves more harm than good.
Humans and AI Should Work Together for You
Your content should be geared toward you and your target audience's needs. AI can't do it alone, and human creators can only do so much in limited amounts of time. Scripted is the only platform offering both human and AI content. We unify the two in a way that maximizes speed and efficiency while providing high-quality, branded content that stands out. With a pool of carefully selected subject matter experts (SMEs) with industry-specific knowledge and a one-stop platform for all your content creation needs, marketers need look no further. AI Empowered, Human Perfected: Craft Authentic Content with Scripted
