Bots Outperform Humans at Solving CAPTCHA
A group of scientists has published a study titled “An Empirical Study and Evaluation of Modern CAPTCHA,” claiming that bots are now better at solving CAPTCHA tests than humans. Researchers from the University of California, Irvine, ETH Zurich, Lawrence Livermore National Laboratory, and Microsoft conducted an experiment to analyze human performance on various CAPTCHA tests. They asked 1,400 people to complete 14,000 tests. Each participant had to solve 10 types of CAPTCHAs: reCAPTCHA, reCAPTCHA v2, two game-based CAPTCHAs from Arkose Labs (one using arrows to rotate an object, another requiring the selection of a vertical object), two hCAPTCHAs (one with simple and one with complex settings), a slider-based CAPTCHA from Geetest, and three with different types of text distortion.
The researchers found no correlation between the time it took to solve a CAPTCHA and how participants rated the test. In other words, people didn’t always prefer the tests they could complete faster. There was an interesting correlation between participants’ age and education level and the time spent solving CAPTCHAs: in many cases, people with advanced degrees solved the tests faster (other groups showed roughly the same results); age increased the time needed by 0.09 seconds for each additional year of life.
At the end of the study, the scientists compared human performance on CAPTCHAs with that of bots, using data from other sources. The results showed that humans performed worse than machines: for example, when people were asked to solve distorted text CAPTCHAs, it took them 9 to 15 seconds and they answered correctly only 50% to 84% of the time. Bots, on the other hand, completed the same tasks in less than a second with an accuracy of 99.8%.
Comparison of CAPTCHA Completion Time and Accuracy: Humans vs. Bots
Additionally, the researchers analyzed 200 popular websites and found that 120 of them use CAPTCHAs. The most common was reCAPTCHA, present on 68 sites and owned by Google.
One of the scientists, Professor Gene Tsudik, shared his thoughts with Time magazine about the experiment’s results: “Why are we still using a technology that is almost universally disliked, so costly (especially in terms of wasted human time), and ineffective against bots?” Tsudik suggested that part of the reason is that CAPTCHA providers, including Google, “make a lot of money selling their services and don’t want to voluntarily give up their business models.” He also suspects that companies using CAPTCHAs to protect services like account creation are reluctant to “admit that using CAPTCHAs is ineffective.”