This is an interesting article, analyzing GPT-3 against a Turing Test. It does quite well, and proves hard to beat. One thing that is obvious is that the intelligence is very capable in general, but its range of responses to the Q & A test is over-constrained by the example prompt, causing it to fail at some challenges that it can easily pass when given a different prompt.
20200728/https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html
That being said, a few obvious and undeniable weaknesses are worth pointing out. Very straightforward, rational, quantitative, and logical reasoning questions cannot be accurately answered. For example, answering questions about the relative weight of objects, it does well most of the time but it has a few notable failures. And, performing arithmetic and simple list manipulations, the AI totally falls on its face when prompted with this. On the other hand, the AI shows that it is curiously good at converting English descriptions of computations to Ruby and Python programs that go and do likewise.
So, that being said, let me put in my own reflection. Unfortunately, the state-of-the-art of artificial intelligence has become a precise mirror of our own human weaknesses. Humans, on the mass market general average, are excellent at generating entertainment materials like fiction novels and “fake news,” but they suffer greatly when it comes to more precise, quantified, organized tasks. Why? It’s simply due to a lack of motivation. Humans don’t want to do quantitative reasoning. It’s not that they can’t do it, it’s that they don’t want to do it.
Interestingly, the very basic element of quantitative reasoning doesn’t really require good math reasoning skills. It’s merely a matter of being able to always think about assigning a number to what you are doing, to count up what you are doing, to label it, and organize it into categories. But alas, this is what your average human resists so greatly. By simply avoiding this type of practice like an expert procrastinator, they effectively never practice the skill, and by simply not practicing the skill, they can act like they are artificially poor performers at a skill which honestly, from an intellectual standpoint, is a easy skill for humans master. What it all comes down to is not forming the required habit.