ChatGPT Still Has a Ways to Go Before Passing Economics 101

As artificial intelligence advances at a rapid pace, you may have heard impressive demos of chatbots like ChatGPT answering complex questions with convincing responses. So you might be shocked to learn that ChatGPT completely bombed a basic economics exam designed by professor Steve Landsburg of the University of Rochester. This failure highlights that while AI tools have come a long way, human-level mastery of conceptual reasoning remains firmly out of reach. As we‘ll see, ChatGPT can mimic human language, but lacks true understanding of economic principles needed to pass Econ 101.

Who is Professor Landsburg and Why is His Exam So Tough?

To understand why ChatGPT fell short, we first need to appreciate why Steve Landsburg‘s introductory economics exam is so challenging. Landsburg has taught economics at Rochester for over 20 years. He is known for emphasizing economic intuition and reasoning over rote memorization.

As Landsburg‘s colleagues describe, his exams are designed to test core concepts:

"Steve‘s exams require students to demonstrate thorough grasp of principles and apply them to analyze novel, complex scenarios," says Professor Amy Davis. "Memorizing definitions isn‘t enough – you need to think quantitatively and logically like an economist."

Landsburg confirms this approach:

"My goal is to evaluate whether students can think in an economically literate way. I want to test their reasoning capacities, not their memories."

With this philosophy, Landsburg has developed an introductory exam that serves as a rigorous test of essential economic thinking. This focus on conceptual understanding is what stumped ChatGPT.

How Does ChatGPT Work and Where Does It Fall Short?

To appreciate why ChatGPT struggled, we need to understand its capabilities and limitations. ChatGPT is a large language model developed by OpenAI using deep learning. It is trained on massive datasets to predict probable next words given previous text. This allows it to generate remarkably human-like responses on almost any topic.

However, under the hood, ChatGPT has no real grasp of the meaning behind words. It cannot truly reason about concepts or make logical inferences outside its training data. It effectively matches patterns between words, without deeper comprehension.

This distinction between generating text and demonstrating comprehension is key. As AI researcher Anima Anandkumar explains:

"ChatGPT has no actual understanding of the semantics and is pattern matching words to words. So of course it will fail on many economics exam questions which require true causal reasoning."

In short, while ChatGPT excels at text generation, it lacks human-level capability for conceptual mastery. Let‘s see how this deficit impacted its performance on Landsburg‘s exam.

Examples of Questions ChatGPT Flunked

Landsburg‘s exam contains a variety of thought-provoking economic analysis problems. Here are two representative examples that ChatGPT famously failed:

Question 5

Prompt:

You are the monopoly owner of an unfenced amusement park where anyone can enter and purchase ride tickets. The demand for ride tickets is Q=100-5P and the marginal cost of providing the rides is P=4Q. You are thinking of building a fence so that you can charge admission to the park. What is the maximum amount you would pay for that fence?

ChatGPT‘s response:

• Correctly calculates optimal ticket price and profit without fence
• Determines consumer surplus with fence is $216.61
• Concludes max to pay for fence is difference in profits = $216.61

Landsburg‘s scoring: 2/10 points

While ChatGPT correctly computed the profit without a fence, it missed the key insight that a monopolist could charge a lower ticket price to increase consumer surplus, thereby justifying a higher spend on the fence. It failed to reason through the dynamics around consumer surplus.

Question 6

Prompt:

A monopoly-owned retail store is located next to a monopoly-owned parking lot…Describe how [the store and parking lot] set prices under various sequencing assumptions.

Key parts of ChatGPT‘s response:

• Incorrect claim that store would adjust demand based on parking price
• Failed to recognize store would lower prices if parking raises prices first
• Vague discussion of "considerations" for joint pricing

Landsburg‘s scoring: 2/10 points

Again this exposes ChatGPT‘s lack of economic reasoning. It made basic errors in describing how two monopolies would adjust pricing, showing no real comprehension of core concepts like demand curves.

ChatGPT scored this poorly across all 9 exam questions. Clearly it has not mastered the causal reasoning and dynamic thinking required in economics. It succeeds only on straightforward computation.

Expert Reaction on the Results

The dismal exam results provoked strong reactions from experts in artificial intelligence and education. Stanford professor Percy Liang remarked:

"It‘s clear we have a long way to go before we can claim real reasoning capability. Models like ChatGPT can parrot lines they‘ve heard before, but lack human judgment."

Some experts pointed out the policy implications:

"This shows we shouldn‘t fear AI systems replacing human roles overnight," said economist Robin Hanson. "There are still many cognitive frontiers where humans have the edge."

Others highlighted the need to revamp educational approaches:

"Rather than relying on ChatGPT, we must focus on developing critical thinking skills in students," noted education reformer Sal Khan. "Problem-based learning is key."

Historical Context on Early AI Systems

To put the results in perspective, it helps to look at previous episodes where AI systems also initially struggled on conceptual reasoning tasks.

When IBM‘s Watson computer first competed on Jeopardy! in 2011, it made several basic errors on categories requiring real-world knowledge. Researchers realized it needed to be trained further.

Similarly, early self-driving car prototypes from the 1980s and 1990s had difficulty recognizing unexpected obstacles on roads. Driving performance was limited to basic scenarios.

In both cases, further advances in data and algorithms over decades eventually improved reasoning capabilities. But mastery required immense effort.

ChatGPT is likely at a similar early stage – impressive potential, but with conceptual gaps that will take years to close. This matches the assessment from OpenAI themselves, with CEO Sam Altman tweeting:

"ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness. It’s a mistake to be relying on it for anything important right now. We have lots of work to do on robustness and truthfulness."

Human vs Artificial Intelligence: Who Wins at Conceptual Reasoning?

Stepping back, experiments like Landsburg‘s economics exam shed light on the differences between human and artificial intelligence.

Humans have a remarkable capacity to learn conceptual knowledge and apply it creatively to tackle novel challenges. Through education, we internalize an intuitive understanding of principles like economics, physics, law or medicine. We can then synthesize those concepts to address new problems. AI today lacks this fluid reasoning ability.

As computer scientist Pedro Domingos says:

"Machine learning models at best capture statistical associations and patterns. But human learning is about gaining real insight into the causal forces at work in the world around us."

For now, conceptual reasoning and expertise remain firmly in humanity‘s court. Creativity, critical analysis, and wisdom depend deeply on true understanding in a way current AI systems do not replicate.

So while machines can beat us at tasks like calculation, chess and trivia, humanity still reigns supreme at higher-order reasoning. Let‘s make sure our education system continues honing these uniquely human cognitive capabilities.

Key Takeaways from ChatGPT‘s Economics Exam Failure

In the race for technological supremacy, it pays to keep some perspective. Results like those from Landsburg‘s exam remind us:

• Advances like ChatGPT are impressive, but still limited compared to human cognition
• True mastery of conceptual reasoning remains out of reach for current AI
• In domains like economics, human experts retain a sizable reasoning edge
• Overhyping AI risks undervaluing human intelligence and education
• Developing critical thinking is key to complement AI‘s text generation skills
• With focus and humility, the best of human and artificial intelligence can be combined

So while ChatGPT fell short on Econ 101, the path ahead is a partnership where each side contributes unique strengths. By maintaining realistic expectations and sound educational foundations, humans remain well-positioned to thrive with AI as a tool rather than a threat.