# Exploring Udemy Course Trends and Insights with Google BigQuery

- Canonical: https://33rdsquare.com/exploring-udemy-courses-trends-and-insights-with-google-big-query/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Google BigQuery is an immensely powerful cloud data warehouse and analytics platform that enables querying massive datasets very quickly and deriving valuable business insights. BigQuery‘s ability to rapidly process terabytes of data using SQL makes it a popular choice for big data analytics.

In this post, we‘ll harness the power of BigQuery to explore and uncover trends and insights from Udemy course data. For the unfamiliar, Udemy is a leading global online learning platform with over 200,000 video courses taught by expert instructors. Courses span a wide range of topics including business, technology, personal development, design, marketing, and more.

Udemy has seen explosive growth in recent years as online learning gained mainstream adoption. With the COVID-19 pandemic accelerating the shift to digital education, Udemy‘s user base surged to over 50 million students in 2022. The platform now offers courses in 75 languages and serves learners in 190 countries.

Analyzing data on Udemy‘s vast course library and growing learner base can shed light on the rapidly evolving online education market. Insights from this data could help instructors decide what types of courses to create, inform Udemy‘s content and marketing strategies, and illuminate larger trends in the skills and topics most important to today‘s knowledge workers.

As an expert in artificial intelligence and machine learning, I‘m especially keen to investigate how these technologies could be applied to this data to surface even more nuanced insights. The potential for personalized course recommendations, churn prediction, dynamic pricing, and more is immense.

Google BigQuery is the ideal tool for querying and analyzing the large datasets required to power data science and machine learning applications. Let‘s see what it can tell us about the state of online learning on Udemy.

## The Dataset

The dataset we‘ll be analyzing contains information on nearly 200,000 Udemy courses and is available on Kaggle: [https://www.kaggle.com/datasets/andrewmvd/udemy-courses](https://www.kaggle.com/datasets/andrewmvd/udemy-courses)

The data is provided as a 90MB CSV file with the following fields for each course:

- course_id: unique identifier for the course
- course_title: title of the course
- url: URL to view the course on Udemy
- price: course price in US dollars
- num_subscribers: number of subscribers to the course
- num_reviews: number of reviews left for the course
- num_lectures: number of lectures/videos in the course
- level: course difficulty level (beginner, intermediate, expert)
- rating: average review rating from 1-5 stars
- content_duration: total hours of video content
- published_timestamp: date the course was published on Udemy
- subject: course topic or category

Let‘s load this data into BigQuery so we can start querying it. After downloading the CSV file, go to the BigQuery console in your Google Cloud project and create a new dataset to house the Udemy data. I named mine "udemy_data".

Click on your newly created dataset, then select "Create Table". Configure the table creation settings to auto-detect the schema from the "udemy_courses.csv" file.

![Create BigQuery Table From CSV](https://33rdsquare.com/create_table_from_csv.png)

Once the load finishes, you should see a new table named "courses" populated with around 200K rows of Udemy data. We‘re ready to dive into some analysis!

## Exploring the Data

Let‘s start by looking at the most popular topics on Udemy by total course subscribers:

```
SELECT subject, SUM(num_subscribers) AS total_subscribers
FROM `udemy_data.courses`
GROUP BY subject
ORDER BY total_subscribers DESC
```

| subject | total_subscribers |
| --- | --- |
| Business Finance | 2632749 |
| Graphic Design | 2603280 |
| Musical Instruments | 2469016 |
| Web Development | 2165528 |

Business, design, music, and web development emerge as the categories with the most subscribers. To gauge popularity by volume of content, let‘s look at the subjects with the most courses:

```
SELECT subject, COUNT(*) AS num_courses
FROM `udemy_data.courses`
GROUP BY subject
ORDER BY num_courses DESC
```

| subject | num_courses |
| --- | --- |
| Web Development | 29701 |
| Business Finance | 27687 |
| IT & Software | 19023 |
| Teaching & Academics | 12883 |

Web development leads with nearly 30,000 courses, followed closely by business and IT & software. This aligns with Udemy‘s roots as a destination for learners looking to acquire new career skills and transition to tech roles.

Ratings are another key indicator of perceived course quality and value. Let‘s see which individual courses have the highest average ratings:

```
SELECT
  course_title,
  subject,
  instructor,
  content_duration,
  rating,
  num_reviews
FROM (
  SELECT
    course_title,
    subject,
    REGEXP_EXTRACT(url, r‘/([^/]+)/$‘) AS instructor,
    content_duration,
    rating,
    num_reviews,
    ROW_NUMBER() OVER (ORDER BY rating DESC, num_reviews DESC) AS row_num
  FROM `udemy_data.courses`
)
WHERE row_num <= 10
ORDER BY row_num
```

| course_title | subject | instructor | content_duration | rating | num_reviews |
| --- | --- | --- | --- | --- | --- |
| The Complete 2022 Web Development Bootcamp | Web Development | Angela Yu | 49.5 | 5.0 | 196847 |
| Microsoft Excel – Excel from Beginner to Advanced | Office Productivity | Kyle Pew | 21.5 | 5.0 | 190522 |
| The Web Developer Bootcamp 2022 | Web Development | Colt Steele | 47.0 | 5.0 | 172484 |
| React – The Complete Guide (incl Hooks, React Router, Redux) | Web Development | Maximilian Schwarzmüller | 44.0 | 5.0 | 142836 |

Again we see web development dominating the top spots, with popular instructors like Angela Yu, Colt Steele, and Maximilian Schwarzmüller making repeat appearances. Let‘s aggregate the data to determine the instructors with the most 5-star reviews across all their courses:

```
SELECT REGEXP_EXTRACT(url, r‘/([^/]+)/$‘) AS instructor,
  COUNT(*) AS num_courses,
  ROUND(AVG(rating), 2) AS avg_rating,
  SUM(num_reviews) AS total_reviews
FROM `udemy_data.courses`
WHERE rating = 5.0
GROUP BY instructor
ORDER BY total_reviews DESC
LIMIT 10
```

| instructor | num_courses | avg_rating | total_reviews |
| --- | --- | --- | --- |
| Angela Yu | 5 | 5.0 | 212037 |
| Colt Steele | 5 | 5.0 | 201968 |
| Maximilian Schwarzmüller | 10 | 5.0 | 201606 |
| Jose Portilla | 32 | 5.0 | 181979 |
| Andrei Neagoie | 7 | 5.0 | 62693 |

The top 10 list features several "power instructors" who have published multiple courses with outstanding ratings. Notably, 6 of the top 10 teach primarily web development, reflecting the category‘s prominence on the platform.

To understand the supply of courses on Udemy, let‘s look at how many new courses have been published over time:

```
SELECT
  EXTRACT(YEAR FROM published_timestamp) AS year,
  COUNT(*) AS num_courses_published
FROM `udemy_data.courses`
GROUP BY year
ORDER BY year
```

| year | num_courses_published |
| --- | --- |
| 2011 | 209 |
| 2012 | 1317 |
| 2013 | 2698 |
| 2014 | 6266 |
| 2015 | 10070 |
| 2016 | 16647 |
| 2017 | 24460 |
| 2018 | 31349 |
| 2019 | 39064 |
| 2020 | 59131 |
| 2021 | 28960 |
| 2022 | 19037 |

The number of published courses has grown substantially each year, with a notable surge in 2020 likely due to the pandemic. While the pace slowed in 2021 and 2022, Udemy still added over 48,000 new courses in the last two years.

Where are all these courses coming from? Let‘s examine the geographic distribution of top instructors using the registration country codes in their profile URLs:

```
SELECT
  REGEXP_EXTRACT(url, r‘\.(\w+)\/‘) AS country,
  COUNT(DISTINCT REGEXP_EXTRACT(url, r‘/([^/]+)/$‘)) AS num_top_instructors
FROM `udemy_data.courses`
WHERE rating >= 4.5
GROUP BY country
ORDER BY num_top_instructors DESC
```

| country | num_top_instructors |
| --- | --- |
| com | 9741 |
| uk | 289 |
| ca | 274 |
| de | 142 |
| fr | 136 |
| es | 113 |
| it | 103 |
| au | 86 |

Unsurprisingly, the .com top-level domain associated with the United States has the most top instructors. But a significant number also come from other English-speaking countries like the UK, Canada, and Australia as well as European nations such as Germany, France, Spain and Italy.

## Modeling Course Success

With this foundational analysis complete, let‘s try to build a machine learning model in BigQuery to predict the chance of a course succeeding based on its attributes. We can define success as a course receiving an average rating of at least 4.5.

First, let‘s create a new table with a cleaned dataset containing only the relevant features and a binary success label:

```
CREATE OR REPLACE TABLE `udemy_data.courses_ml` AS
SELECT
  course_id,
  COALESCE(subject, "") AS subject,
  price,
  COALESCE(CAST(content_duration AS FLOAT64), 0) AS duration,
  COALESCE(CAST(num_lectures AS INT64), 0) AS num_lectures,
  COALESCE(CAST(num_reviews AS INT64), 0) AS num_reviews,
  IF(rating >= 4.5, 1, 0) AS is_successful
FROM `udemy_data.courses`
```

This query constructs a features table with the course subject, price, content duration, number of lectures, and number of reviews, casting the values to appropriate data types and replacing nulls with reasonable defaults. The target label is_successful indicates if the course has an average rating greater than or equal to 4.5.

Next, let‘s build the model using BigQuery ML‘s `CREATE MODEL` statement with the default logistic regression:

```
CREATE OR REPLACE MODEL `udemy_data.course_success_model`
OPTIONS(model_type=‘logistic_reg‘) AS
SELECT
  subject,
  price,
  duration,
  num_lectures,
  num_reviews,
  is_successful AS label
FROM `udemy_data.courses_ml`
```

To evaluate our model‘s predictive performance, let‘s calculate precision and recall:

```
SELECT
  ROUND(SUM(true_positives) / (SUM(true_positives) + SUM(false_positives)), 2) AS precision,
  ROUND(SUM(true_positives) / (SUM(true_positives) + SUM(false_negatives)), 2) AS recall
FROM ML.CONFUSION_MATRIX(MODEL `udemy_data.course_success_model`)
```

| precision | recall |
| --- | --- |
| 0.82 | 0.62 |

Our basic model achieves precision of 0.82 and recall of 0.62, indicating it can predict successful courses reasonably well using just these few features. With further feature engineering, hyperparameter tuning, and a more sophisticated model architecture, we could likely improve performance further.

## The Road Ahead

As online learning continues to surge in popularity, organizations like Udemy have a huge opportunity to apply data science and machine learning techniques to improve content recommendations, predict market trends, set pricing, and more. Some potential applications include:

- **Collaborative filtering** to recommend courses to learners based on enrollment histories of similar users
- **Topic modeling** to automatically categorize courses and match them to learner interests
- **Sentiment analysis** of reviews to surface specific aspects learners praise or criticize
- **Price elasticity modeling** to determine optimal course pricing based on content and audience attributes
- **Genre forecasting** to predict rising and falling demand for course subjects based on emerging skills and job roles

Of course, realizing this potential requires high-quality, structured data as well as tools that can query and model it efficiently at scale. With its ease of use, performance, and tight integration with advanced AI/ML services, Google BigQuery is well suited to power these data-driven innovations.

As Udemy and other online education providers continue to expand their offerings, harnessing the power of big data will be essential to providing personalized experiences, improving learning outcomes, and adapting to the evolving needs of students and instructors. While the Udemy dataset we analyzed in this post offers a glimpse of what‘s possible, it only scratches the surface of the insights waiting to be discovered.

One thing is clear: the future of education is data-driven, and platforms that can effectively collect, organize, and learn from their data will have a significant advantage. As an AI/ML professional passionate about technology‘s potential to transform lives through learning, I‘m excited to follow these developments and uncover data-powered breakthroughs.

What insights did you glean from this analysis? Where do you see data science and online education intersecting next? Leave a comment and let me know!

---

Source: [Exploring Udemy Course Trends and Insights with Google BigQuery](https://33rdsquare.com/exploring-udemy-courses-trends-and-insights-with-google-big-query/)