Reading & Parsing JSON Data With Python: A Comprehensive Tutorial

JSON (JavaScript Object Notation) has become the de facto standard for storing and exchanging data on the web. With the rise of APIs and big data, JSON is now everywhere – web services use it, mobile apps use it, and modern databases like PostgreSQL support it.

As a Python developer, you‘ll inevitably need to handle JSON data. Luckily, Python has excellent support for both parsing JSON into native Python objects and generating JSON from Python objects.

In this comprehensive tutorial, we‘ll cover everything you need to know about handling JSON in Python, including:

• What is JSON and why it‘s useful
• JSON syntax and data structures
• The JSON module in Python‘s standard library
• Loading JSON from strings or files
• Parsing JSON into Python objects like dicts and lists
• Dumping Python objects into JSON
• Custom JSON encoding and decoding
• Tips for working with real-world JSON APIs

So let‘s get started!

What is JSON and Why Use It?

JSON stands for JavaScript Object Notation. It‘s a lightweight text format for representing structured data that is designed to be easy for humans to read and write, and easy for machines to parse and generate.

Here‘s an example of some data represented as JSON:

{
  "name": "John",
  "age": 30,
  "hobbies": ["reading", "hiking", "coding"]
}

As you can see, JSON looks a lot like Python dictionaries and lists. But there are some key differences:

• JSON requires double quotes around keys and strings. Single quotes are invalid.
• JSON has no equivalent to Python‘s None or null types. The closest thing is null in JSON.
• JSON has fewer allowed data types than Python – just objects (dicts), arrays (lists), strings, numbers, booleans, and null.

Some advantages of using JSON include:

• Human readable and editable as plain text
• Very lightweight and compact syntax
• Easy for machines to parse and generate
• Supported out of the box in many programming languages
• Become ubiquitous for web APIs and data interchange

For these reasons, JSON has largely replaced XML as the most common format for sharing or transmitting data between systems.

JSON Syntax and Structures

Before we dive into parsing and generating JSON in Python, let‘s briefly cover the syntax and primitive data structures that make up JSON documents.

Objects

JSON objects are the equivalent of Python dict objects – they are an unordered collection of key-value pairs.

{
  "name": "John",
  "age": 30
}

Object keys must be strings enclosed in double quotes. Values can be any JSON primitive type – strings, numbers, booleans, arrays, objects, or null.

Arrays

JSON arrays correspond to Python list objects – an ordered sequence of JSON primitives.

[1, 2, 3, 4, 5]

["a", "b", "c"]

Strings

JSON strings must be enclosed in double quotes. Some special characters like newlines and tabs must be escaped with backslashes.

"Hello world"

"escaping \n \t characters"

Numbers

JSON supports both integers and floating point numbers. There is no distinction between the two types.

1

3.141592

-10

Boolean

JSON booleans are the keywords true and false.

true

false

Null

In JSON, null represents no value or a missing value. It is roughly equivalent to Python‘s None.

null

That covers the basic JSON syntax and structure. With this foundation, let‘s look at how to work with JSON in Python.

Python‘s JSON Module

Python‘s built-in JSON module provides all the tools you need to work with JSON data. It is available without any additional installations in the standard library.

The json module has four main functions:

• json.load() – Parses a JSON document from a file-like object.
• json.loads() – Parses a JSON document from a string.
• json.dump() – Serializes a Python object to a JSON string and writes it to a file.
• json.dumps() – Serializes a Python object to a JSON string.

So in summary:

• loads() and dumps() work with JSON strings
• load() and dump() work with file-like objects

The load functions parse JSON into Python objects. The dump functions serialize Python objects to JSON.

Let‘s look at examples of using each of these functions.

Loading JSON from Strings

Most often your JSON data will come from an external source like a web API and will already be in a string format.

The json.loads() method parses a JSON string and returns a Python object:

import json

json_string = ‘{"name": "John", "age": 30}‘

python_dict = json.loads(json_string)

print(python_dict)
# {‘name‘: ‘John‘, ‘age‘: 30}

json.loads() can handle JSON objects, arrays, and any other valid JSON structures. Some things to note:

• The JSON string must be a valid JSON document or you‘ll get a ValueError.
• JSON objects get parsed into Python dicts
• JSON arrays become Python lists
• JSON strings, numbers, booleans, and null become equivalent Python types

This makes it easy to work with JSON from APIs in Python. For example, you can make a request to a REST API that returns JSON data, then parse the JSON string directly:

import requests
import json

response = requests.get(‘http://api.example.com/people/1‘)

if response.ok:
  json_data = json.loads(response.text)
  print(json_data[‘name‘])

So json.loads() provides a straightforward way to convert JSON documents into native Python data structures.

Loading JSON from Files

In some cases, you may have JSON saved in files that you want to load into Python objects.

The json.load() method can parse JSON from a file. It works much like json.loads() except it takes a file object instead of a string:

import json

with open(‘data.json‘) as f:
  data = json.load(f)

print(data) 

You can use Python‘s built-in open() function to get a file object, pass that to json.load(), and get a parsed Python object back.

Some things to watch out for when loading JSON from files:

• The file must contain valid JSON or you‘ll get a ValueError
• Make sure to open the file in read mode ‘r‘
• This approach can handle large JSON files incrementally rather than having to load the entire file into memory at once.

So in summary, use json.load() to read JSON documents from files and json.loads() to parse JSON strings.

Dumping Python Objects to JSON

Now let‘s look at serializing Python objects to JSON with json.dump() and json.dumps(). This process is also called JSON encoding.

Dumping to JSON Strings

The json.dumps() method takes a Python object and returns a JSON string representation of that object:

import json 

python_dict = {
  ‘name‘: ‘John‘,
  ‘age‘: 30,
  ‘grades‘: [90, 88, 92]
}

json_string = json.dumps(python_dict)

print(json_string)
# {"name": "John", "age": 30, "grades": [90, 88, 92]}

You can pass any JSON-encodable Python object like dict, list, str, int, float, True, False or None.

Some things to look out for when dumping to JSON strings:

• Any object that isn‘t JSON-encodable will raise a TypeError
• You can use indent and separators kwargs to format the JSON output
• This is useful for generating JSON for web APIs or sending JSON data

So json.dumps() provides an easy way to convert Python objects to JSON strings.

Dumping JSON to Files

To serialize a Python object as JSON to a file, use the json.dump() method. It works the same as json.dumps() but writes the output to a file-like object:

import json

data = {
  ‘name‘: ‘John‘,
  ‘age‘: 30
}

with open(‘data.json‘, ‘w‘) as f:
  json.dump(data, f)  

Some pointers for dumping JSON to files:

• Open the file in write mode ‘w‘. This will overwrite existing contents.
• Use indent to make the output more readable.
• Handy for saving Python data to share as JSON.

So in summary, use json.dump() to serialize Python objects to JSON files and json.dumps() to get JSON strings.

Custom JSON Encoding and Decoding

Python‘s JSON module handles all the built-in types by default. But what if you want to encode custom Python objects as JSON, or decode JSON into custom objects?

For these cases, the JSON module supports:

• Custom JSONEncoders for encoding objects to JSON
• Custom JSONDecoders for decoding JSON to objects

Custom Encoding to JSON

Let‘s look at an example encoding a custom Python object to JSON:

from datetime import datetime

class User:
  def __init__(self, name, joined_at):
    self.name = name
    self.joined_at = joined_at

user = User(‘John‘, datetime(2022, 1, 1))

# Fails with default encoder
json.dumps(user) 

# But we can make a custom encoder
class UserEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, User):
            return {
                ‘name‘: o.name, 
                ‘joined_at‘: o.joined_at.isoformat()  
            }

        # Default handling for other types
        return super().default(o)

json.dumps(user, cls=UserEncoder)
# ‘{"name": "John", "joined_at": "2022-01-01T00:00:00"}‘

So to summarize, making a custom JSON encoder involves:

• Subclassing json.JSONEncoder
• Implementing the .default() method
• Checking the object type and returning a JSON-encodable representation
• Passing the custom encoder to json.dumps()

This allows you to adapt how custom objects get converted to JSON without having to modify those classes.

Custom Decoding from JSON

You can also use custom decoders to deserialize JSON to custom Python objects.

This is useful when you expect JSON data with a certain structure that you want to instantiate into classes immediately.

Here‘s an example decoding some JSON representing a user into a User class instance:

import json
from datetime import datetime

class User:
  def __init__(self, name, joined_at):
    self.name = name
    self.joined_at = joined_at

class UserDecoder(json.JSONDecoder):
  def __init__(self):
    super().__init__(object_hook=self.dict_to_user)

  def dict_to_user(self, d): 
    return User(
        name=d[‘name‘], 
        joined_at=datetime.fromisoformat(d[‘joined_at‘])
    )

json_data = ‘{"name": "John", "joined_at": "2022-01-01T00:00:00"}‘

user = json.loads(json_data, cls=UserDecoder)
print(type(user)) # <class ‘__main__.User‘>

The steps for making a custom JSON decoder are:

• Subclass json.JSONDecoder
• Set the object_hook kwarg in init to point to the decoding method
• Implement the decoding logic in object_hook method
• Pass the decoder class to json.loads()

This gives you full control over parsing JSON into whichever Python objects you need.

Tips for Working with JSON APIs

To wrap up, here are some useful tips for working with JSON from web APIs in Python:

• Use the requests module to make API calls and get JSON responses
• Validate expected JSON structure with a JSON schema validator
• Handle pagination for endpoints that return paginated data
• Use custom encoders and decoders for API objects
• Cache API responses to avoid hitting rate limits
• Use asyncio for making concurrent API requests
• Handle authentication for APIs that require API keys or OAuth
• Read API documentation for usage limits, endpoints, and parameters

And remember – the json module provides everything you need for most tasks involving JSON data in Python. So leverage json.loads(), json.dumps(), and the other functions covered to get the most out of APIs returning JSON.

Conclusion

I hope this tutorial gave you a comprehensive overview of working with JSON data in Python. Here‘s a quick summary of what we covered:

• What is JSON – lightweight text format for data interchange
• JSON syntax – objects, arrays, primitives
• Python‘s JSON module – loads, dumps, load, dump
• Parsing JSON – json.loads(), json.load()
• Generating JSON – json.dumps(), json.dump()
• Custom encoding & decoding – with custom Encoder and Decoder classes
• Tips for handling JSON APIs – requests module, caching, pagination, etc

JSON has become the standard for web services and data transfer thanks to its simple syntax and widespread support. And Python makes it easy to work with JSON thanks to its robust json module.

So you‘re now equipped to handle JSON in your Python projects – integrating web APIs built on JSON, parsing and creating JSON files, sharing data in JSON format, and more. The possibilities are endless!