Building a Deep Learning-based Crowd Counting Model in Python
Crowd counting is an important and challenging problem in computer vision with many useful applications. The goal is to estimate the number of people in an image or video of a crowd. This has traditionally been done by having humans manually count people, which is time-consuming and not always feasible for very large crowds. As a result, there has been great interest in developing automated crowd counting systems using computer vision and deep learning techniques.
Some key application areas of crowd counting include:
- Public safety and security: Monitoring crowds at airports, train stations, stadiums, protests, etc. to detect overcrowding and potentially dangerous situations
- Retail analytics: Analyzing customer traffic at stores and shopping malls to optimize product placement, staffing, etc.
- Smart cities: Measuring crowds at public spaces and events for urban planning and resource allocation
- Social distancing: Monitoring the density and spacing between people to encourage safe social distancing in pandemic situations
However, crowd counting is a difficult task for computers, as crowd images have high variability in terms of perspective, density, occlusion, lighting, etc. For instance, people far away from the camera appear much smaller than those nearby. In dense crowds, people substantially occlude each other. The density of the crowd can vary considerably in different parts of the same image.
Over the years, researchers have developed various approaches to tackle these challenges:
- Detection-based methods: Use a sliding window to detect individual people in the image and count them. Works well for sparse crowds but struggles with high-density crowds and occlusion.
- Regression-based methods: Directly learn a mapping from global image features to the crowd count. Doesn‘t require detecting individuals but has limited ability to handle large variations in crowd density.
- Density estimation-based methods: Estimate the density of the crowd at each pixel and integrate over the density map. Can handle large variations in density but requires annotated density maps for training, which are more costly to obtain than simple counts.
- CNN-based methods: Use convolutional neural networks to learn both local and global features in an end-to-end fashion. Has become the dominant paradigm in recent years due to powerful feature learning and the ability to directly optimize the counting objective.
In this post, we‘ll focus on a state-of-the-art CNN-based method called CSRNet (Congested Scene Recognition Network). Proposed in 2018, CSRNet uses dilated convolutions to aggregate multi-scale contextual information for estimating high-quality crowd density maps and counts.
CSRNet Architecture
The CSRNet architecture consists of two main components: a front-end feature extractor based on the first 10 layers of the VGG-16 network, and a back-end dilated CNN for generating density maps. The input to the network is an image of arbitrary size and the output is a density map of 1/8 resolution.
The key ingredient in CSRNet is the use of dilated convolutions in the back-end network. Dilated convolutions, also known as atrous convolutions, introduce gaps into the convolutional kernels to increase the receptive field without increasing the number of parameters. Formally, for a 2D signal $x[i]$ and a convolutional filter $w[k]$ of size $K$, the dilated convolution operation $F$ on element $i$ with a dilation rate $l$ is defined as:
$$F[i]=\sum_{k=1}^K x[i+l\cdot k]w[k]$$
When $l=1$, this reduces to a standard convolution. Dilated convolutions with progressively increasing dilation rates can efficiently aggregate multi-scale contextual information.
In CSRNet, the dilation rate in the back-end network increases from 2 to 4 to 8 and so on. This allows the network to generate high-resolution density maps by maintaining spatial information, while still having a large enough receptive field to capture the global context of the crowd.
Generating Ground Truth Density Maps
To train the CSRNet model, we need a dataset with ground truth annotations. A commonly used dataset is the ShanghaiTech dataset, which contains 1198 images with a total of 330,165 labeled heads. However, the labels are provided as a sparse set of point annotations for the center of each head, not as density maps.
To convert the point annotations into density maps, we use Gaussian kernels to blur each head annotation, then sum the Gaussian maps to obtain the final density map. Mathematically, the ground truth density map $D_i$ corresponding to image $X_i$ is generated as:
$$Di(p) = \sum{P \in S_i} \mathcal{N}(p; P, \sigma^2)$$
where $S_i$ is the set of annotated points in image $X_i$, $p$ is a pixel location, $P$ is an annotated point, and $\mathcal{N}(p; P, \sigma^2)$ is a 2D Gaussian kernel with mean $P$ and isotropic covariance $\sigma^2$. The Gaussian spread $\sigma$ is chosen based on the perspective map of the image to compensate for scale variations.
Training the CSRNet Model
With the ground truth density maps available, we can now train the CSRNet model. We‘ll use PyTorch to implement the model and training pipeline.
First, we define the model architecture following the description in the CSRNet paper:
class CSRNet(nn.Module):
def __init__(self, load_weights=False):
super(CSRNet, self).__init__()
self.frontend = make_layers(cfg[0])
self.backend = make_layers(cfg[1])
if load_weights:
mod = models.vgg16(pretrained=True)
self._initialize_weights()
for i in range(len(self.frontend.state_dict().items())):
self.frontend.state_dict().items()[i][1].data[:] = mod.state_dict().items()[i][1].data[:]
def forward(self, x):
x = self.frontend(x)
x = self.backend(x)
return xHere, make_layers is a helper function that constructs a sequence of convolutional layers according to the specified configuration cfg. The front-end layers are initialized with pre-trained VGG-16 weights to leverage transfer learning.
Next, we define the dataset and data loader to feed batches of images and ground truth density maps to the model during training:
class CrowdDataset(Dataset):
def __init__(self, root, transform=None):
self.root = root
self.transform = transform
self.img_paths = sorted(glob(os.path.join(root, "*.jpg")))
self.dens_paths = sorted(glob(os.path.join(root, "*.npy")))
def __len__(self):
return len(self.img_paths)
def __getitem__(self, idx):
img = Image.open(self.img_paths[idx]).convert(‘RGB‘)
dens = np.load(self.dens_paths[idx])
if self.transform:
img = self.transform(img)
return img, densThe CrowdDataset class assumes the images and density maps are stored under the same directory with the same filename prefix.
We then define the training loop with the Adam optimizer and MSE loss:
def train(train_loader, model, criterion, optimizer, epoch):
losses = AverageMeter()
model.train()
for i, (imgs, dens) in enumerate(train_loader):
imgs = imgs.cuda()
dens = dens.cuda()
dens = dens.unsqueeze(1)
optimizer.zero_grad()
output = model(imgs)
loss = criterion(output, dens)
loss.backward()
optimizer.step()
losses.update(loss.item(), imgs.size(0))
print(f"Epoch {epoch} Train Loss: {losses.avg:.4f}")
return losses.avgDuring training, we also periodically evaluate the model on a validation set and save the best checkpoint based on validation loss.
Finally, we can run the training pipeline:
data_root = "./data/ShanghaiTech/part_A/"
train_set = CrowdDataset(os.path.join(data_root, "train_data"), transform)
train_loader = DataLoader(train_set, batch_size=1, shuffle=True)
model = CSRNet().cuda()
criterion = nn.MSELoss(reduction=‘sum‘).cuda()
optimizer = Adam(model.parameters(), lr=1e-6)
for epoch in range(0, 400):
train_loss = train(train_loader, model, criterion, optimizer, epoch)With suitable hyperparameters, the CSRNet model can achieve state-of-the-art performance on the ShanghaiTech dataset, with a mean absolute error (MAE) of around 68 on Part_A and 11 on Part_B.
Evaluating the Trained Model
After training, we can evaluate the performance of the CSRNet model on the test set. For each test image, we predict the density map using the trained model and sum over the density map to obtain the crowd count. We then compute the MAE and mean squared error (MSE) between the predicted and ground truth counts over the entire test set.
def evaluate(model, test_loader):
model.eval()
mae = 0
mse = 0
with torch.no_grad():
for imgs, dens in test_loader:
imgs = imgs.cuda()
dens = dens.cuda()
output = model(imgs)
predict_count = torch.sum(output).item()
gt_count = torch.sum(dens).item()
mae += abs(gt_count - predict_count)
mse += (gt_count - predict_count)**2
mae /= len(test_loader)
mse /= len(test_loader)
mse = mse ** 0.5
print(f"MAE: {mae:.2f}, MSE: {mse:.2f}")
return mae, mseHere are some example visualizations of the predicted density maps and crowd counts:
As we can see, the CSRNet model does a good job of estimating the density maps and counts, even for highly crowded scenes. However, there is still room for improvement, especially for extremely dense crowds and small heads far away from the camera.
Future Work
While CSRNet is a powerful crowd counting model, there are several limitations and potential areas for future research:
-
Incorporating additional contextual information: CSRNet only uses the local image context to estimate the density maps. Incorporating additional context such as global crowd statistics, temporal dependencies in video streams, and cross-scene similarities could potentially improve performance.
-
Improving computational efficiency: The current CSRNet model is relatively slow for real-time applications, requiring about 1 second per image on a high-end GPU. Techniques such as model compression, quantization, and neural architecture search could help reduce the computational cost.
-
Handling extreme density variations: CSRNet still struggles with extremely dense crowds and large perspective changes. More advanced multi-scale fusion techniques and perspective-aware losses may help address these issues.
-
Unsupervised domain adaptation: The performance of crowd counting models often degrades when applied to new scenes and datasets. Unsupervised domain adaptation techniques could help bridge the domain gap and improve cross-dataset generalization.
-
Ensuring fairness and privacy: As crowd counting systems become more widely deployed, it is important to consider issues of fairness, transparency, and privacy. Techniques such as federated learning and differential privacy could help protect individual privacy while still enabling useful crowd analytics.
In conclusion, crowd counting is an important and challenging problem in computer vision with many practical applications. CSRNet is a powerful CNN-based approach that achieves state-of-the-art performance on benchmark datasets. However, there is still much room for improvement and innovation in this field. We hope this post has provided a helpful introduction to crowd counting with deep learning and inspires further research and development in this area.
