Predicting Supreme Court Judgements with Machine Learning

The Supreme Court of the United States is the highest federal court in the country and the final arbiter on matters of law and the Constitution. The court‘s rulings can have profound and long-lasting impacts on American society, shaping everything from individual rights to the powers of the government. Given the immense influence of the Supreme Court, there is great interest among legal scholars, policymakers, and the public in understanding and even anticipating the court‘s decisions.

Traditionally, legal experts have relied on their deep knowledge of the law, familiarity with past cases, and understanding of the justices‘ judicial philosophies to forecast potential rulings. However, in recent years, researchers have begun applying cutting-edge artificial intelligence (AI) and machine learning (ML) techniques to the challenge of predicting Supreme Court judgements in a more data-driven, quantitative manner.

The Supreme Court by the Numbers

To appreciate the scale and complexity of Supreme Court decision-making, it‘s helpful to review some key statistics:

• The Supreme Court receives 7,000-8,000 case petitions ("cert petitions") per year
• Of these, the court grants a writ of certiorari and hears oral arguments in only about 80 cases
• Cases heard by the court are often highly contentious matters where the law is unclear
• The court issues formal written opinions in around 70-80 cases per term
• A typical case includes a majority opinion, and may also have concurring and dissenting opinions from individual justices
• The court‘s composition changes over time as justices retire and new justices are appointed

This selective docket means that the cases the Supreme Court chooses to hear tend to be legally complex and difficult matters where strong arguments exist on both sides. If the correct outcome was obvious based on established law, the court would be less likely to grant certiorari. This inherent legal ambiguity makes predicting judgements challenging from the outset.

Applying Machine Learning to Supreme Court Predictions

Despite the inherent difficulties, AI and ML researchers believe that sophisticated computational models trained on prior Supreme Court cases can uncover predictive patterns and forecast future decisions with a high degree of accuracy. The general process involves collecting data on past cases, extracting key features, training ML models to predict case outcomes based on those features, and interpreting the models to understand what factors drive decisions.

Data and Features

The first step is to assemble a comprehensive dataset of Supreme Court cases on which to train predictive models. Fortunately, the court‘s proceedings and decisions are extensively documented and publicly available. Researchers have curated large datasets, such as the Supreme Court Database (SCDB) which contains detailed information on over 8,400 cases from 1946 to 2020.

Key data collected for each case typically includes:

• Case background: parties involved, lower court actions, relevant legal provisions
• Oral argument transcripts
• Written opinions (majority, concurring, dissenting)
• Final case outcome (affirm/reverse lower court)
• Votes of individual justices

With this raw data in hand, the next step is feature engineering – transforming the data into a format suitable for ML models. This process aims to extract the key predictive signals while discarding irrelevant noise.

Much of the predictive power likely lies in the text data – the specific language used in case filings, oral arguments, and written opinions. To convert unstructured text to meaningful features, researchers employ natural language processing (NLP) techniques such as:

• Tokenization: Splitting text into individual words
• Lemmatization: Reducing words to their dictionary form (e.g. "arguing" to "argue")
• Part-of-speech tagging: Labeling words as nouns, verbs, adjectives, etc.
• Named entity recognition: Identifying references to people, places, organizations
• N-grams: Considering phrases in addition to single words (e.g. "Supreme Court", "equal protection")

More advanced language modeling approaches go beyond just counting words to capture deeper semantic meaning and context:

• TF-IDF: Term frequency-inverse document frequency quantifies the importance of each word to a document compared to the full corpus
• Topic modeling: Algorithms like Latent Dirichlet Allocation (LDA) uncover abstract semantic themes across documents
• Word embeddings: Neural models like word2vec and GloVe learn to map words to high-dimensional vector spaces that encode meaning and relationships

Applying these NLP techniques can uncover informative features from legal texts, such as:

• Key themes and topics (search and seizure, due process, federalism)
• Sentiment and tone (certainty, deference to precedent)
• Semantic similarity to previous cases
• References to specific legal doctrines or precedents
• Statements of judicial philosophy

In addition to text features, structured data fields like the court term, circuit court, legal issue area, and participating justices provide valuable context. Category fields are typically one-hot encoded, while numerical variables may be scaled or discretized.

Feature engineering is both an art and science – it requires a combination of deep legal domain knowledge to identify potentially meaningful variables as well as technical skill in NLP and ML to actually extract those signals from raw, unstructured text data.

Predictive Models

With a cleaned and featurized dataset, the next step is training ML models to predict Supreme Court case outcomes (either overall case disposition or how individual justices will vote). This is fundamentally a classification task, predicting a discrete target variable based on a set of input features.

Many classical ML models are well-suited for this type of tabular data prediction:

• Logistic regression
• Naive Bayes
• Support vector machines (SVMs)
• Decision trees and random forests
• Gradient boosted trees (e.g. XGBoost)

In recent years, more advanced deep learning models have also been applied to legal prediction tasks:

• Convolutional neural networks (CNNs): Excel at extracting local patterns and features from text
• Recurrent neural networks (RNNs): Can model the sequential structure of language
• Transformers: Use attention mechanisms to capture long-range dependencies in text
• Graph neural networks (GNNs): Can model citations and relationships between cases

Typically, researchers will test a variety of model architectures and systematically compare their performance on historical data using metrics like:

• Accuracy: Percentage of predictions that match actual outcomes
• Precision and recall: Ability to avoid false positives and false negatives, respectively
• F1 score: Harmonic mean of precision and recall
• ROC AUC: Measures discrimination ability across different prediction thresholds
• Cross entropy/log loss: Captures both prediction accuracy and probabilistic calibration

K-fold cross-validation is commonly used to get a more robust estimate of generalization performance on unseen data. The dataset is split into K subsets, with each subset taking a turn as the held-out validation set while the model is trained on the remaining data. Final model performance is then averaged across the K validation folds.

Hyperparameter tuning techniques like random search and Bayesian optimization can further improve model performance by finding the optimal model configuration (e.g. the number of trees in a random forest or the learning rate of a neural net).

Model Interpretation

In high-stakes domains like law, understanding the "why" behind model predictions is crucial. Legal scholars and practitioners need to know what factors the model identifies as most predictive of case outcomes to assess their face validity and connect them to legal theories and doctrines.

Techniques for interpreting ML models include:

• Feature importance: Quantifies the contribution of each input feature to model predictions. Permutation importance is a model-agnostic approach that measures the drop in performance when a feature is randomly shuffled.

• Partial dependence plots (PDPs): Visualize the marginal effect of each feature on the model‘s predicted outcome, averaging across the values of all other features.

• SHAP (SHapley Additive exPlanations): Unified approach to explaining individual predictions by assigning each feature an importance value for a particular prediction.

• Attention mechanisms: In neural models with attention layers, the learned attention weights indicate which words or phrases the model is focusing on when making a prediction.

• Adversarial examples: Slightly perturbing input features and observing the change in prediction can reveal which features are most important, as well as potential vulnerabilities of the model.

Interpreting model predictions is not always straightforward, especially for complex neural architectures. A first-order approximation is to examine the model‘s learned feature representations and coefficients to understand which textual features and metadata variables are most predictive. However, more advanced interpretation techniques are an active area of ML research.

Qualitative error analysis also plays a key role – examining cases where the model predictions were incorrect and reasoning about potential causes. This can uncover subtle distinctions or contextual factors that the model may be missing.

Notable Research and Results

Numerous studies in recent years have applied ML to Supreme Court prediction, steadily pushing the boundaries of predictive accuracy.

• Katz et al. (2017) used extremely randomized trees to predict case outcomes and justice votes, achieving 70.2% accuracy at the case level and 71.9% at the justice level.

• Kaufman et al. (2019) applied deep learning, using recurrent neural networks and transfer learning to achieve 74% accuracy on case outcomes from 1946-2015.

• Yang et al. (2019) developed a hierarchical attention network to predict outcomes while also identifying the most predictive sentences in case opinions, achieving 73.2% accuracy.

• Zheng et al. (2021) used a combination of language models and structured metadata features in a multi-task learning setup to achieve 75% accuracy on judgement prediction from 1791-2020.

Study	Model	Feature Set	Time Period	Case Outcome Acc.	Justice Vote Acc.
Katz 2017	Extremely randomized trees	Textual (TF-IDF), metadata	1946-2015	70.2%	71.9%
Kaufman 2019	Recurrent neural net	Word embeddings	1946-2015	74.0%	–
Yang 2019	Hierarchical attention net	Word embeddings, metadata	1946-2019	73.2%	–
Zheng 2021	Multi-task learning	BERT embeddings, citation graph	1791-2020	75.0%	80.3%

While these results demonstrate the potential of ML models to predict Supreme Court decisions with reasonable accuracy, it‘s important to note several limitations:

• Model accuracy is still well below 100% and lags human legal expert performance, which can reach 75-85% accuracy depending on the case type and information provided.
• Performance tends to be lower for more recent cases, indicating that models trained on historical data may struggle to generalize to new legal and social contexts.
• The most predictive features often relate to the identity and ideological leanings of justices rather than the specific legal merits of the case, raising questions about what the models are really capturing.
• There is inherent randomness and unpredictability in close Supreme Court decisions, placing an upper bound on achievable accuracy.

Future Directions and Implications

As ML techniques continue to advance, there are many exciting directions for future research on Supreme Court prediction:

• Developing modular architectures that can incorporate both NLP models for text and structured models for background knowledge and context
• Improving few-shot learning to adapt to changing court composition and shifting legal norms over time
• Modeling the hierarchical structure of the court system to share information across SCOTUS and lower courts
• Leveraging multi-task learning to jointly predict case outcomes, justice votes, and opinion content
• Generating plausible synthetic case opinions or dissents for legal counterfactual analysis

Beyond simply predicting outcomes, ML models could potentially uncover latent patterns and inconsistencies in Supreme Court rulings, surfacing areas where the court‘s interpretation of the law has shifted over time. As models achieve greater accuracy and interpretability, they may also become practical tools for lawyers and judges to stress-test legal arguments and identify relevant precedents.

However, it‘s crucial to approach the application of predictive models to the law with caution. There is a risk that placing too much emphasis on "algorithmic jurisprudence" could undermine fundamental legal principles like fairness and impartiality. The role of the Supreme Court is not simply to be predictable, but to carefully reason about each case and faithfully interpret the Constitution.

Moreover, if judicial decision-making starts to be influenced by predictive models trained on historical rulings, it could lead to feedback loops and path dependencies in case law. Imagine a future where justices feel pressure to rule in certain ways to avoid deviating too far from ML predictions and maintain an aura of consistency, even if those predictions are flawed.

ML should be seen as a complement to, not a substitute for, human legal judgement. Predictive models are ultimately tools for uncovering empirical patterns and pressure-testing intuitions. They cannot replace the role of human values, expertise, and argumentation at the heart of judicial reasoning.

Interdisciplinary collaboration between legal scholars, computer scientists, and ethicists is needed to thoughtfully integrate ML into the practice of law in a way that preserves its core tenets. Only by balancing the power of data-driven prediction with the wisdom of human judgement can we build a legal system that is both consistent and just.