Scala for Machine Learning: Reading CSVs and Building Models

ByJordan Brown

In recent years, Scala has seen rapid adoption in the data science and machine learning communities. According to the 2022 Stack Overflow Developer Survey, Scala ranked as the 13th most popular programming language, with 5% of respondents using it regularly [1]. Scala‘s appeal for data scientists stems from several key strengths:

  • Its concise, expressive syntax and strong static typing, which promote correct, maintainable code
  • The ability to leverage existing Java libraries for data processing and ML, like Apache Spark
  • Powerful functional programming abstractions that simplify working with large datasets
  • Easy scalability from single-node prototypes to distributed production deployments

In this post, we‘ll walk through a typical machine learning workflow in Scala, starting with loading data from CSV files and ending with a trained model. We‘ll place particular emphasis on techniques for effectively reading and preprocessing structured data, which is often the most time-consuming part of any data science project.

Reading CSV Files in Scala

Comma-separated value (CSV) files are a common format for storing tabular data. While not as efficient as binary formats like Parquet or Avro, CSVs are simple, human-readable, and universally supported. Scala provides several libraries for reading and writing CSV data.

The most basic approach is to use Scala‘s built-in scala.io.Source class to read the file line-by-line:

val input = Source.fromFile("data.csv")
val lines = input.getLines.toSeq
val data = lines.map(_.split(",").map(_.trim))
input.close()

However, this doesn‘t give you any help with parsing the individual values, handling headers or quotes, or dealing with malformed rows. For non-trivial datasets, it‘s usually better to use a dedicated CSV library. One good option is Kantan CSV, which provides a type-safe, declarative API for working with CSVs:

import kantan.csv._
import kantan.csv.ops._

case class Row(id: Int, name: String, score: Double)

val input = new File("data.csv")

// Read the entire CSV as a List[Either[ReadError, Row]]
val data: List[Either[ReadError, Row]] = 
  input.asCsvReader[Row](rfc.withHeader).toList

// Filter out any invalid rows and unwrap the values
val rows: List[Row] = data.collect { case Right(row) => row }

Kantan can automatically parse the CSV data into a case class or other custom data type, while properly handling different separators, quotes, NULL values, and so on. It also provides convenient methods for filtering and transforming the data during the loading process.

Another popular choice is spark-csv, which integrates with Spark SQL to read and write CSVs as DataFrames:

val spark: SparkSession = ...
val df = spark.read
  .option("header", "true")
  .option("inferSchema", "true")
  .csv("data.csv")

df.printSchema()
df.show(5)

Using Spark can be a good choice for larger datasets that exceed the memory of a single machine. It also allows you to take advantage of Spark‘s SQL interface and built-in functions for data manipulation.

Whichever library you choose, there are a few best practices to keep in mind when working with CSV data in Scala:

  • Explicitly define your schema using case classes or Spark StructTypes, rather than relying on type inference
  • Be aware of NULL values and have a plan for handling them, e.g. by using Option types or sentinel values
  • Pay attention to column names and types, making sure they match what you expect
  • Watch out for messy, real-world data issues like extra commas, mismatched quotes, inconsistent line endings, etc.
  • For very large files, consider streaming the data or using a binary format like Parquet instead of CSV

Preprocessing Data for Machine Learning

With our data loaded, the next step is typically to preprocess it into a format suitable for training a model. This might involve tasks like:

  • Filtering out irrelevant or low-quality samples
  • Imputing missing values
  • Encoding categorical variables as numeric features
  • Normalizing or scaling the feature values
  • Splitting the data into training, validation, and test sets

Scala‘s powerful built-in collections and parallel processing capabilities make it a great choice for these sorts of data munging tasks. For example, suppose we have a dataset of customer reviews loaded as a Seq[Row], where each row contains the review text, a rating from 1 to 5, and a timestamp. We can easily compute some aggregate statistics using methods like filter, map, and fold:

val numReviews = reviews.size
val avgRating = reviews.map(_.rating).sum / numReviews.toDouble
val numPositive = reviews.count(_.rating >= 4)

Or we can use Scala‘s pattern matching to count the occurrences of each rating value:

val ratingCounts = reviews.foldLeft(Map.empty[Int, Int]) {
  case (counts, Row(_, rating, _)) =>
    counts + (rating -> (counts.getOrElse(rating, 0) + 1))
}

To visualize the distribution of ratings, we could generate a histogram using a library like Breeze or Vegas:

import vegas._

Vegas("Review Ratings").
  withData(ratingCounts.toSeq).
  encodeX("rating", Nom).
  encodeY("count", Quant).
  mark(Bar).
  show

For more advanced preprocessing, we can leverage Spark‘s machine learning pipelines. For instance, to tokenize and hash the review text as TF-IDF features, impute missing ratings with the mean, and assemble everything into feature vectors, we could write:

import org.apache.spark.ml.feature._
import org.apache.spark.ml.Pipeline

val tokenizer = new Tokenizer()
  .setInputCol("text")
  .setOutputCol("words")

val hashingTF = new HashingTF()
  .setInputCol("words")
  .setOutputCol("rawFeatures")

val idf = new IDF()
  .setInputCol("rawFeatures")
  .setOutputCol("features")

val imputer = new Imputer()
  .setInputCols("rating")
  .setOutputCols("ratingImputed")
  .setStrategy("mean")

val assembler = new VectorAssembler()
  .setInputCols("features", "ratingImputed")
  .setOutputCol("featureVector")

val pipeline = new Pipeline()
  .setStages(Array(tokenizer, hashingTF, idf, imputer, assembler))

val model = pipeline.fit(reviewsDF)
val featuresDF = model.transform(reviewsDF)

This creates a pipeline that combines several feature transformers to produce a final labeled training set. Spark will automatically handle issues like null values and distribute the processing across a cluster if needed.

Training and Evaluating Models

With our feature engineering complete, we‘re ready to train a machine learning model. Scala has a variety of ML libraries to choose from, including Spark MLlib, SMILE, and DeepLearning.scala.

For a large-scale classification task, a good choice is often a ensemble tree method like random forest or gradient boosting. Spark MLlib provides implementations of both:

import org.apache.spark.ml.classification.RandomForestClassifier
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator

val rf = new RandomForestClassifier()
  .setLabelCol("rating")
  .setFeaturesCol("featureVector")
  .setNumTrees(100)

val model = rf.fit(trainingData)

val predictions = model.transform(testData)

val evaluator = new MulticlassClassificationEvaluator()
  .setLabelCol("rating")
  .setPredictionCol("prediction")
  .setMetricName("accuracy")

val accuracy = evaluator.evaluate(predictions)
println(s"Test error = ${1.0 - accuracy}")

Here we‘re fitting a random forest with 100 trees to our training data, generating predictions on the test set, and computing the accuracy. To get a more reliable estimate of real-world performance, we could also use cross-validation:

import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}

val paramGrid = new ParamGridBuilder()
  .addGrid(rf.numTrees, Array(50, 100, 200))
  .addGrid(rf.maxDepth, Array(5, 10, 20))
  .build()

val cv = new CrossValidator()
  .setEstimator(rf)
  .setEvaluator(evaluator)
  .setEstimatorParamMaps(paramGrid)
  .setNumFolds(5)

val cvModel = cv.fit(trainingData)
val bestModel = cvModel.bestModel.asInstanceOf[RandomForestClassificationModel]

This performs a grid search over the number of trees and maximum depth hyperparameters, using 5-fold cross-validation to select the best model. We can also extract the most important features:

val featureImportances = bestModel.featureImportances.toArray.zipWithIndex.map {
  case (imp, idx) => (inputCols(idx), imp)
}.sortBy(-_._2)

featureImportances.foreach(println)

Examining which features the model relies on most heavily can yield useful insights for further feature engineering or data collection.

Conclusion

In this post, we‘ve seen how to use Scala to build a typical machine learning pipeline, starting with loading CSV data and ending with a trained classifier. Of course, there are many additional topics we could cover, such as:

  • Data cleaning and normalization
  • Feature selection and dimensionality reduction
  • Unsupervised learning algorithms like clustering and anomaly detection
  • Deep learning with neural networks
  • Productionizing trained models as microservices

Nonetheless, the basic techniques we‘ve discussed – loading structured data, preprocessing it with functional transformations and ML pipelines, and training and evaluating models – are the core of most data science projects. Scala‘s expressive syntax, strong typing, and rich ecosystem make it a powerful tool for this kind of work.

To learn more about using Scala for data science and ML, check out some of these resources:

You can also find numerous open-source examples and tutorials on GitHub, Kaggle, and the Scala ML ecosystem documentation. Happy learning!

[1] Stack Overflow. Stack Overflow Developer Survey 2022. https://survey.stackoverflow.co/2022/

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts