Predicting Analytics Salaries in India: A 3-Day Predictive Modeling Case Study

The field of analytics and data science continues to be hot, with strong demand and competitive salaries for professionals with the right skills. But how much do analytics professionals actually make? What are the key factors that drive analytics salaries in the market today?

To answer these questions, we set out to build a web application that could predict analytics salaries based on a user‘s background – and do it all in just 3 days! In this post, we‘ll describe how we built this predictive model, from data preparation to model development to deployment. Along the way, we‘ll share key insights about the analytics market uncovered by the model.

Motivation

The motivation for this project came during a casual team discussion. We realized that we had collected over 23,000 data points from analytics professionals in India over the past year through our interactions and platform. This rich dataset included detailed profiles with demographics, educational background, skills, and salaries.

Rather than let this data sit idle, we thought, why not put it to use to uncover insights about the analytics market in India? And better yet, could we use it to build a predictive model to estimate analytics salaries based on a few key inputs?

We knew it would be an ambitious project, especially with our goal of launching a web application in just 3 days! But our team was excited to put our skills to the test and see what we could produce.

Available Data

Our initial dataset contained 23,451 profiles of analytics professionals, spanning data scientists, machine learning engineers, predictive modelers, business intelligence analysts, and more. For each professional, we had over 30 variables with information such as:

• Demographics: age, gender, city
• Education: degree, major, college tier
• Skills: programming languages, tools, statistical methods
• Professional: job title, industry, years of experience
• Compensation: salary, bonuses, stock options

However, before we could begin modeling, we needed to prepare and clean the data. This is a critical step that is often underestimated in data science projects. In our case, we faced several specific challenges:

• College names were highly inconsistent, written in dozens of variations. We needed to standardize names and categorize colleges into consistent tiers.

• Skills were entered as free-form text, often with many skills jumbled together. We needed to parse the text and categorize skills into a consistent taxonomy.

• Salary and experience data contained many outliers and missing values that needed to be handled appropriately.

To address these issues, we used several data transformation techniques:

• For colleges, we used fuzzy string matching to map variations of the same college to a single canonical name. We then manually classified the cleaned names into tier 1 vs. tier 2 categories.

• For skills, we used natural language processing to parse the text and extract individual skills. We then clustered the thousands of unique skills into a smaller set of archetypes based on co-occurrence. Finally, we classified each person based on their closest skill archetype.

• For numeric outliers, we used standard outlier detection methods such as median absolute deviation. For missing values, we tested both dropping rows vs. imputation with the feature mean.

After rigorous data cleaning, we were left with 19,234 complete profiles with 18 variables for modeling. We were now ready to start the modeling process!

Building the Model

Given the short timeline, we opted to use a regression tree as our initial modeling approach. Regression trees have several advantages:

• Highly interpretable and able to capture non-linear relationships
• Robust to outliers and missing values
• Able to handle a mix of categorical and numeric features
• Computationally efficient to train and easy to implement in a web application

We used a standard regression tree from the scikit-learn library in Python. We chose mean absolute error as our performance metric and used 5-fold cross validation to assess model accuracy.

To determine the optimal tree depth, we used grid search over a range of values. We found a depth of 8 to be optimal, balancing performance and generalization. Our final model achieved a cross-validated mean absolute error of ₹37,000, which is quite good given the wide range of salaries.

In addition to predictive accuracy, the regression tree also provided informative insights about the key drivers of analytics salaries:

• Experience had the highest importance, with salaries rising rapidly in the first 5 years but then plateauing.

• Certain skills, such as deep learning and big data tools, commanded significantly higher salaries than others.

• Tier 1 college graduates earned a substantial premium of ₹1.2 lakhs per year on average.

• Location also played a role, with Bangalore and Mumbai beating out other major metros.

These insights aligned with our intuition but were now backed up by data. The model painted a clear picture of the skills and background that led to top dollars in the analytics market.

Productionizing the Model

With our salary prediction model in hand, the next challenge was to productionize it into an interactive web application. Our goal was to create an engaging experience where users could input their background and skills and see a personalized salary prediction.

To do this, we first needed to modify our blogging platform to support the model. This involved several engineering tasks:

• Creating a new API endpoint to serve predictions from the trained model
• Building a form to capture user inputs in a structured format
• Developing data transformation logic to convert user inputs into a valid feature vector for the model
• Designing an interactive results page to display the predicted salary along with explanations

One technical challenge we faced was mapping user text inputs for skills and colleges to our standardized taxonomies from the training data. To solve this, we used approximate string matching to find the closest valid matches.

With these pieces in place, we then needed to rigorously test the end-to-end application to make sure it was robust to different inputs and edge cases. We spent a full day testing, identifying bugs, and iterating until we had a product that worked smoothly.

The Big Launch

After 3 days of intense work, the moment of truth had arrived. On a Saturday morning, we launched the Analytics Salary Predictor to the world!

We promoted the application through social media, community forums, and our email list. The response was immediate and overwhelming – in the first 48 hours, over 20,000 people tried the application!

The salary predictor clearly struck a chord. We received dozens of positive comments and thanks from the community. Users found the personalized predictions useful for understanding their current market value and planning their careers.

From a marketing perspective, the salary predictor was a big success in building awareness and goodwill for Analytics Vidhya. Thousands of people were exposed to our brand, with many going on to explore our other offerings.

Lessons Learned

This whirlwind 3-day project taught us several valuable lessons about rapid prototyping and deployment of data science applications:

1. Deeply understand your data upfront, including structure, quality issues, and potential gotchas. Invest heavily in data exploration and cleaning to avoid problems downstream.

2. Choose a modeling approach that aligns with your end goal and constraints. Favor simpler, more interpretable models over complex ones for faster development.

3. Plan ahead for productionization from the start. Think through the end-user experience, required API and UI changes, and testing needs.

4. Leverage tools and libraries as much as possible to move fast. Don‘t reinvent the wheel when existing solutions will suffice.

5. Test, test, test! Walk through the application with a new user‘s mindset to catch experience and robustness bugs.

6. Launch is just the first step. Have a plan to monitor, maintain, and improve the application based on real-world usage.

Looking back, I‘m incredibly proud of what our small team was able to accomplish in such a short timeframe. The Analytics Salary Predictor was not only an innovative application of data science, but also drove real value for our users and business.

Future Work

While we‘re pleased with the initial launch, we see many areas for future improvement and expansion:

• Enriching the dataset with additional sources like LinkedIn, GitHub, and Kaggle to capture more dimensions of a candidate‘s skills and achievements

• Experimenting with more sophisticated modeling techniques, such as random forests and gradient boosting, to improve accuracy

• Adding more granular features, such as company size and founding year, to enable more targeted predictions

• Building salary predictors for other geographies outside India, where we have sufficient data

• Layering additional insights and recommendations on top of the salary prediction, such as skill gaps to focus on and companies that are a good fit

We believe the Analytics Salary Predictor is just the first step in harnessing data to help professionals navigate their careers. We‘re excited to push this concept further and uncover new opportunities. Stay tuned for what‘s next!