Pioneering Data Science for Social Good: Dr. Avik Sarkar‘s Inspiring Journey

The field of data science has seen explosive growth in recent years, with companies across industries leveraging data-driven insights to drive innovation and gain a competitive edge. However, the power of data science extends far beyond boosting profits and efficiency. In the right hands, data science can be a force for solving some of society‘s greatest challenges.

One individual who embodies this potential is Dr. Avik Sarkar, the Head of Data Science at NITI Aayog, the policy think tank of the Government of India. Dr. Sarkar‘s journey is a testament to the transformative power of data science when applied to initiatives that serve the greater good.

A Lifelong Passion for Mathematics and Statistics

Dr. Sarkar‘s love affair with numbers began early in life. From his school days, he displayed a keen interest and aptitude for mathematics. This passion led him to pursue a bachelor‘s degree in statistics, followed by a master‘s in applied statistics and informatics from the prestigious Indian Institute of Technology, Bombay. Not one to rest on his laurels, Dr. Sarkar went on to earn a PhD in computer science and statistics.

His academic journey, particularly his master‘s thesis and doctoral research, laid the groundwork for his future endeavors in data science. For his master‘s thesis, Dr. Sarkar tackled the challenge of multi-topic text classification, a prescient choice given the impending explosion of online content in the early 2000s. He developed a hierarchical clustering algorithm that could efficiently categorize articles into a topic taxonomy, even if an article was relevant to multiple topics. This work drew upon techniques from machine learning, information retrieval, and natural language processing.

Dr. Sarkar‘s doctoral research ventured deeper into the realm of natural language processing (NLP), focusing on text mining and statistical modeling of text distribution. He developed novel probabilistic models, such as the Latent Dirichlet Allocation (LDA) model, to uncover hidden thematic structures in large text corpora. These techniques have found wide application in areas like sentiment analysis, content recommendation, and topic modeling.

Applying Data Science to Tackle India‘s Challenges

Prior to joining NITI Aayog, Dr. Sarkar honed his data science skills in senior roles at global companies like Accenture, IBM, and Nokia Siemens. However, it was at NITI Aayog where he found the opportunity to apply his expertise to projects with far-reaching social impact.

As the Head of Data Science at NITI Aayog, Dr. Sarkar leads a horizontal analytics team that works across a wide range of domains, from energy and agriculture to healthcare and education. The scope and variety of the projects are vast, ranging from long-term scenario modeling to real-time analysis of survey data.

One of the key challenges the team tackles is making sense of operational data to paint an accurate picture of the state of the Indian economy. This involves complex scenario modeling, taking into account myriad variables to forecast production and manufacturing in critical industries like oil and automobiles. For instance, the team uses techniques like time-series forecasting and Monte Carlo simulations to model different scenarios and their potential impacts on key economic indicators.

But the team‘s work isn‘t limited to long-term planning. They also leverage data science to address more immediate, operational challenges. In the fight against malnutrition, for example, the team has developed machine learning models that analyze data on malnutrition rates, socioeconomic indicators, and program interventions to identify the districts most in need of additional resources. By providing policymakers with these data-driven insights, the team is helping to optimize the allocation of limited resources for maximum impact.

Another area where Dr. Sarkar‘s team is making a significant impact is in streamlining the survey process. Traditionally, there has been a lag of 2-3 years between conducting surveys and extracting actionable insights from the data. The NITI Aayog data science team is working to close this gap by leveraging techniques from natural language processing and machine learning to automate the analysis of survey responses. This includes tasks like entity recognition, sentiment analysis, and topic modeling, which can quickly surface key themes and trends from large volumes of unstructured text data.

Overcoming Data Challenges in India

While the potential for data science to drive positive change is immense, the journey is not without its challenges, particularly in the Indian context. One of the biggest hurdles Dr. Sarkar and his team face is data quality and accessibility.

Much of the data the team works with comes from operational systems that were not designed with analytics in mind. As a result, the data often suffers from quality issues like missing values, inconsistent formats, and duplication. Data cleaning and pre-processing can consume a significant portion of the team‘s time and resources. A study by IBM found that data scientists spend nearly 60% of their time on data preparation tasks, highlighting the magnitude of this challenge[^1].

[^1]: IBM. (2016). The Quant Crunch: How the Demand for Data Science Skills is Disrupting the Job Market. https://www.ibm.com/downloads/cas/3RL3VXGA

But the challenges don‘t end there. Even getting access to the data in the first place can be a herculean task. Due to concerns around data privacy and security, many government agencies are hesitant to share their data, even with other departments. This siloed approach to data management can greatly hamper the ability to derive cross-cutting insights. A survey by MeitY and Nasscom found that data accessibility and sharing is one of the top barriers to AI adoption in India[^2].

[^2]: NASSCOM, MeitY. (2021). India‘s AI Adoption Index. https://nasscom.in/knowledge-center/publications/indias-ai-adoption-index

The lack of comprehensive, high-quality data also raises the specter of model bias. When data is incomplete or skewed, the resulting models can perpetuate or even amplify existing biases. This is a critical concern in the context of AI for social good, where biased models can lead to inequitable outcomes. Dr. Sarkar‘s team is acutely aware of this challenge and is working to develop frameworks and guidelines to ensure the responsible and ethical use of AI in government.

The Toolkit for Data-Driven Governance

To tackle these challenges and deliver data-driven insights, Dr. Sarkar‘s team relies on a robust toolkit of analytics technologies. For long-term energy modeling projects, they employ specialized tools like ‘Message Models‘ and ‘Times Markel Model‘. These tools allow the team to simulate complex energy systems and forecast demand and supply under different policy and technology scenarios.

When it comes to creating visualizations and dashboards to communicate insights to state governments, the team turns to popular business intelligence platforms like Tableau, Qlik, and Power BI. These tools allow for the creation of interactive, data-rich dashboards that can be easily understood by non-technical audiences. The team also makes heavy use of programming languages like R and Python for statistical modeling, machine learning, and data wrangling tasks.

But the team‘s success is not just about the tools they use. It‘s also about the diverse skill sets and backgrounds they bring to the table. The interdisciplinary nature of the team, with experts in statistics, computer science, and public policy, enables them to approach problems from multiple angles and devise comprehensive solutions. This kind of cross-functional collaboration is essential for tackling the complex, systemic challenges that NITI Aayog aims to address.

Envisioning an Inclusive AI Future for India

Looking to the future, Dr. Sarkar sees both immense potential and significant challenges for AI adoption in India. The key, he believes, is to strive for "AI for all" – to ensure that the benefits of these technologies are inclusive and accessible to all segments of society.

In healthcare, for example, AI and automation could help alleviate the burden on overworked medical professionals and extend quality healthcare to underserved rural areas. Machine learning models could assist in tasks like diagnostic imaging analysis, predicting patient risk, and optimizing resource allocation. However, realizing this potential will require overcoming challenges around data standardization, privacy, and the digital divide in healthcare access.

Agriculture is another sector where AI could drive significant improvements. India is an agrarian economy, with over 50% of the workforce engaged in agriculture[^3]. However, productivity and efficiency in this sector lag behind global benchmarks. Dr. Sarkar‘s team is working on AI-powered solutions to provide farmers with timely, actionable insights on crop health, weather patterns, and market trends. By analyzing satellite imagery and sensor data, machine learning models can predict crop yields, detect pest infestations, and recommend optimal planting and irrigation schedules.

[^3]: World Bank. (2022). Employment in agriculture (% of total employment) – India. https://data.worldbank.org/indicator/SL.AGR.EMPL.ZS?locations=IN

But realizing the full potential of AI in India will require more than just technological solutions. It will require a concerted effort to address the digital divide, enhance data literacy, and create an enabling ecosystem for AI innovation. This includes initiatives to boost digital infrastructure, especially in rural areas, and to equip the workforce with the necessary skills for an AI-powered economy. It will also require the development of robust governance frameworks to address concerns around data privacy, security, and ethics.

A Call to Action for the Data Science Community

Dr. Avik Sarkar‘s journey is a powerful reminder of the potential for data science to drive positive social change. His work at NITI Aayog showcases how data-driven insights can inform policy decisions, optimize resource allocation, and ultimately improve the lives of millions.

But this is just the beginning. The challenges India faces, from poverty and inequality to climate change and public health crises, are complex and multifaceted. Tackling them will require the collective efforts of the data science community.

This is a call to action for data scientists, machine learning engineers, and AI researchers to look beyond commercial applications and consider how their skills can be applied to projects with social impact. Whether it‘s working with government agencies, collaborating with NGOs, or developing open-source tools for public good, there are myriad opportunities to make a difference.

The road ahead is not easy, but the potential rewards are immense. By harnessing the power of data science for social good, we can accelerate India‘s development, unlock the country‘s vast potential, and create a more inclusive, prosperous future for all.

As Dr. Sarkar‘s journey shows, it all starts with a passion for using data to make a difference. Let that be the guiding light for the data science community as we work towards an AI-powered future that benefits all of society.