How to use Python and AI for sentiment analysis: Step-by-step tutorial

Introduction: Understanding sentiment analysis

If you've ever tried to read through hundreds of customer reviews or social media comments to figure out what people actually think about your product, you know how exhausting it can be. People share their opinions constantly across social media, review sites, and forums, but reading through all that text manually is practically impossible, but incredibly important!

That's where sentiment analysis comes in. It's basically a way to use AI to automatically figure out whether a piece of text expresses positive, negative, or neutral feelings. Instead of spending hours reading reviews, you can let a computer do the heavy lifting and tell you what people are actually saying.

In this tutorial, I’m going to talk about how I use a combination of Python and LLMs in a Fabi.ai Smartbook to perform sentiment analysis efficiently and reliably. Python is great at working with structured data and LLMs are great at processing human language. If you want to get right to the meat of it, you can skip ahead to the how-to section where I’ve also included links to a short tutorial video.

‍

What is sentiment analysis?

Sentiment analysis, also known as opinion mining, is a way to automatically figure out the emotional tone of written text. It can tell you whether someone is expressing positive, negative, or neutral feelings about something.

Think of it like having a really fast reader who can go through thousands of customer reviews or social media posts and summarize how people feel about your product. Instead of manually reading through hundreds of comments to see if customers are happy or frustrated, sentiment analysis does this work for you automatically.

When I was a product manager, we often ran user surveys with a freeform input. I had a comb through all the comments to figure out the pattern then manually distill the information. Then if I wanted to build a report off of this and leverage some of the structure inputs, I was out of luck. For example if I wanted to create a grouped bar chart of sentiment by user role, I would have a very hard time.

Types of sentiment analysis

Polarity-based analysis: Classifies text as positive, negative, or neutral 

Emotion-based analysis: Identifies specific emotions like joy, anger, fear, or surprise

Aspect-based analysis: Determines sentiment toward specific features or aspects of a product or service

What is customer sentiment analysis?

Customer sentiment analysis specifically is just sentiment analysis focused specifically on what customers think about your business. This means looking at things like:

• Reviews on Amazon, Yelp, or other platforms
• Comments on your social media posts
• Conversations with customer support
• Survey responses
• App store reviews

The neat thing about customer sentiment analysis is that it goes deeper than star ratings. Someone might give you 4 stars but still complain about specific problems in their written review. Sentiment analysis picks up on those details that ratings alone might miss. In this tutorial we’re going to use an example of a survey response in plain English for our fictional online fashion retailer Superdope.

‍

Why use Python for sentiment analysis?

Python has become the most popular language for sentiment analysis, and there are some really good reasons why:

Lots of helpful libraries

Even though we’re not actually going to use a library in this tutorial, it’s worth noting that Python does have tons of libraries specifically built for text analysis:

• NLTK: Great for all kinds of text processing
• TextBlob: Super simple to use for basic sentiment analysis
• VADER: Works really well with social media text
• spaCy: Fast and professional-grade text processing
• Transformers: Gives you access to the latest AI models

As mentioned above, large language models (transformers) are uniquely well suited for natural language processing. Most of us don’t want to build or spin up our own LLM though for this type of exercise. It’s incredibly expensive and difficult. The fastest and cheapest approach is to leverage a third-party LLM such as ChatGPT, Gemini or Claude.

Easy to learn and read

Python code looks almost like regular English, which means you can focus on solving your problem instead of fighting with complicated syntax. Even if you're relatively new to programming, you can pick up Python pretty quickly.

Great for handling data

Python works really well with large amounts of data, which is exactly what you need when analyzing thousands of reviews or social media posts.

‍

Why combine AI with Python for sentiment analysis?

Python is great for structured data analysis. It’s the most popular data analysis and data science language for good reason. Especially with the help of AI, you can easily build reports and dashboards in a matter of minutes. On the other hand, LLMs are perfect for natural language processing. They’re trained on reams of data and can easily capture the nuance of the human language in ways that more rudimentary models and Python libraries simply can’t.

The challenge comes when you try to combine structured data with an LLM. If you’ve ever uploaded a CSV file to an LLM and asked it to categorize a set of reviews, you’ve likely run into any one of the following issues:

• Hallucination: It may give you a table back, but if you look closely you’ll likely notice that it modifies entries in your table. If you have a table with a handful of fields such as “Name”, “Survey response”, “Age” and “Country”, the LLM will not only look at the “Survey column”, it will look at all fields and regenerate the data. This immediately introduces the risk of hallucination.
• Maintaining structure: If you’re looking to build a dashboard leveraging this sentiment data, you need to guarantee that the data will consistently be output in the same format with the same field names. If you ask the AI to process 10 fields and create a new field called “Sentiment category”, you need to know that it’s going to do that every single time.
• Scale: Not all models can support large files. And the larger the file, the higher the risk of hallucination. The answer to this is typically to split the file into smaller batches, but this can be a lot of work.

At Fabi.ai, we introduced AI cells. These cells can accept a Python DataFrame generated from an uploaded file or a query to your database and a natural language prompt. These AI cells will perform batch inference and produce another Python DataFrame which you can then use for further analysis. In the tutorial below, we’ll be leveraging this feature.

Sentiment analysis: Step-by-step tutorial

Prerequisites

To get started, you should have:

• Basic Python skills and understanding of Python DataFrames
• A Fabi.ai account (you can create one in 1 minute here)

And that’s it! Let’s dive int.

Preparing your data

If your data is stored in a database or data warehouse, you can connect your data to Fabi.ai. You’ll follow the same steps we cover here, but you’ll simply query your data rather than upload a file. In Fabi.ai, when you query data from a data warehouse, it automatically saves it as a Python DataFrame.

‍

Alternatively, as I do in this tutorial, you can simply have the data you want to process as a field in a CSV. In my case, I have my Superdope product reviews saved under a review field in my product_reviews.csv file.

‍

Create a new Smartbook, then either upload your file or create a SQL cell and query your data.

Stringing together Python DataFrames and LLMs

Once you’ve uploaded your file or queried your data from Fabi.ai, it should be stored as a DataFrame. If this is the first cell in your Smartbook, the DataFrame is likely named dataframe1. You can see the name under the cell that you’re using to query or load your data.

‍

Now, below that cell, insert a new AI Enrichment cell and input the following information:

• Input dataframe name: {your_dataframe_name} (eg. dataframe1)
• Column for AI to process: {your_natural_language_field} (eg. reviews)
• Output dataframe name: {your_output_dataframe_name} (eg. processed_data)
• Column containing the AI output: {new_field_name} (eg. sentiment_categories)

‍

And finally, you can enter any prompt in the prompt field. For example, to process the customer product reviews, I simply asked the AI to “Categorize each review as "Positive", "Negative" or "Neutral".”

Further analyzing your sentiment categories in Python

Once you have your new DataFrame with the new AI-generated field that contains your sentiment categories, you can further analyze that DataFrame using more “traditional” Python functionality.

‍

For example if you want to count the number of reviews by sentiment before charting you, you can simply run:

# Create a copy of the dataframe
sentiment_counts_df = processed_reviews.copy()

# Count the number of reviews by sentiment category
sentiment_counts = sentiment_counts_df['sentiment_category'].value_counts().reset_index()
sentiment_counts.columns = ['Sentiment', 'Count']

# Sort by count in descending order
sentiment_counts = sentiment_counts.sort_values('Count', ascending=False)

Python + AI = Perfect combo for sentiment analysis

As you hopefully saw with this tutorial, with tools like Fabi.ai, it’s becoming easier than ever to use the right tool for the job, which in the case of sentiment analysis means using AI for sentiment categorization, and Python for the rest.

Using AI and python for sentiment analysis is a powerful way to get valuable insights into your business to help guide you and your team. Whether you’re analyzing customer survey data, product review data, or any other type of natural language, doing so can help you better understand where to invest your time and money to grow the business.

If you’re trying to perform sentiment analysis using Python today, you can get started with Fabi.ai in less than 5 minutes for free.

‍