Sentiment Analysis with Python and NLTK

Overview

Sentiment analysis scores text as positive, negative, or neutral. NLTK’s VADER (Valence Aware Dictionary and sEntiment Reasoner) is a rule-based model built for social media text. It handles negation, intensifiers, and emoticons without training data. Here is how to scoring individual texts, batch processing, and customizing the lexicon.

When to Use

• You need to classify customer reviews or social media posts
• You want a quick sentiment score without training a model
• You are building a dashboard that tracks sentiment over time
• You need to filter support tickets by urgency or tone

Solution

Install and set up VADER

pip install nltk

import nltk
nltk.download("vader_lexicon")

from nltk.sentiment.vader import SentimentIntensityAnalyzer

sia = SentimentIntensityAnalyzer()

Score a single text

score = sia.polarity_scores("I love this product, it works great!")
print(score)
# {'neg': 0.0, 'neu': 0.536, 'pos': 0.464, 'compound': 0.6249}

score = sia.polarity_scores("Terrible experience, would not recommend.")
print(score)
# {'neg': 0.577, 'neu': 0.423, 'pos': 0.0, 'compound': -0.4767}

The compound score ranges from -1 (most negative) to +1 (most positive). Use it as the aggregate sentiment metric.

Classify sentiment

def classify_sentiment(text):
    score = sia.polarity_scores(text)["compound"]
    if score >= 0.05:
        return "positive"
    elif score <= -0.05:
        return "negative"
    else:
        return "neutral"

texts = [
    "Great service and fast delivery!",
    "The package arrived broken.",
    "It was okay, nothing special.",
]

for text in texts:
    print(f"{classify_sentiment(text):10s} | {text}")

Batch process from CSV

import csv
from nltk.sentiment.vader import SentimentIntensityAnalyzer

sia = SentimentIntensityAnalyzer()

with open("reviews.csv", newline="") as infile, open("scored.csv", "w", newline="") as outfile:
    reader = csv.DictReader(infile)
    writer = csv.DictWriter(outfile, fieldnames=reader.fieldnames + ["sentiment", "compound"])
    writer.writeheader()

    for row in reader:
        score = sia.polarity_scores(row["review"])
        row["compound"] = score["compound"]
        row["sentiment"] = "positive" if score["compound"] >= 0.05 else "negative" if score["compound"] <= -0.05 else "neutral"
        writer.writerow(row)

Handle negation and intensifiers

sia = SentimentIntensityAnalyzer()

print(sia.polarity_scores("The food was good"))
# compound: 0.4404

print(sia.polarity_scores("The food was not good"))
# compound: -0.4404

print(sia.polarity_scores("The food was very good"))
# compound: 0.4927

print(sia.polarity_scores("The food was EXTREMELY good"))
# compound: 0.5671

Customize the lexicon

sia = SentimentIntensityAnalyzer()

# Add domain-specific words
new_words = {
    "buggy": -2.0,
    "crash": -3.0,
    "responsive": 2.0,
    "intuitive": 2.0,
}
sia.lexicon.update(new_words)

print(sia.polarity_scores("The app is buggy and crashes often"))
# Now scores more negative with custom words

Analyze sentiment over time

from nltk.sentiment.vader import SentimentIntensityAnalyzer
from datetime import datetime
import statistics

sia = SentimentIntensityAnalyzer()

posts = [
    {"date": "2026-01-01", "text": "Love the new update!"},
    {"date": "2026-01-02", "text": "Found a bug in the login flow."},
    {"date": "2026-01-03", "text": "Bug is fixed, great support!"},
]

daily_scores = {}
for post in posts:
    date = post["date"]
    score = sia.polarity_scores(post["text"])["compound"]
    daily_scores.setdefault(date, []).append(score)

for date, scores in sorted(daily_scores.items()):
    avg = statistics.mean(scores)
    print(f"{date}: avg={avg:.3f} ({len(scores)} posts)")

Explanation

VADER uses a lexicon of 7,500 words rated by human annotators. Each word has a valence score from -4 (extremely negative) to +4 (extremely positive). VADER combines these scores with five heuristics:

• Punctuation: Exclamation marks increase intensity.
• Capitalization: ALL CAPS increases intensity.
• Degree modifiers: “very” boosts, “somewhat” dampens.
• Negation: “not good” flips the polarity.
• Contrast: “but” shifts focus to the clause after it.

The compound score is a normalized, weighted sum of all lexicon scores in the text. It is the most useful single metric for classification.

Variants

Approach	Training Data	Accuracy	Use When
VADER	None (rule-based)	Good for social media	Quick setup, no training data
TextBlob	Built-in lexicon	Similar to VADER	Simple API, corpus-based
Transformer (HuggingFace)	Pre-trained	High	Production sentiment at scale
Custom classifier	Labeled dataset	Varies	Domain-specific needs

Guidelines

• Use the compound score for classification. It accounts for the full text, not individual words.
• Set thresholds at +0.05 and -0.05 for positive/negative. Adjust based on your data.
• Update the lexicon with domain-specific words. VADER’s default lexicon is built for social media.
• VADER works best on short texts (sentences to paragraphs). For long documents, score paragraph by paragraph.
• Do not use VADER for sarcasm detection. It scores literal word meanings, not implied intent.

Common Mistakes

• Using pos / neg ratios instead of compound. The compound score is normalized and more reliable.
• Not customizing the lexicon for your domain. Words like “sick” mean positive in gaming, negative in healthcare.
• Applying VADER to long documents. It averages sentiment across the whole text, losing local context.
• Ignoring the neu score. A high neutral ratio means the text is mostly informational, not opinionated.
• Comparing VADER scores across languages. VADER is English-only. For Spanish, use pysentimiento or a multilingual transformer.

Troubleshooting

• Model outputs are inconsistent: set temperature to 0 for deterministic tasks, use seed where supported, and version the prompt.
• Prompt injection leaks context: separate user input from system instructions.
• High token costs: cache embeddings, summarize long context, and choose smaller models for simple tasks.
• Retrieval returns irrelevant chunks: tune chunk size, overlap, and metadata filters. Evaluate retrieval metrics separately from generation.
• Evaluation scores do not match human judgment: define clear rubrics, use multiple judges, and track disagreement. Human review is still the ground truth.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply sentiment analysis with python and nltk when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Additional Common Mistakes

• Not handling negation across sentence boundaries — “The food was good. Not really.” VADER scores each sentence independently, missing negation that spans sentences. Process full reviews, not individual sentences.
• Using fixed thresholds for all domains — a 0.05 compound threshold may be too strict for product reviews (which tend to be positive) and too lenient for news articles (which tend to be neutral). Calibrate thresholds per domain using a labeled sample.
• Ignoring the neutral score — a high neu ratio (e.g., 0.9) means the text is mostly informational. These texts should be classified as neutral regardless of the compound score, which may be slightly positive or negative due to noise.
• Not handling repeated punctuation — “Great!!!” scores higher than “Great!” in VADER, which may not be desired. Normalize punctuation before scoring if you want consistent results.
• Comparing VADER scores across different text lengths — VADER’s compound score is normalized, but very short texts (1-3 words) produce extreme scores. Require a minimum text length before scoring.
• Not logging individual word scores — VADER can output per-word valence scores. Log these for debugging unexpected classifications and for building domain-specific lexicons over time.
• Using VADER for sentiment intensity comparison — VADER’s compound score is not linear. A compound of 0.5 is not “twice as positive” as 0.25. Use it for classification, not for ranking sentiment intensity.

Best Practices

• For a deeper guide, see Fine-Tune a Language Model for Code Generation.

• Preprocess text consistently: lowercase all text, normalize whitespace, expand contractions before scoring. This reduces variance in scores for semantically identical inputs.

• Calibrate thresholds per domain: run VADER on a labeled sample of your domain’s text. Plot the distribution of compound scores for positive, negative, and neutral labels. Choose thresholds that minimize misclassification for your specific domain.

• Combine VADER with rule-based filters: use VADER as a first pass, then apply domain-specific rules for known edge cases (e. g. , flip sentiment for detected sarcasm markers, boost sentiment for product-specific positive terms).

• Track sentiment over time: for brand monitoring or product feedback, compute daily/weekly sentiment aggregates. Plot trends to detect shifts in public perception.

• Log scores with context: when storing sentiment results, include the full VADER output (pos, neg, neu, compound), text length, and any preprocessing applied. This enables post-hoc analysis and threshold recalibration.

• Validate against human labels periodically: even if VADER worked well initially, language evolves. Re-validate against fresh human-labeled data every 3-6 months to ensure your thresholds and lexicon are still accurate.

Production Checklist

• Text preprocessing pipeline (lowercase, normalize, expand contractions)
• Domain-specific thresholds calibrated using labeled sample
• Full VADER output logged (pos, neg, neu, compound, text length)
• Minimum text length enforced before scoring
• Punctuation normalized for consistent scoring
• Sentiment trends tracked over time (daily/weekly aggregates)
• Rule-based filters applied for known edge cases (sarcasm, domain terms)
• Batch processing for high-throughput workloads
• Human validation performed every 3-6 months
• Fallback behavior for empty or very short texts

Scaling Considerations

When running sentiment analysis at scale, consider these factors:

• Throughput: VADER processes 10K-50K texts per second on a single CPU core. For higher throughput, use multiprocessing or distribute across workers. For streaming workloads, batch texts into groups of 100-1000 before processing to reduce Python overhead.
• Memory usage: VADER’s lexicon is ~15 KB and loaded once into memory. The main memory bottleneck is the text data itself. For batch processing of large datasets, stream texts from disk or database instead of loading everything into memory.
• Multi-language support: VADER is optimized for English. For other languages, use multilingual models like cardiffnlp/twitter-xlm-roberta-base-sentiment or translate text to English before scoring. Mixing VADER with translated text introduces translation errors that compound sentiment misclassification.
• Real-time dashboards: for brand monitoring, compute sentiment on a rolling window (e. g. , last 24 hours). Alert when sentiment drops below a historical baseline by more than 2 standard deviations.

Cost Estimation

Component	Cost	Notes
VADER (NLTK)	$0	Open-source, runs on CPU
Infrastructure (1 vCPU)	$10-$20/month	AWS t3.small, GCP e2-small
Storage (time-series DB)	$5-$15/month	1M sentiment records/month
Alternative: HuggingFace model	$0.02-$0.10/1K texts	Via HF Inference API

VADER is free and runs locally. For transformer-based sentiment analysis, self-hosting on a single GPU ($200-$400/month) is cheaper than API calls at >1M texts/day.

When Not to Use VADER

• You need aspect-based sentiment: VADER scores the whole text. If you need to know that a review praises the camera but criticizes the battery, use aspect-based sentiment analysis with a transformer model.
• Your text is highly sarcastic or ironic: VADER detects basic negation but misses sarcasm. For social media analysis where sarcasm is common, use a transformer model fine-tuned on sarcastic text.
• You need sentiment intensity comparison: VADER’s compound score is not linear. A score of 0. 5 is not “twice as positive” as 0. 25.
• Your domain uses heavy jargon: VADER’s lexicon is general-purpose. Medical, legal, or technical text may score incorrectly. Supplement with a domain-specific lexicon or switch to a fine-tuned model.
• You need real-time emotion detection: VADER classifies as positive, negative, or neutral. If you need emotions (anger, joy, fear, surprise), use a multi-class emotion classifier like GoEmotions.

Common Production Pitfalls

• Copying the example without adapting it to real data volumes and failure modes.
• Skipping load and error-injection tests before the first production deployment.
• Hard-coding values that should be configurable per environment.
• Forgetting to add logging and monitoring at each step.
• Deploying without a rollback plan or a tested backup strategy.
• Assuming the minimal example will scale without adding caching or batching.
• Not documenting the version and configuration used in production.
• Letting the recipe sit unchanged when dependencies or scale evolve.