Increase user retention: Creating an automated alerting system for at-risk customers

User retention rates for digital products present a significant challenge for businesses today… We’ve all seen the claims that because of AI “there is no more moat” (although I might challenge that, but will save it for another post). With the average retention rate dropping to just over 40% one month after download and plummeting to 30% by month three, companies need proactive strategies to identify and address potential churn before it happens.

While acquisition efforts can create temporary spikes in user activity, true business sustainability comes from retaining existing customers. We’ve all heard of the “leaky bucket issue”: if you lose customers at a similar or faster rate then you acquire them, you’ll quickly run out of cash. This is where an automated alerting system for at-risk customers becomes invaluable – allowing you to identify warning signs early and take meaningful action before users disappear.

In this article, we’ll talk about what user retention is and how it relates to churn and what churn signals to look for. We’ll also share a step-by-step guide on how to create a simple churn alerting system using SQL, Python and email with the help of AI.

What is user retention and the true cost of churn

Let’s just start with a bit of housekeeping: customer retention and churn and two sides of the same coin. So if we talk about decreasing churn, we’re also talking about increasing retention. Now with that out of the way, let’s talk about why these matter to your business.

If you’re reading this, there’s a very good chance that you already have an intuition of how important customer retention is to your business, but in case you need some more concrete evidence, here are some numbers: customer acquisition costs are typically 5-25 times higher than retention costs, depending on your industry. Each churned customer represents not only lost immediate revenue but also:

• Reduced customer lifetime value (CLTV)
• Decreased word-of-mouth referrals
• Lower return on acquisition investments
• Potential negative market feedback

For subscription-based businesses, even a 5% improvement in retention rates can increase profits by 25-95%. This makes churn prevention one of the highest-ROI activities your team can pursue. Put another way, hiring someone full time just to spot and reduce churn can be more effective than a multi-million dollar marketing campaign.

If you’re wondering what those numbers are for your business, simply ask your finance team, there’s a good chance they have those numbers on hand, or perhaps even memorized.

Why traditional retention methods fall short

Oftentimes, companies conflate the reason for churn and the churn signal. Understanding why customers churn is critical and the first step to increasing customer retention. Here are a few ways to understand “why” customers are churning:

• Exit surveys and interviews
• Quarterly business reviews (QBRs) with key accounts
• Feedback from account executives and customer success managers

Understanding why customers churn is important to help provide product and marketing guidance on ways to increase retention, but it doesn’t necessarily help get ahead of future churn that’s going to happen regardless of these efforts.

Key components of an effective early warning system

A comprehensive retention alert system consists of four primary elements:

1. Data collection pipeline: Gathering relevant user behavior signals
2. Risk assessment algorithm: Evaluating signals to determine churn probability
3. Alert generation system: Creating notifications for at-risk accounts
4. Intervention workflow: Ensuring timely, appropriate responses

Let's break down how to build each component step by step.

Tutorial: Building a churn alerting system

Step 1: Identifying your churn signals

The foundation of your system depends on identifying the behavioral indicators that predict eventual churn. These signals will vary across products, but typically fall into several categories:

Engagement metrics

• Login frequency decline: For example a  50%+ drop in login frequency compared to a user's historical baseline
• Session duration reduction: When average time spent in-app drops significantly (e.g., from 10 minutes to 2 minutes)
• Feature abandonment: Decreased usage of core features that previously saw regular engagement
• Inactivity periods: No logins for a specific timeframe (e.g., 14 days for a daily-use app)

Progress indicators

• Stalled onboarding: New users who haven't completed critical setup steps
• Goal completion rates: Decrease in successful task completions
• Adoption plateaus: Users who haven't progressed beyond basic features after 30+ days

Sentiment signals

• Support interactions: Increased ticket volume or severity
• Survey responses: Declining NPS or CSAT scores
• In-app feedback: Negative comments or ratings
• Help documentation usage: Sudden increases in help center visits

Account signals

• Team usage patterns: For B2B products, decreased overall team engagement
• Billing issues: Failed payments or extended payment delays
• Contract changes: Requests for downgrades or flexible terms
• Communication engagement: Declining email open rates or click-throughs

To identify which signals matter most for your product:

1. Analyze historical data from churned customers to identify common pre-churn behaviors
2. Compare patterns between retained customers and those who churned
3. Assign weight to each signal based on its predictive strength
4. Create composite scoring that balances multiple indicators

Of course, you have to first have all the data at your disposal. The key data that might indicate churn likely lives in a variety of platforms, from event tracking tools to your production database. Identifying and leveraging churn signals depends on you having this data, which generally should live in a centralized data warehouse.

In a previous tutorial video we show you how to use survival analysis to identify churn indicators and predict churn. The most powerful churn signals are often unique to your product and user journey. Invest time in understanding the specific "moments of truth" where user commitment is tested in your application.

Step 2: Building your at-risk customer detection system

Once you've identified your key churn signals, you need a system to monitor and flag at-risk accounts. This can range from basic scripts to sophisticated machine learning models depending on your resources and requirements.

Basic implementation approach

For smaller teams or as a starting point, you can build a straightforward risk detection system using SQL and a simple scoring algorithm. Here's a framework for implementation:

1. Create a centralized user activity database or data warehouse
2. Establish regular data pipelines from your product analytics
3. Develop a scoring mechanism to evaluate risk factors
4. Set up scheduled jobs to calculate risk scores
5. Define thresholds for different risk categories

SQL can be a great place to start, but I find that SQL quickly becomes unwieldy when dealing with frequency-based event detection, rolling averages or trends etc. This is where Python comes in handy.

Using AI to generate detection scripts in Python

Even if you just know a bit of Python, you can get a very long way building advanced detection mechanisms with the help of AI.

Even without advanced machine learning, you can create effective detection systems using rule-based approaches that flag users when multiple risk indicators are present simultaneously. You can watch me use AI to build this:

What I’ve done here is ask the AI to create a list of at-risk accounts based on login and new user creation patterns. The AI did a few things for me.

First, it wrote a script to identify organizations with a decreasing number of logins month-over-month:

# Analyze login patterns by organization
login_events = dataframe1[dataframe1['event_type'] == 'login'].copy()
login_events_with_org = login_events.merge(dataframe2[['user_id', 'org_id']], on='user_id', how='inner')
login_events_with_org['year_month'] = login_events_with_org['event_date'].dt.strftime('%Y-%m')

# Count logins by organization and month
monthly_logins = login_events_with_org.groupby(['org_id', 'year_month']).size().reset_index(name='login_count')

# Pivot the data to have months as columns
pivot_logins = monthly_logins.pivot(index='org_id', columns='year_month', values='login_count').reset_index()
pivot_logins = pivot_logins.fillna(0)

# Get the list of months in chronological order
months = sorted([col for col in pivot_logins.columns if col != 'org_id'])
last_month = months[-1]
previous_month = months[-2]

# Identify organizations with declining login trends
orgs_at_risk = []
org_trends = {}

for _, row in pivot_logins.iterrows():
    org_id = row['org_id']
    
    # Check if logins decreased from previous month to last month
    if row[previous_month] > 0 and row[last_month] < row[previous_month]:
        # Calculate percentage decrease
        pct_decrease = (row[previous_month] - row[last_month]) / row[previous_month] * 100
        
        # Only consider significant decreases (e.g., more than 20%)
        if pct_decrease >= 20:
            orgs_at_risk.append(org_id)
            org_trends[org_id] = {
                'previous_month': previous_month,
                'previous_count': row[previous_month],
                'last_month': last_month,
                'last_count': row[last_month],
                'pct_decrease': pct_decrease
            }

The more interesting component of this script being the last part where you can see that the AI is looking at the number of logins from the two months ago to the previous month and flags organizations with a decrease greater than 20%. Keep in mind that your churn signals may look completely different, but you can ask the AI to write a script based on your needs so long as you’ve given it the data it needs.

Then the second part of our churn signal is the new user creation:

# Check for new user creation in the current month (May 2025)
current_month = '2025-05'
recent_users = dataframe2[dataframe2['creation_date'].dt.strftime('%Y-%m') == current_month]
orgs_with_new_users = recent_users['org_id'].unique()
all_orgs = dataframe3['org_id'].unique()
orgs_with_no_new_users = [org for org in all_orgs if org not in orgs_with_new_users]

# Find orgs that meet both criteria
at_risk_orgs = [org for org in orgs_at_risk if org in orgs_with_no_new_users]

In this step we’re also identify customers that meet both churn risk signal criteria.

As you can see, this is a fairly rudimentary and simplistic churn detection mechanism. But you can build off of this and progressively make it more advanced based on the data you have.

Now let’s automate these alerts so that you can proactively get ahead of churn.

Step 3: Automating the alert system

With your detection system in place, the next step is creating automated alerts that notify the right team members about at-risk customers. The goal is to deliver actionable information that enables timely intervention.

At Fabi.ai, we’re big believers in meeting your users where they are. If your account management or customer success team is responsible for increasing user retention and preventing churn for example, it’s important to provide the information at a place and time that works best for them.

Building an email messaging system with Python can be quite involved and generally exceeds what AI is capable of since it requires integration with an email sender, so in this example we’re simply going to use the Fabi.ai built-in email functionality. Once you have the Python DataFrame that has your list of at-risk customers, simply create a new email cell and type out your message with your DataFrame inserted:

Title: Weekly churn alert

Body:
Customers at risk of churning based on login and new user creation patterns:
{{at_risk_customers}}

The {{}} indicates that this is a Python variable that you’ve previously defined.
Watch me build this email here:

If your account management or customer success team prefer getting updated via Slack or Google Sheets, we also offer easy integrations for those.

You can also consider pushing this information back to CRM and ticketing systems used by your team:

• CRM Integration: Automatically create tasks or opportunities in Salesforce, HubSpot, or similar platforms
• Ticketing Systems: Generate support tickets in Zendesk or Jira for high-priority cases
• Customer Success Platforms: Sync risk data with dedicated CS tools like Gainsight or ChurnZero

My personal opinion is that the more individuals have to dig for information and the more repetitive the information, the less likely it is to be used. The most effective weekly alerts are timely and pertinent. For instance, simply sending out a list of all accounts with the risk to each account is likely to be much less effective than just sending the top 10 accounts that need attention. So pushing data back to a CRM or ticketing platform may cause this data to go ignored.

Alert frequency and thresholds

Finding the right balance for alert frequency is critical:

• Too frequent: Creates alert fatigue and diminishes response quality
• Too infrequent: Misses intervention opportunities

Consider these guidelines:

1. Daily alerts for high-risk customers requiring immediate attention
2. Weekly digests for moderate-risk customers showing early warning signs
3. Monthly reviews for trend analysis and strategic planning

Implement progressive thresholds that adjust based on customer value, ensuring your most valuable accounts receive proportionate attention. And again, the more the information changes and the more targeted it is, the more likely it is to be used.

Developing targeted intervention strategies

An alert system is only valuable if it drives effective action. Develop structured intervention strategies for different risk profiles:

High risk intervention (immediate action required)

For customers showing multiple severe warning signs:

1. Executive outreach: Personal call from customer success leadership
2. Health check consultation: Comprehensive usage review and improvement plan
3. Strategic account planning: Custom roadmap to align product with customer goals
4. Potential incentives: Short-term accommodations to demonstrate commitment

Medium-risk intervention (proactive engagement)

For customers showing early warning signs:

1. Success manager check-in: Targeted call to address specific pain points
2. Educational resources: Custom training on underutilized features
3. Usage recommendations: Specific suggestions to increase product value
4. Community connection: Introduction to user groups or success stories

Low-risk intervention (preventative measures)

For customers showing minor deviations from healthy patterns:

1. Automated engagement: Triggered emails with relevant tips or resources
2. Feature highlights: Introduction to new or underutilized capabilities
3. Feedback collection: Short surveys to uncover potential issues early
4. Success stories: Share relevant case studies to inspire deeper usage

The key is matching the intervention to both the risk level and the specific signals that triggered the alert. A customer showing reduced login frequency needs a different approach than one generating increased support tickets.

Continuous system improvement

Your retention alert system should evolve based on outcomes and changing user behaviors:

Track intervention results

Monitor which interventions successfully prevent churn and adjust your approach accordingly:

1. Document each intervention type, timing, and outcome
2. Calculate rescue rates by intervention category
3. Identify patterns in successful retention efforts
4. Refine your playbook based on empirical results

Refine signal weights

Not all churn signals will prove equally predictive over time:

1. Analyze false positives (alerts for customers who didn't churn)
2. Identify missed warnings (customers who churned without triggering alerts)
3. Adjust signal weights to optimize detection accuracy
4. Add new signals as product and user behaviors evolve

Optimize alert system

Fine-tune your alert mechanisms:

1. Survey team members about alert usefulness and actionability
2. Track response rates to different alert formats and channels
3. Test alternative presentation methods for complex risk data
4. Adjust thresholds to balance comprehensiveness with actionability

For example, here are a few next steps I would take in the alerting system I built above:

• Prioritization: I would better indicate the priority of each account
• Assigning ownership: If I had account owner information I would include that information next to each account
• Advanced churn signals: Number of logins and new users created is likely too basic of a signal. I would spend more time looking at the data, using machine learning and talking to the customer success team to identify more precise signals.

Conclusion

Building an automated alerting system for at-risk customers transforms your retention efforts from reactive to proactive. By identifying warning signs early, you create opportunities to address customer concerns before they result in churn.

The most successful implementations balance sophisticated data analysis with straightforward, actionable alerts that drive meaningful interventions. Start with the basics - identifying your key churn signals and setting up simple monitoring - then gradually increase complexity as you learn what works for your specific customer base.

Remember that the ultimate goal isn't just preventing churn but creating such exceptional customer experiences that retention becomes a natural outcome. Your automated alert system is simply a tool to help focus your team's attention where it can have the greatest impact on customer success.

By implementing the approach outlined in this guide, you'll not only improve your retention metrics but also gain deeper insights into your customers' needs and pain points - insights that can drive product improvements and create a virtuous cycle of increasing customer satisfaction and loyalty.

If you have your customer data on hand and are ready to start building a more advanced alerting system, you can send your first email from Fabi.ai in less than 10 minutes for free.

‍