If you aren't automating your support tickets. You are probably wasting a lot of time.

In the fast paced world where we live today, providing efficient support to our customers timely is very important. But why -

Customers primarily leave your software for two main reasons -

  1. They are not satisfied with your product and pricing.
  2. They are kept hanging to get support for their problems.

And with more and more customers, it just becomes impossible to categorize tickets, prioritize them and solving them. And rather than helping our customers with their pain points we are wasting a lot of time in redundant process to maintain tickets.

Today we will aim to address the later, with the rise of LLMs - Intelligent token based models which can answer almost all our query with lightening speed. Can't we use this superpower to empower our softwares?

We should and we will.

Before going deep inside, this week i had a strategy call with one of my client where we discussed on the how to turn software into an engagement system which works on automation.

Ever faced a situation where you have an amazing software but customers are just not using it. To be real - like no retention at all. Download your free retention strategy here -

https://rohangirdhani.com/software-retention

The solution we seek -

We need a LLM that could automate our never ending support tickets, and it should do the following tasks successfully -

  1. Analyze the customer's ticket and do a sentiment analysis for better replies.
  2. Prioritize the tickets and assign to right person.
  3. Send a to the customer in the best human way possible.

In the world of LLMs, AI contribute to storyline generation, and it is important that it adapts with every user interaction to provide an engaging experience.

Article content

Here, is the solution that we need from our AI model, that could analyze the sentiments and prioritize tickets automatically.

Article content

So if you are excited, we will be doing two things now -

  1. Trained multiple models, categorize based on sentiments and prioritize them.
  2. You can use same method to generate replies, to get full code which is not possible to add here on linkedin. Comment "Support" in the comment box and i will send you the link.

So let's do part 1 here: 

# Importing necessary libraries
from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

# Support tickets for analysis
support_tickets = [
    "The product is great, but I'm having difficulty with the setup.",
    "I am extremely frustrated with the service outage!",
    "I love the new features in the latest update! Great job!",
    "The instructions for troubleshooting are clear and helpful.",
    "I'm confused about the product's pricing. Can you provide more details?",
    "The service is consistently unreliable, and it's frustrating.",
    "Thank you for your quick response to my issue. Much appreciated!"
]

# Function to triage tickets based on sentiment
def triage_tickets(support_tickets, sentiment_analyzer):
    prioritized_tickets = {'positive': [], 'negative': [], 'neutral': []}

    for ticket in support_tickets:
        sentiment = sentiment_analyzer(ticket)[0]['label']
        
        if sentiment == 'NEGATIVE':
            prioritized_tickets['negative'].append(ticket)
        elif sentiment == 'POSITIVE':
            prioritized_tickets['positive'].append(ticket)
        else:
            prioritized_tickets['neutral'].append(ticket)

    return prioritized_tickets

# Using the default sentiment analysis model
default_sentiment_analyzer = pipeline('sentiment-analysis')
default_prioritized_tickets = triage_tickets(support_tickets, default_sentiment_analyzer)

# Using a custom sentiment analysis model
custom_model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
custom_model = AutoModelForSequenceClassification.from_pretrained(custom_model_name)
custom_tokenizer = AutoTokenizer.from_pretrained(custom_model_name)
custom_sentiment_analyzer = pipeline('sentiment-analysis', model=custom_model, tokenizer=custom_tokenizer)
custom_prioritized_tickets = triage_tickets(support_tickets, custom_sentiment_analyzer)

# Using the AutoModel for sentiment analysis
auto_model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
auto_model = AutoModelForSequenceClassification.from_pretrained(auto_model_name)
auto_tokenizer = AutoTokenizer.from_pretrained(auto_model_name)
auto_sentiment_analyzer = pipeline('sentiment-analysis', model=auto_model, tokenizer=auto_tokenizer)
auto_prioritized_tickets = triage_tickets(support_tickets, auto_sentiment_analyzer)

# Displaying the prioritized tickets for each sentiment analyzer
for analyzer_name, prioritized_tickets in [('Default Model', default_prioritized_tickets),
                                           ('Custom Model', custom_prioritized_tickets),
                                           ('AutoModel', auto_prioritized_tickets)]:
    print("---------------------------------------------")
    print(f"\nTickets Prioritized Using {analyzer_name}:")
    for sentiment, tickets in prioritized_tickets.items():
        print(f"\n{sentiment.capitalize()} Sentiment Tickets:")
        for idx, ticket in enumerate(tickets, start=1):
            print(f"{idx}. {ticket}")
        print()

The python code above shows the practical implementation of support ticket categorization using the Transformers library. Initially, the code sets up sentiment analysis pipelines employing different models to showcase the library’s flexibility. The default sentiment analyzer relies on the pre-trained model provided by the library. Additionally, two alternative models have been introduced: a custom sentiment analysis model (“nlptown/bert-base-multilingual-uncased-sentiment”) and an AutoModel, demonstrating the ability to customize and utilize external models within the Transformers ecosystem.

Then it defines a function "triage_tickets" which assesses the sentiment of each support ticket using the specified sentiment analyzer and categorizes them into positive, negative, or neutral sentiments. The code then applies this function to the support ticket dataset using each sentiment analyzer, presenting the prioritized tickets based on sentiment for comparison. This approach allows for a comprehensive understanding of sentiment analysis model variations and their impact on ticket triage, emphasizing the versatility and adaptability of the Transformers library in real-world applications.

OUTPUT:

1. Default Model

  • Positive Sentiment Tickets: 3 positive tickets express satisfaction with the product or service.
  • Negative Sentiment Tickets: 4 tickets are negative, indicating issues or frustrations.
  • Neutral Sentiment Tickets: 0 tickets listed.

2. Custom Model

  • Positive Sentiment Tickets: No positive sentiment tickets are listed.
  • Negative Sentiment Tickets: No negative sentiment tickets are listed.
  • Neutral Sentiment Tickets: All tickets, including positive and negative sentiment tickets from the Default Model, are listed here.

3. AutoModel:

  • Positive Sentiment Tickets: No positive sentiment tickets are listed.
  • Negative Sentiment Tickets: No negative sentiment tickets are listed.
  • Neutral Sentiment Tickets: All tickets, including positive and negative sentiment tickets from the Default Model, are listed here.

Article content

You see different models can provide different outputs and it becomes important to choose the right model and train it on the domain knowledge and model will not always replicate the human sentiment. But it does is job well, you can also try Llama model.

Performance Metrics for Evaluation

To exactly assess the model performance, it's important to select the right metrics that could help you make model perform better. Some direct percentage based metrics that you could use -

1. Ticket Categorization Accuracy

  • Definition: Measures the percentage of support tickets correctly categorized by the LLM.
  • Importance: Ensures that the LLM accurately understands and classifies the context of each support ticket.

2. Priority Assignment Accuracy

  • Definition: Evaluate the correctness of priority levels assigned by the LLM based on ticket content.
  • Importance: Reflects the LLM’s ability to identify urgent issues, contributing to effective and timely ticket resolution.

3. Response Time Reduction

  • Definition: Measures the average time saved in responding to support tickets compared to a manual process.
  • Importance: Indicates the efficiency gains achieved by automating responses to common queries using LLMs.

4. Consistency in Responses

  • Definition: Assess the uniformity in responses generated by the LLM for common issues.
  • Importance: Ensures that standard responses generated by the LLM maintain consistency in customer communication.

5. Sentiment Accuracy

  • Definition: Measures the correctness of sentiment analysis in categorizing customer sentiments.
  • Importance: Evaluate the LLM’s ability to interpret and prioritize tickets based on customer emotions accurately.

6. Customer Satisfaction Improvement

  • Definition: Gauges the impact of LLM-driven ticket triage on overall customer satisfaction scores.
  • Importance: Measures the success of LLMOPs in enhancing the customer support experience.

7. False Positive Rate in Sentiment Analysis

  • Definition: Calculates the percentage of tickets wrongly categorized as having negative sentiments.
  • Importance: Highlights potential areas of improvement in sentiment analysis accuracy.

8. False Negative Rate in Sentiment Analysis

  • Definition: Calculates the percentage of tickets wrongly categorized as having positive sentiments.
  • Importance: Indicates areas where sentiment analysis may need refinement to avoid missing critical negative sentiments.

9. Robustness to Domain-Specific Sentiments

  • Definition: Measures the LLM’s adaptability to sentiment nuances specific to the industry or domain.
  • Criteria: Conduct validation tests on sentiment analysis performance using domain-specific data.

10. Ethical Considerations

  • Definition: Evaluate the ethical implications and biases associated with sentiment analysis outputs.
  • Criteria: Consider the fairness and potential biases introduced by the LLM in categorizing sentiments.

I hope, now you can start automating your support tickets, to get the full code using prompt engineering comment "SUPPORT" in the comment box below and i will share the link.

Feel free to repost ♻️ with your network.

Until next time,

Rohan Girdhani - The TechDoc

📰 Three amazing articles to make better software that you must read

  1. Should we build conscious softwares like our movies?
  2. Why is your customer not coming back to your SAAS?
  3. Why every software needs a blueprint?

▶️ Three amazing videos to make better software

  1. Key ingredients to make an amazing software
  2. The only complete audit for your softwares
  3. Decoding Netflix from a consultant's mindset

Thank you so much for reading this edition of the newsletter, if you found this interesting you would also love my course THE SOFTWARE BLUEPRINT.

I keep sharing my insights on making tech easy and better. If you resonate with my content do give me a follow on Youtube and Instagram.

If you a software engineer or manager with at least two years of experience, next sunday on 27th October 11 am IST i am conducting a webinar to help you tap the top career in the software industry. Having served 500+ students with 5 star reviews who have already joined our community and taken the path towards 1% career.

I am inviting you to be a part of this movement. Here is the link -

https://rohangirdhani.com/webinar



View Original on LinkedIn