Avari Blog

Do your subject lines suck? Here’s how you can answer that question with actual statistics.

[fa icon="calendar"] Jul 28, 2018 / by Parry Malm


Writing email subject lines is hard. Writing good email subject lines is even harder. Writing good email subject lines based upon quantifiable metrics is harder still. What am I talking about? Let’s consider this subject line:

“Further Reductions in our SALE”

That’s a subject line that you’ve probably seen in your inbox before, or at least something like it. But how many other ways can you say it?

Here’s a few different ways:

  • Enjoy our extraordinary discount :)
  • Special online-only offer.
  • Advance notice! Your 25% discount ends Friday.
  • ➳ Enjoy these breathtaking sales!
  • I couldn't wait to tell you about this.
  • Who else wants to save?
  • Save 25% - if you're really quick.
  • ☞ Your 25% discount ends Friday.
  • ✬ You better act before July 31st.
  • ✯ The doors close at midnight on Friday.
Those are all plausible subject lines, right?

What about these ones:

  • Benefit from our extraordinary sale. I dare you to ignore this email…
  • FREE DELIVERY. It's all over in 7 days… benefit from this unmissable offer on trendy shoes.
  • 25% off :)
  • We <3 you! The shoes that dreams are made of…
  • Check out our unthinkable discount on trendy shoes. Why you can't resist opening this email!
  • FREE delivery today :) you'd be crazy to ignore this. Enjoy these outstanding offers on shoes.
  • Latest offer! Last Friday I was scared... But our sale has been extended.
  • 25% offer plus shipping is on us.
  • Our sale continues... Benefit from our unmissable saving: search for trendy shoes.
  • FREE shipping! Check out these shoes & save up to 25%

They all sound like something you’d get in your inbox, right?

Here’s a few more to consider:

  • ➡ Trendy shoes are back at Nike - buy them now or regret it forever.
  • Parry we're fascinated by you :) Enjoy Nike's exceptional deal!
  • When it's gone it's gone. Benefit from this heart-stopping saving on cool shoes AND get them shipped for free.
  • New: enjoy our eye-opening deal on trendy shoes.
  • New: don't miss our stupendous saving; look for shoes!
  • Discover how cool shoes made a normal person awesome.
  • You're going to love Nike's new shoes.
  • Nike hearts you. Look for cool shoes: get £10 off
  • Just to let you know - email only :)
  • Hope you're sitting down! Enjoy this superhuman discount on shoes.

Once again, all of these subject lines are plausible. In fact, all of them could indeed be good… or bad! 

How can you tell?


How do email marketers usually test their subject lines?

Split testing subject lines is a good idea. This isn’t news. If you want proof, check out the State of Split Testing report by Howling Mad.

Marketers across the globe agree that split testing your subject lines is the best (and easiest to implement) way to optimise engagement metrics.

However, the way marketers split test subject lines is suboptimal.


Here’s how most email marketers test out email subject lines: They change one word.

For example they’ll test out:

  • A. Hope you're sitting down! Enjoy this superhuman discount on shoes.
  • B. Hope you're sitting down! Enjoy this great discount on shoes.
Sure, one variable is being tested. From an experimental design standpoint, this is good. But, let’s say #1 wins. People will then assume that “superhuman” is inherently better than “great.”

And they’ll always use “superhuman.”

The thing is – there’s at least 80 individual synonyms that you can use in place. What if “superhuman” is simply less bad than “great?”

And, what if the entire subject line is bad in the first place? What you may be doing is polishing a turd. If the original subject line sucked, you’re simply making it less bad.

Hardly ideal, is it?

Testing out the length of a subject line

This is an impossible test. For example, here’s a common test:
  • A. Get a discounted bicycle now from John’s Bikes!
  • B. Get a discounted bicycle now from John’s Bikes and get a free helmet!

Yes, the length is different. But that’s not all that’s different! The second subject line has an additional incentive added into the mix. Therefore what’s actually being tested?

It’s impossible to isolate length as a causal variable. And it’s also a waste of time, because email subject line length has no bearing on response rates. Trust us.


Testing out two completely random things

This is the most common way people test. You write two completely different subject lines, and one wins. Hurrah!
But then you ask yourself, “why did it win?”

Take, for example, this real-world example of a split test that a major publisher ran:

  • A. Subscribe now to <brand name> to save up to $1.50 per issue!
  • B. For the latest Summer fashion trends subscribe now and get the best product reviews around
Version A won with a 12% open uplift. But why did it win? Here are some possible reasons:
  • 'Subscribe' works better earlier than later in the subject line.
  • Exclamation points incite action.
  • Mentioning the price is good.
  • Including the brand name is good.
  • Including the industry name is bad.
  • People don’t care about the content in the publication.
  • 'Save' is a better word than 'get'.
  • 'Latest' is a bad word.
  • Using 15 more characters is worse than 15 fewer.
  • Leading with a second-person verb conjugation is good.
  • Leading with a prepositional clause is bad. 
  • Using the word 'to' three times in a subject line gets awesome results.
  • Using the word 'and' is bad.
  •  ………. and about a million more potential reasons!

All of these factors could be contributory to version A winning. But if you pick the wrong factor, you’ll be making bad decisions for future subject lines. And this is dangerous.


What matters is the emotional power of a subject line.

Remember those 30 subject lines up above? How long would it take you to test them all? And how would you determine which was the best?

Chances are you’d find a few did well, a few didn’t, and you’d spend hours racking your brains trying to figure out which ones were good or bad.

And your conclusion would be imperfect.

This is because what matters is not individual words. They don’t make a difference in isolation. And making assumptions between two random things is a bad idea.

What matters is the higher-order interaction between words and phrases. What matters is what actually makes people respond.

What matters is the emotional power of a subject line.


How to quantify the emotional power of email subject lines

Here at Phrasee, we’ve developed a deep learning algorithm that quantifies the emotional power of subject lines. What does this mean?

We look at your past subject lines to determine what emotional quadrants worked best for you in the past. Then we generate optimal subject lines for you to use in your campaigns.

Take one of the sets of subject lines above. Here’s how they rank in terms of emotional power:


We look at dozens of emotions, and through factor analysis determine the ones most important for your brand.

NOTE: Why 5 emotions? Because more doesn’t add value. For example, there’s huge semantic and statistical crossover between, say, “Fear of missing out” and “Urgency.” Separating them adds confusion to algorithmic models and gives you worse information. This is a statistical fact.

What this information gives you is as follows:
  1. You understand what emotions your audience responds to
  2. You run smart tests using Phrasee’s language generation engine
  3. You get more opens, clicks and conversions
Phrasee optimises your subject lines following this process:
  1. Analysing your previous subject lines using our deep learning algorithms
  2. Generating human-sounding subject lines for you to use in future campaigns
  3. Inputting your ongoing results to progressively optimise the generative algorithms.

All of the subject lines in this blog post were generated by Phrasee. If you want to learn what language actually drives response from your audience, Phrasee can help.

Our goal is to optimise subject lines. With no BS, just science. Get in touch to find out more.

 #EmailCRO is an Email Optimization Experts Series featuring best practice advice and technology insights from email marketing and conversion rate optimization thought leaders and industry experts. Subscribe to the #EmailCRO Twitter list to learn more and join the conversation.


Topics: Email Optimization, Subject Lines

Parry Malm

Written by Parry Malm

Parry is the CEO of Phrasee. He’s a self-described digital anarchist, although according to the Metropolitan Police he’s just an anarchist. He’s worked client-, agency- and tech-side, so he’s either a well-rounded expert, or can’t hold down a job. He probably knows more about email subject lines than most people, a skill set that makes him great at parties. You can find him on Twitter and LinkedIn.

AVARI Blog Updates

Download Guide