RAB Guide: Measuring Radio's Effect

Introduction

This guide has been created in response to the growing number of enquiries received by the Radio Advertising Bureau about researching the effectiveness of radio campaigns.

Typically these can be paraphrased as:

•    How do I go about reseaching the effect of a radio campaign?

•    Are there any special issues I should bear in mind?

•    Is there a right or wrong way to research radio effectiveness?

The booklet is designed as a step-by-step guide for those involved in the advertising process. It aims to provide a better understanding of the effectiveness research area and, of course, it focuses on those issues that can arise when researching radio in particular.

It clearly doesn’t cover every single aspect of the subject, and it also assumes that eventually you will be using the services of a skilled advertising research company. It may be worth asking your research company if they have experience of radio in particular – not all do.

We would particularly like to thank Clark Chapman Research and Millward Brown International for their help in compiling this guide.

Executive summary

Effectiveness research requires clarity of objectives - what are the agreed objectives of the overall campaign and of the radio campaign within this? The research company will need to know precisely what it is you are aiming to measure. N.B. not all objectives can best be measured by consumer survey research (e.g. sales effects).

Radio effectiveness can be measured either using continuous research or in stages (“pre & post”) – the pre-stage is normally the week before the campaign, the post-stage in the week after the campaign finishes.

Consumers tend to misattribute radio advertising memories to other media, particularly TV. This is particularly likely to happen where there is a strong executional link between the two media and/or where there is an established history of TV advertising for the brand.

This tendency to misattribute can be offset by using matched samples of listeners and non-listeners. If response (in terms of, say, advertising awareness or brand perceptions) is greater among listeners than it is among non-listeners, then the effect can be attributed to radio fairly confidently - even if the listeners think the advertising was in another medium.

Whilst radio research can successfully be done using telephone interviewing - ads can be played down the line - in cases where other media are to be included in the research it might be more appropriate to use face-to-face interviewing.

Commercial recognition is a valuable technique – i.e. playing the ads to consumers. This is ultimately a more robust measure of whether they have heard the campaign, and avoids problems of trying to describe the ads. Brand names can also be bleeped out of the commercial, to test whether the campaign is linked to the brand.

Defining the research objectives

The key to any successful research is to have a clear understanding of why the research is being conducted in the first place. In other words, what are you aiming to measure?

It is crucial that you are able to state clearly and precisely what your research objectives are - the success of the research project hinges on this simple premise. Any uncertainty about which issues should be covered by a questionnaire will probably result in a piece of research that, whilst maybe providing lots of interesting findings, does not actually give you the types of answer you were after.

There can be numerous different aspects of the campaign you might wish to research. However in broad terms, radio advertising research aims can be categorised into two types:

• Marketing issues - to what extent has radio helped to achieve the campaign aims?

• Media planning issues - what effect do different media strategies have on the performance of the campaign?

Marketing issues

These vary widely and there is often more than one objective set for a campaign. Typically radio research seeks to find out to what extent did the radio campaign help to…

    Increase sales
    Increase footfall / store traffic
    Increase brand awareness
    Establish / enhance a particular brand identity
    Change consumers’ perceptions about a brand
    Increase market share
    Motivate the salesforce / franchisees / distributors
    Promote distributor acceptance
    Gain share of voice / share of mind
    Generate sales leads
    Encourage brand trial
    Broaden consumer appeal to include new target
    customers
    Publicise a promotion

Not all of these aims are best evaluated with consumer survey research - there are specific tools available for measuring sales effects for example. However consumer research could still be useful in diagnosing results. For example, if there is no significant shift in sales, research can still confirm whether consumers actually noticed or understood the advertising.

Media planning issues

In addition to tracking radio’s contribution to the success of a campaign, as a secondary aim you might also be trying to test and evaluate the effects of using different media strategies.

If this is the case then the types of aspects you might be investigating could be along the following lines:

    What weight of advertising works best?
    Do longer spot lengths work better than shorter spotlengths and if so, to what extent?
    What does radio contribute to a mixed media schedule?
    Can radio arrest campaign decay?
    Is there a difference in effect between burst versus phasing versus continuous activity?
    Does position in break matter?
    How do different daypart strategies effect campaign performance?
    How influential is creative style?
    Are particular days of the week more effective for my brand?
    Does topping and tailing generate greater response?

If you do intend to test a particular media strategy there are three important considerations to note.

Firstly, and most obviously, you must gear the campaign so that you can test the particular media strategy in which you are interested.

Secondly, if you are testing a number of media strategies simultaneously, will you be able to separate the effects of each once you have the research results back (you will need a separate, balanced research “cell” for each media-variable)?

Lastly, when testing different media strategies, bear in mind that you will still be judging the effects in terms of campaign aims. Have the campaign aims been clearly defined?

Whatever your research objectives once you have defined them make sure that they form the core of the questionnaire you use. Any other questions you choose to add to the survey, whilst they may provide some illuminating results are, in the end analysis, of merely secondary importance - icing on the cake.

TIP: SETTING REALISTIC OBJECTIVES

Researchers and planners agree it is extremely difficult to change perceptions of a brand over a short period unless there is something particular and engaging to tell the consumer (Millward Brown call this “new news” advertising).

Most advertising campaigns are tasked with increasing or maintaining awareness of brands which are already familiar to the consumer – and therefore slower to change perceptions.

Objectives must also be realistic about the (usually undesirable) impact of activity by other brands in the sector.

When to do the research

The ideal research method is to monitor advertising activity on a continuous basis, since this allows movements in advertising response to be compared directly to current advertising activity. Often, however, continuous radio research is impractical on grounds of cost unless it forms part of ongoing advertising tracking.

Typically, radio research is conducted in two stages - a pre-campaign piece of research and then a post-campaign study.

The first phase of the research should be conducted as close to the start of the radio campaign as possible - preferably during the week immediately preceding the radio campaign. The results from this phase of the research will establish the base levels of whatever is being measured. For example, it may determine base levels of brand awareness or perceptions about a brand.

The second phase should be conducted as soon as possible after the radio campaign has ended - ideally during the first week after the campaign has come off air.

In some instances you might want to consider conducting more than two phases of research. For example, it might be worth slotting in an additional research phase during a particularly long advertising campaign or sponsorship. Similarly, having done the post-research, you might want to consider adding an additional stage of research some weeks after a campaign has ended in order, say, to track decay in brand awareness.

TIP: NO TIME FOR A PRE-STAGE MEASURE?

There is a “rough and ready” solution for times when a pre-stage has not been possible.

If non-listeners are used as a control sample (i.e. people who would not have been exposed to the radio advertising campaign) they are in a sense equivalent to a pre-stage sample.

In other words the difference, following a radio campaign, between a matched sample of listeners and non-listeners, is equivalent to the difference between listeners before and after a campaign.

If you need to consider this pragmatic solution, remember the following:

• it does not take account of your other media activity

• it doesn’t take account of anything else which happened during the radio campaign (e.g. competitive activity etc)

The research sample

Defining the sample

It’s an obvious point perhaps, but the sample for the research should be the same as the target audience for your advertising/radio campaign. There is little point, for example, in measuring adults’ awareness of a particular campaign when in reality the campaign was geared solely at reaching women; in this instance the findings amongst men are irrelevant since you have not sought to influence men through your campaign.

Sample sizes

Generally speaking, the larger the sample the better. However at some point, of course, the cost of an increased sample size becomes cost prohibitive and contributes little extra to statistical robustness.

As a guide, the table opposite shows details of what would often be recommended as minimum and suggested sample sizes.

Note however that if you intend to analyse the results amongst sub-groups of your target audience a larger sample will probably be required. In addition, the greater the complexity of the task and the range of objectives, the greater the likelihood a larger sample size will be required. The research agency can advise on this.

	Minimum sample size	Suggested sample size
Pre-research
Listeners/test sample	200	400
Non-listeners/control sample	200	400
	400	800
Post-campaign
Listeners/test sample	200	400
Non-listeners/test sample	200	400
	400	800

Total	800	1600

TIP: THE ADVERTISING TARGET AUDIENCE DEFINES THE SAMPLE

If the advertising is tasked with changing the attitudes of ABC1s, is there any reason to gauge the views of C2DEs?

The importance of split samples

Misattribution of advertising

When asked to consider advertising, consumers will turn their thoughts to the most salient source they can think of - this tends to mean TV. Television, as the medium with the most active expectations, tends to dominate memories of advertising, with the result that campaigns in all other media are, to varying extents, attributed to television in the consumer’s mind.

This misattribution is disproportionately likely to happen with radio memories and is accentuated further if the commercials have been heard on local radio; historically, local radio has been presumed not to be a prime source of advertising for national brands.

It is still more likely to happen when radio campaigns are creatively synergistic with TV executions.

Structuring the research correctly offsets this problem.

Split samples: listeners vs non-listeners

The most pragmatic solution to the problem of measuring true radio awareness is to split the sample into two parts: listeners (target consumers who have been listening to the radio stations which carried ad campaign) and non-listeners (people who do not listen to those stations, but who are the same as the listeners in all other respects).

If the only difference between the two sub-samples is their radio listening, then any differences in their awareness or attitudes to the advertised brand can be reasonably attributed to radio - regardless of where they think they have seen or heard the advertising.

When choosing how to split the sample, you have the choice of two options:

Test and control samples in different areas

This involves taking two matched samples of respondents in different geographical areas and comparing their advertising responses – one sample will live in the advertised area, the other in an area where no radio advertising ran.

In this way, it will be possible to compare the results among those who have been exposed to the campaign with the results among those who have not - thus giving you a measure of radio’s effectiveness.

It is important to allow for the media consumption of the samples (e.g. how much TV they watch etc) as well as their demographics, as this could affect response.

The two geographical areas should also be comparable – normally this means picking areas which are “typical” in terms of media and product consumption.

Test and control samples within the same area

In this second approach, all of the research is done within the same area, and this is the most commonly used method.
One part of the sample will comprise people who do listen to the station(s) on your radio schedule, whilst the other part of the sample will comprise people who do not listen to any station on your schedule.

Again, in this way it will be possible to compare the results of those who have been exposed to the campaign to those who have not. This will give you a measure of radio’s effectiveness.

Neither of these two approaches is necessarily better than the other. You might want to bear in mind, however, that the second method has the advantage of questioning people who will have the same history of exposure to your brand. Local distribution levels for the brand will also be the same.

Remember that for some target markets, such as young people, it can be hard to recruit a sample of non-listeners in the same area (sometimes almost all of them do listen to Commercial Radio).

The key point is that the listener and non-listener samples must be matched as closely as possible, in terms of both demographics and media consumption. This should ensure that any differences can confidently be attributed to radio ad exposure.

TIP: HEAVY, MEDIUM & LIGHT LISTENERS

Asking listeners how many hours a day they listen to a given station allows you to categorise them by weight of listening – this can be broken out on the analysis.

This can be useful. For example, if increased response is positive but actually confined to heavy listeners, there may well be a mandate for increasing frequency against medium and light listeners.

Method and questionnaire

Telephone research is often used for assessing the effect of radio campaigns: the method is adaptable and can often be cheaper than face-to-face interviewing. Radio ads can successfully be played down the phone to respondents.

Face-to-face interviewing may also be preferable if respondents need to be shown visual ad material such as stills from TV ads.

Commercial recognition is a valuable technique – i.e. playing the radio ads to consumers – as this is the best “memory jogger” of all. It also delivers a larger sample of people who are identifiable as having definitely heard the campaign: this is useful when analysing them for their attitudes to the brand. For more detail on the commercial recognition method see Appendix 1 on “Radio memories and how to access them”.

N.B. when playing the radio commercials in order to measure commercial recognition, two different approaches can be taken: blind or branded.

blind
By bleeping out all brand references in each execution and asking whether the commercial has been heard before and then asking for the brand name, it is possible to see whether creative treatment has successfully linked the message to the brand.

branded
This allows prompting for brand-specific data, (e.g. attitudes to the advertising/feelings about the proposition), whilst giving a true measure of ad recognition.

A fairly straightforward questionnaire will take around 10-15 minutes to run through - much longer and respondents will begin to lose interest and concentration!

Questionnaire structure

It is important to ensure your research agency’s draft questionnaire includes all your original aims to ensure survey results encompass all factors you wanted to measure.

Pre- and post-campaign questionnaires will largely be the same (although the former can often be shorter).

Key elements of a typical pre- and post-campaign research study would include the following:

•    General questions on the category in which your brand operates

•    General questions on brand usage and advertising awareness

•    General questions on media consumption and specifically questions that can separate listeners to the radio stations used        in the campaign from non-listeners

•    Recall of advertising. At the post-stage, you will be seeking to detect spontaneous and prompted awareness

•    Commercial recognition – playing the ads to respondents

•    Thoughts on what the main message of the ads was

•    Attitudinal statements about the brand and its advertising. These should relate directly to your specific brand communication aims, and the way in which the ad is supposed to communicate them

Appendix 1:

Radio memories and how to access them

Radio campaigns are known to be potentially tricky to evaluate. Jonathan Chapman reports on a recent study which offers guidance.

It is said that local advertisers measure the success of their radio campaigns by what happens to cashflow – if the advertising isn’t making a difference in the tills, it is dropped or changed.

If only everything in life were this uncomplicated. The reality for most national advertisers is that radio’s effect is a lot more difficult to evaluate. In part this is because radio has historically played only a very minor part in the advertising plans, although this is a situation which seems to be changing, as radio’s share of the cake has doubled in the last five years, going from 2% to 5% of revenue.

However the bigger problem is that the noise from the other 95% of advertising tends to get in the way. Every researcher knows that asking about radio advertising awareness can generate unbelievably low recall figures: we commonly see awareness scores which are under-reporting, seemingly because the method hasn’t taken account of the way people consume radio.

Radio is, like posters, almost subliminal in the way it operates. Most listeners are doing something else – driving, eating, chores – with their attention zoning in and out during music, news, ads, weather etc. In addition radio isn’t seen to be the natural home of brand advertising – consumers always think of TV when asked about advertising.

So what can we do about it? What is best practice in terms of measuring memories of radio advertising?
To answer these questions, the Radio Advertising Bureau recently commissioned us to carry out a live test of different methodologies. We surveyed consumer recall and response to six different campaigns which were running on Capital Radio (the test brands involved were Feminax, TV Licensing, Prudential, P&O Stena, Silentnight Beds and Channel 5, but all the figures are averaged across the six).

To offset the possible effect of misattribution we used a split sample of listeners and non-listeners – this is essential to get a clear reading on the role of radio.

Spontaneous awareness scores told us that, on average, 6% of the Capital listeners could tell us that the test brand had been advertising on radio. The level of noise in the numbers here is clear from the fact that 4% of non-listeners answered the same way.

Prompting with test brand names on a showcard helped this to rise to 13%, and it rose again to 19% when listeners were prompted with the medium as well (e.g. “Which, if any, of these have you heard advertised on radio recently?”).

This prompted stage is about as far as many surveys go and, to be fair, 19% is a reasonable level to see significant shifts.

However, as can be seen in the graph, we went two stages further in the survey.

The third stage of prompting was where the interviewers read out a description of the ad, (e.g. “This advert is in the form of a lighthearted drama where a couple are being told at the check-in desk at the airport that they simply cannot bring their bed with them into the aeroplane”), and asked the respondents whether they had heard the ad. Perhaps not surprisingly, scores varied widely on this method, leading to the conclusion that some ads are more describable than others!
The fourth stage was commercial recognition – we literally played the ads to the respondents and asked whether they had heard them. As can be seen in the chart this revealed that, on average, 63% of the listeners could confirm they had heard the campaign (this figure excludes an additional 9% who said “maybe”).

The difference between 6% and 63% is significant in anybody’s language.

Clearly, memories of radio advertising are very sensitive to the method of prompting. So what does this mean for the way we conduct radio advertising research?

We don’t really have a right and wrong method here – we have different results when we use different techniques, and this is a matter of horses for courses.

If you want to know how likely listeners are to think of you as being a radio advertiser, spontaneous awareness scores will tell you this (not surprisingly, the Carphone Warehouse tends to score well on this – the brand has achieved an explicit association with the medium).

Verbally describing radio campaigns to listeners seems to be a seriously unreliable method – and it is actually measuring how describable your ads are, so it’s not particularly useful.

Commercial recognition is clearly the most efficient method in terms of finding out whether people have heard your campaign. This is valuable, because one can look at the correlation between this and brand knowledge or attitudes.

Meanwhile, it seems that the method we normally use is the prompting by brand and medium – this may make sense in terms of economics (it can be added to a TV tracking study) but will inevitably under-report true levels of radio advertising memories. On this basis it should certainly be questioned.

Jonathan Chapman is a director of Clark Chapman Research.

This article is based on FACTFILE from Marketing Week of December 1998, and summarises a special project into the methods used for researching radio campaigns.

Appendix 2:

Measuring the effects of different sorts of radio advertising

by Dale Beaton, Millward Brown International

Increasingly the scope of radio advertising research is expanding so it looks not just at whether an ad has been noticed but at what it communicates and how it is working. Current thinking by Millward Brown suggests that there are three principal routes via which radio advertising works. Each of these types of advertising has different objectives, and uses different techniques to achieve these goals. Subsequently, the research techniques most appropriate to evaluating them shift slightly in line with these objectives.

The first two routes share many similar properties. Both require relatively low mental effort to process them and are usually direct and catchy, with a focus upon mnemonic devices and repetition. The primary objective of announcement ads is to communicate information clearly, and in particular information of a short term nature (e.g. a sale in shop).

The second route is that of Reminder / Reinforcement ads, which serve to remind consumers about a brand or service, building saliency and suggesting situational brand triggers.

The mechanism used in each instance has an implication upon the key area that the research needs to address. The test for Announcement ads is the extent to which the intended information is communicated, whether this is perceived to be useful, and the overall linkage of this information to the brand. Although with Reminder / Reinforcement ads an assessment of these factors also needs to be made, the research fundamentally needs to assess the likelihood of the brand being recalled either randomly, or in the relevant situations (e.g. at point of sale).

The final radio advertising mechanism is that of brand building, or enhancement ads. These ads frequently place greater emphasis upon creativity and originality, and try to create a stronger and clearer set of images. Such ads frequently involve a greater emphasis upon creating a situation or storytelling, and it is here that synergy with TV campaigns is most common.

As a consequence of this creative focus, the research needs to help determine the extent to which the ad engages the listener - for example whether the story is understood, or the humour is appreciated. The clarity of communication, and the emotional response this helps engender need also to be monitored.

In summary, the diagnostic content of the questionnaire and the focus of the data analysis should reflect the intended objectives of the research. An awareness of each of these advertising mechanisms should help identify and target the objectives of this research more tightly.

RAB Guide

Measuring Radio’s Effect