Contents

• >> Introduction
• >> Executive summary
• >> Defining the research objectives
• >> The importance of split samples
• >> Where to do the research
• >> When to do the research
• >> The research sample and sample sizes
• >> Method and questionnaire
• >> Appendix 1:

“Radio memories and how to access them”

Marketing Week FACTFILE from Dec 1998 by Jonathan Chapman, Clark Chapman Research

This guide has been created in response to the growing number of enquiries received by the Radio Advertising Bureau about researching the effectiveness of radio campaigns.It is designed as a step-by-step guide for those involved in the advertising process.

It aims to provide a better understanding of effectiveness research and research and researching radio in particular.

We would particularly like to thank Clark Chapman Research and Millward Brown International for their help in compiling this guide.

Effectiveness research requires clarity of objectives – what are the agreed objectives of the overall campaign and of the radio campaign within this? Remember not all objectives can best be measured by consumer research (e.g. sales effects).Radio effectiveness can be measured either using continuous research or in stages (“pre & post”) – the pre-stage is normally the week before the campaign, the poststage in the week after the campaign finishes.

Consumers tend to misattribute radio advertising memories to other media, particularly TV. This is particularly likely to happen where there is a strong executional link between the two media and/or where there is an established history of TV advertising for the brand.

This tendency to misattribute can be offset by using matched samples of listeners and non-listeners. This way, if the increase in advertising awareness is greater among listeners than it is among non-listeners, then the effect can be attributed to radio fairly confidently – even if the listeners think the advertising was in another medium.Radio research can successfully be done using telephone interviewing – ads can be played down the line. However cases where other media are to be included in the research it might be more appropriate to use face-to-face interviewing.

Commercial recognition is a valuable technique – i.e. playing the ads to consumers. It provides a more robust measure of whether they have heard the campaign, and avoids problems of trying to describe the ads. Brand names can be bleeped out of the commercial, to test whether the campaign is linked to the brand.

The key to any successful research is to have a clear understanding of why the research is being conducted in the first place. In other words, what are you aiming to measure?In broad terms, radio advertising research aims can be categorised into two types:

…Marketing issues – to what extent has radio helped to achieve the campaign aims?

…Media planning issues – what effect do different media strategies have on the performance of the campaign?

MISATTRIBUTION OF ADVERTISING

When asked to consider advertising, consumers will turn their thoughts to the most salient source they can think of – this tends to mean TV. Television, as the medium with the most active expectations, tends to dominate memories of advertising, with the result that campaigns in all other media are, to varying extents, attributed to television in the consumer’s mind.This misattribution is disproportionately likely to happen with radio and is still more likely to happen when radio campaigns are creatively synergistic with TV executions.

AVOIDING MISATTRIBUTION: USING SPLIT SAMPLES

The simplest solution to the problem of measuring true radio awareness is to split your sample into two parts: listeners (target consumers who have been listening to the radio stations which carried the advertising) and non-listeners (people who do not listen to those stations, but who are the same as the listeners in all other respects).If the only difference between the two samples is their radio listening, then any differences in their awareness or attitudes to the advertised brand can be reasonably attributed to radio – regardless of where they think they have seen or heard the advertising.It is particularly important to use split samples where radio is part of a mixed media schedule in order to guage the true radio effect.

WHICH OPTION SHOULD YOU CHOOSE

Neither of the two approaches above is necessarily better than the other. You might want to bear in mind,however, that the second method has the advantage of questioning people who will have the same history of exposure to your brand. Local distribution levels for the brand will also be the same. But remember that for some target markets, such as young people, it can be hard to recruit a sample of non-listeners in the same area (sometimes almost all of them do listen to Commercial Radio).The key point is that the listener and non-listener samples must be matched as closely as possible in terms of demographics, media consumption and weight of exposure to your advertising in other media. This ensures that any differences can confidently be attributed to radio ad exposure.

TIP: HEAVY, MEDIUM & LIGHT LISTENERS Asking listeners how many hours a day they listen to a given station allows you to categorise them by weight of listening – this can be broken out on the analysis.

This can be useful. For example, if increased response is positive but actually confined to heavy listeners, there may well be a mandate for increasing frequency against medium and light listeners.

TEST AND CONTROL SAMPLES IN DIFFERENT AREAS

This involves taking two matched samples of respondents in different geographical areas and comparing their advertising responses – one sample will live in the advertised area, the other in an area where no radio advertising ran.In this way, it will be possible to compare the results among those who have been exposed to the campaign with the results among those who have not - thus giving you a measure of radio’s effectiveness.It is important to match the media consumption of the samples (e.g. how much TV they watch etc) as well as their demographics, as this could affect response. It is equally important to ensure weight of advertising for your brand in all other media is the same for both samples.The two geographical areas should also be comparable – (or “typical”) in terms of media and product consumption as a whole.>

TEST AND CONTROL SAMPLES WITHIN THE SAME AREA

In this second approach, all of the research is done within the same area.One part of the sample will comprise people who do listen to the station(s) on your radio schedule, whilst the other part of the sample will comprise people who do not listen to any station on your schedule.Again, in this way it will be possible to compare the results of those who have been exposed to the campaign to those who have not giving you a measure of radio’s effectiveness.

The ideal research method is to monitor advertising activity on a continuous basis, since this allows movements in advertising response to be compared directly to current advertising activity. Often, however, continuous radio research is impractical on grounds of cost unless it forms part of ongoing advertising tracking.Typically, radio research is conducted in two stages - a pre-campaign and a postcampaign study.

…The pre-campaign study should be conducted as close to the start of the radio campaign as possible – preferably during the week immediately preceding the radio campaign. This will establish the base levels of whatever is being measured (eg brand awareness).

…The post campaign study should be conducted as soon as possible after the radio campaign has ended – ideally during the first week after the campaign has come off air.

In some instances you might want to consider conducting more than two stages of research. For example, it might be worth slotting in an additional research phase during a particularly long advertising campaign or sponsorship. Similarly, having done the post-research, you might want to consider adding an additional stage of research some weeks after a campaign has ended in order, say, to track decay in brand awareness.

TIP: NO TIME FOR A PRE-STAGE MEASURE?

There is a “rough and ready” solution for times when a pre-stage has not been possible.

If non-listeners are used as a control sample (i.e. people who would not have been exposed to the radio advertising campaign) they are in a sense equivalent to a pre-stage sample.

In other words the difference, following a radio campaign, between a matched sample of listeners and non-listeners, is equivalent to the difference between listeners before and after a campaign. If you need to consider this as an option, remember the following:

…it does not take account of your other media activity

…it doesn’t take account of anything else which happened during the radio campaign(e.g. competitive activity etc).

Generally speaking, the larger the sample the better. However at some point, the cost of an increased sample size becomes cost prohibitive and contributes little extra to statistical robustness.The table below shows details of what would often be recommended as minimum and suggested sample sizes. Note however that if you intend to analyse the results amongst sub-groups of your target audi-encea larger sample will probably be required. As a general rule the greater the complexity of the task the larger the sample size needed. Your research agency can advise on this.

TIP: THE ADVERTISING TARGET AUDIENCE DEFINES THE SAMPLE

If the advertising is tasked with changing the attitudes of ABC1s, is there any reason to gauge the views of C2DEs?

Telephone research is often used for assessing the effect of radio campaigns: the method is adaptable and can often be cheaper than face-to-face interviewing. Radio ads can successfully be played down the phone to respondents.

Face-to-face interviewing may also be preferable if respondents need to be shown visual ad material such as stills from TV ads.

Commercial recognition is a valuable technique – i.e. playing the radio ads to consumers – as this is the best “memory jogger” of all. It also delivers a larger sample of people who are identifiable as having definitely heard the campaign: this is useful when analysing them for their attitudes to the brand. For more detail on the commercial recognition method see Appendix 1 on “Radio memories and how to access them”.N.B. when playing the radio commercials in order to measure commercial recognition, two different approaches can be taken: blind or branded.

blind

By bleeping out all brand references in each execution and asking whether the commercial has been heard before and then asking for the brand name, it is possible to see whether creative treatment has successfully linked the message to the brand.

branded

This allows prompting for brand-specific data, (e.g. attitudes to the advertising/feelings about the proposition), whilst giving a true measure of ad recognition.

A fairly straightforward questionnaire will take around 10-15 minutes to run through – much longer and respondents will begin to lose interest and concentration!

QUESTIONNAIRE STRUCTURE

It is important to ensure your research agency’s draft questionnaire includes all your original aims to ensure survey results encompass all factors you wanted to measure.Pre- and post-campaign questionnaires will largely be the same (although the former can often be shorter).Key elements of a typical pre- and post-campaign research study would include the following:

• General questions on the category in which your brand operates
• General questions on brand usage and advertising awareness
• General questions on media consumption and specifically questions that can separate listeners to the radio stations used in the campaign from nonlisteners
• Recall of advertising. At the post-stage, you will be seeking to detect spontaneous and prompted awareness
• Commercial recognition – playing the ads to respondents
• Thoughts on what the main message of the ads was
• Thoughts on what the main message of the ads was

RADIO MEMORIES AND HOW TO ACCESS THEM

Radio campaigns are known to be potentially tricky to evaluate. Jonathan Chapman reports on a recent study which offers guidance.

It is said that local advertisers measure the success of their radio campaigns by what happens to cashflow – if the advertising isn’t making a difference in the tills, it is dropped or changed.

If only everything in life were this uncomplicated. The reality for most national advertisers is that radio’s effect is a lot more difficult to evaluate. In part this is because radio has historically played only a very minor part in the advertising plans, although this is a situation which seems to be changing.

However the bigger problem is that the noise from the other 95% of advertising tends to get in the way. Every researcher knows that asking about radio advertising awareness can generate unbelievably low recall figures: we commonly see awareness scores which are under-reporting, seemingly because the method hasn’t taken account of the way people consume radio.

Radio is, like posters, almost subliminal in the way it operates. Most listeners are doing something else – driving, eating, chores – with their attention zoning in and out during music, news, ads, weather etc. In addition radio isn’t seen to be the natural home of brand advertising – consumers always think of TV when asked about advertising.

So what can we do about it? What is best practice in terms of measuring memories of radio advertising?

To answer these questions, the Radio Advertising Bureau recently commissioned us to carry out a live test of different methodologies. We surveyed consumer recall and response to six different campaigns which were running on Capital Radio (the test brands involved were Feminax, TV Licensing, Prudential, P&O Stena, Silentnight Beds and Channel 5, but all the figures are averaged across the six).

To offset the possible effect of misattribution we used a split sample of listeners and non-listeners – this is essential to get a clear reading on the role of radio.

Spontaneous awareness scores told us that, on average, 6% of the Capital listeners could tell us that the test brand had been advertising on radio. The level of noise in the numbers here is clear from the fact that 4% of non-listeners answered the same way.

Prompting with test brand names on a showcard helped this to rise to 13%, and it rose again to 19% when listeners were prompted with the medium as well (e.g. “Which, if any, of these have you heard advertised on radio recently?”).

This prompted stage is about as far as many surveys go and, to be fair, 19% is a reasonable level to see significant shifts.

However, as can be seen in the graph, we went two stages further in the survey. The third stage of prompting was where the interviewers read out a description of the ad, (e.g. “This advert is in the form of a lighthearted drama where a couple are being told at the check-in desk at the airport that they simply cannot bring their bed with them into the aeroplane”), and asked the respondents whether they had heard the ad.

Perhaps not surprisingly, scores varied widely on this method, leading to the conclusion that some ads are more describable than others!The fourth stage was commercial recognition – we literally played the ads to the respondents and asked whether they had heard them.

As can be seen in the chart this revealed that, on average, 63% of the listeners could confirm they had heard the campaign (this figure excludes an additional 9% who said “maybe”).

The difference between 6% and 63% is significant in anybody’s language.

Clearly, memories of radio advertising are very sensitive to the method of prompting. So what does this mean for the way we conduct radio advertising research?

We don’t really have a right and wrong method here – we have different results when we use different techniques, and this is a matter of horses for courses.

If you want to know how likely listeners are to think of you as being a radio advertiser, spontaneous awareness scores will tell you this (not surprisingly, the Carphone Warehouse tends to score well on this – the brand has achieved an explicit association with the medium).

Verbally describing radio campaigns to listeners seems to be a seriously unreliable method – and it is actually measuring how describable your ads are, so it’s not particularly useful.

Commercial recognition is clearly the most efficient method in terms of finding out whether people have heard your campaign. This is valuable, because one can look at the correlation between this and brand knowledge or attitudes.

Meanwhile, it seems that the method we normally use is the prompting by brand and medium – this may make sense in terms of economics (it can be added to a TV tracking study) but will inevitably under-report true levels of radio advertising memories. On this basis it should certainly be questioned.

Jonathan Chapman is a director of Clark Chapman Research.

This article is based on FACTFILE from Marketing Week of December 1998, and summarises a special project into the methods used for researching radio campaigns.