DIY Surveys

More and more people are discovering the benefits of DIY online surveys; low cost solutions that allow anyone with basic browser skills to create surveys, publish them to the Internet and have the data collected in real-time in a report ready format.

Online survey websites have made a complicated and time consuming process quick and simple. There are now many people with no previous experience of designing surveys either taking the initiative and conducting their own surveys or being delegated the task by others.

As far as the process goes it could not be any easier, but the problem now is making sure that the right questions are being asked.

A respondent can be very critical if they are asked what they consider to be 'stupid questions', and frustrated if they are not given the opportunity to answer correctly, or qualify a particular answer.

Some experienced research professionals consider that DIY surveys are often poorly written and may be doing more harm than good. In some instances their concerns are valid as there are some real shockers out there, but it would be unfair and a backward step to claim that all surveys and questionnaires should be done by professional researchers.

In this article we look at different types of surveys, establish some basic guidelines and look to see where, if anywhere, professional guidance may help.

Low Risk

Every day there are many thousands of useful ad-hoc surveys conducted that are relatively simple and carry little risk even if they are not written perfectly.

Training seminars, presentations and general internal feedback, these are all examples of surveys that are simple to create and carry a low risk for being wrong enough to cause any serious damage. On the whole these types of surveys gather general feedback that can provide useful information.

Survey Galaxy includes a template library that provides a number of templates that can be used as a framework and then customised further to match the exact requirement.

Ad-Hoc surveys can prove very useful for recording the views of just a few people who may have attended a training session or even a few thousand that attended a trade fair.

Often the objective of these types of surveys is to help an organization continually improve, be it fine tuning a training course, measure everyday performance, tweak a sales message or measuring group consensus on a proposed plan of action.

Before issuing a survey, ask:

• What is the objective?
• Do all the survey questions contribute to achieving the objective?
• When the results are analysed, will they help achieve the objective?

Often a major difference between the experienced and inexperienced researcher is that the inexperienced researcher approaches the task in a serial fashion:

1. Write the survey
2. Collate the data
3. Analyse the results

The experienced researcher tends to start with the objective and work backwards:

1. What is the objective?
2. Consider analysis requirements
3. Identify sample requirements
4. Identify questions
5. Design survey

Higher Risk

Employee Satisfaction Surveys

Employee Satisfaction Surveys are a valuable management resource and although Survey Galaxy provides templates, an Employee Satisfaction survey should be tailored to the specific organization.

For organizations experiencing a period of rapid growth, employee surveys are excellent for identifying growing pains; in a period of change they are useful for informing, explaining the need for change and gauging attitudes.

It is very important to get Employee Satisfaction Surveys right; a poor survey could end up doing more bad than good.

Customer Satisfaction Surveys

Many customers like to offer their opinion, and although the feedback may not be always what an organization wants to hear it is nearly always better to know about negativity, so that it can be addressed, and when feedback is positive it is always good to have it confirmed by the people that count.

Customers need to be impressed and reassured; a poorly designed survey could cause damage by sending the wrong signals, also your customer's time is valuable so you should avoid wasting their time by sending surveys that are not properly thought out.

Measuring Performance

Surveys are often at the heart of public organizations that have league tables. Governments the world over are often keen to gather simplistic information and turn it into performance values that may well provide an indication as to performance, but more often than not, just proves who is better at manipulating the results.

For example, in an educational environment, high grades may appear to be the Holy Grail but in reality there are many more factors that need to be considered when comparing educational institutions.

A well designed survey will ensure that the institutions concentrate on providing an excellent service, and less time on working out ways to manipulate the process in order to achieve high rankings.

It is always right to reward good performance but sometimes the better performance was taking something from very poor to good rather than taking something that was already good to excellent.

Market Research

If business decisions are going to be decided on data collected from market research surveys it essential to get the surveys right.

Market Research surveys can often suffer from the questions not having the right balance, the questions being either too simplistic, or over complicated.

A product and service is often a compromise between cost, quality and time and it is therefore important that market research survey questions identify the balance that the target market would prefer.

Common Considerations

From the outset the objectives and goals of a survey needs to be identified and clearly defined and constantly referred back to to ensure that every question is a 'need to know' and not a 'nice to know', or worst still, an irrelevant and meaningless question.

Some surveys ask two questions when they could have achieved the same result with one, such as first asking 'Did you go on holiday', and requesting a 'Yes' or 'No', then following it up with 'If you went on holiday how long did you go for?'.

It is important to try and make a survey as streamlined as possible, the fewer the questions the better.

A survey publisher can know what they mean by a question, but the question must be phrased so that there is no ambiguity and those reading the question consider it in the way it was intended.

Avoid subjective words like 'a lot', 'few', 'sometimes', 'occasionally', 'often', instead define a measurement exactly such as 'once a day' - 'once a day' may be considered 'often' by one respondent, and 'frequently' by another, if the question is open to interpretation the data collected will be compromised.

A simple question like 'Did you go on holiday?' is flawed in that it does not specify any time period. Some respondents will answer 'assuming' the publisher means in the previous twelve months, some might answer 'assuming' some other period of time.

Depending on the objective of the survey a question like 'Do you go on holidays' does not cater for those people that intended to go on holiday but for some exceptional reason were prevented. In those circumstances, should they answer 'Yes' or a 'No'?

For a question to be considered useful they may need to be adequately qualified and supported by a number of other questions that will make the information useful.

Such as an airport asking leaving passengers a simple question such as 'How much did you spend while you were on holiday?' Without knowing additional information the question asked on its own is likely to be meaningless.

• Were they travelling for business, pleasure or both?
• What is their Nationality?
• How long were they in the country?
• What were the numbers and ages of the people the spending covered?
• Did their spending include accommodation?
• What type of accommodation did they use?
• What currency are they using?
• Is their answer approximate or exact?

The above is by no means a definitive list of supporting questions as much will depend on the target respondents and the specific objectives of the survey.

Number of Questions

There is no magic number, a survey should contain the minimum number of questions required to achieve the survey's objective but enough questions so that the results will provide reliable and accurate intelligence.

Mumbo Jumbo

Questions of a technical nature should not be directed at non-technical people, questions that relate to management should not be asked to junior employees and management should not be asked non-management questions.

Jargon is only acceptable if those completing the survey know the language. For example it may be acceptable for a survey aimed at surgeons to use surgical jargon, a survey aimed at military personnel to use military abbreviations. Before using jargon or abbreviations consider if the target respondents know the jargon being used, if there is any doubt, play safe and avoid or be prepared to explain any jargon.

Objectives and Perspective

For each question asked, ask if the answer will help towards the survey's goal and objective, and consider the survey from the viewpoint of the respondents, and different types of respondents, and confirm that questions will be relevant to, and can be properly answered, by them all.

When designing a survey consider respondents who may be very different to you. Consider part-time and full time employees, male and female, old and young, ethnic groups, nationality, currency, religion, those that are novices and those that are experienced.

A useful trick is to have to hand a number of profiles for fictional characters, such as:

• Joe, married with two children
• Henry, single late twenties
• Sarah, high flyer, mid-thirties
• Jane, single, two children, part-time

Then consider each question from the perspective of each profile and check that the question is relevant and the answer options sufficient to allow them all to properly respond.

A Positive 'No Answer'

Consider each question to ensure that respondents have a 'way out', it is always better to receive a positive 'I Don't Know', 'Not Applicable', 'Rather Not Say' than a forced 'Yes' or 'No'.

The 'Other' Dilemma

When a number of closed question answer options are listed, often included at the end of the list is the facility to include another answer, other than those listed.

For each of a survey's closed questions the advantage of including the 'Other' option to cater for the exceptions should be considered, however, also consider the disadvantages of using the 'Other' option.

For example, it is quite common and often 'best practice' that when asking a respondent to select their nationality from a comprehensive list to include a 'Other' option to cater for minor and new countries.

But having the 'Other' option will most certainly result in respondents responding along the lines of 'Welsh', 'English', 'Scottish', 'Irish' when for analysis purposes the survey publisher would prefer them all to have selected 'British' from the listed nationalities.

Before allowing an 'Other' answer option, weigh up the advantages and disadvantages and be prepared to have to 'fix' the data prior to any analysis.

Bias

If the objective is for the survey to be unbiased, ensure there is no bias in the question and for closed questions in the order the answer options are listed; consider randomizing the answer options to reduce the effects of order bias.

Clear, Concise and Self-contained Questions

Avoid referencing other questions, especially if they are no longer visible and so are difficult to reference, instead try to keep each question self-contained.

Don't make questions too long, consider every word in turn and if it is not essential to help clarify the question, remove it.

Professional Advice

Good and effective surveys are not rocket science but there is a lot to consider.

For anything other than 'low risk' surveys consider the benefits of enlisting professional advice at key stages of a project such as having the proposed survey critiqued by a professional.

Because critiques requires a professional's time, critiques can prove to be more expensive than the cost of deploying the survey itself, but don't let a cost effective delivery facility result in cheap and unreliable data.

If the basic survey design considerations are followed, a professional critique should only result in a tweak here or there; or ideally just provide confirmation that the DIY survey has been written like a true professional.

Results Analysis

There are vast numbers of surveys where the results speak for themselves without the need for detailed or complex analysis.

However, for surveys that are concerned with complex issues, professional advice to help interpret the data may be of benefit especially if important decisions are going to be based on the survey findings. Not every organization will have to hand someone with the necessary skill or experience and once again, the savings and simplicity that online surveys bring to the party should not negate the need for expert advice where it is most effective.

For more information or to discuss how online questionnaires can help you please contact surveys@surveygalaxy.com or visit www.surveygalaxy.com the quick, easy and cost effective way to do online questionnaires.