ESOMAR-37 Questions
What is ESOMAR-37 Questions
The primary aim of these 37 Questions is to increase transparency and raise awareness of the key issues for researchers to consider when deciding whether an online sampling approach is fit for their purpose. Put another way, the aim is to help researchers to ensure that what they receive meets their expectations. The questions are also designed to introduce consistent terminology for providers to state how they maintain quality, to enable buyers to compare the services of different sample suppliers.
Notes on the context of the questions explain why the questions should be asked and which issues researchers should expect to be covered in the answer.
It should be noted that these 37 Questions focus on the questions that need to be asked by those buying online samples. If the sample provider is also hosting the data collection you will need to ask additional questions to ensure that your project is carried out in a way that satisfies your quality requirements.
The 37 Questions complement ESOMAR’s Guideline to Online Research which was revised in 2011 to add updated legal and ethical guidance and new sections on privacy notices, cookies, downloadable technology, and interactive mobile.
Company Profile
What experience does your company have in providing online samples for market research?
Research4Insights shortly known as R4I hosts the world’s largest consumer network for digital market research, consisting of more than 1+ million research participants in over 130 countries – accessing more than 4,500 individual market research panels and sample sources. The R4I platform offers a unique and transparent marketplace environment called the Insight Exchange for buyers and sellers of sample to connect and transact by matching a seller’s market research panel and respondent profiling data against the buyer’s sampling needs in real-time. This makes R4I the most effective and efficient way to access sample to conduct digital market research at scale and at speed.
Founded in Texas, USA in 2012, R4I is trusted by over 2,600 agencies, researchers and brands in more than 72 countries worldwide. Since R4I’s inception, innovation has been a cornerstone of the company’s growth.
Panel and sample source partners in the R4I platform include market research agencies, media owners, publishers (digital and traditional), non-profits and companies with access to large-scale web traffic. All panels use consent or some other legal basis for processing personal data and are operated in compliance with ISO 20252. As stated above, these are augmented by carefully selected and vetted supplemental panel-based sources.
All panels accessed via the R4I platform are operated in compliance with accepted industry codes, including, but not limited to ESOMAR and the Insights Association.
Sample sources and recruitment
Please describe and explain the type(s) of online sample sources from which you get respondents. Are these databases? Actively managed research panels? Direct marketing lists? Social networks? Web intercept (also known as river) samples?
Research4Insights actively manages our own research panel and also works with a network of trusted partners. All panelists are validated healthcare professionals, opted-in to receive invitations to surveys via email, and are all actively managed and profiled by R4I to the same exacting standards, including quality of response, engagement and incentives.
If you provide samples from more than one source: How are the different sample sources blended together to ensure validity? How can this be replicated over time to provide reliability? How do you deal with the possibility of duplication of respondents across sources?
Due to the size and strength of our own panel, R4I does not often require additional sample. When we do include samples from external providers, it is only with the prior approval of our clients, and we will only use pre-identified partners who work to our approved standards. In such cases, our technology allows us to detect users taking the same survey with other providers, so we can remove them from the data as required.
Are your sample source(s) used solely for market research? If not, what other purposes are they used for?
The R4I’s Panel is used solely for market research purposes. Under no circumstances do we use our system to advertise or market to survey respondents.
How do you source groups that may be hard to reach on the internet?
R4I actively recruits hard-to-reach respondents (such as younger people and those from ethnic minorities) via a network of partners with access to a wide range of online sources that cater to these groups. These sources have specific experience in recruiting these audiences for online activities.
These recruitment programmers include search engine optimization (SEO), affiliate networks, niche websites, and refer-a-friend programmers.
If on a particular project, you need to supplement your sample(s) with sample(s) from other providers, how do you select those partners? Is it your policy to notify a client in advance when using a third party provider?
The R4I’s Panel was designed to successfully implement, monitor, manage, and deliver projects using multiple sample sources and types – with complete transparency. Our Marketplace users fully control which suppliers are allowed to enter the project via the Marketplace Interface as well as pass-through supplier identifiers so that the sample mix can be tracked within the survey platform. While in-field, suppliers can be added or removed according to project requirements with complete transparency.
Additionally, when working with our Marketplace Services team, we notify all clients about bringing on new partners to complete their projects. We work with our clients to identify any additional partners they are utilizing to ensure we de-duplicate them on our platform.
Sampling & Project Management
What steps do you take to achieve a representative sample of the target population?
R4I offers complete customization of quota set-up, as well as template quota breakouts for frequently used census representations. Using R4I proprietary sampling technology, we begin by framing quotas based upon the census or profile of the required population. This frame is the basis on which our sampling software controls the flow of members into each survey. Our sampling system will randomly select from our available panel, and allocate to surveys according to the quotas set.
Do you employ a survey router?
R4I Panel includes a router. This removes the potential for self-selection on surveys, and increases our ability to deliver lower incidence samples within a short time frame.
If you use a router: Please describe the allocation process within your router. How do you decide which surveys might be considered for a respondent? On what priority basis are respondents allocated to surveys?
Panelists receive an invitation containing a survey or panel link. When they access the link the router will check against quotas on all live surveys and allocate them to a survey for which they qualify. We can manually control prioritization (for example to ensure that a particularly urgent client survey can meet quotas in the necessary time frame), but allocation is predominantly determined by the quotas defined in the frame for each survey. If, following more detailed screening, a respondent doesn’t qualify for the survey to which they are allocated they will be returned to the router and allocated to another survey.
If you use a router: What measures do you take to guard against, or mitigate, any bias arising from employing a router? How do you measure and report any bias?
In any case where we consider there could be a potential for bias resulting from use of a router, we will revert to a manual invitation process. All surveys running on the router will have sampling controls on all variables likely to affect the results.
If you use a router: Who in your company sets the parameters of the router? Is it a dedicated team or individual project managers?
All parameters are set and controlled by the panel team within our Operations department. These are only changed following explicit consultation with all affected parties.
What profiling data is held on respondents? How is it done? How does this differ across sample sources? How is it kept up-to-date? If no relevant profiling data is held, how are low incidence projects dealt with?
Basic profile data is collected at the time of registration and this includes country, language, specialty and workplace name. As soon as possible thereafter we ask members to complete a detailed profiling survey that covers additional details on specialty and applicable subspecialties, years of practice, workplace setting, conditions treated, patient load per condition and procedures conducted.
Beyond this, we have a proprietary system that automatically updates panelist demographics from survey data (typically every 3-6 months) depending on the relative frequency with which particular data points are likely to change.
Please describe your survey invitation process. What is the proposition that people are offered to take part in individual surveys? What information about the project itself is given in the process? Apart from direct invitations to specific surveys (or to a router), what other means of invitation to surveys are respondents exposed to? You should note that not all invitations to participate take the form of emails.
All our panel member invites are personalized and will contain:
- the project name and reference
- a brief description of the project
- the length of interview
- incentive due for completion
- unique survey link and a set of unique login details
- deadline for survey completion
- Targeted e-mail invitation
- Survey link within a respondent-facing platform
- contact information
- privacy and voluntary participation statement
Please describe the incentives that respondents are offered for taking part in your surveys. How does this differ by sample source, by interview length, by respondent characteristics?
All incentives are handled by our supply partners. Each incentive program is unique. Incentives come in the form of:
- Cheque
- PayPal
- Cash
- Gift cards
- Loyalty rewards
- Charitable Donations
- Rewards Points
What information about a project do you need in order to give an accurate estimate of feasibility using your own resources?
The information we need stems from the client’s requirements:
- Who do they wish to survey,in terms of demographics, behaviours and/or attributes? This will enable us to estimate incidence and develop quotas.
- What do they want to know? This will enable us to design the survey(s) to meet their needs and understand the length and nature of the survey(s) we will be running.
- When do they want to know it? Defining an end date will allow us to manage the client’s survey(s) as part of the suite of fieldwork we are running, to ensure optimum response.
Do you measure respondent satisfaction? Is this information made available to clients?
Within projects, panel members have the opportunity to leave comments about the survey completed. We are happy to share relevant feedback from members with clients.
We also closely monitor on-going panel behavior by reviewing attrition and response rates. Employ a dedicated panel support team that deals directly with panelists’ comments and complaints. All member enquiries are logged, categorized and analyzed to provide early warning of emerging issues.
What information do you provide to debrief your client after the project has finished?
We will happily provide clients with disposition reports that contain metrics (such as start and completion rates, terminated rates, quota full rates and so on) and any other details relating to the project fielding.
Data Quality & Validation
Who is responsible for data quality checks? If it is you, do you have in place procedures to reduce or eliminate undesired within survey behaviors, such as (a) random responding, (b) Illogical or inconsistent responding, (c) overuse of item non-response (e.g. “Don’t Know”) or (d) speeding (too rapid survey completion)? Please describe these procedures.
We have several techniques available to us to ensure that clients receive the highest quality data on which to base analysis. Specifically, we have mechanisms to identify and remove:
- Speeders
- Respondent’s IP is outside of the survey country
- Those that fail trap questions
- Habitual non-responses
The suite of quality checks will be defined with the client at the start of a project to ensure they align with any existing quality standards. In addition to removing fraudulent responses from the survey data, those panelists that fail these tests are marked in the database and ultimately removed from the panel should they turn out to be consistent offenders?
How often can the same individual be contacted to take part in a survey within a specified period whether they respond to the contact or not? How does this vary across your sample sources?
Working with closed and in some cases small respondent groups, the issue is not so much one of any individual supplier limiting survey invitations, but the industry as a whole doing so. The reality is, if R4I does not invite a member to a survey then someone else will.
On average, panelists complete a survey no more than once per week, a threshold that ensures continued engagement without over-burdening panelists with too many surveys.
However, we are acutely aware of the potential impact on response rates of over invitations and therefore we monitor the situation very carefully without imposing generic limits. We believe that managing for engagement is the better approach.
How often can the same individual take part in a survey within a specified period? How does this vary across your sample sources? How do you manage this within categories and/or time periods?
Sample buyers determine the frequency of an individual’s activity within a specified period. We de-duplicate within a survey, survey group, or category by IP, participant ID, or cookies. These settings are easily accessed and adjusted through Marketplace Interface.
If you use a router: What measures do you take to guard against, or mitigate, any bias arising from employing a router? How do you measure and report any bias?
In addition to profiling data collected directly from the respondent, R4I also collects respondent data concerning:
- Source Information
- Date of Entry
- Survey participation history
- Individual response rates
Project summaries of this information are available to clients upon request.
Do you have a confirmation of respondent identity procedure? Do you have procedures to detect fraudulent respondents? Please describe these procedures as they are implemented at sample source registration and/or at the point of entry to a survey or router. If you offer B2B samples what are the procedures there, if any?
We applies a number of checks to ensure the quality of survey data, including, for example, trap questions that test panelist responses against pre-collected demographic data.
The suite of quality checks available allow us to detect fraudulent responses as well as duplicate respondents, and where panelists fail quality checks they are marked in the database as suspect and are removed from the panel if they turn out to be consistent offenders.
Policies and compliance
Please describe the ‘opt-in for market research’ processes for all your online sample sources.
We uses a double opt-in procedure across all panels. At the point of the first opt-in panelists are asked to confirm they are happy to accept the Terms and Conditions of membership. Only once they’ve agreed to these will they be sent a confirmation email and a link to double opt-in and activate their panelist account.
Please provide a link to your Privacy Policy. How is your Privacy Policy provided to your respondents?
Our privacy policy appears as a link at the bottom of each page in the website https://research4insights.com/ and is present throughout the screening process.
Our priority is to protect the information of the panelists and to the end, we specify in the privacy policy all the purposes for which we collect personal data and allow panelists to opt-out in case they don’t want us to use their information. In specific cases where clients are asking for contact information of a panelist for A.E., we ask panelists for informed consent first before releasing their information.
Since the GDPR guidelines have gone into effect, we also ask our members to read and actively accept our privacy policy as we’ve made improvements to the way we collect and use private information.
Please describe the measures you take to ensure data protection and data security.
All panelist data, personally identifiable or otherwise, is stored on UK-based servers that are maintained behind secure firewalls, IDS and Threat Management appliances and located in,site that is certified to the latest ISO standard (ISO/IEC 27001:2005). Personally identifiable information is stored separately from all profile data as a means of further securing panelist data.
What practices do you follow to decide whether online research should be used to present commercially sensitive client data or materials to survey respondents?
Though there are no flawless methods of protecting in-survey content, R4I can offer both general and specific protection to web-based client material. First, all R4I panelists are subject to clear restrictions on the re-use of material by virtue of their agreement to our website https://research4insights.com/ where you can see our terms & condition link in footer section. Second, to augment this general protection, R4I can limit the lifetime of any link to audio, video and other content, such that it expires after a certain period of time, and cannot thereafter be accessed.
Are you certified to any specific quality system? If so, which one(s)?
Our third party data center provider is certified to the following standards, providing industry-approved protection to all R4I data:
- ISO 9001:2008 ISO/IEC 27001:2005
- Payment Card Industry Data Security Standard 0
- Member of the British Polling Council
Do you conduct online surveys with children and young people? If so, do you adhere to the standards that ESOMAR provides? What other rules or standards, for example COPPA in the United States, do you comply with?
Panel registration is limited to those over 16 years of age, in line with MRS guidelines.
In cases where our research seeks to understand the views of those under the age of 16, R4I seeks permission from the parent or guardian when screening our own panelist and ensure that guardian know the nature and details of survey.