ONE of the greatest challenges that confront a social science researcher during a pandemic is how to conduct field work. This is particularly most difficult when there are strict lockdowns and mobility of nonessential personnel are restrained by rules and regulations.
After all, conducting academic or political research, including surveys, generally do not qualify as essential work. It is therefore relevant to ask how surveys, such as that conducted by Pulse Asia and the Social Weather Stations, are done, and whether they can be reliable.
As one who teaches social science research methods, I can personally vouch that there are other ways of gathering data during the pandemic. Face-to-face key informant interviews and focus group discussions could be replaced by online interviews conducted via meeting platforms like Zoom and Google Meets. Direct observation, while may not be feasible, can be replaced by asking respondents or research collaborators in the field or on site to personally document by videotaping or writing diaries or field notes of the events that needed to be observed.
But then, there is always the limitation that these observations are no longer distilled and analyzed directly by the researcher and are now done through intermediaries. One of the key requirements in qualitative research, in which direct observation is one method of data gathering, is that the researcher’s role is important in being the primary instrument for conducting the analysis, and thus it is imperative that he or she must be immersed in the actual condition being observed. This is something that would be compromised when they are unable to conduct actual field observations.
As for cross-sectional surveys, even without the pandemic there are already ways that can be administered without face-to-face interactions. Surveys can be conducted either by telephone, or through the release of online surveys to targeted respondents using applications such as Google Forms. In fact, in the US and other countries, surveys are mostly done through these methods of data gathering.
The thing about telephone surveys, however, is that they presuppose that they will not undermine the random nature of the sample, in that they provide everyone in the population an equal chance or probability of being included in the sample. This is because it can be assumed that all voters are accessible because they have either a landline or a mobile phone. Thus, the sample can be truly representative of the population, and that it is not biased in favor of those who can be accessed through their telephone lines to the exclusion of those who cannot. This is precisely why in countries like the US, surveys through telephones and mobile phones can easily take the place of face-to-face surveys.
There, not only are phone lines are more extensive, but mobile phone signals are also more reliable.
That cannot be said, however, of the Philippines. Relying solely on landlines or mobile phones would definitely skew the sample in favor of those who have phone service subscriptions. For those who use mobile phones, the sample would be skewed in favor of those with reliable signals. What further complicates the data gathering process for mobile phones is that while it is easy to attribute a post-paid number to a location, which is important in drawing up a sample that will cover the entire country as it can help the survey taker to determine the proportions drawn from specific geographical areas, such may not be easily done for prepaid numbers.
In addition to the fact that some people would block, or refuse to respond to, unidentified callers, there is also the difficulty of ascertaining the demographic characteristics of the survey respondents, which is important in analyzing cross-sectional survey data, in the case of telephone surveys. There is little assurance that those who answer the call will be truthful about their age or of other information.
Apparently, there are methodological challenges in conducting telephone surveys in the Philippines. Such challenges are also present in online surveys via Google Forms, as it would limit the sample only to those with internet access. Online surveys via social media platforms are never reliable, considering that the representativeness of the respondents relative to the total population is severely compromised. This is precisely why even during the pandemic, survey takers still endeavor to conduct face-to-face data gathering, when it is permitted.
The main challenge in any survey is not only logistical, or in how samples are drawn and whether they truly represent the population, but also in how the questions are framed and how these, and even the survey takers, are perceived by the respondents, particularly in face-to-face interactions. It is possible that the manner people respond would be influenced by who they think the survey-takers are. We call this in social research jargon the Hawthorne Effect.
The accuracy of surveys was tested recently in the 2020 US elections, when there was widespread failure of surveys to correctly reflect the real outcome in many key races. Also further complicating the reliability of surveys is that there are variable results coming from different survey firms.
In the case of the Philippines, and considering the unique challenges that we face, made even more difficult with the onset of the Covid-19 pandemic, it behooves survey firms such as Pulse Asia and SWS to always remind people about the limitations of surveys. I urge these firms to level with the people, and inform them of such limitations, in the same manner that drug or cigarette manufacturers label their products with cautionary warnings.
It would also help if survey firms always detail the manner that they have formed their sample, and how they gathered their data. It is not enough to vaguely refer to these generically. They have an obligation to state these fully, completely and accurately in a language that ordinary people will understand. Media, in turn, should do the same.