Ivory Tower Writing #5: Deciding your methods

This post provides an introduction to the issue of choosing your methods in research.

Once you have decided on a research question, you’re now likely going to start choosing what methods to apply in your research.

Depending on your field of study, your go-to methods will likely be either quantitative or qualitative. Some may say that quantitative methods are better than qualitative methods. Note that neither method is “better” than others absolutely. Imagine trying to cut a steak using a bread knife. Theoretically, you could do it, but then it’ll take a goddamn long time and you’re not doing justice to a prime cut of meat. Instead of insisting on using a bread knife, go buy a proper steak knife. That analogy illustrates the issue of methods: none are absolutely better, they just have different utility depending on what you’re trying to cut. As King, Keohane, and Verba wrote in Designing Social Inquiry (1994):

“All good research can be understood—indeed, is best understood—to derive from the same underlying logic of inference.”

Besides, choosing between qualitative or quantitative isn’t even half the battle.

Qualitative or quantitative?

Described simply, qualitative research pays more attention to data that is usually difficult to quantify. Instead, it relies on interpretation, description, and explanation of sometimes abstract concepts. One example is the idea of justice. What is justice? Who ought to dispense justice? If high-order normative philosophical questions aren’t your cup of tea, other types of qualitative research can be found in history or anthropology. You may want to understand how the Indonesian New Order was like by going through memoirs of dead political activists. Numbers can only tell you how many people died, how much was the price of gas, etc. without establishing a degree of context. Qualitative research attempts to immerse you, the researcher, into the phenomena. Done correctly, qualitative research can generate a lot of insight into a particular phenomena, like de Tocqueville’s seminal work, Democracy in America.

On the other hand, quantitative research pays more attention to numerical data, such as statistics, and tries to draw inferences from that data. It is concerned with data like “How much has the price of bread increased over the years and what does this tell us about inflation?” or “How many times has major wars happened over the last 50 years from 1945?” The tools that they use often include statistical software and datasets. This method is often used in economics and, these days, political science.

Using sub-methods in research

Now that you’ve defined your major approach, it’s time to choose the “sub-methods”, which is a term I came up with to describe the research activities that will help you answer the research question.

Before we proceed, it is beyond the scope of this article to discuss ALL “sub-methods”; you should be familiar with these within your area of discipline. I’m just trying to point out HOW you may want to use these sub-methods in your designing your research.

Let’s return to your research question. For the sake of example, let’s use the one I reverse-engineered from Syailendra’s paper (paywall):

Does disagreement between key Indonesian domestic institutions contribute to a non-balancing response towards an assertive China?

Immediately, there are several variables that you want to clarify. What are the “key domestic institutions”? What is a “non-balancing response”? Is China really “assertive”?

How are you going to describe these variables? You’re likely going to immediately look up government legislation or other scholarly books to understand Indonesia’s key domestic institutions; more books on balancing theory in the Realist canon; and previous research on China’s alleged assertiveness.

But that’s just the beginning. Your main objective now is to actually answer the entirety of the question. Let me walk you through the process.

At this point, you’re probably (you should!) already well-versed in the required theoretical knowledge. You know that states tend to balance each other when confronted with a rising power. They can either align with the rising power, another larger power, or with other smaller powers. But then why is Indonesia not doing that? Is Neorealist theory wrong then? Maybe not. Indonesia could be an outlier. Let’s take a look at the Neorealist canon again. Oh, there’s something here about threat perception. It says that states do not always respond swiftly towards threats; it depends on how they perceive them. That’s a good starting point! (Also, this is something called a literature review and will be addressed in a later post.)

You then posit that there has to be something wrong with how the leaders of the key domestic institutions (such as the legislature, executive, and defense community) perceive China. You think that maybe these three institutions don’t agree with one another on what China is and whether or not it’s a threat. And because of that, according to theory, Indonesia isn’t balancing as it should.

Now, how are you going to SHOW that this disagreement exists? Here’s where your sub-methods come in play.

You may want to dive into the literature again and see how Indonesians have historically perceived China. You find out that China has long been perceived as a threat, although the context is different than what it is today. This sub-method is basically a much more focused literature review, and is usually useful for providing context.

Since your question deals with government institutions, you may want to interview the ministers about how they perceive China. Ask the same question to three different ministers, perhaps, and compare the answers. Oh, you find that they don’t agree on the nature of the China threat. One minister thinks they are a direct threat; another thinks they are a potential economic partner. This reveals that there is indeed a difference in opinion regarding China! This sub-method is the good old interview and is a good way of collecting data that you won’t find in books. This is easier said than done, though, and I’ll get back to this sometime in the future.

If you’ve read Syailendra’s article, you would know he does a mixture of document analysis (i.e. reading statements, following the news, reports, etc.) and interviews. He manages to show that there is indeed a difference in perception of China; however, whether or not this difference of opinion affects Indonesia’s non-response to China remains to be seen.

Other considerations

I’ve described two of my most frequently used sub-methods, but there are others too!

If you’re more quantitative-oriented (but still don’t like numbers), you may be interested in conducting a “coded content analysis”, which basically measures the frequency of a selected “code word” appearing in spoken or written communication. You could choose two code words such as “China” and “threat”, and measure the frequency of appearance in government statements, academic journals, or the news. A higher frequency suggests China is perceived as a threat (according to either government, academics, or the media). One example is Johnston’s article in International Security on China’s perceived assertiveness [PDF].

Note that quantitative methods aren’t the go-to method in this case. This is simply because the nature of the question demands more of the qualitative methods than they do of statistical analysis. So let’s try another research project where quantitative methods are more useful.

Back when I was a research intern at CSIS Jakarta, I had to do a project related to the 2014 Presidential Election. Basically, it was data entry for thousands of questionnaires related to their choice of presidents. Off the top of my head, here’s a research question I could formulate:

What characteristic did voters prefer in the presidential candidates?

In the questionnaire, there were questions like “Why do you prefer Jokowi?” etc. So, all you had to do was tally the responses, and you had your answer. You could go further! Maybe you’re interested in what women prefer in a presidential candidate. Easy. Just go through the dataset again. This is a classic questionnaire, perhaps the go-to sub-method when you want to amass a lot of statistical data on a certain issue. Of course, you need a degree of skill to be able to develop meaningful questionnaires. I may get to this in the future.

But what if we’re trying to establish inference or a causal relationship with statistics? This goes beyond my expertise, although what I know that you would need to test the p-value of your data; but the finer details elude me. Just don’t be too hasty to conclude causality when you’ve only established correlation.

Conclusion

Hopefully, this post helps you decide which methods to choose when writing your paper. Just remember that there is no such thing as a “perfect” method; all methods have their significant strengths and weaknesses. Making sure that you acknowledge these will help you argue your point better.

Thanks for reading, and if you want to expand this list, feel free to write in the comments!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress.com.

Up ↑

%d bloggers like this: