How to Read a (Quantitative) Journal Article

Prepared for Sociology 210
Instructor: Greta Krippner
September 21, 2000

Note: Handout refers to Jeremy Freese, Brian Powell, and Lala Carr Steelman, “Rebel Without Cause or Effect: Birth Order and Social Attitudes,” American Sociological Review 64 (1999): 207-231.

The first thing to realize is that quantitative articles follow a formula. They all have more or less the same structure: an introductory section in which the problem is introduced and the objectives of the paper are previewed; a theoretical section in which the literature that relates to the problem addressed in the paper is described; a data section where the data sources for the analysis are described; the analysis or results section, where the various statistical tests performed are explained and the findings presented; and finally, a discussion or conclusion section in which the main findings are linked back to the theoretical literature.

The most important thing to realize about reading a quantitative article is that (nearly) everything that is presented in the tables is discussed in the text. So read the text along with the tables. The text will draw your attention to which numbers in the tables are important.

Your first task in reading the text is to identify what problem is being addressed by the research. Typically, this will be clear in the first one or two pages. In the Freese paper, the authors identify their problem (pp. 208-9) as testing the effects of birth order on various social attitudes, including conservatism. In addition to identifying what the problem is, try to determine who or what the author is arguing against—i.e., where does the author situate him/herself in existing debates? In the Freese paper, the authors are arguing against Sulloway, who they recognize has made a major contribution by being the first to study the relationship between birth order and social attitudes (p. 208), but who they criticize for suggesting that birth order is more important than standard sociological variables (gender, race, class, age, number of siblings).

Next, you should identify the relevant variables in the study and how they are measured. In the Freese (pp. 213-215) study, the main independent variable is birth order, measured dichotomously—i.e., the respondent is first born or the respondent is not first born. Similarly, the dependent variable, social attitudes, is operationalized using six specific measures: political self-identification, opposition to liberal social movements, conservative views of race and gender, support for existing authority, and “tough mindedness.” Each of these measures of social attitudes is operationalized in turn. For example, Freese et al. (p. 215) ask respondents to indicate how patriotic they are (“How proud are you to be an American?”) as a measure of the variable “support for existing authority.”

The “Results” section is the core of the article. It is also the hardest to read, because most technical. The text will help you to interpret the tables. The first thing you must figure out is how variables are coded—i.e. what does a positive versus a negative coefficient mean? For example, the Freese (p. 215) article notes that measures are coded so that positive coefficients are consistent with the hypothesis that first borns are more conservative in their social attitudes. Negative coefficients, then, do not support the hypothesis. There are two significant coefficients in the first model (p. 216). “Significance” means that the observed effect is strong enough that we can rule out chance as an explanation of the observation. Significant effects are indicated with an asterisk (or several asterisks—meaning we can be even more confident that the observation is not produced by chance). In this case, the first significant coefficient is a positive number. We can interpret this as saying that first borns are more likely to vote for Bush, which supports the hypothesis. On the other hand, the negative coefficient on the significant “tough on crime” measure tells us that first borns are less likely to be tough on crime than later born children—this contradicts the hypothesis. On balance, then, this first model does not lend much credence to birth order theory—only two of 24 measures are significant, and of these two, only one supports the hypothesis that first borns are more conservative. Hmmm….not very convincing, right?

The next thing to notice, however, is that there are various “models.” Specifying different models allows the researchers to take more than one crack at discerning a pattern in the data. In this case, Freese and his co-authors know from other research that variables such as sex, age, race, parents’ education, and sibship size are related to social attitudes. So perhaps there really is a relationship between birth order and conservative attitudes, but it is being obscured by these other variables. The way to handle this possibility is to introduce the various demographic variables as control variables, which means holding them constant so that the effect of birth order can be isolated. This is what Freese et al. are doing in Model 2. But they still don’t find much of a relationship between birth order and social conservatism. Look for the significant coefficients in Model 2. What do they indicate?

Not to be dissuaded, the researchers throw more controls into Model 3 and Model 4. The additional controls specify other factors known to be correlates with social attitudes—parents’ occupational prestige, parents’ marital status, the loss of a parent before age 16, childhood religion, region of the country in which the respondent was raised (MODEL 3); and respondent’s education and occupational prestige (MODEL 4). But in Models 3 and 4, just as in Model 2, only 3 of 24 measures of social attitudes are significant, and they are also in the wrong direction! Remember, because of the way the variables are coded, a negative number contradicts the hypothesis that first borns are more conservative.

So, on this evidence, support for birth order theory is weak. But notice what Freese et al. (pp. 218-219) do next. They now examine each of the variables that served as controls in “Model 2”—sex, age, race, parents’ education, and sibship size—and compare their effect to the effect of birth order. Notice that in Table 2 these variables are no longer functioning as control variables—they are not being held constant, but rather allowed to vary, so that they can be related to variance in the dependent variable. Freese et al. are able to show that these variables are far more powerful predictors of social attitudes than is birth order—for each variable, at least 12 of the measures are significant. However, in looking at the pattern formed by significant measures, Freese et al. (p. 219) note that only age is consistent—the other independent variables tend to contain contradictions. For example, respondents with well educated parents tend to be more liberal on attitudinal measures than respondents with less well educated parents, yet they are also more likely to identify themselves as Republican than Democrat. Freese et al.’s (p. 219) conclusion from all of this is that labels like “conservative” may not actually capture a unified set of values, and that perhaps proponents of birth order theory achieved their results by relying on vague concepts that actually have little purchase in the real world.

Typically, following the main analysis, researchers will try several other tests to establish the robustness of their findings. They want to be sure that the results they are getting are not a quirk of the particular way they manipulated the data. In the Freese paper, the authors establish the robustness of findings by using a different data set—one that has intra-familial data—and by testing a wider variety of measures of social attitudes from the GSS. Neither of these tests changes their results. This increases their confidence that their results are correct.

A final test done by the researchers is for interaction effect. The idea of an interaction effect is that the way a certain variable operates is affected by the presence or absence of another variable. The interaction effect they are testing is birth order and spacing of children: theory suggests that the effect of birth order on social attitudes is most pronounced when there is moderate spacing (2 to 5 years) between adjacent siblings. Again, there is no evidence from their analysis of the data that this is the case.

In sum, in interpreting tables like Table 1 and Table 2 in the Freese paper, there are two things to consider: 1) are any of the variables significant? And 2) if significant, does the given variable affect the dependent variable in the predicted direction?

Vocabulary

Bivariate Relationship: a relationship between two variables considered in isolation from other factors. The Freese paper first examines the bivariate relationship between birth order and each measure of social attitudes (pp. 215-217).

Control Variable: A variable that is held constant so that the effect of a third variable (which you are interested in) on the dependent variable can be observed more clearly. Generally, you control by selecting only individuals who share in common the variable you are controlling for (age, gender, etc.) Whenever a variable is held constant, that variable cannot account for variation in the dependent variable, so you are eliminating its effect from consideration. For example, if you want to explain variation in levels of aggression and you control for gender by studying only males, then the variable “gender” cannot account for any of the observed variation in aggression. This does not mean that there is not a relationship between gender and aggression, but it is not the relationship that you are studying. Holding variables constant is a means of simplifying complex social situations by ruling out variables that are not of immediate interest but that might otherwise explain part of the phenomenon that the investigator wishes to understand. In the Freese study, sex, age, race, parents’ education, and sibship size are included as controls because the authors want to eliminate their affect on social attitudes so that they can observe the effect of birth order directly (p. 215).

Correlate: a variable bearing a relationship to another variable.

Correlation: a measure of the strength and direction of the relationship between two variables. Correlations vary between –1.00 and 1.00. Thus, they may be positive (a direct association), indicating that two variables tend to move in the same direction, or negative (an inverse association), meaning they move in opposite directions. Education is positively correlated to earnings. Smoking is negatively correlated with longevity. Correlations vary in strength: there is a 0.95 correlation in the height of identical twins, but only a .34 correlation between education and earnings (for men age 25-34 in 1988). A correlation of 0 indicates that there is no relationship between two variables. Note: correlation does not imply causation because it does tell us which variable is cause and which is effect.

Dependent Variable: A variable that the researcher tries to explain or predict; the presumed effect of one or more independent variables.

Dichotomous Variable: a variable which can take on only two values, yes or no. In the Freese (p. 213) paper, birth order is a dichotomous value: either the respondent is first-born, or the respondent is not first born.

Explanatory Variable: another term for an independent variable—i.e., a variable which is presumed to be a cause of the dependent (outcome) variable.

Independent Variable: a presumed cause of the dependent variable.

Interaction Effect: An outcome in which the effect of one independent variable on the dependent variable varies according to the value or level of another independent variable. That is, the effects of the variables together differ from the effects of either alone.

Multivariate Relationship: a relationship between many (more than two) variables considered simultaneously.

Null Hypothesis: The hypothesis that there is no relationship between variables you are testing. When Freese et al. (p. 217) talk about the “null results” they mean the evidence provided by their data of no effect of birth order on social attitudes.

Operationalization: the detailed description of research operations or procedures necessary to assign units to the categories of a variable. More concretely, how are abstract concepts (which we represent with a variable such as “social attitudes”) measured? In the Freese (p. 214) study, opposition to a liberal social movement is an operationalization of the concept social attitudes.

Population: The total membership of a defined class of people, objects, or events; also called universe. In the Freese paper (p. 212), which uses GSS data, the population is all non-institutionalized, English-speaking adults in the United States.

Regression: A statistical technique for studying linear relationships among variables. Regression may either consider bivariate relationships (called simple regression) or multivariate relationships (called multiple regression).

Regression Coefficient: A regression coefficient is a way of expressing the effect of one variable on another. Under specific conditions, a regression coefficient has a rather neat interpretation: it indicates how much the dependent variable changes with a one-unit change in the independent variable.

Robustness: Robust results are results that hold up under a wide range of different kinds of tests. You do the statistical test one way and get the result; you do it another way and you produce the same result; you use different data and you still have the same result—then your finding is robust.

Sample: A subset of cases (typically individuals) selected from a population for study. The sample also is used to specify the method of selection—i.e., a random sample or a full probability sample is a sample in which every individual with the desired attributes has an equal probability of being selected. Samples may be further restricted for the purposes of answering certain questions. For example, when measuring beliefs about racial equality, Freese et al. (p. 214) restrict their sample to whites in order to ensure comparability of their results with the research that they are critiquing (which used an all-white sample).

Significance: The effect of a given variable is statistically significant if it is sufficiently strong that we can rule out chance as an explanation of the observed relationship.

Standard Deviation: A measure of how much variability there is in a set of observations. The standard deviation—also called the variance—indicates the average “spread” of observations around the mean. Observations that are tightly clustered around the mean have low variance. Observations that are widely dispersed have high variance.

Note: Many definitions based on Singleton, Straits, and Straits, 1993, Approaches to Social Research.