During the middle of the twentieth century, America was a divided nation, and a growing number of people were losing faith in established social norms and institutions. When I enrolled in Brooklyn College in 1964, I was apprehensive. I was drawn to sociology because it addressed my unease and framed it in questions about social order. The first time I read C. Wright Mills’s book, The Sociological Imagination (Oxford University Press 1959), I imagined that if I truly wanted to understand social order, it would be good if I could see the world through the eyes of everyone who ever lived. After graduating, I enrolled in the M.A. program, in sociology at Brooklyn College. I took a required qualitative methods class with Alfred McClung Lee, who was ASA president from 1976-77, and from him I learned that by studying society through participant observation and open-ended interviews I could begin to learn how people give meaning to their own experience with other people around them. After that I enrolled in a PhD program in sociology at Temple University, where I learned that the sociology I was taught at Brooklyn was atypical. It was the early 1970s and with new technology, methods, and statistics, sociology increasingly was becoming a quantitative field.
Meaning and Measure: Finding Evidence through Research
Evidence from social research is always important, especially when you are doing research that can inform public policy and practice in ways that will enhance rather than diminish our social experience. The challenge is that there are different forms of evidence from social research, depending on how you understand and explain society and on the nature of the question or questions you ask. Quantitative methods view social phenomena as objects and emphasize experimentation to eliminate plausible explanations and are designed for the study of relationships among distinct and clearly defined and measured variables. Qualitative methods view social phenomena as subjects and emphasize finding meaning in symbolic representations and are designed to study common and uncommon patterns among social phenomena that are considered to be analogous. And just as there are different research methods that can be a source of evidence that is recognized and accepted as confirmable, credible, trustworthy, and compelling, similarly, there is more than one setting in which to do social research that yields such evidence.
Lessons Learned from Doing Social Research in Different Settings
After being awarded my PhD in sociology at Temple University in 1977, I took a position on the faculty of a small teaching college in Albany, New York, the state capital. I wanted to do research and teach. Unfortunately, I was one of the only faculty members at my college who was interested in research, so I did not even have the support I would need to do qualitative studies. In 1982, after I was awarded tenure, I received an offer from the New York State government for a research position. I resigned from my teaching position and accepted the offer.
By the time I started working for New York State, the Governor had created the Office of Program Development and Research (OPDR) in the State Division of Criminal Justice Services (DCJS; see https://www.criminaljustice.ny.gov/). Whenever someone is arrested and fingerprinted in the State, a copy of those fingerprints is sent to DCJS. These are coded, computerized, and then examined for “matches” against a computerized database of previous arrests across the State. In 1983, there were already between four and five million sets of fingerprints (representing arrests, not arrestees) in this computerized database, with hundreds of thousands added each year since. The agency uses these data to produce “rap sheets,” which are accounts of individual criminal histories to be sent to the local police agency that made the arrest and to the court that would process the case. Besides collecting arrest data, DCJS collects and maintains relevant data from every criminal justice agency in the State, so for each arrestee the rap sheet becomes the official account of all their known prior contacts with the criminal justice system. The idea was for OPDR to use these data to produce statistical reports and conduct research that would then be used to inform criminal justice policy and practice and make our social world a better place for all people.
Not only would working at OPDR allow me to conduct interesting and possibly influential social policy research, but it would give me an opportunity to learn more about innovations in technology, quantitative methods, and statistical analysis. Ultimately, I spent more than a decade at DCJS. I had a unique opportunity to collaborate with quantitative researchers doing what today would be called mixed-methods studies. Naturally, our research had to be policy-relevant. For example, when prison overcrowding held the attention of government policymakers and practitioners, we did evaluations of what were called “alternative to incarceration” programs. When crack cocaine and homicide held their attention, we did studies of the relationship between drugs and violent crime. We designed studies incorporating statistical analyses of the data available through our agency and semi-structured open-ended interviews with people who had personal experience and a personal stake in what we were studying. It was an opportunity for me to learn more about what statistics could and could not tell us, and what participant respondents could and could not tell us.
OPDR had three bureaus (statistics, research, and policy) that were intended to work in harmony to feed information to the DCJS commissioner, who was also the state director of criminal justice, and therefore the principal criminal justice policy advisor to the governor. My last few years at DCJS, I was appointed chief of the Bureau of Statistical Services. My staff of analysts, programmers, and data entry clerks was responsible for designing, developing, maintaining, and operating a variety of statewide criminal justice databases and statistical reporting programs. We also produced official statistical reports and responded to questions from the governor and state legislature. Our databases were longitudinal, statewide, and included data relevant to all elements of the criminal justice process.
Much of what I learned at DCJS was not surprising, but the experience enhanced my knowledge and understanding of why research is important and of the strengths and limitations of different research methods. I learned how policy research is a reticular process, drawing researchers and policymakers together to identify and implement informed policies and programs. I learned firsthand about the significant challenges of defining and measuring crime for statistical analysis. For example, with so many crimes of all types not reported or not recorded, and sometimes not even recognized or acknowledged, we cannot accurately know or measure the amount of crime in a given place or among a given group of people. Consequently, when we need a measure of crime, we end up using dubious surrogates—either crimes known to the police or victimizations reported by a statistically drawn sample of people.
Eventually, I thought it was time for me to do research in another setting to apply what I had learned at DCJS about how quantitative methods and statistical analyses could inform policy and practice. I resigned from DCJS and began what would be a series of positions in a variety of research settings. At that time, I still thought about being a professor, so immediately after leaving DCJS I accepted a faculty position at the University of Baltimore (UB), an academic setting in a city with illicit drug problems in serious need of attention, but not an R1 research institution.
While I was a professor at UB, I continued to work with researchers from a New York City research institute I had worked with while at DCJS, Narcotic and Drug Research, Inc. With federal funding, we did mixed-methods studies of the relationship between drugs and life experience and about the organization and operation of illicit retail drug markets. We wrote a number of journal articles and policy papers, but I did not have the access I had in New York to policymakers. And while Baltimore was a good place to study drugs and society, UB had limited capacity to support my research.
After five years at UB, I accepted a position to be the director of the Drugs and Crime Research Division at the National Institute of Justice (NIJ) in the U.S. Department of Justice, a federal government agency. There I had access to federal data and policymakers comparable to the access I had to state data and policymakers as a state government employee in New York. As part of my job at NIJ, I was executive director of Arrestee Drug Abuse Monitoring (ADAM), a national program that annually conducted approximately 35,000 interviews and collected urine samples from arrestees within 48 hours of arrest in 35 U.S. counties.
ADAM gave me access to a large database of information on drugs and drug users, and another opportunity to learn more about both the potential and limitations of statistical analysis for informing policy and practice. Among these limitations was the fact that the interview data were self-reported by people who had just been arrested and were sitting in jail awaiting booking. And while the goal of the urine testing was to understand the relationship between drugs and crime, the respondents who were tested were rarely arrested at the time of their crime, thereby limiting the value of the test results as a measure of drug involvement when crimes are committed.
After four years at NIJ, I took positions first as the director of the Center on Crime, Drugs, and Justice at Abt Associates (a for-profit social research organization) and then as a senior vice president at NORC at the University of Chicago (a not-for-profit social research organization). Both organizations are recognized for their policy studies, and especially for their survey research. At both Abt and NORC, I worked, for example, with scholars who had expertise in writing the wording for survey questions. The challenge is that knowing what something means is more important than being able to count it. I learned in real time about the importance and ways of wording your question so that it asks what you think it asks and want it to ask. I learned why it is so important that respondents need to clearly understand what you are asking, how both the researcher and the respondent need to be in agreement about what the question is asking, and how the question you are asking actually needs to provide you with a measure of what you are trying to measure.
I also worked with experts in sampling who would select a random sample of respondents, or at least a representative sample, for our studies. They also were expert in knowing how to account for the shortcomings of a sample. All of this, of course, is mathematical, not substantive, and as a result can inform the research team about the mathematical probability of the sample being appropriate for the analysis, but not necessarily if the selected respondents truly represent the population they are intended to represent. There are mathematical tests that can be run to see if there is bias, for example, if those who declined to participate are different in a mathematically significant way from those who agreed. But that goes back to substantive significance, which again is the question of whether or not you are measuring what you say you are measuring and what is important to measure, and whether or not you are counting the people you say you are counting.
After that, I went back to academic institutions to do research, first as an associate dean for research at Virginia Commonwealth University and now as a distinguished research professor at West Virginia University. Whatever I had learned from my previous positions in various settings about the promise and problems of the different methods of social research as a source of evidence for social policy and practice has helped me in my experience doing research at these academic institutions.
Method and Setting: Social Research in the 2020s
In America during the early decades of the twenty-first century, we are perhaps even more divided than we were and have even less faith in our social norms and institutions than we did during the middle of the twentieth century. So, it is not surprising that after half a century of trying, I still cannot understand how social order is possible. In fact, given the world we live in today, I worry that even if social order is possible, it may not be sustainable. Of course, change is reasonable, progress can be beneficial, and new forms of order should be expected, so it is not necessarily a bad thing for a social order to evolve. Either way, from the experience I have had doing social research in a variety of ways and in a variety of settings, I do know now that the importance of having evidence to support policy, practices, and programs that do enhance and do not debase our social experience may be greater than ever.
That said, if we are to use evidence from social research to make decisions and take actions about social policies and practices that will impact our lives and the lives of other people, we need to recognize that there are different forms of evidence, be able to distinguish trustworthy evidence from evidence that is not, and to acknowledge the strengths and weaknesses of whatever it is we call evidence. This is true no matter what method is used to acquire the evidence, and what setting we are in when we do the research.
Any opinions expressed in the articles in this publication are those of the author and not the American Sociological Association.
