Statistical Biases and Measurement: Introduction

Introduction (PDF)

Welcome to what I believe will be one of the most important sections of the Sources of Statistical Biases Series: Measurement. Besides the existence of confounders, I strongly believe that the measurement of a construct represents one of the largest sources of statistical bias in all scientific disciplines. This belief stems from the numerous researcher degrees of freedom – decision points – that exist when conceptualizing and operationalizing a construct.

The conceptualization of a construct is discipline specific and, often times, is based on the data collection techniques or the available data. Numerous conceptualizations can exist for a single construct, the differences between which contribute to the observation of distinct statistical estimates when conducting a regression analysis. Meaning that how we define our constructs is important when examining the influence of one construct on another construct.

To provide a criminal justice example, our conceptualization of recidivism can drastically impact the outcome of an evaluation. In particular, three important components exist when defining recidivism. First, a conceptualization of recidivism requires one to define a start date for what counts as recidivism. Here are just a few examples of what can be considered a start date for a recidivism event:

1. Date arrested for a crime
2. Date convicted of a crime
3. Date sentenced for a crime
4. Date placed on probation
5. Date placed in prison
6. Date place on parole
7. Date probation was successfully completed
8. Date probation was successfully or unsuccessfully completed
9. Date released from prison
10. Date parole was successfully completed
11. Date parole was successfully or unsuccessfully completed

The distinctions between the possible start dates will influence the period in which the recidivism could have occurred and, in turn, increase or decrease the number of individuals that could have recidivated. After defining the start date, we have to define what counts as recidivism. Here are some options used throughout the correctional literature:

1. Rearrest for non-criminal/ non-probation or parole related violation
2. Rearrest for a technical violation or new crime
3. Rearrest for a new crime (misdemeanor or felony)
4. Rearrest for a new crime (felony)
5. Reconviction for a new crime (misdemeanor or felony)
6. Reconviction for a new crime (felony)
7. Reincarceration for a technical violation or new crime
8. Reincarceration for a new crime (misdemeanor or felony)
9. Reincarceration for a new crime (felony)

As demonstrated throughout the literature, rearrest and measures including technical violations tend to increase the rate of recidivism within a sample. This is because individuals previously involved in the criminal justice system tend to have an increased likelihood of being arrested for a non-criminal behavior and/or experience a probation violation resulting from non-criminal activities (disobeying officer orders, missing probation/parole meetings, failing to maintain employment, etc.). Finally, we have to define the timeframe that we are examining:

1. 1-year from start date
2. 2-years from start date
3. 3-years from start date
4. Etc.

The longer the timeframe, the higher likelihood of an individual recidivating, which is extremely important when studying specialized populations. For instance, sex-offenders tend to sexually-recidivate at an extremely low rate for approximately three years when released back into the community. All of these decision points when conceptualizing recidivism can result in meaningful variation in the magnitude of the association for the causes and effects of recidivism. Recidivism, however, is not unique as many constructs across scientific disciplines can have multiple conceptualizations. Nevertheless, while understanding the distinctions between conceptualizations is important to evaluating and interpreting the resulting coefficients, the specificity this discussion requires is beyond the scope of this series. In particular, simulation analyses would be useless, as we would have to focus in on key constructs across scientific disciplines and discuss how definitional differences bias statistical estimates. As such, the Measurement section of the series will primarily focus on how we operationalize constructs.

The operationalization of a construct is extremely important when generating inferences about causal associations within a population. The importance begins no sooner than with the level of measurement and the creation of a construct. The level of measurement can be (1) dichotomous, (2) ordinal, or (3) continuous,^[i] while a construct can be created various different ways, either being derived from a single item or a representation of multiple items (e.g., aggregation or average). The first five entries of the Measurement section will focus on these topics:

1. Level of Measurement: Dependent Variable
2. Level of Measurement: Independent Variable
3. Measurement Creation: Dichotomies, Ordinal Measures, Aggregate Events, & Variety Scores
4. Measurement Creation: Aggregate Scale, Average Scale, Standardized Scale, Weighted Scale & CFA
5. Measurement Error

After discussing issues related to the measurement of a construct, our discussion will switch gears and focus on how the distribution of a dependent, lagged endogenous, or endogenous variable can influence the estimates derived from various statistical models. 

1. Distributional Assumptions for the Dependent Variable (Linear Regression Models)
2. Distributional Assumptions for the Dependent Variable (Mixed-Effects Linear Models)
3. Distributional Assumptions for the Dependent Variable (Non-Parametric Regression Models)
4. Distributional Assumptions for the Lagged-Endogenous Variables (Structural Equation Modeling)
5. Distributional Assumptions for the Endogenous Variable (Structural Equation Modeling)

Finally, we will end the Measurement section by discussing the clustering of data across space and time. This discussion will focus on how clustering influences statistical estimates, as well as the interaction between clustering and measurement when estimating regression models.

1. Longitudinal Clustering
2. Spatial Clustering

Overall, the discussions and data simulations should provide insight into how measurement can bias the statistical estimates produced by a regression analysis. Now, let’s get into our discussions by focusing on biases related to the level of measurement for the dependent and independent variables.

---

[i] Two brief points, when estimating a statistical model the level of measurement can not traditionally be nominal – except for when estimating multinomial regression model – and often times requires the dichotomization of each option. Interval and ratio level of measurement both describe continuous constructs, but traditionally have distinct distributional assumptions.

License: Creative Commons Attribution 4.0 International (CC By 4.0)