Recent data linkage studies show that survey misreporting is pervasive and biases common analyses. Linked data are usually convenience samples and access is restricted, making them more suitable to document than to solve the problem. I examine how incorporating information from the linked data can improve estimates and how the survey data can still be used without such information. First, I develop a method that combines public use data with an estimated conditional distribution from the linked data. It does not require access to the linked data, is simple to implement and applicable to many estimators. Using administrative SNAP records linked to survey data, I show that this method sharply improves survey estimates. I then use the linked data to evaluate corrections that mainly rely on survey data. Some corrections also yield large error reductions, particularly for univariate statistics. This makes them attractive alternatives when linked data do not exist. Finally, I examine whether estimates can be improved based on similar linked data. For SNAP, I provide evidence that extrapolation using the method developed here improves over survey data and corrections without validation data. Extrapolation reduces average error in the geographic distribution of program spending by up to 90 percent. It suggests larger program effects, such as reducing the poverty rate by almost one additional percentage point.
Creating Improved Survey Data Products Using Linked Administrative-Survey Data (with Michael Davern and Bruce D. Meyer)
Recent research linking administrative to survey data has laid the groundwork for improvements in survey data products. However, the opportunities have not been fully realized yet. In this paper our main objective is to use administrative survey linked microdata to demonstrate the potential of data linkage to reduce survey error cost effectively through model based imputation methods. We use parametric models based on the linked data to create imputed values of Medicaid enrollment and food stamp (SNAP) receipt in public use data. This approach to blending data from surveys and administrative data through models is less likely to compromise confidentiality or violate the terms of the data sharing agreements among the agencies than releasing the linked microdata. We demonstrate that it can yield substantial and measurable improvements of estimate accuracy: Using the blended values to estimate state level Medicaid enrollment and SNAP receipt for sub state areas reduces Root Mean Squared Error (RMSE) of these estimates by 81 percent for Medicaid and by 93 percent for SNAP receipt compared to estimates based on the survey data alone. Given the high level of measurement error associated with these important programs in the United States, producers should consider imputation methods like the ones we describe in this paper to create improved estimates.
Misreporting of Government Transfers: How Important are Survey Design and Geography? (with Bruce D. Meyer)
Recent studies linking household surveys to administrative records have revealed high rates of misreporting of program receipt. These studies often use data from general household surveys for one or two states. Here we use the FoodAPS survey to examine whether the findings of these studies generalize to a survey with a narrow focus and across many states. First, we study how reporting errors differ from other surveys and to what extent the differences are likely due to survey design. We find a lower rate of false negatives (failures to report true receipt) in FoodAPS than in other surveys. As we also find that a shorter time since last SNAP use reduces false negatives, the shorter recall period of FoodAPS likely contributes to its lower error rate. As in other surveys, we find household characteristics such as income, employment, and minority status to predict misreporting. False negative rates vary systematically between interviewers and the differences are not entirely explained by characteristics of the households interviewers are assigned. Second, we examine variation in how administrative and reported SNAP receipt differ between U.S. states. Understanding geographic variation in survey error is important to assess whether we can correct estimates of official statistics, such as program receipt or poverty rates, for the entire U.S. based on linked data from a few states. We document systematic differences between states in unconditional error rates but do not find evidence of substantial differences between states conditional on common covariates. We show that despite the geographic heterogeneity, extrapolating across states can yield more accurate estimates than estimates based on survey reports.
- ProgramsAdditional Tables,
Many important statistics are known from official records for the entire population, but have to be estimated for subpopulations. I describe two simple data combination methods that reduce the substantial sampling error of the commonly used direct survey estimates for small subpopulations. The first estimator incorporates information from repeated cross-sections, while the second estimator uses the knowledge of the statistic for the overall population to improve accuracy of the estimates for subpopulations. To evaluate the estimators, I compare the estimated number of female and elderly recipients of a government transfer program by county to the "true" number from administrative data on all recipients in New York. I find that even the simple estimators substantially improve survey error. Incorporating the statistic of interest for the overall population yields particularly large error reductions and can reduce non-sampling error.
Models with high dimensional sets of fixed effects are frequently used to examine, among others, linked employer-employee data, student outcomes and migration. Estimating these models is computationally difficult, so simplifying assumptions that are likely to cause bias are often invoked to make computation feasible and specification tests are rarely conducted. I present a simple method to estimate large two-way fixed effects (TWFE) and worker-firm match effect models without additional assumptions. It computes the exact OLS solution including estimates of the fixed effects and makes testing feasible even with multi-way clustered errors. An application using German linked employer-employee data illustrates the advantages: The data reject the assumptions of simpler estimators and omitting match effects biases estimates including the returns to experience and the gender wage gap. The results suggest that firm fixed effects, not match effects, are the main channel through which job transitions drive wage dynamics, which underlines the importance of firm heterogeneity for labor market dynamics.
Marginal Voters and Their Racial Prejudice (with Suzanne K. Barth and Kyung H. Park)
This paper studies whether marginal voters are more or less prejudiced in democratic elections in comparison with regular voters. We focus on low-level statewide elections in which marginal voters are plausibly ill-informed in order to mitigate concerns that the estimates reflect unobserved candidate-specific attributes that correlate with race. Both descriptive evidence and falsification tests reaffirm the research design. We find considerable evidence that marginal voters engage in more racial discrimination than regular voters. There is no evidence of bias against female candidates.
Distributional Impact Analysis Toolkit (with Guadalupe Bedoya, Luca Bitarello and Jonathan Davis)
Program evaluations often focus on average treatment effects. However, average treatment effects miss important aspects of policy evaluation, such as the impact on inequality and whether treatment harms some individuals. A growing literature develops methods to evaluate such issues by examining the distributional impacts of programs and policies. This toolkit reviews methods to do so, focusing on their application to randomized control trials. The paper emphasizes two strands of the literature: estimation of impacts on outcome distributions and estimation of the distribution of treatment impacts. The article then discusses extensions to conditional treatment effect heterogeneity, that is, to analyses of how treatment impacts vary with observed characteristics. The paper offers advice on inference, testing, and power calculations, which are important when implementing distributional analyses in practice. Finally, the paper illustrates select methods using data from two randomized evaluations.
Missing data is a frequent problem in economics, either because some variables are missing from a data set or values are missing for some observations. Researchers usually either omit the affected variables and observations or impute them. While the consequences of the former are well understood, the imputation and missing data literature has focused on the conditions under which they lead to unbiased estimates. These conditions often do not hold, but there is little evidence on the circumstances under which missing data methods improve estimates if the conditions for unbiased estimates are violated. I first examine these conditions by discussing the circumstances under which missing data methods can be beneficial and common sources of bias. I then discuss advantages and problems of common missing data methods. Two important problems are that most methods work well for some models, but poorly for others and that researchers often do not have enough information to use imputed observations in common data sets appropriately. To address these problems, I develop a method based on the conditional density that works well for a wide range of models and allows producers of the data to incorporate private information and expertise, but still allows users of the data to adjust the imputations to their application and use them appropriately. Applications to some common problems show that the conditional density method works well in practice.
Work in ProgressPlease send me an email (firstname.lastname@example.org) for further information or a current draft.
A Nonparametric Two-Sample Test of Conditional Independence
A common empirical question is whether the conditional distributions in two or more samples are identical or, equivalently, whether an outcome variable is independent of a discrete variable conditional on covariates. This includes, among other questions, whether a discrete variable such as treatment receipt is conditionally independent of predetermined variables, whether its (conditional) effects are constant or heterogeneous, whether it affects an outcome only through a specific channel, whether missing data is ignorable and whether there are peer effects conditional on the assignment variables. Studies often compare (conditional) means, but this only tests correlation, not independence. I show that testing conditional independence in two (or more) samples is equivalent to testing the equality of two joint distributions. I propose a non-parametric Kolmogorov test that is simple to implement and is consistent against all alternatives that differ at any point in the joint support of the distributions of the covariates. Other non-parametric tests and semi-parametric extensions can easily be based on the same idea. The test can be used for hypotheses that depend on estimated parameters such as a location shift by (conditional) treatment effects or a regression adjustment. Inference can be conducted by simulation.
Errors in Survey Reporting and Imputation and their Effects on Estimates of Food Stamp Program Participation (with Bruce D. Meyer and Robert M. Goerge)
Measuring government benefit receipt in household surveys is important to assess the economic circumstances of disadvantaged populations, program takeup, the distributional effects of government programs, and other program effects. Receipt of food stamps is especially important given the large and growing size of the program and evidence of its effects on labor supply, health and other outcomes. We use administrative data on food stamp participation in two states matched to American Community Survey (ACS), Current Population Survey (CPS), and Survey of Income and Program Participation (SIPP) household data. We find that 23 percent of true recipient households do not report receipt in the SIPP, 35 percent in the ACS, and fully 50 percent do not report receipt in the CPS. Both false negative and false positive reports vary with individual characteristics, leading to complicated biases in food stamp analyses. Our results are also informative about the reasons for misreporting and the success of different survey methods. We then directly examine biases in research finding, in particular the determinants of program receipt using our combined administrative and survey data. Our results differ from conventional estimates in showing higher participation by single parents, non-whites, middle-income households, and other groups. We directly examine one source of potential error, Census Bureau imputations, finding that excluding the imputed observations leads to worse ACS estimates, but has little effect on the CPS and SIPP estimates.
What Leads to Measurement Error? Evidence from Reports of Program Participation in Three Surveys (with Pablo Celhay and Bruce D. Meyer)
Measurement error is often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult for data producers to reduce the extent of errors and for data users to assess the validity of analyses using the data. We study the causes of survey error using high quality administrative data on government transfers linked to three major U.S. surveys. The differences between survey and administrative records show that up to six out of ten cash welfare recipients are missed by the surveys. We find that misreporting by respondents, survey design features, and imputation of missing data induce substantial error. Our results for non-imputed respondents confirm several theories of misreporting, e.g. that errors are related to event recall, forward and backward telescoping, salience of receipt, respondent's degree of cooperation, and the stigma of reporting participation in welfare programs. Our results provide guidance on the conditions under which survey data are likely to be accurate and suggest different ways to reduce survey errors.
Stigma in Welfare Programs (with Pablo Celhay and Bruce D. Meyer)
Stigma is often cited as a main reason why eligible individuals do not participate in welfare programs. It is also advanced as a force that may lead only the most deserving to apply for aid. Yet, there is little convincing evidence for a stigma of program participation. We study how misreporting of SNAP and Public Assistance receipt in household surveys is related to stigma. We posit that program participation should be less stigmatizing when participation in social programs is a common behavior within a person's social network. We empirically test whether misreporting participation is negatively associated with true participation in the same program by neighbors. Our results suggest that welfare participation is stigmatizing, i.e. households that live in areas of high participation are more likely to truthfully report participation. This conclusion is supported by instrumental variable estimates where we take advantage of geocoded data to instrument local participation with distance to the nearest county Department of Social Services office. The evidence for stigma is strong in the case of in-person or phone interviews where we expect stigma to be more pronounced, and weak for mail-in responses. We find no support for the alternative hypothesis that respondents are more cooperative in areas of high participation.
An Empirical Total Survey Error Decomposition Using Data Combination (with Pablo Celhay and Bruce D. Meyer)
Survey error is known to be pervasive and bias even simple, but important estimates of means, rates, and totals such as rates of employment and poverty. To summarize and analyze the extent, sources, and consequences of survey error, we define empirical counterparts of key components of the Total Survey Error framework that can be estimated using data combination and linkage. Specifically, we estimate average total survey error and decompose this average into three sources of error: survey coverage , item non-response and measurement error. For error in dollars received from two large government transfer programs, we use administrative records on the universe of program payments in New York State linked to three major household surveys to estimate the error components we define. We find that total survey error is large and varies in its size and composition, but measurement error always constitutes the dominating error source. Our application shows that data combination and linkage make it possible to routinely measure total survey error and its components. This could help survey producers to cost-effectively reduce survey error and survey users to mitigate the consequences of survey errors or gauge the reliability of their conclusions.
How Does Potential Unemployment Insurance Benefit Duration Affect Re-employment Timing, Sorting and Wages? (with R. Felder and H. Frings)
How potential benefit duration (PBD) of unemployment insurance (UI) affects wages and matching is crucial to evaluate PBD extensions as a policy tool and to understand the causal relation between unemployment, job search and wages. Discontinuities in PBD at age cut offs have allowed recent research to identify the effect of PBD extensions, but even questions as elementary as whether PBD increases or reduces re-employment wages are highly debated and the causal mechanisms through which UI and PBD affect labor market outcomes are unclear. To clarify the nature of PBD effects and to separate unemployment duration, search and selection effects, we decompose wages into individual, firm and match specific wage components. We use the discontinuity in PBD from 12 to 18 month from Schmieder et al. (2016) to identify effects of PBD on these wage components. Adding to evidence on the importance of firms, we find that the entire wage loss operates through the firm fixed effect. Wages do not rebound, so the effect does not appear to be transitory. We then decompose the re-employment wage path into the same components to study re-employment dynamics. The decomposition allows us to better distinguish dynamic selection from dynamic treatment effects by controlling for time-invariant individual characteristics. Even though PBD does not affect the average individual and match fixed effect, it does affect how these re-employment wage components vary with unemployment duration. The individual fixed effect provides evidence of dynamic selection that is not monotone in unemployment duration. Changes in the firm fixed effect are partly offset by changes in the dynamics of the other components, which may reconcile the diverging findings regarding effects on search and the direction of wage effects. Our preliminary results indicate two effects in addition to the previously documented unemployment duration effect. High-wage individuals with prior jobs at high-wage firms that exit to low-wage firms tend to leave unemployment around benefit exhaustion. This sorting implies that the estimated effects of PBD depend on who enters unemployment and sample restrictions. Studies of mass layoffs, for example, will depend on the firm fixed effect of the involved firms. Rather than being driven by this postponement effect, the effect of PBD on wages appears to be mainly due to a downward shift in the firm-specific re-employment wage component at shorter unemployment durations. Overall, our preliminary results show that dynamic selection plays an important role in the effects of PBD as well as in the steep decline of re-employment wages with unemployment duration. We will use the detailed firm- and individual-specific information in our data to shed more light on the nature and underlying causes of both sorting and wage loss in the final paper.
Errors in Reporting and Imputation of Government Benefits and Their Implications (with Pablo Celhay and Bruce D. Meyer)
We document the extent, nature, and consequences of survey errors for receipt of cash welfare and SNAP in three major U.S. household surveys linked to administrative program records. Our results confirm earlier evidence of frequent misreporting of program receipt, particularly failure to report receipt. The surveys poorly capture patterns of participation in multiple programs, even though there is not much evidence of program confusion. Error rates are higher among imputed observations, which also account for a large share of false positive errors. Many household characteristics have significant effects on errors in reporting receipt, both false positives and false negatives. Among others, we find large differences in survey error by race, ethnicity, and income. We provide evidence on the consequences of these errors for models of program receipt. Estimated effects of, among others, income and race are biased downward. On the positive side, our results confirm a tendency to attenuation in models with misclassified dependent variables that preserves many qualitative results in the survey estimates. We then examine error due to item non-response and imputation, as well as whether imputation improves estimates. Item non-respondents have higher receipt rates than the population, even conditional on many covariates. The joint distribution ofs program receipt and these covariates differs between item non-respondents and the entire population, but also between accurate and imputed receipt among item non-respondents. Thus, neither excluding item non-respondents nor using their imputed values yields consistent estimates. For binary choice models of program receipt, estimates from the linked data favor excluding item non-respondents over using their imputed values. The biases in each case are well predicted by the error patterns we document, so such analyses can help researchers make more informed decisions on the use of imputed values.
Misclassification-robust Semiparametric Estimation of Binary Choice Models (with Pavel Cizek)
Misclassification of binary variables is pervasive (see e.g. Marquis and Moore, 1990, Meyer, Goerge and Mittag 2015 and the references therein) and necessarily correlated with the true value of the variable. Since it generally leads to biased estimates (Black, Berger and Scott 2000, Meyer and Mittag 2015), estimation of binary-choice models in the presence of misclassification has been studied both in the parametric, semiparametric (Hausman et al., 1998), and nonparametric context (Lewbel, 2000). In applied studies, the main approach has been to assume constant-rates of misclassification (CRM) as in Hausman et al. (1998). However, this assumption commonly does not hold, resulting in large biases in the parametric estimators that require it (e.g. Cizek, 2008; Cizek and Sadikoglu, 2015, Meyer and Mittag, 2015). Given the difficulties of a general approach to misclassification in cross-sectional data (Lewbel, 2000), consistent point estimates seem only feasible if CRM holds exactly or a model of misclassification is available. Often, neither is the case in applied research, but CRM may be a reasonable approximation. This project aims to evaluate whether recent advances in econometrics can improve the instability and large bias of commonly used estimators. Specifically, the goal is to develop and evaluate robust estimators that are consistent under CRM, but that are insensitive to deviations from CRM in terms of their bias. This would allow researchers to obtain potentially biased, but still reliable estimates from contaminated data. To achieve this, the estimation strategy will be based on the bias-correction approach developed by Gourieroux et al. (2000) in the parametric setting and generalized by Cizek and Sadikoglu (2015) to the semiparametric estimation. The advantage of the approach stems from the fact that an auxiliary estimation criterion can be used, which does not necessarily have to be consistent, but can possess other beneficial properties.
- Estimating Effects of School Quality using Multiple Proxies. With Pedro Bernal and Javaeria A. Qureshi. Labour Economics (2016).
- Misclassification in Binary Choice Models. With Bruce D. Meyer. Journal of Econometrics (2017).Programs
- Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net With Bruce D. Meyer AEJ: Applied Economics Accepted. Web Appendix Tables