Recent validation studies show that survey misreporting is pervasive and biases common analyses. Addressing this problem is further complicated, because validation data are usually convenience samples and access is restricted, making them more suitable to document than to solve the problem. I first use administrative SNAP records linked to survey data to evaluate corrections for misreporting that have been applied to survey data. Second, I develop a method that combines public use data with an estimated conditional distribution from the validation data. It does not require access to the validation data, is simple to implement and applicable to a wide range of econometric models. Using the validation data, I show that this method improves upon both the survey data and the other corrections, particularly for multivariate analyses. Some survey-based corrections also yield large error reductions, which makes them attractive alternatives when validation data do not exist. Finally, I examine whether estimates can be improved based on similar validation data, to mitigate that the population of interest is rarely validated. For SNAP, I provide evidence that extrapolation using the method developed here improves over survey data and corrections without validation data. Deviations from the geographic distribution of program spending are often reduced by a factor of 5 or more. The results suggest substantial differences in program effects, such as reducing the poverty rate by almost one percentage point more, a 75 percent increase over the survey estimate.
Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net (with Bruce D. Meyer)Supplementary Tables
We examine the consequences of underreporting of transfer programs in household survey data for several prototypical analyses of low-income populations. We focus on the Current Population Survey (CPS), the source of official poverty and inequality statistics, but provide evidence that our qualitative conclusions are likely to apply to other surveys. We link administrative data for food stamps, TANF, General Assistance, and subsidized housing from New York State to the CPS at the individual level. Program receipt in the CPS is missed for over one-third of housing assistance recipients, 40 percent of food stamp recipients and 60 percent of TANF and General Assistance recipients. Dollars of benefits are also undercounted for reporting recipients, particularly for TANF, General Assistance and housing assistance. We find that the survey data sharply understate the income of poor households, as conjectured in past work by one of the authors. Underreporting in the survey data also greatly understates the effects of anti-poverty programs and changes our understanding of program targeting, often making it seem that welfare programs are less targeted to both the very poorest and middle income households than they are. Using the combined data rather than survey data alone, the poverty reducing effect of all programs together is nearly doubled while the effect of housing assistance is tripled. We also re-examine the coverage of the safety net, specifically the share of people without work or program receipt. Using the administrative measures of program receipt rather than the survey ones often reduces the share of single mothers falling through the safety net by one-half or more.
Creating Improved Survey Data Products Using Linked Administrative-Survey Data (with Michael Davern and Bruce D. Meyer)
Recent research linking administrative to survey data has laid the groundwork to create improvements in survey data products (microdata, summary files, and report tabulations). However, due to a variety of constraints, including cost of altering existing processing systems, the opportunities have not been fully realized. In this paper we use administrative-survey linked microdata to create estimates of Medicaid enrollment, and food stamp (SNAP) receipt. The Root Mean Squared Error (RMSE) of these estimates reduces by 81 percent for Medicaid enrollment and 93 percent for SNAP receipt compared to estimates based on the survey data alone, demonstrating measureable improvements in survey estimate quality. These substantial gains in data quality are achieved by using a parametric model based on the linked data as the basis for creating blended values. The approach to blending data that we propose does not compromise confidentiality of the data, privacy of the respondents or violate the terms of the data sharing agreements among the agencies.
- ProgramsAdditional Tables,
Many important statistics are known from official records for the entire population, but have to be estimated for subpopulations. I describe two simple data combination methods that reduce the substantial sampling error of the commonly used direct survey estimates for small subpopulations. The first estimator incorporates information from repeated cross-sections, while the second estimator uses the knowledge of the statistic for the overall population to improve accuracy of the estimates for subpopulations. To evaluate the estimators, I compare the estimated number of female and elderly recipients of a government transfer program by county to the "true" number from administrative data on all recipients in New York. I find that even the simple estimators substantially improve survey error. Incorporating the statistic of interest for the overall population yields particularly large error reductions and can reduce non-sampling error.
Models with high dimensional sets of fixed effects are frequently used to examine, among others, linked employer-employee data, student outcomes and migration. Estimating these models is computationally difficult, so simplifying assumptions that are likely to cause bias are often invoked to make computation feasible and specification tests are rarely conducted. I present a simple method to estimate large two-way fixed effects (TWFE) and worker-firm match effect models without additional assumptions. It computes the exact OLS solution including estimates of the fixed effects and makes testing feasible even with multi-way clustered errors. An application using German linked employer-employee data illustrates the advantages: The data reject the assumptions of simpler estimators and omitting match effects biases estimates including the returns to experience and the gender wage gap. Specification tests detect both problems. The results suggest that firm fixed effects, not match effects, are the main channel through which job transitions drive wage dynamics, which underlines the importance of firm heterogeneity for labor market dynamics.
Marginal Voters and Their Racial Prejudice (with Suzanne K. Barth and Kyung H. Park)
This paper studies whether marginal voters are more or less prejudiced in democratic elections in comparison with regular voters. We focus on low-level statewide elections in which marginal voters are plausibly ill-informed in order to mitigate concerns that the estimates reflect unobserved candidate-specific attributes that correlate with race. Both descriptive evidence and falsification tests reaffirm the research design. We find considerable evidence that marginal voters engage in more racial discrimination than regular voters. There is no evidence of bias against female candidates.
Distributional Impact Analysis Toolkit (with Guadalupe Bedoya, Luca Bitarello and Jonathan Davis)
Program evaluations often focus on average treatment effects. However, average treatment effects miss important aspects of policy evaluation, such as the impact on inequality and whether treatment harms some individuals. A growing literature develops methods to evaluate such issues by examining the distributional impacts of programs and policies. This toolkit reviews methods to do so, focusing on their application to randomized control trials. The paper emphasizes two strands of the literature: estimation of impacts on outcome distributions and estimation of the distribution of treatment impacts. The article then discusses extensions to conditional treatment effect heterogeneity, that is, to analyses of how treatment impacts vary with observed characteristics. The paper offers advice on inference, testing, and power calculations, which are important when implementing distributional analyses in practice. Finally, the paper illustrates select methods using data from two randomized evaluations.
Missing data is a frequent problem in economics, either because some variables are missing from a data set or values are missing for some observations. Researchers usually either omit the affected variables and observations or impute them. While the consequences of the former are well understood, the imputation and missing data literature has focused on the conditions under which they lead to unbiased estimates. These conditions often do not hold, but there is little evidence on the circumstances under which missing data methods improve estimates if the conditions for unbiased estimates are violated. I first examine these conditions by discussing the circumstances under which missing data methods can be beneficial and common sources of bias. I then discuss advantages and problems of common missing data methods. Two important problems are that most methods work well for some models, but poorly for others and that researchers often do not have enough information to use imputed observations in common data sets appropriately. To address these problems, I develop a method based on the conditional density that works well for a wide range of models and allows producers of the data to incorporate private information and expertise, but still allows users of the data to adjust the imputations to their application and use them appropriately. Applications to some common problems show that the conditional density method works well in practice.
Work in ProgressPlease send me an email (firstname.lastname@example.org) for further information or a current draft.
Errors in Survey Reporting and Imputation and their Effects on Estimates of Food Stamp Program Participation (with Bruce D. Meyer and Robert M. Goerge)
Measuring government benefit receipt in household surveys is important to assess the economic circumstances of disadvantaged populations, program takeup, the distributional effects of government programs, and other program effects. Receipt of food stamps is especially important given the large and growing size of the program and evidence of its effects on labor supply, health and other outcomes. We use administrative data on food stamp participation in two states matched to American Community Survey (ACS), Current Population Survey (CPS), and Survey of Income and Program Participation (SIPP) household data. We find that 23 percent of true recipient households do not report receipt in the SIPP, 35 percent in the ACS, and fully 50 percent do not report receipt in the CPS. Both false negative and false positive reports vary with individual characteristics, leading to complicated biases in food stamp analyses. Our results are also informative about the reasons for misreporting and the success of different survey methods. We then directly examine biases in research finding, in particular the determinants of program receipt using our combined administrative and survey data. Our results differ from conventional estimates in showing higher participation by single parents, non-whites, middle-income households, and other groups. We directly examine one source of potential error, Census Bureau imputations, finding that excluding the imputed observations leads to worse ACS estimates, but has little effect on the CPS and SIPP estimates.
What Leads to Measurement Error? Evidence from Reports of Program Participation in Three Surveys (with Pablo Celhay and Bruce D. Meyer)
Measurement error is often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult for data producers to reduce the extent of errors and for data users to assess the validity of analyses using the data. We study the causes of survey error using high quality administrative data on government transfers linked to three major U.S. surveys. The differences between survey and administrative records show that up to six out of ten cash welfare recipients are missed by the surveys. We find that misreporting by respondents, survey design features, and imputation of missing data induce substantial error. Our results for non-imputed respondents confirm several theories of misreporting, e.g. that errors are related to event recall, forward and backward telescoping, salience of receipt, respondent's degree of cooperation, and the stigma of reporting participation in welfare programs. Our results provide guidance on the conditions under which survey data are likely to be accurate and suggest different ways to reduce survey errors.
Stigma in Welfare Programs (with Pablo Celhay and Bruce D. Meyer)
Stigma is often cited as a main reason why eligible individuals do not participate in welfare programs. It is also advanced as a force that may lead only the most deserving to apply for aid. Yet, there is little convincing evidence for a stigma of program participation. We study how misreporting of SNAP and Public Assistance receipt in household surveys is related to stigma. We posit that program participation should be less stigmatizing when participation in social programs is a common behavior within a person's social network. We empirically test whether misreporting participation is negatively associated with true participation in the same program by neighbors. Our results suggest that welfare participation is stigmatizing, i.e. households that live in areas of high participation are more likely to truthfully report participation. This conclusion is supported by instrumental variable estimates where we take advantage of geocoded data to instrument local participation with distance to the nearest county Department of Social Services office. The evidence for stigma is strong in the case of in-person or phone interviews where we expect stigma to be more pronounced, and weak for mail-in responses. We find no support for the alternative hypothesis that respondents are more cooperative in areas of high participation.
An Empirical Total Survey Error Decomposition Using Data Combination (with Pablo Celhay and Bruce D. Meyer)
Survey error is known to be pervasive and bias even simple, but important estimates of means, rates, and totals such as rates of employment and poverty. To summarize and analyze the extent, sources, and consequences of survey error, we define empirical counterparts of key components of the Total Survey Error framework that can be estimated using data combination and linkage. Specifically, we estimate average total survey error and decompose this average into three sources of error: survey coverage , item non-response and measurement error. For error in dollars received from two large government transfer programs, we use administrative records on the universe of program payments in New York State linked to three major household surveys to estimate the error components we define. We find that total survey error is large and varies in its size and composition, but measurement error always constitutes the dominating error source. Our application shows that data combination and linkage make it possible to routinely measure total survey error and its components. This could help survey producers to cost-effectively reduce survey error and survey users to mitigate the consequences of survey errors or gauge the reliability of their conclusions.
Misreporting of Government Transfers: How Important are Survey Design and Geography? (with Bruce D. Meyer)
Recent studies linking major US household surveys to administrative records have revealed alarming rates of misreporting of transfer program receipt. These studies use data from one or two states linked to surveys with similar design. We examine to what extent their findings are informative about errors in a survey with a narrow focus and across the entire U.S. The FoodAPS survey provides us with a unique opportunity to study these questions, because its design differs substantially from previously validated data and it is linked to a nationally representative sample of administrative SNAP records. First, we study how reporting errors differ from those found in previously validated surveys and to what extent these differences are likely due to survey design. We document that error rates, particularly failure to report true receipt, are much lower in FoodAPS. As in other surveys, we find key demographic and economic characteristics such as income and household composition predict misreporting. Thus, the resulting measurement error is neither classical nor random conditional on true receipt. We will examine how much of the difference in error rates can be explained by survey design features, such as differences in recall periods, salience, and interview conditions. Preliminary results suggest that differences in recall periods are a plausible reason for a large share of the differences in error rates. Second, we examine variation in misreporting between U.S. states. Understanding geographic variation is important to assess whether the results from prior studies using only one state are likely to generalize to U.S. data. Geographic heterogeneity also determines whether we can correct estimates of official statistics, such as program take-up or poverty rates, for the entire U.S. based on validation data from a few states.
Errors in Reporting and Imputation of Government Benefits and Their Implications (with Pablo Celhay and Bruce D. Meyer)
Misclassification-robust Semiparametric Estimation of Binary Choice Models (with Pavel Cizek)
- Estimating Effects of School Quality using Multiple Proxies. With Pedro Bernal and Javaeria A. Qureshi. Labour Economics (2016).
- Misclassification in Binary Choice Models. With Bruce D. Meyer. Journal of Econometrics (2017).Programs