Recent data linkage studies show that survey misreporting is pervasive and biases common analyses. Linked data are usually convenience samples and access is restricted, making them more suitable to document than to solve the problem. I examine how incorporating information from the linked data can improve estimates and how the survey data can still be used without such information. First, I develop a method that combines public use data with an estimated conditional distribution from the linked data. It does not require access to the linked data, is simple to implement and applicable to many estimators. Using administrative SNAP records linked to survey data, I show that this method sharply improves survey estimates. I then use the linked data to evaluate corrections that mainly rely on survey data. Some corrections also yield large error reductions, particularly for univariate statistics. This makes them attractive alternatives when linked data do not exist. Finally, I examine whether estimates can be improved based on similar linked data. For SNAP, I provide evidence that extrapolation using the method developed here improves over survey data and corrections without validation data. Extrapolation reduces average error in the geographic distribution of program spending by up to 90 percent. It suggests larger program effects, such as reducing the poverty rate by almost one additional percentage point.
Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net (with Bruce D. Meyer)Supplementary Tables
We examine the consequences of underreporting of transfer programs in household survey data for several prototypical analyses of low-income populations. We focus on the Current Population Survey (CPS), the source of official poverty and inequality statistics, but provide evidence that our qualitative conclusions are likely to apply to other surveys. We link administrative data for food stamps, TANF, General Assistance, and subsidized housing from New York State to the CPS at the individual level. Program receipt in the CPS is missed for over one-third of housing assistance recipients, 40 percent of food stamp recipients and 60 percent of TANF and General Assistance recipients. Dollars of benefits are also undercounted for reporting recipients, particularly for TANF, General Assistance and housing assistance. We find that the survey data sharply understate the income of poor households, as conjectured in past work by one of the authors. Underreporting in the survey data also greatly understates the effects of anti-poverty programs and changes our understanding of program targeting, often making it seem that welfare programs are less targeted to both the very poorest and middle income households than they are. Using the combined data rather than survey data alone, the poverty reducing effect of all programs together is nearly doubled while the effect of housing assistance is tripled. We also re-examine the coverage of the safety net, specifically the share of people without work or program receipt. Using the administrative measures of program receipt rather than the survey ones often reduces the share of single mothers falling through the safety net by one-half or more.
Creating Improved Survey Data Products Using Linked Administrative-Survey Data (with Michael Davern and Bruce D. Meyer)
Recent research linking administrative to survey data has laid the groundwork for improvements in survey data products. However, the opportunities have not been fully realized yet. In this paper our main objective is to use administrative survey linked microdata to demonstrate the potential of data linkage to reduce survey error cost effectively through model based imputation methods. We use parametric models based on the linked data to create imputed values of Medicaid enrollment and food stamp (SNAP) receipt in public use data. This approach to blending data from surveys and administrative data through models is less likely to compromise confidentiality or violate the terms of the data sharing agreements among the agencies than releasing the linked microdata. We demonstrate that it can yield substantial and measurable improvements of estimate accuracy: Using the blended values to estimate state level Medicaid enrollment and SNAP receipt for sub state areas reduces Root Mean Squared Error (RMSE) of these estimates by 81 percent for Medicaid and by 93 percent for SNAP receipt compared to estimates based on the survey data alone. Given the high level of measurement error associated with these important programs in the United States, producers should consider imputation methods like the ones we describe in this paper to create improved estimates.
Misreporting of Government Transfers: How Important are Survey Design and Geography? (with Bruce D. Meyer)
Recent studies linking household surveys to administrative records have revealed high rates of misreporting of program receipt. These studies often use data from general household surveys for one or two states. Here we use the FoodAPS survey to examine whether the findings of these studies generalize to a survey with a narrow focus and across many states. First, we study how reporting errors differ from other surveys and to what extent the differences are likely due to survey design. We find a lower rate of false negatives (failures to report true receipt) in FoodAPS than in other surveys. As we also find that a shorter time since last SNAP use reduces false negatives, the shorter recall period of FoodAPS likely contributes to its lower error rate. As in other surveys, we find household characteristics such as income, employment, and minority status to predict misreporting. False negative rates vary systematically between interviewers and the differences are not entirely explained by characteristics of the households interviewers are assigned. Second, we examine variation in how administrative and reported SNAP receipt differ between U.S. states. Understanding geographic variation in survey error is important to assess whether we can correct estimates of official statistics, such as program receipt or poverty rates, for the entire U.S. based on linked data from a few states. We document systematic differences between states in unconditional error rates but do not find evidence of substantial differences between states conditional on common covariates. We show that despite the geographic heterogeneity, extrapolating across states can yield more accurate estimates than estimates based on survey reports.
- ProgramsAdditional Tables,
Many important statistics are known from official records for the entire population, but have to be estimated for subpopulations. I describe two simple data combination methods that reduce the substantial sampling error of the commonly used direct survey estimates for small subpopulations. The first estimator incorporates information from repeated cross-sections, while the second estimator uses the knowledge of the statistic for the overall population to improve accuracy of the estimates for subpopulations. To evaluate the estimators, I compare the estimated number of female and elderly recipients of a government transfer program by county to the "true" number from administrative data on all recipients in New York. I find that even the simple estimators substantially improve survey error. Incorporating the statistic of interest for the overall population yields particularly large error reductions and can reduce non-sampling error.
Models with high dimensional sets of fixed effects are frequently used to examine, among others, linked employer-employee data, student outcomes and migration. Estimating these models is computationally difficult, so simplifying assumptions that are likely to cause bias are often invoked to make computation feasible and specification tests are rarely conducted. I present a simple method to estimate large two-way fixed effects (TWFE) and worker-firm match effect models without additional assumptions. It computes the exact OLS solution including estimates of the fixed effects and makes testing feasible even with multi-way clustered errors. An application using German linked employer-employee data illustrates the advantages: The data reject the assumptions of simpler estimators and omitting match effects biases estimates including the returns to experience and the gender wage gap. The results suggest that firm fixed effects, not match effects, are the main channel through which job transitions drive wage dynamics, which underlines the importance of firm heterogeneity for labor market dynamics.
Marginal Voters and Their Racial Prejudice (with Suzanne K. Barth and Kyung H. Park)
This paper studies whether marginal voters are more or less prejudiced in democratic elections in comparison with regular voters. We focus on low-level statewide elections in which marginal voters are plausibly ill-informed in order to mitigate concerns that the estimates reflect unobserved candidate-specific attributes that correlate with race. Both descriptive evidence and falsification tests reaffirm the research design. We find considerable evidence that marginal voters engage in more racial discrimination than regular voters. There is no evidence of bias against female candidates.
Distributional Impact Analysis Toolkit (with Guadalupe Bedoya, Luca Bitarello and Jonathan Davis)
Program evaluations often focus on average treatment effects. However, average treatment effects miss important aspects of policy evaluation, such as the impact on inequality and whether treatment harms some individuals. A growing literature develops methods to evaluate such issues by examining the distributional impacts of programs and policies. This toolkit reviews methods to do so, focusing on their application to randomized control trials. The paper emphasizes two strands of the literature: estimation of impacts on outcome distributions and estimation of the distribution of treatment impacts. The article then discusses extensions to conditional treatment effect heterogeneity, that is, to analyses of how treatment impacts vary with observed characteristics. The paper offers advice on inference, testing, and power calculations, which are important when implementing distributional analyses in practice. Finally, the paper illustrates select methods using data from two randomized evaluations.
Missing data is a frequent problem in economics, either because some variables are missing from a data set or values are missing for some observations. Researchers usually either omit the affected variables and observations or impute them. While the consequences of the former are well understood, the imputation and missing data literature has focused on the conditions under which they lead to unbiased estimates. These conditions often do not hold, but there is little evidence on the circumstances under which missing data methods improve estimates if the conditions for unbiased estimates are violated. I first examine these conditions by discussing the circumstances under which missing data methods can be beneficial and common sources of bias. I then discuss advantages and problems of common missing data methods. Two important problems are that most methods work well for some models, but poorly for others and that researchers often do not have enough information to use imputed observations in common data sets appropriately. To address these problems, I develop a method based on the conditional density that works well for a wide range of models and allows producers of the data to incorporate private information and expertise, but still allows users of the data to adjust the imputations to their application and use them appropriately. Applications to some common problems show that the conditional density method works well in practice.
Work in ProgressPlease send me an email (firstname.lastname@example.org) for further information or a current draft.
Errors in Survey Reporting and Imputation and their Effects on Estimates of Food Stamp Program Participation (with Bruce D. Meyer and Robert M. Goerge)
Measuring government benefit receipt in household surveys is important to assess the economic circumstances of disadvantaged populations, program takeup, the distributional effects of government programs, and other program effects. Receipt of food stamps is especially important given the large and growing size of the program and evidence of its effects on labor supply, health and other outcomes. We use administrative data on food stamp participation in two states matched to American Community Survey (ACS), Current Population Survey (CPS), and Survey of Income and Program Participation (SIPP) household data. We find that 23 percent of true recipient households do not report receipt in the SIPP, 35 percent in the ACS, and fully 50 percent do not report receipt in the CPS. Both false negative and false positive reports vary with individual characteristics, leading to complicated biases in food stamp analyses. Our results are also informative about the reasons for misreporting and the success of different survey methods. We then directly examine biases in research finding, in particular the determinants of program receipt using our combined administrative and survey data. Our results differ from conventional estimates in showing higher participation by single parents, non-whites, middle-income households, and other groups. We directly examine one source of potential error, Census Bureau imputations, finding that excluding the imputed observations leads to worse ACS estimates, but has little effect on the CPS and SIPP estimates.
What Leads to Measurement Error? Evidence from Reports of Program Participation in Three Surveys (with Pablo Celhay and Bruce D. Meyer)
Measurement error is often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult for data producers to reduce the extent of errors and for data users to assess the validity of analyses using the data. We study the causes of survey error using high quality administrative data on government transfers linked to three major U.S. surveys. The differences between survey and administrative records show that up to six out of ten cash welfare recipients are missed by the surveys. We find that misreporting by respondents, survey design features, and imputation of missing data induce substantial error. Our results for non-imputed respondents confirm several theories of misreporting, e.g. that errors are related to event recall, forward and backward telescoping, salience of receipt, respondent's degree of cooperation, and the stigma of reporting participation in welfare programs. Our results provide guidance on the conditions under which survey data are likely to be accurate and suggest different ways to reduce survey errors.
Stigma in Welfare Programs (with Pablo Celhay and Bruce D. Meyer)
Stigma is often cited as a main reason why eligible individuals do not participate in welfare programs. It is also advanced as a force that may lead only the most deserving to apply for aid. Yet, there is little convincing evidence for a stigma of program participation. We study how misreporting of SNAP and Public Assistance receipt in household surveys is related to stigma. We posit that program participation should be less stigmatizing when participation in social programs is a common behavior within a person's social network. We empirically test whether misreporting participation is negatively associated with true participation in the same program by neighbors. Our results suggest that welfare participation is stigmatizing, i.e. households that live in areas of high participation are more likely to truthfully report participation. This conclusion is supported by instrumental variable estimates where we take advantage of geocoded data to instrument local participation with distance to the nearest county Department of Social Services office. The evidence for stigma is strong in the case of in-person or phone interviews where we expect stigma to be more pronounced, and weak for mail-in responses. We find no support for the alternative hypothesis that respondents are more cooperative in areas of high participation.
An Empirical Total Survey Error Decomposition Using Data Combination (with Pablo Celhay and Bruce D. Meyer)
Survey error is known to be pervasive and bias even simple, but important estimates of means, rates, and totals such as rates of employment and poverty. To summarize and analyze the extent, sources, and consequences of survey error, we define empirical counterparts of key components of the Total Survey Error framework that can be estimated using data combination and linkage. Specifically, we estimate average total survey error and decompose this average into three sources of error: survey coverage , item non-response and measurement error. For error in dollars received from two large government transfer programs, we use administrative records on the universe of program payments in New York State linked to three major household surveys to estimate the error components we define. We find that total survey error is large and varies in its size and composition, but measurement error always constitutes the dominating error source. Our application shows that data combination and linkage make it possible to routinely measure total survey error and its components. This could help survey producers to cost-effectively reduce survey error and survey users to mitigate the consequences of survey errors or gauge the reliability of their conclusions.
Errors in Reporting and Imputation of Government Benefits and Their Implications (with Pablo Celhay and Bruce D. Meyer)
Misclassification-robust Semiparametric Estimation of Binary Choice Models (with Pavel Cizek)
- Estimating Effects of School Quality using Multiple Proxies. With Pedro Bernal and Javaeria A. Qureshi. Labour Economics (2016).
- Misclassification in Binary Choice Models. With Bruce D. Meyer. Journal of Econometrics (2017).Programs