Errors in Reporting and Imputation of Government Benefits and Their Implications (with Pablo Celhay and Bruce D. Meyer)
We document the extent, nature, and consequences of survey errors for receipt of cash welfare and SNAP in three major U.S. household surveys linked to administrative program records. Our results confirm high rates of misreporting of program receipt, particularly failure to report receipt. The surveys inaccurately capture patterns of participation in multiple programs, even though there is little evidence of program confusion. Error rates are higher among imputed observations, which also account for a large share of false positive errors. Many household characteristics have significant effects on errors in reporting receipt, both false positives and false negatives. We find large differences in survey errors by race, ethnicity, income and other household characteristics. We provide evidence on the consequences of these errors for models of program receipt. Estimated effects of income and race are noticeably biased. We then examine error due to item non-response and imputation, as well as whether imputation improves estimates. Item non-respondents have higher receipt rates than the population, even conditional on many covariates. The assumptions for consistent estimates in multivariate models fail both when excluding item non-respondents and when using the imputed values. In binary choice models of program receipt, estimates from the linked data favor excluding item non-respondents rather than using their imputed values. The biases in each case are well predicted by the error patterns we document, so such analyses can help researchers make more informed decisions on the use of imputed values.
Assumptions that a continuous and a discrete variable are independent conditional on covariates are ubiquitous in, among others, program evaluation, discrete choice models, missing data problems and studies of teacher or peer effects. Instead of testing conditional independence, current studies at best compare means, which only tests correlation. I propose an assumption-free, non-parametric Kolmogorov test that is simple to implement and has power against all alternatives at distance 1/√N that differ at any point in the joint support of the distributions of the covariates. Other non- and semi-parametric tests can easily be based on the restriction I test. The test can be used for hypotheses that depend on estimated parameters such as a location shift by (conditional) treatment effects or a regression adjustment. Inference can be conducted by simulation.
Work in ProgressPlease send me an email (email@example.com) for further information or a current draft.
What Leads to Measurement Error? Evidence from Reports of Program Participation in Three Surveys (with Pablo Celhay and Bruce D. Meyer)
Measurement error is often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult for data producers to reduce the extent of errors and for data users to assess the validity of analyses using the data. We study the causes of survey error using high quality administrative data on government transfers linked to three major U.S. surveys. The differences between survey and administrative records show that up to six out of ten cash welfare recipients are missed by the surveys. We find that misreporting by respondents, survey design features, and imputation of missing data induce substantial error. Our results for non-imputed respondents confirm several theories of misreporting, e.g. that errors are related to event recall, forward and backward telescoping, salience of receipt, respondent's degree of cooperation, and the stigma of reporting participation in welfare programs. Our results provide guidance on the conditions under which survey data are likely to be accurate and suggest different ways to reduce survey errors.
Stigma in Welfare Programs (with Pablo Celhay and Bruce D. Meyer)
Stigma is often cited as a main reason why eligible individuals do not participate in welfare programs. It is also advanced as a force that may lead only the most deserving to apply for aid. Yet, there is little convincing evidence for a stigma of program participation. We study how misreporting of SNAP and Public Assistance receipt in household surveys is related to stigma. We posit that program participation should be less stigmatizing when participation in social programs is a common behavior within a person's social network. We empirically test whether misreporting participation is negatively associated with true participation in the same program by neighbors. Our results suggest that welfare participation is stigmatizing, i.e. households that live in areas of high participation are more likely to truthfully report participation. This conclusion is supported by instrumental variable estimates where we take advantage of geocoded data to instrument local participation with distance to the nearest county Department of Social Services office. The evidence for stigma is strong in the case of in-person or phone interviews where we expect stigma to be more pronounced, and weak for mail-in responses. We find no support for the alternative hypothesis that respondents are more cooperative in areas of high participation.
How Does Potential Unemployment Insurance Benefit Duration Affect Re-employment Timing and Wages? (with R. Felder and H. Frings)
How potential benefit duration (PBD) of unemployment insurance (UI) affects wages and matching is crucial to evaluate PBD extensions as a policy tool and to understand the causal relation between unemployment, job search and wages. Recent studies provide evidence from quasi-experiments, but disagree even on the sign of (local average) wage effects. We re-visit the regression discontinuity design of Schmieder et al. (2016), but use more detailed data and a wage decomposition to re-analyze the effects of PBD in a framework that allows for unrestricted heterogeneity of both duration and wage effects. Our (preliminary) results indicate treatment effect heterogeneity that casts doubt on simple mechanisms and complicates learning from (local) average effects. Specifically, we first show that duration effects are heterogeneous, which confirms that treatment changes dynamic selection. Our results so far suggest that PBD almost exclusively prolongs unemployment spells ending close to exhaustion points. We examine pre-determined wage components to (partly) separate dynamic selection from dynamic treatment effects. Dynamic selection is potentially non-monotonic and may create spurious treatment effects in our sample. We find that PBD affects wages only through the firm fixed effect, which adds to evidence on the importance of firms. It suggests that wage losses are likely due to firm attributes (such as bargaining power) rather than individual attributes (such as productivity). Purging (parts of) dynamic selection from re-employment wages shows that the steep wage decline is mainly driven by the firm fixed effect, but also by time-varying unobservables. The marked exhaustion effects have at most a small impact on the effect of PBD on wages. Wage effects appear to accumulate through employment at lower-paying firms at short unemployment durations. Our current results are work in progress, but demonstrate that heterogeneity in both duration and wage effects alters the interpretation of common analyses and thereby potentially reconciles diverging findings. We develop tools to make progress on important questions in the presence of essential heterogeneity.
- Estimating Effects of School Quality using Multiple Proxies. With Pedro Bernal and Javaeria A. Qureshi. Labour Economics (2016).
- Misclassification in Binary Choice Models. With Bruce D. Meyer. Journal of Econometrics (2017).Programs
- Two Simple Methods to Improve Official Statistics for Small Subpopulations. Survey Research Methods (2018). , ProgramsAdd. Tables
- Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net. With Bruce D. Meyer. AEJ: Applied Economics (2019). Web Appendix Tables
- Correcting for Misreporting of Government Benefits. AEJ: Economic Policy (2019)., Parameters Web Appendix
- Misreporting of Government Transfers: How Important are Survey Design and Geography? With Bruce D. Meyer. Southern Economic Journal (2019).
- Creating Improved Survey Data Products Using Linked Administrative-Survey Data. With Michael Davern and Bruce D. Meyer. Journal of Survey Statistics and Methodology (2019).
- Voter Response to Hispanic Sounding Names: Evidence from Down Ballot Statewide Elections. With Suzanne K. Barth and Kyung H. Park. Quarterly Journal of Political Science (2019).AppendixWorking Paper Version,
- A Simple Method to Estimate Large Fixed Effects Models Applied to Wage Determinants. Labour Economics (2019).Longer IZA Working PaperPrograms,Appendix,
- Combining Administrative and Survey Data to Improve Income Measurement. With Bruce D. Meyer in Administrative Records for Survey Methodology, ed. A.Y. Chun, M. Larson, J. Reiter and G. Durrant, Wiley: NY. Forthcoming. IZA Working Paper
- An Empirical Total Survey Error Decomposition Using Data Combination. With Bruce D. Meyer. Journal of Econometrics (accepted).
- Errors in Survey Reporting and Imputation and their Effects on Estimates of Food Stamp Program Participation. With Bruce D. Meyer and Robert M. Goerge. Journal of Human Resources (accepted).
Not in Progress
Distributional Impact Analysis Toolkit (with Guadalupe Bedoya, Luca Bitarello and Jonathan Davis)
Program evaluations often focus on average treatment effects. However, average treatment effects miss important aspects of policy evaluation, such as the impact on inequality and whether treatment harms some individuals. A growing literature develops methods to evaluate such issues by examining the distributional impacts of programs and policies. This toolkit reviews methods to do so, focusing on their application to randomized control trials. The paper emphasizes two strands of the literature: estimation of impacts on outcome distributions and estimation of the distribution of treatment impacts. The article then discusses extensions to conditional treatment effect heterogeneity, that is, to analyses of how treatment impacts vary with observed characteristics. The paper offers advice on inference, testing, and power calculations, which are important when implementing distributional analyses in practice. Finally, the paper illustrates select methods using data from two randomized evaluations.
Missing data is a frequent problem in economics, either because some variables are missing from a data set or values are missing for some observations. Researchers usually either omit the affected variables and observations or impute them. While the consequences of the former are well understood, the imputation and missing data literature has focused on the conditions under which they lead to unbiased estimates. These conditions often do not hold, but there is little evidence on the circumstances under which missing data methods improve estimates if the conditions for unbiased estimates are violated. I first examine these conditions by discussing the circumstances under which missing data methods can be beneficial and common sources of bias. I then discuss advantages and problems of common missing data methods. Two important problems are that most methods work well for some models, but poorly for others and that researchers often do not have enough information to use imputed observations in common data sets appropriately. To address these problems, I develop a method based on the conditional density that works well for a wide range of models and allows producers of the data to incorporate private information and expertise, but still allows users of the data to adjust the imputations to their application and use them appropriately. Applications to some common problems show that the conditional density method works well in practice.