Handling Missing Values in A Dataset

Abstract

In predictive data mining, all issues pertaining to missing values in a dataset must be resolved before modeling can commence. Careful analysis and planning is required in the process of filling missing values to avoid introduction of artificial patterns, which may erroneously be discovered during modeling. In this work, four techniques: “Replacement with mean”, “Replacement with nearest neighbor”, “Replacement with regression analysis” and “Discard columns with missing values” were used to fill missing values in “offences against persons” 1980 - 2008 dataset obtained from the Nigeria Police Force. The dataset was divided into two; test set 1980 - 2003 (containing the filled in values) and holdout sample for validation 2004 – 2008. The test data was used to predict the holdout sample data – the objective being to determine which technique predicted the best match to actual data. The “Discard columns with missing values” technique achieved a correlation coefficient of -0.45 in one run. This work has demonstrated that missing values in a dataset can be handled and need not abort the data mining process.

Keywords: data mining; missing values; data preparation; replacement; correlation