Handling Missing Values in A Dataset

Main Article Content

Emeka, Chinedu E
Okonkwo Obi R


In predictive data mining, all issues pertaining to missing values in a dataset must be resolved before modeling can commence. Careful analysis and planning is required in the process of filling missing values to avoid introduction of artificial patterns, which may erroneously be discovered during modeling. In this work, four techniques: “Replacement with meanâ€, “Replacement with nearest neighborâ€, “Replacement with regression analysis†and “Discard columns with missing values†were used to fill missing values in “offences against persons†1980 - 2008 dataset obtained from the Nigeria Police Force. The dataset was divided into two; test set 1980 - 2003 (containing the filled in values) and holdout sample for validation 2004 – 2008. The test data was used to predict the holdout sample data – the objective being to determine which technique predicted the best match to actual data. The “Discard columns with missing values†technique achieved a correlation coefficient of -0.45 in one run. This work has demonstrated that missing values in a dataset can be handled and need not abort the data mining process.

Keywords: data mining; missing values; data preparation; replacement; correlation


Download data is not yet available.

Article Details