In real-world data, there are some cases where a specific element is missing for a variety of reasons, including corrupted data, information loading issues, or insufficient extraction. One of the biggest problems analysts face is handling the missing values because choosing the appropriate solution results in reliable data models. Let’s examine various approaches to imputing the missing values.
Obtaining a Data Analyst Course is vital for upskilling and staying current in the workplace.
Removing Rows
This approach is frequently used to handle null values. Here, if a row contains a null value for a specific feature or if a column contains more than 70–75% missing values, we either delete the row or the column. Only when there are sufficient samples in the data set is this method recommended. After the data has been deleted, one must ensure that no bias has been added. Removing the data will result in information loss, which will prevent the prediction of the output from producing the desired results.
Pros:
- Completely removing all missing values from the data yields a robust and highly accurate model.
- It is preferable to delete a specific row or column without specific information because it does not carry a lot of weight.
Cons:
- Data and information loss
- Works poorly if the dataset’s overall missing value percentage is high (say, 30%).
Substituting Mean, Median, or Mode
This tactic can be used on a feature that contains numerical information, such as a person’s age or the cost of a ticket. In order to fill in the missing values, we can compute the feature’s mean, median, or mode. This is an estimate that might introduce variation into the data set. But this approach, which produces better results than removing rows and columns, can negate the loss of the data. The three approximations mentioned above can be substituted as a statistical method of handling missing values. This technique is also known as training data leakage. A different method is to approximate it using the deviation of nearby values. If the data is linear, this will function better.
Pros:
- This method is preferable when the amount of data is small.
- It can stop data loss, which would otherwise cause the rows and columns to be removed.
Cons:
- The approximations are added, introducing bias and variance.
- Compared to other multiple-imputations methods, it performs poorly.
