Explore the process of how to detect and remove outliers in data using Python for machine learning tasks. Gain insights into outlier detection techniques, such as statistical methods and visualization tools. Learn how to handle outliers by applying robust statistical measures and preprocessing techniques. Enhance your understanding of outlier impact on machine learning models and improve the accuracy and reliability of your predictions.
Outlier handling depends on the specific context and goals of your analysis, and there is no one-size-fits-all solution. It's important to note that the decision to remove outliers should be made judiciously and should be based on a thorough understanding of the data and the specific goals of your analysis. Removing outliers can alter the distribution and characteristics of your data, so it's crucial to consider the potential implications and document the choices made during the outlier detection and removal process.
You can watch the video-based tutorial with step by step explanation down below.
Load the Dataset
We will read the data from the csv file
df = pd.read_csv('data/winequality.csv')
df.head()The code snippet reads a CSV file named 'winequality.csv' into a Pandas DataFrame object named 'df' and then displaying the first few rows of the DataFrame using the head() function
Next we will see the statistical summary of the DataFrame
df.describe()The describe() function in Pandas provides a statistical summary of the DataFrame, including various descriptive statistics such as count, mean, standard deviation, minimum value, 25th percentile (Q1), median (50th percentile or Q2), 75th percentile (Q3), and maximum value for each numerical column in the DataFrame
We will use the residual sugar column to detect and remove the outliers
Visualize the Data
Next we will plot the data
sns.distplot(df['residual sugar'])This will generate a distribution plot that displays the distribution of values in the 'residual_sugar' column. The plot will include a histogram to visualize the frequency of different values and a smooth curve representing the kernel density estimate
There is a outlier as the plot is completely right skewed
Next we will use boxplot to see the outliers clearly
# to see outliers clearly
sns.boxplot(df['residual sugar'])The code snippet you provided makes use of the sns.boxplot() function from the Seaborn library to create a box plot for the 'residual sugar' variable in the DataFrame df
The box represents the interquartile range (IQR), with the line inside representing the median. The whiskers extend to the minimum and maximum values within 1.5 times the IQR from the first and third quartiles. Any points outside of the whiskers are considered potential outliers
Methods to remove Outliers
There are different methods using which we can remove outliers. Let us see few of them
Z-Score Method
The z-score method is a statistical technique used to detect outliers by measuring how many standard deviations a data point is away from the mean. A z-score tells you how relatively far a data point is from the mean in terms of standard deviations
First we will get the upper and lower limits
# find the limits
upper_limit = df['residual sugar'].mean() + 3*df['residual sugar'].std()
lower_limit = df['residual sugar'].mean() - 3*df['residual sugar'].std()
print('upper limit:', upper_limit)
print('lower limit:', lower_limit)This code snippet calculates the upper limit as the mean plus three times the standard deviation (mean + 3 * std) and the lower limit as the mean minus three times the standard deviation (mean - 3 * std)
These limits define a range beyond which data points are considered outliers based on the z-score method
This is the possible upper and lower limit that we can consider
Next let us find outliers using the limits
# find the outliers
df.loc[(df['residual sugar'] > upper_limit) | (df['residual sugar'] < lower_limit)]The code snippet uses the upper and lower limits calculated earlier to identify outliers in the 'residual sugar' column of the DataFrame df. It uses boolean indexing to filter the DataFrame and select rows where the 'residual sugar' values are outside the calculated limits
The .loc[] method is used to access the rows in df that meet the specified condition
The condition (df['residual sugar'] > upper_limit) | (df['residual sugar'] < lower_limit) checks whether the 'residual sugar' values are greater than the upper limit or less than the lower limit, indicating outliers
The resulting DataFrame outliers will contain only the rows where outliers are present in the 'residual sugar' column
Next we will trim the outliers. Trimming is a data transformation technique where outliers are removed or "trimmed" from the dataset, rather than replacing or imputing their values. Trimming involves setting a threshold or cutoff value, and any data points exceeding this threshold are removed from the dataset
# trimming - delete the outlier data
new_df = df.loc[(df['residual sugar'] <= upper_limit) & (df['residual sugar'] >= lower_limit)]
print('before removing outliers:', len(df))
print('after removing outliers:',len(new_df))
print('outliers:', len(df)-len(new_df))The code snippet performs trimming by removing the outlier data from the DataFrame df based on the upper and lower limits calculated earlier
It creates a new DataFrame named new_df that contains only the rows with 'residual sugar' values within the calculated limits
The code calculates the length of df before removing outliers using len(df)
It then calculates the length of new_df after removing outliers using len(new_df)
Finally, it calculates the number of outliers removed by subtracting the length of new_df from the length of df
By printing these values, you can see the number of rows in df before and after removing outliers, as well as the count of outliers that were removed
Next let us plot the data after trimming outliers
sns.boxplot(new_df['residual sugar'])
Next we will perform capping. Capping, also known as Winsorization, is a technique used to handle outliers by setting a threshold and capping or truncating extreme values to a specified percentile. Capping involves replacing outlier values with less extreme values, thus reducing the impact of outliers on the dataset without entirely removing them
# capping - change the outlier values to upper (or) lower limit values
new_df = df.copy()
new_df.loc[(new_df['residual sugar']>=upper_limit), 'residual sugar'] = upper_limit
new_df.loc[(new_df['residual sugar']<=lower_limit), 'residual sugar'] = lower_limitYou are performing capping by changing the outlier values in the 'residual sugar' column of the DataFrame df to the upper or lower limit values
The DataFrame new_df is created as a copy of the original DataFrame df, and the outlier values are replaced accordingly
new_df is created as a copy of df. The upper_limit and lower_limit values are calculated using the mean and standard deviation of the 'residual sugar' column
The .loc[] method is then used to identify the rows where the 'residual sugar' values exceed the upper limit or fall below the lower limit
The corresponding outlier values are replaced with the upper or lower limit values using the assignment statement
By performing capping in this way, the outlier values in the 'residual sugar' column are replaced with the specified upper or lower limit values, effectively bringing them within the desired range
Next let us plot the data after performing capping
sns.boxplot(new_df['residual sugar'])
Here we have not deleted any of the data rather we have capped it . We can check by printing the length of data
len(new_df)6497
We can see that length of new dataframe is 6497 which is same as the old dataframe
Inter Quartile Range Method
The Interquartile Range (IQR) method is another statistical technique used to detect and handle outliers in a dataset. The IQR represents the range between the first quartile (Q1) and the third quartile (Q3) of a dataset
First let us calculate the first quartile (Q1), third quartile (Q3), and interquartile range (IQR) of the 'residual sugar' column in the DataFrame df
q1 = df['residual sugar'].quantile(0.25)
q3 = df['residual sugar'].quantile(0.75)
iqr = q3-q1In this code, q1 is calculated as the value at the 25th percentile (first quartile) of the 'residual sugar' column using the .quantile() function with a parameter of 0.25
Similarly, q3 is calculated as the value at the 75th percentile (third quartile)
Finally, iqr is computed as the difference between q3 and q1, representing the interquartile range
By calculating the Q1, Q3, and IQR, you obtain important descriptive statistics that can help in understanding the spread and distribution of the 'residual sugar' data.
These values are commonly used in the Interquartile Range (IQR) method for outlier detection and other data analysis techniques
q1, q3, iqr(1.8, 8.1, 6.3)
These are the values of Q1, Q3, and IQR for the 'residual sugar' data in your DataFrame
Next let us calculate the upper and lower limit using the Interquartile Range (IQR) method
upper_limit = q3 + (1.5 * iqr)
lower_limit = q1 - (1.5 * iqr)
lower_limit, upper_limitupper_limit is computed by adding 1.5 times the IQR to Q3 (q3 + (1.5 * iqr)), while lower_limit is calculated by subtracting 1.5 times the IQR from Q1 (q1 - (1.5 * iqr))
By printing these values, you can obtain the specific lower and upper limits that define the range within which data points are considered non-outliers according to the IQR method
(-7.6499999999999995, 17.549999999999997)
Next let us plot the data
sns.boxplot(df['residual sugar'])
Next we will find the outliers using upper and lower limit calculated earlier
# find the outliers
df.loc[(df['residual sugar'] > upper_limit) | (df['residual sugar'] < lower_limit)]
Next let us perform trimming of the outliers
# trimming - delete the outlier data
new_df = df.loc[(df['residual sugar'] <= upper_limit) & (df['residual sugar'] >= lower_limit)]
print('before removing outliers:', len(df))
print('after removing outliers:',len(new_df))
print('outliers:', len(df)-len(new_df))The code snippet performs outlier removal using the calculated upper and lower limits based on the Interquartile Range (IQR) method. It creates a new DataFrame named new_df that includes only the rows with 'residual sugar' values within the calculated limits
By printing the lengths of df and new_df, you can see the number of rows in the DataFrame before and after removing outliers. Additionally, the difference in lengths (len(df) - len(new_df)) gives you the count of outliers that were removed
Next let us plot the data after trimming outliers
sns.boxplot(new_df['residual sugar'])
Next let us perform capping of the outliers
# capping - change the outlier values to upper (or) lower limit values
new_df = df.copy()
new_df.loc[(new_df['residual sugar']>upper_limit), 'residual sugar'] = upper_limit
new_df.loc[(new_df['residual sugar']<lower_limit), 'residual sugar'] = lower_limitThe code snippet performs capping by replacing the outlier values in the 'residual sugar' column of the DataFrame df with the upper or lower limit values. The DataFrame new_df is created as a copy of the original DataFrame df, and the outlier values are modified accordingly
Next let us plot the data after performing capping
sns.boxplot(new_df['residual sugar'])
Percentile Method
The percentile method can be used to handle outliers in a dataset. The percentile method involves setting a threshold based on percentiles and capping or truncating the outlier values accordingly
First let us calculate the upper and lower limit
upper_limit = df['residual sugar'].quantile(0.99)
lower_limit = df['residual sugar'].quantile(0.01)
print('upper limit:', upper_limit)
print('lower limit:', lower_limit)The quantile() function in pandas is used to calculate the desired percentiles of the 'residual sugar' column in the DataFrame df
upper_limit is calculated as the value at the 99th percentile (0.99) of the 'residual sugar' column, and lower_limit is calculated as the value at the 1st percentile (0.01)
By printing these values, you can obtain the specific upper and lower limits that define the range within which data points are considered non-outliers according to the percentile method. These limits are calculated based on the specified percentiles and can be used to handle outliers in the 'residual sugar' column of your dataset
Next let us plot the data
sns.boxplot(df['residual sugar'])
Next we will find the outliers using upper and lower limit calculated earlier
# find the outliers
df.loc[(df['residual sugar'] > upper_limit) | (df['residual sugar'] < lower_limit)]
Next let us perform trimming of the outliers
# trimming - delete the outlier data
new_df = df.loc[(df['residual sugar'] <= upper_limit) & (df['residual sugar'] >= lower_limit)]
print('before removing outliers:', len(df))
print('after removing outliers:',len(new_df))
print('outliers:', len(df)-len(new_df))The code snippet filters the DataFrame df based on the upper and lower limits calculated using the percentile method. It creates a new DataFrame named new_df that includes only the rows with 'residual sugar' values within the calculated limits
Next let us plot the data after trimming outliers
sns.boxplot(new_df['residual sugar'])
Next let us perform capping of the outliers
# capping - change the outlier values to upper (or) lower limit values
new_df = df.copy()
new_df.loc[(new_df['residual sugar']>upper_limit), 'residual sugar'] = upper_limit
new_df.loc[(new_df['residual sugar']<lower_limit), 'residual sugar'] = lower_limitThe code you provided performs capping by replacing the outlier values in the 'residual sugar' column of the DataFrame df with the upper or lower limit values. The DataFrame new_df is created as a copy of the original DataFrame df, and the outlier values are modified accordingly
Next let us plot the data after performing capping
sns.boxplot(new_df['residual sugar'])
Next let us plot the distplot for both old and new dataframe
sns.distplot(df['residual sugar'])
sns.distplot(new_df['residual sugar'])
Final Thoughts
Outliers are data points that deviate significantly from the majority of the dataset, and they can have a significant impact on statistical measures and model performance
It's crucial to have a good understanding of the data and the domain in which it is collected. Outliers may arise due to various reasons, such as measurement errors, data entry mistakes, or genuinely rare events. Understanding the nature of the data helps in making informed decisions about whether to remove or retain outliers
There are several methods available for detecting outliers, including statistical techniques like z-score, modified z-score, and box plots. These methods help identify observations that fall outside a certain threshold. Additionally, domain-specific knowledge and visual exploration of the data can also aid in outlier detection.
Outliers can significantly influence statistical measures such as mean, variance, and correlation coefficients. Therefore, it is essential to assess the impact of outliers on the analysis and decide whether their presence distorts the results. Sometimes, outliers may contain valuable information, and removing them can lead to biased or inaccurate conclusions
Once outliers are detected, the next step is to decide how to handle them. There are several approaches such as Remove outliers (trimming), Transform data(capping), and Treat separately
While outlier detection and removal can be valuable, it is essential to exercise caution and be aware of potential pitfalls such as Overzealous removal, Sample size and statistical power and Outlier definition
In summary, detecting and removing outliers should be approached with careful consideration of the data, domain knowledge, and the goals of the analysis. It is a crucial step in data preprocessing, but it requires judgment and an understanding of the potential impact on subsequent analyses or models
In this article we have explored how we can detect and remove outliers using Z-score method , Inter Quartile Range method and Percentile method and we have also seen how we can perform trimming and capping in each of this methods.
Get the project notebook from here
Thanks for reading the article!!!
Check out more project videos from the YouTube channel Hackers Realm