In this article we will learn how to remove the rows with special characters i.e; if a row contains any value which contains special characters like @, %, &, $, #, +, -, *, /, etc. then drop such row and modify the data. To drop such types of rows, first, we have to search rows having special characters per column and then drop. To search we use regular expression either [@#&$%+-/*] or [^0-9a-zA-Z]. Let’s discuss the whole procedure with some examples :
Example 1:
This example consists of some parts with code and the dataframe used can be download by clicking data1.csv or shown below.
Python3
# importing packageimport pandas as pd# load datasetdf = pd.read_csv("data1.csv")# view datasetprint(df) |
Output:
Select rows with columns having special characters value
Python3
# select the rows # if Name column# has special charactersprint(df[df.Name.str.contains(r'[@#&$%+-/*]')]) |
Output:
Python3
# select the rows # if Grade column# has special charactersprint(df[df.Grade.str.contains(r'[^0-9a-zA-Z]')]) |
Output:
Merging of selected rows
Python3
# merge the selected rows# by using orprint(df[df.Name.str.contains(r'[^0-9a-zA-Z]') | df.Grade.str.contains(r'[@#&$%+-/*]')]) |
Output:
Remove the merged selected rows
Python3
# drop the merged selected rowsprint(df.drop(df[df.Name.str.contains(r'[^0-9a-zA-Z]') | df.Grade.str.contains(r'[^0-9a-zA-Z]')].index)) |
Output:
Example 2: This example uses a dataframe which can be download by clicking data2.csv or shown below :
Python3
# importing packageimport pandas as pd# load datasetdf = pd.read_csv("data2.csv")# view datasetprint(df)# select and then merge rows # with special charactersprint(df[df.ID.str.contains(r'[^0-9a-zA-Z]') | df.Name.str.contains(r'[^0-9a-zA-Z]') | df.Age.str.contains(r'[^0-9a-zA-Z]') | df.Country.str.contains(r'[^0-9a-zA-Z]')])# drop the rowsprint(df.drop(df[df.ID.str.contains(r'[^0-9a-zA-Z]') | df.Name.str.contains(r'[^0-9a-zA-Z]') | df.Age.str.contains(r'[^0-9a-zA-Z]') | df.Country.str.contains(r'[^0-9a-zA-Z]')].index)) |
Output :
