數據分析-數據預處理

處理重復值

duplicated( )查找重復值

import pandas as pd
a=pd.DataFrame(data=[['A',19],['B',19],['C',20],['A',19],['C',20]],columns=['name','age'])
print(a)
print('--------------------------')
a=a.duplicated()
print(a)

只判斷全局不判斷每個

any()

import pandas as pd
a=pd.DataFrame(data=[['A',19],['B',19],['C',20],['A',19],['C',20]],columns=['name','age'])
print(a)
print('--------------------------')
a=any(a.duplicated())
print(a)

drop_duplicates( )刪除重復值

參數inplace 是否在原數據上修改

import pandas as pd
a=pd.DataFrame(data=[['A',19],['B',19],['C',20],['A',19],['C',20]],columns=['name','age'])
print(a)
print('--------------------------')
b=a.drop_duplicates(inplace=False)
a.drop_duplicates(inplace=True)
print(a)
print('--------------------------')
print(b)

處理缺失值

NaN表示缺失值

import pandas as pd
a=pd.read_csv(r'text.csv')
print(a)

isnull( )判斷所有位置元素是否缺失

import pandas as pd
a=pd.read_csv(r'text.csv')
print(a.isnull())

any( )判斷行列元素是否缺失

import pandas as pd
a=pd.read_csv(r'text.csv')
print(a.isnull().any())
print(a.isnull().any(axis=1))

del( )dropna( )刪除

import pandas as pd
a=pd.read_csv(r'text.csv')
del a['name']
print(a)

import pandas as pd
a=pd.read_csv(r'text.csv')
b=a.dropna(axis=0)
print(b)
c=a.dropna(axis=1)
print(c)

del( )刪除指定列，dropna( )刪除含有缺失值的列（行）

fillna( )缺失值填補

import pandas as pda=pd.read_csv(r'text.csv')a=a.fillna('wu')print(a)

根據上（下）數據填充

pad / ffill: 按照上一行進行填充
backfill / bfill: 按照下一行進行填充

import pandas as pda=pd.read_csv(r'text.csv')print(a)print('---------------------')b=a.fillna(method='pad')print(b)print('---------------------')c=a.fillna(method='bfill')print(c)

數值型數據填充

平均值mean()

每列的平均值填充

import pandas as pda=pd.read_csv(r'text.csv')
print(a)
print('---------------------')
a=a.fillna(a.mean())
print(a)

中位數median( )

import pandas as pd
a=pd.read_csv(r'text.csv')
print(a)print('---------------------')
a=a.fillna(a.median( ))
print(a)

字符型數據填充

眾數mode( )

import pandas as pd
a=pd.read_csv(r'text.csv')
print(a)
print('---------------------')
for i in a.columns:a[i] = a[i].fillna(a[i].mode()[0])print(a)

數據變換

map( )數據轉換

import pandas as pd
data={'sex':[1,0,1,1,0]}
a=pd.DataFrame(data)
a['sex-T']=a['sex'].map({1:'男',0:'女'})
print(a)

啞變量

import pandas as pd
data={'sex':['男','女','男','女','保密']}
a=pd.DataFrame(data)
a=pd.get_dummies(a)
print(a)