How can one idiomatically run a function like get_dummies
, which expects a single column and returns several, on multiple DataFrame columns?
5 Answers
With pandas 0.19, you can do that in a single line :
pd.get_dummies(data=df, columns=['A', 'B'])
Columns
specifies where to do the One Hot Encoding.
>>> df
A B C
0 a c 1
1 b c 2
2 a b 3
>>> pd.get_dummies(data=df, columns=['A', 'B'])
C A_a A_b B_b B_c
0 1 1.0 0.0 0.0 1.0
1 2 0.0 1.0 0.0 1.0
2 3 1.0 0.0 1.0 0.0
- 9,292
- 3
- 27
- 34
Since pandas version 0.15.0, pd.get_dummies
can handle a DataFrame directly (before that, it could only handle a single Series, and see below for the workaround):
In [1]: df = DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'],
...: 'C': [1, 2, 3]})
In [2]: df
Out[2]:
A B C
0 a c 1
1 b c 2
2 a b 3
In [3]: pd.get_dummies(df)
Out[3]:
C A_a A_b B_b B_c
0 1 1 0 0 1
1 2 0 1 0 1
2 3 1 0 1 0
Workaround for pandas < 0.15.0
You can do it for each column seperate and then concat the results:
In [111]: df
Out[111]:
A B
0 a x
1 a y
2 b z
3 b x
4 c x
5 a y
6 b y
7 c z
In [112]: pd.concat([pd.get_dummies(df[col]) for col in df], axis=1, keys=df.columns)
Out[112]:
A B
a b c x y z
0 1 0 0 1 0 0
1 1 0 0 0 1 0
2 0 1 0 0 0 1
3 0 1 0 1 0 0
4 0 0 1 1 0 0
5 1 0 0 0 1 0
6 0 1 0 0 1 0
7 0 0 1 0 0 1
If you don't want the multi-index column, then remove the keys=..
from the concat function call.
- 106,362
- 32
- 216
- 184
-
Nice, I like the multi-index column. – chrisb Jun 08 '14 at 19:33
Somebody may have something more clever, but here are two approaches. Assuming you have a dataframe named df
with columns 'Name' and 'Year' you want dummies for.
First, simply iterating over the columns isn't too bad:
In [93]: for column in ['Name', 'Year']:
...: dummies = pd.get_dummies(df[column])
...: df[dummies.columns] = dummies
Another idea would be to use the patsy package, which is designed to construct data matrices from R-type formulas.
In [94]: patsy.dmatrix(' ~ C(Name) + C(Year)', df, return_type="dataframe")
- 39,034
- 8
- 55
- 56
Unless I don't understand the question, it is supported natively in get_dummies by passing the columns argument.
- 4,762
- 6
- 32
- 51
-
You don't need to specify the `columns` argument. By default, it will encode all categorical-like columns (string/categorical) – joris Mar 12 '16 at 20:52
-
That is true, I assumed @Emre wanted to control what columns got dummied. – sapo_cosmico Mar 13 '16 at 16:51
The simple trick I am currently using is a for-loop.
First separate categorical data from Data Frame by using select_dtypes(include="object")
,
then by using for loop apply get_dummies
to each column iteratively
as I have shown in code below:
train_cate=train_data.select_dtypes(include="object")
test_cate=test_data.select_dtypes(include="object")
# vectorize catagorical data
for col in train_cate:
cate1=pd.get_dummies(train_cate[col])
train_cate[cate1.columns]=cate1
cate2=pd.get_dummies(test_cate[col])
test_cate[cate2.columns]=cate2
- 5,195
- 62
- 50
- 120
- 9
- 2