Handling unknown values for label encoding

Question

How can I handle unknown values for label encoding in sk-learn? The label encoder will only blow up with an exception that new labels were detected.

What I want is the encoding of categorical variables via one-hot-encoder. However, sk-learn does not support strings for that. So I used a label encoder on each column.

My problem is that in my cross-validation step of the pipeline unknown labels show up. The basic one-hot-encoder would have the option to ignore such cases. An apriori pandas.getDummies /cat.codes is not sufficient as the pipeline should work with real-life, fresh incoming data which might contain unknown labels as well.

Would it be possible to use a CountVectorizer for this purpose?

Can you catch the exception, log it (or whatever), then move on? Or just ignore them? — wwii, Oct 29 '16 at 16:54
If a predictive model is deployed as an API it is very likely it will be confronted with yet unknown labels of features. How can I deal with that in sklearn? Would you suggest to propagate the error to the API? — Georg Heiler, Oct 29 '16 at 17:03
@GeorgHeiler, Have you tried looking into `DictVectorizer` which does binary one hot encoding of string features? You would need to input a list of dictionaries however. So, select the subset where the categorical values are present and do something like `df[cat_cols].to_dict(orient='records')` to create a mapping of list of dicts which could then be feeded to the `DictVectorizer`. These could also be included in a pipeline too which could be used by scikit-learn estimators. — Nickil Maveli, Oct 29 '16 at 17:36
@NickilMaveli I experimented a little bit with it but did not yet get it to work. — Georg Heiler, Oct 29 '16 at 18:07

marbel · Accepted Answer · 2019-09-16T18:02:25.190

16

EDIT:

A more recent simpler/better way of handling this problem with scikit-learn is using the class sklearn.preprocessing.OneHotEncoder

from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(train)

enc.transform(train).toarray()

Old answer:

There are several answers that mention pandas.get_dummies as a method for this, but I feel the labelEncoder approach is cleaner for implementing a model. Other similar answers mention using DictVectorizer for this, but again converting the entire DataFrame to dict is probably not a great idea.

Let's assume the following problematic columns:

from sklearn import preprocessing
import numpy as np
import pandas as pd

train = {'city': ['Buenos Aires', 'New York', 'Istambul', 'Buenos Aires', 'Paris', 'Paris'],
        'letters': ['a', 'b', 'c', 'd', 'a', 'b']}
train = pd.DataFrame(train)

test = {'city': ['Buenos Aires', 'New York', 'Istambul', 'Buenos Aires', 'Paris', 'Utila'],
        'letters': ['a', 'b', 'c', 'a', 'b', 'b']}
test = pd.DataFrame(test)

Utila is a rarer city, and it isn't present in the training data but in the test set, that we can consider new data at inference time.

The trick is converting this value to "other" and including this in the labelEncoder object. Then we can reuse it in production.

c = 'city'
le = preprocessing.LabelEncoder()
train[c] = le.fit_transform(train[c])
test[c] = test[c].map(lambda s: 'other' if s not in le.classes_ else s)
le_classes = le.classes_.tolist()
bisect.insort_left(le_classes, 'other')
le.classes_ = le_classes
test[c] = le.transform(test[c])
test

  city  letters
0   1   a
1   3   b
2   2   c
3   1   a
4   4   b
5   0   b

To apply it to new data all we need is to save a le object for each column which can be easily done with Pickle.

This answer is based on this question which I feel wasn't totally clear to me, therefore added this example.

edited Sep 16 '19 at 18:02

answered Dec 14 '16 at 06:07

marbel

6,933
5
46
65

You mean the difference is that your solution requires only one column in memory whereas my solution needs all columns in memory? – Georg Heiler Dec 14 '16 at 06:09
Sehr Would you suggest other or null? I think null is good as well because the preprocessing code will handle it of called before the label encoder. – Georg Heiler Dec 14 '16 at 06:11
I feel this is a safer solution compared to `get_dummies` or `DictVectorizer`. I'm thinking on using one-k encoding after the labelEncoder, so whatever value will do. – marbel Dec 14 '16 at 06:14
That lambda mapping is not particularly efficient on large datasets. – Flair Aug 25 '17 at 16:34
3

If there's a "larger" value than `other` inside `le.classes_`, would `bisect.insort_left(le_classes, 'other')` end up inserting `other` before that element? If that's the case, the code for the elements after `other` would change, compromising the integrity of the code map. – ldavid Nov 21 '17 at 10:50
How to apply this to OneHotEncoding? For many categorical variables LabelEncoding is not suitable. – Nick Jun 30 '18 at 17:30
2

Watch out to not do `le.fit_transform(train[c])` before modifying the labelEncoder. Otherwise the mapping of the labelEncoder is not the same on train and test. Instead do `le.fit(train[c])` and then, after modifying the labelEncoder do `le.transform(train[c])`. – Pierre Gourseaud Aug 01 '18 at 10:35
@Arius, A workaround would be to add a row of 'other' to the Xtrain array before calling in fit_transform and then modifying the Xtest to have 'other' for new values. But that would require deleting this additional sample before training. – shahensha Apr 21 '19 at 16:22
1

Change `le.classes_ = le_classes` to `le.classes_ = np.array(le_classes)` in the old answer to preserve the usefulness of `preprocessing.LabelEncoder().inverse_transform()` – MaxPi Dec 01 '20 at 18:39
feel free to edit it! – marbel Dec 11 '20 at 15:25

Handling unknown values for label encoding

1 Answers1

Linked