[bug] ordinal encoder example code incorrect?

[suzannejb]

Enter the chapter number

2

Enter the page number

encoding example within sklearn section

What is the cell's number in the notebook

na

Enter the environment you are using to run the notebook

Jupyter on Windows

Describe your issue

the following code:
from sklearn.preprocessing import OrdinalEncoder

ordinal_encoder = OrdinalEncoder()
housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat)

seems like, instead of using housing_cat, the var used should be housing_full[["ocean_proximity"]]

Enter what you expected to happen

No response

If you found a workaround, describe it here

No response

[suzannejb]

also, I was compelled to change the ocean_proximity to ordinal and fix missing values in total_bedrooms before splitting into training and test sets.

[ageron]

Thanks for your feedback @suzannejb .

You generally don't want to train a model on the test data. Since housing_full contains the entire dataset, this includes test data, so we shouldn't call fit() or fit_transform() with data from housing_full. Instead, housing_cat is based only on the training set, so that's what we want.

You can certainly convert the ocean_proximity attribute to ordinal before working on the data, but the OrdinalEncoder takes care of that (at least in recent versions of Scikit-Learn), and I find it easier to keep track of the categories if they're represented as strings.

More generally, I prefer to split the data before doing any cleanup, keeping the test data quite "raw". The goal is to ensure that it's a good representative of the data we will get in production. This forces me to write pipelines that takes care of cleaning up the data, taking care of category encoding and other things. The benefit is that it lets you catch issues before your code reaches production, and the same pipeline can be used in production.

Hope this answers your questions.