Example usage
To use ols_regressor in a project:
from ols_regressor.regressor import LinearRegressor
from ols_regressor.cross_validate import cross_validate
import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, StandardScaler, OrdinalEncoder
from sklearn.compose import make_column_transformer
from sklearn.model_selection import train_test_split
In this instructional guide, we aim to demonstrate the practical application of our OLS_Regressor tool through an in-depth analysis of a comprehensive vehicle dataset gathered in Australia for the year 2023. This rich dataset encapsulates a wide range of information, including the latest car prices in Australia, encompassing a diverse array of brands, models, vehicle types, and distinctive features prevalent in the Australian automotive market.
The core focus of our study is to accurately predict the market price of vehicles using the basic information provided in the dataset. To achieve this, we employ our OLS_Regressor package. As the name indicates, this package is adept at fitting a linear regression model by utilizing the Ordinary Least Squares (OLS) method. It is further equipped with several analytical tools, including methods like predict for price estimation and score for evaluating model performance. In addition to these features, we have innovatively designed a bespoke cross_validate method, specifically tailored for the hyperparameter tuning process to enhance the model’s accuracy and efficiency.
Moreover, our package has been carefully crafted to align with the design patterns and methodologies used in the renowned scikit-learn package, with some minor yet significant modifications. This alignment ensures familiarity for users experienced with scikit-learn, while our enhancements offer additional value and unique capabilities.
Data Preprocessing
We have already worked on the data for an initial preprocessing which mainly consists of dropping some useless columns. The details of the initial preprocessing can be found in the data folder in the Github repository.
The initially preprocessed data is stored in the data/preprocessed_data.csv file. We read it using read_csv method provided by pandas.
data = pd.read_csv("../data/preprocessed_data.csv")
data.columns
Index(['Brand', 'UsedOrNew', 'Transmission', 'DriveType', 'FuelType',
'BodyType', 'Doors', 'Seats', 'Engine_cylinder_number',
'Engine_total_volume', 'ExteriorColour', 'Year', 'Kilometres', 'Price',
'fuel_comsumption_liter', 'fuel_comsumption_km'],
dtype='object')
The Price column is our response in this study. Therefore, we removed it from the data to create X and y for training purposes.
The data are then split into training and test parts using the train_test_split method provided by scikit-learn.
X, y = data.drop(columns=["Price"]), data["Price"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
A noticeable fact of our data is that the columns belong to various types of features. For an OLS model, the input features should only be numeric. The code snippet below is part of a data preprocessing pipeline. It demonstrates the use of different transformers for various types of features: categorical, ordinal, numeric, and features to be dropped. Let’s break down what each part of the code does:
Categorical Features:
categorical_featuresare those variables that represent categories, such as the brand or type of fuel. These features are transformed usingOneHotEncoder, which converts categorical data into a format that can be used by machine learning algorithms. Thehandle_unknown="ignore"parameter ensures that if any unknown category is encountered during transformation, it will be ignored rather than throwing an error.
Ordinal Features:
ordinal_featuresare categorical features but with a clear ordering or ranking (e.g., number of doors in a car). These are transformed usingOrdinalEncoder. Thehandle_unknown="use_encoded_value", unknown_value=999parameter settings imply that if an unknown category is encountered, it will be assigned a value of 999.
Numeric Features:
numeric_featuresare continuous numbers (e.g., year of manufacture, kilometers driven). These are standardized usingStandardScalerto normalize their range and distribution, making them more suitable for many machine learning algorithms.
Dropping Features:
drop_featuresspecifies the features to be excluded from the model. In this case,fuel_comsumption_literis being dropped. The"drop"transformer is used for this purpose.
Column Transformer (
ct):make_column_transformeris used to apply these transformations to the appropriate columns in the dataset. It creates a single transformer object (ct) which applies all the specified transformations to the dataset in a streamlined manner.
Transforming the Training Data (
X_train_encoded):Finally,
X_train_encodedis created by applying thecttransformer toX_train. This results in a transformed training dataset with one-hot encoded, ordinal encoded, standardized, and dropped features, making it ready for use in a machine learning model.
This preprocessing step is crucial for preparing the data correctly, ensuring that the machine learning model we choose to apply next can learn effectively from this structured and cleaned data.
# Categorical features should be encoded with OneHotEncoder
categorical_features = ['Brand', 'UsedOrNew', 'Transmission', 'DriveType', 'FuelType',
'BodyType', 'ExteriorColour']
# Ordinal features should be encoded with OrdinalEncoder
ordinal_features = ['Doors', 'Seats', 'Engine_cylinder_number']
# Numeric features should be normalized with StandardScaler
numeric_features = ['Year', 'Kilometres', 'Engine_total_volume', 'fuel_comsumption_liter']
# Since this feature contains only 100 for all observations, we simply drop it
drop_features = ['fuel_comsumption_liter']
# make up a column transformers based on the feature types
ct = make_column_transformer(
(OneHotEncoder(handle_unknown="ignore", sparse_output=False), categorical_features),
(OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=999), ordinal_features),
(StandardScaler(), numeric_features),
("drop", drop_features)
)
# fit the column transformer on the training data
X_train_encoded = ct.fit_transform(X_train)
X_train_encoded
array([[ 0. , 0. , 0. , ..., 1.49214463,
1.41067538, 1.51222474],
[ 0. , 0. , 0. , ..., -0.44649103,
-0.66911493, -0.71054051],
[ 0. , 0. , 0. , ..., -0.51863016,
-1.01574664, -1.7538793 ],
...,
[ 0. , 0. , 0. , ..., -0.00508252,
-0.43802711, -0.61981539],
[ 0. , 0. , 0. , ..., 0.65793342,
-0.43802711, -0.84662817],
[ 0. , 0. , 0. , ..., -0.2266645 ,
-1.24683446, 0.37816084]])
X_test_encoded = ct.transform(X_test)
X_test_encoded
array([[ 0. , 0. , 0. , ..., -1.07234979,
0.48632413, -0.07546472],
[ 0. , 0. , 0. , ..., 0.61690914,
0.71741194, 0.06062295],
[ 0. , 0. , 0. , ..., -1.22961588,
-0.43802711, 0.55961106],
...,
[ 0. , 0. , 0. , ..., -1.22920882,
-1.01574664, -0.34764006],
[ 0. , 0. , 0. , ..., -0.4935322 ,
-0.43802711, -0.52909028],
[ 0. , 0. , 0. , ..., -0.08272971,
-0.43802711, -0.34764006]])
Fitting the model on the training data
The fit function in the ols_regressor package will calculate the coefficients for the linear regression model using the Ordinary Least Squares (OLS) method. It converts the input features and target values into NumPy arrays. The function then augments the feature matrix with an intercept term and computes the model coefficients using the OLS formula. The resulting coefficients are stored in the self.coef attribute, representing the weights that minimize the sum of squared differences between the predicted and actual target values.
The use of this function is demonstrated below.
model = LinearRegressor()
model.fit(X_train_encoded, y_train)
array([ 3.31243354e+04, -3.93475549e+04, -3.24646306e+04, 1.59195143e+05,
-2.52374160e+04, -2.26802997e+04, -5.99355379e+04, 2.01100254e+05,
-2.02462376e+04, -5.23111567e+04, 4.22082867e+04, -4.60241522e+04,
-4.31256017e+04, -2.12256237e+04, -1.41643442e+04, -5.61456850e+04,
1.46990577e+04, -4.78145771e+04, -1.41355978e+04, 4.71981281e+05,
-4.21568751e+04, -3.55554888e+04, -5.88606157e+04, -5.23137170e+04,
-6.34807607e+04, -1.50773693e+04, -5.60401968e+04, -9.49431836e+03,
-5.27748543e+04, -1.11349923e+03, -4.11487545e+04, -3.26223859e+04,
1.93720780e+04, -3.69586647e+04, -1.60671583e+04, -4.53015637e+04,
2.06220655e+05, -3.02795242e+04, 5.11685049e+04, -2.09534812e+04,
-4.06118736e+04, -3.87486782e+04, -5.02415146e+04, 3.00881145e+05,
-6.54027526e+03, -2.28377048e+04, 5.18866116e+04, -4.98507558e+04,
-5.64256849e+04, 4.87344184e+04, -3.78121374e+04, 3.34400983e+05,
-1.58124464e+04, -3.47243053e+04, -3.79463997e+04, -3.72809066e+04,
-4.09005689e+04, 7.16243906e+04, -3.74947841e+04, -5.06594715e+04,
3.13868013e+04, -4.65583709e+04, -7.29321231e+03, -3.92775463e+04,
2.02678637e+05, -3.65082293e+04, -3.93350061e+04, -3.71545261e+04,
-5.40726596e+04, -4.03327576e+04, -4.18205677e+04, -4.81874292e+04,
-2.56369156e+04, -2.94822937e+04, -3.54756138e+04, -3.03336255e+04,
1.71682757e+04, 1.27992729e+04, 3.15680269e+03, 1.39944692e+04,
1.91298703e+04, 1.01794390e+04, 1.00936079e+04, 5.58341852e+02,
1.16575046e+04, 6.35446083e+02, -4.86143321e+03, 2.89130557e+04,
1.93153546e+03, -1.76582261e+04, 4.83442504e+04, -8.28144290e+03,
-6.56112112e+03, -8.70227872e+03, 7.63422290e+03, 1.75910941e+03,
2.00107348e+04, -5.09721911e+03, 5.34617099e+03, 7.21259705e+03,
-3.23498899e+03, -1.47466834e+03, 8.85509516e+02, 8.28712095e+01,
1.27848049e+03, 5.43480724e+02, 1.44402244e+03, 1.73957897e+03,
-1.04399667e+03, 1.51303238e+03, -1.82436390e+04, 2.26900371e+03,
4.76843876e+03, 1.73619578e+03, 2.60459814e+03, 1.44716379e+04,
6.14042915e+01, 6.11615801e+03, -7.87241904e+02, -3.60291161e+02,
-1.09781613e+03, 1.54652243e+04, 3.17809714e+02, 3.28258662e+02,
2.00271074e+02, 1.51714423e+03, 6.50019241e+03, 6.62422153e+03,
-9.63099633e+03, 3.04591320e+03, 8.36030092e+01])
model.coef
array([ 3.31243354e+04, -3.93475549e+04, -3.24646306e+04, 1.59195143e+05,
-2.52374160e+04, -2.26802997e+04, -5.99355379e+04, 2.01100254e+05,
-2.02462376e+04, -5.23111567e+04, 4.22082867e+04, -4.60241522e+04,
-4.31256017e+04, -2.12256237e+04, -1.41643442e+04, -5.61456850e+04,
1.46990577e+04, -4.78145771e+04, -1.41355978e+04, 4.71981281e+05,
-4.21568751e+04, -3.55554888e+04, -5.88606157e+04, -5.23137170e+04,
-6.34807607e+04, -1.50773693e+04, -5.60401968e+04, -9.49431836e+03,
-5.27748543e+04, -1.11349923e+03, -4.11487545e+04, -3.26223859e+04,
1.93720780e+04, -3.69586647e+04, -1.60671583e+04, -4.53015637e+04,
2.06220655e+05, -3.02795242e+04, 5.11685049e+04, -2.09534812e+04,
-4.06118736e+04, -3.87486782e+04, -5.02415146e+04, 3.00881145e+05,
-6.54027526e+03, -2.28377048e+04, 5.18866116e+04, -4.98507558e+04,
-5.64256849e+04, 4.87344184e+04, -3.78121374e+04, 3.34400983e+05,
-1.58124464e+04, -3.47243053e+04, -3.79463997e+04, -3.72809066e+04,
-4.09005689e+04, 7.16243906e+04, -3.74947841e+04, -5.06594715e+04,
3.13868013e+04, -4.65583709e+04, -7.29321231e+03, -3.92775463e+04,
2.02678637e+05, -3.65082293e+04, -3.93350061e+04, -3.71545261e+04,
-5.40726596e+04, -4.03327576e+04, -4.18205677e+04, -4.81874292e+04,
-2.56369156e+04, -2.94822937e+04, -3.54756138e+04, -3.03336255e+04,
1.71682757e+04, 1.27992729e+04, 3.15680269e+03, 1.39944692e+04,
1.91298703e+04, 1.01794390e+04, 1.00936079e+04, 5.58341852e+02,
1.16575046e+04, 6.35446083e+02, -4.86143321e+03, 2.89130557e+04,
1.93153546e+03, -1.76582261e+04, 4.83442504e+04, -8.28144290e+03,
-6.56112112e+03, -8.70227872e+03, 7.63422290e+03, 1.75910941e+03,
2.00107348e+04, -5.09721911e+03, 5.34617099e+03, 7.21259705e+03,
-3.23498899e+03, -1.47466834e+03, 8.85509516e+02, 8.28712095e+01,
1.27848049e+03, 5.43480724e+02, 1.44402244e+03, 1.73957897e+03,
-1.04399667e+03, 1.51303238e+03, -1.82436390e+04, 2.26900371e+03,
4.76843876e+03, 1.73619578e+03, 2.60459814e+03, 1.44716379e+04,
6.14042915e+01, 6.11615801e+03, -7.87241904e+02, -3.60291161e+02,
-1.09781613e+03, 1.54652243e+04, 3.17809714e+02, 3.28258662e+02,
2.00271074e+02, 1.51714423e+03, 6.50019241e+03, 6.62422153e+03,
-9.63099633e+03, 3.04591320e+03, 8.36030092e+01])
Cross Validation
Sometimes, we may need to get both the train score and validation score for hyperparameter tuning. Therefore, we need cross-validation to get the validation performance while avoiding overfitting. Our implementation of cross-validation is somewhat similar to the implementation of scikit-learn. The cross_validate function accepts five arguments:
model: The model to perform cross-validationX: The predictors of training data. It should be a 2D numpy arrayy: The response of training data. It should be a 1D numpy arraycv: The number of folds used for cross-validationrandom_state: The random state of the random shuffling in the cross-validation
Notice that our cross_validate does not require a return_train_score argument. The train scores are automatically returned in the cross-validation results.
cv_results = cross_validate(model, X_train_encoded, y_train.to_numpy(), 5, 42)
pd.DataFrame(cv_results)
| train_score | test_score | fit_time | score_time | |
|---|---|---|---|---|
| 0 | 0.669816 | 0.664145 | 0.029977 | 0.002372 |
| 1 | 0.688293 | 0.537996 | 0.017776 | 0.002241 |
| 2 | 0.666635 | 0.670863 | 0.021107 | 0.002276 |
| 3 | 0.715619 | 0.488649 | 0.017541 | 0.002277 |
| 4 | 0.665131 | 0.687616 | 0.021016 | 0.002270 |
| 5 | 0.672295 | -19.209700 | 0.033301 | 0.000342 |
Predicting with the Fitted Model
Now that our regression model has been fitted, it is time to utilize it for making predictions on unseen data. The predict function within the ols_regressor package has been designed for this purpose. This function expects an array-like matrix X of shape (n_samples, n_features) as input so that we can compute the predicted target values with the coefficients stored in the self.coef attribute. The predict function will return an array containing the model’s predictions based on the provided input features.
The use of this function is demonstrated below.
model.predict(X_test_encoded)
array([55898.43408883, 75474.05804947, 67526.80375404, ...,
44614.05367761, 33326.25381787, 23928.94731972])
Scoring the Fitted Model
Here we use the score function within the ols_regressor package. The function takes in X(n_samples, n_features) and y_pred(n_samples, ) as input and calculates the coefficient of determination for the prediction.
model.score(X_test_encoded, y_test)
0.5705212165508947