2  The Whole Game

The exercises for this chapter only uses the food delivery data.

Exercise 2.1  

Deliveries - Optimizing different Metrics

For the three models, also measure:

• The root mean squared error (RMSE) and
• Coefficient of determination (a.k.a. \(R^2\)).

Would you have picked a different tuning parameter value for the neural network?

Do the models rank differently?

Exercise 2.2  

Deliveries - Transforming the outcome

Determine a better scale for the outcome data to make its distribution more symmetric.

Do you see a benefit in performance? How do you quantify that?

Exercise 2.3  

Deliveries - What Not to Do: Calibration Edition

Re-predict the training set and use those values to estimate the calibration.

If you apply the calibration to the training set, what is the MAE?

How does that compare to the MAE values shown in the website’s analysis? Which do you think is correct and why?

Exercise 2.4  

Deliveries - Tuning Cubist

We (arbitrarily) decided to use a 100-member ensemble in the materials.

Like the neural network, try tuning the model across the ensemble size and determine if any size seems better than others (in terms of MAE).