Skip to main content
ML Evaluation Statistics Insurance
View as Markdown Suggest changes

XGBoost Beats the Tweedie GLM by Half a Point, and It's Real

· Reading time: 10 min
XGBoost Beats the Tweedie GLM by Half a Point, and It's Real

TL;DR

A premium is a mean, so the metric that picks a pricing model must be minimised by the mean and weight policies by exposure. That rules out MAE, MSE and the Gini, and leaves the Tweedie deviance, read as D2D^2, the share of the constant model’s deviance a model explains. On the French motor portfolio freMTPL2, a Poisson frequency × gamma severity XGBoost explains 4.54% in cross-validation, the Tweedie GLM 4.10%. Tiny, but paired by fold it is 5.3 standard errors, and on the untouched test set the gain is 0.42 points with a 95% interval from 0.23 to 0.61. The model’s premium is driven by bonus-malus, almost entirely through claim frequency. TabICL, a tabular foundation model that never trains on the data, trails because it sees only 10,000 policies; with the same 10,000, it gets 92% of the trees’ D2D^2.

Third of four posts on pricing freMTPL2. The first cleans the data, the second caps large claims at 18,000; everything here is on that cleaned, capped target.

Choosing the metric

Three requirements

We price μ(x)=E[Y∣x,e]\mu(x) = \mathbb E[Y \mid x, e], the expected claim cost per policy-year Y=S/eY = S/e. The metric must

  • (i) be minimised in expectation by the true mean, so a lower score means a better premium;
  • (ii) weight policies by the information they carry: a month of exposure says less than a year;
  • (iii) not be decided by a few claims.

The usual suspects fail:

MetricMinimised byFails
Absolute errorthe median, zero for most policies(i)
Squared error, RMSEthe mean(ii), (iii): every residual counts the same, though variance grows with the premium
Log-scale errorthe geometric mean, below the mean(i)
Gini of the ordered Lorenz curveany increasing transform of the mean(i): μ^\hat\mu and 10 μ^10\,\hat\mu score the same

The Gini is popular in pricing because it measures how well a tariff separates risks, and a competitor with a better ranking takes the policies we overcharge. But it can’t tell a right premium from one ten times too high, and a better ranking already lowers the deviance below. (iii) can’t be met by any metric that meets (i): the scores minimised by the mean are exactly the Bregman divergences, and under each of them a claim moves the comparison in proportion to its size. That is what the cap is for.

The Tweedie deviance, read as D2D^2

For a power p∈(1,2)p \in (1, 2), the Tweedie unit deviance and its exposure-weighted mean over a set AA of held-out policies are

dp(y,μ^)=2(y2−p(1−p)(2−p)−y μ^1−p1−p+μ^2−p2−p),Dp(μ^;A)=∑i∈Aei dp(yi(c),μ^i)∑i∈Aei.d_p(y, \hat\mu) = 2 \left( \frac{y^{2-p}}{(1-p)(2-p)} - \frac{y\, \hat\mu^{1-p}}{1-p} + \frac{\hat\mu^{2-p}}{2-p} \right), \qquad D_p(\hat\mu; A) = \frac{\sum_{i \in A} e_i\, d_p\big(y_i^{(c)}, \hat\mu_i\big)}{\sum_{i \in A} e_i}.

dpd_p is a Bregman divergence, so (i) holds whatever the true distribution of YY. Under a Tweedie model, Var(Y∣x,e)=ϕ μ(x)p/e\mathrm{Var}(Y \mid x, e) = \phi\, \mu(x)^p / e and DpD_p is the log-likelihood up to constants: each residual is divided by its variance, which is (ii). p=1.7p = 1.7 is fixed once from a moment estimate on the training data; recomputed for every pp from 1.2 to 1.9, the ranking below doesn’t change.

DpD_p depends on pp and on the currency, so it is reported as D2=1−Dp(μ^;A)/Dp(yˉ;A)D^2 = 1 - D_p(\hat\mu; A) / D_p(\bar y; A), with yˉ\bar y the mean premium of the fitting data. D2=0D^2 = 0 is the constant model. Models are compared by the difference of their D2D^2, not by a ratio, which would make 0.4 points look like an 11% improvement.

Calibration, the second check

The deviance barely sees the level. Scale the true mean by ss: the first derivative of the expected deviance vanishes at the mean, so

E[dp(Y,sμ)]−E[dp(Y,μ)]=(s−1)2 μ2−p+O((s−1)3).\mathbb E\big[d_p(Y, s\mu)\big] - \mathbb E\big[d_p(Y, \mu)\big] = (s - 1)^2\, \mu^{2-p} + O\big((s-1)^3\big).

A tariff 3% too high loses almost nothing in deviance and misprices the whole portfolio by 3%. So the deviance picks the model, and calibration, total predicted over total observed, overall and by decile, checks that the winner charges the right level.

The candidates

All models predict the annual capped pure premium, and all but TabICL use exposure as a weight, which for a log link has the same gradient as a log⁡ei\log e_i offset.

ModelSpecification
Constantμ^=yˉ\hat\mu = \bar y
Tweedie GLMlog⁡μ(x)=β0+h(x)⊤β\log \mu(x) = \beta_0 + h(x)^\top \beta, Tweedie deviance, small ridge penalty
Poisson × Gamma GLMPoisson frequency λ(x)\lambda(x) and gamma mean claim size m(x)m(x), each log-linear in h(x)h(x); μ^=λ^ m^\hat\mu = \hat\lambda\, \hat m
Tweedie XGBoostlog⁡μ(x)=∑t=1Tft(x)\log \mu(x) = \sum_{t=1}^{T} f_t(x), regression trees, Tweedie loss
Poisson × Gamma XGBoostthe GLM pair with each linear predictor replaced by a sum of trees
TabICLa pretrained transformer’s prediction given a context of training rows

h(x)h(x) is the GLM design: cubic B-splines in driver age, bonus-malus and vehicle age, log density, and one-hot categories with thin levels grouped. The trees take the raw features with native categorical splits, are monotone in bonus-malus, and stop early on an inner validation split; their other hyperparameters are defaults, not tuned.

TabICL is a tabular foundation model under a BSD licence. It isn’t trained on the portfolio: it stores 10,000 random policies as context, and a prediction is a forward pass that attends to them. It takes no weights, so exposure is an input, set to one at prediction, and a one-feature GLM on other fitting rows recalibrates its too-sharp raw predictions.

The result

Cross-validation

Each model is fitted on four of five folds and scored on the fifth:

ModelD2D^2Calibration
Constant0.00%1.001
Tweedie GLM4.10%1.007
Poisson × Gamma GLM3.86%0.997
Tweedie XGBoost4.37%1.001
Poisson × Gamma XGBoost4.54%0.988
TabICL2.94%1.047

Single-digit D2D^2 is normal for a pure premium: most of the variation is whether and how much a policy happens to claim, which no feature predicts. What matters is the gap to the GLM, fold by fold:

−2.0−1.5−1.0−0.50+0.5Poisson × Gamma GLMTweedie XGBoostPoisson × Gamma XGBoostTabICLD² minus the Tweedie GLM's, percentage points

Each model’s D2D^2 minus the Tweedie GLM’s on the same held-out fold (dots) and the mean (bar), in percentage points.

Is it real?

A fold with an unlucky claim is hard for every model, so models are compared paired by fold. For the best model bb and a model mm, with μ^(−k)\hat\mu^{(-k)} fitted without fold FkF_k,

Δk=Dp(μ^m(−k);Fk)−Dp(μ^b(−k);Fk),SE=sΔK.\Delta_k = D_p\big(\hat\mu_m^{(-k)}; F_k\big) - D_p\big(\hat\mu_b^{(-k)}; F_k\big), \qquad \mathrm{SE} = \frac{s_\Delta}{\sqrt K}.

The rule is the one-standard-error rule of Breiman et al.: take the simplest model, in the order of the table, within one SE of the best. Simpler models are easier to explain and to file with a regulator, so they win ties. The Tweedie GLM is 5.3 standard errors behind, the Tweedie XGBoost 2.3. The folds share most of their training data, so these errors are somewhat optimistic, but not fivefold. Splitting frequency and severity helps the trees and costs the GLMs a quarter point.

On the test set

The winner and the GLM were refitted on all training policies and scored once on the test set: 133,721 policies set aside before anything was fitted. A bootstrap over test policies, 200 resamples with the same policies for both models, gives the intervals:

  • Poisson × Gamma XGBoost explains 5.21% (4.64% to 5.81%), the Tweedie GLM 4.79% (4.17% to 5.39%).
  • The intervals overlap, but both models are scored on the same resampled policies, so most of the noise cancels in their difference: the gain is 0.42 points, from 0.23 to 0.61, positive in all 200 resamples.
Observed per policy-year25501002004002550100200400charged 44, claimed 31charged 317, claimed 328Charged per policy-year (capped premium, log scale)

Observed against charged capped premium per decile of equal exposure in the test set, log scales; on the dotted diagonal a decile pays what it is charged.

Observed cost rises from 31 to 328 per policy-year across the deciles, so the ordering holds on unseen policies, and no decile is off by more than 15. The line climbs slightly faster than the diagonal, the cautious under-spread that early stopping produces. In total the model predicts 1.017 times the capped cost, but the constant model predicts 1.038: the test policies simply claimed less than the training policies.

What the model learned

Trees have no coefficients, so the model is read in two ways. Permutation importance shuffles one feature across the test policies: shuffling bonus-malus drops D2D^2 from 5.21% to −2.97%, 71% of the total importance, followed by driver age with 10%. Partial dependence shows the shape: set a feature to one value for every policy, average the premium, divide by the average premium. For a multiplicative model that is the tariff’s relativity curve.

Bonus-malus012345507090106Driver age0.60.81.01.21.420406080Premium relative to the average premium (partial dependence)

Partial dependence on bonus-malus and driver age, relative to the average premium; each panel has its own scale.

  • Bonus-malus is the tariff. From 50 to 106 the premium rises by a factor of 6.7. Bonus-malus moves only with the policyholder’s own claims, so it measures the individual claim frequency no other factor can.
  • Driver age is not U-shaped. It falls from 1.14 at 20 to 0.75 near 28, rises to 1.24 near 49 and ends at 1.30 at 80. The middle peak fits parents whose children drive the car too, and part of the young-driver effect is already in their high starting bonus-malus.
  • It is all frequency. Bonus-malus moves claim frequency by a factor of 6.3 and mean claim size by 1.07. The rating factors decide how often a policy claims, hardly how much.

TabICL: the learner or the data?

TabICL’s context of 10,000 policies, about 365 of them with a claim, is 2.3% of a fitting set. To separate the learner from the data, the GLM and the boosted pair were refitted on random subsamples of the fitting folds, the smallest being exactly TabICL’s context:

Held-out D² (%)234510,00030,000100,000all 427,906Poisson × Gamma XGBoostTweedie GLMTabICLFitting policies (log scale)

Held-out D2D^2 against the number of fitting policies, mean over five folds; TabICL is the point at its context size. The GLM is shown from 30,000 policies.

  • With equal data, it is close to the trees. On the same 10,000 rows TabICL reaches 2.9%, the boosted pair 3.2%: 92% of it, without being trained on the portfolio.
  • The context limits it. The boosted pair gains 1.5 points from 10,000 policies to the full fitting set, about the whole gap to TabICL. TabICL can’t follow: every predicted row attends to every context row, so on a laptop GPU it sees 43 times fewer policies.

A tabular foundation model is a learned prior: it does most of the work when rows are few and loses its head start as data grows. For a portfolio this size, boosted trees are the better use of the data.

What the GLM keeps

The boosted model wins by a margin that is small and real. What it gives up is a tariff that can be filed line by line, so the Tweedie GLM stays as the reference, and the obvious next step is a GLM that keeps most of the gain by adding the interactions the trees find, such as driver age with bonus-malus. And the largest uncertainty in the premium isn’t in this comparison at all: it is the large-claim loading, which multiplies every premium and is known only to within about ±20%.

AI Chat

Messages you send are processed by the Google Gemini API to generate responses. Do not share sensitive personal data. See the privacy policy for details.