tommyodland.com

Decline Curve Analysis

Tommy Odland — Fri, 09 Oct 2026 00:00:00 +0200

Abstract. Decline Curve Analysis (DCA) is about fitting parametric curves to petroleum production time series, usually with the goal of forecasting future production. DCA dates back to at least the 1940s, and is more accessible than ever with modern optimization methods. It is a simple, transparent and fast forecasting method that only requires time series data.

Still, the statement “let’s use DCA” leaves many details unspecified: how should data be preprocessed? Which parametric curve should one use? What kind of loss function is appropriate? How to incorporate uncertainty? In this article we provide one set of answers to these questions.

Download the PDF

1. Introduction

Figure 1. An Arps curve model fit to production data. DCA is most commonly used on individual well time series, but here we perform DCA on the aggregated field level.

Decline Curve Analysis (DCA) is the process of fitting parametric curves to well production time series data. The idea dates back to when Arps (1945)¹¹ introduced his three-parameter curve model:

\begin{equation} \label{eqn:arps} \tag{1} f(t; q, h, D) = q(1 + hDt)^{-1/h} \end{equation}

DCA is simple: it requires no geological, nor any physics-based, model. We simply take observed production and extrapolate with a curve model $f$. But simple does not always mean easy: Should we use the well-known Arps curve, or perhaps some other curve model instead? Which loss function? How should data be pre-processed? How to handle outliers? How to quantify uncertainty? In this article we discuss these questions.

Equinor (2025)¹² provides an open-source Python implementation of many of the ideas presented in this article. I hope you apply the ideas (and code) in your own forecasting.

1.1 Simplicity in forecasting

You might be surprised that we, in 2026, discuss what is essentially an eighty year old forecasting method. Has the world not progressed? Are better and more sophisticated approaches not available? The answer is nuanced: we’ll first discuss forecasting in general, then forecasting petroleum time series specifically.

It is not always the case that novel methods lead to better results. On many problems across industries, simple and well-established methods often produce the best results.¹ Green & Armstrong (2015)¹³ reviewed papers on forecasting and found little evidence that complexity improves accuracy. Fama & French (2010)¹⁴ found no evidence that active management of stock portfolios outperformed the market average. Yan (2022)¹⁵ argues that complex methods are often chosen because they signal effort, mastery and innovation: complexity sells, but simplicity is an advantage. Simple methods also have an advantage in real-world deployments: they are easier to understand and use, easier to build and scale, and easier to maintain.

What about forecasting petroleum production specifically? Are engineers professionals whose expertise and advanced models drastically improve forecasts, or are they more like stock market investors—believing that they outperform the market when they rarely do? The answer is somewhere in between the two extremes, but the second possibility must be considered. Bratvold et al. (2020)¹⁶ showed that early stage forecasts of petroleum projects are systematically optimistic and overconfident. Odland & Hollund (2026)¹⁷ showed that on 70% of wells in a sample of thousands, simple automatic DCA outperformed engineering forecasts. Unfortunately we do not know the full answer, since there is little public data available for researchers to compare forecasting methods on, and the literature is somewhat disappointing.

1.2 A critique of the literature

I have chosen to not cite many papers related to DCA, since I believe most of it is low-quality. Novel methods are rarely compared against strong baselines, such as a properly tuned DCA. Train/test splits are often absent or have time leaks. Code is seldom provided. Evaluation is rarely statistically meaningful: ten or twenty wells is not enough data for a meaningful conclusion. In summary, basic scientific method is often lacking in papers published on DCA and related topics.²

To remedy this, the goal must be clearly stated: we wish to predict future production as accurately as possible. Whether the theory is clean or the methods are novel is unimportant if the forecasts are inaccurate. In a real-world engineering organization, the cost of complexity must be considered: complex methods require more data inputs, are harder to reason about, and have higher operational costs.

Having said all this, I have nothing against physical theories or fancy methods. My plea is only that engineers use scientific methodology when choosing forecasting methods: evaluate on many wells, on a test set, benchmarking against a strong baseline. If nothing else, I hope the ideas in this article provide a baseline to benchmark your own ideas against.

2. Basic mathematical model

In this section we outline a model that we’ll extend on in two ways: first by curve fitting, then by statistical modeling. The model is frequency-agnostic, but in reality data typically arrives either daily or monthly. Each day or month is called a period. Within each period $i$, we observe the total production $p_i$ as well as an uptime fraction $0 < \tau_i \leq 1$. The unit of production is arbitrary. The uptime denotes how much of a time period the well is producing. For instance, if a well was shut down for $3$ out of $30$ days, the uptime would be $\tau = 0.9$.

Example 1 (Observed data). If we observe monthly data with structure

period    production    uptime
 2026-01   0.482         0.92
 2026-02   0.435         0.95
 2026-03   0.445         1.00

then $\boldsymbol{p} = (0.482, 0.435, 0.445)$ and $\boldsymbol{\tau} = (0.92, 0.95, 1.00)$. If the well is always producing, then $\tau_i = 1$ for all $i$ and the equations simplify.

We define the cumulative uptime as $\xi_i = \sum_{j=1}^i \tau_j$. Suppose we have a decline curve model $f(t; \boldsymbol{\theta})$, such as the Arps curve, that represents production rate (the units of $f$ are production per time unit). Ignoring noise and assuming our model is correct:

Production in the first period is $p_1 = \int_{0}^{\tau_1} f(t; \boldsymbol{\theta}) \, dt = \int_{\xi_0}^{\xi_1} f(t; \boldsymbol{\theta}) \, dt$.
Production in the second period is $p_2 = \int_{\tau_1}^{\tau_1 + \tau_2} f(t; \boldsymbol{\theta}) \, dt = \int_{\xi_1}^{\xi_2} f(t; \boldsymbol{\theta}) \, dt$.
Production in the third period is $p_3 = \int_{\tau_1 + \tau_2}^{\tau_1 + \tau_2 + \tau_3} f(t; \boldsymbol{\theta}) \, dt = \int_{\xi_2}^{\xi_3} f(t; \boldsymbol{\theta}) \, dt$.

One approach would be to work with the antiderivative $F(t; \boldsymbol{\theta}) = \int_0^t f(z; \boldsymbol{\theta}) \, dz$, by setting $p_i = F(\xi_i; \boldsymbol{\theta}) - F(\xi_{i-1}; \boldsymbol{\theta})$. Another approach, which we pursue, is to apply the midpoint rule of numerical integration to obtain

\begin{equation*} p_i = \int_{\xi_{i-1}}^{\xi_{i}} f(t; \boldsymbol{\theta}) \, dt \approx f\left( \frac{\xi_{i-1} + \xi_{i}}{2}; \boldsymbol{\theta} \right) \tau_i. \end{equation*}

When $f$ is nearly linear at small time segments, this approximation is very good.³ Dividing both sides by $\tau_i$ and introducing $t_i = (\xi_{i-1} + \xi_{i}) / 2 = \xi_i - \tau_i / 2$ as the midpoint, we obtain a curve fitting problem on production rates $y_i$:

\begin{equation*} y_i = \frac{p_i}{\tau_i} = f\left( t_i; \boldsymbol{\theta} \right) \end{equation*}

2.1 Data preprocessing

We now introduce two preprocessing rules that transform raw data $(\boldsymbol{p}, \boldsymbol{\tau})$ into $(\boldsymbol{t}, \boldsymbol{y}, \boldsymbol{w})$ for curve fitting. Each $(t_i, y_i)$ is a data point and $w_i$ is its associated weight. The choice of rule depends on the question the forecaster wants to answer. In addition to these two rules, standard data cleaning is needed: missing values, zeros, inconsistencies, etc.

Producing time transform. This data preprocessing rule answers the question “If the well has 100% uptime in the future, what will it produce?” On the horizontal (time) axis, we fit to midpoints and account for uptime by contracting the axis. On the vertical (production rate) axis, we fit to production rates. Data is weighted by the uptime fraction.

\begin{align*} t_i = \xi_i - \tau_i / 2 \quad y_i = \frac{p_i}{\tau_i} \quad w_i = \tau_i \end{align*}

Calendar time transform. This data preprocessing rule answers the question “If the uptime in the future is roughly equal to the uptime in the past, what will the well produce in the future?” On the horizontal axis, we fit to midpoints within each period without accounting for uptime. On the vertical axis, we fit to production values. Data is not weighted by the uptime fraction.

\begin{align*} t_i = i - 0.5 \quad y_i = p_i \quad w_i = 1 \end{align*}

Notice that the calendar time transform is equivalent to setting $\tau_i = 1$ for all periods $i$ and applying the producing time transform. The crucial difference between the two rules is whether (1) time passes only when the well is producing, or if it (2) passes irrespectively of whether the well is producing or not.

Example 2 (Data transformations). Suppose $\boldsymbol{p} = (100, 50, 25)$, $\boldsymbol{\tau} = (1.0, 0.5, 0.25)$ and we use a simple curve model $f\left( t; \theta_1 \right) = \theta_1$ with a least-squares loss function. The producing time transform yields

\begin{align*} \boldsymbol{t} = (0.5, 1.25, 1.625) \quad \boldsymbol{y} = (100, 100, 100) \quad \boldsymbol{w} = (1.0, 0.5, 0.25) \end{align*}

and the forecast becomes $100$ units of production in the future, since that is what the well will produce if it has 100% uptime in the future. If we instead use the calendar time transform, the forecast would be $(100 + 50 + 25) / 3 \approx 58.33$ units per period in the future.

2.2 Naive least-squares fit

Having established a mathematical model and data preprocessing rules, we are now in a position where we could use a weighted least-squares loss function and minimize

\begin{equation} \label{eqn:leastsquares} \tag{2} \mathcal{L}\left( \boldsymbol{\theta} \right) = \sum_i w_i \left( y_i - f\left( t_i; \boldsymbol{\theta} \right) \right)^2 \end{equation}

to perform DCA. However, we do not recommend this approach, and will introduce alternatives that improve forecasting accuracy.

Two directions are possible: we can modify the curve model $f$ or the loss function $\mathcal{L}$. While many papers compare curve models, the largest improvements in forecasting accuracy are obtained by modifying the loss function. Before we venture down that path we will discuss curve models.

3. Curve models

The Arps model is the most popular choice of curve model, but other parametric functions can also be used. In this section we aim to demystify some common curve models, discuss their properties and re-parametrize them so their parameters are expressed as unbounded real numbers.

3.1 The constant model

The constant model is simply

\begin{equation*} \log f\left( t; \theta_1 \right) = \theta_1. \end{equation*}

The logarithm ensures that the parameter $\theta_1$ is unbounded and the outputs are positive. This model should not be used for real-world forecasting, but it is a useful test-case.

3.2 The exponential model

Figure 2. Optimization log-landscape for two parametrizations of the exponential curve. The $(C, k)$ -parametrization has a thin, narrow valley around the optimum. The $(\theta_1, \theta_2)$ -parametrization is closer to quadratic. The difference is more pronounced than the figure shows, since the axes of the $(C, k)$ -parametrization have been scaled for visibility. If unscaled, the valley is so narrow that it is nearly impossible to see on a plot.

An exponential function such as $f(t; C, k) = C \exp(-kt)$ is a viable DCA candidate. However, if this parametrization is used, two issues crop up. The first is that $C$ and $k$ must both be positive, so a constrained optimization algorithm must be used. The second issue is that $C$ and $k$ might be on vastly different scales, causing first-order optimization methods to struggle.

One solution is to scale the input data to mitigate the different scales of $C$ and $k$. Another solution is to use an alternative parametrization, such as

\begin{equation*} \log f(t; \theta_1, \theta_2) = \theta_1 - \theta_2 - t \exp(-\theta_2) \end{equation*}

which avoids both of the problems above. See Figure 2 for a visualization of the optimization landscapes associated with these two parametrizations.

3.3 The Arps model

The Arps model is convenient and often matches real-world data. That being said, the main reason for its popularity is arguably historical entrenchment. There is nothing inherently superior about the Arps curve, and many papers show equally good forecasts when using other, similar curve models.⁵

The most common parametrization of the Arps model is

\begin{equation*} f(t; q, h, D) = q(1 + hDt)^{-1/h}, \end{equation*}

where $q >0$, $D >0$ and $0 < h \leq 1$. When $h < 1$, the integral is finite and evaluates to

\begin{equation*} \int_0^T f(t; q, h, D) \, dt = \frac{q \left[ (1 + ThD)^{\frac{h-1}{h}} - 1 \right]}{(h-1) D}. \end{equation*}

This parametrization suffers from the same issues as the naive exponential parametrization, and numerical issues can arise as $h \to 0$ if implemented naively. An alternative parametrization, due to Lee & Mallick (2021)¹⁸, avoids these issues:⁴

\begin{equation*} \log f(t; \theta_1, \theta_2, \theta_3) = \theta_1 - \theta_2 - \left( 1 + e^{-\theta_3} \right) \log \left( 1 + t e^{\theta_3 - \theta_2} \right). \end{equation*}

It comes from the following change of variables:

\begin{align*} \theta_1 &= \log \left( q / ((1 - h) D) \right) & & h= 1 / (1 + \exp(-\theta_3)) \\ \theta_2 &= \log \left( 1 / ((1 - h) D) \right) & & D= 1 / ((1 - h) \exp(\theta_2) ) \\ \theta_3 &= \log \left( h / (1 - h) \right) & & q= \exp(\theta_1) (1-h ) D. \end{align*}

A few comments are in order:

The Arps model is equivalent to the generalized Pareto distribution. If you want to experiment with other curve models, look to probability density functions and modify them as needed. Two popular alternatives, not discussed further in this article, are the stretched exponential curve and the Weibull curve.
The Arps model can be thought of as a relaxation of the definition $\exp x := \lim_{n \to \infty} \left(1 + \frac{x}{n}\right)^n$, where we choose $n = 1/h$ and instead of taking the limit $h \to 0$ we leave $h$ as a free parameter. As $h \to 0$, or equivalently as $\theta_3 \to -\infty$, the Arps model converges to the exponential model.
Since $\int_{0}^{\infty} f(t; \boldsymbol{\theta}) \, dt = \exp (\theta_1)$, prior belief about the Estimated Ultimate Recovery (EUR) of a well can be encoded as a prior distribution on $\theta_1$.
Shifting the log-model up or down can be accomplished by changing $\theta_1$, since $\log f(t; \theta_1, \theta_2, \theta_3) + z = \log f(t; \theta_1 + z, \theta_2, \theta_3)$. Pointwise P10 and P90 curves can therefore be obtained by specifying $\theta_1^{\text{P10}}$ and $\theta_1^{\text{P90}}$.

4. Curve fitting

In Section 2.2 we introduced the least squares loss and claimed that it is inappropriate for DCA. In this section we extend the loss with a non-statistical curve fitting approach. Later we will consider a statistical perspective.

Figure 3. Unweighted vs. weighted least squares fit on production in the Statfjord field. We use aggregated field data from the Norwegian Offshore Directorate (factpages.sodir.no). It would be more meaningful to look at individual wells, but these data are not public.

4.1 Weighting recent data

When forecasting the future, the recent past matters more than the distant past. Based on this observation, we introduce a weighting function $W(t_i, \gamma)$ and modify the loss function in $\eqref{eqn:leastsquares}$ to read

\begin{equation*} \mathcal{L}\left( \boldsymbol{\theta}; \gamma \right) = \sum_i W(t_i, \gamma) \, w_i \left( y_i - f\left( t_i; \boldsymbol{\theta} \right) \right)^2, \end{equation*}

where the weight function is an exponential decay going back in time, given by

\begin{equation*} W(t_i, \gamma) \propto 2^{(t_i - \max_i t_i) / \gamma}. \end{equation*}

The parameter $\gamma > 0$ is the half-life. It controls how far back in time we go until a data point receives a relative weight of 50%. Half-life lets us focus in on recent data without using a hard cutoff, and its effect is shown in Figure 3.

A good starting point for $\gamma$ is to set it equal to the forecasting horizon. For instance, if the forecasting horizon is one year into the future, and data has monthly frequency, then $\gamma = 12$ is a good value. A more sophisticated approach is to treat $\gamma$ as a hyper-parameter and tune it on a test set that matches the forecasting horizon in length.

4.2 Fitting on log scale

Often, fitting log-model to log-production is sensible since it penalizes multiplicative errors instead of additive. With this loss, predicting e.g. 20% over the observed production rate $y_i$ is equally bad, no matter the value of $y_i$.

\begin{equation*} \mathcal{L}\left( \boldsymbol{\theta}; \gamma \right) = \sum_i W(t_i, \gamma) \, w_i \left( \log y_i - \log f\left( t_i; \boldsymbol{\theta} \right) \right)^2 \end{equation*}

While fitting on log-scale often leads to better out-of-sample predictions, it introduces a theoretical misalignment: If $X$ is a random variable, then $\exp \operatorname{mean}_i \log X_i \leq \operatorname{mean}_i X_i$. Fitting in log-space and then exponentiating the result, without accounting for the variance, underestimates the mean. However, the primary goal is to forecast the test set to high accuracy—a clean theory is not a goal in and of itself. The effect of fitting in log-space is shown in Figure 4. Test set performance can help decide whether one should fit in log-space or not.⁶

Figure 4. The effect of fitting the Arps model in the original space (without taking logarithm) vs. fitting in log space (fitting log-model to log-data).

4.3 The median curve

Outliers show up in real-world data for various reasons. Fitting in log-space dampens their effect. We can also fit a median curve instead of a mean curve. To accomplish this, we change the $2$ to $1$ in the loss function exponent—minimizing Mean Absolute Error (MAE) instead of Mean Squared Error (MSE). More generally, we can introduce a hyperparameter $1 \leq p \leq 2$ that we tune by cross-validation:

\begin{equation*} \mathcal{L}\left( \boldsymbol{\theta}; \gamma, p \right) = \sum_i W(t_i, \gamma) \, w_i \lvert \log y_i - \log f\left( t_i; \boldsymbol{\theta} \right) \rvert^p \end{equation*}

Test-set evaluation can still be measured by RMSE. Another option is to change the evaluation metric from MSE to MAE.

Figure 5. Fitting a mean curve (minimizing MSE) and a median curve (minimizing MAE) on the same data set. The fit was performed in log-space with exponential data weighting. Notice how the mean curve is influenced by the downward outliers, while the median curve is not. Whether these low-production periods should be considered outliers or not can be determined by cross validation instead of subjective assessment - choose the $p$ that leads to the best out-of-sample forecasting performance.

4.4 Regularization

To produce robust forecasts of wells with little data, and help the optimizer converge, we can add a regularization term $P(\boldsymbol{\theta})$ to the loss. With a quadratic regularization term, the loss becomes:

\begin{align*} \mathcal{L}\left( \boldsymbol{\theta}; \gamma, p, \alpha \right) &= \sum_i W(t_i, \gamma) \, w_i \lvert \log y_i - \log f\left( t_i; \boldsymbol{\theta} \right) \rvert^p + \alpha P(\boldsymbol{\theta}) \\ P(\boldsymbol{\theta}) &= \left( \boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}} \right)^T \boldsymbol{\Sigma}^{-1} \left( \boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}} \right) \end{align*}

Regularization strength is controlled by the hyperparameter $\alpha \geq 0$, which can be set manually or tuned by cross validation. A simple but effective way of determining the prior mean $\boldsymbol{\mu}_{\boldsymbol{\theta}}$ is by a two stage process: first do a pilot-fit where all wells are concatenated into one data set, then use the observed mean $\boldsymbol{\mu}_{\boldsymbol{\theta}}$ to inform each individual well.⁷ The matrix $\boldsymbol{\Sigma}^{-1}$ can be determined by a pilot fit, or it can simply be set to the identity matrix.

Taking all these ideas into account (data weighting, fitting on log-scale, allowing a median curve and adding a regularization term), the final loss function becomes

\begin{equation} \label{eqn:dcaloss} \tag{3} \begin{aligned} \mathcal{L}(\boldsymbol{\theta}; \gamma, p, \alpha) &= \sum_i W(t_i, \gamma)\, w_i \lvert \log y_i - \log f\left( t_i; \boldsymbol{\theta} \right) \rvert^p + \alpha P(\boldsymbol{\theta}) \\ W(t_i, \gamma) &\propto 2^{(t_i - \max_i t_i)/\gamma} \\ P(\boldsymbol{\theta}) &= \left(\boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}}\right)^T \boldsymbol{\Sigma}^{-1} \left(\boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}}\right) \end{aligned} \end{equation}

The weights $W(t_i, \gamma)$ can be scaled so the final weight is $1$, or so that $\sum_i W(t_i, \gamma)\, w_i = \sum_i w_i$. Least-squares loss is a special case of this more general loss function. A practitioner can choose which of the ideas presented in this section to implement or discard, depending on their forecasting problem. For more tips and tricks related to curve fitting in general, see Odland (2025).²²

5. A statistical model

In this section we present a statistical model that forecasts future production with uncertainty. Before we go down this path: remember that forecasted uncertainty relies on the assumption that the statistical model is correct. If the model is unreasonable, the forecast will be unreasonable too.

The curve fitting approach explored in Section 4 is forgiving when forecasts are aggregated, since errors tend to cancel when forecasting a group of wells: if one well is over-forecasted, the next might be under-forecasted, and so on. This is not the case for uncertainty. In each individual well, uncertainty is often under-estimated, and these errors do not cancel when wells are aggregated.

5.1 Independent errors

Our first attempt at a statistical model assumes independent errors. We can specify a forward model, that we can simulate from, as

\begin{align*} \log \left( y_i \right) &= \log \left( f\left(t_i; \boldsymbol{\theta} \right) \right) + \sqrt{\tau_i} \sigma \epsilon_i \\ \epsilon_i &\sim N(0, 1). \end{align*}

When we employ additive errors on the log-scale, we assume multiplicative errors and guarantee that predicted production is positive.⁸ Once parameters are estimated, the expected value can be computed as

\begin{equation} \label{eqn:expected_value_arps} \tag{4} p_i / \tau_i = \mathbb{E} \left[ \exp \left( \log \left( f\left(t_i; \boldsymbol{\theta} \right) \right) + \sqrt{\tau_i} \sigma \epsilon_i \right) \right] = f\left(t_i; \boldsymbol{\theta} \right) \exp \left( \tau_i \sigma^2 / 2 \right), \end{equation}

where the last equality is due to the fact that $\mathbb{E} \left[ \exp N(\mu, \sigma^2) \right] = \exp \left( \mu + \sigma^2 / 2\right)$. When forecasting with uncertainty, Monte-Carlo simulation should be performed since the sum of log-normal distributions does not have an analytical equation.

The square-root term $\sqrt{\tau_i}$ comes from thinking of noise as a Wiener process within each period on the log scale: $W(\xi_i) - W(\xi_{i-1}) \sim N(0, \tau_i \sigma^2)$. Alternatively, we may think of summing $n$ independent normal variables: the standard deviation grows like $\sqrt{n}$.

With this model, observing (or forecasting) data on a fine grid and a coarse grid is not equivalent. The issue is that the sum of two log-normals is not log-normal. Assuming $f$ is approximately constant over two periods and forecasting the expected value yields

\begin{align*} \text{2 periods of length 1: }& \frac{p_i }{1} + \frac{p_{i+1} }{1} = f \exp \left( \sigma^2 / 2 \right) + f \exp \left( \sigma^2 / 2 \right) \\ \text{1 period of length 2: }& \frac{p_j }{2} = f \exp \left( 2 \sigma^2 / 2 \right) \end{align*}

The requirement $p_i + p_{i+1} = p_j$ translates into $2 f \exp \left( \sigma^2 / 2 \right) = 2 f \exp \left( 2 \sigma^2 / 2 \right)$, and this only holds when $\sigma$ is small. In practice every value of $\tau_i$ is often similar, and we can use a (possibly weighted) mean $\bar{\boldsymbol{\tau}}$ when forecasting before interpolating onto a unit grid.

5.2 Autocorrelated errors

Figure 6. An AR(1) process sampled at equidistant discrete time, and a discretized OU process sampled at non-equidistant times with equivalent parameters. Notice how the process stays above or below trend for a while, but eventually reverts back to trend level. This phenomenon is observed in real-world production data.

Simulating from the model in Section 5.1 does not yield realistic results, because wells rarely exhibit independent errors. Therefore, forecasting from the model does not yield realistic uncertainty either. Real data is typically above or below trend for longer time periods, as shown in Figure 6.

On discrete, equidistant time intervals, we can model correlated noise using an AR(1) process centered around zero:

\begin{equation*} X_{i} = \phi X_{i-1} + \sigma \epsilon_i, \quad \epsilon_i \sim N(0, 1). \end{equation*}

The parameter $\sigma > 0$ determines how much the next value $X_i$ is perturbed away from the previous value, while $0 \leq \phi < 1$ determines mean reversion. The first conditional moments of $X_i$ are given by

\begin{equation*} \mathbb{E}\left[ X_{i} \mid X_{i-n} \right] = X_{i-n} \phi^n \quad \operatorname{Var}\left[ X_{i} \mid X_{i-n} \right] = \sigma^2 \frac{1 - \phi^{2n}}{1 - \phi^2} \end{equation*}

Taking the limit $n \to \infty$ we obtain the long-running steady state moments

\begin{equation*} \mathbb{E}\left[ X_{i} \right] = 0 \quad \operatorname{Var}\left[ X_{i} \right] = \frac{\sigma^2}{1 - \phi^2} \end{equation*}

Since we allow uptime $\tau_i$ to be different in each time period, our time intervals are not equidistant. Assuming that uptime does not vary too much, one approximation is to use $X_{i} = \phi X_{i-1} + \sqrt{\tau_i} \sigma \epsilon_i$, but this process grows unbounded if $\tau_i \to \infty$. A more appropriate model can be found by discretizing the Ornstein-Uhlenbeck (OU) process, which is a continuous time analogue of AR(1). The stochastic differential equation that defines the OU process is

\begin{equation*} d x_{i} = -\vartheta x_{i} dt + \sigma dW_i, \end{equation*}

where $W_i$ is a Wiener process. The solution yields the expectation and variance on an arbitrary, non-equidistant grid with spacing $\tau_i$:

\begin{equation*} \mathbb{E}\left[ x_{i} \mid x_{i-1} \right] = x_{i-1} e^{-\tau_i \vartheta} \quad \operatorname{Var}\left[ x_{i} \mid x_{i-1} \right] = \sigma^2 \frac{ \left( 1 - e^{-2 \vartheta \tau_i} \right)}{2 \vartheta}. \end{equation*}

In the limit $\tau_i \to \infty$, we obtain the stationary moments

\begin{equation*} \mathbb{E}\left[ x_{i} \right] = 0 \quad \operatorname{Var}\left[ x_{i} \right] = \frac{ \sigma^2}{2 \vartheta}. \end{equation*}

See Holý & Tomanová (2025) for more information about these equations.²⁰ Notice that if $\vartheta \tau_i \approx 0$, then we can write $e^{-2 \vartheta \tau_i} \approx 1 - 2 \vartheta \tau_i$ and the variance becomes $\operatorname{Var}\left[ x_{i} \mid x_{i-1} \right] \approx \sigma^2 \tau_i$, which leads to a $\sqrt{\tau_i}-$approximation of the standard deviation.

The exact discretized noise model, which we will use in the following model, is

\begin{equation*} X_i = \exp(-\vartheta \tau_i) X_{i-1} + \sigma \sqrt{\frac{1 - \exp(-2 \vartheta \tau_i)}{2 \vartheta}} \epsilon_i \end{equation*}

5.3 Statistical model with autocorrelation

The final statistical model uses the discretized OU process, but with a Generalized Normal (GN) shock term $\epsilon_i$. The reason for the GN distribution is the same as allowing $1 \leq p \leq 2$ in the exponent in the curve fit model: it makes inference robust to outliers and enables predicting more heavy-tailed production. We retain the OU variance structure as a convenient parametrization rather than a consequence of the stochastic differential equation.

Figure 7. A simulation from the statistical model with autocorrelated errors. Notice that the simulation tends to stay above or below the mean value (defined by the Arps curve) before regressing back. This kind of behavior is also seen in real-life wells.

The forward model, which we can simulate from, becomes

\begin{align} \log \left( y_i \right) &= \log \left( f\left(t_i; \boldsymbol{\theta} \right) \right) + \eta_i \\ \eta_i &= \exp(-\vartheta \tau_i) \eta_{i-1} + \sigma \sqrt{\frac{1 - \exp(-2 \vartheta \tau_i)}{2 \vartheta}} \epsilon_i \tag{5} \label{eqn:statistical_model} \\ \epsilon_i &\sim \operatorname{GN}(0, 1, \beta). \end{align}

The first equation defines the trend, the second the correlated noise process and the third the shock. The shock term $\epsilon_i$ is drawn with unit scale; the actual scale is applied as a multiplier outside, using $\alpha X \sim \operatorname{GN}(0, \alpha, \beta)$ for $X \sim \operatorname{GN}(0, 1, \beta)$. In the special case $\beta=2$ the GN reduces to a Normal distribution.

Initial condition. Starting at the first index $i=1$, what is the correct value of the first noise term $\eta_1$? To answer this, we assume that the long running, stationary distribution of $\eta$ is normal. This is a fair approximation as long as $\exp(-\vartheta \tau_i)$ is not too close to zero (in that case the stationary distribution is closer to a GN-distribution). This approximation only applies to the first data point, and is therefore often not very important.

Let $\phi := \exp(-\vartheta)$. The variance of GN is given by $\operatorname{Var} \left[ \operatorname{GN}(0, 1, \beta) \right] = \Gamma(3/\beta) / \Gamma(1/\beta)$, where $\Gamma$ is the Gamma function. The variance of the steady state can be found by taking variance of the recursion

\begin{align*} \operatorname{Var} \left[ \eta_i \right] &= \operatorname{Var} \left[ \phi^{\tau_i} \eta_{i-1} \right] + \operatorname{Var} \left[ \sigma \sqrt{\frac{1 - \phi^{2\tau_i}}{2 \vartheta}} \epsilon_i \right] \\ &= \phi^{2 \tau_i} \operatorname{Var} \left[ \eta_{i-1} \right] + \sigma^2 \frac{1 - \phi^{2\tau_i}}{2 \vartheta} \operatorname{Var} \left[ \epsilon_i \right] \\ &= \phi^{2 \tau_i} \operatorname{Var} \left[ \eta_{i-1} \right] + \sigma^2 \frac{1 - \phi^{2\tau_i}}{2 \vartheta} \frac{\Gamma(3/\beta)}{\Gamma(1/\beta)}. \end{align*}

At steady state we must have $\operatorname{Var} \left[ \eta_i \right] = \operatorname{Var} \left[ \eta_{i-1} \right] = \operatorname{Var} \left[ \eta \right]$. Substituting this into the equation above and solving yields

\begin{equation*} \operatorname{Var} \left[ \eta \right] = \frac{\sigma^2}{2 \vartheta} \frac{\Gamma(3/\beta)}{\Gamma(1/\beta)}. \end{equation*}

To summarize, for the initial condition we assume

\begin{equation} \label{eqn:statistical_model_initial} \tag{6} \eta_1 \sim N \left( 0, \sigma_1^2 \right) \text{ with } \sigma_1^2 = \frac{\sigma^2}{2 \vartheta} \frac{\Gamma(3/\beta)}{\Gamma(1/\beta)}. \end{equation}

Using Equations $\eqref{eqn:statistical_model}$ and $\eqref{eqn:statistical_model_initial}$ we can simulate from the model. One such simulation is shown in Figure 7. In other words: given parameters, we can generate data. We now discuss the inverse problem: determining parameters, given data.

Figure 8. The first figure shows the posterior mean curve. The second figure shows samples from the posterior distribution of $\boldsymbol{\theta}$ , plotted as curves. The third shows posterior predictive simulations: each line shows a simulation using parameters drawn from the posterior of $\boldsymbol{\Theta}$ . A slight weakness of the model is revealed: while history is mostly smooth with some downward outliers, the noise of the model captures this as higher overall noise. The fourth figure shows pointwise percentiles over the cumulative forecasts with uncertainty, estimating the distribution of EUR as $t \to \infty$ .

5.4 Parameter inference

This section explains how to infer parameters from the model defined by $\eqref{eqn:statistical_model}$ above. Collecting all parameters into $\boldsymbol{\Theta} := (\boldsymbol{\theta}, \vartheta, \sigma, \beta)$ and defining $z_i := \log (y_i)$ to ease notation, the log-likelihood becomes

\begin{align*} \log P(z_1, z_2, z_3, \ldots \mid \boldsymbol{\Theta}) &= \log \left( P(z_1 \mid \boldsymbol{\Theta}) \, P(z_2 \mid z_1, \boldsymbol{\Theta} \,) P(z_3 \mid z_2, \boldsymbol{\Theta} ) \, \ldots \right) \\ &= \log \left( P(z_1 \mid \boldsymbol{\Theta} ) \right) + \log \left( P(z_2 \mid z_1, \boldsymbol{\Theta}) \right) + \log \left( P(z_3 \mid z_2, \boldsymbol{\Theta}) \right) + \ldots \end{align*}

The log-likelihood of the $i$‘th term is

\begin{align*} \log P \left( z_i \mid z_{i-1}, \boldsymbol{\Theta} \right) &= \log \operatorname{GN} \left( z_i \mid \mu_i, \alpha_i, \beta \right) \\ \mu_i &= \log \left( f\left(t_i; \boldsymbol{\theta} \right) \right) + \exp(-\vartheta \tau_i) \eta_{i-1} \\ \alpha_i &= \sigma \sqrt{\frac{1 - \exp(-2 \vartheta \tau_i)}{2 \vartheta}}, \end{align*}

where $\eta_{i-1} = z_{i-1} - \log \left( f\left(t_{i-1}; \boldsymbol{\theta} \right) \right)$. Setting $\phi := \exp(-\vartheta)$, we can write:

\begin{equation*} \mu_i = \log \left( f\left(t_i; \boldsymbol{\theta} \right) \right) + \phi ^ {\tau_i} \left[ z_{i-1} - \log \left( f\left(t_{i-1}; \boldsymbol{\theta} \right) \right) \right] \end{equation*}

This gives us $\mu_i$ in terms of previous data, and is analogous to Prais–Winsten estimation in AR(1) regression.

Initial condition. The log-likelihood of the first term is

\begin{align*} \log P \left( z_1 \mid \boldsymbol{\Theta} \right) &= \log N \left( z_1 \mid \mu_1, \sigma_1^2 \right) \\ \mu_1 &= \log \left( f\left(t_1; \boldsymbol{\theta} \right) \right) \quad \sigma_1^2 = \frac{\sigma^2}{2 \vartheta} \frac{\Gamma(3/\beta)}{\Gamma(1/\beta)}. \end{align*}

Some general comments on parameter inference:

We use likelihood weights $W(t_i, \gamma) \, w_i$, as defined in $\eqref{eqn:dcaloss}$, to account for observation quality and recency. The $w_i$ factor was given in Section 2.1, and corresponds to either $\tau_i$ in the producing time transform, or simply $1$ in the calendar time transform.
The model parameters $\boldsymbol{\theta}$ become non-identifiable as $\exp(-\vartheta \tau_i) \to 1$. If the error term $\boldsymbol{\eta}$ does not revert back to the trend, but instead grows unbounded (like Brownian motion), then observed data can be explained by any decline curve plus a drifting error term. A prior should be put on $\vartheta$ to keep it away from zero.
There are two main approaches to infer parameters: maximum likelihood point estimation or sampling the posterior with MCMC. Parameter uncertainty leads to more meaningful probabilistic forecasts, but is slower and requires a MCMC library.

5.5 Forecasting

In this section we spell out some details about how to use the statistical model to forecast and how to obtain percentile and mean production rate curves. See also Equinor (2025).¹²

Probabilistic forecasting. Algorithm 1 demonstrates how to generate future production simulations. From simulation trajectories $\{\boldsymbol{p}_j\}_{j=1}^{S}$ we can extract pointwise percentiles and expected values for each future period $i$, as well as cumulative sums, EUR, etc.

Assuming $n$ observed periods, we generate $S$ simulations that each go $T$ periods into the future. Instead of sampling the posterior, it is possible to use the maximum a posteriori (MAP) estimate of parameters $\boldsymbol{\Theta}$. Instead of setting $\tau \gets 1$ (unit spacing) on the future grid, it is possible to use a weighted⁹ mean to estimate a $\tau$ and use that for future simulations. For instance, if $\tau \approx 2/3$ on historical data, we can use the same value of $\tau$ when forecasting, then interpolate onto a unit grid to forecast period by period.

Algorithm 1 (Posterior predictive simulation).

Require: Observed data $\{(t_i, y_i)\}_{i=1}^{n}$, posterior $p(\boldsymbol{\Theta} \mid \text{data})$; forecast horizon $T$; simulations $S$.
Ensure: Simulation trajectories $\boldsymbol{p}_1, \boldsymbol{p}_2, \ldots, \boldsymbol{p}_S$.

for $j = 1, 2, \ldots, S$ do (outer loop: posterior predictive draws)
1. Sample $\boldsymbol{\Theta}_j$ from the posterior (sample with MCMC, or use MAP)
2. $\eta \gets \log(y_n) - \log\!\big(f(t_n;\boldsymbol{\theta}_j)\big)$ (initial $\eta$-value)
3. $\tau \gets 1$ (future grid has unit spacing)
4. $\boldsymbol{p}_j \gets \boldsymbol{0} \in \mathbb{R}^{T}$ (allocate production vector)
5. for $k = 1, 2, \ldots, T$ do (inner loop: future periods)
  1. $\epsilon \sim \mathrm{GN}(0,\,1,\,\beta_j)$ (sample shock term)
  2. $\phi_j \gets \exp(-\vartheta_j \tau)$ (redefine for notational clarity)
  3. $\eta \gets \phi_j \eta + \sigma_j \sqrt{\left( 1 - \phi_j^2 \right) / \left( 2 \vartheta_j \right)} \, \epsilon$ (update noise $\eta$)
  4. $t \gets t_n + k \tau$ (compute $t_i$ on the forecast grid)
  5. $p_{j, n} \gets \tau \exp\left( \log f(t; \boldsymbol{\theta}_j) + \eta \right)$ (forecast in log-space, then exponentiate)

Percentile curves. Recall from Section 3.3 that a vertical shift in log-space of the Arps curve $f$ can be expressed as adding to the $\theta_1$ parameter:

\begin{equation*} \log f(t; \theta_1, \theta_2, \theta_3) + z = \log f(t; \theta_1 + z, \theta_2, \theta_3). \end{equation*}

At steady state, the noise process $\eta$ has variance given by $\eqref{eqn:statistical_model_initial}$. Assuming that $\eta$ is Normal, we can derive percentile curves and an expected value curve. Displaying P5, P50, P95 curves, along with the expected curve, is meaningful when engineers quality control production rate curves.¹⁰

Example 3 (P95 Arps curve). The P95 of a Normal distribution is located at $1.645$ standard deviations. Given MAP parameter estimates $(\hat{\theta}_1, \hat{\sigma}, \hat{\beta}, \hat{\vartheta})$ we can create a P95 curve:

\begin{equation*} \theta_1^{\text{P95}} = \hat{\theta}_1 + 1.645 \hat{\sigma} \sqrt{\frac{\Gamma(3/\hat{\beta})}{2 \hat{\vartheta} \Gamma(1/\hat{\beta}) }}. \end{equation*}

Notice that the P50 curve is simply $\theta_1^{\text{P50}} = \hat{\theta}_1$. Using MCMC, one could compute percentiles over $\{(\theta_1 + \eta)\}_{j=1}^S$, where $\eta$ is defined as in $\eqref{eqn:statistical_model_initial}$.

Example 4 (Expected Arps curve). Assume that $\eta \sim N \left( 0, \hat{\sigma}_1^2 \right)$, where $\hat{\sigma}_1^2$ is given by $\eqref{eqn:statistical_model_initial}$. Analogous to $\eqref{eqn:expected_value_arps}$, but with steady-state $\eta$ as the noise term, we obtain:

\begin{align*} p_i / \tau_i = \mathbb{E} \left[ \exp \left( \log \left( f\left(t_i; \hat{\boldsymbol{\theta}} \right) \right) + \eta \right) \right] &= f\left(t_i; \hat{\boldsymbol{\theta}} \right) \exp \left( \hat{\sigma}_1^2 / 2 \right) \\ &= \exp \left( \log \left( f(t_i; \hat{\boldsymbol{\theta}} ) \right) + \hat{\sigma}_1^2 / 2 \right). \end{align*}

In other words: in expectation, adding a Normal $\eta$ in log-space is equivalent to shifting $\theta_1$ by $\hat{\sigma}_1^2 / 2$. The expected pointwise production rate is determined by an Arps curve with parameter

\begin{equation*} \theta_1^{\text{EXP}} = \hat{\theta}_1 + \hat{\sigma}_1^2 / 2 = \hat{\theta}_1 + \frac{\hat{\sigma}^2 \Gamma(3/\hat{\beta})}{4 \hat{\vartheta} \Gamma(1/\hat{\beta})}, \end{equation*}

where in the last equality we expanded the definition of $\hat{\sigma}_1$.

6. Summary

Eighty years after Arps (1945),¹¹ DCA is still a powerful forecasting method despite its simplicity; or perhaps because of it. In this article we walked through the choices that turn the phrase “let’s use DCA” into a concrete model, and presented ideas on data preprocessing, re-parametrization of curve models, up-weighting recent observations and fitting in log-space. We presented two main directions: (1) curve fitting and (2) a statistical model to forecast with uncertainty. The individual ideas are far from novel, but together they form a baseline that is hard to beat.

Appendix: Stan Code

Below is Stan code for the statistical model with autocorrelation.

functions {

    // Log of the reparametrized Arps curve (theta-parametrization)
    real log_arps(real t, real theta1, real theta2, real theta3) {
        // log(1 + t * exp(x)) = log_sum_exp(0, log(t) + x)
        real log_term = log_sum_exp(0.0, log(t) + theta3 - theta2);
        return theta1 - theta2 - (1.0 + exp(-theta3)) * log_term;
    }

    // Log-density of the Generalized Normal distribution
    real generalized_normal_lpdf(real x, real mu, real alpha, real beta) {
        real z = abs(x - mu) / alpha;
        return log(beta) - log(2.0) - log(alpha) - lgamma(inv(beta)) - pow(z, beta);
    }

    // Conditional OU variance
    real ou_conditional_variance(real sigma_sq, real vartheta, real tau) {
        // 1 - exp(x) = -expm1(x)
        return sigma_sq * (-expm1(-2.0 * vartheta * tau)) / (2.0 * vartheta);
    }
}

data {
    int<lower=1> N;                // Number of observations
    vector<lower=0>[N] y;          // Production rates y_i
    vector<lower=0>[N] t;          // Midpoint times t_i
    vector<lower=0>[N] tau;        // Uptime fractions 0 < tau_i <= 1
    real<lower=0> gamma_hl;        // Half-life (same time units as t)
    vector[3] mu_theta;            // Prior mean for (theta1, theta2, theta3)
    matrix[3, 3] Sigma_theta_inv;  // Prior precision (inverse covariance)
}

transformed data {
    vector[N] log_y = log(y); // Pre-compute log production rates

    // Pre-compute likelihood weights
    vector[N] weights = tau .* exp((t - max(t)) * (log(2.0) / gamma_hl));
    weights *= sum(tau) / sum(weights); // Normalization
}

parameters {
    // Arps curve parameters (all unbounded)
    real theta1;
    real theta2;
    real theta3;

    // OU noise process parameters
    real<lower=0> vartheta;            // Mean-reversion rate
    real<lower=0> sigma;               // Diffusion scale of OU process
    real<lower=1, upper=2> beta_gn;    // GN shape: 2 = Normal, 1 = Laplace
}

model {

    // PRIORS
    // Multivariate normal prior on theta
    {
        [theta1, theta2, theta3]' ~ multi_normal_prec(mu_theta, Sigma_theta_inv);
    }

    // Prior on vartheta: must stay away from 0 for identifiability
    // As vartheta -> 0, the OU process becomes Brownian motion and theta
    // becomes non-identifiable.
    vartheta ~ normal(1.5, 0.5); // exp(-vartheta) not too close to 0 or 1

    // Weakly informative priors on sigma and beta_gn
    sigma ~ lognormal(-1, 1);
    (beta_gn - 1) ~ beta(2, 1); // beta = 2 more likely than beta = 1

    // LIKELIHOOD
    {
        // --- First observation (initial condition) ---
        real log_f_prev = log_arps(t[1], theta1, theta2, theta3);
        real var_stationary = (square(sigma) / (2.0 * vartheta))
        * tgamma(3.0 / beta_gn) / tgamma(1.0 / beta_gn);
        real sigma1 = sqrt(var_stationary);

        target += weights[1] * normal_lpdf(log_y[1] | log_f_prev, sigma1);

        // --- Subsequent observations ---
        for (i in 2:N) {
            real log_f_i = log_arps(t[i], theta1, theta2, theta3);

            // OU discretization
            real phi_i = exp(-vartheta * tau[i]);

            // Conditional mean: Prais-Winsten form
            // mu_i = log f(t_i) + phi_i * (log y_{i-1} - log f(t_{i-1}))
            real eta_prev = log_y[i - 1] - log_f_prev;
            real mu_i = log_f_i + phi_i * eta_prev;

            // Conditional std dev from OU transition
            real cond_var = ou_conditional_variance(square(sigma), vartheta, tau[i]);
            real alpha_i = sqrt(cond_var);

            // weighted GN log-likelihood
            target += weights[i]
            * generalized_normal_lpdf(log_y[i] | mu_i, alpha_i, beta_gn);

            log_f_prev = log_f_i;
        }
    }
}

It is simultaneously true that sophisticated methods lead to better results on many problems too. ↩
This article is also guilty of not having a strong experimental section. Creating such a section without high-quality, open data is hard. The reader is referred to Odland & Hollund (2026) for some experimental results on these ideas. This is not a peer reviewed paper, and I am advocating for simple DCA—the antithesis of published papers arguing for fancy models. ↩
The midpoint rule underestimates for convex, decreasing curves. ↩
The final factor $\log ( 1 + t e^{\theta_3 - \theta_2} )$ can overflow if $\theta_3 - \theta_2$ is large. In a practical implementation, one should use the log-sum-exp trick or apply the identity $\log (1 + t e^x) = x + \log (t + e^{-x})$ when $x$ is large. ↩
Some literature insists that certain physical regimes, e.g. transient flow in low-permeability systems or gas reservoirs where compressibility changes with pressure, correspond to certain values of $h$ in the Arps curve. But this has no bearing on forecasting: fixing $h$ based on estimated reservoir characteristics instead of leaving it open as a much-needed degree of freedom when fitting to data almost surely lead to worse forecasts. The physical justifications are often either post-hoc explanations for why certain fitted values of $h$ appear plausible, or circular reasoning. ↩
It is sometimes suggested that fitting cumulative production, or rate as a function of cumulative production, is an alternative to fitting rates as a function of time. However, the error structure such a model imposes is difficult to justify, since errors accumulate across observations. I have yet to encounter a clear empirical or theoretical case for any advantage over fitting rates. I take no issue with plotting these functions. ↩
A more theoretically appealing approach is to do partial pooling of parameters with a hyper-prior; see Gelman & Hill (2006). The open-source ADCA Python package (Equinor, 2025) instead performs a simple two stage process with a pilot fit, since it is faster and does not require Markov Chain Monte Carlo (MCMC). ↩
In curve fitting, we can choose to fit on log-scale or not. In the statistical framework, we must work on log scale to ensure positive predictions if errors are Normal. ↩
Using the same exponentially decaying weights as we used elsewhere in this article. ↩
Beware that the sum of percentiles is not the percentile of a sum. We cannot use the independent, pointwise P95 production rates to estimate the P95 of the total production. The probability of a well producing at the 95-th percentile in every future period is vanishingly small. To compute the percentile of the sum, use Monte-Carlo simulation as sketched in Algorithm 1. ↩
Arps, J. J.(1945). “Analysis of Decline Curves.” Transactions of the AIME, 160, 228–247. doi:10.2118/945228-G ↩↩
Equinor (2025). decline-curve-analysis [Source Code]. https://github.com/equinor/decline-curve-analysis ↩↩
Green, K. C. & Armstrong, J. S.(2015). “Simple versus complex forecasting: The evidence.” Journal of Business Research, 68, 1678–1685. ↩
Fama, E. F. & French, K. R.(2010). “Luck versus Skill in the Cross-Section of Mutual Fund Returns.” The Journal of Finance, 65(5), 1915–1947. doi:10.1111/j.1540-6261.2010.01598.x ↩
Yan, Z. (2022). “Simplicity is An Advantage but Sadly Complexity Sells Better.” eugeneyan.com. https://eugeneyan.com/writing/simplicity/ ↩
Bratvold, R. B., Mohus, E., Petutschnig, D. & Bickel, E. (2020). “Production Forecasting: Optimistic and Overconfident - Over and Over Again.” SPE Reservoir Evaluation & Engineering, 23, 799–810. ↩
Odland, T. & Hollund, K. U.(2026). “Automated Decline Curve Analysis - DCA from a Data Science Perspective.” EAGE Annual Conference & Exhibition, 1–5. doi:10.3997/2214-4609.202639021 ↩
Lee, S. Y. & Mallick, B. (2021). “Bayesian Hierarchical Modeling: Application Towards Production Results in the Eagle Ford Shale of South Texas.” Sankhya B. doi:10.1007/s13571-020-00245-8 ↩
Norwegian Offshore Directorate. Open Data from the Norwegian Offshore Directorate. https://factpages.sodir.no. Licensed under the Norwegian Licence for Open Government Data (NLOD) 2.0. ↩
Holý, V. & Tomanová, P. (2025). “Estimation of Ornstein-Uhlenbeck Process Using Ultra-High-Frequency Data with Application to Intraday Pairs Trading Strategy.” arXiv:1811.09312. doi:10.48550/arXiv.1811.09312 ↩
Gelman, A. & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press. ↩
Odland, T. (2025). “A Recipe for Curve Fitting.” https://tommyodland.com/articles/2025/a-recipe-for-curve-fitting/ ↩

Minimal models and dumb deployments

Tommy Odland — Fri, 11 Sep 2026 00:00:00 +0200

The most common mistake I encounter in practical data science work is a certain type of unwarranted complexity in model choice and deployment. All organizations, but especially those that are not technology companies, should avoid this if they can. As Dijkstra said in the 70s: “The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible.“

Let’s admit it: when it comes to data science, many organizations have underwhelming track records. Startups and FAANG-like companies might be different, but in the typical organization, initiatives tend to become messy. Data science is not unique in this regard: the average corporate codebase, data warehouse, internal documentation, etc. is often not that great either.

Still, data science is especially plagued by complexity because it sits on top of, and depends on, other systems. Bad design, conceptual degradation, backwards-incompatible changes, and lack of documentation tends to accumulate and cause problems. The paper Machine Learning: The High Interest Credit Card of Technical Debt lists some concrete reasons why data science is hard in practice.

So how should an organization approach its first data science initiatives?

Avoid complexity

Organizations often trip up because they overcomplicate matters; they focus on the wrong things at the wrong time. Some inexperienced organizations immediately go the “best practice” route: multiple deployment environments, elaborate monitoring, automatic re-training, API-endpoints, clusters, etc. Often this is not needed.

The same faulty thinking tends to afflict the models too. On regression tasks, boosting models with thousands of base learners is the default choice. Hyperparameters (which are often in conflict with each other or are slight variations of the same basic idea) are all at once tuned uncritically by fancy Bayesian optimization algorithms that take ages to run. Dozens or hundreds of input features are used, whether they are meaningfully predictive or not.

The resulting model is slow to train, hard to explain and difficult to maintain. This leads to mountains of busywork for data scientists.

Aim for simplicity

The average organization should choose a ridiculously simple model as the starting point. On regression tasks, this means a linear model or a decision tree. Then it should be “deployed” in the dumbest way possible. If a batch run on a weekly schedule is good enough, then just run it manually from a laptop. If real-time or a nightly run is needed, convert the model to an SQL statement. Follow “good enough” practices, but do nothing more. Then give it a few months and see if it delivers any value to the organization. This is commandment 5 (Keep it simple) in my ten commandments of data science.

Let’s illustrate this with the California housing dataset. The goal is to use eight features to predict house prices. For reference, a dummy model that predicts the mean achieves an out-of-sample Root Mean Squared Error (RMSE) of 1.17, while a boosting model (widely considered the best choice on tabular data) achieves 0.64. These models give realistic upper and lower bounds on the RMSE.

Linear regression

Linear regression achieves an RMSE of 0.83, which is two thirds of the way between the dummy and the boosting model. The Lasso variant selects four out of eight features. The resulting model can be deployed anywhere, for instance as an SQL expression in a database:

SELECT
  ((0.393414 * MedInc) +
   (0.016651 * HouseAge) +
   (-0.003222 * AveOccup) +
   (-0.021127 * Latitude) +
   0.797172) AS predicted_price
FROM california_housing_data;

This model is super interpretable, so the business side of the organization will be delighted when they inevitably wish to understand the model. Predictive power can be improved by adding non-linear transformations of the features and interaction terms, but we won’t pursue that here.

Decision trees

Do you need interactions and/or non-linearities? But you still want an interpretable model that can be deployed anywhere? Use a simple decision tree! The tree below achieves an RMSE of 0.84, which is very similar to the linear model’s RMSE, and it only uses two features. Here is the decision tree written in SQL:

SELECT
CASE WHEN MedInc <= 5.04 THEN
  CASE WHEN MedInc <= 3.07 THEN
    1.36
  ELSE
    CASE WHEN AveOccup <= 2.37 THEN
      2.79
    ELSE
      1.88
    END
  END
ELSE
  CASE WHEN MedInc <= 6.82 THEN
    CASE WHEN AveOccup <= 2.74 THEN
      3.39
    ELSE
      2.56
    END
  ELSE
    4.22
  END
END
AS predicted_price
FROM california_housing_data;

Like the regression model, the tree is extremely interpretable. Below it is shown as a figure.

Two approaches to improvement

Broadly speaking there are two approaches to improving any situation. One is to start from an abstract (and perhaps unattainable) ideal and work backwards, e.g. ask yourself “How does Google do data science?” and then try to emulate it. The other approach is to start where you are, assess your resources and skills honestly, and ask “What is a small step I can take to improve?” Both methods have their place, but the second one is often more concrete and less risky. Practical project experience helps a lot in balancing the two. Many data science projects fail because complexity is “spent” on the wrong things.

Sometimes a simple model with a dumb deployment is not only a good first step, but the best place to end up too. It’s fast to build, requires little upkeep, is interpretable and has few data dependencies. On many problems performance will match a complex approach. It might be unglamorous and unfashionable, but it will quietly keep working—and boy can it save you money and time.

We’ll end with another Dijkstra quote, my personal favorite: “Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.“

Notes on Ensemble Smoothers

Tommy Odland — Sat, 01 Aug 2026 00:00:00 +0200

Ensemble smoother algorithms are used to approximate a posterior distribution. Given a prior over parameters, a forward model (e.g. a petroleum reservoir model) and real-world observations: what is the posterior over the parameters? Ensemble-based algorithms describe the posterior with a set of samples, and the most common use-case is when the forward model is black-box and expensive to evaluate.

The notes are organized as follows: first we examine the Gauss-linear case, where we can compute an analytical posterior in arbitrarily high dimensions. The equation for the Gaussian posterior is the starting point for all ensemble methods. We derive the Ensemble Smoother (ES)—an algorithm that retains some properties of the Gauss-linear case. Finally we examine EnIF, which is similar to ES but works with precision matrices rather than covariances, and leans on the statistical theory of high-dimensional estimation.

Introduction
The Gaussian posterior
- Exact posterior distribution $p(\boldsymbol{x} \mid \boldsymbol{d})$
The Ensemble Smoother
- Computation
The Ensemble Information Filter
- Computation

$$ \newcommand{\R}{\mathbb{R}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\N}{\mathbb{N}} $$

Introduction

The Ensemble Kalman Filter was introduced by Evensen (1994)⁹ and the Ensemble Smoother was introduced by Van Leeuwen and Evensen (1996)¹⁰. Four years later, in Burgers et al. (1998)¹¹, it was shown that the original update equation was wrong—the covariance did not match in the Gauss-linear case.¹

New methods were proposed to deal with non-linearity. One such variant is the Iterative Ensemble Smoother (IES) of Chen and Oliver (2012¹²; 2013¹³). The idea behind IES is to move towards the maximum of the posterior in several small steps instead of a single large one, like a reduced step length optimization method. Another variant is the Ensemble Smoother with Multiple Data Assimilation (ESMDA), proposed in Emerick and Reynolds (2013)¹⁴. It also makes several small steps, but shortens step length implicitly by increasing the covariance matrix of the observations. Either way, the core idea is that linearizing, taking a small step, re-linearizing, taking another step, etc. is better than one linearization followed by one large step.

In high dimensions with few samples, empirical correlations are mostly noise and very little signal. This phenomenon is called spurious correlations, and all the methods above suffer from it. Many procedures have been proposed to alleviate it, such as setting all correlations that are below $3 / \sqrt{r}$ to zero (Vossepoel et al., 2025)¹⁵. Another idea is to look towards the statistics of high-dimensional models, see for instance Hastie et al. (2009, Ch. 18)¹⁶. This is what Lunde (2025)¹⁷ does in the Ensemble Information Filter (EnIF), which uses graph properties to estimate sparse precision matrices and high-dimensional regression to estimate a sparse linearization of the forward model.

The papers above can be a bit tough to read: notation can be a bit inconsistent, some proofs are not as detailed as I’d like, sometimes there is no code to look at, etc. That is why I have decided to write these notes; maybe they can help clear up some confusion. If nothing else, you’ll get a second perspective. You can find well-documented and production-ready Python code for many of these algorithms at the GitHub repository github.com/equinor/iterative_ensemble_smoother/.

The Gaussian posterior

With a Gaussian prior, a Gaussian noise model and a linear forward model, we’re in the happy Gauss-linear case where an analytical expression for the posterior can be derived. In this section we establish notation and illustrate several expressions for the posterior.

Let $\boldsymbol{x} \in \R^m$ be a vector of parameters. The forward model is denoted as $h: \R^m \to \R^n$, and applying it to the $m$ parameters returns a vector of $n$ responses $\boldsymbol{y} \in \R^n$.

The observation vector $\boldsymbol{d} \in \R^n$ is equal to the response plus a random noise component $\boldsymbol{\epsilon}_d$, which is often called the observation noise or measurement noise. The full equation is then $\boldsymbol{d} = h(\boldsymbol{x}) + \boldsymbol{\epsilon}_d$, where $\boldsymbol{\epsilon}_d \sim \mathcal{N}(\boldsymbol{0}, \Sigma_{\epsilon})$.

Think for instance of a petroleum reservoir model. The parameters $\boldsymbol{x}$ are uncertain static properties of the subsurface, e.g., porosity. The forward model $h$ is a reservoir flow simulator. It solves the fluid flow equations over the grid and reports predicted production data $\boldsymbol{y}$, for instance oil production at a well. A single evaluation of $h$ may take hours of compute, and we only have access to its inputs and outputs. The match is never exact, so the real-world observation $\boldsymbol{d}$ departs from the simulator response by a noise term $\boldsymbol{\epsilon}_d$. The inverse problem, called history matching, is: given the observed production $\boldsymbol{d}$, how can we update our knowledge of the parameters $\boldsymbol{x}$? Characterizing the posterior distribution of $\boldsymbol{x}$ given $\boldsymbol{d}$ is the subject of the rest of these notes.

Exact posterior distribution $p(\boldsymbol{x} \mid \boldsymbol{d})$

In this section we assume that the prior distribution over parameters $p(\boldsymbol{x})$ is multivariate Gaussian, and that the forward model $h$ is linear. Then $\boldsymbol{d} = H \boldsymbol{x} + \boldsymbol{\epsilon}_d = \boldsymbol{y} + \boldsymbol{\epsilon}_d$, where $H \in \R^{n \times m}$ is a matrix. The posterior distribution $p(\boldsymbol{x} \mid \boldsymbol{d})$ is also multivariate Gaussian.

In words, we will answer “Suppose I have a prior belief over parameters $p(\boldsymbol{x})$ and I have observed an observation $\boldsymbol{d}$—how can I update my prior to a posterior?”

Remark 1 (Response covariance). Recall from Petersen and Pedersen (2012, §6.2)¹⁸ that $\operatorname{Cov}\left[A\boldsymbol{x}, B\boldsymbol{y}\right] = A \operatorname{Cov}\left[\boldsymbol{x}, \boldsymbol{y}\right] B^T$ when $\boldsymbol{x}$ and $\boldsymbol{y}$ are random variables. When both arguments are identical, we’ll use a shorthand notation: $\operatorname{Cov}\left[\boldsymbol{x}\right] := \operatorname{Cov}\left[\boldsymbol{x}, \boldsymbol{x}\right] = \operatorname{Var}\left[\boldsymbol{x}\right]$. Similarly, when there is no ambiguity we’ll simply write $\Sigma_x$ instead of $\Sigma_{xx}$ for the covariance of $\boldsymbol{x}$.

In both equations below we’ll use the fact that $\boldsymbol{\epsilon}_d$ is independent of $\boldsymbol{x}$. The covariance in $\boldsymbol{d}$ is the covariance of $\boldsymbol{x}$, propagated through the linear map $H$, plus the covariance of the observation noise $\boldsymbol{\epsilon}_d$.

\begin{align*} \Sigma_{d} &= \operatorname{Cov}\left[ \boldsymbol{d} \right] = \operatorname{Cov}\left[ H\boldsymbol{x} + \boldsymbol{\epsilon}_d \right] \\ &= \operatorname{Cov}\left[ H\boldsymbol{x} \right] + \operatorname{Cov}\left[ \boldsymbol{\epsilon}_d \right] = H \operatorname{Cov}\left[\boldsymbol{x}, \boldsymbol{x}\right] H^T + \Sigma_{\epsilon} \\ &= H \Sigma_x H^T + \Sigma_{\epsilon}. \end{align*}

Similarly, the cross-covariance can be expressed as:

\begin{align*} \Sigma_{xd} &= \operatorname{Cov}\left[ \boldsymbol{x}, \boldsymbol{d} \right] = \operatorname{Cov}\left[ \boldsymbol{x}, H\boldsymbol{x} + \boldsymbol{\epsilon}_d \right] \\ &= \operatorname{Cov}\left[ \boldsymbol{x}, \boldsymbol{x} \right] H^T = \Sigma_x H^T \end{align*}

Note also that $\Sigma_{xy} = \operatorname{Cov}\left[ \boldsymbol{x}, \boldsymbol{y} \right] = \operatorname{Cov}\left[ \boldsymbol{x}, H\boldsymbol{x} \right] = \operatorname{Cov}\left[\boldsymbol{x}, \boldsymbol{x}\right] H^T = \Sigma_{xd}$.

Suppose the prior over the parameters $\boldsymbol{x}$ and the observation $\boldsymbol{d}$, conditioned on the parameters, are given by multivariate Gaussians:

\begin{align} \label{eqn:model_def_gaussian} \tag{1} p(\boldsymbol{x}) &= \mathcal{N}\left( \boldsymbol{x} \mid \boldsymbol{\mu}_x, \Sigma_x \right) \\ p(\boldsymbol{d} \mid \boldsymbol{x}) &= \mathcal{N}( \boldsymbol{d} \mid H\boldsymbol{x}, \Sigma_{\epsilon}) \end{align}

Then the posterior distribution over parameters, conditioned on the response, is also a multivariate Gaussian $p(\boldsymbol{x} \mid \boldsymbol{d}) = \mathcal{N} (\boldsymbol{\mu}_{x \mid d}, \Sigma_{x \mid d})$. The posterior mean and covariance are:

\begin{align} \boldsymbol{\mu}_{x \mid d} &= \Sigma_{x \mid d} \left( H^T \Sigma_\epsilon^{-1} \boldsymbol{d} + \Sigma_x^{-1} \boldsymbol{\mu}_x \right) \label{eqn:posterior_mean_precision} \tag{2} \\ \Sigma_{x \mid d} &= \left( \Sigma_x^{-1} + H^T \Sigma_\epsilon^{-1} H \right)^{-1} \label{eqn:posterior_cov_precision} \tag{3} \end{align}

The equations above are well known, and can for instance be found in Bishop (2006, §2.3.3)¹⁹.² In the equations above, the posterior mean $\boldsymbol{\mu}_{x \mid d}$ is expressed in terms of precision matrices $\Sigma_x^{-1}$ and $\Sigma_\epsilon^{-1}$ rather than covariance matrices $\Sigma_x$ and $\Sigma_\epsilon$.

Remark 2 (Posterior covariance). We show the Schur complement form of the posterior covariance $\Sigma_{x \mid d}$. Starting with equation $\eqref{eqn:posterior_cov_precision}$, we first use the Woodbury identity, then the results from Remark 1:

\begin{align*} \Sigma_{x \mid d} &= \left( \Sigma_x^{-1} + H^T \Sigma_\epsilon^{-1} H \right)^{-1} \\ &= \Sigma_x - \Sigma_x H^T \left( \Sigma_\epsilon + H \Sigma_x H^T \right)^{-1} H \Sigma_x \\ &= \Sigma_x - \Sigma_{xd} \Sigma_d^{-1} \Sigma_{dx} \end{align*}

The Kalman gain form is an alternative way to express the update equations above, and we’ll spend the rest of this section deriving it. It’s more commonly used in the Kalman filter literature, and the update is given by:

\begin{align} \boldsymbol{\mu}_{x \mid d} &= \boldsymbol{\mu}_{x} + K (\boldsymbol{d} - H \boldsymbol{\mu}_{x}) \label{eqn:posterior_mean_kalman} \tag{4} \\ K &:= \Sigma_x H^T \left( \Sigma_\epsilon + H \Sigma_x H^T \right)^{-1} = \Sigma_{xd} \Sigma_d^{-1} \label{eqn:kalman_gain} \tag{5} \end{align}

In the equation above, $H \boldsymbol{\mu}_{x}$ is the predicted mean, $(\boldsymbol{d} - H \boldsymbol{\mu}_{x})$ is the innovation term and $K$ is the Kalman gain. In words: the posterior equals the prior, plus a linear combination of the innovation (residuals).

We will now derive the Kalman update in equation $\eqref{eqn:posterior_mean_kalman}$. The first step is to show that $K = \Sigma_{x \mid d}H^T \Sigma_{\epsilon}^{-1}$. To see this, we start by assuming that both expression are equal:

\begin{equation} \label{eqn:push_through_kalman} \tag{6} \Sigma_{x \mid d}H^T \Sigma_{\epsilon}^{-1} = K = \Sigma_x H^T \left( \Sigma_\epsilon + H \Sigma_x H^T \right)^{-1} \end{equation}

We right-multiply both sides with $\left( \Sigma_\epsilon + H \Sigma_x H^T \right)$, then left-multiply both sides with $\Sigma_{x \mid d}^{-1}$ and finally expand both sides to show that they both equal $H^T + H^T \Sigma_\epsilon^{-1} H \Sigma_x H^T$.³

The second step is to realize that, using Remark 2, we can write

\begin{align*} \Sigma_{x \mid d}\Sigma_{x}^{-1} &= \Sigma_{x \mid d} \left( \Sigma_{x \mid d}^{-1} - H^T \Sigma_{\epsilon}^{-1} H \right) \\ &= I - \Sigma_{x \mid d} H^T \Sigma_{\epsilon}^{-1} H = I - KH. \end{align*}

With these two results in hand, we start with equation $\eqref{eqn:posterior_mean_precision}$ and show that it can indeed be transformed into equation $\eqref{eqn:posterior_mean_kalman}$. The update equation for the mean becomes

\begin{align*} \boldsymbol{\mu}_{x \mid d} &= \Sigma_{x \mid d} \left( H^T \Sigma_\epsilon^{-1} \boldsymbol{d} + \Sigma_x^{-1} \boldsymbol{\mu}_x \right) \\ &= \Sigma_{x \mid d} H^T \Sigma_\epsilon^{-1} \boldsymbol{d} + \Sigma_{x \mid d} \Sigma_x^{-1} \boldsymbol{\mu}_x \\ &= K \boldsymbol{d} + (I - KH) \boldsymbol{\mu}_x = \boldsymbol{\mu}_x + K (\boldsymbol{d} - H\boldsymbol{\mu}_x ) \end{align*}

In summary then, the Kalman-style update equations are:

\begin{align*} \boldsymbol{\mu}_{x \mid d} &= \boldsymbol{\mu}_x + K (\boldsymbol{d} - H\boldsymbol{\mu}_x ) \\ \Sigma_{x \mid d} &= (I - KH) \Sigma_x \end{align*}

We have seen that there are different ways to express the update equations: the precision form of equation $\eqref{eqn:posterior_cov_precision}$, the Schur form of Remark 2 and the Kalman form in equation $\eqref{eqn:posterior_mean_kalman}$. Which one should we use? It depends on what we wish to show or prove, or how we want to structure our computations. For instance, the Ensemble Smoother—which we discuss next—uses a Kalman style update with covariances.

The Ensemble Smoother

The Ensemble Smoother (ES) describes the posterior with an ensemble of realizations (i.e., a set of samples) instead of $\boldsymbol{\mu}_{x \mid d}$ and $\Sigma_{x \mid d}$. First we draw $r$ realizations $\{\boldsymbol{x}_j\}_{j=1}^{r}$ from the prior distribution $p(\boldsymbol{x})$. Then we propagate them through the forward model $h$ and obtain $\boldsymbol{y}_j = h(\boldsymbol{x}_j)$ for each realization $j=1, \ldots, r$. Finally, we perform data assimilation; a Kalman-style update that transforms the prior ensemble to the posterior ensemble by conditioning on the observation.

The upside of ES is that we don’t need knowledge of the forward model $h$ to compute a posterior; $h$ can be a black-box function. The downside is that the update equation is derived assuming (1) a linear $h$, (2) Gaussian distributions and (3) infinitely many realizations. ES won’t actually sample the true, analytical posterior except in the happy case when conditions (1), (2) and (3) are all satisfied. Instead we approximate it.⁴ Since the forward model is expensive to compute, the number of realizations $r$ is much smaller than the number of parameters $m$ and responses $n$. This leads to poor estimates of covariances due to spurious correlations.

We will now derive the update equation for ES, which computes the posterior realizations given the prior realizations. First, some notation: let $X \in \R^{m \times r}$ be a matrix with $r$ realizations (ensemble members); one per column.⁵ Let $Y = h(X) = [ h(\boldsymbol{x}_1) \mid h(\boldsymbol{x}_2) \mid \cdots \mid h(\boldsymbol{x}_r)] \in \R^{n \times r}$ be a matrix of responses, where $h$ is applied to each column.

Statement 1. The matrix form of the ES update equation, which updates every realization (column) in $X$, is

\begin{equation} \label{eqn:ensemble_smoother_update} \tag{7} X_{\text{posterior}} = X_{\text{prior}} + \hat{\Sigma}_{xy} ( \Sigma_\epsilon + \hat{\Sigma}_y )^{-1} (\boldsymbol{d} \boldsymbol{1}^T - (Y + E)) \end{equation}

where each column of $E$ is independently sampled observation $\boldsymbol{\epsilon}_d$. In other words, $E_{\cdot, j} \sim \mathcal{N}(\boldsymbol{0}, \Sigma_\epsilon)$ for every column $j$. The vector $\boldsymbol{1}$ is a vector of $r$ ones, so $\boldsymbol{d} \boldsymbol{1}^T$ is a matrix with the observation $\boldsymbol{d}$ repeated in each of the $r$ columns. The matrices $\hat{\Sigma}_{xy}$ and $\hat{\Sigma}_y$ are estimates of the cross-covariance and the covariance. The most common estimate for, e.g., $\hat{\Sigma}_{xy}$ is the empirical cross covariance matrix $\bar{X}\bar{Y}^T / (r-1) \in \R^{m \times n}$, where $\bar{X}$ is a centered $X$ in which every parameter (row) has its mean subtracted.⁶ $\bar{Y}$ is a centered $Y$ and $\hat{\Sigma}_{y}$ is computed similarly.

We’ll spend the rest of this section showing that equation $\eqref{eqn:ensemble_smoother_update}$ is the correct update equation in the following sense: when we use it on a linear model and the number of realizations $r$ goes to infinity, the posterior mean and posterior covariance matches the analytical posterior.

To derive equation $\eqref{eqn:ensemble_smoother_update}$, we start by examining the Gaussian update equation for the mean:

\begin{align*} \boldsymbol{\mu}_{x \mid d} &= \boldsymbol{\mu}_x + K (\boldsymbol{d} - H\boldsymbol{\mu}_x ) \\ &= \boldsymbol{\mu}_x + \Sigma_x H^T \left( \Sigma_\epsilon + H \Sigma_x H^T \right)^{-1} (\boldsymbol{d} - H\boldsymbol{\mu}_x ) \end{align*}

We use the results from Remark 1 and substitute $\Sigma_x H^T = \Sigma_{xy}$ and $H \Sigma_x H^T = \operatorname{Cov}\left[ H\boldsymbol{x} \right] = \operatorname{Cov}\left[ \boldsymbol{y} \right] = \Sigma_y$. This gives us Kalman gain in the form $K = \Sigma_{xy} \left( \Sigma_\epsilon + \Sigma_y \right)^{-1}$.

The update equation for each realization $j$, based on equation $\eqref{eqn:ensemble_smoother_update}$, is given by

\begin{equation*} \boldsymbol{x}^{j}_{\text{posterior}} = \boldsymbol{x}^{j}_{\text{prior}} + K(\boldsymbol{d} - (\boldsymbol{y}^j + \boldsymbol{\epsilon}_d^j)), \end{equation*}

where $\boldsymbol{\epsilon}_d^j \sim \mathcal{N}(\boldsymbol{0}, \Sigma_\epsilon)$. To see why this is the correct update and why we have to perturb the response $\boldsymbol{y}^j$ with observation noise $\boldsymbol{\epsilon}_d^j$, we have to prove that in the Gauss-linear case, this update equation recovers the same posterior mean and posterior covariance as the analytical equations in $\eqref{eqn:posterior_mean_precision}$ and $\eqref{eqn:posterior_cov_precision}$. To this end, suppose that $h$ is linear, so $\boldsymbol{y}^j = H \boldsymbol{x}^{j}_{\text{prior}}$.

We update each realization with the equation

\begin{align*} \boldsymbol{x}_{\text{posterior}} &= \boldsymbol{x}_{\text{prior}} + K(\boldsymbol{d} - (H \boldsymbol{x}_{\text{prior}} + \boldsymbol{\epsilon}_d)) \\ \boldsymbol{x}_{\text{prior}} &\sim \mathcal{N}(\boldsymbol{\mu}_x, \Sigma_x) \\ \boldsymbol{\epsilon}_d &\sim \mathcal{N}(\boldsymbol{0}, \Sigma_\epsilon), \end{align*}

where $\boldsymbol{d}$ is no longer a random variable, but an actual real-world observation. The residual term $(\boldsymbol{d} - (H \boldsymbol{x}_{\text{prior}} + \boldsymbol{\epsilon}_d))$ is the actual value minus the predicted value.

First we show that the posterior mean equals equation $\eqref{eqn:posterior_mean_kalman}$:

\begin{align*} \mathbb{E} \left[ \boldsymbol{x}_{\text{posterior}} \right] &= \mathbb{E} \left[ \boldsymbol{x}_{\text{prior}} + K(\boldsymbol{d} - (H \boldsymbol{x}_{\text{prior}} + \boldsymbol{\epsilon}_d)) \right] \\ &= \mathbb{E} \left[ \boldsymbol{x}_{\text{prior}} \right] + K \left( \mathbb{E} \left[ \boldsymbol{d} \right] - \mathbb{E} \left[ H \boldsymbol{x}_{\text{prior}} \right] - \mathbb{E} \left[ \boldsymbol{\epsilon}_d \right] \right) \\ &= \boldsymbol{\mu}_x + K \left( \boldsymbol{d} - H \mathbb{E} \left[ \boldsymbol{x}_{\text{prior}} \right] - \boldsymbol{0} \right) \\ &= \boldsymbol{\mu}_x + K(\boldsymbol{d} - H \boldsymbol{\mu}_x ) \end{align*}

Second we show that the posterior covariance equals equation $\eqref{eqn:posterior_cov_precision}$. More specifically, we show it to be equal to the Schur complement form derived in Remark 2. Below we use the fact that $K\boldsymbol{d}$ is constant and that $\operatorname{Cov} \left[ \boldsymbol{x}_{\text{prior}}, \boldsymbol{\epsilon}_d \right] = 0$ when we distribute the covariance over the terms.

\begin{align*} \operatorname{Cov} \left[ \boldsymbol{x}_{\text{posterior}} \right] &= \operatorname{Cov} \left[ \boldsymbol{x}_{\text{prior}} + K( \boldsymbol{d} - (H \boldsymbol{x}_{\text{prior}} + \boldsymbol{\epsilon}_d) ) \right] \\ &= \operatorname{Cov} \left[ (I - KH) \boldsymbol{x}_{\text{prior}} + K \boldsymbol{d} - K \boldsymbol{\epsilon}_d \right] \\ &= \operatorname{Cov} \left[ (I - KH) \boldsymbol{x}_{\text{prior}} \right] + \operatorname{Cov} \left[ -K \boldsymbol{\epsilon}_d \right] \\ &= (I - KH) \Sigma_{x} (I - KH)^T + K \Sigma_{\epsilon} K^T \\ &= \Sigma_{x} - \Sigma_{x} H^T K^T - K H \Sigma_{x} + K H \Sigma_{x} H^T K^T + K \Sigma_{\epsilon} K^T \\ &= \Sigma_{x} - \underbrace{\Sigma_{x} H^T}_{\Sigma_{xd}} K^T - K \underbrace{H \Sigma_{x}}_{\Sigma_{dx}} + K \underbrace{\left( H \Sigma_{x} H^T + \Sigma_{\epsilon} \right) }_{\Sigma_{d}} K^T \\ &= \Sigma_{x} - \Sigma_{xd} K^T - K \Sigma_{dx} + K \Sigma_{d} K^T \end{align*}

The equalities shown in underbraces were all derived in Remark 1. From equation $\eqref{eqn:kalman_gain}$, we observe that $K \Sigma_{d} = (\Sigma_{xd} \Sigma_{d}^{-1}) \Sigma_{d} = \Sigma_{xd}$. The last term, $K \Sigma_{d} K^T$, can therefore be re-written as $\Sigma_{xd} K^T$ and cancels the second term. We are left with

\begin{equation*} \operatorname{Cov} \left[ \boldsymbol{x}_{\text{posterior}} \right] = \Sigma_{x} - K \Sigma_{dx} = \Sigma_{x} - \Sigma_{xd} \Sigma_{d}^{-1} \Sigma_{dx} = \Sigma_{x \mid d}, \end{equation*}

where the last equality follows from Remark 2.

In summary, we have shown that if we use equation $\eqref{eqn:ensemble_smoother_update}$ to update the ensemble, then the resulting posterior mean and covariance over the realizations matches the analytical posterior in the Gauss-linear case.

Computation

To implement ES we must compute equation $\eqref{eqn:ensemble_smoother_update}$ given in Statement 1. Substituting empirical population estimates for $\hat{\Sigma}_{xy}$ and $\hat{\Sigma}_y$, the update equation becomes

\begin{equation*} X_{\text{posterior}} = X_{\text{prior}} + \frac{\bar{X}\bar{Y}^T}{r-1} \left( \Sigma_\epsilon + \frac{\bar{Y}\bar{Y}^T}{r-1} \right)^{-1} \left( \boldsymbol{d} \boldsymbol{1}^T - (Y + E) \right). \end{equation*}

This equation should never be implemented naïvely. For instance, the inverse should not be explicitly formed and neither should the huge cross-covariance matrix $\bar{X}\bar{Y}^T$ (order of matrix multiplication matters).

Emerick and Reynolds (2012, Appendix A)²⁰ write about computation, and explain two tricks: (1) scaling the system of equations and (2) subspace inversion. Emerick (2016, Appendix B)²¹ also discusses computation. But their sentence “it is computationally more efficient to rescale based only on the diagonal elements” is not as clear as it could be: using only the diagonal elements means we don’t solve the original equation.

The best way I know to solve this system is to scale with the Cholesky factor of $\Sigma_{\epsilon}$, then do a subspace inversion with a single SVD. This is different from both papers above, and is documented iterative_ensemble_smoother.

The Ensemble Information Filter

One issue with ES is that in equation $\eqref{eqn:ensemble_smoother_update}$ we have to estimate $\Sigma_{xy}$ and $\Sigma_{y}$. When $m$ and $n$ are larger than $r$, empirical covariance is a poor estimator that picks up spurious correlations—the update ends up being dominated by noise.

What if we avoid covariances and instead work with precision matrices and $H$? After all, there is a substantial literature on high-dimensional regression that we can lean on to estimate $H$. Also, parameters with conditional independence structure are often better described by precision matrices $\Lambda_x$ rather than covariance matrices $\Sigma_{x}$, see Rue and Held (2005, Ch. 2)²². The following ideas are due to Lunde (2025)¹⁷: (1) estimate $H$ with sparse high dimensional regression and (2) think in terms of precision rather than covariance. We’ll refer to his work as the Ensemble Information Filter (EnIF). The information terminology means that we parametrize in terms of precision rather than covariance. We’re discussing smoothers, so the filter does not exactly match this document—but we’ll stick with EnIF rather than inventing a new abbreviation.

Let us approximate $\boldsymbol{y} = h(\boldsymbol{x})$ as $\boldsymbol{y} = H \boldsymbol{x} + \boldsymbol{r}$. Here $\boldsymbol{r}$ is the residual (the non-linear part of $h$) and $H$ is a sparse matrix.⁷ If we model $\boldsymbol{r} \sim \mathcal{N}(\boldsymbol{0}, \Sigma_{r})$, then a large $\Sigma_{r}$ means the linear model is poor.⁸

Statement 2. The matrix form of the EnIF update equation, which updates every realization (column) in $X$, is

\begin{align} X_{\text{posterior}} &= X_{\text{prior}} + ( \Lambda_x + H^T \tilde{\Lambda} H )^{-1} H^T \tilde{\Lambda} (\boldsymbol{d} \boldsymbol{1}^T - (Y + E)) \label{eqn:ensemble_smoother_update_inf} \tag{8} \\ \tilde{\Lambda} &= (\Sigma_{d \mid x})^{-1} = \left( \Sigma_{\epsilon} + \Sigma_{r} \right)^{-1} \label{eqn:ensemble_smoother_update_precision_def} \tag{9} \end{align}

where $Y = h(X_{\text{prior}})$ applied column-wise and each column of $E$ is independently sampled observation $\boldsymbol{\epsilon}_d$. In other words, $E_{\cdot, j} \sim \mathcal{N}(\boldsymbol{0}, \Sigma_\epsilon)$ for every column $j$.

$H$, $\Lambda_x$ and $\Sigma_{r}$ must be estimated. The forward map $H$ is estimated by regressing $Y = h(X_{\text{prior}})$ on $X_{\text{prior}}$, typically with a sparse high dimensional method. The precision $\Lambda_x$ must be estimated, taking into account the parameter ensemble $X$ as well as neighborhood (graph) information (sparsity pattern in $\Lambda_x$). The residual covariance $\Sigma_{r}$ can be estimated as $\operatorname{diag}\left( \operatorname{Cov}\left[ Y - HX \right] \right)$, where $H$ is the estimated forward map. Correcting for degrees of freedom is an option when estimating $\Sigma_{r}$.

We will now sketch why equation $\eqref{eqn:ensemble_smoother_update_inf}$ is the correct update equation, in the sense that posterior mean and covariance matches under the Gauss-linear model. The analytical model given the assumption $\boldsymbol{y} = H \boldsymbol{x} + \boldsymbol{r}$ becomes:

\begin{align*} p(\boldsymbol{x}) &= \mathcal{N}\left( \boldsymbol{x} \mid \boldsymbol{\mu}_x, \Sigma_x \right) \\ p(\boldsymbol{d} \mid \boldsymbol{x}) &= \mathcal{N}( \boldsymbol{d} \mid H\boldsymbol{x}, \Sigma_{\epsilon} + \Sigma_{r}) \end{align*}

This is analogous to equation $\eqref{eqn:model_def_gaussian}$, but with $\Sigma_{\epsilon}$ replaced by $\Sigma_{\epsilon} + \Sigma_{r}$. The posterior distribution $p(\boldsymbol{x} \mid \boldsymbol{d}) = \mathcal{N} (\boldsymbol{\mu}_{x \mid d}, \Sigma_{x \mid d})$ over parameters is given by

\begin{align} \boldsymbol{\mu}_{x \mid d} &= \Sigma_{x \mid d} \left( H^T \left( \Sigma_{\epsilon} + \Sigma_{r} \right)^{-1} \boldsymbol{d} + \Sigma_x^{-1} \boldsymbol{\mu}_x \right) \label{eqn:posterior_mean_precision_inf} \tag{10} \\ \Sigma_{x \mid d} &= \left( \Sigma_x^{-1} + H^T \left( \Sigma_{\epsilon} + \Sigma_{r} \right)^{-1} H \right)^{-1} \label{eqn:posterior_cov_precision_inf} \tag{11} \end{align}

From equation $\eqref{eqn:push_through_kalman}$ and the result in Remark 1, we observe that we can write the Kalman gain as a function of precision matrices and $H$ as

\begin{equation*} K = \Sigma_{x \mid d} H^T \Sigma_{d \mid x}^{-1} = ( \Lambda_x + H^T \tilde{\Lambda} H )^{-1} H^T \tilde{\Lambda} \end{equation*}

where $\tilde{\Lambda}$ is defined as in equation $\eqref{eqn:ensemble_smoother_update_precision_def}$ and $\Lambda_x = \Sigma_{x}^{-1}$ by definition. At this point it’s quite easy to mimic the proof for the ES update equation, to show that equation $\eqref{eqn:ensemble_smoother_update_inf}$ also is correct, in the sense that

\begin{equation*} \mathbb{E} \left[ \boldsymbol{x}_{\text{posterior}} \right] = \boldsymbol{\mu}_{x \mid d} \qquad \operatorname{Cov} \left[ \boldsymbol{x}_{\text{posterior}} \right] = \Sigma_{x \mid d} \end{equation*}

The two differences between EnIF and ES, under the Gauss-linear assumption, are that (1) the Kalman gain relies on precision and $H$ instead of covariance and (2) the covariance of $\boldsymbol{r}$ is accounted for. We have no theoretical guarantees that EnIF is better than ES, in the sense that the update equation is equivalent but written in a different form. The hope is that if we are able to compute high-quality estimates of $H$ and $\Lambda_x$, the resulting update will also have high quality.

Computation

Here are some notes on computation of the EnIF update.

Subproblem 1 (Estimate $\Lambda_x$)

If the realizations in $X_\text{prior}$ are known to be multivariate Gaussian, e.g. $\boldsymbol{x}_\text{prior} \sim \mathcal{N}\left( \boldsymbol{x} \mid \boldsymbol{\mu}_x, \Sigma_x \right)$ with a known $\Sigma_x$, then $\Lambda_x = \Sigma_x^{-1}$ and no estimation is needed.
If $X_\text{prior}$ is given without information about its distribution, but the adjacency matrix $G$ is given ($G_{ij} = 1$ iff $(\Lambda_x)_{ij} \neq 0$), then we have to solve the problem “estimate $\Lambda_x$, given $X_\text{prior}$ and $G$.” This can for instance be done with the algorithm given in Hastie et al. (2009, §17.3.1)¹⁶ or the triangular transport approach given in Lunde (2025)¹⁷.
If $X_\text{prior}$ is given but no information about the graph structure $G$ is given, then the graph structure can be estimated. A good starting point for information about this problem is Hastie et al. (2009, §17.3.2)¹⁶.

Subproblem 2 (Estimate $H$)

The problem is to find $H \in \mathbb{R}^{n \times m}$ such that $Y \approx H X$. This can be formulated as $n$ regressions, one for each response: $X^T H^T \approx Y^T$ in matrix notation or $X^T \boldsymbol{h}_{i, \cdot} \approx \boldsymbol{y}_{i, \cdot}$ in vector notation. For each response $i$, we solve the regression problem to obtain $\boldsymbol{h}_{i, \cdot}$, the $i$‘th row in $H$.

This regression should be

Sparse, since then the posterior precision $\Lambda_x + H^T \tilde{\Lambda} H$ will retain most of the sparsity of the prior $\Lambda_x$. Sparsity also means a more interpretable model.
Fast, since both $m$ and $n$ can be huge.
High dimensional, since $r$ is small compared to $m$ and $n$.

Some kind of Lasso or boosted linear regression is a good choice. See Lunde (2025, Appendix E)¹⁷ for more details. The information criterion in Bühlmann (2006)²³ could perhaps be used.

Main computation

The algorithm inputs are $X_{\text{prior}}$, an estimated $H$, the observation noise covariance matrix $\Sigma_{\epsilon}$ and responses $Y := h(X_\text{prior})$.

Compute the residual matrix $R = Y - HX_{\text{prior}}$ and estimate $\Sigma_r$ from it. For instance by choosing $\Sigma_r$ to be diagonal, i.e., taking the variance over each row (response-residual) in $R$.
Compute $\tilde{\Lambda} = \left( \Sigma_{\epsilon} + \Sigma_{r} \right)^{-1}$. If either $\Sigma_{\epsilon}$ or $\Sigma_{r}$ is dense, then do not explicitly compute the inverse like this, but instead use e.g. Cholesky factorization to solve systems of equations. If they are both diagonal, then exploit that fact in computation instead of actually forming the diagonal matrix.
Solve

\begin{equation*} ( \Lambda_x + H^T \tilde{\Lambda} H ) Z = H^T \tilde{\Lambda} (\boldsymbol{d} \boldsymbol{1}^T - (Y + E)) \end{equation*}

or, equivalently, the column version for each column $j$

\begin{equation*} ( \Lambda_x + H^T \tilde{\Lambda} H ) \boldsymbol{z}_j = H^T \tilde{\Lambda} (\boldsymbol{d} - (\boldsymbol{y}_j + \boldsymbol{\epsilon}_d)) \end{equation*}

If $\tilde{\Lambda}$ is diagonal, then exploit that fact. Since $( \Lambda_x + H^T \tilde{\Lambda} H )$ is sparse, large and symmetric positive definite, it can be solved with Cholesky or with Conjugate gradients. 4. Set $X_{\text{posterior}} = X_{\text{prior}} + Z$.

If you want to jump ahead, the issue was that in equation $\eqref{eqn:ensemble_smoother_update}$ the original formulation had $\boldsymbol{d} - \boldsymbol{y}$ in the innovation term instead of the correct $\boldsymbol{d} - (\boldsymbol{y} + \boldsymbol{\epsilon}_d)$. ↩
We can derive the posterior mean $\boldsymbol{\mu}_{x \mid d}$ by maximizing the log-posterior $\log ( p(\boldsymbol{x} \mid \boldsymbol{d}))$, which is equal to $\log ( p(\boldsymbol{d} \mid \boldsymbol{x}) \, p(\boldsymbol{x}))$ (up to a constant) by Bayes’ theorem. Simply take the gradient of $(\boldsymbol{d} - H \boldsymbol{x})^T \Sigma_\epsilon^{-1} (\boldsymbol{d} - H \boldsymbol{x}) + (\boldsymbol{x} - \boldsymbol{\mu}_x)^T \Sigma_x^{-1} (\boldsymbol{x} - \boldsymbol{\mu}_x)$ with respect to $\boldsymbol{x}$, equate it to $\boldsymbol{0}$ and solve. ↩
A more direct, constructive way of showing this is to appeal to the push-through identity, see for instance the Wikipedia page on the Woodbury matrix identity. ↩
This approximation comes without any guarantees, so engineering judgment is often required. We potentially perform an update so full of noise that keeping the prior would’ve been better. ↩
In statistics and machine learning, samples are typically associated with rows and features with columns. Here that convention is transposed; each parameter gets a row and each realization a column. ↩
Some authors write the centered matrix $\bar{X}$ as $X (I - \boldsymbol{1} \boldsymbol{1}^T / r)$. While centering can be expressed as matrix multiplication, it is both hard to read and an unwise implementation strategy. ↩
You might be wondering why we do not include a bias term in the approximation. It’s not needed, since it will cancel in the innovation term in ES-style updates like equation $\eqref{eqn:ensemble_smoother_update}$ anyway. ↩
The reason we did not include residuals $\boldsymbol{r}$ in the ES derivation is that they become part of $\Sigma_{y}$ in equation $\eqref{eqn:ensemble_smoother_update}$ anyway. Under the assumption $\boldsymbol{y} = H \boldsymbol{x}$ we have $\Sigma_{y} = \operatorname{Cov}\left[ \boldsymbol{y} \right] = \operatorname{Cov}\left[ H \boldsymbol{x} \right] = H \Sigma_{x} H^T$. Had we instead assumed $\boldsymbol{y} = H \boldsymbol{x} + \boldsymbol{r}$, then $\Sigma_{y} = \operatorname{Cov}\left[ \boldsymbol{y} \right] = \operatorname{Cov}\left[ H \boldsymbol{x} + \boldsymbol{r} \right] = H \Sigma_{x} H^T + \Sigma_{r}$. Either way, in the ES update equation $\eqref{eqn:ensemble_smoother_update}$ we rely on an empirical $\Sigma_{y}$, which captures the residuals. It is only now that we wish to rely on $H$ instead of $\Sigma_{y}$ that we have to decompose $\Sigma_{y}$ into variance due to $\boldsymbol{x}$ mapped through $H$ (the $H \Sigma_{x} H^T$ term) and variance due to residuals (the $\Sigma_{r}$ term). ↩
G. Evensen, “Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics,” Journal of Geophysical Research, vol. 99, no. C5, pp. 10143–10162, 1994. ↩
P. J. Van Leeuwen and G. Evensen, “Data assimilation and inverse methods in terms of a probabilistic formulation,” Monthly Weather Review, vol. 124, no. 12, pp. 2898–2913, 1996. ↩
G. Burgers, P. J. Van Leeuwen, and G. Evensen, “Analysis scheme in the ensemble Kalman filter,” Monthly Weather Review, vol. 126, no. 6, pp. 1719–1724, 1998. ↩
Y. Chen and D. S. Oliver, “Ensemble Randomized Maximum Likelihood Method as an Iterative Ensemble Smoother,” Mathematical Geosciences, vol. 44, pp. 1–26, 2012. ↩
Y. Chen and D. S. Oliver, “Levenberg–Marquardt Forms of the Iterative Ensemble Smoother for Efficient History Matching and Uncertainty Quantification,” Computational Geosciences, vol. 17, pp. 689–703, 2013. ↩
A. A. Emerick and A. C. Reynolds, “Ensemble Smoother with Multiple Data Assimilation,” Computers & Geosciences, vol. 55, pp. 3–15, 2013. ↩
F. C. Vossepoel, G. Evensen, and P. J. van Leeuwen, “Adaptive Correlation- and Distance-Based Localization for Iterative Ensemble Smoothers in a Coupled Nonlinear Multiscale Model,” Monthly Weather Review, vol. 153, no. 11, pp. 2593–2609, 2025. ↩
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. New York: Springer, 2009. ↩↩↩
B. Å. S. Lunde, “An Ensemble Information Filter: Retrieving Markov-information from the SPDE Discretisation,” arXiv:2501.09016, 2025. ↩↩↩↩
K. B. Petersen and M. S. Pedersen, The Matrix Cookbook, Technical University of Denmark, version 20121115, 2012. ↩
C. M. Bishop, Pattern Recognition and Machine Learning. New York: Springer, 2006. ↩
A. A. Emerick and A. C. Reynolds, “History matching time-lapse seismic data using the ensemble Kalman filter with multiple data assimilations,” Computational Geosciences, vol. 16, pp. 639–659, 2012. ↩
A. A. Emerick, “Analysis of the performance of ensemble-based assimilation of production and seismic data,” Journal of Petroleum Science and Engineering, vol. 139, pp. 219–239, 2016. ↩
H. Rue and L. Held, Gaussian Markov Random Fields: Theory and Applications. New York: Chapman and Hall/CRC, 2005. ↩
P. Bühlmann, “Boosting for High-Dimensional Linear Models,” Annals of Statistics, vol. 34, 2006. ↩

Hvordan fungerer opptak til høyere utdanning?

Tommy Odland — Mon, 20 Jul 2026 00:00:00 +0200

I Norge har vi en svært god prosess for opptak til høyere utdanning. Ved å ta i bruk en sentralisert mekanisme har vi laget et system som er raskt, objektivt og korrupsjonsfritt. Dette systemet, kalt Samordna Opptak, fordeler hvert år omtrent 150 000 søkere til 1 400 studier. Hver søker sender simpelthen inn en rangert liste over sine ønsker, og en algoritme fordeler deretter studieplassene.

I denne artikkelen skal vi se nærmere på følgende spørsmål:

Hva er fordelene med et sentralisert opptak? Hvorfor gjør vi ikke som USA og desentraliserer opptaket?
Hvordan fungerer egentlig algoritmen? Den er tross alt med på å bestemme fremtiden til over hundre tusen mennesker hvert eneste år.
Hvilke effektivitetsgevinster gir man fra seg i algoritmevalget? Og på hvilken måte mister vi effektivitet ved å bruke kjønnspoeng og kvoter?

Sentralisert vs. desentralisert opptak
- Desentralisert opptak i USA
- Forutsetninger for bruk av algoritmer
Algoritmen: Utsatt Aksept
- Eksempel: Utsatt Aksept
Pareto-effektivitet
- Ineffektivitet grunnet stabilitetskrav
- Ineffektivitet grunnet kjønnspoeng
Oppsummering og referanser
Appendiks A: Digitalisering, algoritmer og offentlig sektor
Appendiks B: Boston-mekanismen

Sentralisert vs. desentralisert opptak

At opptaket er sentralisert betyr at søkere forholder seg til en sentral autoritet, heller enn å individuelt søke til hvert studie. Ifølge en Verdensbank-rapport (2024) bruker omtrent 90 land sentralisert opptak til høyere utdanning. De fleste av disse er rike, industrialiserte land.

I Norge har vi hatt sentralisert opptak siden midten av 90-tallet. Det er lett å ta gevinstene for gitt, men i enkelte andre land er opptaksprosessen fremdeles manuell, og derfor langt mer krevende å navigere, samt mindre transparent.

Desentralisert opptak i USA

I USA søker man individuelt til hvert studie eller universitet. Hvert studie rangerer søkere etter til dels subjektive kriterier, som ofte heller ikke er allmennkjente (essay, fritidsaktiviteter, diverse kvoter, osv). Det er flere runder, f.eks. er det ofte en “early admissions” runde først. Dette er et uoversiktlig og komplekst system for alle parter, og en kommersiell industri har oppstått som direkte resultat av dette.

I USA er studiestedene aktive strategiske aktører som tilbyr plasser i håp om å kapre de beste studentene. Studenter må også tenke strategisk: “bør jeg akseptere et tilbud nå, i tilfelle jeg ikke får et bedre tilbud senere?” I Norge trenger ikke studenter å tenke strategisk i det hele tatt—de oppgir bare en rangert liste og algoritmen ordner resten. Dette er fordi algoritmen som brukes i Norge er strategisikker (man kan ikke oppnå et bedre resultat ved å oppgi en uærlig rangering), og fordi vi har felles tidsfrister som alle studiesteder må forholde seg til.

Forutsetninger for bruk av algoritmer

Det sitter selvsagt ikke ansatte på et kontor og manuelt fordeler hundre tusen studieplasser. Jobben gjøres av en algoritme: en veldefinert sekvens av konkrete steg. I Norge har vi vært kloke nok til å koke opptaksprosessen ned til noen få objektive kriterier, f.eks. karaktersnitt og alderspoeng, som en algoritme kan prosessere.

Forutsetningene for å kunne bruke en algoritme godt i slike settinger er:

Alle studiesteder må være med. Vi bør ikke ha parallelle opptaksprosesser på siden.
Rangeringskriteriene må være transparente, objektive og ikke for kompliserte.
Selve algoritmen må være god.

Påstander som “ikke for kompliserte” og “algoritmen må være god” er vage. I resten av artikkelen skal vi konkretisere disse utsagnene og se på et enkelt eksempel.

Algoritmen: Utsatt Aksept

Algoritmen som brukes i Samordna Opptak er med høy sannsynlighet en variant av Deferred Acceptance, eller Utsatt Aksept på norsk. Jeg fant dessverre ingen offisiell kilde som bekrefter dette, så det er kvalifisert gjetning. Likevel er jeg rimelig sikker, fordi algoritmen brukes av mange andre land, og Wikipedia hinter til Utsatt Aksept. Dette er en utmerket algoritme med gode egenskaper (strategisikkerhet og fri for misunnelse), som brukes verden over til lignende opptaksprosesser.

Algoritme (Utsatt Aksept).

Input til algoritmen:

for hver student, en rangert liste over studier
for hvert studie, en rangert liste over studenter
for hvert studie, en kapasitet

Studenter fordeles på studier i tre steg:

Hver student søker på sitt førsteønske, og hvert studie tar tentativt inn studentene som er høyest rangert, mens resten blir avvist.
Hver avviste student søker på sitt neste ønske, og hvert studie ser på både de nye søkerne og de som allerede er tentativt akseptert. Hver studie tar inn studentene som er rangert høyest, mens resten blir avvist.
Steg 2 gjentas til alle studenter enten er akseptert eller har blitt avvist fra alle studier.

Eksempel: Utsatt Aksept

Her er et eksempel med 4 studenter og 3 studier som viser hvordan Utsatt Aksept fungerer. Vi ser bort fra kvotene, og tar kun med karakterpoeng og kjønnspoeng. I virkeligheten har vi også språkpoeng, realfagspoeng, alderspoeng og tilleggspoeng, men dette endrer ikke hvordan algoritmen fungerer—det endrer kun studienes rangering av studentene.

Her er et lite datasett med søkere og studieplasser:

Søkere
------
Anna  (poeng: 42): ['Sykepleie', 'Lærer', 'Ingeniør']
Eva   (poeng: 42): ['Lærer', 'Sykepleie', 'Ingeniør']
Elias (poeng: 41): ['Lærer', 'Ingeniør', 'Sykepleie']
Emma  (poeng: 40): ['Lærer', 'Ingeniør', 'Sykepleie']

Studieplasser
-------------
Sykepleie: 1 plass
Ingeniør:  1 plass
Lærer:     1 plass

Ettersom vi inkluderer kjønnspoeng, antar vi at følgende tilleggspoeng:

Kvinner får 2 kjønnspoeng når de søker ingeniør.
Menn får 2 kjønnspoeng når de søker sykepleie.

Uten kjønnspoeng ville studienes rangering vært karakterbasert og helt identisk, altså ['Anna', 'Eva', 'Elias', 'Emma'] for alle studiene. Dersom flere studenter har likt antall poeng, avgjør vi rekkefølgen alfabetisk. I virkeligheten er det alder som brukes som tie-breaker, se $ 7-1 (4) i forskrift om opptak til høgare utdanning.

Med kjønnspoeng tatt i betraktning rangerer hvert studie søkerne slik:

Studieplassenes rangering
-------------------------
Sykepleie: 'Elias', 'Anna', 'Eva', 'Emma'
Ingeniør:  'Anna', 'Eva', 'Emma', 'Elias'
Lærer:     'Anna', 'Eva', 'Elias', 'Emma'

Nå har både studentene og studiene en rangering av hverandre, og vi kan kjøre Utsatt Aksept. Her er stegene som algoritmen utfører, forklart i detalj:

Anna søker sykepleie og blir akseptert. Eva, Elias og Emma søker lærer. Lærerstudiet rangerer Eva høyest av disse tre og aksepterer henne. Elias og Emma blir avvist fra lærerstudiet.
Elias og Emma søker begge ingeniør. Ingeniørstudiet rangerer Emma høyest av disse to og aksepterer henne. Elias blir avvist.
Elias søker sykepleie. Sykepleie rangerer Elias foran Anna (som har plassen tentativt), så Elias blir akseptert og Anna blir avvist.
Anna søker lærer. Lærerstudiet rangerer Anna foran Eva, så Anna blir akseptert og Eva blir avvist.
Eva søker sykepleie. Sykepleie rangerer Elias foran Eva, så Eva blir avvist.
Eva søker ingeniør. Ingeniør rangerer Eva foran Emma, så Emma blir avvist.
Emma søker sykepleie. Sykepleie rangerer Elias foran Emma, så Emma blir avvist. Emma har nå blitt avvist fra alle studiene, og algoritmen er ferdig.

Resultatet blir følgende fordeling:

Studieplassenes opptak
----------------------
Sykepleie: 'Elias'  (3. ønske)
Ingeniør:  'Eva'    (3. ønske)
Lærer:     'Anna'   (2. ønske)

Ingen fikk sitt førsteønske, og bare Anna fikk sitt andreønske. Emma fikk ikke studieplass i det hele tatt.

Pareto-effektivitet

Du synes kanskje ikke at løsningen ovenfor var særlig god? Tross alt fikk ingen sitt førsteønske, og bare én person fikk sitt andreønske. En måte å kvantifisere om en løsning er god på, er så sjekke om den er Pareto-effektiv (eller bare effektiv).

Definisjon (Pareto-effektivitet).

En fordeling er Pareto-effektiv dersom ingen kan bytte plass og bli mer fornøyde, uten at det går ut over noen andre.

Fordelingen ovenfor er ikke Pareto-effektiv, fordi både Anna og Elias ville blitt mer fornøyde om de byttet plass.

Ineffektivitet grunnet stabilitetskrav

Elias og Anna rangerte studiene slik:

Søkere
------
Anna  (poeng: 42): ['Sykepleie', 'Lærer', 'Ingeniør']
Elias (poeng: 41): ['Lærer', 'Ingeniør', 'Sykepleie']

På tross av at Anna ønsker å bli sykepleier og Elias ønsker å bli lærer, ender Anna opp på lærerstudiet og Elias opp på sykepleierstudiet. Dersom Anna og Elias bytter plass, får begge to sitt førsteønske!

Hvorfor kan vi ikke bare la de bytte plass? Det finnes en algoritme som baserer seg på slike bytter, den kalles Bytteringer (Top Trading Cycles på Engelsk). Om vi bruker Bytteringer ville vi fått løsningen:

Studieplassenes opptak
----------------------
Sykepleie: 'Anna'  (1. ønske)
Ingeniør:  'Eva'   (3. ønske)
Lærer:     'Elias' (1. ønske)

Problemet med å innfri byttet er at da vil Eva bli misunnelig på Elias. Når misunnelse (eller justified envy på Engelsk) oppstår, sier vi at løsningen ikke er stabil, fordi en student og ett studie foretrekker begge hverandre fremfor hva de ble tildelt.

Her er problemet med fordelingen vi får dersom vi innfrir byttet: Eva rangerte lærerstudiet foran ingeniørstudiet. Lærerstudiet rangerte Eva foran Elias. Elias har “tatt” plassen som Eva hadde prioritet på hos lærerstudiet!

Dette illustrerer den uunngåelige avveiningen mellom å prioritere enkeltindividers rettigheter til å ikke bli forbigått i køen (når rangeringen til studieplassene tolkes som en kø) og å innfri Pareto-effektivitet.

Algoritmen Utsatt Aksept respekterer enkeltindividers rettigheter i prioriteringslisten på bekostning av Pareto-effektivitet.
Algoritmen Bytteringer garanterer Pareto-effektivitet, men respekterer ikke enkeltindividers rettigheter (den kan skape misunnelse).

Det finnes ingen løsning på dette problemet som både (1) unngår at studenter blir misunnelige og (2) er Pareto-effektive. Det finnes derfor heller ingen algoritme som kan innfri begge disse egenskapene samtidig for et vilkårlig problem. I de aller fleste land, inkludert i Norge, bruker man Utsatt Aksept fordi man anser (1) som viktigere enn (2).

Ineffektivitet grunnet kjønnspoeng

Det er en annen kilde til ineffektivitet: kjønnspoeng.

Enten man er for eller i mot kjønnspoeng politisk, så er det greit å være klar over at kjønnspoeng ofte overstyrer individers egne ønsker. Ved å bruke kjønnspoeng ignorerer staten til en viss grad individers ønsker og ofrer effektivitet for å innfri politiske mål rundt kjønnsbalanse.

Uten kjønnspoeng ville studiene rangert studentene identisk etter kun karaktersnitt:

Studieplassenes rangering
-------------------------
Sykepleie: 'Anna', 'Eva', 'Elias', 'Emma'
Ingeniør:  'Anna', 'Eva', 'Elias', 'Emma'
Lærer:     'Anna', 'Eva', 'Elias', 'Emma'

Løsningen ville blitt:

Studieplassenes opptak (uten kjønnspoeng)
-----------------------------------------
Sykepleie: 'Anna'  (1. ønske)
Ingeniør:  'Elias' (2. ønske)
Lærer:     'Eva'   (1. ønske)

Dette er en bedre løsning enn den vi fikk med kjønnspoeng; faktisk blir alle tre studentene mer fornøyde.

Blir samfunnet bedre av å gi Eva hennes førsteønske (lærer), eller av å sende henne til tredjeønsket (ingeniør) fordi hun er kvinne og vil bidra til kjønnsbalanse blant ingeniører? Det er et politisk spørsmål, og din rettferdighetssans er med på å bestemme svaret.

Oppsummering og referanser

De fleste land bruker i dag sentraliserte opptak til høyere utdanning, og Norge er ett av dem. Det store unntaket er USA, der privat rådgivning i opptaksprosessen er en egen bransje som håver inn omtrent 30 milliarder NOK per år. Algoritmen som sannsynligvis brukes i Norge er Utsatt Aksept, som er en variant av Gale-Shapley algoritmen. Den er strategisikker og stabil, men ikke Pareto-effektiv.

Når alle studiene rangerer søkerne identisk—som er tilfellet om man kun baserer seg på karakterpoeng—så har vi Pareto-effektivitet: ingen grupper kan innbyrdes bytte plass og bli mer fornøyde. Dersom studiene derimot har forskjellige rangeringer av søkerne, f.eks. på grunn av kjønnspoeng, vil det oppstå et effektivitetstap. Noen søkere vil ønske å innbyrdes bytte plass, men staten sier implisitt at deres ønsker må vike—selv når et bytte ville vært win-win for de involverte og ikke gått ut over noen andre.

To gode referanser, skrevet av de samme forskerne, er:

Matching Markets: Theory and Practice. Abdulkadiroğlu, Atila, and Tayfun Sönmez. 2013. Se spesielt seksjon 4.
School Choice: A Mechanism Design Approach. Abdulkadiroğlu, Atila, and Tayfun Sönmez. 2003.

Min egen artikkel Mangelfull evaluering av inntaksmodeller kan også være interessant.

Appendiks A: Digitalisering, algoritmer og offentlig sektor

Et hjertesukk: på tross av mye prat om digitalisering og effektivisering i offentlig sektor, er det for lite fokus på enkle tiltak med høy nytteverdi.

Anta at Samordna Opptak sparer hver søker for om lag 10 timers arbeid, sammenlignet med individuelle søknader til hvert studie. Det tilsvarer omtrent 500 MNOK hvert eneste år. Dette er lite sammenlignet med den samfunnsøkonomiske genvisten av å ha en transparent, ikke-strategisk prosess som allokerer talent til riktig sted. (Om karakterer er ensbetydende med talent kan diskuteres, men prosessen er i hvert fall fri for korrupsjon, vennetjenester og pengemakt.) Samordna Opptak er fantastisk god digitalisering og algoritmebruk.

Samtidig står det i en rapport fra 2022 at datasystemene til Samordna Opptak ikke har “fulgt den teknologiske utviklingen”. Deler av systemet er utviklet og vedlikeholdt av én og samme person. Rapporten sier at “faren for total kollaps blir større for hver dag som går”.

Samfunnet kunne tjent mye på å bruke algoritmer for sentraliserte opptak i andre situasjoner. Noen åpenbare eksempler er:

barnehageopptak
fastlegebytter
opptak til militæret
fordeling av studentboliger
interne jobbmarkeder
tildeling av praksisplasser til studenter
og mye mer

Du kan lese mer om dette på Alloker.no, som er et prosjekt jeg har jobbet med som implementerer algoritmer som Utsatt Aksept bak et API.

Appendiks B: Boston-mekanismen

Boston-mekanismen er en algoritme som ikke bør brukes. Den fungerer slik:

Algoritme (Boston-mekanismen).

Studentene sender inn en rangert liste over studiene de ønsker. Algoritmen behandler så personene i iterasjoner:

Alle studentenes førsteønsker vurderes. Hvert studie fyller plassene sine med de høyest prioriterte studentene blant dem som har studiet som førsteønske. Studenter som ikke får plass, avvises.
Avviste studenter vurderes etter sitt andreønske, men kun for gjenværende plasser på studiene etter forrige iterasjon.
Slik fortsetter prosessen til studieplassene er fylt opp.

Dersom vi kjører algoritmen på vårt eksempel:

Sykepleierstudiet velger Anna. Lærerstudiet velger Eva. Elias og Emma avvises.
Ingeniørstudiet velger Emma. Elias avvises. Alle studieplassene er nå fylt opp.

Løsningen blir:

Studieplassenes opptak (Boston-mekanismen)
------------------------------------------
Sykepleie: 'Anna'  (1. ønske)
Ingeniør:  'Emma'  (2. ønske)
Lærer:     'Eva'   (1. ønske)

Dette er en god løsning i den forstand at mange får sine ønsker oppfylt. Dessverre er mekanismen ikke strategisikker, så vi kan ikke anta at de oppgitte ønskene er de reelle ønskene når en slik algoritme blir brukt i virkeligheten. Å bruke Boston-mekanismen, eller å prøve å finne opp sin egen algoritme, er en dårlig idé.

KI-billettkontroll

Tommy Odland — Sun, 19 Jul 2026 00:00:00 +0200

Dersom 5% sniker på bussen i område A, og 2% sniker i område B - hvordan bør Ruter fordele billettkontrollørene sine?

De kan f.eks.: (a) sende alle til område A for å ta flest mulige snikere, (b) fordele kontrollørene proporsjonalt eller (c) se bort fra historiske data og sende halvparten til hvert område. Ulempen med (a) er at publikum etter hvert vil skjønne at ingen lenger kontrollerer område B. Ulempen med (c) er at man tar færre snikere enn hva man kunne gjort. En mellomting, noe sånn som (b), er nok mest fornuftig. (Alle tre løsningene kan parametriseres.)

Denne problemstillingen skapte debatt da Ruter-sjefen sa til Aftenposten at deres KI-modell “forurenses med tilfeldige data for å gjøre billettkontrollene sosialt akseptable.” Det er uklart hva “sosialt akseptable” betyr, men noen politikere reagerte, og det kom et tilsvar fra Ruter. Der sier de, med ganske mange ord, at de gjør noe som ligner på (b): de gjennomfører kontroller bredt, samtidig som de prioriterer flere kontroller i områder med høyere snikandel.

Jeg skrev om Ruters KI-billettkontroll for tre år siden, i 2023, og la frem 8 ulike løsninger på problemstillingen som alle er i konflikt med hverandre—det finnes ikke én rettferdig løsning.

Ruter burde utdypet vage begreper som “KI-modell” og “sosialt akseptabelt”, og vært mer konkrete om algoritmene de bruker og målene de ønsker å oppnå. Det kunne ha startet en interessant diskusjon om algoritmer og kontroll. Når Skatteetaten, Mattilsynet eller Lånekassen utfører kontroll, bør de velge (a), (b) eller (c)? Er tilfeldig kontroll det eneste rettferdige, fordi alle individer skal ha like lite mistanke rettet mot seg a priori? Eller kan erfaringsdata brukes til å effektivisere kontroller? Hvis ja, i hvor stor grad? Kan alle typer data brukes? Hva skal epsilon være i en epsilon-grådig strategi?

Dette er god bruk av grunnleggende statisti… ehm… KI-modeller, og kunne nok ha effektivisert en del kontrollprosesser—men da må man først bli enige om hva man vil oppnå.

Ten lessons from twenty years of strength training

Tommy Odland — Mon, 22 Jun 2026 00:00:00 +0200

I started strength training in 2006. Over twenty years I’ve learned that the principles of strength are simple. Almost everything you need to know can be summarized in two pages. Following the ten lessons below will make you stronger than 90-99% of the population. Here are my two pages:

Start easy

Start with light weights if you’re a beginner. During the first months, establish a routine of going to the gym and learn to lift correctly. Don’t rush into heavy weights—you’ll have plenty of time for that later.

The same advice applies if you’re getting back to lifting after a break. When gyms re-opened after COVID-19, I lifted the bar on my first workout. Then I increased to 40 kg, then 60 kg, 70 kg, etc. Start too heavy and you might get injured, or experience debilitating soreness. Either one hinders the steady progress required to build strength.

Squatting 285 kg at 108 kg bodyweight in 2010.

Stay consistent

Priority number one is to keep training. Half the game is showing up 2-4 times a week. If you get injured, come up with a plan to train around it. Set your ego aside. Don’t risk training through pain.

You must also avoid burnout. Hitting the gym six days a week and eating like a bodybuilder can be motivating. But if you burn out and quit as a result, you’ve lost the game. Slow and steady is usually better.

Push yourself

Since you are now consistent, the next step is to set goals, make plans and push yourself. Muscles grow by adapting to stress via progressive overload. The best way to induce stress is to increase the weight. If you lift 100 kg today and your goal is to reach 140 kg in one year, then you must increase the weight by around 0.75 kg per week. This is the outline of a long-term plan.

There are several ways to drive progress. Suppose you can comfortably lift 4 sets of 5 repetitions at 80 kg today. Try 4x5 at 82.5 kg next week, then 4x5 at 85 kg, 4x5 at 87.5 kg, etc. until you plateau. Don’t rush it—the idea is to start easy and gradually push yourself. You can alternatively increase repetitions instead of weight: do 4x1 at 90 kg, then 4x2 at 90 kg, etc. until you can do 4x6 at 90 kg—then increment the weight to 100 kg and start over again with 4x1. A third option is to combine weights and repetitions, e.g. 4x5 at 80 kg, 5x5 at 80 kg, 4x5 at 82.5 kg, 5x5 at 82.5 kg, etc.

Deadlifting 262.5 kg at 104 kg bodyweight in 2022.

Two steps forward, one step back

When your strength plateaus—which it invariably will—drop the weight (or repetitions) and start over. Suppose you lift 4x5 at 80 kg today, and 82.5 kg the next week, then 85 kg, etc. and eventually you fail at 95 kg. What should you do next? Try to drop down to 85 kg and start increasing by 2.5 kg per week again. You’ll likely reach 100 kg this time.

I take a big step back and start light in every new training program (every 8-12 weeks). Some programs add a small step back every four weeks. The “two steps forward, one step back’’ idea can be used at different resolutions in a long-term plan, and can be applied to weight, volume (total number of repetitions) and intensity (% of one-rep max).

Switch it up

Introduce slight variations to stay motivated and get unstuck when you stagnate. If you train 5x5, try increasing the repetitions (e.g., 4x8) or decreasing them (e.g., 5x3). Switch out barbell exercises for their dumbbell counterparts. Experiment with unilateral alternatives, for instance heavy lunges instead of squats. Train with bands or chains for a while. But don’t experiment too much or too often—stick with something long enough for it to work.

Movements, not muscles

Want big arms? Gyms are littered with men who feverishly curl with little to show for it. But I have yet to see anyone with small arms bench three plates.

The body responds well to heavy compound exercises: bench press, military press, rows, chin-ups, squats, deadlifts, etc. These exercises train many muscles simultaneously and let you move serious weight. I bench four plates, but never train arms directly.

Benching 182.5 kg at 104 kg bodyweight in 2022.

Compete or cooperate

Compete with yourself. Strength is easy to measure, so take advantage of it. Test your strength, weigh yourself and take some pictures every three months. Document your journey and make note of what works for you.

Strength training can be solitary, but it does not have to be. Train with others, build a community and share information. Compete with your friends or sign up for a meet. Read books and articles, participate in online forums, or publish a training log.

Food makes you look good

If your goal is to look good, then diet matters more than training. I enjoy tweaking my training programs, but it honestly has no effect on how I look—only my diet does.

To master your diet, start by writing down everything you eat for a week. Buy a food scale and weigh your food. Weigh yourself daily, but use a three-day average to assess progress.

Once you’ve mapped out how you eat today, start adding or subtracting food. It’s easier to control your diet if you eat the same meals every day, so lock them down if you can. I fix all my meals except one per day, where I only impose a weight limit. Don’t lose more than 0.5-1% of your bodyweight per week. Don’t gain more than 0.25-0.5% per week.

Healthy foods often have low caloric density, and calories are by far the most important factor. Eat healthy most of the time. “Healthy’’ means food your grandmother or a bodybuilder would approve of. Eat enough protein. Don’t drink calories. Don’t do anything too extreme, like swearing off entire food groups, avoiding social gatherings or dropping weight sharply. It rarely works out long term.

At ~92 kg I look pretty good. At 110 kg? Not so much.

Mute the marketers

People will try to sell you supplements, clothes, gear and entertainment. Buy if you like, but it’s mostly marketing designed to exploit your psychology.

For instance, very few supplements have strong scientific backing. Whey is a cheap source of protein and creatine has documented effects. That’s about it.

As for clothes and gear, buy a pair of Adidas lifting shoes, Rehband knee sleeves and a belt. They will last a decade or more. What kind of T-shirt you wear to the gym hardly matters.

Fitness influencers peddle a mix of minutiae, drama and genuinely good information. In the end, your attention is sold to their advertisers. Use social media feeds with caution: the information does not stick well and comparison is the thief of joy. For information, read a few high-quality books. For motivation, make some real-world friends.

Keep learning

If you enjoyed this article, then The Art of Lifting and The Science of Lifting by Nuckols are short reads in the same spirit. Bigger Leaner Stronger by Matthews is an overall guide to training and nutrition. The books by Eric Helms also fall into this category. Rippetoe wrote three books I enjoyed: Starting Strength covers technique on the main lifts, Practical Programming teaches strength program design and Strong Enough? is a delightful collection of no-bullshit, down-to-earth essays. Dan John has also written many great essays. Atomic Habits by Clear covers habit formation, which is a big part of training and lifestyle.

Unsure where to start? Set some goals based on the strength standards at ExRx.net or strengthlevel.com. Choose a program like Starting Strength, StrongLifts, 5/3/1, Texas Method, etc. Good luck!

Optimal bus stop spacing

Tommy Odland — Sat, 16 May 2026 00:00:00 +0200

Did you know that building new roads can increase traffic congestion in a city? It’s called Braess’s paradox.

In a similar vein—though perhaps not as counterintuitive—adding bus stops to a bus route can increase travel time. That’s the main takeaway from the article “The United States needs fewer bus stops”. It’s great when we can improve a system by subtracting from it.

How can adding a stop increase travel time? The reason is that travel time is a function of two competing mechanisms:

If there are many bus stops, you don’t have to walk far until you get to one
However, with many bus stops, the bus will also have to stop many times on your journey

Simulation

Let us simulate this on a 10 kilometer stretch of road. We assume that population density is uniform along the road, so a journey can be randomly generated by drawing 2 numbers; a start and an end.

The figure below shows 25 random journeys, along with 6 equidistant bus stops 2000 meters apart.

The figure above outlines the basic simulation setup. We’ll also make the following assumptions:

The bus has infinite passenger capacity and always stops at every bus stop.
The bus dwells for 45 seconds at each bus stop.
The bus has a speed of 25 kilometers per hour (slow, but realistic for a city).
A person’s walking speed is 5 kilometers per hour.

Instead of 25 journeys, let us look at one million journeys. The figure below shows the distribution of travel time when traveling by foot and on the bus—a fine-grained histogram with a million journeys.

No journey ever takes more than two hours, since every person walks at a speed of 5 kilometers per hour, and the road is 10 kilometers long. Most journeys are faster by bus, but this is not the case for very short journeys—in that case walking is faster. The average travel time with the bus is 0.35 hours (21 minutes), which is significantly shorter than the average walking time of 0.67 hours.

What if we add more bus stops? Does it help? Let us try with 16 bus stops.

With 16 bus stops the average spacing goes down to 667 meters. The average travel time also goes down to 0.26 hours.

What if we keep adding bus stops? Will travel time continue to decrease? Let’s go all the way to 51 bus stops.

The figure above shows that more bus stops are not better, since with 51 bus stops the travel time is up to 0.36 hours. In fact, a road with 51 bus stops is worse than one with 6 bus stops!

Here is the figure you might have been waiting for; the average travel time as a function of bus stops.

The optimal number of bus stops is 16. Does 16 stops—a spacing of 667 meters—seem reasonable? I think so; you never have to walk for more than 333 meters to catch the bus, and on average you walk 167 meters. This corresponds to about 4 and 2 minutes of walking, respectively.

Analytical solution

The bus stop problem is excellent because it can be approached either by computer simulation or analytical mathematics (deriving equations). In fact, the two approaches complement each other nicely.

Let $n$ be the number of bus stops and $L$ be the length of the road (10 kilometers). Then $\ell = L / (n-1)$ is the equidistant spacing between each bus stop. Furthermore, let $v_w$ be the walking velocity and $v_b$ be the bus velocity. Finally, we denote the dwell time of the bus, measured in hours, by $d$.

The expected total travel time is the sum of three different terms:

Starting at a random location, the average distance to the closest bus stop is $\ell / 4$. We multiply this by two, since we have to walk both to the nearest bus stop at the start of our journey, and from the final bus stop at the end of our journey. The average time walking to and from the bus is therefore $\ell / (2 v_w)$.
The average distance traveled on the bus is roughly $L / 3$. The total time on the bus, excluding dwelling times, is therefore $L / (3 v_b)$.
The number of stops on a journey of length $L/3$ is $L / (3 \ell)$, so the total time on the bus due to dwelling at bus stops is $d L / (3 \ell)$.

The skeptical reader is encouraged to verify the claims above more thoroughly. If we put all this together, we obtain the average time $T$, as a function of bus stops $n$:

\begin{align*} T(n) &= \frac{\ell}{2 v_w} + \frac{L}{3 v_b} + \frac{d L}{3 \ell} \\ &= \frac{L}{(n-1) 2 v_w} + \frac{L}{3 v_b} + \frac{d (n-1)}{3} \end{align*}

Differentiating and solving $T'(n^{\star}) = 0$, we obtain

\begin{align*} n^{\star} = 1 + \sqrt{\frac{3 L}{2 v_w d}} \end{align*}

This analytical equation matches the simulation results nicely, as shown in the figure below.

Notice how the optimal number of bus stops grows like the square root of $L$. If the road segment becomes four times as long, only twice as many bus stops are needed. This is because the average journey also becomes longer, due to the assumption that journeys are uniformly drawn from the line segment.

Why the discrepancy between the simulations and the equation? It’s because bus journeys were simulated by picking the fastest out of four possible journeys. Given a starting point for a journey, a person can either go left or go right to catch the bus. Similarly, a person can either exit the bus before or after the final endpoint of the journey. This gives four possible journeys, and in each simulation I computed the fastest one.

Deltidsstillinger

Tommy Odland — Tue, 14 Apr 2026 00:00:00 +0200

NRK har publisert en sak om overtidsregler for deltidsansatte, og skriver at det er fastlåste posisjoner mellom arbeidsgivere (NHO, KS, Virke og Spekter) og arbeidstakere (Fagforbundet og LO). I denne artikkelen presenterer vi et enkelt kompromiss.

Dagens praksis med merarbeid og overtid

Anta at Ola jobber 50% stilling (20 timer per uke om vi antar en 40-timers uke) og tjener 300 kroner per time.

Hvis han jobber 10 timer over avtalt arbeidstid regnes dette som merarbeid, ikke overtid. Han vil få utbetalt $300 \times (20 + 10) = 9000$ kroner.

Om han derimot jobber 25 timer mer, jobber han 5 timer overtid (over grensa på 40 timer). Dersom satsen for overtid er 40%, får han utbetalt $300 \times 40 + 300 \times 5 \times 1.4 = 14 100$ kroner.

EU-domstolens konklusjon

I to dommer fra EU-domstolen regnes alt arbeid utover avtalt arbeidstid som overtid, ikke merarbeid.

Med disse reglene vil Ola få følgende utbetalt:

Om han jobber 10 timer over avtalt arbeidstid på 20 timer, får han $300 \times 20 + 300 \times 10 \times 1.4 = 10 200$ kroner.
Om han jobber 25 timer over avtalt arbeidstid på 20 timer, får han $300 \times 20 + 300 \times 25 \times 1.4 = 16 500$ kroner.

Problemstillingen

I dag er det ingen strategisk grunn til at en bedrift bør ansette i 100% stilling. Det er bedre å ha to ansatte i 50% stilling, fordi da har arbeidsgiver en viss fleksibilitet til å justere opp arbeidsmengden uten å betale ekstra. Om bedriften trenger 50 arbeidstimer, vil det koste:

Med én ansatt: $300 \times 40 + 300 \times 10 \times 1.4 = 16 200$.
Med to ansatte: $2 \times (300 \times 20 + 300 \times 5) = 15 000$.

I regimet som EU-domstolen legger opp til, er det likegyldig om man har en eller to ansatte. Prisen blir uansett:

Med én ansatt: $300 \times 40 + 300 \times 10 \times 1.4 = 16 200$.
Med to ansatte: $2 \times (300 \times 20 + 300 \times 5 \times 1.4) = 16 200$.

Det er også likegyldig hvem av de ansatte som jobber ekstra. Dersom Ola har 60% stilling og Kari har 40% stilling, så har det ingen betydning for arbeidsgiver om det er Ola eller Kari som jobber ekstra. Det er ingen gevinst i å balansere merarbeidet/overtiden.

Et kompromiss

La funksjonen $p$ være overtidsprosent som funksjon av timer jobbet $x$, og anta at stillingsprosenten (i timer) er gitt ved $s$. Et kompromiss er å velge en lineær funksjon $p(x) = ax + b$.

Dersom vi krever at $p(s) = 100$ og $p(40) = 140$, så får vi

\begin{equation*} p(x;s) = \frac{40}{40 - s} (x - s) + 100 \end{equation*}

Når $x \leq s$ definerer vi $p(x;s) = 100$, og når $x \geq 40$ definerer vi $p(x;s) = 140$. Dersom Ola jobber $t$ timer, skal han ha $300 \int_0^t (p(x) / 100) \, dx$ kroner i lønn. For 30 timer får han 9 300 kroner, sammenlignet med 9 000 i dag og 10 200 under EU-løsningen.

Partene vil neppe bli enige om et slikt kompromiss i praksis, men det har noen fordeler:

Det koster alltid arbeidsgiver å planlegge dårlig og bruke ansatte utover avtalt stillingsprosent, enten de jobber deltid eller heltid.
Arbeidsgiver har insentiv til å gi ansatte høyere stillingsprosent, heller enn å konsekvent bruke merarbeid/overtid.
Arbeidsgiver betaler ikke like mye for å bruke ekstra arbeidskraft fra en ansatt i 50% stilling, som hvis samme arbeidskraft blir hentet fra en ansatt med 100% stilling. Hvem vil du helst bli operert av på sykehuset: fulltidslegen som er på tredje ekstravakt, eller deltidslegen som er på første ekstravakt?
Dersom arbeidsgiver har to personer ansatt i 50% stilling og trenger to vakter ekstra, lønner det seg å fordele dem likt og gi én vakt til Ola og én til Kari. Med andre ord lønner det seg å balansere overtiden på de ansatte, i motsetning til både dagens regime og EU-domstolens tolkning.

Du kan lese LOs argumenter her og noen motargumenter fra arbeidsgiverne her.

Continuous interest with continuous deposits

Tommy Odland — Wed, 25 Mar 2026 00:00:00 +0100

Writing a savings calculator is a great programming exercise for beginners. They’re easy to implement in spreadsheets or programming languages like Python, and teach people about the power of compounding interest. In this article we’ll investigate a discretization error that such calculators often make and try to fix it.

A simple savings calculator

A simple savings calculator might look like this in Python:

interest_rate = 1.1  # 10% annual interest
saved_per_year = 10  # 10 units of money saved per year
years = 20

saved = 0
for year in range(1, years + 1):
    saved = saved * interest_rate + saved_per_year
    print(f"Year: {year:>2}  Saved: {saved:.2f}")

Running the code produces the following output:

Year:  1  Saved: 10.00
Year:  2  Saved: 21.00
Year:  3  Saved: 33.10
...
Year: 18  Saved: 455.99
Year: 19  Saved: 511.59
Year: 20  Saved: 572.75

But there’s a simplification in this calculation which causes us to underestimate the total amount saved. Did you spot it?

The discretization error

The “error” in the script above is noticeable in the first year of the computation, so let’s focus on the first year. We assumed that one year passes before the amount saved per year is deposited into the account. In reality, it would likely be deposited monthly as the person saves money throughout the year.

The bank might pay you the interest owed monthly or yearly, but their internal computation is done every single day. Therefore depositing as early as possible (when you get your salary) will help accumulate more interest on your deposit throughout the year. The same applies to mutual funds.

If the yearly interest rate is $r$, then the monthly interest rate is $r^{1/12}$ and the yearly interest rate is $r^{1/365}$. This is because interest rates compound multiplicatively, so the geometric mean, not the arithmetic mean, is the correct computation.

Back to our savings calculator: in that first year, if we deposit monthly and compute the interest monthly, we do not end up with $10$ units of money at the end of the first year. Instead we end up with $10.45$ units of money:

periods = 12
interest_rate = (1.1)**(1/periods)
saved_per_year = 10 / periods

saved = 0
for period in range(1, periods + 1):
    saved = saved * interest_rate + saved_per_year
    print(f"Period: {period:>2}  Saved: {saved:.2f}")

...
Period: 12  Saved: 10.45

What happens if we compound daily ($365$ times) instead? Or hourly ($365 \times 12$ times)? Or every second ($365 \times 12 \times 3600$ times)? Here is a figure showing the multiplicative factor that money grows by at the end of one year, as a function of the number of times we compound the interest, using $r=1.1$:

For instance, if we compound $12$ times the multiplicative factor is $\approx 1.045$. Observe that the factor converges to a finite number instead of blowing up to infinity. We’ll show that, as we compound interest rates and deposit more and more often, the factor converges to

\begin{align*} (r - 1) / \ln (r), \end{align*}

which for an interest rate $r=1.1$ becomes $0.1 / \ln(1.1) = 1.0492$ (compare with the figure above).

Continuous deposits

Let $a$ be the total amount deposited in a year, let $r$ be the yearly interest rate and $n$ be the number of times interest is compounded.

The naïve savings calculator above used $n=1$.
In terms of realism for a savings calculator, $n=12$ is a good choice if salary is paid every month and then deposited.
From the bank’s perspective, whether it’s a bank account or a mutual fund, $n=365$ is the upper limit.
We’re interested in what happens as $n \to \infty$. This is optimistic because it overestimates the discrete reality, but $n \to \infty$ is still a much better approximation of $n=12$ than $n=1$ (again: see the figure).

Let $a$ be the yearly added value. We can look for a pattern as we increase the number $n$:

\begin{align*} \text{n=1} \quad & a \\ \text{n=2} \quad & \left( \frac{a}{2} \right) r^{1/2} + \frac{a}{2} \\ \text{n=3} \quad & \left( \left( \frac{a}{3} \right) r^{1/3} + \frac{a}{3} \right) r^{1/3} + \frac{a}{3} \\ \text{n=4} \quad & \left( \left( \left( \left( \frac{a}{4} \right) r^{1/4} + \frac{a}{4} \right) \right) r^{1/4} + \frac{a}{4} \right) r^{1/4} + \frac{a}{4} \\ \end{align*}

These are all geometric series, and can be written as:

\begin{align*} \text{n=1} \quad & a \\ \text{n=2} \quad & \frac{a}{2} \left( r^{1/2} + 1 \right) \\ \text{n=3} \quad & \frac{a}{3} \left( r^{2/3} + r^{1/3} + 1 \right) \\ \text{n=4} \quad & \frac{a}{4} \left( r^{3/4} + r^{2/4} + r^{1/4} + 1 \right) \\ \end{align*}

The pattern for a general $n$ emerges, and (provided we assume $r\neq 1$) we can use the formula for a sum of a geometric series to obtain a closed-form expression:

\begin{align*} f_n(a, r) = \frac{a}{n} \sum_{i=0}^{n-1} \left( r^{1/n} \right)^i = \frac{a}{n} \frac{\left( r^{1/n} \right)^n - 1}{r^{1/n} - 1} = \frac{a}{n} \frac{r - 1}{r^{1/n} - 1} \end{align*}

What does this converge to as $n \to \infty$? It all comes down to what the denominator $n (r^{1/n} - 1)$, since that’s the only part of the equation that depends on $n$. The answer is $\ln(r)$, and there are many ways to show this: L’Hôpital’s Rule and Taylor Series are both possible methods. Here’s one way that I like: one of the many definitions of the exponential function $e^x$ is that

\begin{align*} e^x = \lim_{n \to \infty} \left(1+\frac{x}{n}\right)^n \end{align*}

so then, since $\ln(x)$ is the inverse of $e^x$ for positive $x$, we must have

\begin{align*} x = e^{\ln (x)} = \lim_{n \to \infty} \left(1+\frac{\ln (x)}{n}\right)^n \end{align*}

and if we rearrange this expression and substitute $x$ for $r$, we discover that $\lim_{n \to \infty} n (r^{1/n} - 1) = \ln(r)$.

The final result is that

\begin{align*} \lim_{n \to \infty} f_n(a, r) = \lim_{n \to \infty} \frac{a(r - 1)}{n (r^{1/n} - 1)} = \frac{a (r - 1)}{\ln (r)}. \end{align*}

We’re now in a position to create a more realistic savings calculator, in which money is continually deposited and interest is continually accrued:

import math

interest_rate = 1.1
saved_per_year = 10
years = 20

saved = 0
for year in range(1, years + 1):
    saved = saved * interest_rate
    saved += saved_per_year * (interest_rate - 1) / math.log(interest_rate)
    print(f"Year: {year:>2}  Saved: {saved:.2f}")

Year:  1  Saved: 10.49
Year:  2  Saved: 22.03
Year:  3  Saved: 34.73
...
Year: 18  Saved: 478.43
Year: 19  Saved: 536.76
Year: 20  Saved: 600.93

The difference is shown graphically in the animated figure below.

Solution by integration

We can obtain the same answer as above by realizing that as $n$ becomes large we can replace the sum with a Riemann integral:

\begin{align*} f_n(a, r) = \frac{a}{n} \sum_{i=0}^{n-1} r^{i/n} \approx a \int_{0}^{1} r^{x} \, dx \end{align*}

If you know the indefinite integral of $r^x$, then it’s easy to compute

\begin{align*} \int_{0}^{1} r^{x} \, dx = \int_{0}^{1} \exp(x \ln(r)) \, dx = \frac{1}{\ln(r)} \exp(x \ln(r)) \Biggr|^{x = 0}_{x = 1} = \frac{r-1}{\ln(r)}. \end{align*}

It is comforting that both approaches give identical results. Let’s look at a third approach that also leads to the same answer, but is more general and more powerful: differential equations.

Solution by differential equations

Let us summarize what we have learned so far. When we account for continuous deposits and continuous interest rates, we do not have $a$ units of money after the first year. Instead we end up with $a (r-1) / \ln(r)$ units of money. The factor $(r-1) / \ln(r)$ accounts for the difference between discrete yearly deposits and continuous deposits throughout the year. When $r=1$ this factor is undefined, but if we assume $r \neq 1$ then we get this graph:

In the figure above, I also plotted the first-order Taylor expansion around $r \approx 1$, which is $g(r) = 1/2 + r/2$.

The factor that “fixes” the first year, but can be applied every single year in our savings calculator. If we re-write the recursive definition into a sum, and again use the sum of a geometric series, we end up with an expression for an arbitrary year $n$:

\begin{align*} \text{n=1} \quad & a \frac{r-1}{\ln(r)} \\ \text{n=2} \quad & \left( a \frac{r-1}{\ln(r)} \right) r + a \frac{r-1}{\ln(r)} \\ \text{n=3} \quad & \left( \left( a \frac{r-1}{\ln(r)} \right) r + a \frac{r-1}{\ln(r)} \right) r + a \frac{r-1}{\ln(r)} \\ \text{n} \quad & a \frac{r-1}{\ln(r)} \left( r^{n-1} + r^{n-2} + \cdots + 1 \right) = a \frac{r-1}{\ln(r)} \frac{r^n - 1}{r-1} = a \frac{r^n - 1}{\ln(r)} \end{align*}

Perhaps the most powerful approach to studying continuous deposits and continuous interest rates is by using differential equations. Let $M(t)$ be money as a function of time, which is now a continuous function of $t$. Let $r$ be the yearly interest rate and $t$ be measured in years. Before we proceed to the general case, let us study two simple cases first:

Deposits without any interest.

\begin{align*} M(t) = at \qquad M'(t)= a \end{align*}

Interest without any deposits.

\begin{align*} M(t) = M_0 r^t \qquad M'(t) = \ln(r) M_0 r^t = \ln(r) M(t) \end{align*}

Both deposits and interest. Above we saw that the rate of change in a bank account is driven by $a$ as well as $\ln(r) M(t)$. We sum these to obtain the (first order linear ordinary) differential equation

\begin{align*} M'(t) = \ln(r) M(t) + a. \end{align*}

The solution to this differential equation (verify it!) is

\begin{align*} M(t) = M_0 r^t + \frac{a (r^t - 1)}{\ln(r)}, \end{align*}

where $M_0 = M(0)$ is the initial money in the account. This matches our previous result perfectly, where we assumed $M_0 = 0$.

As a final note, if $a$ is negative and we’re depleting the account, then a relevant question is “When will we go broke?”. This amounts to solving $M(t) = 0$ for an unknown $t$. In the general case, let $M(t) = M_t$, then we can solve for $t$ to obtain:

\begin{align*} t = \frac{1}{\ln (r)} \ln \left( \frac{M_t \ln(r) + a}{M_0 \ln(r) + a} \right) \end{align*}

Finally, here is an updated Python snippet that uses the function $M(t)$.

import math


def M(t, r, a, M0):
    r_power_t = math.exp(t * math.log(r))
    return M0 * r_power_t + a * (r_power_t - 1) / math.log(r)


interest_rate = 1.1
saved_per_year = 10
years = 20

for year in range(1, years + 1):
    saved = M(t=year, r=interest_rate, a=saved_per_year, M0=0)
    print(f"Year: {year:>2}  Saved: {saved:.2f}")

Summary and references

This is an example of how the first thing one might implement is not necessarily correct. I’ve taught students about geometric series, Excel and Python using this example a few times, and I was aware of the tension between computing the interest and deposit at the end of the year (which underestimates) and at the start of the year (which overestimates).

For students familiar with calculus and geometric series, working out the limit of $n \to \infty$ is a nice problem. In the end, assuming $n \to \infty$ is not really correct either—it probably overestimates slightly compared to the reality (which is likely $n=12$ or so). In many cases the corrective factor of $(r - 1) / \ln (r)$ is likely worth implementing since it’s closer to the truth than the naive $n=1$ computation. We ignored most issues related to numerical stability in this article, but in a proper implementation $r^{(1/n)}$ and $(r-1) / \ln(r)$.

Everything can be summarized by this set of equations: one discrete difference equation and one continuous differential equation. The equations and their closed-form solutions are:

\begin{align*} \boxed{M_n = r M_{n-1} + a} \Rightarrow M_n &= M_0 r^n + a \frac{r^n - 1}{r - 1} \\ \boxed{M'(t) = \ln(r) M(t) + a} \Rightarrow M(t) &= M_0 r^t + a \frac{r^t - 1}{\ln(r)} \end{align*}

Some further reading:

Modeling Change One Step at a Time, from Topics in Mathematical Modeling
Differential Equations by Chasnov
Continuous-repayment mortgage on Wikipedia
Compound interest on Wikipedia

Skatter og bidrag

Tommy Odland — Fri, 13 Feb 2026 00:00:00 +0100

Denne artikkelen er et tankeeksperiment der vi skal endre det norske skatte- og bidragssystemet. I dag styrer mange mekanismer pengeflyten mellom borgere og staten. Et forenklet bilde ser slik ut:

Kompleksiteten i et slikt system har store kostnader. Staten må holde styr på alle skattene og bidragsordningene: den må oppdatere lovverk, endre satser, vurdere og innvilge søknader, vedlikeholde digital infrastruktur, osv. Borgere må forstå systemet, være informert om støtteordningene og navigere søknadsprosesser. Begge parter kunne vært tjent med en forenkling.

Vi skal kutte nesten alle skatter og samle dem sammen i en funksjon $T(x)$. Deretter skal vi kutte nesten alle støtteordninger og samle dem sammen i en funksjon $B(x)$.

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k} \qquad B(x) = be^{-x/b} \end{align*}

Det er ikke åpenbart at disse funksjonene er i stand til å modellere store deler av forholdet mellom staten og borgerne. Vi skal prøve likevel. Resultatet blir at forholdet mellom staten og individet forenkles til følgende diagram.

Alle skatter og bidrag styres av tre parametre $H$, $k$ og $b$, og vi skal estimere disse. Underveis skal vi gi skattelette til folk flest, “ta de rike”, forenkle offentlig sektor, sørge for at det alltid lønner seg å jobbe, og kreve formuesskatt fra alle borgere. Så her er det noe for alle, uansett politisk ståsted.

Inntekt
Bidrag og støtteordninger
En realistisk beregning
- Oppsummering: Bidrag og støtteordninger
Formuesskatt
Oppsummering

Inntekt

For å regne ut skatt på en vanlig borgers inntekt må man ta hensyn til trygdeavgift, minstefradrag, personfradrag, skatt på alminnelig inntekt og trinnskatt. Dette er et komplisert regnestykke, og i artikkelen Smooth taxes without brackets viste jeg at beregningen kan reduseres til formelen

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k}, \end{align*}

der $T(x)$ er skatten man betaler om man har en brutto inntekt på $x$ kroner. Parametrene $H$ og $k$ bestemmer hvor aggressiv skatten er. $H$ er den maksimale marginalskatten ettersom $H = \lim_{x \to \infty} T'(x)$. Figuren nedenfor viser at funksjonen er en meget god tilpasning til dagens inntektsskatt.

I figuren ovenfor viser vi også inntektsfordelingen, som er hentet fra figur 2 i denne SSB-artikkelen. Boks 2.2 i Skatter og avgifter 2025 viser statens egen visualisering av marginalskatten. Denne skattekalkulatoren har 27 felter som kan fylles ut, så den fulle beregningen er enda mer komplisert.

Matematiske egenskaper

Her kommer en rask oppsummering av hvilke spørsmål funksjonen $T(x)$ svarer på og hvilke egenskaper som er hensiktsmessige for en slik skattefunksjon. Se Smooth taxes without brackets for en grundigere gjennomgang.

Nettoinntekt: $x - T(x)$ svarer på spørsmålet “Hvor mye penger har jeg igjen etter at jeg har betalt skatt?”
Marginalskatt: $T'(x)$ svarer på “Hvor stor andel må jeg skatte på neste krone jeg tjener?”
Skatteprosent: $T(x) / x$ svarer på “Hvilken andel av total inntekt betaler jeg i skatt?”

Eksempel. Dersom $x=800 \text{ TNOK}$, så er:

\begin{align*} T(x) = 227 \text{ TNOK} \quad T'(x) = 0.423 \quad T(x)/x = 0.284 \end{align*}

Funksjonen $T(x)$ har to viktige egenskaper bakt inn i definisjonen, for all verdier av $0 < H \leq 1$ og $k > 0$:

Gode insentiver. En skatt $T(x)$ har gode insentiver dersom marginalskatten $T'(x) < 1$ for alle $x$, fordi da ender man alltid opp med økt nettoinntekt dersom man tjener mer.
Progressiv beskatning. En skatt $T(x)$ er progressiv dersom skatteprosenten $T(x)/x$ øker når $x$ øker. Dersom Ola tjener mer enn Kari, så betaler Ola også en større prosentandel av sin inntekt i skatt.

Hva kan vi forenkle?

Fra en borgers perspektiv er kun skattenivået relevant, ikke hva staten kaller de ulike skattene og fradragene eller hva som finansierer hva. Dersom vi bruker $T(x)$ kan vi ta bort:

Minstefradrag
Personfradrag
Skatt på alminnelig inntekt
Trinnskatt
Arbeidsgiveravgift (det er i praksis en personskatt på 14.1 %)

La oss også samle all inntekt i $x$: pensjonsgivende inntekt, utbytte fra selskaper, aksjegevinst, styrehonorarer, osv. De som eier et selskap kan i dag velge å enten ta ut penger som inntekt eller som utbytte. Inntekt gir pensjonssparing gjennom folketrygden og rett til sykepenger, men utbytte har lavere marginalskatt. Det er ofte optimalt å først maksimere pensjonssparing ved å ta ut lønn opp til 7.1 G og deretter ta utbytte. Denne typen optimalisering er kun tilgjengelig for et fåtall. Jo enklere skatten er, desto mindre muligheter er det for optimalisering.

Om vi er villige til å fjerne noen flere, mer målrettede insentivordninger, kan vi fjerne:

Soner og næringsgrupperinger i arbeidsgiveravgift (14 koeffisienter)
BSU fradrag
Fagforeningskontingent
Foreldrefradrag
Reiseutgifter

Noen av disse fradragene bør modelleres som bidrag, og vi kan legge dem tilbake senere.

Vanlige folks tur

Vi kan gi skattelette til store deler av befolkningen om vi velger en høyere verdi for maksimal marginalskatt $H$ og løser for $k$ slik at statens inntekter forblir identiske med dagens nivå ($H \approx 0.47$).

Medianpersonen har en inntekt på omtrent 640 TNOK før skatt. Går vi fra dagens skatteregime til $H=0.55$, vil medianpersonen få skattelette på omtrent 3.8 TNOK per år. Setter vi heller $H=0.65$ får medianpersonen skattelette på 7.4 TNOK per år. Alle som er under 80 persentilen vil få skattelette, mens de som er over må skatte mer. I dagens system har DNB-sjefen, som tjente 16.9 MNOK i fjor, samme marginalskatt som en dyktig ingeniør (eller lege, advokat, osv.) som tjener en tiendedel av dette.

Tabellen sammenligner dagens scenario med to alternativer. Statens inntekter er identiske.

	$H=0.474$	$H=0.55$	$H=0.65$
Skattelette for medianperson	0 NOK	3800 NOK	7400 NOK
Andel som får skattelette	0 %	79 %	80 %
Marginalskatt DNB-sjefen	47 %	55 %	65 %
Marginalskatt ingeniør	47 %	53 %	60 %
Skatteprosent DNB-sjefen	46 %	53 %	63 %
Skatteprosent ingeniør	37 %	40 %	43 %

Oppsummering: Inntekt

Vi samler alt en person tjener i én felles inntekt $x$ og beskatter den med $T(x)$, enten det er ordinær pensjonsgivende inntekt, utbytte fra selskaper, avkastning på aksjefond, styrehonorar, godtgjørelser eller annet. Da vil en rik investor med 10 MNOK i aksjegevinst skatte en høyere andel enn en småsparer, noe som ikke er tilfellet i dag. Mange muligheter for optimalisering forsvinner, og de som får inntekt gjennom utbytte, avkastning og honorarer får samme beregning som folk flest. Ikke minst blir alt mye enklere. Vi kan endre $H$ og gi skattelette til 80 % av arbeidsstyrken, men insentivene må også vurderes.

Vi kan også inkludere arv i $x$, ettersom det også er inntekt. Det ville vært en dobbeltbeskatning av penger, men ikke en dobbeltbeskatning av personer. Dersom man arver en villa med nedbetalt lån i Oslo med verdi 20 MNOK, ville man måtte låne 9.1 MNOK i boligen for å betale skatten. Dette ville utjevnet forskjeller og forhindret opphopning av kapital. Alternativt kan vi droppe arveskatt og heller endre formuesskatten, noe vi kommer til senere i artikkelen.

Bidrag og støtteordninger

Det eksisterer mange støtteordninger for mennesker som av ulik grunn har lav inntekt, enten i en periode eller permanent. NAV har over 200 tjenester, og noen av disse er:

Uføretrygd
Sykepenger
Arbeidsavklaringspenger (AAP)
Dagpenger
Tiltakspenger

I disse fem kategoriene ble det utbetalt 27.3 milliarder NOK til 166 000 Oslo-borgere i 2024, noe som tilsvarer 164 TNOK per person. I noen bydeler er andelen som mottar utbetalinger relatert til arbeidsliv og sykdom over 40 %, og på landsbasis er den 36 %. Andre støtteordninger inkluderer pleiepenger, omsorgspenger, svangerskapspenger, yrkesskadeerstatning, foreldrepenger og økonomisk sosialhjelp.

Kritikk mot NAV går ofte på at dagens system:

Er krevende å navigere
Har dårlige insentiver

Systemet blir lettere å navigere om vi fjerner kompleksitet. Problemstillingen med insentiver kan forstås ved å se på enkeltsaker som viser såkalte bidragsfeller (welfare traps). I noen tilfeller lønner det seg ikke å jobbe fordi da forsvinner bidragene:

NAVs egen rapport fra 2023 anslår at 1 av 4 dagpengemottakere kan tape på å jobbe én dag. Mange mennesker lever i økonomiske bidragsfeller.

Å fange de fattige

Det er fort gjort å konstruere en bidragsfelle. Intensjonene må være så gode at man ikke tenker nøye nok over insentivene. Dersom bidraget er betinget på lønn (enten direkte eller indirekte via arbeidsevne) og kuttes skarpt når inntekt øker, så sitter man i fella. Dersom bidraget derimot senkes sakte, så unngår man feller.

Eksempel - konstant bidragsfelle. Hva om vi gir alle som tjener mindre enn 500 TNOK en utbetaling på 500 TNOK? Anta at vi har fem personer med ulik inntekt. La $B(x)$ være et bidrag som funksjon av bruttoinntekt $x$. Vi setter $B(x)$ lik 500 TNOK dersom $x$ er mindre enn eller lik 500 TNOK, og null hvis ikke. Visualiseringen viser hva som skjer:

Når man overstiger 500 TNOK i inntekt, så forsvinner hele bidraget $B(x)$ (øverste underfigur til høyre). Det rasjonelle er å ligge så nær 500TNOK som mulig, men aldri gå over. Dersom summen $x - T(x) + B(x)$ synker er det ikke noe insentiv for å tjene mer.

Eksempel - lineær bidragsfelle. Hva om vi velger en annen funksjon? Vi kan kompensere de som tjener mindre enn 500 TNOK med en sum slik at alle ender opp med minst 500 TNOK. Matematisk er funksjonen $B(x) = \max(500\text{ TNOK} -x, 0)$.

For en som har null i inntekt lønner det seg ikke å jobbe i det hele tatt, med mindre personen klarer å tjene ca. 665 TNOK. Selv da får han ikke betydelig mer enn hva han fikk med null inntekt.

Bidrag $B(x)$ med gode egenskaper

Det finnes bidrag $B(x)$ som i kombinasjon med $T(x)$ ikke fører til bidragsfeller. Funksjoner på formen

\begin{align*} B(x) = b e^{-x/b} + C \qquad b < \frac{1}{Hk} \end{align*}

har gode egenskaper: insentivene er gode og de er billige for staten.

Summen av skatt og bidrag har gode insentiver dersom det alltid lønner seg å jobbe. Matematisk er kravet at

\begin{align*} \underbrace{(x + \epsilon) + B(x + \epsilon) - T(x + \epsilon)}_{\text{netto med $\epsilon$ mer}} > \underbrace{x + B(x) - T(x)}_{\text{netto}} \end{align*}

Dersom vi stokker om på likningen og tar grenseverdien $\epsilon \to 0$ får vi kravet

\begin{align*} T'(x) - B'(x) < 1. \end{align*}

Vi deriverer skatten $T(x)$ og substituerer $T'(x) = H\left(1 - e^{-kx} \right)$. Da blir kravet at

\begin{align*} B'(x) > H\left(1 - e^{-kx} \right) - 1. \end{align*}

Denne likningen sier at bidragene ikke kan synke fort mot null når inntekten $x$ øker. Dette gir intuitivt mening; bidrag som forsvinner fort fører til bidragsfeller. Alle funksjoner $B(x)$ som oppfyller dette kravet er gyldige, og kvalifisert gjetning leder oss til $B(x) = b e^{-x/b} + C$, som oppfyller kravet $T'(x) - B'(x) < 1$ når $b < 1/(Hk)$ (se appendiks).

For eksempel er $b=200 \text{ TNOK}$ og $C=0$ gyldig, og gir følgende omfordeling via skatter og bidrag:

Konstantleddet $C$ kalles gjerne borgerlønn (universal basic income), mens $b e^{-x/b}$ er i kombinasjon med $T(x)$ en negativ skatt. Problemet med $B(x) = C$ er at det er dyrt for staten, fordi alle får bidrag. I resten av artikkelen setter vi $C=0$ og antar at $B(x) = b e^{-x/b}$. Legg merke til at $B(0) = b$, altså er $b$ summen som en person uten inntekt vil få utbetalt.

Verdien $b$ må balansere to hensyn: den må være høy nok til at det går an å leve på, samtidig må den være lav nok til å insentivere til arbeid.

Hva kan vi forenkle?

Dersom vi innfører et universelt bidrag $B(x) = b e^{-x/b}$ kan vi fjerne eller kutte kraftig i uføretrygd, sykepenger, arbeidsavklaringspenger, dagpenger, tiltakspenger og økonomisk sosialehjelp. NAV har 12 000 årsverk og 15 mrd. i driftsutgiver, og deler av dette kan kuttes. Tiden og ressursene som brukes på å sende inn og behandle søknader forsvinner helt.

Staten betaler også bidrag til friske mennesker for å søtte en rekke aktiviteter. Eksempelvis var det 12 000 søkere på statens kunstnerstipend i 2024, og i 2025 ble det bevilget nesten 500 millioner kroner. Mange slike ordninger kan fjernes. Mennesker som ønsker å lage kunst eller gjøre noe helt annet (som kan være vel så samfunnsnyttig men ikke anerkjent av staten), står fritt til å gjøre det—dersom de er villige til å leve på bidraget $B(x)$. Dersom kunsten selger, vil de få mer inntekt, miste bidrag og betale skatt—men de vil alltid ende opp med mer penger.

En realistisk beregning

I denne seksjonen forsøker vi grovt å regne ut hva $B(x)$ vil koste. Her er noen store utgiftsposter, hentet fra NAVs årsrapport 2024:

Uføretrygd: 125 269 MNOK
Sykepengetilfeller: 68 068 MNOK
Arbeidsavklaringspenger: 46 926 MNOK
Dagpenger: 13 300 MNOK
Hjelpemidler: 10 321 MNOK

Til sammen er dette 264 milliarder kroner. I følge SSB er 622 000 personer i yrkesaktiv alder utenfor arbeidslivet (omtrent 18 %). Fordeler vi 264 milliarder på 622 000 personer kan hver person få 424 TNOK.

Ettersom $B(x)$ også deler ut penger til de som ikke har null i inntekt, må vi ta hensyn til de 82 % som er i arbeidslivet. Jeg fant ingen gode data som viser inntektsfordeling for alle i Norge, men om vi baserer oss på fulltidsekvivalenter for de 82 % som er i arbeidslivet og kombinerer det med de 18 % som er utenfor, kommer vi frem til at 264 milliarder kan finansiere $b = 271 \text{ TNOK}$. At vi bruker fulltidsekvivalenter gjør at $b$ overestimeres. At vi verken inkluderer flere støtteordninger eller administrative kostnader gjør at $b$ underestimeres. Disse effektene kansellerer hverandre til en viss grad, og i sum er nok $b$ estimert noelunde riktig.

Men hvor mye bør staten dele ut? Det finnes ingen klar fasit, men her er to ankerpunkter:

SIFOs referansebudsjett er på omtrent 190 TNOK i året (eksl. bolig, ferier og helsetjenester). Boliginformasjon kan vi finne hos SSB, som estimerer typisk boligkostnad for eiere til 130 TNOK per år og leietakere til 111 TNOK per år. En person trenger derfor omtrent 190 + 120 = 310 TNOK per år for å leve greit.
Lavinntekstsgrensa måler relativ fattigdom og er definert til å være 60% av medianinntekt etter skatt. Grensa er på 285 TNOK i 2025. Oppsummert har staten råd til $b \approx 271 \text{ TNOK}$, og dette nivået samsvarer godt med SIFOs referansebudsjett og fattigdomsgrensen.

Oppsummering: Bidrag og støtteordninger

Bidrag og støtteordninger er i hovedsak til for syke og arbeidsledige. Dagens systemer kan være vanskelige å navigere, kreve mye dokumentasjon, og kan disinsentivere arbeid. Staten må bedømme hva som er verdifullt for samfunnet, selv om det kanskje ikke er kommersielt levedyktig. Stipendordninger kan føre til habilitetsutfordringer og staten må bruke ressurser på å bestemme hvem som får hva.

Vi innfører en bidragsfunksjon $B(x)$, som sammen med $T(x)$ gir negativ skatt til de som tjener lite. Dette krever ingen søknadsprosess eller vurdering fra statens side. Vi fjerner ikke tiltak som jobbsøkerkurs og økonomisk veiledning, ei heller bidrag til rullestoler, proteser, osv. Bidragene kunne vært satt til omtrent 300 TNOK per år, men det eksakte beløpet er en subjektiv vurdering. I hvilken grad barn, studenter og pensjonister omfattes av ordningen kan diskuteres. Det enkleste er å gi bidrag til disse gruppene og fjerne eksisterende støtteordninger.

Formuesskatt

Denne delen av artikkelen er uten sammenheng med resten, som i hovedsak handler om inntekt og bidrag, men la oss se på formueskatt også.

Den marginale formueskatten har to trinn: ett på 1.76 MNOK (1 %) og ett på 20.7 MNOK (1.1 %). Den er med andre ord langt mindre progressiv enn inntektsskatt, og den slår inn relativt tidlig. Effekten er nedjustert av verdsettingsrabatter; eksempelvis har primærbolig en rabatt på hele 75 % mens aksjer har en rabatt på 20 %.

De rike blir rikere på grunn av renters-rente effekten. Den brede aksjeindeksen S&P 500 har hatt en gjennomsnittlig inflasjonsjustert avkastning på 7.65 % de siste 30 årene. En rik person med 100 MNOK investert vil derfor få 7.65 MNOK i avkastning. Anta at denne avkastningen realiseres og beskattes med 3.46 MNOK (med $T(x)$) slik at personen står igjen med en gevinst på 4.2 MNOK. Formuen er nå 100 + 4.2 = 104.2 MNOK, og formueskatten blir 1.1 MNOK, så personen står igjen med 103.1 MNOK. Ved å sette pengene i en bred indeks som S&P 500, som ikke krever noe arbeid eller oppfølging, kan personen hente ut 3.1 MNOK per år. Formuen vil ha samme avkastning hvert eneste år.

Figuren nedenfor viser den marginale formueskatten og simuleringer av ulike formuer i S&P 500. Formuene vokser lineært på log skala, altså eksponentielt.

Dagens marginale formueskatt er i praksis flat, og da vil formuene enten vokse evig (om formueskatten er lav) eller gå mot null (om formuesskatten er høy). Ved å konstruere en marginal formuesskatt som er mer progressiv kan vi begrense de rikes formuer.

La $y$ være formue. Ved å bruke en sigmoid $z = 1 / (1 + e^{-z})$ på log-skala kan vi konstruere en formuesskatt:

\begin{align*} W'(y) = \frac{H}{1 + (y / y_0)^{-k}} \qquad W(y) = \frac{H y^{1 + k}}{(k+1) y_0^k} {_2F_1}\left(1, 1 + 1/k, 2 + 1/k, -(y/y_0)^k\right) \end{align*}

Parameteren $H$ er maksimal marginal formuesskatt, $y_0$ er den formuen som gir $W'(y_0) = H/2$ og $k > 0$ bestemmer hvor raskt sigmoid-funksjonen vokser. Den marginale formueskatten $W'(y)$ er gitt av en relativt uskyldig formel, mens $W(y)$ er uttrykt ved hjelp av en hypergeometrisk funksjon ${_2F_1}$ (se appendiks). Man trenger ikke å vite noe om hypergeometriske funksjoner for å bruke eller forstå denne funksjonen.

Den progressive marginale formuesskatten $W'(y)$ er vist i figuren nedenfor, og sammenlignet med veksten til ulike formuer. Parametrene er satt ut fra øyemål for å vise effekten på en tydelig måte. Merk at veksten til ekstremt store formuer begrenses, mens mindre formuer får vokse.

Med $W(y)$ betaler alle formuesskatt, men de fleste betaler veldig lite. En formue på 100 TNOK gir 19 kr i formueskatt, 1 MNOK gir 1200 kr, 10 MNOK gir 64 TNOK og 100 MNOK gir 2.2 MNOK.

Alle eksemplene i denne seksjonen antar at formuen er likvid. Dette er ikke tilfellet dersom formuen består av aksjer i selskaper som ikke er på børs, eller om formuen er i eiendom. Formuesskatt kan føre til at man mister kontrollen over sitt eget selskap eller mister sin egen bolig. En bolig gir bruksverdi til eieren på en annen måte enn et selskap, så argumentet for formuesskatt på eiendom er sterkere enn argumentet for formuesskatt på illikvide aksjer. Dersom vi verken har formueskatt eller arveskatt, kan en rik familie sette formuen i aksjemarkedet og oppnå eksponentiell vekst for all fremtid.

Oppsummering

I dette tankeeksperimentet endret vi skattesystemet ved å innføre to funksjoner: $T(x)$ for skatt og $B(x)$ for bidrag. Parametrene ($H$, $k$ og $b$) eliminerer kompleksiteten i skatteberegninger, mange av NAVs støtteordninger og en rekke andre overføringsmekanismer. Den generelle idéen om borgerlønn og negativ inntektsskatt er gammel, mens de spesifikke funksjonene og deres egenskaper er mitt eget bidrag. Les gjerne Civita-notat nr. 38 om borgerlønn for en grundig gjennomgang av borgerlønn i norsk sammenheng.

Økonomiske insentiver som Finnmarksfradraget i skatteberegningen og sletting av gjeld for beboere i Finnmark kan modelleres ved å velge litt andre parametre (f.eks. $b$ eller $C$) i Finnmark. Her bør staten vise måtehold, ettersom mye av poenget forsvinner dersom man får en eksplosjon i antall parametre. Vi viste også at en sigmoid-funksjon $W(y)$ som marginal formueskatt begrenser ekstrem formuesvekst.

Noen av fordelene med et slikt system er:

Å øke maksimal marginalskatt $H$ kan gi skattelette til omtrent 80 % av befolkningen.
Skatteoptimaliseringsmuligheter som primært kommer de rike til gode fjernes.
Bidragsfeller som i dag fanger mange NAV-brukere blir eliminert.
Den administrative besparelsen ville vært stor. Tusenvis av årsverk kuttes og borgere slipper søknadsprosesser.

Det er imidlertid et tankeeksperiment og en øvelse i matematikk, ikke et gjennomtenkt politisk forslag. Mange av kompleksitetene vi ønsker å fjerne reflekterer legitime forsøk på å balansere konkurrerende samfunnsmål. Noen vil mene at det er moralsk riktig å gi støtte ut fra hvem som fortjener det eller ikke, selv om det koster å gjennomføre slike vurderinger. En mer aggressiv formueskatt kan utløse kapitalflukt. Mange ville brukt økonomisk trygghet til å investere i utdanning, starte bedrifter eller ta seg av familie. Andre ville potensielt redusert arbeidsinnsatsen eller i verste fall ty til destruktiv atferd. En implementering ville krevd utredninger av atferdsresponser, overgangsordninger og konsekvenser. Men hvis det er et land som har råd til å utrede, så er det Norge.

Referanser og videre lesing

Utbetalinger til personer i Norge - Statistikk for 2024 - NAV
Årsrapport 2024 - NAV
Nasjonalbudsjettet 2025 - Regjeringen
Hvorfor snakker vi ikke om borgerlønn? - NRK Ytring (2014)
Borgerlønn - Civita-notat nr. 38 (2017) 50 tips som kan kutte skatten din - smartepenger.no
Negative income tax - Wikipedia
Is There Empirical Evidence on How the Implementation of a Universal Basic Income (UBI) Affects Labour Supply? A Systematic Review - Artikkel

Appendiks A: Bevis for at bidragsfunksjonen har gode insentiver

I dette appendikset skal vi vise at når vi velger funksjoner

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k} \qquad \text{og}\qquad B(x) = Abe^{-x/b} + C \end{align*}

med $0 \leq H < 1$, $b > 0$, $k > 0$ og $A > 0$, så er har vi alltid gode insentiver fordi

\begin{align*} T'(x) - B'(x) < 1 \end{align*}

for alle $x > 0$ dersom $Hkb < A$ og $0 < A \leq 1$. Det naturlige er å velge $A=1$, fordi da synker bidraget raskest og det blir billigst for staten.

Bevis. Vi deriverer funksjonene og setter dem inn i ulikheten. Da får vi

\begin{align*} f(x) &= -T'(x) + B(x) + 1 > 0 \\ f(x) &= H\left(e^{-kx} - 1 \right) - A e^{-x/b} + 1 > 0. \end{align*}

Vi ser også at

\begin{align*} f'(x) &= -Hk e^{-kx} + \frac{A}{b} e^{-x/b} \\ f'(x) &= 0 \, \Rightarrow \, x (k - \frac{1}{b}) = \ln\left( \frac{Hkb}{A} \right). \end{align*}

Dersom $f(x)$ er positiv og økende når $x=0$, konvergerer mot et positivt tall og har maksimalt ett ekstremalpunkt når $x > 0$, så er $f(x) > 0$ for alle $x > 0$ og ulikheten er gyldig.

Vi observerer at:

$f(0) = -A + 1 \geq 0 \Rightarrow A \leq 1$
$f'(0) = -Hk + A/b > 0 \Rightarrow A > Hkb$
$\lim_{x \to \infty} f(x) = 1 - H > 0$
$f(x)$ har ett ekstremalpunkt fordi $f'(x) = 0$ har én løsning

Dette gir oss kravene (1) $A \leq 1$ og (2) $A > Hkb$. I artikkelen valgte vi $A=1$, og da blir kravet for $b$ at $b < 1/(Hk)$.

Appendiks B: Formuesskatten

I dette appendikset skal vi utlede formuesskatten $W(y)$ fra den marginale formuesskatten $W'(y)$. Vi definerte den marginale formuesskatten som

\begin{align*} W'(y) = \frac{H}{1 + (y / y_0)^{-k}} = \frac{H}{1 + \exp(-k \ln (y / y_0))}. \end{align*}

Målet er å evaluere integralet

\begin{align*} \int_0^y W'(z) \, dz = H \int_0^y \frac{1}{1 + \exp(-k \ln (z / y_0))} \, dz \end{align*}

for å finne den totale formuesskatten ved et beløp $Y$. Først gjør vi substitusjonen $u = \ln(z/y_0)$, som transformerer integralet til

\begin{align*} H y_0 \int_{-\infty}^{\ln(y/y_0)} \frac{e^u}{1 + e^{-ku}} \, du = H y_0 \int_{-\infty}^{\ln(y/y_0)} \frac{e^u e^{ku}}{1 + e^{ku}} \, du. \end{align*}

Deretter gjør vi substitusjonen $z = e^{ku}$, som transformerer integralet til

\begin{align*} \frac{H y_0}{k} \int_{0}^{(y/y_0)^k} \frac{z^{1/k}}{1 + z} \, dz \end{align*}

og dette er et kjent integral som kan formuleres som en hypergeometrisk funksjon. Vi kan bruke identiteten

\begin{align*} \int \frac{x^{1/k}}{1 + x} \, dx = \frac{k x^{1 + 1/k}}{k+1} \, {_2F_1}\left( 1, 1 + 1/k, 2 + 1/k, -x \right) \end{align*}

som gir oss vårt endelig svar:

\begin{align*} W(y) = \int_0^y W'(z) \, dz = \frac{H y^{1 + k}}{(k+1) y_0^k} \, {_2F_1}\left(1, 1 + 1/k, 2 + 1/k, -(y/y_0)^k\right) \end{align*}

Appendiks C: Kode

Her er Python-kode som implementerer hovedidéene i artikkelen.

import dataclasses  # Python 3.12
from abc import ABC, abstractmethod
import numpy as np  # 2.3.3
import scipy as sp  # 1.16.2


class Transfer(ABC):
    """Abstract base class for transfer functions (taxes, benefits, etc.)."""

    @abstractmethod
    def marginal(self, x):
        """Marginal rate at income x."""
        pass

    @abstractmethod
    def absolute(self, x):
        """Absolute value at income x."""
        pass

    def average(self, x):
        """Average rate: absolute(x) / x."""
        return self.absolute(x) / x


@dataclasses.dataclass
class Tax(Transfer):
    """Class representing a tax function T(x).

    Examples
    --------
    >>> T = Tax(k=2.787914e-06, H=0.474)
    >>> T.absolute(500_000)  # Current norwegian tax system: 113643.99
    np.float64(109160.85121307125)
    >>> T.absolute(1_000_000)  # Current norwegian tax system: 313248.14
    np.float64(314445.0244402999)
    """

    # Defaults that give best-fit to current tax system in 2025,
    # weighted by income of population.
    k: float = 2.8745e-06
    H: float = 0.4772

    def __post_init__(self):
        assert self.k > 0
        assert self.H > 0

    def marginal(self, x):
        """Marginal tax rate T'(x)."""
        H, k = self.H, self.k
        return H - H * np.exp(-k * x)

    def absolute(self, x):
        """Absolute value of tax T(x)."""
        H, k = self.H, self.k
        return (H / k) * (np.exp(-k * x) + k * x - 1)


@dataclasses.dataclass
class Benefit(Transfer):
    """Class representing a benefit function B(x).

    Examples
    --------
    >>> B = Benefit(300_000)
    >>> B.absolute(0)
    np.float64(300000.0)
    >>> B.absolute(300_000)  # exp(-1) * 300_000 = 110363
    np.float64(110363.8323514327)
    """

    b: float = 271_000  # Somewhat realistic value

    def __post_init__(self):
        assert self.b > 0

    def marginal(self, x):
        """Marginal benefit rate B'(x)."""
        return -np.exp(-x / self.b)

    def absolute(self, x):
        """Absolute value of benefits B(x)."""
        return self.b * np.exp(-x / self.b)


@dataclasses.dataclass
class WealthTax(Transfer):
    """Class representing a wealth tax function W(y).

    Uses a logistic-based wealth tax with hypergeometric function
    for absolute tax calculation.

    Examples
    --------
    >>> W = WealthTax(y_0=10_000_000, H=0.03, k=1.0)
    >>> W.marginal(10_000_000)  # At threshold, marginal rate is H/2
    np.float64(0.015)
    >>> W.average(2_000_000)  # Average tax on 2 MNOK wealth
    np.float64(0.0026517664809068064)
    >>> W.average(20_000_000)  # Average tax on 20 MNOK wealth
    np.float64(0.013520815669978353)
    """

    y_0: float = 10_000_000  # Wealth threshold
    H: float = 0.03  # Maximum marginal rate
    k: float = 1  # Steepness parameter

    def __post_init__(self):
        assert 0 <= self.H < 1
        assert self.y_0 > 0
        assert self.k > 0

    def marginal(self, y):
        """Marginal wealth tax rate W'(y)."""
        y_0, H, k = self.y_0, self.H, self.k
        return H / (1 + np.exp(-k * np.log(y / y_0)))

    def absolute(self, y):
        """Absolute value of wealth tax W(y)."""
        y_0, H, k = self.y_0, self.H, self.k
        power = (y / y_0) ** k
        factor = H * y * power / (k + 1)
        return factor * sp.special.hyp2f1(1, 1 + 1 / k, 2 + 1 / k, -power)


if __name__ == "__main__":
    import pytest

    pytest.main(["--doctest-modules", __file__])

    def numerical_derivative(func, x, h=1e-4):
        return (func(x + h) - func(x - h)) / (2 * h)

    # Test that derivatives are correct for all classes with defaults
    for cls in [Tax, Benefit, WealthTax]:
        instance = cls()
        x = np.logspace(3, 7)
        analytical = instance.marginal(x)
        numerical = numerical_derivative(instance.absolute, x)
        assert np.allclose(numerical, analytical, rtol=0.01)

Practical Linear Algebra with NumPy

Tommy Odland — Fri, 30 Jan 2026 00:00:00 +0100

Whether you’re solving differential equations, training neural networks, inferring parameters in a statistical model or solving optimization problems, you’ll find linear algebra at the core of almost every routine. Still, mistakes in the application of linear algebra are all too common. Even though it’s a novice numerical linear algebra error, I’ve seen professors solve $A \boldsymbol{x} = b$ as b = np.linalg.inv(A) @ x and call it a day.

Here’s a list of beginner linear algebra mistakes, followed by some tips for writing better and faster numerical code. Remember to start with a correct implementation before working on speed.

Don’t compare floats
Don’t multiply diagonal matrices
Don’t add diagonal matrices
Don’t multiply matrices in arbitrary order
Don’t compute a product if you only need the diagonal
Don’t form the inverse matrix
Don’t use memory you don’t need
Don’t use pure Python loops
Do use the Woodbury identity
Do think about what the meaning is
Do time everything
Do read the documentation
Do consider approximations
Resources and further reading

Don’t compare floats

Floats are an approximation of real numbers, and comparing them bit-by-bit is too stringent:

np.power(np.sqrt(13), 2) == 13.0  # False
np.isclose(np.power(np.sqrt(13), 2), 13.0)  # True

A = np.random.randn(10, 10)
Q, R = np.linalg.qr(A)
np.all(Q @ R == A)  # False
np.allclose(Q @ R, A)  # True

Use numpy.isclose and numpy.allclose in general, and consider numpy.testing.assert_allclose for testing.

Don’t multiply diagonal matrices

The multiplication $A \operatorname{diag}(\boldsymbol{v})$ is just a column-wise scaling of $A$ by $\boldsymbol{v}$. Similarly, $\operatorname{diag}(\boldsymbol{v}) A$ is a row-wise scaling. Implement the scaling directly instead of multiplying with the diagonal matrix, since using a diagonal matrix mostly does unnecessary work by multiplying the entries of $A$ with zeros.

import numpy as np
import scipy as sp

A = np.random.randn(1000, 500)
v = np.random.randn(500)

A @ np.diag(v)  # 6.3 ms ± 1.01 ms per loop
A * v  # 369 µs ± 35.9 µs per loop

assert np.allclose(A @ np.diag(v), A * v)

Don’t add diagonal matrices

Don’t explicitly form $I$ when computing $A + I$. If $A$ has size $n \times n$, then you’re adding $n$ ones and $n^2 - n$ unnecessary zeroes. Instead use numpy.fill_diagonal and do it like this:

A = np.random.randn(1000, 1000)
I = np.eye(1000)

A + I  # 1.07 ms ± 68.2 µs per loop
np.fill_diagonal(A, A.diagonal() + 1)  # 8.66 μs ± 141 ns per loop

This trick can be used to speed up adding arbitrary diagonal matrixes too.

Don’t multiply matrices in arbitrary order

Matrix multiplication is associative, so the expressions $(AB)\boldsymbol{x}$ and $A(B\boldsymbol{x})$ evaluate to the same value—but the computational cost is not the same. Your can either work out the optimal ordering by hand, or let numpy.linalg.multi_dot do the job for you.

A = np.random.randn(1000, 500)
B = np.random.randn(500, 2000)
x = np.random.randn(2000)

(A @ B) @ x  # 18.8 ms ± 1.11 ms per loop
A @ (B @ x)  # 378 µs ± 114 µs per loop
A @ B @ x  # 19 ms ± 766 µs per loop
np.linalg.multi_dot([A, B, x])  # 378 µs ± 110 µs per loop

Consider another example: computing $A \boldsymbol{v} \boldsymbol{v}^T A$. Here numpy.linalg.multi_dot won’t be of much help, since $\boldsymbol{v}$ and $\boldsymbol{v}^T$ are the same 1 dimensional array in NumPy. Should we implement the computation as $A (\boldsymbol{v} \boldsymbol{v}^T) A$ or $(A \boldsymbol{v}) (\boldsymbol{v}^T A)$?

A = np.random.randn(3000, 3000)
v = np.random.randn(3000)

A @ np.outer(v, v) @ A  # 910 ms ± 61.6 ms per loop
np.outer(A @ v, A.T @ v)  # 41.5 ms ± 3.95 ms per loop

assert np.allclose(A @ np.outer(v, v) @ A, np.outer(A @ v, A.T @ v))

In the first approach we perform one outer product and two matrix multiplications.
In the second approach we perform one outer product and two matrix-vector multiplications.

If $A \in \mathbb{R}^{m \times n}$ and $\boldsymbol{v} \in \mathbb{R}^{n}$, then computing the matrix-vector multiplication $A \boldsymbol{v}$ uses $\mathcal{O}(mn)$ operations. If $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times k}$, then computing the matrix-matrix multiplication $A B$ uses $\mathcal{O}(mnk)$ operations. So a matrix-vector multiplication is $\mathcal{O}(k)$ faster than a matrix-matrix multiplication.

Don’t compute a product if you only need the diagonal

Consider computing $\operatorname{diag}(A B)$. Forming the full product $A B$ only to extract the diagonal is wasteful. Instead, compute the diagonal of the product as (B.T * A).sum(axis=1) in NumPy:

A = np.random.randn(5000, 10000)
B = np.random.randn(10000, 5000)

np.diag(A @ B)  # 3.55 s ± 151 ms per loop
(B.T * A).sum(axis=1)  # 188 ms ± 5.44 ms per loop

assert np.allclose(np.diag(A @ B), (B.T * A).sum(axis=1))

Don’t form the inverse matrix

There are several reasons why you don’t invert that matrix: it’s slow and it’s not numerically stable. Instead of explicitly solving $A\boldsymbol{x} = b$ by forming and multiplying by $A^{-1}$, use a solver.

A = np.random.randn(2000, 2000)
x = np.random.randn(2000)
b = A @ x

np.linalg.inv(A) @ x  # 355 ms ± 52.5 ms per loop
np.linalg.solve(A, b)  # 86.7 ms ± 8.33 ms per loop

Now let’s look at a slightly more advanced computation, which combines the idea of not forming inverses with the idea of not computing products in arbitrary order. We want to compute the product $A B^{-1} C$, where $A \in \mathbb{R}^{m \times n}$, $B \in \mathbb{R}^{n \times n}$ and $C \in \mathbb{R}^{n \times k}$.

If $m > k$, we should compute $A (B^{-1} C)$, which uses $\mathcal{O}( kn(m+n) )$ floating point operations.
If $k > m$, we should compute $(A B^{-1}) C$, which uses $\mathcal{O}( mn(n +k) )$ floating point operations.

Below we solve the case when $k \gg m$ without forming the inverse.

m, n, k = 10, 1000, 100_000

A = np.random.randn(m, n)
B = np.random.randn(n, n)
C = np.random.randn(n, k)

(A @ np.linalg.inv(B)) @ C  # 233 ms ± 56.4 ms per loop
A @ (np.linalg.inv(B) @ C)  # 2.32 s ± 145 ms per loop

np.linalg.solve(B.T, A.T).T @ C  # 157 ms ± 7.29 ms per loop
A @ np.linalg.solve(B, C)  # 2.71 s ± 174 ms per loop

Don’t use memory you don’t need

If you can, use inplace operations to avoid filling up the memory with unnecessary arrays:

v = np.random.randn(100)
v -= 1  # Mutate v without creating any new arrays
v = v - 1  # Create a new array (v - 1) in memory, assign it to v

Inplace operations are also typically a bit faster. Win-win.

Don’t use pure Python loops

Avoid pure Python loops in numerical code. If the obvious approach is to loop, first try to vectorize the computation. If that fails, look into Numba or Cython.

Here’s an example. The code below computes the empirical cross-covariance between two data sets. Each variable is a column, each row is a set of observations.

def covariance_loop(A, B):
    A = A - np.mean(A, axis=0)
    B = B - np.mean(B, axis=0)
    cov = np.zeros((A.shape[1], B.shape[1]))

    for j in range(A.shape[0]):
        cov += np.outer(A[j, :], B[j, :])

    return cov / (A.shape[0] - 1.0)


A = np.random.randn(1000, 1000)
B = np.random.randn(1000, 2000)

covariance_loop(A, B)  # 3.62 s ± 241 ms per loop

The code above computes an outer product plus a matrix-matrix addition. This is done for each row.

Although the asymptotic complexity is the same, the code below is much faster because it eliminates the Python loop and expresses the computation as a single matrix-matrix product instead.

def covariance(A, B):
    A = A - np.mean(A, axis=0)
    B = B - np.mean(B, axis=0)

    return A.T @ B / (A.shape[0] - 1.0)


covariance(A, B)  # 48.3 ms ± 8.75 ms per loop

Do use the Woodbury identity

Learn to use the Woodbury matrix identity and related formulas. The Woodbury equation is:

\begin{equation} \left(A + UCV \right)^{-1} = A^{-1} - A^{-1}U \left(C^{-1} + VA^{-1}U \right)^{-1} VA^{-1} \end{equation}

If $P$ and $R$ are positive definite, then the following holds:

\begin{equation} \left(P^{-1} + B^T R^{-1} B \right)^{-1} B^T R^{-1} = P B^T (B P^T B + R)^{-1} \end{equation}

A typical use case is when we have inverted $A$, and want to compute the inverse of a rank 1 update of $A$. In other words we wish to compute $\left(A + \boldsymbol{v}\boldsymbol{v}^T \right)^{-1}$, and we already have $A^{-1}$. This is a special case of the Woodbury matrix identity, called the Sherman–Morrison formula:

A = np.random.randn(1000, 1000)
v = np.random.randn(1000)
A_inv = np.linalg.inv(A)

np.linalg.inv(A + np.outer(v, v))  # 50.2 ms ± 6.32 ms per loop


def rank1update(A_inv, v):
    return A_inv - np.outer(A_inv @ v, v.T @ A_inv) / (1 + v.T @ A_inv @ v)

rank1update(A_inv, v)  # 8.61 ms ± 2.9 ms per loop

Do think about what the meaning is

Just like $A \operatorname{diag}(\boldsymbol{v})$ simply scales the columns of $A$ with the entries of $\boldsymbol{v}$, more tangled expressions occasionally have simple interpretations. Consider the expression $A(I - \frac{1}{N}\boldsymbol{1}\boldsymbol{1}^T)$. Looks fancy, right? It simply subtracts the mean from each row. Instead of forming $I$, then forming $\boldsymbol{1}\boldsymbol{1}^T$, performing the subtraction, multiplying, etc—simply implement the row scaling directly.

If you’re working in a thousand dimensions, its easy to get lost. Work out simple examples in three or four dimensions before generalizing.

A = np.random.randn(1000, 1000)
factor = np.eye(1000) - np.outer(np.ones(1000), np.ones(1000)) / 1000

A @ factor  # 25 ms ± 9.58 ms per loop
A - np.mean(A, axis=1, keepdims=True)  # 1.61 ms ± 60.4 µs per loop

assert np.allclose(A @ factor, A - np.mean(A, axis=1, keepdims=True))

Here’s another example from the wild. I once saw someone implement

v = np.exp(np.random.randn(5_000))
sp.linalg.cholesky(np.diag(v**2))  # 724 ms ± 99.9 ms per loop

because that’s what a paper said. But the Cholesky decomposition of a diagonal matrix is just the square of the diagonal, so the entire factorization is a waste of time—it does nothing at all.

Do time everything

We’ve been using the ubiquitous %timeit Ipython magic command in almost every example so far. If you’re in doubt about whether one approach or the other is better, then implement both and time them.

Consider solving the Ridge regression optimization problem

\begin{equation} \underset{\boldsymbol{\beta}}{\operatorname{minimize}} \quad \lVert X \boldsymbol{\beta} - \boldsymbol{y} \rVert^2_W + \lVert \boldsymbol{\beta} \rVert^2, \end{equation}

where $\lVert \boldsymbol{x} \rVert^2_W = \boldsymbol{x}^T W \boldsymbol{x}$ denotes the weighted squared norm. We assume that $W$ is a diagonal matrix with entries given by a vector $\boldsymbol{w}$.

We will examine three approaches to solving this problem.

The first approach is to differentiate the objective function and set it equal to zero. We obtain the normal equations $(X^T W X + I) \boldsymbol{\beta} = X^T W \boldsymbol{y}$. Forming the normal equations and solving could look like this:

X = np.random.randn(10_000, 1000)
y = np.random.randn(10_000)
w = np.exp(np.random.randn(10_000))

def solve_normal_eqns(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""
    XT_W = X.T * w  # Form X.T @ diag(w)
    lhs = XT_W @ X
    lhs.flat[:: lhs.shape[0] + 1] += 1  # Add identity
    rhs = XT_W @ y
    return sp.linalg.solve(
        lhs,
        rhs,
        assume_a="pos",
        overwrite_a=True,
        overwrite_b=True,
    )

solve_normal_eqns(X, w, y)  # 321 ms ± 97.2 ms per loop

The second approach is to realize that

\begin{equation} \lVert X \boldsymbol{\beta} - \boldsymbol{y} \rVert^2_W + \lVert \boldsymbol{\beta} \rVert^2 = \lVert W^{1/2} X \boldsymbol{\beta} - W^{1/2} \boldsymbol{y} \rVert^2 + \lVert \boldsymbol{\beta} \rVert^2 = \left\lVert \begin{bmatrix} W^{1/2} X \boldsymbol{\beta} - W^{1/2} \boldsymbol{y} \\ \boldsymbol{\beta} \end{bmatrix} \right\rVert^2 = \left\lVert \begin{bmatrix} W^{1/2} X \\ I \end{bmatrix} \boldsymbol{\beta} - \begin{bmatrix} \boldsymbol{y} \\ \boldsymbol{0} \end{bmatrix} \right\rVert^2 \end{equation}

and make a call to a least squares solver with these matrices directly.

def solve_lstsq(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""
    squared_w = w**0.5
    lhs = np.vstack(((X.T * squared_w).T, np.eye(X.shape[1])))
    rhs = np.hstack((squared_w * y, np.zeros(X.shape[1])))
    beta, *_ = sp.linalg.lstsq(
        lhs,
        rhs,
        overwrite_a=True,
        overwrite_b=True,
    )
    return beta

solve_lstlsq(X, w, y)  # 1.34 s ± 429 ms per loop

The third approach uses the singular value decomposition (SVD). Let $W^{1/2}X = U D V^T$, then the factor $(X^T W X + I)$ in the normal equations may be written as

\begin{equation} (X^T W X + I) = (W^{1/2}X)^T (W^{1/2}X) + I = (U D V^T)^T (U D V^T) + I = VD^2V^T + VV^T = V (D^2 + I) V^T \end{equation}

and $(X^T W X + I)^{-1} = V^T (D^2 + I)^{-1} V$. Plugging this back into the normal equations, we obtain the solution as

\begin{equation} \boldsymbol{\beta} = (X^T W X + I)^{-1} X^T W \boldsymbol{y} = V (D^2 + I)^{-1} D U^T W^{1/2} \boldsymbol{y}. \end{equation}

In the implementation, we exploit the fact that $(D^2 + I)^{-1}$, $D$ and $W^{1/2}$ are diagonal matrices:

def solve_svd(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""
    w_sq = w**0.5
    W_sq_X = (X.T * w_sq).T  # Form W^1/2 @ X
    U, d, VT = sp.linalg.svd(W_sq_X, full_matrices=False, overwrite_a=True)
    inv_diag = d / (1 + d**2)  # Form (D^2 + I)^-1 @ D
    return np.linalg.multi_dot([VT.T * inv_diag, U.T, (w_sq * y)])

solve_svd(X, w, y)  # 2.47 s ± 329 ms per loop

The approaches return the same answer:

assert np.allclose(solve_normal_eqns(X, w, y), solve_lstsq(X, w, y))
assert np.allclose(solve_normal_eqns(X, w, y), solve_svd(X, w, y))

The third approach using the SVD is what you might find in a paper. However, sklearn implements approach one, which is significantly faster. Under the hood, scipy.linalg.solve calls the LAPACK routine sposv, which uses a Cholesky decomposition. It’s typically instructive to implement a few different approaches, then test them and time them. We have not discussed the numerical stability of the least-squares problem here, but have a look at the references at the end of this article for more information about that issue.

Do read the documentation

Consider again the function solve_normal_eqns(X, w, y) that we used to solve the normal equations $(X^T W X + I) \boldsymbol{\beta} = X^T W \boldsymbol{y}$ with. It turns out that scipy.linalg.solve(a, b) has a parameter lower=False, and the documentation reads:

If False (default), the calculation uses only the data in the upper triangle of a; entries below the diagonal are ignored.

It also turns out that there’s a BLAS routine with the cryptic name dsyrk that only computes the upper-triangular part of $\alpha A A^T$.

We can improve the speed of our solver by only computing the upper-triangular part of $X^T W X + I$ as follows:

def solve_normal_eqns2(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""

    # Compute upper triangular entries in X.T @ diag(w) @ X
    XT_W_X = sp.linalg.blas.dsyrk(alpha=1, a=X.T * w**0.5)

    # Add identity matrix
    XT_W_X.flat[:: XT_W_X.shape[0] + 1] += 1

    return sp.linalg.solve(
        XT_W_X,
        X.T @ (w*y),
        assume_a="pos",
        lower=False, # Solver only uses upper triangular elements
        overwrite_a=True,
        overwrite_b=True,
    )

X = np.random.randn(10_000, 5000)
y = np.random.randn(10_000)
w = np.exp(np.random.randn(10_000))

solve_normal_eqns(X, w, y)  # 4.78 s ± 393 ms per loop
solve_normal_eqns2(X, w, y)  # 3.33 s ± 436 ms per loop

If you’re working with linear algebra, familiarize yourself with all the functions in scipy.linalg. When using a function, read carefully through the arguments. If speed and accuracy truly matters, look into how functions are implemented and research the specific problem you’re working on carefully.

Do consider approximations

Sometimes we don’t need exact answers. For instance, we can approximately solve the Ridge regression problem by running five Conjugate Gradient iterations on the normal equations.

X = np.random.randn(10_000, 1000)
y = np.random.randn(10_000)
w = np.exp(np.random.randn(10_000))
XT_W = X.T * w


def matvec(v):
    # Implements (X.T @ diag(v) @ X + I) @ v
    return np.linalg.multi_dot([XT_W, X, v]) + v


A = sp.sparse.linalg.LinearOperator((X.shape[1], X.shape[1]), matvec=matvec)

# 56.9 ms ± 13 ms per loop
approx, _ = sp.sparse.linalg.cg(A, XT_W @ y, tol=0.0, maxiter=5)

# 295 ms ± 55.1 ms per loop
exact = solve_normal_eqns(X, w, y)

The Pearson correlation between the exact and approximate answer is approximately $0.998$. What we gain in speed might make up for the loss of accuracy, but this is highly problem dependent. An approximate singular value decomposition can be obtained using sklearn.utils.extmath.randomized_svd.

Resources and further reading

Three excellent books:

Numerical Linear Algebra by Trefethen et al (1997).
Matrix Computations by Golub et al (2013).
Accuracy and Stability of Numerical Algorithms by Nicholas J. Higham (2002).

Two reference manuals:

An open-source book on vectorization:

From Python to Numpy

Books

Tommy Odland — Mon, 15 Dec 2025 00:00:00 +0100

Here’s an inspirational, and perhaps true, quote by Stephen Boyd:

When you end up being trained in just a handful of areas, you can be unbelievably effective across fifty fields. If you know linear algebra, optimization, probability and statistics, and computer science – those are the topics. You don’t have to know anything else.

If you know those things, and really know them, and if you’ve seen a bunch of applications across different fields, you are so valuable it’s crazy. You can be hired instantly into a hundred fields. But here’s the even cooler thing, and this is the Silicon Valley part of it: you can create new fields that don’t even exist. Please train yourself broadly. Learn these things.

Below is a reading list for aspiring data scientists and others interested in mathematical modeling, statistics, optimization, scientific programming, etc.

Only my favorite books are included. Books I would recommend.
Only general topics useful to many practitioners are included. Category theory, game theory and geostatistics is cool – but not that useful to the average practitioner.
I have read the majority of the material, but not everything.
The resources are sorted by difficulty within reach category, but only roughly.

The nomenclature is:

📗 Book.
📄 Paper.
📺 Video lecture(s).
🤖 Book contains many code examples.
⭐ Life changing.

Mathematics

📗 Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares by Boyd et al (2018).
📗 Advanced Engineering Mathematics by Michael Greenberg (1998).
📗 ⭐ Introduction to Applied Mathematics by Gilbert Strang (1986).
📗 The Princeton Companion to Mathematics by Gowers et al (2008).
📄 An Introduction to the Conjugate Gradient Method Without the Agonizing Pain by Jonathan Richard Shewchuk (1994).

Scientific computing

📗 Numerical Methods for Scientists and Engineers by R. W. Hamming (1987).
📗 🤖 From Python to Numpy by Nicolas P. Rougier (2017).
📗 Numerical Linear Algebra by Trefethen et al (1997).
📗 Matrix Computations by Golub et al (2013).

Python and programming

📗 The Pragmatic Programmer by Hunt et al (2019).
📗 A Philosophy of Software Design by John Ousterhout (2021).
📗 🤖 Fluent Python: Clear, Concise, and Effective Programming by Luciano Ramalho (2015).
📗 🤖 Effective Python: 90 Specific Ways to Write Better Python by Brett Slatkin (2019).
📺 So you want to be a Python expert? by James Powell (2017).
📺 Transforming Code into Beautiful, Idiomatic Python by Raymond Hettinger (2013).
📄 Good enough practices in scientific computing by Wilson et al (2017).
📄 No Silver Bullet - Essence and Accident in Software Engineering by Fred Brooks (1986).

Machine learning

📗 Artificial Intelligence: A Modern Approach by Norvig et al (2009).
📗 Pattern Classification by Duda et al (2000).
📗 Pattern Recognition and Machine Learning by Christopher M. Bishop (2006).
📗 The Elements of Statistical Learning by Hastie et al (2016).
📄 A Few Useful Things to Know about Machine Learning by Pedro Domingos (2012).
📄 Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead by Cynthia Rudin (2018).
📄 Factorization Machines by Steffen Rendle (2010).
📄 Machine Learning: The High Interest Credit Card of Technical Debt by Sculley et al (2014).

Deep learning etc.

📗 🤖 Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron (2022).
📗 Reinforcement Learning by Sutton (2018).
📺 DeepMind x UCL | Reinforcement Learning Course by David Silver (2015).

Statistics

📗 Statistical Inference by Casella et al (2001).
📗 Regression and Other Stories by Gelman et al (2020).
📗 Data Analysis Using Regression and Hierarchical Models by Gelman et al (2006).
📗 All of Statistics by Larry Wasserman (2010).
📗 🤖 ⭐ Statistical Rethinking by Richard McElreath (2020).
📺 Statistical Rethinking by Richard McElreath (2022).
📄 The garden of forking paths by Gelman et al (2019).
📄 Statistical Modeling: The Two Cultures by Leo Breiman (2001).
📄 Hierarchical generalized additive models in ecology by Pedersen et al (2019).
📄 What are the most important statistical ideas of the past 50 years? by Gelman et al (2021).
📄 A Conceptual Introduction to Hamiltonian Monte Carlo by Michael Betancourt (2018).

Optimization

📗 The Basics of Practical Optimization by Adam B. Levy (2022).
📗 Optimization Modelling: A Practical Approach by Sarker et al (2007).
📗 Essentials of Metaheuristics by Sean Luke (2012).
📗 Linear Programming: Foundations and Extensions by Robert J. Vanderbei (2021).
📗 Convex Optimization by Boyd et al (2004).
📗 Numerical Optimization by Nocedal et al (2006).
📄 Formulating Integer Linear Programs: A Rogues’ Gallery by Brown et al (2007).
📄 Bus Routing Optimization Helps Boston Public Schools Design Better Policies by Bertsimas et al (2020).

Algorithms

📗 The Algorithm Design Manual by Steven S. Skiena (2021).
📗 ⭐ Algorithms by Dasgupta et al (2006).
📗 🤖 Competitive Programming by Halim et al.
📗 Introduction to Algorithms by Cormen et al (2022).
📗 Network Flows: Theory, Algorithms, and Applications by Ahuja et al (1993).

Popular science

📗 Flaws and Fallacies in Statistical Thinking by Stephen K. Campbell (2012).
📗 Innumeracy: Mathematical Illiteracy and Its Consequences by John Allen Paulos (2001).
📗 Science Fictions by Stuart Ritchie (2020).
📗 Concepts of Modern Mathematics by Ian Stewart (1995).
📗 Euler: The Master of Us All by William Dunham (1999).
📗 The Book of Why by Judea Pearl et al (2018).

Miscellaneous

📗 How Big Things Get Done by Flyvbjerg (2023).
📗 The Visual Display of Quantitative Information by Edward R. Tufte (2001).
📗 On Writing Well: The Classic Guide to Writing Nonfiction by William Zinsser (2016).
📗 Handbook of Writing for the Mathematical Sciences by Nicholas J. Higham (2019).
📗 How to Solve It by Polya et al (2014).
📗 The Matrix Cookbook by Pedersen et al (2012).
📗 Opt Art: From Mathematical Optimization to Visual Design by Robert Bosch (2019).
📄 School Choice: A Mechanism Design Approach by Abdulkadiroğlu et al (2003).

Fastlegebytter

Tommy Odland — Wed, 05 Nov 2025 00:00:00 +0100

Oppdatering: To dager etter at jeg skrev denne artikkelen ble jeg informert om innlegget “Enkelt å forbedre fastlegeordningen - med en algoritme” i Dagens Næringsliv. Forfatterne argumenterer for mye av det samme som det jeg gjør nedenfor. Innlegget deres er veldig godt skrevet, og de har også skrevet en 80-siders forskningsartikkel “Designing Dynamic Reassignment Mechanisms: Evidence from GP Allocation”. De beregner at 15 % (ca. 45 000 mennesker) kunne sluppet venteliste ved å endre på algoritmen!

Det er ikke uvanlig å vente i årevis på fastlegebytte.

I 2024 kunne vi lese at over 300 000 står på venteliste. På helsenorge.no er det 608 fastleger registrert i Oslo fylke, og 60 % av disse har fulle lister. En lege har typisk plass til 1 300 pasienter, og det er 816 000 plasser i Oslo fylke totalt. Kun 4,5 % av disse plassene er ledige. Situasjonen er enda verre andre steder i landet. I 2022 var det ille i Rogaland, der kun 3 % av fastlegene hadde ledige plasser. I Bergen og Trondheim var det ikke en eneste ledig plass.

Et stort problem med fulle lister er at dagens ordning ikke tillater bytter mellom pasienter på fulle lister. Dette er ineffektivt fordi det låser markedet. I denne artikkelen skal jeg forklare dette utsagnet og foreslå en løsning på problemet.

Bytter mellom fulle lister

Her er et eksempel på en situasjon der vi kunne kuttet ventetiden drastisk. Ola er på lista til lege A. Kari er på lista til lege B. Ola ønsker å bytte fra lege A til lege B. Kari ønsker å bytte motsatt vei; fra lege B til lege A.

I dag må noen forlate listen til én av legene før Ola, Kari eller Per får innvilget søknad om fastlegebytte. Men Ola og Kari hadde ikke trengt å vente (potensielt i årevis) - de kunne ha byttet plass i dag! Vi trenger bare en algoritme som leter etter sykluser i bytte-grafen og gjennomfører byttene.

Er du bekymret for Per i figuren ovenfor? Den nye algoritmen er bedre for både Ola og Kari, men Per har potensielt stått i kø lengre enn Ola. Kan Per havne i en situasjon der han må vente lengre enn med dagens algoritme? Svaret er ja - det er mulig å konstruere et slikt eksempel:

Anta at pasientene melder ønske om bytte i rekkefølgen Kari, Per og Ola. Vi står i en fastlåst situasjon som vist i figuren ovenfor, der ingen får bytte fordi det ikke er ledige plasser. Om noen dager vil lege A få en ledig plass, men det vet vi ikke enda.

Dagens algoritme. Dersom vi ikke kjører syklus-algoritmen er situasjonen fastlåst frem til lege A får en ledig plass. Da får Kari sitt ønske innvilget ettefulgt av Per, mens Ola står fremdeles i kø.
Foreslått algoritme. Dersom vi kjører syklus-algoritmen bytter Ola og Kari umiddelbart plass. Når lege A får en ledig plass påvirker det ikke Per. Han står fremdeles i kø.

Per må vente til lege B får en ledig plass som ikke umiddelbart fylles fordi den er del av en bytte-syklus. Oppsummert kan det å gjennomføre sykluser-bytter mellom fulle lister fjerne (potensielt lange) stier i grafen som ville innfridd flere ønsker.

Er dette eksempelet et godt argument mot den foreslåtten algoritmen? Jeg mener at det ikke er det. Per er kanskje misfornøyd dersom virkeligheten utspiller seg som ovenfor, men langt flere er misfornøyde med dagens regime. Realiteten er at ingen på venteliste vet om de er en del av en syklus (slik som Ola og Kari) eller ikke (slik som Per), og mange bytter ødelegger ikke for personer som Per. Markedet er i dag helt låst i byer som Bergen, der kun 3 av 274 fastleger har ledige plasser. I slike situasjoner ville syklus-bytter betydd mye for pasientene.

Lengre sykluser

Ovenfor byttet vi plass på to personer: Ola og Kari. Vi kan bytte plass på et vilkårlig antall personer så lenge de lager en syklus i bytte-grafen.

Av og til må vi ta et valg, fordi én persons bytteønske kan brukes i flere sykluser, slik som i figuren nedenfor.

I slike situasjoner kan vi enten velge den lengste syklusen (grønn), eller vi kan velge den syklusen der pasientene har ventet lengst på å få bytte. Det er et verdispørsmål, ikke et matematisk spørsmål - og uansett hva vi velger blir utfallet bedre enn i dag for de aller fleste.

Et annet valg vi må ta er om vi utfører syklus-bytter ofte (f.eks. hver dag) eller sjeldent (f.eks. hver måned). Å bytte ofte gir umiddelbar bevegelse i markedet. Å bytte litt mer sjeldent kan føre til at det bygger seg opp lengre syklusen i grafen som vi kan utnytte. En siste idé er å tillate mer enn ett ønske: hvorfor ikke la pasientene rangere flere leger som de kunne tenke seg å bytte til?

Oppsummering og referanser

På tross av målsetninger om å digitalisere og effektivisere helsesektoren, er det fremdeles noen frukter som henger så lavt at de nesten tar i bakken. Syklus-bytter hjelper ikke de som er villige til å stå i kø i fem år for å være pastient hos den beste legen i byen. Men det hjelper når Ola flytter fra en bydel til en annen og vil bytte lege, mens Kari flytter motsatt vei. Algoritmer som baserer seg på å fjerne sykluser har vært kjent siden 70-tallet, da Top Trading Cycle (TTC) først ble publisert. Selv om TTC er Pareto-effektiv, gir den ikke nødvendigvis de lengste syklusene. Optimal kidney exchange handler om å finne de lengste syklusene.

Boka Who Gets What - and Why: The New Economics of Matchmaking and Market Design av Alvin Roth er en folkelig introduksjon til markeder uten penger og uten delelige varer. Fagfeltet heter mekanismedesign, og er godt kjent gjennom Wikipedia-artikler, bøker og en Nobelpris til Roth i 2012. På tross av dette er denne kompetansen tilsynelatende lav, både i offentlig og privat sektor. Dette skrev jeg også om i artikkelen “Mangelfull evaluering av inntaksmodeller”, der jeg argumenterte mot å bruke politikernes hjemmesnekra modeller til å avgjøre skjebnen til tusenvis at VGS-elever.

De fleste overvurderer betydningen av sitt eget fagfelt. Jeg er neppe noe unntak, fordi jeg tror matematikk kan spille en rolle i en effektiv helsesektor. PS: Les gjerne artikkelen min om turnusplaner for sykepleiere!

Korrespondanse med Norsk helsenett SF

Jeg kontaktet Norsk helsenett SF (Helsenorge) om problemstillingen. Ut fra svaret tolker jeg at denne artikkelen fremstiller dagens system riktig.

For meg virker det som en åpenbar forbedring å tillate bytter, og jeg vet ikke hvorfor Helsenorge ikke gjør det. Det kan hende de har gode grunner til å beholde dagens system, eller at de har andre prioriteringer, eller at de har IT-systemer som er vanskelige å endre, eller mangler kompetanse til å implementere en slik algoritme.

Her er korrespondansen:

Hei,

Dersom to (eller flere) personer innbyrdes ønsker å bytte fastlege, 
skjer byttet umiddelbart?

Eksempel: Ola har A som fastlege. Kari har B som fastlege. 
Både A og B har helt fulle lister. Ola ønsker å bytte fra A til B. 
Kari ønsker å bytte fra B til A. Ingen andre som står til kø til verken 
A eller B må vente lengre dersom Ola og Kari får bytte umiddelbart. 
Er algoritmen smart nok til å gjøre slike bytter, eller må man vente
til en på en ledig plass fordi noen først forlater listene?

// Tommy

--------------------------------

Hei,

Det er dessverre ikke mulig å bytte plass med noen. 
Dersom ønsket lege ikke har ledig plass, må man stå på venteliste.

Vennlig hilsen
Veiledning Helsenorge

Monte Carlo modeling in Python with probabilit

Tommy Odland — Fri, 17 Oct 2025 00:00:00 +0200

probabilit is a Python package for Monte Carlo modeling that I helped create. It’s great for prototyping uncertainty calculations in some types of problems, such as those encountered in undergraduate engineering courses. In addition to a high-level modeling language, it contains algorithms for inducing correlations between variables, as well as computing the nearest correlation matrix.

Here is what probabilit can do:

It has a high-level modeling language that lets us model equations and expressions using probability distributions
It supports quasi Monte Carlo sampling: Latin Hypercube, Sobol, and Halton sequences
It can correlate variables, e.g. induce a correlation between a uniform and a normal distribution

Here is what probabilit does not do:

It’s not suited for modeling complex environments, like multi-agent systems, queues, differential equations, etc.
It’s not as memory efficient nor as fast as writing pure numpy simulations

This article is a tutorial demonstrating the most important features of probabilit. We’ll be using version 0.1.0. Beware that in future versions some things might change slightly. However, the overall ideas will remain the same, and all the Python code in this tutorial is executable (it was created from a jupyter notebook).

In [1]:

import numpy as np
import matplotlib.pyplot as plt

Monte Carlo modeling examples¶

Let us jump into probabilit with some modeling examples.

Braking distance of a car¶

Problem. Suppose we’re driving down the highway at approximately 100 km/h when suddenly a family of ducks appear and we have to slam the brakes. What is the braking distance?

Deterministic answer. It turns out that the relevant equation, obtained by realizing that the braking energy must cancel out the kinetic energy, is d = v^2 / (2 * mu * g), where

d = braking distance (m)
v = velocity (m/s)
mu = coefficient of friction (dimensionless)
g = gravitational acceleration (m/s²)

A deterministic answer is straightforward to compute in Python:

In [2]:

def braking_distance(velocity_kmh, mu, g=9.81):
    velocity_ms = velocity_kmh / 3.6  # Convert to m/s
    return velocity_ms**2 / (2 * mu * g)


velocity_kmh = 100
mu_dry = 0.7  # Coefficient of friction on dry road
distance_dry = braking_distance(velocity_kmh, mu_dry)
print(f"{distance_dry:.1f} meters")

56.2 meters

Probabilistic answer.

With probabilit we can compute braking distance as a distribution. More specifically we’ll be able to draw samples from the braking distance distribution. All we have to do is allow some of the input parameters to be distributions:

We do not trust the speedometer to be exact, so we set velocity to be a triangular distribution.
We do not trust that the coefficient of kinetic friction is exactly 0.7, so we use a normal distribution.

These distributions are somewhat arbitrary. Any distribution in scipy can be used from within probabilit, with the syntax
Distribution("scipy_distr_name", scipy_distr_arg1, scipy_distr_arg2, ...).

In [3]:

from probabilit import Distribution

# Replace velocity and coefficient of friction with distributions
velocity_kmh = Distribution("triang", loc=95, scale=10, c=0.5)
mu_dry = Distribution("norm", loc=0.7, scale=0.03)

# We can use the same function as earlier
distance_dry = braking_distance(velocity_kmh, mu_dry)

# Sample from the answer. This samples the distributions above
# and propagates the samples through the expression defined by the
# 'braking_distance' function. "lhs" = latin hypercube sampling
samples = distance_dry.sample(2500, random_state=42, method="lhs")

plt.figure(figsize=(6, 2.5))
plt.hist(samples, bins="auto", density=True, edgecolor="black")
plt.xlabel("Braking distance")
plt.tight_layout()
plt.show()

In [4]:

import pandas as pd
# We can also get summary statistics
(
    pd.Series(samples, name="braking_distance")
    .describe(percentiles=[0.01, 0.05, 0.95, 0.99])
    .to_frame()
    .T.round(1)
)

Out[4]:

	count	mean	std	min	1%	5%	50%	95%	99%	max
braking_distance	2500.0	56.3	3.3	47.1	49.3	50.9	56.2	62.0	64.3	72.3

With uncertainty added to some of the input parameters, the answer is not nearly as certain as it seemed in the deterministic analysis. We have learned that braking distance could reasonably be anywhere between 51 meters and 62 meters or so. This is the power of (simple) Monte Carlo analysis.

When distance_dry is sampled, the ancestor nodes velocity_kmh and mu_dry are also sampled. In the process, the attribute samples_ is set on all nodes. This is useful if we want to inspect the samples drawn on ancestor variables. For instance, here is a simple sensitivity analysis where we study the effect of the input parameters on the output:

In [5]:

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(6, 3), sharey=True)

ax1.scatter(velocity_kmh.samples_, distance_dry.samples_, s=3, zorder=5)
ax2.scatter(mu_dry.samples_, distance_dry.samples_, s=3, zorder=5)

ax1.set_ylabel("Distance")
ax1.set_xlabel("Velocity")
ax2.set_xlabel("Coefficient of friction")
ax1.grid(True, ls="--", zorder=0, alpha=0.5)
ax2.grid(True, ls="--", zorder=0, alpha=0.5)
fig.tight_layout()
plt.show()

Mutual fund¶

Problem. Suppose we save 100 units of money per month for 10 years and put the money in a mutual fund. The yearly interest rate is around 7 % after accounting for inflation. How much money will we have after 10 years?

Probabilistic solution. The interest rate will not be exactly 7 % every year, so we can add some uncertainty. Notice how probabilit expressions can be used in for-loops without any issues.

In [6]:

from probabilit import Lognormal, plot

YEARS = 10
SAVED_PER_MONTH = 100

money_saved = 0
for year in range(YEARS):
    interest_rate = Lognormal(mean=1.07, std=0.05)
    money_saved = money_saved * interest_rate + SAVED_PER_MONTH * 12

# Sample and plot the result
samples = money_saved.sample(999, random_state=42, method="lhs")
pairgrid = plot(money_saved)  # returns a seaborn pairgrid
pairgrid.axes[0, 0].set_xlabel("Money saved")
pairgrid.axes[0, 0].set_ylabel("")
plt.show()

Exercise. Change the example above so that money saved per month is a distribution. Choose values that are realistic for your own financial situation and study how much money you can save over the next decade.

Bird survival¶

This final example illustrates compound distributions, where one distribution is an argument to another.

Problem. Suppose we have a distribution of eggs per nest for a certain species of bird. We also have a survival percentage for each egg. What is the distribution of the total number of birds per nest that live till adulthood?

Probabilistic solution. We set up separate distributions and pass them as arguments to each other:

In [7]:

# Number of eggs per nest
eggs_per_nest = Distribution("poisson", mu=2)

# Assume we are uncertain of the survival probability too
# According to one source online, around 10-30% of eggs survive
prob_survive = Distribution("beta", a=2, b=2, loc=0.1, scale=0.2)

# Here is the compound distribution, which depends on two distributions
adult_birds_per_nest = Distribution("binom", n=eggs_per_nest, p=prob_survive)

# Calling plot() without sampling will sample a copy of the nodes
plot(
    eggs_per_nest,
    prob_survive,
    adult_birds_per_nest,
    sample_kwargs={"random_state": 42, "method": "lhs"},
    height=1.5,
    aspect=1.2,
);

The discrete nature of the distributions make them hard to visualize without jitter in the default plot call, but hopefully you get the idea.

Exercise. Go to Wikipedia’s examples of compound distributions. Verify computationally that this statement is true: “Compounding a Gaussian distribution with variance distributed according to an exponential distribution yields a Laplace distribution.”

How does probabilit work?¶

Given an expression, probabilit creates a computational graph that is evaluated lazily. For instance, the expression Constant(5) ** 2 is converted to Power(Constant(5), Constant(2)), where Power is a Transform node. Other transform nodes are Exp, Log, Sin, Abs, Add, Divide, etc. There are also Constant nodes and Distribution nodes.

In [8]:

from probabilit import Constant

expression = Constant(5) ** 2 + Distribution("uniform") / Constant(3)
print(expression)

Add(Power(Constant(5), Constant(2)), Divide(Distribution("uniform"), Constant(3)))

The computational graph can be visualized as a tree:

In [9]:

from probabilit import treeprint
treeprint(expression)

Add
   ├──Power
   │  ├──Constant(5)
   │  └──Constant(2)
   └──Divide
      ├──Distribution("uniform")
      └──Constant(3)

Not every probabilit expression is a tree. For instance, if a is a node then b = abs(a) + a creates a graph that is not strictly speaking a tree. All probabilit graphs are DAGs (directed acyclic graphs), since they cannot contain cycles.

A probabilit expression can be converted to a networkx MultiDiGraph.

In [10]:

import networkx as nx

# Create a MultiDiGraph
G = expression.to_graph()

# Draw information
pos = nx.spring_layout(G, seed=0)
labels = {node: type(node).__name__ for node in G.nodes()}

# Draw the graph
plt.figure(figsize=(6, 3))
nx.draw(
    G,
    pos,
    with_labels=True,
    node_color="lightblue",
    node_size=500,
    font_size=6,
    font_weight="bold",
    arrows=True,
    arrowsize=12,
    edge_color="gray",
    labels=labels,
)
plt.show()

It’s important to understand that no computation is performed until .sample() is called on a node. The expression above is just a graph with instructions about how to compute a result. Once .sample() is called, the actual work is done:

In [11]:

expression.sample(10, random_state=42)

Out[11]:

array([25.12484671, 25.31690477, 25.24399798, 25.19955283, 25.05200621,
       25.05199817, 25.0193612 , 25.28872538, 25.20037167, 25.23602419])

When the call asks for the sink node expression to be sampled, all ancestors are sampled in turn and results are propagated through the graph. This has both memory and performance overhead, which probabilit sacrifices in favor of a clean API and a pleasant modeling experience.

Latin HyperCube sampling (LHS)¶

Probabilit supports all major sampling methods in the scipy quasi-monte carlo module.

Below we show the difference between pseudo-random sampling (method=None) and Latin HyperCube (method="lhs").

In [12]:

uniform = Distribution("uniform")
exponential = Distribution("expon")

# Sample a sum to set `.samples_` on distributions for plotting
(uniform + exponential).sample(500, random_state=42, method=None)
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5});

In [13]:

(uniform + exponential).sample(500, random_state=42, method="lhs")
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5});

Notice how the low-discrepancy sequence produced by LHS leads to distributions of samples that looks very much like the theoretical distributions. In many Monte Carlo simulations, using LHS is an excellent choice because we obtain empirical distributions that match the theoretical ones with fewer samples drawn than what we would achieve had we used pseudo-random sequences.

Inducing correlations¶

Probabilit comes with Correlator objects that can be used to correlate variables. They work within the framework of the modeling API, or outside of it (where numpy arrays can be used). This means that these highly performant algorithms can be used outside of the modeling language too.

When all variables are Normal the Cholesky correlator can be a good choice. While it obtains the desired correlation, it does not preserve the marginal distributions. This is a major problem many real-world applications, so be careful when using the Cholesky correlator! Notice in the example below that the exponential distribution contains negative values after inducing correlations.

In [14]:

from probabilit.correlation import Cholesky

# Correlation matrix with desired Pearson correlation of 0.8
corr_mat = np.array([[1.0, 0.8], [0.8, 1.0]])

expression = (uniform + exponential).correlate(uniform, exponential, corr_mat=corr_mat)

expression.sample(500, random_state=42, method="lhs", correlator=Cholesky())
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5})

# Observed Pearson correlation
np.corrcoef(uniform.samples_, exponential.samples_)

Out[14]:

array([[1. , 0.8],
       [0.8, 1. ]])

Above, the marginal of the uniform distribution was preserved. This is a happy accident since we defined that distribution before we defined the exponential distribution in the code. If we switch them, then we will preserve the marginals of the exponential but destroy the marginal of the uniform. In general the Cholesky comes with no guarantees of preserving marginals.

Exercise. Switch the order and define the exponential distribution first, then the uniform distribution. Use the Cholesky correlator and notice how the marginal of the uniform is destroyed.

Iman-Conover¶

The Iman-Conover transformation is almost always a better choice than Cholesky. It leaves the marginal distributions unchanged, while attempting to induce the desired correlation. Iman-Conover is essentially a heuristic where data is transformed to rank-space, then Cholesky is used, then the data is transformed back. The end result is a permutation of the samples in each distribution. The idea is that by choosing a permutation that induces correlation in rank space, the permutation will also get close to the desired correlation in the original space. This typically works pretty well in most cases, but the algorithm comes without any optimality guarantees.

In [15]:

from probabilit.correlation import ImanConover

expression = (uniform + exponential)
expression.correlate(uniform, exponential, corr_mat=corr_mat)

expression.sample(500, random_state=42, method="lhs", correlator=ImanConover())
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5})

np.corrcoef(uniform.samples_, exponential.samples_)

Out[15]:

array([[1.        , 0.70443783],
       [0.70443783, 1.        ]])

Above we do not achieve the exact correlation that we wanted, but we get pretty close. The marginals are unaffected by the transformation.

Inducing correlations by permutation¶

Since Iman-Conover is just a clever and fast heuristic to pick a good permutation of the samples (rows) within a variable (column) in a matrix, perhaps we can do even better if we search the space of all permutations? This space is huge, so we can only afford to search a small part of it. The Permutation correlation does this, and by spending some time on computation we often end up with even better correlations than what Iman-Conover achieves. Notice below that we get extremely close to the desired correlation.

In [16]:

from probabilit.correlation import Permutation

expression = (uniform + exponential).correlate(uniform, exponential, corr_mat=corr_mat)

correlator = Permutation(iterations=5_000, random_state=42)
expression.sample(500, random_state=42, method="lhs", correlator=correlator)
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5})

np.corrcoef(uniform.samples_, exponential.samples_)

Out[16]:

array([[1.        , 0.78002787],
       [0.78002787, 1.        ]])

Finally, let us compare ImanConover with the Composite algorithm. The composite approach is what probabilit does by default. It first uses Iman-Conover to find a good starting point, then proceeds by running Permutation on that starting point for a number of iterations.

In [17]:

variables = [Distribution("lognorm", s=1) for _ in range(10)]

# Set up correlation matrix
corr_mat = np.ones((10, 10)) * 0.8
np.fill_diagonal(corr_mat, 1.0)

expression = sum(variables).correlate(*variables, corr_mat=corr_mat)

# Iman-Conover
correlator = ImanConover()
expression.sample(999, random_state=42, method="lhs", correlator=correlator)
observed_corr = np.corrcoef(np.array([v.samples_ for v in variables]))
RMSE = np.sqrt(np.mean((corr_mat - observed_corr) ** 2))
print(f"RMSE (observed-desired) for Iman-Conover: {RMSE:.4f}")

RMSE (observed-desired) for Iman-Conover: 0.1011

In [18]:

from probabilit.correlation import Composite

# Composite() => ImanConover(), then Permutation()
correlator = Composite(random_state=0, iterations=5_000, tol=0.01)
expression.sample(999, random_state=42, method="lhs", correlator=correlator)
observed_corr = np.corrcoef(np.array([v.samples_ for v in variables]))
RMSE = np.sqrt(np.mean((corr_mat - observed_corr) ** 2))
print(f"RMSE norm (observed-desired) for Composite: {RMSE:.4f}")

RMSE norm (observed-desired) for Composite: 0.0367

If we are willing to spend some additional time on computation (here 2.5 seconds instead of 15 milliseconds), then we can obtain correlations that are much closer to what we desired. Often this is worth it, as speed is rarely a bottleneck in these types of analyses.

Nearest correlation matrix¶

Just like the ImanConover and Permutation correlator classes can be used without using the modeling API, the nearest_correlation_matrix can also be used without using the modeling API. This algorithm is useful when users (e.g. a domain expert) proposes correlations, but they end up proposing a correlation matrix that is not valid. This happens quite frequently, since specifying a high-dimensional, non-trivial correlation matrix that is positive definite is hard.

In [19]:

from probabilit.correlation import nearest_correlation_matrix

rng = np.random.default_rng(0)

n = 5
proposed_corr = rng.uniform(low=-1, high=1, size=(n, n))

# Make it symmetric
proposed_corr = np.triu(proposed_corr) + np.triu(proposed_corr, 1).T
np.fill_diagonal(proposed_corr, 1.0)

(pd.DataFrame(proposed_corr) * 100).round(1)

Out[19]:

	0	1	2	3	4
0	100.0	-46.0	-91.8	-96.7	62.7
1	-46.0	100.0	45.9	8.7	87.0
2	-91.8	45.9	100.0	-93.3	45.9
3	-96.7	8.7	-93.3	100.0	-15.5
4	62.7	87.0	45.9	-15.5	100.0

In [20]:

np.linalg.eigvals(proposed_corr)  # Negative eigenvalues => not positive definite

Out[20]:

array([-1.05418258, -0.00675076,  1.41936444,  2.20321725,  2.43835165])

In [21]:

nearest_corr = nearest_correlation_matrix(proposed_corr)
(pd.DataFrame(nearest_corr) * 100).round(1)

Out[21]:

	0	1	2	3	4
0	100.0	-35.0	-42.5	-51.7	35.2
1	-35.0	100.0	49.9	11.4	75.0
2	-42.5	49.9	100.0	-51.0	26.8
3	-51.7	11.4	-51.0	100.0	-32.1
4	35.2	75.0	26.8	-32.1	100.0

Summary and references¶

probabilit has two layers:

A high-level modeling language, built on top of Python
Low-level functionality for correlating variables, finding nearest correlation matrices, etc

Depending on what kind of Monte Carlo analysis you want to do, one or both (or none!) of these might be useful.

Probabilit is a bit like Tensorflow in that it uses a lazily evaluated computational graph. The idea of building a Monte Carlo analysis tool in this fashion is due to Feda. Integrating LHS and ImanConover into the tool was also his idea. Knut provided feedback on practical methods and distributions used in some types of Monte Carlo analysis, e.g. the importance of Triangular and PERT distributions. The idea behind the Permutation correlation and how to speed it up by not re-calculating the correlation matrix in every iteration (an idea we did not have time to go into in this tutorial) is my own, but these are not novel ideas.

To read more on these subjects, I recommend:

The probabilit repository. Have a look at the implementations. Create an issue or PR if you find a bug :)
To find the nearest correlation matrix, we use Equation (3) in the paper An augmented Lagrangian dual approach for the H-weighted nearest correlation matrix problem
An excellent explanation of the Iman Conover algorithm is found in The geometry of the Iman-Conover transformation
The book Global Sensitivity Analysis by Saltelli et al. is not directly related to Monte Carlo modeling, but Monte Carlo is often used for sensitivity analysis of models, and therefore this book might be of interest

A Recipe for Curve Fitting

Tommy Odland — Sat, 04 Oct 2025 00:00:00 +0200

Inspired by A Recipe for Training Neural Networks, here are some tips and tricks for curve fitting. Assume we are fitting a non-linear, parametric function $f(x; \boldsymbol{\theta})$ to data $y$. The problem could be anything. Many highly technical problems are illuminated by curve fitting, but it helps to think of everyday examples such as: personal savings vs. time, apartment price vs. square meters, salary vs. experience, strength vs. bodyweight, world population vs. year, etc.

Modeling

Determine the purpose. Do you want to extrapolate, or do you want to interpolate? Is prediction the overall goal, or is it inference of latent variables? In other words: raw predictive power or understanding underlying phenomena? Is uncertainty needed? If yes, then what kind? A posterior conditional distribution $p(x | \hat{\boldsymbol{\theta}})$ or a posterior predictive distribution $p(x | \boldsymbol{\theta}) p(\boldsymbol{\theta} | X)$ that accounts for parameter uncertainty? Is curve fitting a one-time job, or will it every night or on demand? Will it run on a single data set, or perhaps on thousands of datasets?
Read the literature. Before deciding on the function $f$, search the web and review papers. What functions are used on similar problems? Avoid ad-hoc models like unregularized high-degree polynomials or very intricate functions. Apply Occam’s razor.
Know basic functions and how to combine them. Know the behavior of basic building blocks like linear functions, powers, exponentials, logarithms, sigmoids, hyperbolic functions, etc. Sometimes you can transition between them in more natural ways than the convex combination $\alpha f(x) + (1- \alpha) g(x)$. For instance, the power transform $(x^\alpha - 1) / \alpha$ lies between the affine function $x-1$ when $\alpha=1$ and the logarithmic function $\ln x$ when $\alpha \to 0$. Similarly, the generalized pareto distribution lies “between” the exponential $e^{-x}$ and the hyperbola $(1 + x)^{-1}$. Probability density functions and their antiderivatives are a rich source of modeling functions. Know how to combine functions to create new models.
Use an identifiable model. Avoid using non-identifiable models like $f(x; \boldsymbol{\theta}) = \theta_1 \exp(x - \theta_2)$, since they do not have a unique solution. This seems obvious, but I’ve seen papers where non-identifiable models are used without priors.
Choose an appropriate loss. The least squares loss $\mathcal{L}(\boldsymbol{\theta}) = \sum_i (y_i - f(x_i; \boldsymbol{\theta}))^2$ has nice properties, but might not be what you want. Know when to use $p$-norms, huber loss, pseudohuber, geometric loss, harmonic loss and statistical losses. Remember that fitting log-data to log-model with a loss like $(\log y_i - \log f(x_i; \boldsymbol{\theta}))^2$ is not equivalent to fitting data to a model without logs. Determining the appropriate loss function is a big part of the job. Always align the primary metric with the loss. If you care about MAE as the metric, it makes little sense to use RMSE as the loss.
Use a prior, or if the model is not statistical, a regularization term. A prior encodes prior knowledge and helps the solver numerically. It also lets you get a solution when you have little data or no data at all. The 2-norm $\lVert \boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}} \rVert_2$ is a good start and $(\boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}})^T A (\boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}})$ can be even better. Often $A$ is diagonal.
Deal with parameter constraints. There are several options: (1) use an optimizer that handles constraints natively, (2) re-parametrize the function as $f(x; t(\boldsymbol{\theta}))$ with $t$ mapping from unbounded real numbers to the desired bounds or (3) add a barrier function to the objective. For instance, if $\theta_1 \in [0, 1]$, then you can (1) use an algorithm like BFGS-B (the B stands for “box constraints”), (2) transform with e.g. a sigmoid like $\theta_1 = t(z) = (1 + e^{-z})^{-1}$ or (3) add barriers such as $-\log(\theta_1) - \log (1 - \theta_1 )$ to the objective. The log-barrier is equivalent to a beta-prior.
Weight the data. Consider a weighted loss like $\sum_i w_i (y_i - f(x_i; \boldsymbol{\theta}))^2$ if (1) some data points are more important than others or (2) you are working on a temporal forecasting problem. Beware of the difference between (a) inverse variance weights and (b) weighting the log-likelihood. They are the same when estimating the mean of a normal, but not in general (see also this post).
Consider partial pooling. Use hierarchical models to automatically regularize with hyperpriors. This is relevant if you have many curve fitting instances that you can conceptualize as being drawn from some population, and you want to learn the distribution of that population. It’s also extremely useful if some instances have little data.

Validation

Cross validate. Split randomly if you’re interpolating. Use a time based split if you’re forecasting. Always match cross validation to how the model will be used. Compute metrics and study them.
Use your eyes. Curve fitting can be visualized! Sample prior curves and plot them, plot the initial guess, curve fit “by hand” a few times, fit on every instance you have in your database. Visually look for errors, edge cases, etc.
Implement the forward model. Implement data generation based on your model if it’s statistical. This is called a prior predictive check. Can you tell a simulation apart from real-world observed data? How? If you can, does it give you ideas for how to improve the model?
Test parameter estimation. Some property-based tests: (1) generate a random $\boldsymbol{\theta}$, use it to generate a dataset, fit on the dataset. Is $\lVert \boldsymbol{\theta} - \hat{\boldsymbol{\theta}} \rVert$ small? It should be, especially in the limit of much data. (2) Test that $\mathcal{L}(\hat{\boldsymbol{\theta}}) \leq \mathcal{L}(\boldsymbol{\theta})$.
Plot parameter-space. You can plot the contours of the loss as a function of two parameters in $\boldsymbol{\theta}$ at a time, while fixing the remaining parameters at $\hat{\boldsymbol{\theta}}$. Sometimes studying the Hessian is worthwhile.
Plot residuals. When are residuals large? Are residuals homoscedastic? Should you model heteroscedasticity?

Optimization

Understand the optimization algorithm. Have at least a rudimentary understanding of the algorithms you use. Read the documentation and perhaps a summary paper. Play with the optimizer on toy problems. Get a feeling for how it works, and more importantly how and when it fails. Study the full verbose output of the algorithm.
Put variables on the same scale. First order methods (gradient methods) depend critically on the condition number. Pure second order methods (Newton’s method) on the other hand are affine invariant and in principle unaffected by the condition number (numerics and quasi-Newton methods make this untrue in reality). Standardize or take logs/exp/sigmoid/etc as needed.
De-correlate variables. Sometimes you can de-correlate variables through re-parametrization of the curve function.
Use a prior. Even if you lack prior knowledge in the modeling sense, weak regularization tends to help the optimizer.
Pick a good starting point. Graphical trial and error often works. You can also solve a simpler problem, e.g. solve a least squares problem first and use that solution as the starting point for your $p=1$ norm problem. The curve function $f$ can sometimes be relaxed by removing non-linearities. You could use (y[-1] - y[0])/(x[-1] - x[0]) as initial guess for slope, mean(y) as initial guess for bias, etc. Stress test the solver on bad initial solutions too.
Implement gradients and/or Hessian. Take the time to work out gradients. Use a CAS like SymPy or WolframAlpha along with pen and paper. Verify gradients numerically with scipy.optimize.check_grad or similar functions. If the gradient is too unwieldy, consider autograd with one of the many available packages: jax, autograd, PyTorch, TensorFlow or Stan.
Use numerically stable implementations. Watch out for expressions like $\ln(1 + e^{\theta})$ that overflow, catastrophic cancellation and other phenomena. See The Right Way to Calculate Stuff for more examples.

Curve fitting seems easy, but this perception can be deceptive. It’s not uncommon that people fail to get the desired results, often because they call standard routines without knowing how they work. I hope the tips and tricks above can help. Experience helps too: work on diverse practical problems and try to solve them. Make notes on what works in practice.

Lønna til norske utviklere i 2025

Tommy Odland — Wed, 17 Sep 2025 00:00:00 +0200

I denne artikkelen skal vi gå gjennom betraktninger og metoder for å lage en lønnskalkulator for IT-utviklere. Utålmodige lesere kan gå rett til Lønnskalkulatoren og se hovedfunnene i figuren nedenfor.

Nettsiden kode24 har lenge hatt årlige spørreundersøkelser om lønn og publisert data i etterkant. Vi kan finne datasett for 2021, 2022, 2023, 2024 og 2025 på nettsidene.

Jeg har tidligere gjort flere analyser av spørreundersøkelsene:

I 2021 gjorde jeg en analyse i det probabilistiske programmeringsspråket Stan
I 2023 brukte jeg generaliserte additive modeller
I 2024 gjentok jeg analysen fra 2023 med det nye datasettet
Basert på 2024-tallene hjalp jeg også til med å lage kodejobb.no’s Lønnskalkulator 2024

Det meste som kan sies har blitt sagt: konsulenter tjener noe mer enn fast ansatte, bosatte i Oslo får en lønnspremie, ledere og arkitekter tjener mest, kjønn er (heldigvis) ubetydelig, og antall år med erfaring dominerer alle andre variabler. Størsteparten av variasjonen i lønn skyldes variabler som spørreundersøkelsene ikke plukker opp. Til sammen forklarer variablene i lønnsundersøkelsene mellom 30% og 50% av variasjonen i lønna; hvor mye avhenger av datasettet og modellen. Les mine tidligere analyser for en nøye gjennomgang.

Årets pro-bono bidrag til IT-miljøet blir å slå sammen alle datasettene og lage en robust lønnskalulator. Den er robust fordi den (1) baserer seg på nesten ti tusen observasjoner og (2) tar automatisk høyde for lønnsvekst i fremtiden.

Datasett

Det er fem datasett for fem ulike år. “Data engineering” er alltid mer jobb enn man håper, og innsatsen for å slå sammen disse datasettene var ikke noe unntak. Jeg måtte:

Sammenstille lønnstall på format som 600 000 med 600.000,00 kr og lignende.
Sammenstille små variasjoner i spørsmålene fra år til år, f.eks. I hvilket fylke ligger jobben din? og I hvilket fylke jobber du?.
Ta hensyn til at det noen år var spurt om lønn med bonus og andre år lønn uten bonus. Dette fikser jeg med en latent variabel i modelleringen, men vi går ikke inn på det i denne artikkelen.
Fjerne tvilsomme observasjoner (omtrent 1.5% av dataene, der lønn var under 400 TNOK eller over 2000 TNOK).
Noen har sannsynligvis tolket spørsmålet “Hvor mange års relevant, formell utdannelse har du?” til å inkludere barneskolen, ungdomsskolen og VGS. Dette må fikses etter beste evne (jeg håper og tror ingen har vært student på et universitet i mer enn femten år).
Ta hensyn til regionreformen og dens delvise omgjørelse.
Slå sammen fagdisipliner, f.eks. var arkitektur et alternativ noen år, mens det het arkitekt andre år.
Slå sammen spørsmål: tidligere var jobb (in-house, konsulent, osv.) og sektor (privat, offentlig) to spørsmål—nå er det slått sammen til ett spørsmål.

Etter en grundig datavask står vi igjen med 9610 observasjoner. En del av disse er nok de samme personene som svarer på nytt, men det tenker jeg at er greit i denne sammenhengen. Som alltid er det masse forbehold: dataene er selvrapporterte, kode24’s lesere er ikke representative for utviklere generelt, osv. Nedenfor er lønn og erfaring visualisert for alle årene.

Medianlønna har økt de siste fem årene, men det har også gjennomsnittlig erfaring blant kode24’s lesere.

år	erfaring	lønn
2021	8.2	690
2022	8.3	725
2023	8.8	790
2024	8.5	850
2025	9.4	880

En regresjon på log-lønna avslører at den gjennomsnittlige lønnsveksten har vært omtrent 6.4% de siste årene, men dette tar ikke hensyn til at erfaringen også har økt. Om vi korrigerer for dette blir lønnsøkningen omtrent 5.1% per år.

Lønnskalkulatorer

En kalkulator implementerer vanligvis en kjent formel. For eksempel implementerer en skattekalkulator formelen for skatt (noe som kan være utfordrende å finne ut av). På samme måte kan en lønnskalkulator implementere “formelen for lønn”, men den vet vi ikke hva er. Derfor må vi først bruke en modell til å lære en formel fra datasettet, og deretter implementere denne. Et alternativ er å ikke implementere noen formel i det hele tatt, men heller bruke en såkalt ikke-parametrisk modell og vise brukeren sammenlignbare personer. Vi skal se på disse to alternativene i denne artikkelen, men det finnes andre muligheter også.

Uansett fremgangsmåte bør en god kalkulator:

Predikere godt. Estimert lønn bør matche faktisk lønn for så mange som mulig. Dette kan kryssvalideres.
Kommunisere usikkerheten. I dette datasettet har hver prediksjon stor usikkerhet, og det må brukeren forstå.

Ikke-parametriske modeller

Anta at en typisk utvikler ønsker å vite hva han eller hun kan forvente å tjene:

{'år': 2025, 'fylke': 'Oslo', 'jobb': 'in-house, privat sektor',
 'erfaring': 6, 'utdannelse': 3, 'fag': 'Fullstack'}

K-nærmeste naboer

Vi kan bruke K-nærmeste naboer med $k=10$ på 2025-data for å vise ti personer som ligner. Disse er

erfaring	utdannelse	fylke	jobb	fag	lønn
5	3	Oslo	in-house, privat sektor	Fullstack	775
7	3	Oslo	in-house, privat sektor	Fullstack	803
6	3	Oslo	in-house, privat sektor	Fullstack	819
5	3	Oslo	in-house, privat sektor	Fullstack	877
6	3	Oslo	in-house, privat sektor	Fullstack	895
6	3	Oslo	in-house, privat sektor	Fullstack	900
6	3	Oslo	in-house, privat sektor	Fullstack	914.375
5	3	Oslo	in-house, privat sektor	Fullstack	950
6	3	Oslo	in-house, privat sektor	Fullstack	1000
6	3	Oslo	in-house, privat sektor	Fullstack	1400

Her var vi heldige, fordi det er mange personer i Oslo med lignende egenskaper. Medianlønna blant disse ti personene er 897 TNOK, og 80% ligger i intervallet mellom 800 og 1040 TNOK.

Beslutningstre

Dersom vi bruker et beslutningstre i stedet, får vi følgende tabell. Modellen er satt til å alltid returnere minst ti personer, og for denne spørringen returnerer den 15. Sammenlignet med K-nærmeste naboer er beslutningstreet mer opptatt av erfaring, og mindre opptatt av fag og jobb.

erfaring	utdannelse	fylke	jobb	fag	lønn
6	3	Oslo	frilans / selvstendig næringsdrivende	DevOps og Automatisering	500
6	3	Oslo	in-house, privat sektor	UX og Design	650
6	3	Oslo	konsulent	Ledelse	735
6	3	Oslo	konsulent	Frontend	780
6	3	Oslo	in-house, privat sektor	Fullstack	819
6	3	Oslo	konsulent	Testing	830
6	3	Oslo	in-house, privat sektor	Fullstack	895
6	3	Oslo	in-house, privat sektor	Frontend	900
6	3	Oslo	in-house, privat sektor	Fullstack	900
6	3	Oslo	in-house, privat sektor	Fullstack	914.375
6	3	Oslo	konsulent	Fullstack	918
6	3	Oslo	konsulent	Frontend	940
6	3	Oslo	in-house, privat sektor	Fullstack	1000
6	3	Oslo	in-house, privat sektor	Fullstack	1400
6	3	Oslo	konsulent	Fullstack	1400

Medianlønna blant disse 15 personene er 900 TNOK, og 80% ligger i intervallet mellom 684 og 1240 TNOK.

Sammenligning: K-nærmeste naboer vs. beslutningstre

Vi kan sammenligne Mean Absolute Error (MAE) på en 10-folds kryssvalidering for å se hvor mange personer modellen bør returnere for å gi gode prediksjoner.

Vi observerer at:

Beslutningstrær er klart bedre enn K-nærmeste naboer, med lavere MAE over hele linja.
Så lenge vi returnerer omtrent 20 personer så får vi gode prediksjoner.
Jo flere personer vi returnerer, desto bedre kan vi estimere et intervall og si noe om lønnspennet.

Grunnen til at beslutningstrær er bedre enn K-nærmeste naboer er fordi førstnevnte automatisk velger hvilke variabler som er viktige (erfaring er helt klart viktigst), mens sistnevnte må gå like langt ut i alle dimensjoner for å finne naboer.

Både K-nærmeste naboer og beslutningstrær har noen felles svakheter i denne problemstillingen:

Nær endepunktene i hver variabel er de nødt til å se innover for å gjøre prediksjoner. For eksempel, dersom man har 45 års erfaring vil modellen ikke finne noen personer som har mer erfaring, kun personer som har mindre erfaring. Det samme gjelder de som har null år med erfaring eller null år med utdannelse. Dette er spesielt kritisk i høye dimensjoner, fordi da er sannsynligvis alle datapunktene nær endepunktene—jo flere ting vi måler, desto mer er alle mennesker ulike på hver sin måte!
Vi brukte 2025-data til nå, men vi ønsker å bruke hele datasettet. Vi vet at tidligere år har lavere lønn på grunn av generell lønnsøkning i samfunnet, men 2025-tallene er nær endepunktet på variabelen “år”. Derfor vil modellene ofte kun hente informasjon fra 2025 og ikke bruke hele datasettet. Begge modellen er lokale heller enn globale, og evner ikke å bruke informasjon fra tidligere år på en god måte.

Parametriske modeller

En parametrisk modell lærer parametrene i en formel som beskriver et datasett. Fordelen er at hele datasettet kan komprimeres ned til én kort formel som beskriver sammenhengene. Ulempen er at man må selv spesifisere en fornuftig formel. Parametriske modeller er globale og ekstrapolerer vanligvis godt, og dette er en fordel i vår sammenheng—spesielt med tanke på årstall inn i fremtiden og lønnsinflasjon.

Første idé er å bruke en Ridge-modell og minimere

\begin{align*} \text{minimize} \, &\lVert \text{lønn} - \boldsymbol{\mu} \rVert_2^2 + \alpha \lVert \boldsymbol{\beta} \rVert_2^2 \\ \mu &= \beta_0 + f(\text{erfaring}) + \beta_{\text{utdannelse}} \text{utdannelse} + \beta_{\text{fylke}} + \beta_{\text{jobb}} + \beta_{\text{fag}} \\ f(z) &= \beta_1 z + \beta_2 z^2 + \beta_3 z^3 + \cdots \end{align*}

og dette er et godt utgangspunkt som gir helt kurante resultater. Høyere-ordens polynomer er vanligvis ingen god idé, fordi de kan overtilpasse og svinge voldsomt. Det er derfor viktig å plotte polynomet (noe som ble gjort men ikke er med i artikkelen).

Hvordan kan vi forbedre denne modellen? For det første vil vi finne medianlønna heller enn gjennomsnittslønna, fordi medianen er mindre påvirket av ekstreme observasjoner og er ofte et bedre valg i slike sammenhenger. Det var derfor vi brukte Mean Absolute Error (MAE) heller enn Mean Squared Error (MSE) tidligere; MAE er minimert av medianen.

For det andre er en tradisjonell regresjonsmodell påvirket av feil i absolutte tall, mens multiplikative feil gir mer mening i lønnssammenheng. Med andre ord, å bomme med 100 TNOK når lønna er 500 TNOK er en stor feil, mens å bomme med 100 TNOK når lønna er 1200 TNOK er ikke like ille, fordi det er en mindre multiplikativ feil.

Vi endrer modellen til å bruke en $\ell_1$-norm på likelihood-leddet og tilpasse log-lønn heller enn lønn:

\begin{align*} \text{minimize} \, &\lVert \log \left( \text{lønn} \right) - \boldsymbol{\mu} \rVert_1 + \alpha \lVert \boldsymbol{\beta} \rVert_2^2 \end{align*}

Dette minner om en Lasso, men med normene byttet om: vi har $\ell_1$-norm på likelihood-leddet og $\ell_2$-norm på regulariseringsleddet. Her er koeffisientene i modellen (klikk på figuren for å se større versjon):

Legg merke til at lønnsveksten estimeres til 5.1% per år.

Å sammenligne mellom kategorier innebærer å konvertere prosentene til relative tall, ikke å legge dem sammen additivt. For eksempel, dersom man går fra en in-house offentlig stilling til å bli konsulent, kan man forvente en lønnsøkning på omtrent

\begin{align*} \frac{1 + 5.9\%}{1 - 5.4\%} = \frac{1.059}{0.946}= 1.119 = 11.9\% . \end{align*}

Her er residualene til modellen:

Dette er relativt pene residualer, fordi det ikke er stor endring i residualene over lønnsintervallet, såkalt heteroskadist… hetroskedasisti…. heteroskedastisitet. Polynomet som ble brukt var et fjerdegradspolynom, som på grunn av regularisering oppfører seg pent når erfaring er mellom null og førti år.

La oss kjøre vår “typiske” utvikler gjennom alle tre modellene:

{'år': 2025, 'fylke': 'Oslo', 'jobb': 'in-house, privat sektor',
 'erfaring': 6, 'utdannelse': 3, 'fag': 'Fullstack'}

Vi observerer noe ulike estimater av medianlønn og intervaller fra de ulike modellene:

KNN: 897 TNOK med 80% intervall mellom 800 og 1040 TNOK.
Tre: 900 TNOK med 80% intervall mellom 684 og 1240 TNOK.
Lineær modell: 918 TNOK med 80% intervall mellom 748 og 1136 TNOK.

Intervallene fra de ikke-parametriske modellene (KNN og tre) er mindre robuste, ettersom de er basert på henholdsvis 10 og 15 observasjoner. Intervallet fra den lineære modellen er basert på alle datapunktene, og har høyere kvalitet.

La oss nå sammenligne MAE for alle modellene, samt en svak dummy-modell som alltid predikerer medianen og en kraftig gradient boosting modell.

Modell / Metrikk	MAE
Dummy	202
KNN	152
Tre	128
Lineær	119
Gradient boosting	114

Gradient boosting har lavest MAE, men den lineære modellen er bare marginalt verre. Likevel har den noen klare fordeler:

Det estimerte intervallet (usikkerheten) er nok mer robust og realistisk.
Den er veldig enkel å implementere i Javascript. Man bruker bare formelen.

Oppsummering

Kode24 har gjort en kjempejobb med å samle inn og publisere data på utvikleres lønn. I denne artikkelen kombinerte vi fem år med lønnsdata til ett datasett med 9610 observasjoner, etter datavask og sammenstilling av variabler på tvers av år.

Vi sammenlignet to ulike tilnærminger til lønnskalkulatorer:

Ikke-parametriske modeller (KNN og beslutningstrær) som viser faktiske, sammenlignbare personer
Parametriske modeller (log-lineær regresjon med $\ell_1$-loss) som bruker en global formel

Den enkle log-lineære modellen (MAE: 119) presterte bedre enn de to ikke-parametriske modellene, fordi MAE var lavere og intervallene var mer robuste. Faktisk hadde modellen nesten like lav MAE som gradient boosting (MAE: 114), men med fordelen av å være tolkbar og enkel å implementere i JavaScript. Beslutningstrær var gode for å finne sammenlignbare personer, betydelig bedre enn K-nærmeste naboer.

Når det gjelder metodikk for lønnskalkulatorer er lærdommen klar: prioriter enkelhet og transparens. Brukere trenger to ting: (1) fornuftige prediksjoner og (2) ærlig kommunikasjon om usikkerheten. Kalkulatoren implementerer den enkle lineære modellen.

Selv om jeg fulgte bud (5) og bud (9) har jeg dårlig samvittighet for å ha brukt et fjerdegradspolynom i modelleringen uten progressiv regularisering—men jeg slapp unna med det på partial dependence plottet. Aldri igjen.

People rating people

Tommy Odland — Mon, 04 Aug 2025 00:00:00 +0200

Ratings and rankings are everywhere in daily life. We rank sports teams, chess players, movies and TV-shows, doctors, local businesses, and sometimes even each other.

This article is about computing an overall rating in a setting where people have individually rated each other. Examples include dating apps like Tinder, e-commerce websites where people act as both sellers and buyers, and other platforms where we want to quantify the trustworthiness or quality of members.

We distinguish between two scenarios:

Symmetry: if person $a$ rates $b$, then $b$ must also rate $a$. Their rating values may differ.
No symmetry: if person $a$ rates $b$, then it does not necessarily follow that $b$ rates $a$ back.

In this article we’ll consider the more general non-symmetric scenario. Given individual ratings that are either positive or negative, the goal is to produce an overall rating of everyone. The figure below shows 5 people, where blue arrows indicate positive ratings and red arrows indicate negative ratings.

In the figure above, we expect $a$ to be highly rated and $c$ to be poorly rated.

Simple aggregation

The simplest thing we can do is to count the number of ratings and create a ratio $r_i$ between $0$ and $1$ for each person.

We define a binary matrix $P$ consisting of positive ratings and a matrix $N$ of negative ratings.

\begin{align*} P_{ij} &= \begin{cases} 1 & \text{if person } i \text{ is rated positively by person } j \\ 0 & \text{otherwise} \end{cases} \\ N_{ij} &= \begin{cases} 1 & \text{if person } i \text{ is rated negatively by person } j \\ 0 & \text{otherwise} \end{cases} \end{align*}

The rating vector $\boldsymbol{r}$ is computed by summing positive ratings and dividing by total ratings. One issue with the naive approach is that someone who has few ratings might end up with a total rating of $0$ or $1$, so we employ the rule of succession by adding two dummy ratings: one positive and one negative. The equation becomes

\begin{align*} r_i &= \frac{1 + \sum_j P_{ij}}{2 + \sum_j P_{ij} + \sum_j N_{ij}} \end{align*}

Or alternatively, in matrix notation, where $\boldsymbol{1}$ and $\boldsymbol{2}$ denote vectors of ones and twos and division is entry-wise:

\begin{align*} \boldsymbol{r} &= \frac{\boldsymbol{1} + P \boldsymbol{1}}{\boldsymbol{2} + P \boldsymbol{1} + N \boldsymbol{1}} \end{align*}

Here’s a piece of Python code that implements this computation on the graph structure shown in the initial figure:

import numpy as np

P = np.array([
     [0, 1, 1, 1, 1], # <- Positive ratings given to A
     [0, 0, 0, 0, 0], # <- Positive ratings given to B
     [0, 0, 0, 0, 0],
     [0, 1, 1, 0, 0],
     [1, 0, 0, 0, 0], # <- Positive ratings given to E
    ])

N = np.array([
     [0, 0, 0, 0, 0], # <- Negative ratings given to A
     [0, 0, 1, 0, 0], # <- Negative ratings given to B
     [0, 1, 0, 1, 1],
     [1, 0, 0, 0, 1],
     [0, 0, 0, 0, 0], # <- Negative ratings given to E
    ])

# Simple aggregation
ratings = (1 + np.sum(P, axis=1)) / (2 + np.sum(P, axis=1) + np.sum(N, axis=1))
print(ratings.round(2))  # [0.83 0.33 0.2  0.5  0.67]

The results are superimposed on the graph in the figure below.

The popular person $a$ gets the highest rating $0.83$ as expected, and unpopular person $c$ gets the lowest rating $0.2$ as expected.

Weighted aggregation

Perhaps all users in a system should not be considered equally trustworthy. In other words, if $a$ rates $b$ positively, then our opinion of $b$ might also depend on $a$.

If $a$ is a long-time user, who is positively rated by others and who has given balanced ratings to others, then we should trust $a$’s ratings.
If $a$ is a new user, or is negatively rated by others, or rates everyone negatively, then we should perhaps not trust $a$’s ratings as much.

We weight each user with a weight $w_j \in [0, 1]$ before we aggregate their individual ratings:

\begin{align*} r_i &= \frac{1 + \sum_j P_{ij} w_j}{2 + \sum_j P_{ij} w_j + \sum_j N_{ij} w_j} \end{align*}

How do we choose the weights $w_j$? There are many possibilities, but the weights $w_j$ must somehow represent the trustworthiness of person $j$.

Weighting by discrimination

One idea is to look at how much discrimination each user applies. Whose ratings do you trust more: someone who rates everyone positively, or someone who rates half the population negatively and the other half positively? If you agree with me that the second person provides more useful information, then one option is to compute a scaled version of the entropy:

\begin{align*} H(p, n) = \frac{1}{\ln 2} \left[ \frac{p}{p+n} \ln \left( \frac{p+n}{p} \right) + \frac{n}{p+n} \ln \left(\frac{p+n}{n}\right) \right], \end{align*}

where $n$ is the number of negative ratings the person has given and $p$ is the number of positive ratings.

Observe that $H(p=k, n=k) = 1$ for all $k \geq 1$ and that $H(p=k, n=0) = H(p=0, n=k) = 0$ for all $k \geq 1$. In other words, this weighting function accomplishes what we set out to do: users with high discriminatory power are given a weight of one, while users who show no discrimination in their ratings are given a weight of zero.

from scipy.stats import entropy

def H(n, p):
    return entropy([n, p]) / np.log(2)

print(H(5, 5))  # 1.0
print(H(0, 5))  # 0.0

weights = entropy(np.stack((np.sum(P, axis=0), np.sum(N, axis=0))), axis=0) / np.log(2)

# Weighted positive and negative sums
P_w = P * weights
N_w = N * weights

# Weighted ratings
ratings = (1 + np.sum(P_w, axis=1)) / (2 + np.sum(P_w, axis=1) + np.sum(N_w, axis=1))
print(ratings.round(2))  # [0.83 0.34 0.21 0.49 0.67]

In this particular problem instance, the results do not differ much from the previous ratings.

Entropy is by no means the only possible quality measure—it depends strongly on the specific use-case. A number of more complicated weighting functions could be used. For instance, we could also weight a user by:

how many ratings they have given in total
how long they have been users in the system/platform
how much information they have on their profile
how well they themselves are rated by others.

All of these signal high levels of trustworthiness. We will now pursue option (4): we’ll weight each person’s ratings by how highly they themselves are rated.

Weighted aggregation by rating

The final model that we’ll consider is the most advanced one. The idea is that the rating person $a$ gives to person $b$ should be weighted by the rating person $a$ has. Building on the matrix notation we established earlier, we write the equation as:

\begin{align*} \boldsymbol{r} &= \frac{\boldsymbol{1} + P \boldsymbol{r}}{\boldsymbol{2} + P \boldsymbol{r} + N \boldsymbol{r}} \end{align*}

In an e-commerce platform, this means that if $a$ is highly rated and gives a positive rating, then that is worth more than a positive rating from a user with mixed or negative ratings.
In a dating app like Tinder where ratings signify attractiveness, the interpretation is that if someone who is attractive rates us favorably, then that rating is worth more than a rating from someone who is not considered attractive by others.

Notice the recursive definition: to weight the ratings given by person $a$ to others, we need to know the overall rating of person $a$—but that quantity is exactly what we’re trying to compute in the first place!

Solution by fixed-point iteration

The simplest and fastest way to compute $\boldsymbol{r}$ is to iterate the equation until it converges. Starting with an initial guess $\boldsymbol{r}_0 = \boldsymbol{1} / 2$ we compute

\begin{align*} \boldsymbol{r}_{k+1} &= \frac{\boldsymbol{1} + P \boldsymbol{r}_{k}}{\boldsymbol{2} + P \boldsymbol{r}_{k} + N \boldsymbol{r}_{k}} \end{align*}

repeatedly until convergence, as measured by $\lVert \boldsymbol{r}_{k+1} - \boldsymbol{r}_{k} \rVert < \epsilon$.

epsilon = 1e-12
ratings = np.ones(N.shape[0]) / 2
for iteration in range(99):
    new_ratings = (1 + P @ ratings) / (2 + P @ ratings + N @ ratings)
    if np.linalg.norm(new_ratings - ratings) < epsilon:
        break
    ratings = new_ratings

print(ratings.round(2))  # [0.74 0.44 0.29 0.42 0.63]

Convergence is linear, and after 13 iterations the procedure terminates.

All three ratings induce rankings on the people, and they are compared in the figure below.

Notice how:

Person $b$ moves up compared to the simple rating. He is given a negative rating by $c$, but $c$ has a low overall rating and this lessens the weight given to $c$’s opinion of $b$.
Person $d$ moves down, because the negative rating from $a$ matters a lot and the positive rating from $c$ matters little.

Summary and references

Rating and ranking is interesting because it is part science and part art. There’s no fixed answer: every scenario is different and must be modeled appropriately.

The problem considered in this article, where people rate people, is special in two ways: (1) there is just one group and (2) ratings are not symmetric. The defining equation is

\begin{align*} \boldsymbol{r} &= \frac{\boldsymbol{1} + P \boldsymbol{r}}{\boldsymbol{2} + P \boldsymbol{r} + N \boldsymbol{r}}, \end{align*}

which is non-linear in the vector $\boldsymbol{r}$ and may contain millions of unknown variables. Despite this, it’s solved by fixed-point iteration in a few seconds on a laptop.

I’ve previously written about how patients can rate doctors. In that problem there were two separate groups: doctors and patients. One group rates the other group. I’ve also written about rating football teams, which is a symmetric problem. If team $a$ plays $b$, then $b$ necessarily plays $a$ too.

The PageRank algorithm ranks websites, and it’s what made Google famous in the late 90s. The paper A Survey on PageRank Computing is an interesting read. Similar algorithms include HITS, SALSA and EigenTrust. The book Who’s #1?: The Science of Rating and Ranking gives an overview of simple methods, with a focus on sports. When users interact with each other, the field is sometimes called reputation systems. The paper The Digitization of Word of Mouth: Promise and Challenges of Online Feedback Mechanisms explores the online reputation mechanisms of the early internet.

The ideas in the paper Propagation of Trust and Distrust are similar to those expressed in this article, but the authors focus more on local propagation while we solve a global problem. Reputation and ranking is closely related to content discovery, recommendation systems and collaborative filtering, and the book Recommender Systems by Aggarwal is a handy reference. This has social implications too, which the paper On Social Machines for Algorithmic Regulation explores.

Appendix A: Empirical results

I created random sparse rating matrices for a million people. I set the density to $p=10^{-5}$, so each matrix has approximately $\left( 10^6 \right)^2 \times p = 10^7$ non-negative entries. Convergence is shown in the figure below.

Below is the computation on the third largest strongly connected component of the Epinions social network dataset. The full dataset has 131,828 nodes and 841,372 and is solved quickly, but it’s too large to visualize in full.

Appendix B: Solution by Newton’s method

Newton’s method and its variants can be an alternative to fixed-point iteration. While Newton’s method exhibits quadratic convergence (fixed-point iteration is linear), the time spent per iteration is so high that it ends up being an inferior choice in practice. Also, if the starting point is not good ($\boldsymbol{r} = 1/2$ tends to work, while $\boldsymbol{r} = 0$ does not) the algorithm will fail. Regardless, here’s how it can be done.

The matrix equation

\begin{align*} \boldsymbol{r} &= \frac{1 + P \boldsymbol{r}}{2 + P \boldsymbol{r} + N \boldsymbol{r}}, \end{align*}

is equivalent to solving

\begin{align*} F(\boldsymbol{r}) = \boldsymbol{r} \odot \left( \boldsymbol{2} + P \boldsymbol{r} + N \boldsymbol{r} \right) - \left( \boldsymbol{1} + P \boldsymbol{r} \right) = \boldsymbol{0} \end{align*}

which we can do with Newton’s method:

\begin{align*} \boldsymbol{r}_{k+1} = \boldsymbol{r}_{k} - J(\boldsymbol{r}_{k})^{-1} F(\boldsymbol{r}_{k}). \end{align*}

To find the Jacobian matrix $J_{ij}(\boldsymbol{r}) = \partial F_i(\boldsymbol{r}) / \partial \boldsymbol{r}_{j}$, we first write out the components of $F(\boldsymbol{r})$.

\begin{align*} F_i(\boldsymbol{r}) = r_i \left( 2 + \sum_k P_{ik} r_k + \sum_k N_{ik} r_k\right) - 1 - \sum_k P_{ik} r_k \end{align*}

Differentiating with respect to $r_j$, then switching back to matrix notation, we obtain:

\begin{align*} J(\boldsymbol{r}) = \partial_{r_j} F_i(\boldsymbol{r}) & = \delta_{ij} \left( 2 + \sum_k P_{ik} r_k + \sum_k N_{ik} r_k\right) + r_i \left( P_{ij} + N_{ij} \right) - 0 - P_{ij} \\ &= 2 I + \operatorname{diag}(P \boldsymbol{r} + N \boldsymbol{r}) + \operatorname{diag}(\boldsymbol{r}) (P + N) - P \end{align*}

This can be implemented in Python as:

def F_and_J(ratings, P, N):
    """Returns a tuple (F, J), with function value F and its jacobian J."""
    Pr = P @ ratings
    Pr_Nr = Pr + N @ ratings
    F = ratings * (2 + Pr_Nr) - 1 - Pr
    J = ratings * (P + N) - P
    np.fill_diagonal(J, J.diagonal() + 2 + Pr_Nr)
    return F, J

epsilon = 1e-12
ratings = np.ones(N.shape[0]) / 2
for iteration in range(99):
    F, J = F_and_J(ratings, P, N)
    new_ratings = ratings - np.linalg.solve(J, F)
    if np.linalg.norm(new_ratings - ratings) < epsilon:
        break
    ratings = new_ratings
print(ratings.round(2))  # [0.74 0.44 0.29 0.42 0.63] <- Same as fixed-point

Note that the code above explicitly stores zeros (it does not use sparse matrices), and would therefore be unsuitable for large matrices.

Appendix C: Properties of the fixed-point iteration

In millions of computational experiments with random matrices $P$ and $N$, the fixed-point iteration seemingly always converges to a unique fixed point. For a long time I conjectured that the fixed point iteration always converges to a unique, attracting fixed point. However, it turns out this conjecture is false.

A counterexample is the following structure: There are three groups and at least 16 people in each group. Within each group, everyone rates everyone else negatively. There are no self-ratings.

With 16 people in each group, the algorithm cycles between $(0.509, 0.366, 0.509)$ and $(0.29, 0.585, 0.29)$ when started at a random $\boldsymbol{r}$. With 15 people in each group, the algorithm does not cycle, but will eventually converge.

Appendix D: Code

Here is some Python 3.12 code for the model.

import numpy as np  # Version: 2.1.3


class RatingProblem:

    def __init__(self, N, P):
        """Initialize a problem given positive binary ratings P
        and negative binary ratings N."""
        assert (N.shape == P.shape) and (N.shape[0] == N.shape[1])
        self.P, self.N = P, N

    @classmethod
    def random(cls, n, seed=None):
        """Create a random problem instance of size n."""
        rng = np.random.default_rng(seed)

        # Create adjacency matrix with random density
        density = rng.random()
        adj = (rng.random(size=(n, n)) < density).astype(int)

        # Zero out diagonal (can't rate yourself)
        np.fill_diagonal(adj, 0)

        # Randomly split into positive and negative ratings
        split = rng.random(size=(n, n)) < rng.random()
        return cls(N=adj * (1 - split), P=adj * split)

    def __call__(self, r):
        assert len(r) == self.N.shape[0]
        P, N = self.P, self.N
        return (1 + P @ r) / (2 + P @ r + N @ r)

    def jacobian(self, r):
        """Compute the Jacobian matrix at r."""
        assert len(r) == self.N.shape[0]

        P, N = self.P, self.N
        Pr, Nr = P @ r, N @ r

        # Vectorized jacobian computation
        numerator = (1 + Nr)[:, None] * P - (1 + Pr)[:, None] * N
        denominator = (2 + Pr + Nr) ** 2
        return numerator / denominator[:, None]

    def solve(self, r_0=None, epsilon=1e-12):
        """Solve using fixed-point iteration."""
        r_0 = np.ones(self.N.shape[0]) / 2 if r_0 is None else r_0

        assert len(r_0) == self.N.shape[0]

        ratings = r_0.copy()
        for iteration in range(99):
            new_ratings = self(ratings)
            if np.linalg.norm(new_ratings - ratings) < epsilon:
                break
            ratings = new_ratings
        return ratings


if __name__ == "__main__":

    rng = np.random.default_rng(0)

    # Test that different starting points converge to the same value
    for test in range(10**4):

        n = rng.integers(4, 20)
        problem = RatingProblem.random(n=n, seed=test)
        r1 = rng.random(size=n) * rng.random()
        r2 = rng.random(size=n) * rng.random()

        s1 = problem.solve(r1)
        s2 = problem.solve(r2)
        assert np.allclose(s1, s2)

        if test % 1_000 == 0:
            print(".", end="")

Hva kan jeg bruke matte til?

Tommy Odland — Thu, 17 Jul 2025 00:00:00 +0200

Elever spør ofte “når får jeg bruk for dette?” eller “hva skal jeg bruke matematikk til?“

Det er et godt spørsmål som ofte går ubesvart.

Det er flere måter å svare på slike spørsmål på. Her en én type svar: en liste med 100 konkrete problemstillinger der matematikk er nyttig.

Noen problemstillinger er relativt enkle, som å bygge en boks med størst mulig volum. Andre er vanskelige, som å spille Go. Å vanne plenen fremstår som et tøyeste problem, men illustrerer hvor dypt man kan gå om man begynner å undersøke en problemstilling. Andre er store og mer seriøse problemer, som å lande en rakett.

Det er på ingen måte ment å være en uttømmende eller spesielt seriøs liste :)

Telle
Spille Go
Spille dart
Sortere tall
Lage kunst
Løse Soduko
Forstå politikk
Vanne plenen
Spille dataspill
Styre en missil
Lagre fiskemat
Lande en rakett
Løse en labyrint
Lage en labyrint
Vedde på hester
Lage en matplan
Føre et regnskap
Regne med flyttall
Gå inn i veddemål
Fargelegge et kart
Teste en hypotese
Lage en søkemotor
Rangere klatreruter
Dele en taxiregning
Bygge en database
Arrangere ekteskap
Designe en auksjon
Simulere stigespillet
Planlegge bussruter
Bygge en søkemotor
Regne på et boliglån
Forstå språkmodeller
Rangere sjakkspillere
Bygge en atomreaktor
Digital bildebehandling
Manipulere valgkretser
Lage en aksjeportefølje
Undersøke dødsårsaker
Bygge elektriske kretser
Anbefale filmer og serier
Lage et treningsprogram
Forstå kunstig intelligens
Matche spillere i dataspill
Løse en differensialligning
Analysere dating-markedet
Lage turnus for sykepleiere
Designe effektive algoritmer
Forstå verden gjennom data
Verdtsettelse av fritidsboliger
Finne minimum av en funksjon
Lage vaktplaner for et sykehus
Designe et demokratisk system
Finne korteste reisevei offshore
Vedde penger på fotballkamper
Designe effektive datastrukturer
Effektivisere robotiserte varehus
Få mest mulig bolig for pengene
Modellere et petroleumsreservoir
Vedlikeholde roterende maskiner
Lagre store mengder informasjon
Lage et program som spiller sjakk
Planlegge sitteplasser i barnehagen
Forsvare infrastruktur mot terrorister
Innholdsbevisst reskalering av bilder
Forstå hvordan et aksjefond fungerer
Bygge en prosessor til en datamaskin
Lage en valgomat for kommunevalget
Bygge skyskrapere som ikke kollapser
Effektivisere Lånekassens bokontroller
Bygge en boks med størst mulig volum
Sette sammen en ungdomsskoleklasse
Predikere hvor billettkontrollører bør stå
Analysere data fra spørreundersøkelser
Estimere kostnadene til et stort prosjekt
Bestemme når man bør pensjonere seg
Predikere hvem som finner fotballkamper
Verdsette fritidsboliger med maskinlæring
Sette sammen det beste Pokémon-partiet
Logge tastetrykk ved hjelp av en mikrofon
Sette best mulig pris på en Airbnb-leilighet
Lage en Bayesiansk sannsynlighetsmodell
Plassere hjertestartere på best mulig steder
Forstå DNA-sekvenser til mennesker og dyr
Finne ut om du blir snytt når du kjøper Twist
Spare 125 millioner på å borre en rett tunnel
Programmere og studere logikken til en heis
Regne ut om det er best å eie eller leie bolig
Sette sammen forsikringer for marin shipping
Projisere høye dimensjoner til få dimensjoner
Lage en inntaksmodell for videregående skole
Bestemme hvor mye et bakeri bør bake hver dag
Kutte tekstilruller slik at man får minst mulig svinn
Designe en kaffekopp som holder godt på varmen
Regne ut sannsynligheten for suksessfull putt i golf
Designe loot-boxes i dataspill for å maksimere profitt
Finne ut hvilke faktorer som er assosiert med høy lønn
Plassere returpunkter for glass- og metallembalasje i byen
Lage timeplan for en skole for best mulig ressursutnyttelse
Teste om man klarer å smake forskjell på brus med og uten sukker
Forstå hvorfor man ikke alltid bør stemme på politikerne man ønsker

Who’s the best bettor?

Tommy Odland — Sat, 07 Jun 2025 00:00:00 +0200

A friend of mine is part of a group that arranges bets at their workplace. Each person proposes bets they believe will come true. The bets have European-style odds $o$ and are from a betting exchange. The money placed on each bet is always the same, so we arbitrarily take it to be one unit. Each bet leads to an outcome $y = 1$ (win) or $y = 0$ (loss).

Given histories of bets for each person, how can we decide who the best bettor is?

For instance, given three people Bob, Alice and Mark with histories:

person    odds (o)             outcomes (y)
bob       (2, 2, 2)            (1, 1, 1)    
alice     (3, 3)               (1, 1)    
mark      (2, 2, 2, 2, 2)      (1, 1, 1, 1, 0)

Who is the best bettor? Both Bob and Alice have won all their bets, but Alice took higher risk. Mark placed more bets that came through, but one of his bets was wrong.

In this article we will define the likelihood of an outcome over a sequence of bets. Then we create a latent variable that encodes the skill of a bettor, relative to the probability given by the odds. Given a sequence of bets and outcomes we can then solve for this latent variable, and the best bettor is the person with the greatest value.

The probability of an outcome

A person performs a sequence of $n$ bets with odds $o_i$ and attains outcomes $y_i$ for each bet $i=1, 2, \ldots, n$. Ignoring the margin of the bookmaker, the odds can be converted to a probabilities by the equation $p_i = 1 / o_i$. Each bet is independent, so the probability of the joint outcome is

\begin{equation*} P(\boldsymbol{p}, \boldsymbol{y}) = \prod_i^n p_i^{y_i} (1 - p_i)^{1 - y_i}. \end{equation*}

Insight

When a bet will be made. Suppose a person believes the true probability of winning a bet is $\tilde{p}$. With European-style odds the net gain is $(o-1)$ if a bet is won and $-1$ if it is lost. A person will only bet if the expected payout is larger than zero, that is, if

\begin{equation*} P(\text{win}) (\text{amount won}) + P(\text{loss}) (\text{amount lost}) = \tilde{p} (o - 1) + (1 - \tilde{p}) (-1) \geq 0. \end{equation*}

Solving for $\tilde{p}$ we learn that a person will only bet if $\tilde{p} \geq 1/o = p$. In other words, they will bet if their own personal probability is greater than the one deduced from the odds.

Introducing the latent insight $\theta$. We will assume that each person has some insight $\theta$, which is a latent (unobserved) variable representing their additional information. We would like to infer how much insight each person has, and rank their betting ability by their insight. One idea is to set $\tilde{p} = p + \theta$, but we prefer $\theta$ to be an unbounded variable and the probabilities $\tilde{p}$ to be restricted to the domain $[0, 1]$. To accomplish this we use the logistic function (sigmoid or expit) and its inverse the logit. The functions are:

\begin{equation*} \operatorname{expit}(z) = \frac{1}{1 + \exp(-z)} \quad \text{and} \quad \operatorname{logit}(z) = \log\left( \frac{z}{1 - z} \right). \quad \end{equation*}

To ensure that $\tilde{p}$ ends up in the domain $[0, 1]$ we transform the probabilities to logits on $\mathbb{R}$, add the insight $\theta$, and transform the result back to $[0, 1]$:

\begin{equation*} \tilde{p}(\theta, p) = \operatorname{expit}\left( \operatorname{logit}(p) + \theta \right) = \frac{p e^{\theta}}{p e^{\theta} - p + 1} \end{equation*}

Substituting $\tilde{p}(\theta, p)$ into the probability of the joint outcome, we obtain the negative log-likelihood function

\begin{align*} - \log P( \boldsymbol{p}, \boldsymbol{y} \mid \theta) &= - \sum_i^n y_i \log \left( \tilde{p}_i(\theta, p_i) \right) + (1 - y_i) \log \left( 1 - \tilde{p}_i(\theta, p_i) \right) \\ &= \sum_i^n - y_i \log{\left(\frac{p_i e^{\theta}}{p_i e^{\theta} - p_i + 1} \right)} + \left(y_i - 1\right) \log{\left(\frac{1 - p_i}{p_i e^{\theta} - p_i + 1} \right)}. \end{align*}

The likelihood is the probability of observing $(\boldsymbol{p}, \boldsymbol{y})$ given a value of $\theta$.

Why a prior is needed. Suppose someone has made three bets with odds $\boldsymbol{o} = (2, 2, 2)$ and outcomes $\boldsymbol{y} = (1, 1, 1)$. Plotting the negative log-likelihood as a function of $\theta$ we get the following figure. Minimizing the negative log-likelihood leads to $\theta \to \infty$. In a sense, the person is able to claim “I’m great at betting! I knew all along that the true probability was one for all these bets!”

This is analogous to estimating the probability of heads for a sequence of coin tosses as the number of heads divided by the total number of coin tosses. If we get three heads, then the estimated probability is one, and that is unreasonable.

Putting a prior on insight

It’s far-fetched to believe that someone who wins a bet or two really has great insight. To fix this issue of $\theta$ going to infinity, we put a prior distribution on it. We center a normal distribution around zero, and since the standard deviation is somewhat arbitrary we choose unity. In other words we choose $\theta \sim \mathcal{N}(1, 0)$ as the prior. A single squared term is added to the negative log-likelihood (now a log-posterior), which becomes

\begin{align*} - \log P(\theta \mid \boldsymbol{p}, \boldsymbol{y}) &\propto - \sum_i^n y_i \log \left( \tilde{p}_i(\theta, p_i) \right) + (1 - y_i) \log \left( 1 - \tilde{p}_i(\theta, p_i) \right) + \theta^2 \\ &\propto \theta^{2} - \sum_i^n \theta y_i + \log{\left(p_i e^{\theta} - p_i + 1 \right)}. \end{align*}

As a result, the negative log-posterior now has a finite minimizer: We now have a reasonable statistical model.

To summarize, we assumed that each person had some insight $\theta$ that led them to believe that the true probabilities $\tilde{p}_i$ are higher than the $p_i$ deduced from the odds. Then we compute the maximum a posteriori (MAP) estimate of $\theta$, which is the most probable value of $\theta$ given the observed sequence of bets. A higher value of $\theta$ means the person has insight and beats the bookmaker, $\theta$ around zero means no additional insight is available apart from the one deduced by the odds, and a negative value of $\theta$ means the person consistently underperforms.

Cases

We will now go through some concrete cases and check if the model is sensible.

Placing more and more bets. Suppose the winning streak is increased by one bet at a time. The table below shows odds, outcomes and MAP estimates of the insight $\theta$.

case  odds (o)       outcomes (y)  insight (theta)
1     (,)            (,)            0.0000
2     (2,)           (1,)           0.2223
3     (2, 2)         (1, 1)         0.4011
4     (2, 2, 2)      (1, 1, 1)      0.5491
5     (2, 2, 2, 2)   (1, 1, 1, 1)   0.6748

No bets placed implies zero insight, as there is no reason to believe the bettor is any better than randomly guessing. As the number of consecutive wins increases, the posterior mode moves away from the prior and we believe more and more that the person really has insight and outperforms the bookmaker.

Equal gains. Suppose earlier wins can be used to be more money on the next bets. Here are three bettors who all multiplied their money by $16$.

case  odds (o)       outcomes (y)  insight (theta)
1     (16,)          (1,)           0.4526
2     (4, 4)         (1, 1)         0.6179
3     (2, 2, 2, 2)   (1, 1, 1, 1)   0.6748

The model assigns the most insight to the person who placed four bets. Suppose each person has some small insight $\epsilon$. Then the probability $(1/16 + \epsilon) \geq (1/2 + \epsilon)^4 = (1/2)^4 + (1/2)^3 \epsilon + \mathcal{O}(\epsilon^2)$ when $\epsilon$ is small. The probability that the person who places four bets ends up with four wins is lower, and it is therefore more impressive. In other words, the model favors winning on many smaller bets rather than one big bet, even though the final monetary gain is identical.

Although the amount of money placed on each bet is the same in our setup, the result above makes even more sense if we were betting on a sequence and re-investing our wins. A longer sequence of smaller wins is more beneficial than one large win, since we can re-invest our previous earnings along the way.

A win and a loss. Now consider four bettors who all win one bet and lose one bet.

case  odds (o)       outcomes (y)  insight (theta)
1     (2, 2)         (1, 0)         0.0
2     (3, 3)         (1, 0)         0.1358
3     (4, 4)         (1, 0)         0.2088
4     (5, 5)         (1, 0)         0.2559

The model assigns the most insight to the person who took the most risk. This also makes sense.

Case 1 represents winning once with odds two (doubling the money), then losing one unit of money. This bettor is back to where he started.
Case 5 represents winning once with odds five (returning five times the original amount of money), then losing one unit of money. This bettor is still up by four units of money.

Some random cases. Below is a table with nine cases, sorted by the inferred insight $\theta$ of each bettor. These cases are all pretty random, and are shown so the reader can study how different odds and outcomes measure up against each other.

case  odds (o)       outcomes (y)  insight (theta)
1     (2, 2, 2)      (0, 0, 1)     -0.182
2     (2, 2)         (1, 0)         0.0
3     (2, 2, 2)      (0, 1, 1)      0.182
4     (2, 2, 2)      (1, 0, 1)      0.182
5     (5, 5)         (1, 0)         0.2559
6     (8,)           (1,)           0.4113
7     (16,)          (1,)           0.4526
8     (3, 3)         (1, 1)         0.5386
9     (2, 2, 2)      (1, 1, 1)      0.5491

Conclusion

To determine who the best bettor is, we modeled a bettor’s insight $\theta$ as a latent variable. The insight is added to the probability deduced from the odds, and the best bettor is the one with the greatest insight. In order to avoid $\theta$ becoming very large or very small in small sequences with only wins or losses, we put a prior on it. The skill of a bettor is the maximum a posteriori estimate of $\theta$.

There might be other interesting ways of solving this problem, but I think this approach works well. It’s conceptually simple, all test cases look good, and it’s simple to implement and compute. Further work that might be interesting includes: making the optimization routine faster, studying the equations in greater depth, or computing the entire posterior distribution instead of finding the value that maximizes it.

Code

Here’s a Python 3.11 code snippet if you want to run the model.

import numpy as np  # 1.26.4
import scipy as sp  # 1.13.0
import functools


def log_posterior(theta, odds, y):
    """Compute the log-posterior of theta, given `odds` and `y`."""

    # Convert from odds to base probabilities
    p = 1 / odds
    # Convert to logit scale using logit(z) = log(z / (1-z))
    logits = sp.special.logit(p)
    # Add `theta` (insight) and convet back using expit(z) = 1/(1+exp(-z))
    p_tilde = sp.special.expit(logits + theta)
    # Compute the log-likelihood
    log_likelihood = y * np.log(p_tilde) + (1 - y) * np.log(1 - p_tilde)

    # A more numerically stable version as theta -> oo is:
    # log_likelihood = -theta * (1 - y) - np.logaddexp(
    #     np.log(p), -theta + np.log1p(-p))

    # Add prior and return the negative log-posterior
    return -np.sum(log_likelihood) + theta**2


# Example case
odds = np.array([2, 2, 2])
y = np.array([1, 1, 1])

# Optimize the log-posterior for optimal theta
fun = functools.partial(log_posterior, odds=odds, y=y)
result = sp.optimize.minimize_scalar(fun=fun)
print(result.x)  # 0.549107...

Fraværsgrensa

Tommy Odland — Thu, 22 May 2025 00:00:00 +0200

Fraværsgrensa i videregående skole er på 10%. Det betyr at dersom en elev har udokumentert fravær på 10% eller mer i et fag, så mister eleven karakter i faget. I Reddit-tråden “Videregående skole er no bullshit” har en elev fått varsel fordi han eller hun var borte én dag, som utløste 3 timers fravær i kroppsøving og oversteg 5%.

Kroppsøving inneholder 56 undervisningstimer totalt, mens f.eks. matematikk har 140 timer. Det betyr at fraværsgrensen har ulik effekt avhengig av hvor mange skoledager som inneholder faget:

Om kroppsøving er på 3 timer per dag, så vil 2 dager med fravær betyr at man mister karakteren, fordi $6 / 56 \approx 0.107 \geq 0.1$.
Om matematikk er på 2 timer per dag, så vil 7 dager med fravær betyr at man mister karakteren, fordi $14 / 140 = 0.1 \geq 0.1$.

Selv om det er mye fravær som kan unntas fraværsgrensen, er det rimelig å anta at noe udokumentert fravær vil forekomme. Denne artikkelen svarer på spørsmålet: “gitt at udokumentert fravær inntreffer tilfeldig, hva er sannsynligheten for at en elev mister karakteren i et fag?”

Sannsynligheten for å miste karaktergrunnlaget i et fag

La oss gjøre to forenklinger før vi fortsetter:

Vi antar at hver dag med et fag inneholder like mange timer. Dette er tilfellet om man har 3 timer gym hver onsdag. Det er ikke tilfellet om man for eksempel har 2 timer matte på mandager og 1 time på torsdager. Denne antagelsen betyr at vi slipper å telle opp antall timer innad i hver dag og antall timer i faget totalt. Dette gjør utregningene litt enklere fordi vi slipper å forholde oss til faktiske timeplaner, som uansett vil variere fra skole til skole.
Vi antar at udokumentert fravær inntreffer tilfeldig med en viss sannsynlighet, og at fravær er uavhengig. Med andre ord: om en elev hadde fravær i går påvirker ikke sannsynligheten for at eleven har fravær i dag.

Figuren nedenfor viser sannsynligheten for å miste karakter et fag, som funksjon av antall dager med faget. $P(\text{fravær})$ er sannsynligheten for fravær på en gitt dag, og utregningen er visualisert for 1%, 2% og 3%.

Den overordnede trenden er at sannsynligheten for å miste karakter er høyere desto mindre antall dager man har med faget. Det stemmer med de innledende observasjonene om kroppsøving og matematikk.

Vi skal nå forklare knekkene i grafen: før man har 11 dager med et fag, vil én fraværsdag føre til at man overstiger grensa på 10%. På 11 dager har man litt rom for fravær ettersom én dag ikke lengre gjør at man overstiger grensa. Situasjonen blir likevel verre fra 11 dager til og med 20 dager. I dette intervallet vil to eller flere fraværsdager utløse grensa, og med flere dager er det større sannsynlighet for at det skjer—derfor stiger grafen. På 21 dager har man igjen litt mer slingring, og slik fortsetter mønsteret.

Utregning

La sannsynligheten for fravær være $p = P(\text{fravær})$, la fraværsgrensa være $g=0.1$ og la antall dager med faget være $d$. Antall fraværsdager $X \sim \text{Binom}(p, d)$ er da en stokastisk variabel med binomisk sannsynlighetsfordeling.

Sannsynligheten for at man mister karakter er $P(X \geq \lceil g d \rceil)$, der $\lceil \cdot \rceil$ er matematisk notasjon for å runde opp, f.eks. så er $\lceil 1.7 \rceil = 2$. Vi regner oss frem til sannsynligheten ved hjelp av den kumulative sannsynlighetsfordelingen.

\begin{equation*} P(X \geq \lceil g d \rceil) = 1 - P(X < \lceil g d \rceil) = 1 - (P(X \leq \lceil g d \rceil) - P(X = \lceil g d \rceil)). \end{equation*}

Eksempel
Om vi setter $p=0.02$ og antall dager lik $d=13$, så er

\begin{align*} P(X \geq \lceil (0.1) 13 \rceil) &= P(X \geq 2 ) \\ &= 1 - P(X < 2) \\ &= 1 - \left[ P(X = 0) + P(X = 1) \right] \\ &= 1 - \left[ \binom{13}{0} 0.02^{0} (1 - 0.02)^{13} + \binom{12}{1} 0.02^{1} (1 - 0.02)^{12} \right] \\ &= 1 - \left[ 1 (0.98)^{13} + 12 0.02 (0.98)^{12} \right] \\ &\approx 1 - \left[ 0.769 + 0.188 \right] \approx 0.0426455. \end{align*}

Hva bør fraværsgrensa være?

Dette er et verdispørsmål som matematikk alene ikke kan svare på, men la oss gjøre noen antagelser og undersøke hva en rimelig fraværsgrense som funksjon av antall dager $d$ bør være:

Anta at vi ønsker at $P(\text{ingen karakter}) \leq 0.001$, gitt et normalt udokumentert fravær.
Anta at noe udokumentert fravær vil forekomme, f.eks. at $p = 0.02$ eller lignende er normalnivå.

Da kan vi spørre “Hva er minste (strengeste) fraværsgrense $g$ vi kan ha som oppfyller $P(\text{ingen karakter}) \leq 0.001$?”

Svaret er gitt grafisk i figuren nedenfor, for tre ulike verdier av $p$ og ulike verdier av dager $d$ med et gitt fag.

Å velge $P(\text{ingen karakter})$ og $p$ er en skjønnsmessig vurdering. Uansett er konklusjonen at antall dager med faget spiller en viktig rolle. Desto lavere (strengere) fraværsgrense, desto høyere sannsynlighet for at man har uflaks og mister karaktergrunnlaget, selv når sannsynligheten for fravær $p$ i utgangspunktet er lav.

Personlig ville jeg heller straffet elever med nedsatt karakter dersom fraværet blir stort, heller enn å ta bort hele karaktergrunnlaget. Dette gir en mer glidende overgang, i motsetning til en hard statisk grense på 10%. Det er litt rart at politikerne ikke har valgt en annen mekanisme enn en statisk grense på 10% for alle fagene. Kanskje de selv hadde fravær? All matematikken vi har brukt i denne artikkelen er i hvert fall VGS-pensum.

Kode

# Python 3.12.9,
import numpy as np  # numpy: 2.1.3
import scipy as sp  # scipy: 1.15.1


def prob_lose_grade(num_days, prob_absence=0.01, limit_absence=1 / 10):
    """Probability of losing your grade, given a course with `num_days` days."""
    # Limit for losing grades, in days. Hitting this means losing the grade.
    days_limit = int(np.ceil(num_days * limit_absence))
    # Probability of absence is Binomial
    distr = sp.stats.binom(n=num_days, p=prob_absence)
    # P(X >= limit) = 1 - P(X < limit) = 1 - (CDF(limit) - P(X=limit))
    return float(1 - distr.cdf(days_limit) + distr.pmf(days_limit))


def find_limit(num_days, prob_absence=0.01, max_prob_lose_grade=0.001):
    """Find largest fraværsgrense (limit), such that the probability of
    losing grade is <= target_value."""

    def objective(limit):
        return prob_lose_grade(num_days, prob_absence, limit) - max_prob_lose_grade

    # The goal is to find the largest possible limit that ensures that the
    # return value of prob_lose_grade is smaller than or equal to target_value.
    optimal_limit = sp.optimize.bisect(objective, a=0, b=1) + 1e-8
    # print(f"The absence limit should be set to {optimal_limit:.4f}")

    prob_lose = prob_lose_grade(num_days, prob_absence, optimal_limit)
    # print(f"Then the prob of losing the grade is {prob_lose:.4f}")
    assert prob_lose < max_prob_lose_grade
    return float(optimal_limit)


if __name__ == "__main__":
    # Test case - single day
    assert np.isclose(prob_lose_grade(1, prob_absence=0.01, limit_absence=1 / 10), 0.01)

    # Test case - two days
    prob = 0.01 * 0.01 + 2 * 0.01 * (1 - 0.01)
    assert np.isclose(prob_lose_grade(2, prob_absence=0.01, limit_absence=1 / 10), prob)

    # Test case - tree days
    p = 0.01
    prob = p * p * p + 3 * p**2 * (1 - p) + 3 * p * (1 - p) ** 2
    assert np.isclose(prob_lose_grade(3, prob_absence=0.01, limit_absence=1 / 10), prob)

How do Voting Advice Applications work?

Tommy Odland — Mon, 07 Apr 2025 00:00:00 +0200

This is a translation of my Norwegian article about VAAs. The figures have not been translated, but are hopefully understandable from the context.

This article presents the mathematics behind voting advice applications (VAAs). VAAs are essentially questionnaires: a user expresses agreement or disagreement with statements about politics, and the VAAs tells the user what parties they align with.

This article consists of three main parts:

Part 1: VAAs and geometry
Part 2: How does a VAA explain the results?
Part 3: How should a VAA present questions to the user?

The article begins at a basic mathematical level and becomes progressively more challenging. The advantage of this approach is that there is something to learn for everyone regardless of level. The disadvantage is that the article does not have a single clear target audience.

PS: In 2023, I visualized the parties in the Norwegian municipal election. The focus of that article was insight into party politics. This article focuses more on the design of VAAs.

Part 1: VAAs and geometry

Opinions and matrices

To create a VAA, we ask political parties to which degree they agree with various statements. The results are quantified by translating responses like “Strongly agree” into numbers. The answers can then be visualized in a table with one row per statement and one column per party. Below, “Strongly agree” is translated to $2$ and “Strongly disagree” to $-2$.

The table forms a matrix $X$. The rows and columns have a geometric interpretation as vectors. A vector is a list of numbers, e.g., $(1, 2)$, and can be interpreted geometrically as an arrow or as a point.

Statements and parties

Statements as vectors

To visualize statements and parties as vectors, we must limit ourselves to two or three dimensions. Higher dimensions cannot be visualized.

Let’s study geometrically what the parties SV and Sp think about a selection of statements.

In the upper right of the figure, we find statements that both parties agree with. In the lower right, we find statements that SV agrees with, but Sp disagrees with. We can make similar observations for the other two quadrants.

Which statements are good and which are bad?

A good statement allows us to distinguish parties from each other:

In the figure above, “Build railway northward to Tromsø” (Norwegian: “Bygg jernbane nordover til Tromsø.”) is a poor statement. Regardless of whether you agree or disagree, the statement does not clarify which of the two parties you are politically closest to. Trivial statements, such as “A just society is good,” are agreed upon by all parties. Therefore, they provide no information. The same applies to trivial statements that all parties disagree with, such as “The county municipality should be abolished” (Norwegian: “Fylkeskommunen bør avskaffes.”) in the figure above.
“Drop the ferry-free E39 along the west coast” (Norwegian: “Dropp ferjefri E39 langs Vestlandet.”) is a good statement because it discriminates between the parties. If you agree, you are politically closer to SV. If you disagree, you are closer to Sp.

In two dimensions (two parties), statements are good if they are far from the subspace spanned by the vector $\boldsymbol{e} = (1, 1)$, i.e., the diagonal line $y=x$. At the end of this article, we will generalize this idea to construct an algorithm that finds a good ordering of questions.

Parties as vectors

Above, we considered the rows (statements) in the table as vectors. Let’s now look at the complementary vector space where the columns (parties) are vectors. We choose two random statements: one about wolves (“Det skytes for mye ulv her i landet.” / “Too many wolves are shot in this country.”) and one about state ownership in companies (“Staten bør eie flere bedrifter.” / “The state should own more companies.”).

Two parties are similar if they are close to each other in the space of opinions in the figure above. For example, we can measure distance as the distance on the horizontal axis plus the distance on the vertical axis (a mathematician might call this “Manhattan distance” or $p=1$ norm).

Parties R and AP are close to each other. The distance is $0 + 1 = 1$.
Parties R and V are far from each other. The distance is $3 + 4 = 7$.

The distance in this space is the key to how a VAA determines which parties you most agree with when you take the test. This is what we will cover in part 2.

Part 2: How does a VAA explain the results?

Explanation: ranking parties by distance

Above, we saw that party similarity can be quantified by looking at the distance between parties. We visualized this in two dimensions, but the calculation is the same in any number of dimensions.

Here is a matrix with the numbers from the first figure in this article:

\begin{equation*} X = \begin{bmatrix} \text{AP} & \text{Sp} & \text{Frp} & \text{H}\\ -1 & 1 & -2 & -1\\ 2 & 1 & -2 & -2\\ -2 & -2 & 2 & 2\\ 1 & -1 & -2 & -2\\ 1 & 2 & 2 & -2\\ -1 & -2 & -2 & 2 \end{bmatrix} \end{equation*}

If you take the VAA and your opinions are the vector $\boldsymbol{y} = (-1, -2, 2, -2, 0, 0)$, then your distance to AP is $13$, to Sp $14$, to Frp $5$, and to H $4$. In other words, you are likely on the political right and should consider voting for Frp or H.

The distance $d_j$ between your opinions $\boldsymbol{y}$ and a party $j$’s opinions is given by the sum of the distances for each statement $i$:

\begin{equation*} d_j = \sum_i | X_{ij} - y_i | \end{equation*}

Distances, $p$-norms, and weighting

We can generalize to an arbitrary $p$-norm and add a weighting $w_i$:

\begin{equation*} d_j = \left( \sum_i w_i | X_{ij} - y_i |^p \right)^{(1/p)} \end{equation*}

The weights $w_i$ are typically equal to $1$ for most statements, while $w_i = 2$ for core issues or issues that are particularly important to you. For issues where you have no opinion, we can set $w_i = 0$. The value chosen for $p$ depends on how you want to measure distance.

If you believe that $\boldsymbol{y}$ is closer to party $A$ in the example below, you should choose $p=2$. If you think $\boldsymbol{y}$ is closer to $B$, you should choose $p=1$.

\begin{align*} \boldsymbol{y} &= (0, 0, 0, 0) \\ A &= (1, 1, 1, 1) \\ B &= (0, 0, 0, 3) \end{align*}

This is a subjective assessment, and personally, I lean toward $p=1$. Both of NRK and VG do the same; they both use $p=1$ in their VAAs.

The calculations ultimately result in a list of parties, sorted by distance:

1. H   (distance 4 )
2. Frp (distance 5 )
3. AP  (distance 13)
4. Sp  (distance 14)

Such a list, sorted by distance, is one possible way to explain the user’s results. But it’s not the only option; the next section covers another type of explanation.

Explanation: Your position as a combination of party positions

The parties’ vectors set directions in the space of opinions. After taking a VAA, you also get a position in this space. (“Ditt standpunkt” means “Your standpoint” or “Your position” in English.)

In the figure above, your position is $\boldsymbol{y} = (1, -2)$.

One way to reach your position with vector arithmetic is to take MDG’s position and add KrF’s. This can be observed geometrically in the figure above, or algebraically:

\begin{align} \text{MDG} + \text{KrF} = \boldsymbol{y} \\ (2, -1) + (-1, -1) &= (1, -2) \end{align}

Drafting an optimization model

Here is an alternative idea to a distance-based explanation: we express your position as a combination of party positions. We are looking for a combination of parties that describes you by taking us as close to your position as possible. In other words, we are search for a vector $\boldsymbol{\beta}$ that solves the optimization problem

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \|. \end{align}

This model has some practical challenges:

It allows complicated explanations in the form of long vectors $\boldsymbol{\beta}$. For example, we can reach $\boldsymbol{y}$ through the linear combination
\begin{align} \text{V} + \text{R} + \text{H} + \text{Ap} + \text{MDG} = \boldsymbol{y}. \end{align}
The model allows negative directions because $\boldsymbol{\beta}$ can consist of negative numbers. This is difficult to interpret. For example, $-\text{R} = \boldsymbol{y}$ in the example above, but you naturally want to know who you agree with, rather than being told that you disagree with the party $\text{R}$.
The model allows explanations where the sum of the parties’ contributions $\boldsymbol{\beta}$ exceeds $1$. This is the case in $\text{MDG} + \text{KrF} = \boldsymbol{y}$ and in the long explanation above. Again, it is difficult to interpret such an answer.

All these problems can be solved if we limit ourselves to convex combinations.

Final optimization model

We fix the problems in the above model by adding some constraints. Rather than look for any linear combination of parties, we look for the shortest convex combination:

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \| + \epsilon \| \boldsymbol{\beta} \| \\ & \text{subject to} && \beta_j \geq 0 && \\ & && \sum_j \beta_j = 1. && \end{align}

The regularization term $\epsilon \| \boldsymbol{\beta} \|$ means that we prefer simple (short) explanations. The two constraints on $\boldsymbol{\beta}$ are the definition of a convex combination, and ensure that we avoid negative directions and sums that exceed $1$. The sum $\sum_j \beta_j$ must equal $1$, rather than be less than $1$, because if $\boldsymbol{y} = (0, 1)$, $A = (0, 1)$, and $B = (0, 2)$, we expect $\boldsymbol{\beta} = (1, 0)$ as the answer, not $\boldsymbol{\beta} = (0, 0.5)$.

With the $2$-norm on both terms in the objective function, the solution to the optimization problem becomes:

\begin{align} 0.69 \, \text{V} + 0.23 \, \text{H} + 0.08 \, \text{Frp} = \boldsymbol{y}. \end{align}

If you go back to the figure and compare the geometry with this solution, you’ll see that the solution makes sense. That said, there are often no single correct approach in mathematical modeling, and other variations of such a model might be appropriate.

The optimization model on a larger example

Let’s run the optimization model on the same example as before:

\begin{align*} [X \mid \boldsymbol{y}] = \begin{bmatrix} \text{AP} & \text{Sp} & \text{Frp} & \text{H} & \boldsymbol{y} \\ -1 & 1 & -2 & -1 & -1\\ 2 & 1 & -2 & -2 & -2\\ -2 & -2 & 2 & 2 & 2\\ 1 & -1 & -2 & -2 & -2 \\ 1 & 2 & 2 & -2 & 0\\ -1 & -2 & -2 & 2 & 0 \end{bmatrix} \end{align*}

The answer is given below and corresponds well with the ranking we obtained earlier.

1. H   (51%)
2. Frp (45%)
4. Sp  (4%)
3. AP  (0%)

The difference between distance-based and combination-based explanation

We have seen two ways to explain a user’s results in a VAA:

Distance-based explanation - ranks parties by distance from the user
Combination-based explanation - the user is expressed as the nearest convex combination of parties

The difference between the explanations is illustrated in the figure below.

Distance-based explanation says that $\boldsymbol{y}$ is closest to $B$, then $A$, and finally $C$. Distance-based explanation is mutually independent; the distance to $B$ does not affect the distance to $A$. If $B$ had disappeared, the distance to $A$ would remain unchanged.
Combination-based explanation expresses the position $\boldsymbol{y}$ as a convex combination. The answer in the figure above becomes $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0, 0.7, 0.3)$. The optimization model’s explanation is mutually dependent. If $B$ had disappeared, the explanation would have relied on $A$, but when $B$ exists, $A$ is not needed at all to explain $\boldsymbol{y}$.

The figure below shows an example where the explanations become completely different.

Measured by distance, $C$ is closer to $\boldsymbol{y}$ than both $A$ and $B$. Yet $\boldsymbol{y}$ is explained using $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0.5, 0.5, 0)$ without using $C$. Shortest distance tells you which single party is closest. The optimization model tells you which coalition of parties is closest to you.

Part 3: How should a VAA present questions to the user?

In this section, we present an algorithm that selects statements that discriminate well between parties. There may be two reasons for wanting this:

We want to use a selection of statements. If parties have answered many questions, we may want to limit ourselves to a smaller selection in the VAA. How can we choose a small subset of good statements that differentiate between parties?
Early termination. We want the user to be able to end the VAA after any number of questions. How do we ensure that the user has been presented with a diverse set of statements even if they do not answer every question?

To motivate the algorithm, we start by looking at the same example as earlier in the article.

If we ask about a railway to Tromsø or whether the county municipality should be abolished, we gather no information because the user’s answer does not discriminate between parties. The subspace spanned by the vector $\boldsymbol{e} = (1, 1)$ consists of such trivial statements. The best statement to present to the user is therefore “Drop the ferry-free E39 along the west coast,” since this statement lies furthest from $\operatorname{span} (\{ \boldsymbol{e} \} )$.

Algorithm for selecting statements in sequence

Let’s generalize from two dimensions to an arbitrary number of dimensions:

Start with the trivial subspace spanned by the statement $\boldsymbol{e} = (1, 1, \ldots, 1)$.
The first statement selected is the one that is furthest from $\operatorname{span} (\{ \boldsymbol{e} \} )$. This is $\boldsymbol{v}_1 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, where $P_S$ is a projection matrix that projects a vector onto $S$. The formula for such a projection matrix is $P_S = S S^{+} = S (S^T S)^{-1} S^T$, where the columns of the matrix $S$ are the vectors that span the space we want to project onto (here just $\boldsymbol{e}$).
The next statement selected should be far from both the trivial subspace $\boldsymbol{e}$ and far from the previously selected statement $\boldsymbol{v}_1$. So we choose the statement that is furthest from $\operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1 \} )$. In mathematical notation, this becomes $\boldsymbol{v}_2 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, where $S$ in this iteration is a matrix with columns $\boldsymbol{e}$ and $\boldsymbol{v}_1$.
We continue like this, selecting statements that are furthest from those we have already chosen. If $\operatorname{rank} \operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1, \ldots, \boldsymbol{v}_n \} )$ becomes equal to the dimensionality of the space (number of parties), then all remaining statements are in the space. Then the distance $\| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2 \approx 0$ for all remaining statements $\boldsymbol{v}_j$. The solution is to treat the selected statements as a FIFO queue: we remove the first selected statement $\boldsymbol{v}_1$ and try again. Then the next statement we select is always far from the most recently selected statements.

Python code for the algorithm is included at the end of the article. The figure below illustrates the concept in three dimensions. The next statement selected is always the one that is furthest from the space spanned by the trivial direction and previously selected statements. (In english, the first figure says “Start with (1, 1, 1)”, the next says “Choose first statement” and the third says “Choose second statement”.)

Results - bad statements

Let’s look at an example with statements that are similar to each other.

The first two statements are very similar. The third statement about Erna is on the opposite side of the vector space but still does not provide much new information because it is close to the space spanned by the first two statements. In other words: the question about Erna thus reveals little new information, despite the statement being on the opposite side of the vector space.

Results - good statements

Here is a selection of statements that are different from each other, identified by the algorithm above.

The first statement is far from trivial because half of the parties disagree and the other half agree. The second statement is answered differently by six parties and differs from both the trivial statement and the first statement. The algorithm continues to select statements that are consistently different from what the VAA user has already been presented with.

Summary and references

Voting advice applications quantify political opinions. Party responses can be represented as a matrix, where the rows (statements) and columns (parties) can be interpreted as vectors. Such a geometric interpretation gives us valuable insight: we can define what it means for a person to be close to a party, and we can formalize what it means for a statement to discriminate between parties.

The idea of viewing parties as vectors and ranking users by proximity is obvious, and for the 2019 municipal election, NRK used $p=1$ as a metric. (The fact that NRKbeta’s article in the link above describes middle school mathematics in phrases like “explain the calculation to me who doesn’t have math anxiety” and “I get a headache!” is sad.) The idea of describing a user as a convex combination of parties is my own. In practice, it is more difficult to implement in code and more challenging for an average user to understand. Simple algorithms have inherent value, and distance calculation provides a simple and understandable explanation.

The idea of viewing statements as vectors and selecting statements that are far from each other is my own. The inspiration comes from QR decomposition with pivoting, which is an algorithm for orthogonalizing a matrix. The pivot columns in QR can be seen as a greedy approximation of D-optimality in optimal experimental design, which maximizes the volume of a high-dimensional parallelepiped. The differences between my algorithm and QR decomposition are that (1) we always stay away from the trivial statement $\boldsymbol{e} = \boldsymbol{1}$ and that (2) we use a FIFO queue. Again I believe simplicitiy has inherent value: I like that this algorithm does not adapt the sequence of questions to what the user responded, since I do not like the idea of a personalized VAA.

This article focused mostly on mathematics, but a VAA must of course contain politically relevant statements, good graphic design, potentially the ability to skip statements or weight statements up, subjective analyses of the results, and so on.

Code

Here is a simple implementation of the algorithm that finds statements that are far from each other. I have not emphasized runtime, numerical stability, caching of calculations, etc. The code is certainly good enough to showcase the idea, but for larger datasets, it should be rewritten.

import collections  # Python 3.12
import numpy as np  # NumPy 2.1.3


def away_from_last_k(A, k=None, verbose=False):
    """Yields the column indices of A that are as far away from a continually
    updated subspace spanned by (1, 1, 1, ...) and a First-In-First-Out (FIFO)
    queue of length k. When a column index is yielded, it is added to the queue.

    Parameters
    ----------
    A : np.ndarray
        A matrix (2D array) with columns [v1 | v2 | ... ].
    k : int, optional
        Numbers of elements to keep in the FIFO queue. None means no limit.
    verbose : bool, optional
        Whether to print information. The default is False.

    Examples
    --------
    >>> A = np.array([[3, 0, 0], [0, 2, 0], [0, 0, 1], [2.5, 0, 0]]).T
    >>> list(away_from_last_k(A, k=None))
    [0, 1, 3, 2]
    >>> list(away_from_last_k(A, k=0)) #  Distance from span({(1, 1, 1)}) only
    [0, 3, 1, 2]
    >>> list(away_from_last_k(A, k=1))
    [0, 1, 3, 2]
    """
    subspace = collections.deque([], maxlen=k)  # Vector in FIFO subspace
    ones = np.ones(A.shape[0])  # Vector of ones, always in subspace
    remaining = dict(enumerate(A.T))  # Remaining column indices of A

    def distance_from_subspace(tuple_):
        idx, v = tuple_
        S = np.vstack([ones] + list(subspace)).T  # Full space as col. mat.
        P_S = S @ np.linalg.pinv(S)  # Projection matrix onto span(S)
        return np.linalg.norm(P_S @ v - v)  # Distance between v and span(S)

    while remaining:
        if verbose:
            inds = list(remaining.keys())
            print(f"\nCol idx of {len(remaining)} remaining vectors: {inds}")
            print(f"Vectors in subspace (excl. ones): {len(subspace)}")

        # Get idx of col vector furthest from S = [ones | subspace]
        idx, _ = max(remaining.items(), key=distance_from_subspace)
        maxdist = distance_from_subspace((idx, remaining[idx]))
        if np.isclose(maxdist, 0):
            if verbose:
                print("Distance was 0. Popping from subspace.")
            subspace.popleft()
            continue

        if verbose:
            print(f"Col idx {idx} furthest from subspace. Dist: {maxdist:.6f}")
        subspace.append(remaining.pop(idx))
        yield idx

Hvordan fungerer en valgomat?

Tommy Odland — Mon, 07 Apr 2025 00:00:00 +0200

Denne artikkelen presenterer matematikken bak valgomater. Den består av tre hoveddeler:

Del 1: Valgomater og geometri
Del 2: Hvordan forklarer en valgomat resultatene?
Del 3: Hvordan bør en valgomat presentere spørsmål for brukeren?

Artikkelen begynner på lavt matematisk nivå og blir progressivt mer krevende. En fordel med dette er at det er noe å lære for alle, uansett nivå. Ulempen er at artikkelen ikke har én tydelig målgruppe.

PS: I 2023 visualiserte jeg partiene i kommunevalget. Fokuset i den artikkelen var innsikt i partienes politikk. Denne artikkelen retter seg mer mot design av valgomater.

PSS: Idéene i denne artikkelen er implementert i min egen Valgomat for Stortingsvalget 2025.

Del 1: Valgomater og geometri

Meninger og matriser

For å lage en valgomat spør man politiske partier hvor enige de er i ulike påstander. Resultatet kvantifiseres ved at svar som “Helt enig!” blir oversatt til tall. Svarene kan deretter visualiseres i en tabell med en rad per påstand og en kolonne per parti. Nedenfor er “Helt enig” oversatt til $2$ og “Helt uenig” oversatt til $-2$.

Tabellen utgjør en matrise $X$. Radene og kolonnene har en geometrisk tolkning som vektorer. En vektor er en liste med tall, e.g. $(1, 2)$, og kan tolkes geometrisk som en pil eller som et punkt.

Påstander og partier

Påstander som vektorer

For å visualisere påstander og partier som vektorer må vi holde oss til to eller tre dimensjoner. Høyere dimensjoner lar seg ikke visualisere.

La oss studere geometrisk hva partiene SV og Sp mener om et utvalg påstander.

Oppe til høyre i figuren finner vi påstander som begge partiene er enige i. Nede til høyre finner vi påstander som SV er enige i, men som Sp er uenige i. Vi kan gjøre tilsvarende observasjoner for de to andre kvadrantene.

Hvilke påstander er gode og dårlige?

En god påstand lar oss skille partiene fra hverandre:

I figuren ovenfor er “Bygg jernbane nordover til Tromsø” en dårlig påstand. Uansett om du er enig eller uenig, avklarer ikke påstanden hvilket av de to partiene du er nærmest politisk. Trivielle påstander, som “Et rettferdig samfunn er bra”, er alle partiene enige i. Derfor gir de heller ingen informasjon. Det samme gjelder trivielle påstander som alle partiene er uenige i, for eksempel “Fylkeskommunen bør avskaffes” i figuren ovenfor.
“Dropp ferjefri E39 langs Vestlandet” er en god påstand fordi den diskriminerer mellom partiene. Om du er enig ligger du nærmere SV politisk. Er du derimot uenig ligger du nær Sp.

I to dimensjoner (to partier) er påstander gode dersom de ligger langt fra underrommet som spennes av vektoren $\boldsymbol{e} = (1, 1)$, altså den diagonale linja $y=x$. På slutten av denne artikkelen skal vi generalisere denne idéen for å konstruere en algoritme som finner en god rekkefølge å stille spørsmål på.

Partier som vektorer

Ovenfor betraktet vi radene (påstandene) i tabellen som vektorer. La oss nå se på det komplementære vektorrommet der kolonnene (partiene) er vektorer. Vi velger to tilfeldige påstander: en om ulv og en om statens eierskap i bedrifter.

To partier ligner hverandre dersom de ligger nær hverandre i meningsrommet. Vi kan for eksempel måle avstand som distanse på den horisontale aksen pluss distanse på den vertikale aksen.

Partiene R og AP er nær hverandre. Distansen er $0 + 1 = 1$.
Partiene R og V er langt fra hverandre. Distansen er $3 + 4 = 7$.

Distansen i dette rommet er nøkkelen til hvordan en valgomat avgjør hvilke partier du er mest enig med når du tar valgomaten. Det er dette vi nå skal ta for oss i del 2.

Del 2: Hvordan forklarer en valgomat resultatene?

Forklaring: rangering av partiene etter avstand

Ovenfor så vi at partienes likhet kan kvantifiseres ved å se på avstanden mellom partiene. Vi visualiserte dette i to dimensjoner, men beregningen er den samme i et vilkårlig antall dimensjoner.

Her er en matrise med tallene fra den første figuren i denne artikkelen:

Hvis du tar valgomaten og dine meninger er vektoren $\boldsymbol{y} = (-1, -2, 2, -2, 0, 0)$, så blir din avstand til AP $13$, Sp $14$, Frp $5$ og H $4$. Du ligger med andre ord sannsynligvis på høyresiden politisk, og bør vurdere å stemme Frp eller H.

Distansen $d_j$ mellom dine meninger $\boldsymbol{y}$ og et parti $j$ sine meninger er gitt av summen av avstandene mellom hver påstand $i$:

\begin{equation*} d_j = \sum_i | X_{ij} - y_i | \end{equation*}

Avstander, $p$-normer og vekting

Vi kan generalisere til en vilkårlig $p$-norm og legge til en vekting $w_i$:

\begin{equation*} d_j = \left( \sum_i w_i | X_{ij} - y_i |^p \right)^{(1/p)} \end{equation*}

Vektene $w_i$ er gjerne lik $1$ for de fleste påstander, mens $w_i = 2$ for kjernesaker eller saker som er spesielt viktige for deg. For saker der du ikke har noen mening kan vi sette $w_i = 0$. Verdien som velges for $p$ er avhengig av hvordan man vil måle distanse.

Dersom du mener at $\boldsymbol{y}$ er nærmere partiet $A$ i eksempelet nedenfor bør du velge $p=2$. Mener du derimot at $\boldsymbol{y}$ er nærmere $B$ bør du velge $p=1$.

\begin{align*} \boldsymbol{y} &= (0, 0, 0, 0) \\ A &= (1, 1, 1, 1) \\ B &= (0, 0, 0, 3) \end{align*}

Dette er en subjektiv vurdering, og personlig lener jeg mot $p=1$. Det samme gjør både NRK og VG; begge bruker $p=1$ i sine valgomater.

Beregningene resulterer til slutt i en liste av partier, sortert etter distanse:

1. H   (avstand 4 )
2. Frp (avstand 5 )
3. AP  (avstand 13)
4. Sp  (avstand 14)

En slik liste, sortert etter avstand, er én mulig måte å forklare brukerens resultater på. Men det er ikke den eneste; neste seksjon tar for seg en annen type forklaring.

Forklaring: Ditt standpunkt som en kombinasjon av partienes standpunkter

Partienes vektorer setter retning i rommet av meninger. Etter å ha tatt en valgomat får du også en plassering i dette rommet.

I figuren ovenfor er ditt standpunkt $\boldsymbol{y} = (1, -2)$.

Én måte å nå ditt standpunkt på med vektorregning er å ta MDGs standpunkt og legge på KrFs. Dette kan observeres geometrisk i figuren ovenfor, eller algebraisk:

\begin{align} \text{MDG} + \text{KrF} = \boldsymbol{y} \\ (2, -1) + (-1, -1) &= (1, -2) \end{align}

Utkast til optimeringsmodell

Her er en alternativ idé til distansebasert forklaring: vi uttrykker ditt standpunkt som en kombinasjon av partienes standpunkter. Vi leter da etter en kombinasjon av partier som beskriver deg ved at de tar oss så nært ditt standpunkt som mulig. Med andre ord er vi på jakt etter en vektor $\boldsymbol{\beta}$ som løser optimeringsproblemet

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \|. \end{align}

Denne modellen har noen praktiske utfordringer:

Den tillater kompliserte forklaringer i form av lange vektorer $\boldsymbol{\beta}$. For eksempel kan vi nå frem til $\boldsymbol{y}$ ved den lineære kombinasjonen
\begin{align} \text{V} + \text{R} + \text{H} + \text{Ap} + \text{MDG} = \boldsymbol{y}. \end{align}
Modellen tillater negative retninger fordi $\boldsymbol{\beta}$ kan bestå av negative tall. Dette er vanskelig å tolke. For eksempel er $-R = \boldsymbol{y}$ i eksempelet ovenfor, men du ønsker naturligvis å vite hvem du er enig med, heller enn å få vite at du er uenig med Rødt.
Modellen tillater forklaringer der summen av partienes bidrag $\boldsymbol{\beta}$ overskrider $1$. Dette er tilfellet i $\text{MDG} + \text{KrF} = \boldsymbol{y}$ og i den lange forklaringen ovenfor. Igjen er det vanskelig å tolke et slikt svar.

Alle disse problemene kan løses om vi begrenser oss til konvekse kombinasjoner.

Endelig optimeringsmodell

Vi fikser problemene til modellen ovenfor ved å utvide den. Heller enn å leter etter en hvilken som helst lineær kombinasjon av partier, leter vi etter den korteste konvekse kombinasjonen:

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \| + \alpha \| \boldsymbol{\beta} \| \\ & \text{subject to} && \beta_j \geq 0 && \\ & && \sum_j \beta_j = 1. && \end{align}

Begge noremen ovenfor er $2$-normer. Setter vi $\alpha = \epsilon$ (et lite positivt tall) i regulariseringsleddet, så vil modellen foretrekke løsninger der elementene i $\boldsymbol{\beta}$ jevnt over er små. Dersom $\alpha \to \infty$ så velges $\boldsymbol{\beta} \propto \boldsymbol{1}$, dersom $\alpha = -\epsilon$ vil modellen foretrekke løsninger som bruker så få partier som mulig, og dersom $\alpha \to -\infty$ er vi tilbake til en typisk valgomat som velger ett parti. Merk at når $\alpha < 0$ så er ikke problemet konvekst. Vi setter $\alpha = \epsilon$ i resten av artikkelen.

De to bibetingelsene på $\boldsymbol{\beta}$ er definisjonen på en konveks kombinasjon, og gjør at vi unngår negative retninger og summer som overskrider $1$. Summen $\sum_j \beta_j$ må være lik $1$, heller enn mindre enn $1$, fordi dersom $\boldsymbol{y} = (0, 1)$, $A = (0, 1)$ og $B = (0, 2)$ så forventer vi $\boldsymbol{\beta} = (1, 0)$ som svar, ikke $\boldsymbol{\beta} = (0, 0.5)$.

Med $2$-norm på begge leddene i objektivfunksjonen blir løsningen på optimeringsproblemet:

\begin{align} 0.69 \, \text{V} + 0.23 \, \text{H} + 0.08 \, \text{Frp} = \boldsymbol{y}. \end{align}

Om du går tilbake til figuren og sammenligner geometrien med denne løsningen, ser du at løsningen er fornuftig. Når det er sagt er det ingen absolutt fasit i matematisk modellering og andre variasjoner av en slik modell kan være hensiktsmessige.

Optimeringsmodellen på et større eksempel

La oss kjøre optimeringsmodellen på samme eksempel som tidligere:

Svaret er gitt nedenfor, og samsvarer godt med rangeringen som vi fikk tidligere.

1. H   (51 %)
2. Frp (45 %)
4. Sp  ( 4 %)
3. AP  ( 0 %)

Forskjellen på distansebasert og kombinasjonsbasert forklaring

Vi har sett to måter å forklare en brukers resultater på i en valgomat:

Distansebasert forklaring - rangerer partiene etter distanse fra brukeren. Svarer på spørsmålet “Hvilket enkeltparti er nærmest meg?”
Kombinasjonsbasert forklaring - brukeren uttrykkes som den nærmeste konvekse kombinasjonen av partiene. Svarer på spørsmålet “Hvilken koalisjon av partier er nærmest meg?”

Forskjellen mellom forklaringene er illustrert i figuren nedenfor.

Distansebasert forklaring sier at $\boldsymbol{y}$ er nærmest $B$, deretter $A$ og til slutt $C$. Distansebasert forklaring er innbyrdes uavhengig; avstanden til $B$ påvirker ikke avstanden til $A$. Dersom $B$ hadde forsvunnet ville avstanden til $A$ vært uendret.
Kombinasjonsbasert forklaring uttrykker standpunktet $\boldsymbol{y}$ som en konveks kombinasjon. Svaret i figuren ovenfor blir $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0, 0.7, 0.3)$. Optimeringsmodellens forklaring er innbyrdes avhengig. Dersom $B$ hadde forsvunnet ville forklaringen lent seg på $A$, men når $B$ finnes trengs ikke $A$ i det hele tatt for å forklare $\boldsymbol{y}$.

Figuren nedenfor viser et eksempel der forklaringene blir helt ulike.

Målt i distanse er $C$ nærmere $\boldsymbol{y}$ enn både $A$ og $B$. Likevel forklares $\boldsymbol{y}$ ved hjelp av $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0.5, 0.5, 0)$ uten å bruke $C$. Korteste distanse forteller deg hvilket enkeltparti som er nærmest. Optimeringsmodellen forteller deg hvilken kombinasjon av partier som står deg nærmest.

Del 3: Hvordan bør en valgomat presentere spørsmål for brukeren?

I denne seksjonen presenterer vi en algoritme som velger påstander som diskriminerer godt mellom partiene. Det kan være to grunner til at man ønsker dette:

Man skal bruke et utvalg påstander. Om partiene har svart på mange spørsmål, ønsker vi gjerne å begrense oss til et utvalg. Hvordan velger vi gode påstander som differensierer mellom partiene?
Tidlig avslutning. Man ønsker at brukeren skal kunne avslutte valgomaten etter et vilkårlig antall spørsmål. Hvordan garanterer vi at brukeren har blitt presentert for et allsidig sett med påstander?

For å motivere algoritmen begynner vi med å se på samme eksempel som tidligere i artikkelen.

Spør vi om jernbane til Tromsø eller om fylkeskommunen bør avskaffes, innhenter vi ingen informasjon fordi brukerens svar ikke diskriminerer mellom partiene. Underrommet som spennes av vektoren $\boldsymbol{e} = (1, 1)$ består av slike trivielle påstander. Den beste påstanden å presentere for brukeren er derimot “Dropp ferjefri E39 langs Vestlandet”, ettersom denne påstanden ligger lengst i fra $\operatorname{span} (\{ \boldsymbol{e} \} )$.

Algoritme for å velge påstander i rekkefølge

La oss generalisere fra to dimensjoner til et vilkårlig antall dimensjoner:

Start med det trivielle underrommet som spennes av påstanden $\boldsymbol{e} = (1, 1, \ldots, 1)$.
Den første påstanden som velges er den som er lengst fra $\operatorname{span} (\{ \boldsymbol{e} \} )$. Dette er $\boldsymbol{v}_1 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, der $P_S$ er en projeksjonsmatrise som projiserer en vektor på $S$. Formelen for en slik projeksjonsmatrise er $P_S = S S^{+} = S (S^T S)^{-1} S^T$, der kolonnene i matrisen $S$ er vektorene som spenner rommet vi ønsker å projisere på (her bare $\boldsymbol{e}$).
Den neste påstanden som velges skal være langt fra både det trivielle underrommet $\boldsymbol{e}$, og langt fra den forrige valgte påstanden $\boldsymbol{v}_1$. Vi velger altså påstanden som er lengst fra $\operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1 \} )$. I matematisk notasjon blir dette $\boldsymbol{v}_2 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, der $S$ i denne iterasjonen er en matrise med kolonner $\boldsymbol{e}$ og $\boldsymbol{v}_1$.
Slik fortsetter vi og velger påstander som er lengst fra de vi allerede har valgt. Dersom $\operatorname{rank} \operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1, \ldots, \boldsymbol{v}_n \} )$ blir lik dimensjonaliteten til rommet (antall partier), så er alle gjenværende påstander i rommet. Da blir distansen $\| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2 \approx 0$ for alle gjenværende påstander $\boldsymbol{v}_j$. Løsningen er å behandle de valgte påstandene som en FIFO-kø: vi fjerner den første valgte påstanden $\boldsymbol{v}_1$ og prøver på nytt. Da er neste påstand vi velger alltid langt fra de siste valgte påstandene.

Python-kode for algoritmen er vedlagt nederst i artikkelen. Figuren nedenfor illustrerer konseptet i tre dimensjoner. Den neste påstanden som velges er alltid den som er lengst i fra rommet som spennes av den trivielle retningen og tidligere valgte påstander.

Resultater - dårlige påstander

La oss se på et eksempel med påstander som innbyrdes ligner på hverandre.

De første to påstandene er veldig like. Den tredje påstanden om Erna er på motsatt side av vektorrommet, men gir likevel ikke mye ny informasjon fordi det er nær rommet som spennes av de første to spørsmålene. Spørsmålet om Erna avdekker altså lite ny informasjon, på tross av at påstanden er på motsatt side av vektorrommet.

Resultater - gode påstander

Her er et utvalg påstander som er ulike hverandre, avdekket av algoritmen ovenfor.

Den første påstanden er langt fra triviell, fordi halvparten av partiene er uenige og den andre halvparten er enige. Den andre påstanden er ulikt besvart av seks partier, og skiller seg både fra den trivielle påstanden og den første påstanden. Slik fortsetter algoritmen med å velge påstander som hele tiden er annerledes enn det brukeren av valgomaten har blitt presentert for.

Oppsummering og referanser

Valgomater kvantifiserer politiske meninger. Partienes svar kan representeres som en matrise, der radene (påstander) og kolonnene (partier) kan tolkes vektorer. En slik geometrisk tolkning gir oss verdifull innsikt: vi kan definere hva det vil si at en person er nær et parti og vi kan formalisere hva det vil si at en påstand diskriminerer mellom partier.

Idéen som å se på partier som vektorer og rangere brukeren etter nærhet er rimelig åpenbar, og for kommunevalget 2019 brukte NRK $p=1$ som metrikk. (At NRKbetas artikkel pakker ungdomsskolematematikk inn i fraser som “forklar utregningen for meg som ikke har matteskrekk” og “Jeg får vondt i hodet!” er sørgelig.) Idéen om å beskrive en bruker som en konveks kombinasjon av partier er min egen. I praksis er det vanskeligere å implementere i kode, samt mer krevende å forstå for en gjennomsnittlig bruker. Enkle algoritmer har iboende verdi og distanse-beregning gir en enkel og forståelig forklaring.

Idéen om å se på påstander som vektorer og velge ut påstander som er langt fra hverandre er min egen. Inspirasjonen kommer fra QR-dekomposisjon med pivotering, som er en algoritme for å ortogonalisere en matrise. Pivot-kolonnene i QR kan sees på som en grådig approksimasjon av D-optimalitet i optimal eksperimentdesign, som maksimerer volumet av en høydimensjonell parallellepiped. Forskjellene mellom min algoritme og QR-dekomposisjon er at (1) vi alltid holder oss borte fra den trivielle påstanden $\boldsymbol{e} = \boldsymbol{1}$ og at (2) vi bruker en FIFO-kø.

Denne artikkelen fokuserte mest på matematikk, men en valgomat må selvsagt ha politisk relevante påstander, godt grafisk design, potensielt mulighet for å hoppe over påstander eller vekte påstander opp, subjektive analyser av resultatene, og så videre.

Kode

Her er en enkel implementasjon av algoritmen som finner påstander som er langt fra hverandre. Jeg har ikke vektlagt kjøretid, numerisk stabilitet, caching av beregninger, etc. Koden er absolutt god nok for bruksområdet, men for større datasett bør man skrive om.

import collections  # Python 3.12
import numpy as np  # NumPy 2.1.3


def away_from_last_k(A, k=None, verbose=False):
    """Yields the column indices of A that are as far away from a continually
    updated subspace spanned by (1, 1, 1, ...) and a First-In-First-Out (FIFO)
    queue of length k. When a column index is yielded, it is added to the queue.

    Parameters
    ----------
    A : np.ndarray
        A matrix (2D array) with columns [v1 | v2 | ... ].
    k : int, optional
        Numbers of elements to keep in the FIFO queue. None means no limit.
    verbose : bool, optional
        Whether to print information. The default is False.

    Examples
    --------
    >>> A = np.array([[3, 0, 0], [0, 2, 0], [0, 0, 1], [2.5, 0, 0]]).T
    >>> list(away_from_last_k(A, k=None))
    [0, 1, 3, 2]
    >>> list(away_from_last_k(A, k=0)) #  Distance from span({(1, 1, 1)}) only
    [0, 3, 1, 2]
    >>> list(away_from_last_k(A, k=1))
    [0, 1, 3, 2]
    """
    subspace = collections.deque([], maxlen=k)  # Vectors in FIFO subspace
    ones = np.ones(A.shape[0])  # Vector of ones, always in subspace
    remaining = dict(enumerate(A.T))  # Remaining column indices of A

    def distance_from_subspace(tuple_):
        idx, v = tuple_
        S = np.vstack([ones] + list(subspace)).T  # Full space as col. mat.
        P_S = S @ np.linalg.pinv(S)  # Projection matrix onto span(S)
        return np.linalg.norm(P_S @ v - v)  # Distance between v and span(S)

    while remaining:
        if verbose:
            inds = list(remaining.keys())
            print(f"\nCol idx of {len(remaining)} remaining vectors: {inds}")
            print(f"Vectors in subspace (excl. ones): {len(subspace)}")

        # Get idx of col vector furthest from S = [ones | subspace]
        idx, _ = max(remaining.items(), key=distance_from_subspace)
        maxdist = distance_from_subspace((idx, remaining[idx]))
        if np.isclose(maxdist, 0):
            if verbose:
                print("Distance was 0. Popping from subspace.")
            subspace.popleft()
            continue

        if verbose:
            print(f"Col idx {idx} furthest from subspace. Dist: {maxdist:.6f}")
        subspace.append(remaining.pop(idx))
        yield idx

Rettferdig fordeling av boliglån med ulik eierandel

Tommy Odland — Fri, 07 Mar 2025 00:00:00 +0100

Anta at to personer skal kjøpe en bolig sammen, men har ulik egenkapital og/eller ulik betalingsevne på boliglånet. Hva er en økonomisk rettferdig fordeling?

Ulik egenkapital, lik fordeling av lånet

I DNBs artikkel “del boliglånet rettferdig” blir følgende fordeling foreslått som rettferdig dersom partene ikke har lik egenkapital. Alle tall er i tusen kroner.

	Johan	Andrea
Andel av lån	1 500	1 500
Egenkapital	800	200
Total andel av kjøpesum	2 300	1 700
Andel ift. totalsum	2 300 / 4 000	1 700 / 4 000
Eierandel	57,5 %	42,5 %

Men dette kan oppleves som urettferdig. For å forstå hvorfor gjør vi eksempelet mer ekstremt—hva om Johan har mye mer penger enn Andrea?

Da kan vi få en fordeling av lån og egenkapital som for eksempel ser slik ut:

	Johan	Andrea
Andel av lån	100	100
Egenkapital	3 800	0
Total andel av kjøpesum	3 900	100
Andel ift. totalsum	3 900 / 4 000	100 / 4 000
Eierandel	97,5 %	2,5 %

Johan har gått inn med 97,5 %, og skal få 97,5 % av salget. Men Andrea har likevel gjort et kupp! Selv om hun får lite når boligen selges, så hun bor i praksis nesten gratis. Hun bruker tross alt halve boligen i hverdagen, men betaler nesten ikke noe for det.

En mer rettferdig fordeling?

For å korrigere for skjevheten som oppstår når Andrea disponerer halve boligen kan vi anse henne som delvis å være leietaker. Hun betaler da leie for den andelen av hennes 50 % bruksrett som ikke er dekket av eierandelen. Utregningen ser slik ut i Python:

# Egenkapital for begge personene
egenkapital_A = 3_800
egenkapital_B = 0
egenkapital = egenkapital_A + egenkapital_B

# Andel av lån for begge personene
lån_A = 100
lån_B = 100
lån = lån_A + lån_B

# Utregninger
boligverdi = egenkapital + lån
eierskap_A = egenkapital_A + lån_A
eierskap_B = egenkapital_B + lån_B

print(f"Boligens verdi: {boligverdi:.2f}")
print(f"Belåningsgrad: {lån / boligverdi:.2%}\n")
print(f"Person A eier {eierskap_A / boligverdi:.1%}")
print(f"Person B eier {eierskap_B / boligverdi:.1%}\n")
print(f"Person A skal betale {lån_A / lån:.1%} av lånet hver måned")
print(f"Person B skal betale {lån_B / lån:.1%} av lånet hver måned\n")

# Total leiepris per måned for tilsvarende bolig
leiepris = 15
leieprosent_A = max(1 - 2 * (eierskap_A / boligverdi), 0)
leieprosent_B = max(1 - 2 * (eierskap_B / boligverdi), 0)

if leieprosent_A > 0:
    print(f"Person A er {leieprosent_A:.1%} leietaker")
    print(f"Person A betaler {leieprosent_A * leiepris / 2:.2f} til person B i leie")
if leieprosent_B > 0:
    print(f"Person B er {leieprosent_B:.1%} leietaker")
    print(f"Person B betaler {leieprosent_B * leiepris / 2:.2f} til person A i leie")

Kjører man koden blir resultatet:

Boligens verdi: 4000.00
Belåningsgrad: 5.00%

Person A eier 97.5%
Person B eier 2.5%

Person A skal betale 50.0% av lånet hver måned
Person B skal betale 50.0% av lånet hver måned

Person B er 95.0% leietaker
Person B betaler 7.12 til person A i leie

I dette eksempelet ville en tilsvarende bolig kostet 15 tusen å leie per måned. Andrea disponerer halve boligen, og av denne halvdelen leier hun i praksis 95 % og eier 5 %. Hun eier altså 5 % av sin 50 % bruksandel, som blir 0,05 * 0,5 = 2,5 % av boligens totalverdi. Hun må betale 7,12 tusen per måned i leie til Johan.

Det originale eksempelet

La oss gå tilbake til det originale eksempelet der Andrea går inn med 200 tusen i egenkapital og Johan går inn med 800 tusen. Tallene ser da slik ut:

egenkapital_A = 800
egenkapital_B = 200
lån_A = 1500
lån_B = 1500

Utregningen blir nå at Andrea eier 42,5 % av boligen. Men med en antatt månedlig leiekostnad på 15 tusen må hun både (1) betale ned halvparten av lånet per måned og (2) betale 1,13 tusen kroner i leie til Johan ettersom han gikk inn med en større andel egenkapital.

Et ekstremt eksempel

Hva om Andrea ikke går inn med noe penger i det hele tatt?

egenkapital_A = 1_000
egenkapital_B = 0
lån_A = 3_000
lån_B = 0

Da må Andrea betale 7,5 tusen i leie til Johan hver måned, fordi det er halvparten av boligens antatte leiekostnad. Hadde vi fulgt DNBs logikk ville hun bodd gratis.

Konklusjon

DNBs eksempel tar ikke hensyn til at parten som eier den minste andelen i boligen disponerer halve boligen. For å korrigere for dette kan vi anse parten som eier minst som å delvis være leietaker. Om leietakeren eier 42,5 % av boligen men 50 % av lånet, slik som i DNBs eksempel, så får hun 42,5 % av salgssummen og betaler 42,5 % av vedlikeholdskostnadene. Hun betaler også 50 % av lånet, pluss leie til den andre parten.

Dersom egenkapital er begrensningen, er en annen mulighet at den ene parten låner penger av den andre parten. Da kan begge gå inn med 50 % i både egenkapital og lån, men det er tatt opp et privat lån på siden. I praksis kan det oppleves kjipt å være delvis leietaker, spesielt om eierbrøken uansett er tilnærmet lik 50 %. Da er det nok mer praktisk å følge DNBs utregning.

Rent matematisk er lærdommen at man bør konstruere ekstreme eksempler for å sjekke om en utregning gir mening. DNBs utregning feiler på denne testen og er åpenbart utrettferdig når eierbrøken blir for skjev.

Se også “Slik deler dere boliglånet og utgiftene riktig” hos Pengeverkstedet. Artikkelen sier at totalen er lik egenkapital pluss lån, og korrigerer ikke for at hver part disponerer halvparten av boligen.
Om du vil ha en utfordring kan du forsøke å generalisere logikken ovenfor. Hva blir utregningen om mer enn to personer går sammen med ulik egenkapital og/eller ulik betalingsevne på lån?

Strømstøtte, fastpris, makspris og insentiver

Tommy Odland — Tue, 04 Feb 2025 00:00:00 +0100

Strømprisen har tatt stor plass i media de siste årene. Arbeiderpartiet foreslår nå å bytte ut strømstøtte med fastpris, mens Fremskrittspartiet og Kristelig Folkeparti vil ha makspris.

I denne artikkelen skal vi visualisere disse ordningene, fikse de svake insentivene ved hjelp av litt enkel matematikk på VGS-nivå, og se at politikernes tenkemåte rundt strømpriser har fellestrekk med hvordan de tenker på f.eks. fraværsgrensa.

Konstant pris, lineær pris og insentiver

I Oslo kommune er det to måter å fastsette vann- og avløpsgebyrer på:

Konstant pris basert på en arealberegning, som er uavhengig av forbruk
Lineær pris proporsjonal med forbruk (pluss en fastpris)

I motsetning til vannpris, er strømpris alltid basert på kundens forbruk. Om prisen var konstant, ville det ikke vært noe økonomisk insentiv for å isolere, holde temperaturen nede, kjøpe varmepumpe, eller spare strøm på andre måter.

Strømprisen har en annen egenskap som vannprisen ikke har; den er ustabil over tid. Prisen varierer innad i døgnet, innad året, og fra år til år.

Dagens strømstøtte

Figuren nedenfor viser prisen per kvartal de siste ti årene. Dataene er fra NRKs.no / SSB.

Dagens strømstøtte-ordning er også vist i figuren. Opp til 73 øre betaler man alt selv, deretter betaler man 10% av overskytende. La oss kalle spotprisen for $x$ og nettoprisen som kunden betaler for $y$. Da kan formelen for nettopris skrives som

\begin{equation*} y(x) = \min\left( x, 73 + 0.1 (x - 73) \right). \end{equation*}

Edit: Fra og med 1. januar 2025 er strømstøtten 75 øre, ikke 73 øre.

Fastpris

Arbeidetpartiet lanserte januar 2025 forslag om fastprisavtaler på 40 øre/kWh. En slik avtale er visualisert i denne artikkelen og gjenskapt i figuren nedenfor. Formelen er enkel; uansett hva spotprisen $x$ er, blir nettoprisen $y$ lik $40$ øre/kWh.

\begin{equation*} y(x) = 40 \end{equation*}

Makspris

Frp og KrF ønsker 100 prosent kompensasjon for priser over 50 øre/kWh, altså en makspris. Formelen blir

\begin{equation*} y(x) = \min\left(x, 50 \right), \end{equation*}

og resultatet av en slik regel er vist i figuren nedenfor.

Insentiver og den deriverte

Fastprisforslaget får kritikk for å ta bort insentiver. Kritikken kommer fra professorer, økonomer, og fra Støre selv, som tidligere uttalte at:

“Makspris på strøm løser ikke dette problemet, makspris gjør ikke noe med gassprisene og det fyller heller ikke vannmagasinene.”

Dette er riktig, men vi må huske at det er snakk om en fastpris per kilowattime, ikke en fastpris per måned. En fastpris per kilowattime insentiverer til lavt forbruk; kutter man forbruket til halvparten, betaler man også halvparten. Derimot forsvinner insentiver for å bruke strøm når den er billig, heller enn når den er dyr, under en slik ordning.

Politikere klamrer seg ofte til harde grenser som fastpriser og makspriser når de vil regulere et marked. Ofte tenker de at det må gå en hard grense mellom regulering og frie markeder i seg selv. Her er noen eksempler der harde grenser kunne vært myknet opp:

Skoleopptak. Politikerne i Oslo har i mange år ment at skoleopptak enten må være helt karakterbasert, eller basert på ren loddtrekning. En mellomting er mulig med vektet loddtrekning, som jeg skrev om i 2020.
Fraværsgrensa. En fraværsprosent på 9% er uproblematisk, men med 11% fravær mister man hele standpunktkarakteren. En mykere overgang ville vært mulig ved å redusere karakteren noe før man mister hele karakteren.
Arbeidsavklaringspenger (AAP). Du kvalifiserer om du er 50% syk, men ikke om du er 49% syk.
Toll. Frem til 1. januar 2024 var det 350-kronersgrense for toll- og avgiftsfritak på varer kjøpt fra utlandet.

Ofte er harde grenser både nødvendig, enkelt og praktisk; for eksempel 18-års aldersgrense på alkohol. Andre ganger kan harde grenser skape perverse insentiver og uheldige tilpasninger, som når elever strategisk planlegger fravær rett under grensen. Da bør vi vurdere mer gradvise overganger som bedre reflekterer de underliggende realitetene. Dette er en enkel form for mekanismedesign.

I våre modeller $y(x)$ er insentiver analogt med den deriverte, som er stigningen til funksjonen. En flat funksjon dreper koblingen mellom input og output, som her er bruttopris og nettopris. For å insentivere strømforbruk når det er billig, kan vi glatte ut Frp og KrFs makspris.

Glatt makspris

Vi kan glatte ut makspris-forslaget til Frp og KrF for å beholde insentiver, og samtidig garantere at prisen aldri overstiger 50 øre/kWh. Insentivene blir svake, men det er bedre enn ingen ting.

Vi starter med softplus-funksjonen $s(x; \alpha) = \log \left(1 + \exp\left( \alpha x \right) \right) / \alpha$, skifter og skalerer til

\begin{equation*} y(x) = 50 - s(50 - x; \alpha) \end{equation*}

og velger en fornuftig $\alpha$-verdi for å kontrollere hvor glatt funksjonen blir. Resultatet er vist i figuren nedenfor.

Logaritmisk pris

Enda sterkere insentiver får man om man ikke begrenser seg til en makspris på 50 øre/kWh. Det politikerne egentlig ønsker å gjøre er å skjerme forbrukerne mot den enorme prisvariasjonen. Grunnen til at de strekker seg mot fastpris og makspris heller enn en logaritme, kan være at de ikke vet hva en logaritme er, eller tror at befolkningen ikke vet hva en logaritme er.

For å begrense svingningene, men samtidig beholde insentiver i større grad enn med makspris, kan vi bruke en logaritmisk pris. Formelen kan skrives som

\begin{equation*} y(x) = \log \left( 1 + \alpha x \right) / \alpha , \end{equation*}

der $\alpha$-verdien kontrollerer styrken. En logaritmisk funksjon vil begrense prisen kraftig, men la kunden føle toppene mer enn med makspris. Logaritmen er en enkel funksjonell form som oppnår ønsket effekt, men mange andre funksjoner kan også oppnå lignende effekter og være vel så gode.

Oppsummering

Hvordan strømmarkedet skal fungere er en politisk beslutning, og de fleste problemene er uten tvil politisk skapt heller enn noe som kan fikses med matematikk. Når det er sagt er det fremdeles interessant å analysere politikernes tankemåte matematisk, og det er nok lurt å lytte til økonomenes betraktninger. Politikere tyr ofte til harde grenser som fastpris og makspris, men vi har sett at enkel matematikk kan gi oss mer nyanserte løsninger.

Vi skisserte to prismekanismer som både beskytter forbrukerne mot ekstreme prissvingninger og samtidig bevarer insentiver for fornuftig strømbruk: en myk makspris og en logaritmisk pris. Samme tankegang kunne forbedret andre områder der harde grenser skaper uheldige virkninger.

Beer tasting

Tommy Odland — Fri, 10 Jan 2025 00:00:00 +0100

In this article we’ll analyze data from a blind test. Blind tests are fun experiments that can determine, for instance, if family members or friends can distinguish between:

Pepsi and Cola
Red and white wines
Brand-name and generic foods

Analysis of such data is interesting, especially when you’ve designed the experiment and collected the data yourself. You might learn some statistics along the way, and that knowledge could prove useful in industry settings as well as in everyday situations.

The beer tasting experiment

A friend wanted to determine her true beer preferences, so we decided to run a blind test:

We bought 6 different brands of beer.
She tasted each brand on 3 different occasions.
The ordering of the 18 tastings was randomized.
Each of the 18 tastings was conducted on a separate day.

In every tasting, the unknown beer brand was given a rating on a scale from 1 to 10. Knowledge of which brands were given which rating was not disclosed until the experiment was finished. The results of the full experiment are shown in the figure below, and the full dataset is available at the end of this article.

The Oslo-based beer brand “Schous” is the winner, with an average rating of

\begin{align*} (7 + 7 + 8)/3 = 22/3 \approx 7.33 \end{align*}

Can my friend tell beers apart?

Let us examine the figure above a bit more critically. The ratings are somewhat consistent, in the sense that they do not look like random integers between 1 and 10. However, it is surprising that the very same beer is awarded both the lowest observed rating (3) and the highest observed rating (8).

This raises the question: can my friend consistently tell beers apart? Let’s test this with a permutation hypothesis test. We’ll test if the explained variance $r^2$ in the dataset is higher than what could expect if knowledge of beer brands contained no information.

Let $y_i$ be the rating at tasting $i$. Defint the variable $x_{ij}$ to be a one-hot encoding of the brands: it is equal to $1$ if beer brand $j$ was tasted in tasting $i$, otherwise it is equal to $0$. We’ll assume a linear model with structure

\begin{align*} y_i = \sum_j \beta_j x_{ij} + \epsilon_i, \end{align*}

where $\epsilon_i$ is normally distributed noise. The coefficients $\beta_j$ have a natural interpretation: they are the average ratings of each brand $j$.

Fitting this model to the data gives an explained variance $r^2 = 0.375$. The beer brand explains around one third of the total variane in the ratings. To put this number into perspective, consider that:

A perfectly consistent beer-rater would achieve $r^2 = 1$ on a training set (and on a test set), since the beer brand would explain their ratings perfectly and account for all variation.
A completely random beer-rater would on average achieve $r^2 \approx 0$ on a test set, since the beer brand would not explain the ratings at all. However, on a training set they would achieve $r^2 > 0$ since there are six degrees of freedom in the brand coefficients $\beta_j$, and these will overfit and capture some of the variance.

Let the null hypothesis $H_0$ be that the beer brands have no effect on the ratings at all. Permuting the rows of the data set randomly (so there is no relationship between brands and ratings) produces the following distribution of $r^2$ under the null hypothesis.

If $H_0$ is true (brands have no effect), we will on average observe an $r^2$ of around $0.292$. The variation in $r^2$-values in the permuted data sets is large, and anything between $0.08$ and $0.57$ is relatively common (coverage of 90%).
The observed $r^2$ is $0.375$, which is certainly higher than the average, but by no means extreme.
The $p$-value, which is the probability of observing a value of $0.375$ or higher, given that $H_0$ is true, is around $0.289$.

Scientists typically reject $H_0$ if the $p$-value is lower than $0.05$. Here the $p$-value is much larger; this level of consistency in the ratings is well within the realm of what could happen even if my friend cannot tell beers apart. It might be that any apparent consistency is attributable to chance rather than genuine tasting ability.

Summary

The brand “Schous” is the winner, but there is a concerning amount of variation in the ratings within each brand. It’s not clear that my friend can consistently tell beers apart in this experiment. The $p$-value is so high that any scientific paper would report a “no effect” conclusion, failing to discard $H_0$. That being said, I am also reluctant to decisively claim that my friend cannot tell beers apart. While $H_0$ is mathematically convenient, it imposes a somewhat one-sided burden of proof.

People’s tasting abilities may be less consistent than they believe. The paper An Examination of Judge Reliability at a major U.S. Wine Competition investigates this to some extent. A larger, more involved experiment might be needed to figure out if my friend can tell beers apart. For now, the data suggests Schous remains the preferred choice.

Dataset

Here is the dataset used in this article in .csv format:

order,beer_brand,rating
1,Frydenlund,3
2,CB,7
3,Pokal,5
4,Tuborg,3
5,Frydenlund,5
6,Tuborg,8
7,Frydenlund,5
8,Hansa,6
9,CB,7
10,CB,5
11,Pokal,3
12,Schous,7
13,Hansa,3
14,Schous,8
15,Hansa,8
16,Tuborg,4
17,Schous,7
18,Pokal,5

Python code

Here’s a sketch of Python code to perform the statistical test. Python 3.12.8 was used with Pandas 2.2.3, NumPy 1.26.4 and scikit-learn 1.5.2.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
import numpy as np

# Read dataset
df = pd.read_csv("dataset.csv")

# Create X and y
encoder = OneHotEncoder(drop=None)
X = encoder.fit_transform(df[["beer_brand"]])
y = df["rating"].values

# Fit linear model, compute r2
linreg = LinearRegression(fit_intercept=False)
linreg.fit(X, y)
r2 = linreg.score(X, y)
print(f"r^2={r2:.3f}")  # r^2=0.375

# Randomly permute data and fit models
rng = np.random.default_rng(42)
r2_scores = []
for simulation in range(999):
    y_permuted = rng.permutation(y)
    linreg.fit(X, y_permuted)
    r2_scores.append(linreg.score(X, y_permuted))

p_value = np.mean(np.array(r2_scores) >= r2)
print(f"p_value={p_value:.3f}")  # p_value=0.311

Papers read in 2024

Tommy Odland — Sat, 14 Dec 2024 00:00:00 +0100

I try to read a few math books every year, but the more I learn, the more overlap I see in their content. In 2024 I decided to organize with a group of friends and read papers instead.

We read 17 papers together, and I skimmed many more to decide on which 17 to read in the group. Below is the list of papers, which might be useful if you want to learn some new data science related conceps, or if you want you organize a reading club of your own.

Lessons learned

Here’s what I learned:

Most papers are not ideal for learning; the authors are not great writers and they have limited space to convey their ideas.
Most practitioners should read books rather than papers to gain general knowledge.
Finding papers that are on an appropriate technical level and of interest to everyone in a group is difficult.

Before reading papers, it might be useful to read some guides on reading papers:

How to Read a Paper (2007)
How to Read a Technical Paper (2009)
A guide to reading scientific papers (2016)
How to read Machine Learning and Deep Learning Research papers (2021)

Here is the list of papers that we read. Honorable mentions are papers that I was interested in, but ended up not choosing for the group. Papers that I for some reason enjoyed more than others get a star (⭐).

Data Science

We chose to start with two papers on the culture of data science and practical tips for data science projects.

50 Years of Data Science. David Donoho (2017)
⭐ Good enough practices in scientific computing. Wilson et al. (2017)

Honorable mentions

A few useful things to know about machine learning. Pedro Domingos (2012)
⭐ Statistical Modeling: The Two Cultures. Leo Breiman (2001)

Recommendation engines

We looked into recommendation engines: the Netflix price, matrix factorization and extensions to neural networks.

Matrix Factorization Techniques for Recommender Systems. Koren et al. (2009)
⭐ Factorization Machines. Steffen Rendle (2010)
Neural Collaborative Filtering. He et al. (2017) [arXiv]

Honorable mentions

Collaborative Filtering for Implicit Feedback Datasets. Hu et al. (2008)
Neural Factorization Machines for Sparse Predictive Analytics. He et al. (2017)

Continuous optimization

These articles cover practical methods for solving optimization problems.

An overview of gradient descent optimization algorithms. Sebastian Ruder (2016)
A comparison of numerical optimizers for logistic regression. Thomas P. Minka (2003)
An Interior-Point Method for Large-Scale l1-Regularized Least Squares. Kim et al. (2007)

Honorable mentions

Enhancing Sparsity by Reweighted L1 Minimization. Candes et al. (2007)

Operations research

In the age of data science, operations research is underrated.

UPS Optimizes Delivery Routes. Holland et al. (2017)
Increasing the Responsiveness of Firefighter Services …. van den Berg et al. (2017)
Data analysis and optimization for (citi)bike sharing. O’Mahony et al. (2015)

Honorable mentions

Defending Critical Infrastructure. Brown et al. (2006)

Information and compression

Information theory has a rich history and is the foundation of many modern techniques.

A Mathematical Theory of Communication. Claude E. Shannon (1948)
Arithmetic coding for data compression. Witten et al. (1987)

Statistics

More statistics never hurts, and seeing how practitioners attack problems is always interesting.

Should the Olympic sprint skaters run 500 meter twice?. Nils Lid Hjort (1994)
⭐ A Conceptual Introduction to Hamiltonian Monte Carlo. Michael Betancourt (2017)
MCMC using Hamiltonian dynamics. Radford M. Neal (2012)
⭐ Hierarchical generalized additive models in ecology. Pedersen et al. (2019)

Honorable mentions

Mindless statistics. Gerd Gigerenzer (2004)
Prediction, Estimation, and Attribution. Bradley Efron (2020)
Hyperparameters: Optimize, or Integrate Out? David J.C. MacKay (1996)

Solving NRK’s game “Former”

Tommy Odland — Tue, 26 Nov 2024 00:00:00 +0100

NRK published a fun game on their website called “Former” (English: “Shapes”). The gameplay is straightforward: the player clicks on a cell on the board, and as a result every connected cell (up, down, left and right) with the same color vanishes. The surrounding shapes fall down and the player is ready for the next move.

The goal is to make all shapes vanish in as few clicks as possible.

In this article we formulate the game as a graph search problem, show how to reduce the size of the graph and attempt to solve it using various graph search algorithms. The full code is available at github.com/tommyod/NRK-former-game.

Games as directed graphs

Games like these can be represented as directed graphs: nodes represent game states and directed edges represent moves (clicks). The initial node is the first board we see and the terminal node is the empty board.

Any path from the initial node to the terminal node is a solution.
The optimal solution is the shortest solution path.

The original game has 9 rows and 7 columns, which leads to a huge graph and therefore a huge search space. In the game instance shown in the animation above:

There are 39 possible first moves
There are 1,446 possible sequences of two moves
There are 51,162 possible sequences of three moves
There are 1,730,312 possible sequences of four moves
There are 55,950,299 possible sequences of five moves

We count one move as one click on any cell in a connected group of cells of the same color. The search space continues to grow as we search for sequences of moves. The branching factor is roughly 35, so at depth 10 we can expect there to be around $35^{10} \approx 10^{15}$ nodes.

A simpler instance

Let’s examine a simple 2-color instance on a board with 2 rows and 2 columns. Below is the full search space for such a problem instance.

The initial node is at the top and the terminal node is at the bottom of the figure. There are 24 nodes in total. The game can be solved in 3 or 4 moves, depending on which path we choose. The original game instance has billions of such nodes, and solving it comes down to searching the graph efficiently.

Reducing the search space

The first observation we make is that a click on any group of connected colors leads to the same node. It does not matter which specific cell we click as long as it’s connected to a group of the same color, so we might as well only consider the top-left cell in any group of connected colors.

The second observation is that any permutation of colors represents the same state. If there are $4$ colors, there are $4! = 4 \times 3 \times 2 \times 1 = 24$ ways to permute them. Applying this observation we can reduce the search space from 24 to 12 nodes in this specific problem instance.

The third observation is that horizontal flips represent the same state. If we’ve solved the game at some node $n$, then we’ve also solved the node $n'$ with flipped columns. There are two such states (flipped and not flipped), so this can only reduce the search space by a factor of 2. By applying this observation we can reduce the search space down to 8 nodes.

Based on these observations we can define a canonical representation of each game state (node) where we re-label the colors and flip the game. There is a tradeoff between:

the computation time spent transforming a game state into canonical form
the storage of game states as we search the graph

Re-coloring reduces the size of the search space by quite a bit, so we’ll apply this transformation. We’ll skip the flipping and full canonicalization.

Optimal solutions

The simplest algorithm that is guaranteed to produce an optimal solution is breadth-first search (BFS). It first explores all nodes one move away from the initial node, then all nodes two moves away, then all nodes three nodes away, and so forth. When we get to an empty board we know that no shorter path exists.

Unfortunately, BFS holds too many nodes in memory and explores every path. A smarter approach is to use A* search. All nodes are put in a priority queue, and we continually pop off the node with the smallest value of

\begin{equation*} f(n) = g(n) + h(n), \end{equation*}

where $n$ is a node, $g(n)$ is the length of the path so far and $h(n)$ is an admissible heuristic that estimates the total remaining path.

Admissible heuristics

If the heuristic $h$ never overestimates the length of the remaining path, then A* is guaranteed to return an optimal solution. Such a heuristic is called admissible.

An obvious admissible $h$ is the unique number of colors left on the board. This never overestimates the number of remaining moves needed, because at least one click is required to remove all cells of a given color.
An even better $h$ is described in the code below. It’s based on the following idea: if any given color appears on both sides of a column not containing that color, then at least two moves are needed to get rid of that color.

def consecutive_groups(iterable):
    """Count how many consecutive groups of True there are.

    Examples
    --------
    >>> consecutive_groups([False, True, True, False, False, True, True])
    2
    """
    return sum(1 for key, group in itertools.groupby(iterable) if key)


def a_star_heuristic(board: Board) -> int:
    """A lower bound on how many moves are needed to solve.

    This function examines each color in turn. For each color, count whether
    it appears in each column. For each group of columns, separated by
    columns where the color does not appear, we need at least one click.

    Examples
    --------
    >>> board = Board([[1, 2, 1],
    ...                [2, 2, 2],
    ...                [1, 2, 1]])
    >>> a_star_heuristic(board)
    3
    >>> board = Board([[1, 3, 1],
    ...                [2, 3, 1],
    ...                [1, 3, 1]])
    >>> a_star_heuristic(board)
    4
    """
    # Transpose the grid
    matrix = list(map(list, zip(*board.grid)))

    # Get the unique integers larger than zero in the matrix
    unique_integers = set(c for col in matrix for c in col if c > 0)

    # For every integer, see if it exists in each column
    integer_in_col = (((c in col) for col in matrix) for c in unique_integers)

    # Count groups of integers separated by other integers
    return sum(consecutive_groups(integer) for integer in integer_in_col)

Unfortunately, even with this heuristic A* is too slow for boards of a decent size. The figure below shows solution times for A* and BFS as a function of board size. Solution times grow exponentially. Whether we re-label colors or not seems to have little practical impact on solution times.

Good-enough solutions in limited time

A heuristic graph search algorithm (not to be confused with the heuristic function $h$ in A* search) is an algorithm that attempts to find a good solution. With heuristics, there are no guarantees that the optimal solution is found.

Some simple heuristics are:

Random search. Always choose a random move.
Greedy best-first search. Always choose the move that clears the most cells.
Randomized best-first search. Choose a move at random, but use weighted sampling and weigh the moves by the number of cells they remove. Setting weight = cells_removed**power for some power >= 0 is one way to parametrize this graph search. If power = 0 then the search behaves like random search, and if power is very large then it behaves like greedy best-first search. Setting power=None means using no randomness.

The figure below shows what happens when we solve the large $9 \times 7$ board 100 times using different values of power.

Setting power=5 and power=10 both yield solution solution paths with 17 moves.

We implement three algorithms that yield good-enough solutions quickly. They are called anytime beam search, heuristic search and Monte Carlo search, and all are more clever than simply repeating randomized best-first search many times.

Anytime beam search

Beam search is a simple algorithm:

Start at the root, expand all children. Keep only the beam_width children that are promising, for instance those that have the largest values of cells_cleared / num_moves.
From the beam_width nodes under consideration, expand all children. Filter them again and keep only the beam_width most promising ones.
Repeat until a solution is found.

One disadvantage of beam search is that for large values of beam_width we have to wait a while until we get a solution. An anytime algorithm is one that yields good-enough solutions quickly, and if it runs for longer it improves on the solution. To make beam search into an anytime algorithm we run it for beam_width=1, 2, 4, 8, ..., keeping track of the best solution seen so far and yielding as better solutions are found.

Heuristic search

This algorithm yields better and better solutions as they are found. If it runs long enough, it will eventually find an optimal solution. In practice it will run out of memory before it’s guaranteed to explore every path, so there are no guarantees. It works roughly like this:

Start by running a single deterministic best-first search. This gives a quick upper bound best_solution_length_so_far on the optimal solution that we can use to prune solutions with later.
Sort the priority queue by a non-admissible heuristic function, for instance cells_cleared / num_moves. Always consider moves that maximize this function, hoping they lead to a good solution.
Let h(n) be the admissible heuristic function discussed earlier. If num_moves + h(n) >= best_solution_length_so_far, then we can prune node n.

Monte Carlo tree search

Monte Carlo tree search also yields better and better solutions as they are discovered. Like the algorithm above, it also eventually finds an optimal solution but provides no guarantees when run for a short time. It works roughly like this:

Start with a deterministic best-first search to establish an upper bound used for pruning.
Randomly sample nodes in the explored tree, going down guided by the so-called upper confidence bound, which trades of exploration and exploitation.
When a new unexplored node is reached, run a randomized best-first search from the node down to the bottom of the tree to score the node. If this results in a new best solution, update the upper bound. Propagate the score cells_cleared / num_moves up in the explored tree to the root. This score affects the upper confidence bound used in the next iteration.
Prune by using the admissible heuristic h(n) as the tree is explored.

Running all algorithms for around four minutes on the game instance of size $9 \times 7$, we get the following results.

Anytime beam search finds the best-known solution from NRK in two seconds.
Heuristic search finds a solution of length 17 in around two minutes, but is unable to discover any better solutions in the remaining time.
Monte Carlo search finds a solution of length 14 in around twenty seconds.

Here is the solution sequence for the 12 move solution (click to enlarge).

The NRK record was 12 moves, and we managed to solve the board in 12 moves. On this particular instance, running for longer does not lead to further improvements. On other instances we are not always so lucky; sometimes we are unable to find the best known solution, at least in a few minutes of compute time.

Summary, notes and references

The NRK game “Former” can be formulated as a graph search problem, but the graph is huge. There are billions and billions of nodes in the graph, so complete enumeration is out of the question. While A* search is effective similar puzzles (e.g. the 15 puzzle), here it fails because we were unable to design a great admissible heuristic function.

Heuristic search algorithms are still able to produce good-enough solutions in a short time. We implemented three such algorithms with different flavors: anytime beam search, heuristic search and Monte Carlo search. In the end we managed to find the record solution of 12 moves on this instance. These algorithms can be combined: heuristic search and Monte Carlo search perform better if an upper bound is established quickly, so running beam search as a first step could help.

The game is similar to SameGame, and the paper Single-Player Monte-Carlo Tree Search proposes Monte Carlo search for this game. An implementation by Odin at github.com/odinhg/nrk-former-sp-mcts implements this algorithm. Another paper is Solving SameGame and its Chessboard Variant, which also uses Monte Carlo search. The sgbust program is a SameGame solver written in C++ that uses beam search. A good introductory reference for game solving is Chapter 3 “Solving Problems by Searching” in Artificial Intelligence: A Modern Approach. Another avenue to go down is reinforcement learning, see for instance the book Reinforcement Learning: An Introduction.

Thanks to Andreas for introducing me to the game, and to Eivind and Gunvor who independently discovered the admissible heuristic.

Appendix: more boards

Above we considered the NRK game instance for the 16th of November. Below we show performance on more game instances from four other days.

The average value

Tommy Odland — Sun, 10 Nov 2024 00:00:00 +0100

Presentation slides and video of a talk I gave about notions of averages (one-parameter models), loss functions and simple machine learning problems.

Below is a transcript with slides, if you prefer reading over listening.

The average value
Table of contents
The Pythagorean averages
The $p$-norm
Huber, mellowmax and quantiles
ETA for file downloads
Ranking products by ratings
Summary

The average value

Hello, my name is Tommy and this is a recording of a talk I gave at an IT conference. The title is “The average value” and the intended audience is intermediate-level data scientists. In this recording, I’ll go through the presentation informally. I’ll assume you have some mathematical background and are interested in data science. If you want to look at these slides or check the references, you can download them from this URL.

The average value—this is the equation for it, and if that was all there was to say, the talk would end here. This slide introduces some notation, but obviously to make a presentation out of this, we need to generalize away from this equation.

We’ll do that by looking at the optimization problem that this equation—the arithmetic average—solves. That’s the least squares optimization problem. In general, in machine learning, when we do regression tasks, it’s very common that we have squared loss and some kind of function $f$ where we want to compute the optimal parameters for $f$. Here, $f$ could be a tree-based model, linear regression, or neural network, and somehow we find the best parameters $\theta$.

However, there is typically no closed-form solution. For the arithmetic average there is a closed-form equation, but for many more complicated models, you use something like gradient descent or Newton’s method. You end up with a result, but there’s no equation you can use to just compute the results—it’s an iterative procedure. In many problems, there’s no guarantee that you have a global minimum or that you always end up with the same one.

Basically, this talk isn’t really about the average value—it’s about setting the model $f$ to be a single parameter model. We set $f(x,\theta) = \theta$ and ask ourselves “what else can we do?”. If we fix the model to be the simplest possible model we can think of, what else can we do? We can tweak the loss function!

Here’s the table of contents for the presentation:

First, we’ll look at the Pythagorean averages—the arithmetic average, geometric average, and harmonic average—and see that there are loss functions that induce these averages in a very natural way
Then we’ll look at the $p$-norm, which you might know about—it’s a way to measure distance, and we can use that to get the notion of the median and the midpoint
We’ll examine Huber, Mellox, and quantiles, which are one-parameter models that go further away from the Pythagorean averages and $p$-norm
Finally, we’ll look at some real-world examples aimed at developers: (1) ETA for file downloads and (2) Ranking products by ratings on websites

The Pythagorean averages

Let’s start with the Pythagorean averages.

The arithmetic average is the most common average, and it minimizes squared loss. There’s an assumption that if you’re twice as far off, it’s four times worse because you’re squaring the residuals. It has really nice mathematical properties—basically, if you have squared loss, you have a quadratic, so when you differentiate it, you get something linear that you can solve immediately. Newton’s method can solve this in one iteration—you can one-shot these squared problems!

This figure is important to understand—it shows for each data point (1, 3, and 6) the loss, and then shows the sum of these three losses over the three data points. That’s the solid blue line, and this is the loss function. If you minimize it, you get the arithmetic average, shown in the dashed line.

One final comment in the footnote: how do we get this squared loss in the first place? One way is to look at the normal distribution, which is essentially an exponential to the power of squares. But you also end up with averages in Poisson models and binomial models, so there are many types of assumptions about data that lead to arithmetic averages.

The geometric average minimizes multiplicative loss. If you take the logarithm of the data and the logarithm of the model and compare those, you get the multiplicative loss. Here, being one order of magnitude below is equally bad as being one order of magnitude above. The geometric average is like the arithmetic average but with multiplication—it’s the number that you have to multiply with itself $n$ times to get the product of the data.

The harmonic average minimizes relative loss, where you measure errors relative to the true value. This is equivalent to doing weighted least squares where the weights are given by $1/y_i$. This particular loss—the relative loss—is not the only loss function that induces this notion of average. The loss in the footnotes will also lead to the same harmonic average, but the relative loss one is more natural and commonly used.

Let’s look at these three loss functions in a different problem setting. Suppose you have resources to distribute—maybe you have bakeries and predictions for tomorrow’s bread sales. You have a central bakery where you bake bread and want to distribute it to cover every store.

Unfortunately, you don’t have enough bread to cover every store, so you need to distribute according to some rule. You would like your distribution $\theta$ to be as close as possible to the demand of each store $y$, so you use a loss function. Depending on which loss function you use, you get three different distributions.

I’m not sure what the correct answer would be—it would probably depend on the bakeries’ goals. Would you use squared loss, multiplicative loss, or relative loss in the real world? I’ll leave that up to you to think about.

We’ve seen several loss functions, and it might be interesting to discuss how you would choose between them. Every loss function has the property that if your prediction is very close to the target, it’s good, and if it’s far off, it’s bad.

How do you choose between loss functions? One thing I like to do is consider the null space—by that, I mean considering which types of errors are equally bad and whether that makes sense in your application. For instance, if the true values $y$ are 2 and 8:

Squared errors would imply that going two up in the positive direction on either prediction is equally bad
Multiplicative errors mean that predicting 4 (8/2) or 16 (8*2) is equally bad
Relative errors mean that subtracting 2 from the first prediction or adding 8 to the second one is equally bad, because those represent equal units of true values

Here are some practical examples:

If you’re trying to predict medicine dosages, you probably want to use relative or multiplicative errors, since different drugs work on different dosage scales
If you’re predicting economic outcomes or investments, you probably don’t want to use multiplicative or relative errors because money isn’t relative—one unit of money is the same whether it comes from a small project or a huge project
For financial predictions, you might want to use absolute errors rather than squared errors, because being twice as far off means losing twice as much, not four times as much (unless you want to consider the utility of money, but let’s skip that for now)
When people look at interest rates or summarize computer performance benchmarks (CPUs, etc.), they almost always use the geometric average

The $p$-norm

Having finished the first part about Pythagorean averages, let’s talk about the $p$-norm.

We’re going to generalize this equation in a different direction by solving this optimization problem for different values of $p$ instead of $p=2$ (which gives us the arithmetic average).

If we set $p$ equal to one, we get the absolute error. This is an absolute value function placed on each data point, and when you sum it up, it’s piecewise linear. If there’s an odd number of data points, there’s a unique maximum; if there’s an even number of data points, it’s flat between the two middle points—there’s no unique minimum.

The median has an interesting property: if I were to change the data point 6 and move it up to 8 or down to 4, as long as it doesn’t cross 3, the median stays the same. The median as a function completely ignores the location of every data point except for the middle one. Obviously, if 6 crosses 3, then something happens with the function, but it’s essentially blind to many of the data points.

Choosing $p=1.5$ is one of many ways to create a function between a median and an average. It’s also the first optimization problem we’ve seen where there is no closed-form solution—here you would use an iterative optimization routine to find the minimum.

Choosing $p=2$ is the squared error, which we’ve seen before.

As you increase $p$ further, the functions become more and more sharp.

You get to an interesting property in the limit as $p$ becomes very large: minimizing the $p$-norm becomes equivalent to minimizing the maximum. If you want to minimize the maximum distance, you want to make the worst case as good as possible, and you do that by placing yourself at the midpoint. This is one of the measures of centrality you might have learned in high school—the midpoint.

How can we combine these Pythagorean losses and $p$-norms? Let’s put them in a table:

Mean absolute error
Mean squared error
Maximum error (not used much in machine learning but used in engineering where you want to optimize the worst case, like in airplane design)
Mean squared logarithmic error
Mean absolute percentage error
Weighted mean squared error

Some combinations aren’t listed but could be imagined. For instance, with $p=1$ and multiplicative error, you’d get something like the mean absolute logarithmic error. The point is that you can construct and combine these functions.

How would you do that? You can first look at an individual data point and consider what kind of loss function you’re interested in, then use a $p$-norm to combine them. The $p$-norm is a way to combine the individual losses.

Think about what properties you want in your application:

With absolute error, residuals of $(4,0)$ are as bad as $(3,1)$ and $(2,2)$. With money, this makes sense—if you’re off by four, you probably don’t care how it’s distributed if it’s money lost
It also means that $(9,0)$ is better than $(5,5)$, even though errors in $(5,5)$ are more balanced
Maximum error just looks at the largest value, so $(9,0)$ would be as bad as $(9,5)$ and $(9,9)$, while $(8,8)$ would be better than $(9,0)$

Huber, mellowmax and quantiles

Now let’s look at even more esoteric notions of averages, maxima, minima, and everything in between.

One issue with squared errors is that they can be overly sensitive to outliers. One remedy for this is to use the smooth Huber loss, which acts like a square when you’re close to zero but like the absolute value when you zoom out. This makes it less sensitive to outliers while still maintaining a unique minimum even with an even number of data points.

There’s an opposite problem that might occur in some applications: squared errors are very flat around zero, so if you have a small residual and you square it, it becomes even smaller. If you’re regularizing coefficients (like the LASSO does) and want to drive them all the way to zero, it’s very nice to have something that acts like the absolute value close to zero. The reverse Huber has this effect—it’s like the absolute value when you’re close to zero and acts like a square if you’re far away.

Mellowmax and mellowmin are smooth generalizations of maximum and minimum. They are functions that:

As $\alpha$ goes to infinity, the function approaches the maximum
As $\alpha$ goes to negative infinity, the function approaches the minimum
Around zero, the function approaches the arithmetic average

If you want something that acts in between the arithmetic average and the maximum, mellowmax is a nice way to do it. The maximum only depends on the maximum value in the dataset, but sometimes when you want to minimize the maximum, you want to bring all values down. One way to do that is to drive down the maximum value, but as a function, the maximum is blind to every other value apart from the largest one. If you use mellowmax instead, it’s actually a bit sensitive to other values, which can better inform the optimization routine.

Sometimes it’s very nice to relax these functions that are piecewise linear or have derivatives that are zero for most inputs. The median only depends on one or two values for most inputs, and if you want something more smooth, this is a nice option.

Let’s quickly cover quantiles—I assume you know about them. They’re a way to be somewhere between the median and the minimum/maximum. They put an absolute value function on every data point but scale it up and down on the left and right.

A small generalization: there’s no reason we have to use the absolute value—we could use squared residuals too. If $p=2$, you would call it expectile, but in general, you don’t have to use $p=1$ or $p=2$. You could in principle use any value for $p$.

ETA for file downloads

Let’s look at our first practical example: ETA for file downloads.

Suppose you’re a developer writing a routine that will download files, copy files, or install software. The computer is doing chunks of work, and you have the time that each chunk takes. You know the amount of work done and how much work will be done in total (like the total megabytes for all files to download). You want to estimate how long this will take.

To solve this, we need to handle:

Non-equidistant arrivals (times or pieces of work)
We’ll assume a linear model is reasonable (parameters can change—if someone else starts downloading, your speed might drop, but after it drops we assume it’s still linear with a different rate)
Very importantly, when a new data point arrives, we must be able to update our model in constant time—we can’t iterate over all previous data points, as there could be millions

Here’s one approach: we can compute the average rate of work. With two pieces of work (two files downloaded), we can compute the slope for each piece (work done over time elapsed). By minimizing this function, we get $\theta$, which is the average amount of work done per time.

I weight this by $\Delta t$ (time) because when I solve that optimization problem, it leads to an equation with nice symmetry in $\theta$. It should make intuitive sense that downloading a file in 1 second contains some information, but downloading the next file in 30 seconds contains much more information—so large $\Delta t$ should be weighted more heavily.

We use $w_i$ to weigh the most recent data more heavily. We can split it up like this: the most recent data point is attached at the end, so we have the previous estimate and some new data. We can quickly update by keeping track of our previous computation and adding to it.

We then take a linear model constrained to go through the last data point seen. We have one degree of freedom left, which we use on the slope with this estimate. We set the weights proportional to some number to the power of $i$ over $\gamma$, where $\gamma$ is the half-life. This parameterized the weights so the most recent data is weighted more heavily.

Here’s how it performs:

If the speed changes when we see more data, obviously it’s not a very good model
If it’s linear all the way through, it’s quite accurate
With some deviation from linearity, it picks up an average rate somewhere in between
With noise, it goes through the last data point and picks up an average rate that’s quite close

As more data arrives, it will actually improve because it’ll pick up on new regimes with different rates of change. This is a simple algorithm—very fast and accurate in many real-world cases. It depends on the application, but it’s probably more advanced than many file ETA estimates I’ve seen, which tend to be jittery and jump around too much.

Ranking products by ratings

Our final example is ranking products by ratings.

This is a fascinating real-world example of how you can use averages effectively. I looked at Norwegian e-commerce websites where users can give ratings. All these websites use a naive average rating, which produces some strange results:

The average of a single 5 is 5, but the average of many 5s and a few 4s would be slightly less than 5
This means a product with one 5-star rating would rank higher than a product with hundreds of 5-star ratings and a few 4-star ratings
I would trust the product with more ratings to be better, as one rating doesn’t give much certainty about true quality
Many websites also place products with no ratings at the very bottom, which I think is wrong—a new product with no ratings is likely average, not the worst product

I didn’t find any Norwegian e-commerce websites that do this correctly (I didn’t look at international ones). One of the few sites that gets this right is the IMDb Top 250 list—they account for the number of ratings and use an equation similar to what we’re going to deduce.

We want our average rating to have two properties:

It should take into account the number of ratings
A product without any ratings is likely typical, not really bad

Here’s a crucial idea: what are we really doing when we’re trying to rank products? We’re going to frame this as a prediction problem—we’re predicting the next rating. If you were to buy this product and rate it, what would your rating be? That would be the next rating for this product.

So we’re casting it as a prediction problem and saying that ranking products is equivalent to predicting the next rating and sorting by that prediction. We’ll do this by:

Introducing a prior average $\bar{y}$ and a pseudo count $\alpha$
Solving this optimization problem
The solution shows the meaning of the pseudo count—it’s added to $n$ as dummy ratings, and the value of these dummy ratings is $\bar{y}$ (the prior average rating)

To determine the hyperparameters $\bar{y}$ and $\alpha$, we need to do cross-validation. I assume you know about leave-one-out and k-fold cross-validation. We’re going to do “leave-next-out” cross-validation.

It’s very important that your cross-validation procedure be as similar as possible to how you want to use the model. Since we defined our notion of ranking as predicting the next value, we’ll simulate that exactly:

Start with no ratings, try to predict the first one
Get to know the first rating, look at the squared residual
Use that first data point to predict the second one
Get to know the second data point and compare
Continue this process

This is essentially a time series split where you learn one more data point at a time. Both of these can be computed in linear time for these simple models, so we can do this very quickly.

Let’s apply it to a real dataset. There’s a Norwegian website called Legelisten that ranks doctors.

I found 17 doctors with 212 total ratings. I used our average with some weights, parameterizing those weights by the half-life $\bar{y}$. Using leave-next-out cross-validation, I found:

Optimal number of pseudo-ratings is around $\alpha=2$
Optimal half-life is around $\gamma =7$ years
Prior rating is $\bar{y}=3.8$ (on a scale from 1 to 5)

The loss when using the arithmetic average as our predictor was 1.77. With this regularized temporal average, it’s 1.55—a very nice reduction in loss that leads to better predictions of the next rating.

You might ask why not make this into a big machine learning project using all kinds of covariates, but I don’t think people would like that. If you rank doctors using information about their gender and age, it’s probably not good. This solution is something people will understand and won’t feel discriminates unfairly based on age, gender, or other factors. It’s still super simple, just more nuanced and created from thinking through the problem more carefully.

Looking at the results (blue is the old website rating, green is my proposed rating):

The third doctor: One person rated this doctor with a 5. The old ranking puts them first; we say no—one rating isn’t much, so we pull them down. The doctor with many 5s deserves to be ranked higher.
The tenth doctor is interesting because they’re pulled away from the grand average. Typically without temporal weighting, you pull towards the grand average, but this doctor has gotten more positive reviews lately—lots of 5s and 4s, 44 ratings in total. This doctor has improved over the years, so the half-life is forgiving the older low ratings and pulling up towards the more recent 5s.
The seventeenth doctor hasn’t been rated once. The old ranking would place them last; we say no—with no ratings, we believe you’re in the middle of the pack. We place them there, certainly better than someone with just many 1-star ratings.

Summary

Let’s summarize the key points.

Only simple problems have closed-form solutions. Don’t worry so much about how to solve things when you start modeling—look at what properties you want to model instead. Solving is left to an optimization algorithm.
There’s a good reason why people like squared errors—they have nice mathematical properties. Not everything I showed today can be solved efficiently for bigger models.
In machine learning, the solution depends on both the model and the loss function. Many data scientists focus heavily on the model, discussing which is better in detail. While the model is important, choosing the wrong loss function can be just as problematic. For instance, if errors are truly relative in your application but you choose RMSE, or you have an economic application but use the wrong loss function, you’re probably modeling the whole thing incorrectly.
Adapt your approach to the problem. Combine existing ideas but don’t reinvent—there are many good books and sources. People have thought about amazing things, so read up on that. If you find yourself inventing something you believe is totally new to solve a practical problem, you’re probably reinventing something—do a literature search.
Always start simple. Never use models you don’t understand—you’ll have a hard time explaining them to others, and they might behave in ways you don’t fully understand. There’s still plenty of room to have fun with simple models. Even today, some e-commerce sites are billion-dollar businesses and don’t implement averages correctly.

The final comment I want to make is that if you go back to the first slide and remove the restriction of the one-parameter model, everything we talked about still holds. All the loss functions and modeling approaches are still relevant, but by choosing one-parameter models, I was able to create nice plots and talk about things simply. Many of these concepts are still relevant for other models, even though computationally it’s not always easy to optimize.

I really hope you enjoyed my presentation. Thanks a lot for watching, and again, you can download the slides and look at the references if you want to. Thank you.

Smooth taxes without brackets

Tommy Odland — Wed, 16 Oct 2024 00:00:00 +0200

I recently implemented a calculator for Norwegian income tax, and it was harder than I anticipated. The implementation required around 30 lines of code and 20 constants.

In this article we use the term “income tax” to refer to everything the government deducts from gross income. To compute income tax, one must take into account the brackets, the national insurance contributions, the minimum standard deduction, the personal allowance, and so forth.

In this article we’ll simplify the income tax calculation to one line of code and two constants.

We’ll start by looking empirically at the Norwegian income tax, then we’ll study properties that a tax scheme should have. In the end we’ll construct a smooth tax function with two adjustable parameters $H$ and $k$:

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k}. \end{align*}

We’ll show that this function has good properties and that for suitable values of $H$ and $k$ it approximates the current income tax very well. Below is a figure showing the average tax rate and the marginal tax rate induced by the function $T(x)$ above. You’ll understand this figure better as you read the next sections.

Introduction

The tax function $T(x)$

Let $T: \mathbb{R}_+ \to \mathbb{R}_+$ be the income tax as a function of gross income $x$. For instance, if Alice earns a gross income of $x=100$ units of money, then she might have to pay $T(100) = 20$ in taxes, leaving her with a net income of $x - T(x) = 100 - 20 = 80$.

If we have such a function $T(x)$, then we can deduce three other interesting functions that answer specific questions:

The net income $x - T(x)$: “How much money do I have left after paying taxes?”
The average tax rate $T(x) / x$: “What percentage of my total income do I pay in taxes?”
The marginal tax rate $T'(x) \approx T(x+1) - T(x)$: “If I earn one more unit of money, what fraction of that unit of money goes to taxes?”

Proportional and progressive tax

A tax scheme is called proportional if the tax rate is a fixed percentage. In our notation, this would mean that $T(x) = cx$ for some constant $c$.

A tax scheme is called progressive if those who earn more pay a higher percentage in taxes. In addition to progressive tax schemes, we can define non-regressive and strictly progressive tax schemes.

Non-regressive: the average tax rate $T(x)/x$ never decreases.
Progessive: the average tax rate $T(x)/x$ never decreases, and it increases at least once in the domain.
Strictly progressive: the average tax rate $T(x)/x$ always increases.

Mathematically, these requirements correspond to conditions on the derivatives of the average tax rate. For instance, a non-regressive tax function $T(x)$ has the constraint that

\begin{align} \label{progressive} \tag{1} \left( \frac{T(x)}{x} \right)' \geq 0 \Rightarrow T'(x) - \frac{T(x)}{x} \geq 0. \end{align}

The second inequality comes from applying the product rule of differentiation. Having established some notation and concepts that we’ll use later, we will now empirically examine the Norwegian income tax.

The Norwegian income tax

The Norwegian income tax has several deductions, a few if-then-else rules and a (mostly) increasing marginal tax rate. Once all the rules are implemented and combined, the result can be visualized as a function. The implementation was tested against the calculator Skatteberegning 2024, and in the background the income distribution from 2022 is shown.

The jumps on the marginal tax rate $T'(x)$ make the figure a little disorderly, but notice that:

It’s progressive, since the average tax rate $T(x)/x$ never decreases and it does increase.
It’s not strictly progressive because $T(x)/x$ has two flat regions.
The limit evaluates to $\lim_{x \to \infty} T(x) / x = 0.474$.
The marginal tax rate is discontinuous, and actually jumps down once.

The government often changes the effective tax rates: they adjust deductions, the lower and upper limits on the brackets and rates within each bracket.

To get an idea of how erratic these changes are, consider the highest bracket that extends to infinity. In 2020 the lower limit of the highest bracket was 999 550, then in 2021 it was raised to 1 021 550. It doubled to 2 000 000 in 2022, was reduced by half a million to 1 500 000 in 2023 and cut further to 1 350 001 in 2024.

Properties that smooth taxes should have

To implement the Norwegian income tax I had to read up on tax rules from several sources online. I also had to test the implementation against an online calculator multiple times before I got it right. Wouldn’t it be nice if $T(x)$ was just a simple formula? No jumps and kinks—just a smooth, simple function.

For instance, given a constant $k >0$, what if $T(x) = x(1 - e^{-xk})$? That way the average tax rate becomes $T(x)/x = 1 - e^{-xk}$, which seems reasonable… right?

It turns out that this is a seriously bad idea, as we can see in this figure:

The net income $x - T(x) = x - x(1 - e^{-xk})$ is maximized at $x^{\star} = 1/k$. There is no incentive to earn more money than this, since you lose money by earning more. This is a terrible property for a tax system to have—back to the drawing board!

Property 1: Good incentives

Let’s set a simple rule: earning more should always mean keeping more after taxes. In other words, if we increase our gross income from $x$ to $x + \epsilon$, then we should end up with more net income after paying taxes:

\begin{align*} \underbrace{(x + \epsilon) - T(x + \epsilon)}_{\text{net income with $\epsilon$ more}} > \underbrace{x - T(x)}_{\text{net income}} \end{align*}

Rearranging this expression we find that the condition is equivalent to

\begin{align*} \frac{T(x + \epsilon) - T(x)}{\epsilon} < 1, \end{align*}

and in the limit $\epsilon \to 0$ the requirement says that the marginal tax rate $T'(x)$ must obey $T'(x) < 1$. If this inequality is satisfied, then higher pre-tax income always leads to higher post-tax income. In other words, there’s always an incentive to increase one’s gross income $x$.

Property 2: Progressive taxation

Another natural constraint to impose is that $T'(x)$ should be non-decreasing, since then the tax scheme must be non-regressive. To see why this is true, note that if $T'(x)$ is non-decreasing and non-negative, then $T(x) = \int_0^{x} T'(\tau) \, d \tau \leq x T'(x)$. Rearranging this inequality gives $T'(x) \geq T(x)/ x$, which is an inequality that we have seen before in Equation \eqref{progressive} in the introduction. It is exactly what is needed for a tax scheme to be non-regressive.

Note that the converse is not true: a tax scheme can be non-regressive even if $T'(x)$ decreases. For instance, the Norwegian marginal tax does decrease at one point, but the tax is still non-regressive overall. If $T'(x)$ is not only non-decreasing, but also increases at some point, then the resulting tax scheme is guaranteed to be progressive.

Due to these constraints on the marginal tax rate $T'(x)$, it’s easier to find good functions $T(x)$ if we start with $T'(x)$ and integrate to obtain $T(x) = \int_0^{x} T'(\tau) \, d \tau$.

Constructing tax functions

Recall that the marginal tax rate $T'(x)$ says how much we’ll be taxed on the next unit of money earned. For instance, $T'(x=100 \, 000) = 0.25$ in the Norwegian income tax, so if we earn $100\, 000$ NOK the next unit of money will be taxed by $0.25$.

Proposing a function. Let’s again consider the function $1 - e^{-xk}$, but this time as the marginal tax rate $T'(x)$ instead of as an average tax rate:

\begin{align*} T'(x) = 1 - e^{-xk}. \end{align*}

This function satisfies both our properties: it’s always less than one, and it never decreases. If we integrate it we obtain

\begin{align*} T(x) = \int_0^{x} T'(\tau) \, d \tau = \int_0^{x} (1 - e^{-\tau k}) \, d \tau = \frac{e^{-kx} - 1}{k} + x. \end{align*}

Notice that our proposed marginal tax rate $1 - e^{-xk}$ is zero when $x=0$ and approaches one as $x\to \infty$. The interpretation is that there is no tax initially as one starts to earn income, but if one approaches infinite income then units of money earned toward infinity will be taxed more and more.

Generalizing ever so slightly, we can change the marginal tax rate to $T'(x) = H - (H-L)e^{-k x}$. This way the marginal tax starts at a low value $L$ and converges to a high value $H$. Integrating, we obtain the tax function

\begin{align*} T(x) = \int_0^{x} T'(\tau) \, d \tau = \int_0^{x} H - (H-L)e^{-k \tau} \, d \tau = \frac{(H-L) e^{-kx} + H(kx - 1) + L}{k}. \end{align*}

The current policy today in Norway is that the first units of money earned are not taxed. Therefore, let us set $L=0$ and optimize for $H$ and $k$ by curve fitting to the Norwegian tax rate:

The smooth model based on $T(x)$ is shown in dashed lines. The error is never more than 5300 NOK, so this approximation is pretty good. It’s easy to implement if you want to play around with it in Excel or Python:

import numpy as np


def T(x):
    """Norwegian income tax for 2024. 
    Max error < 5300 NOK, mean absolute error ~1400."""
    H, k = 0.4772, 2.8745e-06
    return H * (np.exp(-k * x) + k * x - 1) / k

Summary

Income tax computation is quite complex. To implement it one needs around 30 lines of code with deductions, rules and tax brackets, all defined using around 20 constants.

In this post we examined smooth tax functions $T(x)$. One such function is

\begin{align*} T(x; H, k) = \frac{H(e^{-kx} + kx - 1)}{k}, \end{align*}

which mimics the overall behavoir of the current income tax calculation with good accuracy.

Maybe it could be an alternative to today’s system? After all, a single formula is easy to implement compared to the current rule set with various deductions, constants and tax brackets. Everyone has access to the exponential function on their phones or computers. A simple function makes it harder for politicians to obfuscate the tax system by introducing new rules and conditions. They could still modify the tax rate by modifying constants such as $H$ and $k$.

One argument against a smooth function is that the average person does not understand the mathematics of it. This might be true, but then again: the average person probably does not understand what a bracketed marginal tax system is either.

Keep it simple!

If I was a benevolent dictator, I would:

Simplify tax formulas and other computations performed by the government as much as possible
Favor simple, smooth functions (2-3 parameters) over discretizations and bracketing
Require government agencies that use formulas to publish them in math and code, with tests
Ensure that properties such as good incentives are baked into the models
Run an exhaustive test suite to guarantee good incentives across all combinations of gross income and all manner of government payouts, ensuring that citizens are never punished for earning more and working more (this does happen, and it might be a sign of overly complex systems that have not been thought through carefully)

Notes and references

Continuous tax brackets were discussed on the Personal Finance & Money Stack Exchange back in 2020.
The paper Smooth Income Tax Schedules: Derivation and Consequences by Schwarz et al discusses how various countries approach taxation and propose using smooth functions.
A Quora post with the fantastic title The US Federal Tax Code is More Complicated Than Calculus. So Let’s Use Calculus also looks at smooth functions. The author proposes shifting and scaling a function of the form $T'(x) = 1 - 1/(x + 1)$, whereas we shifted and scaled $T'(x) = 1 - e^{-x}$.
Any sensible cumulative probability density function can be used to construct $T'(x)$, offering a flexible approach to designing smooth tax functions. I chose the exponential function because it’s relatively simple to work with.

Arithmetic coding in Python

Tommy Odland — Thu, 19 Sep 2024 00:00:00 +0200

Arithmetic coding is a lossless data compression algorithm. It encodes a sequence of symbols into a sequence of bits. After reading the 1987 paper “Arithmetic coding for data compression” I decided to implement the algorithm in clean, modern Python.

In this article I’ll demonstrate how to use my Python implementation, how well arithmetic coding works when the symbol frequency is skewed, and how to encode “Crime and Punishment.” We also compare arithmetic coding with the more well-known Huffman coding. The implementation is available at: github.com/tommyod/arithmetic-coding

A simple example

Here we create a message to encode and put an End Of Message (EOM) symbol "<EOM>" at the end of it. We also need to set up a table of symbol frequencies. Arithmetic coding exploits the frequent symbols and assigns fewer bits to them, though unlike Huffman encoding arithmetic encoding does not actually use a fixed bit sequence per symbol.

>>> import collections
>>> from arithmetic_coding import ArithmeticEncoder
>>> message = ["A", "B", "A", "A", "A", "<EOM>"]
>>> frequencies = collections.Counter(message)
>>> encoder = ArithmeticEncoder(frequencies=frequencies, bits=12)
>>> bits = list(encoder.encode(message))
>>> bits
[1, 1, 0, 0, 0, 0, 1, 1]

We can decode these bits back to symbols and we get the original message back.

>>> list(encoder.decode(bits))
['A', 'B', 'A', 'A', 'A', '<EOM>']

This examples encodes the message to 8 bits. A naive approach would be to assign a unique 2-bit sequence to each symbol, e.g., {"A": 0b00, "B": 0b01, "<EOM>": 0b10}. Since the message contains 6 symbols, we would need 6 * 2 = 12 bits with this naive approach.

Infrequent symbols

To see how the encoder exploits the frequency of symbols, consider this example. We initialize ArithmeticEncoder(bits=32) because we need more internal bits in the buffer of the encoder to represent low-frequency symbols.

>>> message = ["A"] + ["B"] * 1_000_000 + ["<EOM>"]
>>> frequencies = collections.Counter(message)
>>> encoder = ArithmeticEncoder(frequencies=frequencies, bits=32)
>>> bits = list(encoder.encode(message))
>>> len(bits)
43

Amazingly, a message with a million symbols is encoded to only 43 bits! Having seen how efficient arithmetic coding is with skewed frequencies, let’s apply it to a real-world example.

Encoding “Crime and Punishment”

Let’s download the English translation of “Crime and Punishment” by Fyodor Dostoevsky.

>>> import requests
>>> import string
>>> url = "https://www.gutenberg.org/cache/epub/2554/pg2554.txt"
>>> response = requests.get(url)
>>> symbols = (s for s in response.text if s in set(string.printable))
>>> message = list(symbols) + ["<EOM>"]
>>> len(message)
1164281

For this example, we’ll use the actual frequencies from the text, though in practice one might use a more general frequency table for the English language.

>>> frequencies = collections.Counter(message)
>>> len(frequencies)
83
>>> frequencies["e"], frequencies["q"]
(104735, 766)

Naive encoding

The Python code above shows that there are 83 distinct printable symbols in the book. Since $2^7 = 128$, a naive approach would be to use seven bits for each symbol, regardless of its frequency. We would then encode "A" to 0b0000000, "B" to 0b0000001, "C" to 0b0000010 and so forth. This approach would encode the book using 1164281 * 7 = 8149967 bits.

Arithmetic encoding

Instead of using the naive approach, the arithmetic encoder exploits that symbols like "e" are much more frequent than symbols like "q". In around three seconds we’re able to encode all the 1.05 million symbols in the book:

>>> encoder = ArithmeticEncoder(frequencies=frequencies, bits=32)
>>> bits = list(encoder.encode(message))
>>> len(bits)
5259420

The result has around 5.26 million bits, and the bits/symbol ratio is 4.52.

Shannon entropy as a lower bound on bits

The Shannon entropy, which is a lower bound on how many bits we need on average to encode a message of this length with these frequencies, is

\begin{equation*} -\sum_i f_i \log_2 \left( p_i \right), \end{equation*}

where $f_i$ is the frequency (count) of symbol $i$ and $p_i = f_i / \sum_j f_j$ is the probability of symbol $i$. The bound only applies if we assume that each symbol is independent of the others—if we assume higher-order statistics, higher compression ratios might be possible.

Computing this value for “Crime and Punishment”, we find that the best we can hope for is

>>> import math
>>> total = sum(frequencies.values())
>>> probs = {s: c / total for (s, c) in frequencies.items()}
>>> shannon_bound = -sum(frequencies[s] * math.log2(probs[s]) 
...                     for s in frequencies.keys())
>>> shannon_bound
5259418.898940693

Huffman encoding

We also use the dahuffman Python library to produce a Huffman encoding, which maps each symbol to a unique code value of variable length. Notice how common symbols like "e" are mapped to short code values, while rare symbols like "B" correspond to long code values.

>>> from dahuffman import HuffmanCodec
>>> codec = HuffmanCodec.from_frequencies(frequencies)
>>> codec.print_code_table()
Bits Code                    Value Symbol
   3 000                         0 'e'
   4 0010                        2 'h'
   4 0011                        3 'i'
   5 01000                       8 'u'
   9 010010000                 144 'R'
  10 0100100010                290 'Y'
  10 0100100011                291 'B'
  ...
>>> symbol_to_bitsize = {code:bitsize for (code, (bitsize, _))
...                      in codec.get_code_table().items()}
>>> sum(symbol_to_bitsize[symbol] for symbol in message)
5298110

Huffman encodes the book to around 5.3 million bits.

Arithmetic coder with an adaptive model

If we do not want to assume known frequencies, we can use a simple adaptive model. The frequency count for each symbol is initialized to one. As the encoder processes symbols it increments the counts to update the probability model. The decoder mimics this process exactly, but in reverse. We still need to know all distinct symbols that the encoder and decoder expects to see beforehand though (unless we iterate over the data twice).

>>> symbols = list(set(message))
>>> encoder = ArithmeticEncoder(symbols, bits=32)
>>> bits = list(encoder.encode(message))
>>> len(bits)
5260166
>>> decoded = list(encoder.decode(bits))
>>> decoded == message
True

A comparison table

Comparing all the methods we’ve mentioned, we get the following table.

Compression Method	Bits Used
Naive encoding (7 bits/symbol)	8,149,967
Huffman encoding	5,298,110
Arithmetic encoding (adaptive)	5,260,166
Arithmetic encoding (fixed)	5,259,420
Shannon entropy bound	5,259,418.9

Naively encoding each symbol using a fixed bit length is wasteful, since it does not account for the fact that some symbols are much more frequent than others. Huffman encoding improves greatly upon this, but arithmetic encoding is even better and is able to get within two bits of the Shannon bound.

Summary

Arithmetic coding maps a sequence of symbols into a sequence of bits. Unlike Huffman coding, there is no one-to-one correspondence between a single symbol and a single bit sequence. Both methods exploit the fact that some symbols are typically more common than others.

With arithmetic coding there are two types of probability models: fixed and adaptive. A fixed model uses a static frequency count, computed by either looping the data in advance or by observing a larger corpus of text. A dynamic model starts out with equal probabilities for all symbols, and adapts to the specific message as it iterates through the symbols. Regardless of which model is used, the logic of the decoder must exactly match the logic of the encoder.

tommyodland.com

Decline Curve Analysis

1. Introduction

1.1 Simplicity in forecasting

1.2 A critique of the literature

2. Basic mathematical model

2.1 Data preprocessing

2.2 Naive least-squares fit

3. Curve models

3.1 The constant model

3.2 The exponential model

3.3 The Arps model

4. Curve fitting

4.1 Weighting recent data

4.2 Fitting on log scale

4.3 The median curve

4.4 Regularization

5. A statistical model

5.1 Independent errors

5.2 Autocorrelated errors

5.3 Statistical model with autocorrelation

5.4 Parameter inference

5.5 Forecasting

6. Summary

Appendix: Stan Code

Minimal models and dumb deployments

Avoid complexity

Aim for simplicity

Linear regression

Decision trees

Two approaches to improvement

Notes on Ensemble Smoothers

Introduction

The Gaussian posterior

Exact posterior distribution \(p(\boldsymbol{x} \mid \boldsymbol{d})\)

The Ensemble Smoother

Computation

The Ensemble Information Filter

Computation

Hvordan fungerer opptak til høyere utdanning?

Sentralisert vs. desentralisert opptak

Desentralisert opptak i USA

Forutsetninger for bruk av algoritmer

Algoritmen: Utsatt Aksept

Eksempel: Utsatt Aksept

Pareto-effektivitet

Ineffektivitet grunnet stabilitetskrav

Ineffektivitet grunnet kjønnspoeng

Oppsummering og referanser

Appendiks A: Digitalisering, algoritmer og offentlig sektor

Appendiks B: Boston-mekanismen

KI-billettkontroll

Ten lessons from twenty years of strength training

Start easy

Stay consistent

Push yourself

Two steps forward, one step back

Switch it up

Movements, not muscles

Compete or cooperate

Food makes you look good

Mute the marketers

Keep learning

Optimal bus stop spacing

Simulation

Analytical solution

Deltidsstillinger

Dagens praksis med merarbeid og overtid

EU-domstolens konklusjon

Problemstillingen

Et kompromiss

Continuous interest with continuous deposits

A simple savings calculator

The discretization error

Continuous deposits

Solution by integration

Solution by differential equations

Summary and references

Skatter og bidrag

Inntekt