tommyodland.com

Ten lessons from twenty years of strength training

Tommy Odland — Mon, 22 Jun 2026 00:00:00 +0200

I started strength training in 2006. Over twenty years I’ve learned that the principles of strength are simple. Almost everything you need to know can be summarized in two pages. Following the ten lessons below will make you stronger than 90-99% of the population. Here are my two pages:

Start easy

Start with light weights if you’re a beginner. During the first months, establish a routine of going to the gym and learn to lift correctly. Don’t rush into heavy weights—you’ll have plenty of time for that later.

The same advice applies if you’re getting back to lifting after a break. When gyms re-opened after COVID-19, I lifted the bar on my first workout. Then I increased to 40 kg, then 60 kg, 70 kg, etc. Start too heavy and you might get injured, or experience debilitating soreness. Either one hinders the steady progress required to build strength.

Squatting 285 kg at 108 kg bodyweight in 2010.

Stay consistent

Priority number one is to keep training. Half the game is showing up 2-4 times a week. If you get injured, come up with a plan to train around it. Set your ego aside. Don’t risk training through pain.

You must also avoid burnout. Hitting the gym six days a week and eating like a bodybuilder can be motivating. But if you burn out and quit as a result, you’ve lost the game. Slow and steady is usually better.

Push yourself

Since you are now consistent, the next step is to set goals, make plans and push yourself. Muscles grow by adapting to stress via progressive overload. The best way to induce stress is to increase the weight. If you lift 100 kg today and your goal is to reach 140 kg in one year, then you must increase the weight by around 0.75 kg per week. This is the outline of a long-term plan.

There are several ways to drive progress. Suppose you can comfortably lift 4 sets of 5 repetitions at 80 kg today. Try 4x5 at 82.5 kg next week, then 4x5 at 85 kg, 4x5 at 87.5 kg, etc. until you plateau. Don’t rush it—the idea is to start easy and gradually push yourself. You can alternatively increase repetitions instead of weight: do 4x1 at 90 kg, then 4x2 at 90 kg, etc. until you can do 4x6 at 90 kg—then increment the weight to 100 kg and start over again with 4x1. A third option is to combine weights and repetitions, e.g. 4x5 at 80 kg, 5x5 at 80 kg, 4x5 at 82.5 kg, 5x5 at 82.5 kg, etc.

Deadlifting 262.5 kg at 104 kg bodyweight in 2022.

Two steps forward, one step back

When your strength plateaus—which it invariably will—drop the weight (or repetitions) and start over. Suppose you lift 4x5 at 80 kg today, and 82.5 kg the next week, then 85 kg, etc. and eventually you fail at 95 kg. What should you do next? Try to drop down to 85 kg and start increasing by 2.5 kg per week again. You’ll likely reach 100 kg this time.

I take a big step back and start light in every new training program (every 8-12 weeks). Some programs add a small step back every four weeks. The “two steps forward, one step back’’ idea can be used at different resolutions in a long-term plan, and can be applied to weight, volume (total number of repetitions) and intensity (% of one-rep max).

Switch it up

Introduce slight variations to stay motivated and get unstuck when you stagnate. If you train 5x5, try increasing the repetitions (e.g., 4x8) or decreasing them (e.g., 5x3). Switch out barbell exercises for their dumbbell counterparts. Experiment with unilateral alternatives, for instance heavy lunges instead of squats. Train with bands or chains for a while. But don’t experiment too much or too often—stick with something long enough for it to work.

Movements, not muscles

Want big arms? Gyms are littered with men who feverishly curl with little to show for it. But I have yet to see anyone with small arms bench three plates.

The body responds well to heavy compound exercises: bench press, military press, rows, chin-ups, squats, deadlifts, etc. These exercises train many muscles simultaneously and let you move serious weight. I bench four plates, but never train arms directly.

Benching 182.5 kg at 104 kg bodyweight in 2022.

Compete or cooperate

Compete with yourself. Strength is easy to measure, so take advantage of it. Test your strength, weigh yourself and take some pictures every three months. Document your journey and make note of what works for you.

Strength training can be solitary, but it does not have to be. Train with others, build a community and share information. Compete with your friends or sign up for a meet. Read books and articles, participate in online forums, or publish a training log.

Food makes you look good

If your goal is to look good, then diet matters more than training. I enjoy tweaking my training programs, but it honestly has no effect on how I look—only my diet does.

To master your diet, start by writing down everything you eat for a week. Buy a food scale and weigh your food. Weigh yourself daily, but use a three-day average to assess progress.

Once you’ve mapped out how you eat today, start adding or subtracting food. It’s easier to control your diet if you eat the same meals every day, so lock them down if you can. I fix all my meals except one per day, where I only impose a weight limit. Don’t lose more than 0.5-1% of your bodyweight per week. Don’t gain more than 0.25-0.5% per week.

Healthy foods often have low caloric density, and calories are by far the most important factor. Eat healthy most of the time. “Healthy’’ means food your grandmother or a bodybuilder would approve of. Eat enough protein. Don’t drink calories. Don’t do anything too extreme, like swearing off entire food groups, avoiding social gatherings or dropping weight sharply. It rarely works out long term.

At ~92 kg I look pretty good. At 110 kg? Not so much.

Mute the marketers

People will try to sell you supplements, clothes, gear and entertainment. Buy if you like, but it’s mostly marketing designed to exploit your psychology.

For instance, very few supplements have strong scientific backing. Whey is a cheap source of protein and creatine has documented effects. That’s about it.

As for clothes and gear, buy a pair of Adidas lifting shoes, Rehband knee sleeves and a belt. They will last a decade or more. What kind of T-shirt you wear to the gym hardly matters.

Fitness influencers peddle a mix of minutiae, drama and genuinely good information. In the end, your attention is sold to their advertisers. Use social media feeds with caution: the information does not stick well and comparison is the thief of joy. For information, read a few high-quality books. For motivation, make some real-world friends.

Keep learning

If you enjoyed this article, then The Art of Lifting and The Science of Lifting by Nuckols are short reads in the same spirit. Bigger Leaner Stronger by Matthews is an overall guide to training and nutrition. The books by Eric Helms also fall into this category. Rippetoe wrote three books I enjoyed: Starting Strength covers technique on the main lifts, Practical Programming teaches strength program design and Strong Enough? is a delightful collection of no-bullshit, down-to-earth essays. Dan John has also written many great essays. Atomic Habits by Clear covers habit formation, which is a big part of training and lifestyle.

Unsure where to start? Set some goals based on the strength standards at ExRx.net or strengthlevel.com. Choose a program like Starting Strength, StrongLifts, 5/3/1, Texas Method, etc. Good luck!

Optimal bus stop spacing

Tommy Odland — Sat, 16 May 2026 00:00:00 +0200

Did you know that building new roads can increase traffic congestion in a city? It’s called Braess’s paradox.

In a similar vein—though perhaps not as counterintuitive—adding bus stops to a bus route can increase travel time. That’s the main takeaway from the article “The United States needs fewer bus stops”. It’s great when we can improve a system by subtracting from it.

How can adding a stop increase travel time? The reason is that travel time is a function of two competing mechanisms:

If there are many bus stops, you don’t have to walk far until you get to one
However, with many bus stops, the bus will also have to stop many times on your journey

Simulation

Let us simulate this on a 10 kilometer stretch of road. We assume that population density is uniform along the road, so a journey can be randomly generated by drawing 2 numbers; a start and an end.

The figure below shows 25 random journeys, along with 6 equidistant bus stops 2000 meters apart.

The figure above outlines the basic simulation setup. We’ll also make the following assumptions:

The bus has infinite passenger capacity and always stops at every bus stop.
The bus dwells for 45 seconds at each bus stop.
The bus has a speed of 25 kilometers per hour (slow, but realistic for a city).
A person’s walking speed is 5 kilometers per hour.

Instead of 25 journeys, let us look at one million journeys. The figure below shows the distribution of travel time when traveling by foot and on the bus—a fine-grained histogram with a million journeys.

No journey ever takes more than two hours, since every person walks at a speed of 5 kilometers per hour, and the road is 10 kilometers long. Most journeys are faster by bus, but this is not the case for very short journeys—in that case walking is faster. The average travel time with the bus is 0.35 hours (21 minutes), which is significantly shorter than the average walking time of 0.67 hours.

What if we add more bus stops? Does it help? Let us try with 16 bus stops.

With 16 bus stops the average spacing goes down to 667 meters. The average travel time also goes down to 0.26 hours.

What if we keep adding bus stops? Will travel time continue to decrease? Let’s go all the way to 51 bus stops.

The figure above shows that more bus stops are not better, since with 51 bus stops the travel time is up to 0.36 hours. In fact, a road with 51 bus stops is worse than one with 6 bus stops!

Here is the figure you might have been waiting for; the average travel time as a function of bus stops.

The optimal number of bus stops is 16. Does 16 stops—a spacing of 667 meters—seem reasonable? I think so; you never have to walk for more than 333 meters to catch the bus, and on average you walk 167 meters. This corresponds to about 4 and 2 minutes of walking, respectively.

Analytical solution

The bus stop problem is excellent because it can be approached either by computer simulation or analytical mathematics (deriving equations). In fact, the two approaches complement each other nicely.

Let $n$ be the number of bus stops and $L$ be the length of the road (10 kilometers). Then $\ell = L / (n-1)$ is the equidistant spacing between each bus stop. Furthermore, let $v_w$ be the walking velocity and $v_b$ be the bus velocity. Finally, we denote the dwell time of the bus, measured in hours, by $d$.

The expected total travel time is the sum of three different terms:

Starting at a random location, the average distance to the closest bus stop is $\ell / 4$. We multiply this by two, since we have to walk both to the nearest bus stop at the start of our journey, and from the final bus stop at the end of our journey. The average time walking to and from the bus is therefore $\ell / (2 v_w)$.
The average distance traveled on the bus is roughly $L / 3$. The total time on the bus, excluding dwelling times, is therefore $L / (3 v_b)$.
The number of stops on a journey of length $L/3$ is $L / (3 \ell)$, so the total time on the bus due to dwelling at bus stops is $d L / (3 \ell)$.

The skeptical reader is encouraged to verify the claims above more thoroughly. If we put all this together, we obtain the average time $T$, as a function of bus stops $n$:

\begin{align*} T(n) &= \frac{\ell}{2 v_w} + \frac{L}{3 v_b} + \frac{d L}{3 \ell} \\ &= \frac{L}{(n-1) 2 v_w} + \frac{L}{3 v_b} + \frac{d (n-1)}{3} \end{align*}

Differentiating and solving $T'(n^{\star}) = 0$, we obtain

\begin{align*} n^{\star} = 1 + \sqrt{\frac{3 L}{2 v_w d}} \end{align*}

This analytical equation matches the simulation results nicely, as shown in the figure below.

Notice how the optimal number of bus stops grows like the square root of $L$. If the road segment becomes four times as long, only twice as many bus stops are needed. This is because the average journey also becomes longer, due to the assumption that journeys are uniformly drawn from the line segment.

Why the discrepancy between the simulations and the equation? It’s because bus journeys were simulated by picking the fastest out of four possible journeys. Given a starting point for a journey, a person can either go left or go right to catch the bus. Similarly, a person can either exit the bus before or after the final endpoint of the journey. This gives four possible journeys, and in each simulation I computed the fastest one.

Deltidsstillinger

Tommy Odland — Tue, 14 Apr 2026 00:00:00 +0200

NRK har publisert en sak om overtidsregler for deltidsansatte, og skriver at det er fastlåste posisjoner mellom arbeidsgivere (NHO, KS, Virke og Spekter) og arbeidstakere (Fagforbundet og LO). I denne artikkelen presenterer vi et enkelt kompromiss.

Dagens praksis med merarbeid og overtid

Anta at Ola jobber 50% stilling (20 timer per uke om vi antar en 40-timers uke) og tjener 300 kroner per time.

Hvis han jobber 10 timer over avtalt arbeidstid regnes dette som merarbeid, ikke overtid. Han vil få utbetalt $300 \times (20 + 10) = 9000$ kroner.

Om han derimot jobber 25 timer mer, jobber han 5 timer overtid (over grensa på 40 timer). Dersom satsen for overtid er 40%, får han utbetalt $300 \times 40 + 300 \times 5 \times 1.4 = 14 100$ kroner.

EU-domstolens konklusjon

I to dommer fra EU-domstolen regnes alt arbeid utover avtalt arbeidstid som overtid, ikke merarbeid.

Med disse reglene vil Ola få følgende utbetalt:

Om han jobber 10 timer over avtalt arbeidstid på 20 timer, får han $300 \times 20 + 300 \times 10 \times 1.4 = 10 200$ kroner.
Om han jobber 25 timer over avtalt arbeidstid på 20 timer, får han $300 \times 20 + 300 \times 25 \times 1.4 = 16 500$ kroner.

Problemstillingen

I dag er det ingen strategisk grunn til at en bedrift bør ansette i 100% stilling. Det er bedre å ha to ansatte i 50% stilling, fordi da har arbeidsgiver en viss fleksibilitet til å justere opp arbeidsmengden uten å betale ekstra. Om bedriften trenger 50 arbeidstimer, vil det koste:

Med én ansatt: $300 \times 40 + 300 \times 10 \times 1.4 = 16 200$.
Med to ansatte: $2 \times (300 \times 20 + 300 \times 5) = 15 000$.

I regimet som EU-domstolen legger opp til, er det likegyldig om man har en eller to ansatte. Prisen blir uansett:

Med én ansatt: $300 \times 40 + 300 \times 10 \times 1.4 = 16 200$.
Med to ansatte: $2 \times (300 \times 20 + 300 \times 5 \times 1.4) = 16 200$.

Det er også likegyldig hvem av de ansatte som jobber ekstra. Dersom Ola har 60% stilling og Kari har 40% stilling, så har det ingen betydning for arbeidsgiver om det er Ola eller Kari som jobber ekstra. Det er ingen gevinst i å balansere merarbeidet/overtiden.

Et kompromiss

La funksjonen $p$ være overtidsprosent som funksjon av timer jobbet $x$, og anta at stillingsprosenten (i timer) er gitt ved $s$. Et kompromiss er å velge en lineær funksjon $p(x) = ax + b$.

Dersom vi krever at $p(s) = 100$ og $p(40) = 140$, så får vi

\begin{equation*} p(x;s) = \frac{40}{40 - s} (x - s) + 100 \end{equation*}

Når $x \leq s$ definerer vi $p(x;s) = 100$, og når $x \geq 40$ definerer vi $p(x;s) = 140$. Dersom Ola jobber $t$ timer, skal han ha $300 \int_0^t (p(x) / 100) \, dx$ kroner i lønn. For 30 timer får han 9 300 kroner, sammenlignet med 9 000 i dag og 10 200 under EU-løsningen.

Partene vil neppe bli enige om et slikt kompromiss i praksis, men det har noen fordeler:

Det koster alltid arbeidsgiver å planlegge dårlig og bruke ansatte utover avtalt stillingsprosent, enten de jobber deltid eller heltid.
Arbeidsgiver har insentiv til å gi ansatte høyere stillingsprosent, heller enn å konsekvent bruke merarbeid/overtid.
Arbeidsgiver betaler ikke like mye for å bruke ekstra arbeidskraft fra en ansatt i 50% stilling, som hvis samme arbeidskraft blir hentet fra en ansatt med 100% stilling. Hvem vil du helst bli operert av på sykehuset: fulltidslegen som er på tredje ekstravakt, eller deltidslegen som er på første ekstravakt?
Dersom arbeidsgiver har to personer ansatt i 50% stilling og trenger to vakter ekstra, lønner det seg å fordele dem likt og gi én vakt til Ola og én til Kari. Med andre ord lønner det seg å balansere overtiden på de ansatte, i motsetning til både dagens regime og EU-domstolens tolkning.

Du kan lese LOs argumenter her og noen motargumenter fra arbeidsgiverne her.

Continuous interest with continuous deposits

Tommy Odland — Wed, 25 Mar 2026 00:00:00 +0100

Writing a savings calculator is a great programming exercise for beginners. They’re easy to implement in spreadsheets or programming languages like Python, and teach people about the power of compounding interest. In this article we’ll investigate a discretization error that such calculators often make and try to fix it.

A simple savings calculator

A simple savings calculator might look like this in Python:

interest_rate = 1.1  # 10% annual interest
saved_per_year = 10  # 10 units of money saved per year
years = 20

saved = 0
for year in range(1, years + 1):
    saved = saved * interest_rate + saved_per_year
    print(f"Year: {year:>2}  Saved: {saved:.2f}")

Running the code produces the following output:

Year:  1  Saved: 10.00
Year:  2  Saved: 21.00
Year:  3  Saved: 33.10
...
Year: 18  Saved: 455.99
Year: 19  Saved: 511.59
Year: 20  Saved: 572.75

But there’s a simplification in this calculation which causes us to underestimate the total amount saved. Did you spot it?

The discretization error

The “error” in the script above is noticeable in the first year of the computation, so let’s focus on the first year. We assumed that one year passes before the amount saved per year is deposited into the account. In reality, it would likely be deposited monthly as the person saves money throughout the year.

The bank might pay you the interest owed monthly or yearly, but their internal computation is done every single day. Therefore depositing as early as possible (when you get your salary) will help accumulate more interest on your deposit throughout the year. The same applies to mutual funds.

If the yearly interest rate is $r$, then the monthly interest rate is $r^{1/12}$ and the yearly interest rate is $r^{1/365}$. This is because interest rates compound multiplicatively, so the geometric mean, not the arithmetic mean, is the correct computation.

Back to our savings calculator: in that first year, if we deposit monthly and compute the interest monthly, we do not end up with $10$ units of money at the end of the first year. Instead we end up with $10.45$ units of money:

periods = 12
interest_rate = (1.1)**(1/periods)
saved_per_year = 10 / periods

saved = 0
for period in range(1, periods + 1):
    saved = saved * interest_rate + saved_per_year
    print(f"Period: {period:>2}  Saved: {saved:.2f}")

...
Period: 12  Saved: 10.45

What happens if we compound daily ($365$ times) instead? Or hourly ($365 \times 12$ times)? Or every second ($365 \times 12 \times 3600$ times)? Here is a figure showing the multiplicative factor that money grows by at the end of one year, as a function of the number of times we compound the interest, using $r=1.1$:

For instance, if we compound $12$ times the multiplicative factor is $\approx 1.045$. Observe that the factor converges to a finite number instead of blowing up to infinity. We’ll show that, as we compound interest rates and deposit more and more often, the factor converges to

\begin{align*} (r - 1) / \ln (r), \end{align*}

which for an interest rate $r=1.1$ becomes $0.1 / \ln(1.1) = 1.0492$ (compare with the figure above).

Continuous deposits

Let $a$ be the total amount deposited in a year, let $r$ be the yearly interest rate and $n$ be the number of times interest is compounded.

The naïve savings calculator above used $n=1$.
In terms of realism for a savings calculator, $n=12$ is a good choice if salary is paid every month and then deposited.
From the bank’s perspective, whether it’s a bank account or a mutual fund, $n=365$ is the upper limit.
We’re interested in what happens as $n \to \infty$. This is optimistic because it overestimates the discrete reality, but $n \to \infty$ is still a much better approximation of $n=12$ than $n=1$ (again: see the figure).

Let $a$ be the yearly added value. We can look for a pattern as we increase the number $n$:

\begin{align*} \text{n=1} \quad & a \\ \text{n=2} \quad & \left( \frac{a}{2} \right) r^{1/2} + \frac{a}{2} \\ \text{n=3} \quad & \left( \left( \frac{a}{3} \right) r^{1/3} + \frac{a}{3} \right) r^{1/3} + \frac{a}{3} \\ \text{n=4} \quad & \left( \left( \left( \left( \frac{a}{4} \right) r^{1/4} + \frac{a}{4} \right) \right) r^{1/4} + \frac{a}{4} \right) r^{1/4} + \frac{a}{4} \\ \end{align*}

These are all geometric series, and can be written as:

\begin{align*} \text{n=1} \quad & a \\ \text{n=2} \quad & \frac{a}{2} \left( r^{1/2} + 1 \right) \\ \text{n=3} \quad & \frac{a}{3} \left( r^{2/3} + r^{1/3} + 1 \right) \\ \text{n=4} \quad & \frac{a}{4} \left( r^{3/4} + r^{2/4} + r^{1/4} + 1 \right) \\ \end{align*}

The pattern for a general $n$ emerges, and (provided we assume $r\neq 1$) we can use the formula for a sum of a geometric series to obtain a closed-form expression:

\begin{align*} f_n(a, r) = \frac{a}{n} \sum_{i=0}^{n-1} \left( r^{1/n} \right)^i = \frac{a}{n} \frac{\left( r^{1/n} \right)^n - 1}{r^{1/n} - 1} = \frac{a}{n} \frac{r - 1}{r^{1/n} - 1} \end{align*}

What does this converge to as $n \to \infty$? It all comes down to what the denominator $n (r^{1/n} - 1)$, since that’s the only part of the equation that depends on $n$. The answer is $\ln(r)$, and there are many ways to show this: L’Hôpital’s Rule and Taylor Series are both possible methods. Here’s one way that I like: one of the many definitions of the exponential function $e^x$ is that

\begin{align*} e^x = \lim_{n \to \infty} \left(1+\frac{x}{n}\right)^n \end{align*}

so then, since $\ln(x)$ is the inverse of $e^x$ for positive $x$, we must have

\begin{align*} x = e^{\ln (x)} = \lim_{n \to \infty} \left(1+\frac{\ln (x)}{n}\right)^n \end{align*}

and if we rearrange this expression and substitute $x$ for $r$, we discover that $\lim_{n \to \infty} n (r^{1/n} - 1) = \ln(r)$.

The final result is that

\begin{align*} \lim_{n \to \infty} f_n(a, r) = \lim_{n \to \infty} \frac{a(r - 1)}{n (r^{1/n} - 1)} = \frac{a (r - 1)}{\ln (r)}. \end{align*}

We’re now in a position to create a more realistic savings calculator, in which money is continually deposited and interest is continually accrued:

import math

interest_rate = 1.1
saved_per_year = 10
years = 20

saved = 0
for year in range(1, years + 1):
    saved = saved * interest_rate
    saved += saved_per_year * (interest_rate - 1) / math.log(interest_rate)
    print(f"Year: {year:>2}  Saved: {saved:.2f}")

Year:  1  Saved: 10.49
Year:  2  Saved: 22.03
Year:  3  Saved: 34.73
...
Year: 18  Saved: 478.43
Year: 19  Saved: 536.76
Year: 20  Saved: 600.93

The difference is shown graphically in the animated figure below.

Solution by integration

We can obtain the same answer as above by realizing that as $n$ becomes large we can replace the sum with a Riemann integral:

\begin{align*} f_n(a, r) = \frac{a}{n} \sum_{i=0}^{n-1} r^{i/n} \approx a \int_{0}^{1} r^{x} \, dx \end{align*}

If you know the indefinite integral of $r^x$, then it’s easy to compute

\begin{align*} \int_{0}^{1} r^{x} \, dx = \int_{0}^{1} \exp(x \ln(r)) \, dx = \frac{1}{\ln(r)} \exp(x \ln(r)) \Biggr|^{x = 0}_{x = 1} = \frac{r-1}{\ln(r)}. \end{align*}

It is comforting that both approaches give identical results. Let’s look at a third approach that also leads to the same answer, but is more general and more powerful: differential equations.

Solution by differential equations

Let us summarize what we have learned so far. When we account for continuous deposits and continuous interest rates, we do not have $a$ units of money after the first year. Instead we end up with $a (r-1) / \ln(r)$ units of money. The factor $(r-1) / \ln(r)$ accounts for the difference between discrete yearly deposits and continuous deposits throughout the year. When $r=1$ this factor is undefined, but if we assume $r \neq 1$ then we get this graph:

In the figure above, I also plotted the first-order Taylor expansion around $r \approx 1$, which is $g(r) = 1/2 + r/2$.

The factor that “fixes” the first year, but can be applied every single year in our savings calculator. If we re-write the recursive definition into a sum, and again use the sum of a geometric series, we end up with an expression for an arbitrary year $n$:

\begin{align*} \text{n=1} \quad & a \frac{r-1}{\ln(r)} \\ \text{n=2} \quad & \left( a \frac{r-1}{\ln(r)} \right) r + a \frac{r-1}{\ln(r)} \\ \text{n=3} \quad & \left( \left( a \frac{r-1}{\ln(r)} \right) r + a \frac{r-1}{\ln(r)} \right) r + a \frac{r-1}{\ln(r)} \\ \text{n} \quad & a \frac{r-1}{\ln(r)} \left( r^{n-1} + r^{n-2} + \cdots + 1 \right) = a \frac{r-1}{\ln(r)} \frac{r^n - 1}{r-1} = a \frac{r^n - 1}{\ln(r)} \end{align*}

Perhaps the most powerful approach to studying continuous deposits and continuous interest rates is by using differential equations. Let $M(t)$ be money as a function of time, which is now a continuous function of $t$. Let $r$ be the yearly interest rate and $t$ be measured in years. Before we proceed to the general case, let us study two simple cases first:

Deposits without any interest.

\begin{align*} M(t) = at \qquad M'(t)= a \end{align*}

Interest without any deposits.

\begin{align*} M(t) = M_0 r^t \qquad M'(t) = \ln(r) M_0 r^t = \ln(r) M(t) \end{align*}

Both deposits and interest. Above we saw that the rate of change in a bank account is driven by $a$ as well as $\ln(r) M(t)$. We sum these to obtain the (first order linear ordinary) differential equation

\begin{align*} M'(t) = \ln(r) M(t) + a. \end{align*}

The solution to this differential equation (verify it!) is

\begin{align*} M(t) = M_0 r^t + \frac{a (r^t - 1)}{\ln(r)}, \end{align*}

where $M_0 = M(0)$ is the initial money in the account. This matches our previous result perfectly, where we assumed $M_0 = 0$.

As a final note, if $a$ is negative and we’re depleting the account, then a relevant question is “When will we go broke?”. This amounts to solving $M(t) = 0$ for an unknown $t$. In the general case, let $M(t) = M_t$, then we can solve for $t$ to obtain:

\begin{align*} t = \frac{1}{\ln (r)} \ln \left( \frac{M_t \ln(r) + a}{M_0 \ln(r) + a} \right) \end{align*}

Finally, here is an updated Python snippet that uses the function $M(t)$.

import math


def M(t, r, a, M0):
    r_power_t = math.exp(t * math.log(r))
    return M0 * r_power_t + a * (r_power_t - 1) / math.log(r)


interest_rate = 1.1
saved_per_year = 10
years = 20

for year in range(1, years + 1):
    saved = M(t=year, r=interest_rate, a=saved_per_year, M0=0)
    print(f"Year: {year:>2}  Saved: {saved:.2f}")

Summary and references

This is an example of how the first thing one might implement is not necessarily correct. I’ve taught students about geometric series, Excel and Python using this example a few times, and I was aware of the tension between computing the interest and deposit at the end of the year (which underestimates) and at the start of the year (which overestimates).

For students familiar with calculus and geometric series, working out the limit of $n \to \infty$ is a nice problem. In the end, assuming $n \to \infty$ is not really correct either—it probably overestimates slightly compared to the reality (which is likely $n=12$ or so). In many cases the corrective factor of $(r - 1) / \ln (r)$ is likely worth implementing since it’s closer to the truth than the naive $n=1$ computation. We ignored most issues related to numerical stability in this article, but in a proper implementation $r^{(1/n)}$ and $(r-1) / \ln(r)$.

Everything can be summarized by this set of equations: one discrete difference equation and one continuous differential equation. The equations and their closed-form solutions are:

\begin{align*} \boxed{M_n = r M_{n-1} + a} \Rightarrow M_n &= M_0 r^n + a \frac{r^n - 1}{r - 1} \\ \boxed{M'(t) = \ln(r) M(t) + a} \Rightarrow M(t) &= M_0 r^t + a \frac{r^t - 1}{\ln(r)} \end{align*}

Some further reading:

Modeling Change One Step at a Time, from Topics in Mathematical Modeling
Differential Equations by Chasnov
Continuous-repayment mortgage on Wikipedia
Compound interest on Wikipedia

Skatter og bidrag

Tommy Odland — Fri, 13 Feb 2026 00:00:00 +0100

Denne artikkelen er et tankeeksperiment der vi skal endre det norske skatte- og bidragssystemet. I dag styrer mange mekanismer pengeflyten mellom borgere og staten. Et forenklet bilde ser slik ut:

Kompleksiteten i et slikt system har store kostnader. Staten må holde styr på alle skattene og bidragsordningene: den må oppdatere lovverk, endre satser, vurdere og innvilge søknader, vedlikeholde digital infrastruktur, osv. Borgere må forstå systemet, være informert om støtteordningene og navigere søknadsprosesser. Begge parter kunne vært tjent med en forenkling.

Vi skal kutte nesten alle skatter og samle dem sammen i en funksjon $T(x)$. Deretter skal vi kutte nesten alle støtteordninger og samle dem sammen i en funksjon $B(x)$.

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k} \qquad B(x) = be^{-x/b} \end{align*}

Det er ikke åpenbart at disse funksjonene er i stand til å modellere store deler av forholdet mellom staten og borgerne. Vi skal prøve likevel. Resultatet blir at forholdet mellom staten og individet forenkles til følgende diagram.

Alle skatter og bidrag styres av tre parametre $H$, $k$ og $b$, og vi skal estimere disse. Underveis skal vi gi skattelette til folk flest, “ta de rike”, forenkle offentlig sektor, sørge for at det alltid lønner seg å jobbe, og kreve formuesskatt fra alle borgere. Så her er det noe for alle, uansett politisk ståsted.

Inntekt
Bidrag og støtteordninger
En realistisk beregning
- Oppsummering: Bidrag og støtteordninger
Formuesskatt
Oppsummering

Inntekt

For å regne ut skatt på en vanlig borgers inntekt må man ta hensyn til trygdeavgift, minstefradrag, personfradrag, skatt på alminnelig inntekt og trinnskatt. Dette er et komplisert regnestykke, og i artikkelen Smooth taxes without brackets viste jeg at beregningen kan reduseres til formelen

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k}, \end{align*}

der $T(x)$ er skatten man betaler om man har en brutto inntekt på $x$ kroner. Parametrene $H$ og $k$ bestemmer hvor aggressiv skatten er. $H$ er den maksimale marginalskatten ettersom $H = \lim_{x \to \infty} T'(x)$. Figuren nedenfor viser at funksjonen er en meget god tilpasning til dagens inntektsskatt.

I figuren ovenfor viser vi også inntektsfordelingen, som er hentet fra figur 2 i denne SSB-artikkelen. Boks 2.2 i Skatter og avgifter 2025 viser statens egen visualisering av marginalskatten. Denne skattekalkulatoren har 27 felter som kan fylles ut, så den fulle beregningen er enda mer komplisert.

Matematiske egenskaper

Her kommer en rask oppsummering av hvilke spørsmål funksjonen $T(x)$ svarer på og hvilke egenskaper som er hensiktsmessige for en slik skattefunksjon. Se Smooth taxes without brackets for en grundigere gjennomgang.

Nettoinntekt: $x - T(x)$ svarer på spørsmålet “Hvor mye penger har jeg igjen etter at jeg har betalt skatt?”
Marginalskatt: $T'(x)$ svarer på “Hvor stor andel må jeg skatte på neste krone jeg tjener?”
Skatteprosent: $T(x) / x$ svarer på “Hvilken andel av total inntekt betaler jeg i skatt?”

Eksempel. Dersom $x=800 \text{ TNOK}$, så er:

\begin{align*} T(x) = 227 \text{ TNOK} \quad T'(x) = 0.423 \quad T(x)/x = 0.284 \end{align*}

Funksjonen $T(x)$ har to viktige egenskaper bakt inn i definisjonen, for all verdier av $0 < H \leq 1$ og $k > 0$:

Gode insentiver. En skatt $T(x)$ har gode insentiver dersom marginalskatten $T'(x) < 1$ for alle $x$, fordi da ender man alltid opp med økt nettoinntekt dersom man tjener mer.
Progressiv beskatning. En skatt $T(x)$ er progressiv dersom skatteprosenten $T(x)/x$ øker når $x$ øker. Dersom Ola tjener mer enn Kari, så betaler Ola også en større prosentandel av sin inntekt i skatt.

Hva kan vi forenkle?

Fra en borgers perspektiv er kun skattenivået relevant, ikke hva staten kaller de ulike skattene og fradragene eller hva som finansierer hva. Dersom vi bruker $T(x)$ kan vi ta bort:

Minstefradrag
Personfradrag
Skatt på alminnelig inntekt
Trinnskatt
Arbeidsgiveravgift (det er i praksis en personskatt på 14.1 %)

La oss også samle all inntekt i $x$: pensjonsgivende inntekt, utbytte fra selskaper, aksjegevinst, styrehonorarer, osv. De som eier et selskap kan i dag velge å enten ta ut penger som inntekt eller som utbytte. Inntekt gir pensjonssparing gjennom folketrygden og rett til sykepenger, men utbytte har lavere marginalskatt. Det er ofte optimalt å først maksimere pensjonssparing ved å ta ut lønn opp til 7.1 G og deretter ta utbytte. Denne typen optimalisering er kun tilgjengelig for et fåtall. Jo enklere skatten er, desto mindre muligheter er det for optimalisering.

Om vi er villige til å fjerne noen flere, mer målrettede insentivordninger, kan vi fjerne:

Soner og næringsgrupperinger i arbeidsgiveravgift (14 koeffisienter)
BSU fradrag
Fagforeningskontingent
Foreldrefradrag
Reiseutgifter

Noen av disse fradragene bør modelleres som bidrag, og vi kan legge dem tilbake senere.

Vanlige folks tur

Vi kan gi skattelette til store deler av befolkningen om vi velger en høyere verdi for maksimal marginalskatt $H$ og løser for $k$ slik at statens inntekter forblir identiske med dagens nivå ($H \approx 0.47$).

Medianpersonen har en inntekt på omtrent 640 TNOK før skatt. Går vi fra dagens skatteregime til $H=0.55$, vil medianpersonen få skattelette på omtrent 3.8 TNOK per år. Setter vi heller $H=0.65$ får medianpersonen skattelette på 7.4 TNOK per år. Alle som er under 80 persentilen vil få skattelette, mens de som er over må skatte mer. I dagens system har DNB-sjefen, som tjente 16.9 MNOK i fjor, samme marginalskatt som en dyktig ingeniør (eller lege, advokat, osv.) som tjener en tiendedel av dette.

Tabellen sammenligner dagens scenario med to alternativer. Statens inntekter er identiske.

	$H=0.474$	$H=0.55$	$H=0.65$
Skattelette for medianperson	0 NOK	3800 NOK	7400 NOK
Andel som får skattelette	0 %	79 %	80 %
Marginalskatt DNB-sjefen	47 %	55 %	65 %
Marginalskatt ingeniør	47 %	53 %	60 %
Skatteprosent DNB-sjefen	46 %	53 %	63 %
Skatteprosent ingeniør	37 %	40 %	43 %

Oppsummering: Inntekt

Vi samler alt en person tjener i én felles inntekt $x$ og beskatter den med $T(x)$, enten det er ordinær pensjonsgivende inntekt, utbytte fra selskaper, avkastning på aksjefond, styrehonorar, godtgjørelser eller annet. Da vil en rik investor med 10 MNOK i aksjegevinst skatte en høyere andel enn en småsparer, noe som ikke er tilfellet i dag. Mange muligheter for optimalisering forsvinner, og de som får inntekt gjennom utbytte, avkastning og honorarer får samme beregning som folk flest. Ikke minst blir alt mye enklere. Vi kan endre $H$ og gi skattelette til 80 % av arbeidsstyrken, men insentivene må også vurderes.

Vi kan også inkludere arv i $x$, ettersom det også er inntekt. Det ville vært en dobbeltbeskatning av penger, men ikke en dobbeltbeskatning av personer. Dersom man arver en villa med nedbetalt lån i Oslo med verdi 20 MNOK, ville man måtte låne 9.1 MNOK i boligen for å betale skatten. Dette ville utjevnet forskjeller og forhindret opphopning av kapital. Alternativt kan vi droppe arveskatt og heller endre formuesskatten, noe vi kommer til senere i artikkelen.

Bidrag og støtteordninger

Det eksisterer mange støtteordninger for mennesker som av ulik grunn har lav inntekt, enten i en periode eller permanent. NAV har over 200 tjenester, og noen av disse er:

Uføretrygd
Sykepenger
Arbeidsavklaringspenger (AAP)
Dagpenger
Tiltakspenger

I disse fem kategoriene ble det utbetalt 27.3 milliarder NOK til 166 000 Oslo-borgere i 2024, noe som tilsvarer 164 TNOK per person. I noen bydeler er andelen som mottar utbetalinger relatert til arbeidsliv og sykdom over 40 %, og på landsbasis er den 36 %. Andre støtteordninger inkluderer pleiepenger, omsorgspenger, svangerskapspenger, yrkesskadeerstatning, foreldrepenger og økonomisk sosialhjelp.

Kritikk mot NAV går ofte på at dagens system:

Er krevende å navigere
Har dårlige insentiver

Systemet blir lettere å navigere om vi fjerner kompleksitet. Problemstillingen med insentiver kan forstås ved å se på enkeltsaker som viser såkalte bidragsfeller (welfare traps). I noen tilfeller lønner det seg ikke å jobbe fordi da forsvinner bidragene:

NAVs egen rapport fra 2023 anslår at 1 av 4 dagpengemottakere kan tape på å jobbe én dag. Mange mennesker lever i økonomiske bidragsfeller.

Å fange de fattige

Det er fort gjort å konstruere en bidragsfelle. Intensjonene må være så gode at man ikke tenker nøye nok over insentivene. Dersom bidraget er betinget på lønn (enten direkte eller indirekte via arbeidsevne) og kuttes skarpt når inntekt øker, så sitter man i fella. Dersom bidraget derimot senkes sakte, så unngår man feller.

Eksempel - konstant bidragsfelle. Hva om vi gir alle som tjener mindre enn 500 TNOK en utbetaling på 500 TNOK? Anta at vi har fem personer med ulik inntekt. La $B(x)$ være et bidrag som funksjon av bruttoinntekt $x$. Vi setter $B(x)$ lik 500 TNOK dersom $x$ er mindre enn eller lik 500 TNOK, og null hvis ikke. Visualiseringen viser hva som skjer:

Når man overstiger 500 TNOK i inntekt, så forsvinner hele bidraget $B(x)$ (øverste underfigur til høyre). Det rasjonelle er å ligge så nær 500TNOK som mulig, men aldri gå over. Dersom summen $x - T(x) + B(x)$ synker er det ikke noe insentiv for å tjene mer.

Eksempel - lineær bidragsfelle. Hva om vi velger en annen funksjon? Vi kan kompensere de som tjener mindre enn 500 TNOK med en sum slik at alle ender opp med minst 500 TNOK. Matematisk er funksjonen $B(x) = \max(500\text{ TNOK} -x, 0)$.

For en som har null i inntekt lønner det seg ikke å jobbe i det hele tatt, med mindre personen klarer å tjene ca. 665 TNOK. Selv da får han ikke betydelig mer enn hva han fikk med null inntekt.

Bidrag $B(x)$ med gode egenskaper

Det finnes bidrag $B(x)$ som i kombinasjon med $T(x)$ ikke fører til bidragsfeller. Funksjoner på formen

\begin{align*} B(x) = b e^{-x/b} + C \qquad b < \frac{1}{Hk} \end{align*}

har gode egenskaper: insentivene er gode og de er billige for staten.

Summen av skatt og bidrag har gode insentiver dersom det alltid lønner seg å jobbe. Matematisk er kravet at

\begin{align*} \underbrace{(x + \epsilon) + B(x + \epsilon) - T(x + \epsilon)}_{\text{netto med $\epsilon$ mer}} > \underbrace{x + B(x) - T(x)}_{\text{netto}} \end{align*}

Dersom vi stokker om på likningen og tar grenseverdien $\epsilon \to 0$ får vi kravet

\begin{align*} T'(x) - B'(x) < 1. \end{align*}

Vi deriverer skatten $T(x)$ og substituerer $T'(x) = H\left(1 - e^{-kx} \right)$. Da blir kravet at

\begin{align*} B'(x) > H\left(1 - e^{-kx} \right) - 1. \end{align*}

Denne likningen sier at bidragene ikke kan synke fort mot null når inntekten $x$ øker. Dette gir intuitivt mening; bidrag som forsvinner fort fører til bidragsfeller. Alle funksjoner $B(x)$ som oppfyller dette kravet er gyldige, og kvalifisert gjetning leder oss til $B(x) = b e^{-x/b} + C$, som oppfyller kravet $T'(x) - B'(x) < 1$ når $b < 1/(Hk)$ (se appendiks).

For eksempel er $b=200 \text{ TNOK}$ og $C=0$ gyldig, og gir følgende omfordeling via skatter og bidrag:

Konstantleddet $C$ kalles gjerne borgerlønn (universal basic income), mens $b e^{-x/b}$ er i kombinasjon med $T(x)$ en negativ skatt. Problemet med $B(x) = C$ er at det er dyrt for staten, fordi alle får bidrag. I resten av artikkelen setter vi $C=0$ og antar at $B(x) = b e^{-x/b}$. Legg merke til at $B(0) = b$, altså er $b$ summen som en person uten inntekt vil få utbetalt.

Verdien $b$ må balansere to hensyn: den må være høy nok til at det går an å leve på, samtidig må den være lav nok til å insentivere til arbeid.

Hva kan vi forenkle?

Dersom vi innfører et universelt bidrag $B(x) = b e^{-x/b}$ kan vi fjerne eller kutte kraftig i uføretrygd, sykepenger, arbeidsavklaringspenger, dagpenger, tiltakspenger og økonomisk sosialehjelp. NAV har 12 000 årsverk og 15 mrd. i driftsutgiver, og deler av dette kan kuttes. Tiden og ressursene som brukes på å sende inn og behandle søknader forsvinner helt.

Staten betaler også bidrag til friske mennesker for å søtte en rekke aktiviteter. Eksempelvis var det 12 000 søkere på statens kunstnerstipend i 2024, og i 2025 ble det bevilget nesten 500 millioner kroner. Mange slike ordninger kan fjernes. Mennesker som ønsker å lage kunst eller gjøre noe helt annet (som kan være vel så samfunnsnyttig men ikke anerkjent av staten), står fritt til å gjøre det—dersom de er villige til å leve på bidraget $B(x)$. Dersom kunsten selger, vil de få mer inntekt, miste bidrag og betale skatt—men de vil alltid ende opp med mer penger.

En realistisk beregning

I denne seksjonen forsøker vi grovt å regne ut hva $B(x)$ vil koste. Her er noen store utgiftsposter, hentet fra NAVs årsrapport 2024:

Uføretrygd: 125 269 MNOK
Sykepengetilfeller: 68 068 MNOK
Arbeidsavklaringspenger: 46 926 MNOK
Dagpenger: 13 300 MNOK
Hjelpemidler: 10 321 MNOK

Til sammen er dette 264 milliarder kroner. I følge SSB er 622 000 personer i yrkesaktiv alder utenfor arbeidslivet (omtrent 18 %). Fordeler vi 264 milliarder på 622 000 personer kan hver person få 424 TNOK.

Ettersom $B(x)$ også deler ut penger til de som ikke har null i inntekt, må vi ta hensyn til de 82 % som er i arbeidslivet. Jeg fant ingen gode data som viser inntektsfordeling for alle i Norge, men om vi baserer oss på fulltidsekvivalenter for de 82 % som er i arbeidslivet og kombinerer det med de 18 % som er utenfor, kommer vi frem til at 264 milliarder kan finansiere $b = 271 \text{ TNOK}$. At vi bruker fulltidsekvivalenter gjør at $b$ overestimeres. At vi verken inkluderer flere støtteordninger eller administrative kostnader gjør at $b$ underestimeres. Disse effektene kansellerer hverandre til en viss grad, og i sum er nok $b$ estimert noelunde riktig.

Men hvor mye bør staten dele ut? Det finnes ingen klar fasit, men her er to ankerpunkter:

SIFOs referansebudsjett er på omtrent 190 TNOK i året (eksl. bolig, ferier og helsetjenester). Boliginformasjon kan vi finne hos SSB, som estimerer typisk boligkostnad for eiere til 130 TNOK per år og leietakere til 111 TNOK per år. En person trenger derfor omtrent 190 + 120 = 310 TNOK per år for å leve greit.
Lavinntekstsgrensa måler relativ fattigdom og er definert til å være 60% av medianinntekt etter skatt. Grensa er på 285 TNOK i 2025. Oppsummert har staten råd til $b \approx 271 \text{ TNOK}$, og dette nivået samsvarer godt med SIFOs referansebudsjett og fattigdomsgrensen.

Oppsummering: Bidrag og støtteordninger

Bidrag og støtteordninger er i hovedsak til for syke og arbeidsledige. Dagens systemer kan være vanskelige å navigere, kreve mye dokumentasjon, og kan disinsentivere arbeid. Staten må bedømme hva som er verdifullt for samfunnet, selv om det kanskje ikke er kommersielt levedyktig. Stipendordninger kan føre til habilitetsutfordringer og staten må bruke ressurser på å bestemme hvem som får hva.

Vi innfører en bidragsfunksjon $B(x)$, som sammen med $T(x)$ gir negativ skatt til de som tjener lite. Dette krever ingen søknadsprosess eller vurdering fra statens side. Vi fjerner ikke tiltak som jobbsøkerkurs og økonomisk veiledning, ei heller bidrag til rullestoler, proteser, osv. Bidragene kunne vært satt til omtrent 300 TNOK per år, men det eksakte beløpet er en subjektiv vurdering. I hvilken grad barn, studenter og pensjonister omfattes av ordningen kan diskuteres. Det enkleste er å gi bidrag til disse gruppene og fjerne eksisterende støtteordninger.

Formuesskatt

Denne delen av artikkelen er uten sammenheng med resten, som i hovedsak handler om inntekt og bidrag, men la oss se på formueskatt også.

Den marginale formueskatten har to trinn: ett på 1.76 MNOK (1 %) og ett på 20.7 MNOK (1.1 %). Den er med andre ord langt mindre progressiv enn inntektsskatt, og den slår inn relativt tidlig. Effekten er nedjustert av verdsettingsrabatter; eksempelvis har primærbolig en rabatt på hele 75 % mens aksjer har en rabatt på 20 %.

De rike blir rikere på grunn av renters-rente effekten. Den brede aksjeindeksen S&P 500 har hatt en gjennomsnittlig inflasjonsjustert avkastning på 7.65 % de siste 30 årene. En rik person med 100 MNOK investert vil derfor få 7.65 MNOK i avkastning. Anta at denne avkastningen realiseres og beskattes med 3.46 MNOK (med $T(x)$) slik at personen står igjen med en gevinst på 4.2 MNOK. Formuen er nå 100 + 4.2 = 104.2 MNOK, og formueskatten blir 1.1 MNOK, så personen står igjen med 103.1 MNOK. Ved å sette pengene i en bred indeks som S&P 500, som ikke krever noe arbeid eller oppfølging, kan personen hente ut 3.1 MNOK per år. Formuen vil ha samme avkastning hvert eneste år.

Figuren nedenfor viser den marginale formueskatten og simuleringer av ulike formuer i S&P 500. Formuene vokser lineært på log skala, altså eksponentielt.

Dagens marginale formueskatt er i praksis flat, og da vil formuene enten vokse evig (om formueskatten er lav) eller gå mot null (om formuesskatten er høy). Ved å konstruere en marginal formuesskatt som er mer progressiv kan vi begrense de rikes formuer.

La $y$ være formue. Ved å bruke en sigmoid $z = 1 / (1 + e^{-z})$ på log-skala kan vi konstruere en formuesskatt:

\begin{align*} W'(y) = \frac{H}{1 + (y / y_0)^{-k}} \qquad W(y) = \frac{H y^{1 + k}}{(k+1) y_0^k} {_2F_1}\left(1, 1 + 1/k, 2 + 1/k, -(y/y_0)^k\right) \end{align*}

Parameteren $H$ er maksimal marginal formuesskatt, $y_0$ er den formuen som gir $W'(y_0) = H/2$ og $k > 0$ bestemmer hvor raskt sigmoid-funksjonen vokser. Den marginale formueskatten $W'(y)$ er gitt av en relativt uskyldig formel, mens $W(y)$ er uttrykt ved hjelp av en hypergeometrisk funksjon ${_2F_1}$ (se appendiks). Man trenger ikke å vite noe om hypergeometriske funksjoner for å bruke eller forstå denne funksjonen.

Den progressive marginale formuesskatten $W'(y)$ er vist i figuren nedenfor, og sammenlignet med veksten til ulike formuer. Parametrene er satt ut fra øyemål for å vise effekten på en tydelig måte. Merk at veksten til ekstremt store formuer begrenses, mens mindre formuer får vokse.

Med $W(y)$ betaler alle formuesskatt, men de fleste betaler veldig lite. En formue på 100 TNOK gir 19 kr i formueskatt, 1 MNOK gir 1200 kr, 10 MNOK gir 64 TNOK og 100 MNOK gir 2.2 MNOK.

Alle eksemplene i denne seksjonen antar at formuen er likvid. Dette er ikke tilfellet dersom formuen består av aksjer i selskaper som ikke er på børs, eller om formuen er i eiendom. Formuesskatt kan føre til at man mister kontrollen over sitt eget selskap eller mister sin egen bolig. En bolig gir bruksverdi til eieren på en annen måte enn et selskap, så argumentet for formuesskatt på eiendom er sterkere enn argumentet for formuesskatt på illikvide aksjer. Dersom vi verken har formueskatt eller arveskatt, kan en rik familie sette formuen i aksjemarkedet og oppnå eksponentiell vekst for all fremtid.

Oppsummering

I dette tankeeksperimentet endret vi skattesystemet ved å innføre to funksjoner: $T(x)$ for skatt og $B(x)$ for bidrag. Parametrene ($H$, $k$ og $b$) eliminerer kompleksiteten i skatteberegninger, mange av NAVs støtteordninger og en rekke andre overføringsmekanismer. Den generelle idéen om borgerlønn og negativ inntektsskatt er gammel, mens de spesifikke funksjonene og deres egenskaper er mitt eget bidrag. Les gjerne Civita-notat nr. 38 om borgerlønn for en grundig gjennomgang av borgerlønn i norsk sammenheng.

Økonomiske insentiver som Finnmarksfradraget i skatteberegningen og sletting av gjeld for beboere i Finnmark kan modelleres ved å velge litt andre parametre (f.eks. $b$ eller $C$) i Finnmark. Her bør staten vise måtehold, ettersom mye av poenget forsvinner dersom man får en eksplosjon i antall parametre. Vi viste også at en sigmoid-funksjon $W(y)$ som marginal formueskatt begrenser ekstrem formuesvekst.

Noen av fordelene med et slikt system er:

Å øke maksimal marginalskatt $H$ kan gi skattelette til omtrent 80 % av befolkningen.
Skatteoptimaliseringsmuligheter som primært kommer de rike til gode fjernes.
Bidragsfeller som i dag fanger mange NAV-brukere blir eliminert.
Den administrative besparelsen ville vært stor. Tusenvis av årsverk kuttes og borgere slipper søknadsprosesser.

Det er imidlertid et tankeeksperiment og en øvelse i matematikk, ikke et gjennomtenkt politisk forslag. Mange av kompleksitetene vi ønsker å fjerne reflekterer legitime forsøk på å balansere konkurrerende samfunnsmål. Noen vil mene at det er moralsk riktig å gi støtte ut fra hvem som fortjener det eller ikke, selv om det koster å gjennomføre slike vurderinger. En mer aggressiv formueskatt kan utløse kapitalflukt. Mange ville brukt økonomisk trygghet til å investere i utdanning, starte bedrifter eller ta seg av familie. Andre ville potensielt redusert arbeidsinnsatsen eller i verste fall ty til destruktiv atferd. En implementering ville krevd utredninger av atferdsresponser, overgangsordninger og konsekvenser. Men hvis det er et land som har råd til å utrede, så er det Norge.

Referanser og videre lesing

Utbetalinger til personer i Norge - Statistikk for 2024 - NAV
Årsrapport 2024 - NAV
Nasjonalbudsjettet 2025 - Regjeringen
Hvorfor snakker vi ikke om borgerlønn? - NRK Ytring (2014)
Borgerlønn - Civita-notat nr. 38 (2017) 50 tips som kan kutte skatten din - smartepenger.no
Negative income tax - Wikipedia
Is There Empirical Evidence on How the Implementation of a Universal Basic Income (UBI) Affects Labour Supply? A Systematic Review - Artikkel

Appendiks A: Bevis for at bidragsfunksjonen har gode insentiver

I dette appendikset skal vi vise at når vi velger funksjoner

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k} \qquad \text{og}\qquad B(x) = Abe^{-x/b} + C \end{align*}

med $0 \leq H < 1$, $b > 0$, $k > 0$ og $A > 0$, så er har vi alltid gode insentiver fordi

\begin{align*} T'(x) - B'(x) < 1 \end{align*}

for alle $x > 0$ dersom $Hkb < A$ og $0 < A \leq 1$. Det naturlige er å velge $A=1$, fordi da synker bidraget raskest og det blir billigst for staten.

Bevis. Vi deriverer funksjonene og setter dem inn i ulikheten. Da får vi

\begin{align*} f(x) &= -T'(x) + B(x) + 1 > 0 \\ f(x) &= H\left(e^{-kx} - 1 \right) - A e^{-x/b} + 1 > 0. \end{align*}

Vi ser også at

\begin{align*} f'(x) &= -Hk e^{-kx} + \frac{A}{b} e^{-x/b} \\ f'(x) &= 0 \, \Rightarrow \, x (k - \frac{1}{b}) = \ln\left( \frac{Hkb}{A} \right). \end{align*}

Dersom $f(x)$ er positiv og økende når $x=0$, konvergerer mot et positivt tall og har maksimalt ett ekstremalpunkt når $x > 0$, så er $f(x) > 0$ for alle $x > 0$ og ulikheten er gyldig.

Vi observerer at:

$f(0) = -A + 1 \geq 0 \Rightarrow A \leq 1$
$f'(0) = -Hk + A/b > 0 \Rightarrow A > Hkb$
$\lim_{x \to \infty} f(x) = 1 - H > 0$
$f(x)$ har ett ekstremalpunkt fordi $f'(x) = 0$ har én løsning

Dette gir oss kravene (1) $A \leq 1$ og (2) $A > Hkb$. I artikkelen valgte vi $A=1$, og da blir kravet for $b$ at $b < 1/(Hk)$.

Appendiks B: Formuesskatten

I dette appendikset skal vi utlede formuesskatten $W(y)$ fra den marginale formuesskatten $W'(y)$. Vi definerte den marginale formuesskatten som

\begin{align*} W'(y) = \frac{H}{1 + (y / y_0)^{-k}} = \frac{H}{1 + \exp(-k \ln (y / y_0))}. \end{align*}

Målet er å evaluere integralet

\begin{align*} \int_0^y W'(z) \, dz = H \int_0^y \frac{1}{1 + \exp(-k \ln (z / y_0))} \, dz \end{align*}

for å finne den totale formuesskatten ved et beløp $Y$. Først gjør vi substitusjonen $u = \ln(z/y_0)$, som transformerer integralet til

\begin{align*} H y_0 \int_{-\infty}^{\ln(y/y_0)} \frac{e^u}{1 + e^{-ku}} \, du = H y_0 \int_{-\infty}^{\ln(y/y_0)} \frac{e^u e^{ku}}{1 + e^{ku}} \, du. \end{align*}

Deretter gjør vi substitusjonen $z = e^{ku}$, som transformerer integralet til

\begin{align*} \frac{H y_0}{k} \int_{0}^{(y/y_0)^k} \frac{z^{1/k}}{1 + z} \, dz \end{align*}

og dette er et kjent integral som kan formuleres som en hypergeometrisk funksjon. Vi kan bruke identiteten

\begin{align*} \int \frac{x^{1/k}}{1 + x} \, dx = \frac{k x^{1 + 1/k}}{k+1} \, {_2F_1}\left( 1, 1 + 1/k, 2 + 1/k, -x \right) \end{align*}

som gir oss vårt endelig svar:

\begin{align*} W(y) = \int_0^y W'(z) \, dz = \frac{H y^{1 + k}}{(k+1) y_0^k} \, {_2F_1}\left(1, 1 + 1/k, 2 + 1/k, -(y/y_0)^k\right) \end{align*}

Appendiks C: Kode

Her er Python-kode som implementerer hovedidéene i artikkelen.

import dataclasses  # Python 3.12
from abc import ABC, abstractmethod
import numpy as np  # 2.3.3
import scipy as sp  # 1.16.2


class Transfer(ABC):
    """Abstract base class for transfer functions (taxes, benefits, etc.)."""

    @abstractmethod
    def marginal(self, x):
        """Marginal rate at income x."""
        pass

    @abstractmethod
    def absolute(self, x):
        """Absolute value at income x."""
        pass

    def average(self, x):
        """Average rate: absolute(x) / x."""
        return self.absolute(x) / x


@dataclasses.dataclass
class Tax(Transfer):
    """Class representing a tax function T(x).

    Examples
    --------
    >>> T = Tax(k=2.787914e-06, H=0.474)
    >>> T.absolute(500_000)  # Current norwegian tax system: 113643.99
    np.float64(109160.85121307125)
    >>> T.absolute(1_000_000)  # Current norwegian tax system: 313248.14
    np.float64(314445.0244402999)
    """

    # Defaults that give best-fit to current tax system in 2025,
    # weighted by income of population.
    k: float = 2.8745e-06
    H: float = 0.4772

    def __post_init__(self):
        assert self.k > 0
        assert self.H > 0

    def marginal(self, x):
        """Marginal tax rate T'(x)."""
        H, k = self.H, self.k
        return H - H * np.exp(-k * x)

    def absolute(self, x):
        """Absolute value of tax T(x)."""
        H, k = self.H, self.k
        return (H / k) * (np.exp(-k * x) + k * x - 1)


@dataclasses.dataclass
class Benefit(Transfer):
    """Class representing a benefit function B(x).

    Examples
    --------
    >>> B = Benefit(300_000)
    >>> B.absolute(0)
    np.float64(300000.0)
    >>> B.absolute(300_000)  # exp(-1) * 300_000 = 110363
    np.float64(110363.8323514327)
    """

    b: float = 271_000  # Somewhat realistic value

    def __post_init__(self):
        assert self.b > 0

    def marginal(self, x):
        """Marginal benefit rate B'(x)."""
        return -np.exp(-x / self.b)

    def absolute(self, x):
        """Absolute value of benefits B(x)."""
        return self.b * np.exp(-x / self.b)


@dataclasses.dataclass
class WealthTax(Transfer):
    """Class representing a wealth tax function W(y).

    Uses a logistic-based wealth tax with hypergeometric function
    for absolute tax calculation.

    Examples
    --------
    >>> W = WealthTax(y_0=10_000_000, H=0.03, k=1.0)
    >>> W.marginal(10_000_000)  # At threshold, marginal rate is H/2
    np.float64(0.015)
    >>> W.average(2_000_000)  # Average tax on 2 MNOK wealth
    np.float64(0.0026517664809068064)
    >>> W.average(20_000_000)  # Average tax on 20 MNOK wealth
    np.float64(0.013520815669978353)
    """

    y_0: float = 10_000_000  # Wealth threshold
    H: float = 0.03  # Maximum marginal rate
    k: float = 1  # Steepness parameter

    def __post_init__(self):
        assert 0 <= self.H < 1
        assert self.y_0 > 0
        assert self.k > 0

    def marginal(self, y):
        """Marginal wealth tax rate W'(y)."""
        y_0, H, k = self.y_0, self.H, self.k
        return H / (1 + np.exp(-k * np.log(y / y_0)))

    def absolute(self, y):
        """Absolute value of wealth tax W(y)."""
        y_0, H, k = self.y_0, self.H, self.k
        power = (y / y_0) ** k
        factor = H * y * power / (k + 1)
        return factor * sp.special.hyp2f1(1, 1 + 1 / k, 2 + 1 / k, -power)


if __name__ == "__main__":
    import pytest

    pytest.main(["--doctest-modules", __file__])

    def numerical_derivative(func, x, h=1e-4):
        return (func(x + h) - func(x - h)) / (2 * h)

    # Test that derivatives are correct for all classes with defaults
    for cls in [Tax, Benefit, WealthTax]:
        instance = cls()
        x = np.logspace(3, 7)
        analytical = instance.marginal(x)
        numerical = numerical_derivative(instance.absolute, x)
        assert np.allclose(numerical, analytical, rtol=0.01)

Practical Linear Algebra with NumPy

Tommy Odland — Fri, 30 Jan 2026 00:00:00 +0100

Whether you’re solving differential equations, training neural networks, inferring parameters in a statistical model or solving optimization problems, you’ll find linear algebra at the core of almost every routine. Still, mistakes in the application of linear algebra are all too common. Even though it’s a novice numerical linear algebra error, I’ve seen professors solve $A \boldsymbol{x} = b$ as b = np.linalg.inv(A) @ x and call it a day.

Here’s a list of beginner linear algebra mistakes, followed by some tips for writing better and faster numerical code. Remember to start with a correct implementation before working on speed.

Don’t compare floats
Don’t multiply diagonal matrices
Don’t add diagonal matrices
Don’t multiply matrices in arbitrary order
Don’t compute a product if you only need the diagonal
Don’t form the inverse matrix
Don’t use memory you don’t need
Don’t use pure Python loops
Do use the Woodbury identity
Do think about what the meaning is
Do time everything
Do read the documentation
Do consider approximations
Resources and further reading

Don’t compare floats

Floats are an approximation of real numbers, and comparing them bit-by-bit is too stringent:

np.power(np.sqrt(13), 2) == 13.0  # False
np.isclose(np.power(np.sqrt(13), 2), 13.0)  # True

A = np.random.randn(10, 10)
Q, R = np.linalg.qr(A)
np.all(Q @ R == A)  # False
np.allclose(Q @ R, A)  # True

Use numpy.isclose and numpy.allclose in general, and consider numpy.testing.assert_allclose for testing.

Don’t multiply diagonal matrices

The multiplication $A \operatorname{diag}(\boldsymbol{v})$ is just a column-wise scaling of $A$ by $\boldsymbol{v}$. Similarly, $\operatorname{diag}(\boldsymbol{v}) A$ is a row-wise scaling. Implement the scaling directly instead of multiplying with the diagonal matrix, since using a diagonal matrix mostly does unnecessary work by multiplying the entries of $A$ with zeros.

import numpy as np
import scipy as sp

A = np.random.randn(1000, 500)
v = np.random.randn(500)

A @ np.diag(v)  # 6.3 ms ± 1.01 ms per loop
A * v  # 369 µs ± 35.9 µs per loop

assert np.allclose(A @ np.diag(v), A * v)

Don’t add diagonal matrices

Don’t explicitly form $I$ when computing $A + I$. If $A$ has size $n \times n$, then you’re adding $n$ ones and $n^2 - n$ unnecessary zeroes. Instead use numpy.fill_diagonal and do it like this:

A = np.random.randn(1000, 1000)
I = np.eye(1000)

A + I  # 1.07 ms ± 68.2 µs per loop
np.fill_diagonal(A, A.diagonal() + 1)  # 8.66 μs ± 141 ns per loop

This trick can be used to speed up adding arbitrary diagonal matrixes too.

Don’t multiply matrices in arbitrary order

Matrix multiplication is associative, so the expressions $(AB)\boldsymbol{x}$ and $A(B\boldsymbol{x})$ evaluate to the same value—but the computational cost is not the same. Your can either work out the optimal ordering by hand, or let numpy.linalg.multi_dot do the job for you.

A = np.random.randn(1000, 500)
B = np.random.randn(500, 2000)
x = np.random.randn(2000)

(A @ B) @ x  # 18.8 ms ± 1.11 ms per loop
A @ (B @ x)  # 378 µs ± 114 µs per loop
A @ B @ x  # 19 ms ± 766 µs per loop
np.linalg.multi_dot([A, B, x])  # 378 µs ± 110 µs per loop

Consider another example: computing $A \boldsymbol{v} \boldsymbol{v}^T A$. Here numpy.linalg.multi_dot won’t be of much help, since $\boldsymbol{v}$ and $\boldsymbol{v}^T$ are the same 1 dimensional array in NumPy. Should we implement the computation as $A (\boldsymbol{v} \boldsymbol{v}^T) A$ or $(A \boldsymbol{v}) (\boldsymbol{v}^T A)$?

A = np.random.randn(3000, 3000)
v = np.random.randn(3000)

A @ np.outer(v, v) @ A  # 910 ms ± 61.6 ms per loop
np.outer(A @ v, A.T @ v)  # 41.5 ms ± 3.95 ms per loop

assert np.allclose(A @ np.outer(v, v) @ A, np.outer(A @ v, A.T @ v))

In the first approach we perform one outer product and two matrix multiplications.
In the second approach we perform one outer product and two matrix-vector multiplications.

If $A \in \mathbb{R}^{m \times n}$ and $\boldsymbol{v} \in \mathbb{R}^{n}$, then computing the matrix-vector multiplication $A \boldsymbol{v}$ uses $\mathcal{O}(mn)$ operations. If $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times k}$, then computing the matrix-matrix multiplication $A B$ uses $\mathcal{O}(mnk)$ operations. So a matrix-vector multiplication is $\mathcal{O}(k)$ faster than a matrix-matrix multiplication.

Don’t compute a product if you only need the diagonal

Consider computing $\operatorname{diag}(A B)$. Forming the full product $A B$ only to extract the diagonal is wasteful. Instead, compute the diagonal of the product as (B.T * A).sum(axis=1) in NumPy:

A = np.random.randn(5000, 10000)
B = np.random.randn(10000, 5000)

np.diag(A @ B)  # 3.55 s ± 151 ms per loop
(B.T * A).sum(axis=1)  # 188 ms ± 5.44 ms per loop

assert np.allclose(np.diag(A @ B), (B.T * A).sum(axis=1))

Don’t form the inverse matrix

There are several reasons why you don’t invert that matrix: it’s slow and it’s not numerically stable. Instead of explicitly solving $A\boldsymbol{x} = b$ by forming and multiplying by $A^{-1}$, use a solver.

A = np.random.randn(2000, 2000)
x = np.random.randn(2000)
b = A @ x

np.linalg.inv(A) @ x  # 355 ms ± 52.5 ms per loop
np.linalg.solve(A, b)  # 86.7 ms ± 8.33 ms per loop

Now let’s look at a slightly more advanced computation, which combines the idea of not forming inverses with the idea of not computing products in arbitrary order. We want to compute the product $A B^{-1} C$, where $A \in \mathbb{R}^{m \times n}$, $B \in \mathbb{R}^{n \times n}$ and $C \in \mathbb{R}^{n \times k}$.

If $m > k$, we should compute $A (B^{-1} C)$, which uses $\mathcal{O}( kn(m+n) )$ floating point operations.
If $k > m$, we should compute $(A B^{-1}) C$, which uses $\mathcal{O}( mn(n +k) )$ floating point operations.

Below we solve the case when $k \gg m$ without forming the inverse.

m, n, k = 10, 1000, 100_000

A = np.random.randn(m, n)
B = np.random.randn(n, n)
C = np.random.randn(n, k)

(A @ np.linalg.inv(B)) @ C  # 233 ms ± 56.4 ms per loop
A @ (np.linalg.inv(B) @ C)  # 2.32 s ± 145 ms per loop

np.linalg.solve(B.T, A.T).T @ C  # 157 ms ± 7.29 ms per loop
A @ np.linalg.solve(B, C)  # 2.71 s ± 174 ms per loop

Don’t use memory you don’t need

If you can, use inplace operations to avoid filling up the memory with unnecessary arrays:

v = np.random.randn(100)
v -= 1  # Mutate v without creating any new arrays
v = v - 1  # Create a new array (v - 1) in memory, assign it to v

Inplace operations are also typically a bit faster. Win-win.

Don’t use pure Python loops

Avoid pure Python loops in numerical code. If the obvious approach is to loop, first try to vectorize the computation. If that fails, look into Numba or Cython.

Here’s an example. The code below computes the empirical cross-covariance between two data sets. Each variable is a column, each row is a set of observations.

def covariance_loop(A, B):
    A = A - np.mean(A, axis=0)
    B = B - np.mean(B, axis=0)
    cov = np.zeros((A.shape[1], B.shape[1]))

    for j in range(A.shape[0]):
        cov += np.outer(A[j, :], B[j, :])

    return cov / (A.shape[0] - 1.0)


A = np.random.randn(1000, 1000)
B = np.random.randn(1000, 2000)

covariance_loop(A, B)  # 3.62 s ± 241 ms per loop

The code above computes an outer product plus a matrix-matrix addition. This is done for each row.

Although the asymptotic complexity is the same, the code below is much faster because it eliminates the Python loop and expresses the computation as a single matrix-matrix product instead.

def covariance(A, B):
    A = A - np.mean(A, axis=0)
    B = B - np.mean(B, axis=0)

    return A.T @ B / (A.shape[0] - 1.0)


covariance(A, B)  # 48.3 ms ± 8.75 ms per loop

Do use the Woodbury identity

Learn to use the Woodbury matrix identity and related formulas. The Woodbury equation is:

\begin{equation} \left(A + UCV \right)^{-1} = A^{-1} - A^{-1}U \left(C^{-1} + VA^{-1}U \right)^{-1} VA^{-1} \end{equation}

If $P$ and $R$ are positive definite, then the following holds:

\begin{equation} \left(P^{-1} + B^T R^{-1} B \right)^{-1} B^T R^{-1} = P B^T (B P^T B + R)^{-1} \end{equation}

A typical use case is when we have inverted $A$, and want to compute the inverse of a rank 1 update of $A$. In other words we wish to compute $\left(A + \boldsymbol{v}\boldsymbol{v}^T \right)^{-1}$, and we already have $A^{-1}$. This is a special case of the Woodbury matrix identity, called the Sherman–Morrison formula:

A = np.random.randn(1000, 1000)
v = np.random.randn(1000)
A_inv = np.linalg.inv(A)

np.linalg.inv(A + np.outer(v, v))  # 50.2 ms ± 6.32 ms per loop


def rank1update(A_inv, v):
    return A_inv - np.outer(A_inv @ v, v.T @ A_inv) / (1 + v.T @ A_inv @ v)

rank1update(A_inv, v)  # 8.61 ms ± 2.9 ms per loop

Do think about what the meaning is

Just like $A \operatorname{diag}(\boldsymbol{v})$ simply scales the columns of $A$ with the entries of $\boldsymbol{v}$, more tangled expressions occasionally have simple interpretations. Consider the expression $A(I - \frac{1}{N}\boldsymbol{1}\boldsymbol{1}^T)$. Looks fancy, right? It simply subtracts the mean from each row. Instead of forming $I$, then forming $\boldsymbol{1}\boldsymbol{1}^T$, performing the subtraction, multiplying, etc—simply implement the row scaling directly.

If you’re working in a thousand dimensions, its easy to get lost. Work out simple examples in three or four dimensions before generalizing.

A = np.random.randn(1000, 1000)
factor = np.eye(1000) - np.outer(np.ones(1000), np.ones(1000)) / 1000

A @ factor  # 25 ms ± 9.58 ms per loop
A - np.mean(A, axis=1, keepdims=True)  # 1.61 ms ± 60.4 µs per loop

assert np.allclose(A @ factor, A - np.mean(A, axis=1, keepdims=True))

Here’s another example from the wild. I once saw someone implement

v = np.exp(np.random.randn(5_000))
sp.linalg.cholesky(np.diag(v**2))  # 724 ms ± 99.9 ms per loop

because that’s what a paper said. But the Cholesky decomposition of a diagonal matrix is just the square of the diagonal, so the entire factorization is a waste of time—it does nothing at all.

Do time everything

We’ve been using the ubiquitous %timeit Ipython magic command in almost every example so far. If you’re in doubt about whether one approach or the other is better, then implement both and time them.

Consider solving the Ridge regression optimization problem

\begin{equation} \underset{\boldsymbol{\beta}}{\operatorname{minimize}} \quad \lVert X \boldsymbol{\beta} - \boldsymbol{y} \rVert^2_W + \lVert \boldsymbol{\beta} \rVert^2, \end{equation}

where $\lVert \boldsymbol{x} \rVert^2_W = \boldsymbol{x}^T W \boldsymbol{x}$ denotes the weighted squared norm. We assume that $W$ is a diagonal matrix with entries given by a vector $\boldsymbol{w}$.

We will examine three approaches to solving this problem.

The first approach is to differentiate the objective function and set it equal to zero. We obtain the normal equations $(X^T W X + I) \boldsymbol{\beta} = X^T W \boldsymbol{y}$. Forming the normal equations and solving could look like this:

X = np.random.randn(10_000, 1000)
y = np.random.randn(10_000)
w = np.exp(np.random.randn(10_000))

def solve_normal_eqns(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""
    XT_W = X.T * w  # Form X.T @ diag(w)
    lhs = XT_W @ X
    lhs.flat[:: lhs.shape[0] + 1] += 1  # Add identity
    rhs = XT_W @ y
    return sp.linalg.solve(
        lhs,
        rhs,
        assume_a="pos",
        overwrite_a=True,
        overwrite_b=True,
    )

solve_normal_eqns(X, w, y)  # 321 ms ± 97.2 ms per loop

The second approach is to realize that

\begin{equation} \lVert X \boldsymbol{\beta} - \boldsymbol{y} \rVert^2_W + \lVert \boldsymbol{\beta} \rVert^2 = \lVert W^{1/2} X \boldsymbol{\beta} - W^{1/2} \boldsymbol{y} \rVert^2 + \lVert \boldsymbol{\beta} \rVert^2 = \left\lVert \begin{bmatrix} W^{1/2} X \boldsymbol{\beta} - W^{1/2} \boldsymbol{y} \\ \boldsymbol{\beta} \end{bmatrix} \right\rVert^2 = \left\lVert \begin{bmatrix} W^{1/2} X \\ I \end{bmatrix} \boldsymbol{\beta} - \begin{bmatrix} \boldsymbol{y} \\ \boldsymbol{0} \end{bmatrix} \right\rVert^2 \end{equation}

and make a call to a least squares solver with these matrices directly.

def solve_lstsq(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""
    squared_w = w**0.5
    lhs = np.vstack(((X.T * squared_w).T, np.eye(X.shape[1])))
    rhs = np.hstack((squared_w * y, np.zeros(X.shape[1])))
    beta, *_ = sp.linalg.lstsq(
        lhs,
        rhs,
        overwrite_a=True,
        overwrite_b=True,
    )
    return beta

solve_lstlsq(X, w, y)  # 1.34 s ± 429 ms per loop

The third approach uses the singular value decomposition (SVD). Let $W^{1/2}X = U D V^T$, then the factor $(X^T W X + I)$ in the normal equations may be written as

\begin{equation} (X^T W X + I) = (W^{1/2}X)^T (W^{1/2}X) + I = (U D V^T)^T (U D V^T) + I = VD^2V^T + VV^T = V (D^2 + I) V^T \end{equation}

and $(X^T W X + I)^{-1} = V^T (D^2 + I)^{-1} V$. Plugging this back into the normal equations, we obtain the solution as

\begin{equation} \boldsymbol{\beta} = (X^T W X + I)^{-1} X^T W \boldsymbol{y} = V (D^2 + I)^{-1} D U^T W^{1/2} \boldsymbol{y}. \end{equation}

In the implementation, we exploit the fact that $(D^2 + I)^{-1}$, $D$ and $W^{1/2}$ are diagonal matrices:

def solve_svd(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""
    w_sq = w**0.5
    W_sq_X = (X.T * w_sq).T  # Form W^1/2 @ X
    U, d, VT = sp.linalg.svd(W_sq_X, full_matrices=False, overwrite_a=True)
    inv_diag = d / (1 + d**2)  # Form (D^2 + I)^-1 @ D
    return np.linalg.multi_dot([VT.T * inv_diag, U.T, (w_sq * y)])

solve_svd(X, w, y)  # 2.47 s ± 329 ms per loop

The approaches return the same answer:

assert np.allclose(solve_normal_eqns(X, w, y), solve_lstsq(X, w, y))
assert np.allclose(solve_normal_eqns(X, w, y), solve_svd(X, w, y))

The third approach using the SVD is what you might find in a paper. However, sklearn implements approach one, which is significantly faster. Under the hood, scipy.linalg.solve calls the LAPACK routine sposv, which uses a Cholesky decomposition. It’s typically instructive to implement a few different approaches, then test them and time them. We have not discussed the numerical stability of the least-squares problem here, but have a look at the references at the end of this article for more information about that issue.

Do read the documentation

Consider again the function solve_normal_eqns(X, w, y) that we used to solve the normal equations $(X^T W X + I) \boldsymbol{\beta} = X^T W \boldsymbol{y}$ with. It turns out that scipy.linalg.solve(a, b) has a parameter lower=False, and the documentation reads:

If False (default), the calculation uses only the data in the upper triangle of a; entries below the diagonal are ignored.

It also turns out that there’s a BLAS routine with the cryptic name dsyrk that only computes the upper-triangular part of $\alpha A A^T$.

We can improve the speed of our solver by only computing the upper-triangular part of $X^T W X + I$ as follows:

def solve_normal_eqns2(X, w, y):
    """Solve |X @ beta - y|_w^2 + |beta|^2 for beta."""

    # Compute upper triangular entries in X.T @ diag(w) @ X
    XT_W_X = sp.linalg.blas.dsyrk(alpha=1, a=X.T * w**0.5)

    # Add identity matrix
    XT_W_X.flat[:: XT_W_X.shape[0] + 1] += 1

    return sp.linalg.solve(
        XT_W_X,
        X.T @ (w*y),
        assume_a="pos",
        lower=False, # Solver only uses upper triangular elements
        overwrite_a=True,
        overwrite_b=True,
    )

X = np.random.randn(10_000, 5000)
y = np.random.randn(10_000)
w = np.exp(np.random.randn(10_000))

solve_normal_eqns(X, w, y)  # 4.78 s ± 393 ms per loop
solve_normal_eqns2(X, w, y)  # 3.33 s ± 436 ms per loop

If you’re working with linear algebra, familiarize yourself with all the functions in scipy.linalg. When using a function, read carefully through the arguments. If speed and accuracy truly matters, look into how functions are implemented and research the specific problem you’re working on carefully.

Do consider approximations

Sometimes we don’t need exact answers. For instance, we can approximately solve the Ridge regression problem by running five Conjugate Gradient iterations on the normal equations.

X = np.random.randn(10_000, 1000)
y = np.random.randn(10_000)
w = np.exp(np.random.randn(10_000))
XT_W = X.T * w


def matvec(v):
    # Implements (X.T @ diag(v) @ X + I) @ v
    return np.linalg.multi_dot([XT_W, X, v]) + v


A = sp.sparse.linalg.LinearOperator((X.shape[1], X.shape[1]), matvec=matvec)

# 56.9 ms ± 13 ms per loop
approx, _ = sp.sparse.linalg.cg(A, XT_W @ y, tol=0.0, maxiter=5)

# 295 ms ± 55.1 ms per loop
exact = solve_normal_eqns(X, w, y)

The Pearson correlation between the exact and approximate answer is approximately $0.998$. What we gain in speed might make up for the loss of accuracy, but this is highly problem dependent. An approximate singular value decomposition can be obtained using sklearn.utils.extmath.randomized_svd.

Resources and further reading

Three excellent books:

Numerical Linear Algebra by Trefethen et al (1997).
Matrix Computations by Golub et al (2013).
Accuracy and Stability of Numerical Algorithms by Nicholas J. Higham (2002).

Two reference manuals:

An open-source book on vectorization:

From Python to Numpy

Books

Tommy Odland — Mon, 15 Dec 2025 00:00:00 +0100

Here’s an inspirational, and perhaps true, quote by Stephen Boyd:

When you end up being trained in just a handful of areas, you can be unbelievably effective across fifty fields. If you know linear algebra, optimization, probability and statistics, and computer science – those are the topics. You don’t have to know anything else.

If you know those things, and really know them, and if you’ve seen a bunch of applications across different fields, you are so valuable it’s crazy. You can be hired instantly into a hundred fields. But here’s the even cooler thing, and this is the Silicon Valley part of it: you can create new fields that don’t even exist. Please train yourself broadly. Learn these things.

Below is a reading list for aspiring data scientists and others interested in mathematical modeling, statistics, optimization, scientific programming, etc.

Only my favorite books are included. Books I would recommend.
Only general topics useful to many practitioners are included. Category theory, game theory and geostatistics is cool – but not that useful to the average practitioner.
I have read the majority of the material, but not everything.
The resources are sorted by difficulty within reach category, but only roughly.

The nomenclature is:

📗 Book.
📄 Paper.
📺 Video lecture(s).
🤖 Book contains many code examples.
⭐ Life changing.

Mathematics

📗 Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares by Boyd et al (2018).
📗 Advanced Engineering Mathematics by Michael Greenberg (1998).
📗 ⭐ Introduction to Applied Mathematics by Gilbert Strang (1986).
📗 The Princeton Companion to Mathematics by Gowers et al (2008).
📄 An Introduction to the Conjugate Gradient Method Without the Agonizing Pain by Jonathan Richard Shewchuk (1994).

Scientific computing

📗 Numerical Methods for Scientists and Engineers by R. W. Hamming (1987).
📗 🤖 From Python to Numpy by Nicolas P. Rougier (2017).
📗 Numerical Linear Algebra by Trefethen et al (1997).
📗 Matrix Computations by Golub et al (2013).

Python and programming

📗 The Pragmatic Programmer by Hunt et al (2019).
📗 🤖 Fluent Python: Clear, Concise, and Effective Programming by Luciano Ramalho (2015).
📗 🤖 Effective Python: 90 Specific Ways to Write Better Python by Brett Slatkin (2019).
📺 So you want to be a Python expert? by James Powell (2017).
📺 Transforming Code into Beautiful, Idiomatic Python by Raymond Hettinger (2013).

Machine learning

📗 Artificial Intelligence: A Modern Approach by Norvig et al (2009).
📗 Pattern Classification by Duda et al (2000).
📗 Pattern Recognition and Machine Learning by Christopher M. Bishop (2006).
📗 The Elements of Statistical Learning by Hastie et al (2016).
📄 A Few Useful Things to Know about Machine Learning by Pedro Domingos (2012).
📄 Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead by Cynthia Rudin (2018).
📄 Machine Learning: The High Interest Credit Card of Technical Debt by Sculley et al (2014).

Deep learning etc.

📗 🤖 Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron (2022).
📗 Reinforcement Learning by Sutton (2018).
📺 DeepMind x UCL | Reinforcement Learning Course by David Silver (2015).

Statistics

📗 Statistical Inference by Casella et al (2001).
📗 Regression and Other Stories by Gelman et al (2020).
📗 Data Analysis Using Regression and Hierarchical Models by Gelman et al (2006).
📗 All of Statistics by Larry Wasserman (2010).
📗 🤖 ⭐ Statistical Rethinking by Richard McElreath (2020).
📺 Statistical Rethinking by Richard McElreath (2022).
📄 The garden of forking paths by Gelman et al (2019).
📄 Statistical Modeling: The Two Cultures by Leo Breiman (2001).
📄 Hierarchical generalized additive models in ecology by Pedersen et al (2019).
📄 What are the most important statistical ideas of the past 50 years? by Gelman et al (2021).
📄 A Conceptual Introduction to Hamiltonian Monte Carlo by Michael Betancourt (2018).

Optimization

📗 The Basics of Practical Optimization by Adam B. Levy (2022).
📗 Optimization Modelling: A Practical Approach by Sarker et al (2007).
📗 Essentials of Metaheuristics by Sean Luke (2012).
📗 Linear Programming: Foundations and Extensions by Robert J. Vanderbei (2021).
📗 Convex Optimization by Boyd et al (2004).
📗 Numerical Optimization by Nocedal et al (2006).
📄 Formulating Integer Linear Programs: A Rogues’ Gallery by Brown et al (2007).
📄 Bus Routing Optimization Helps Boston Public Schools Design Better Policies by Bertsimas et al (2020).

Algorithms

📗 The Algorithm Design Manual by Steven S. Skiena (2021).
📗 ⭐ Algorithms by Dasgupta et al (2006).
📗 🤖 Competitive Programming by Halim et al.
📗 Introduction to Algorithms by Cormen et al (2022).
📗 Network Flows: Theory, Algorithms, and Applications by Ahuja et al (1993).

Popular science

📗 Flaws and Fallacies in Statistical Thinking by Stephen K. Campbell (2012).
📗 Innumeracy: Mathematical Illiteracy and Its Consequences by John Allen Paulos (2001).
📗 Science Fictions by Stuart Ritchie (2020).
📗 Concepts of Modern Mathematics by Ian Stewart (1995).
📗 Euler: The Master of Us All by William Dunham (1999).
📗 The Book of Why by Judea Pearl et al (2018).

Miscellaneous

📗 The Visual Display of Quantitative Information by Edward R. Tufte (2001).
📗 On Writing Well: The Classic Guide to Writing Nonfiction by William Zinsser (2016).
📗 Handbook of Writing for the Mathematical Sciences by Nicholas J. Higham (2019).
📗 How to Solve It by Polya et al (2014).
📗 The Matrix Cookbook by Pedersen et al (2012).
📗 Opt Art: From Mathematical Optimization to Visual Design by Robert Bosch (2019).
📄 School Choice: A Mechanism Design Approach by Abdulkadiroğlu et al (2003).

Fastlegebytter

Tommy Odland — Wed, 05 Nov 2025 00:00:00 +0100

Oppdatering: To dager etter at jeg skrev denne artikkelen ble jeg informert om innlegget “Enkelt å forbedre fastlegeordningen - med en algoritme” i Dagens Næringsliv. Forfatterne argumenterer for mye av det samme som det jeg gjør nedenfor. Innlegget deres er veldig godt skrevet, og de har også skrevet en 80-siders forskningsartikkel “Designing Dynamic Reassignment Mechanisms: Evidence from GP Allocation”. De beregner at 15 % (ca. 45 000 mennesker) kunne sluppet venteliste ved å endre på algoritmen!

Det er ikke uvanlig å vente i årevis på fastlegebytte.

I 2024 kunne vi lese at over 300 000 står på venteliste. På helsenorge.no er det 608 fastleger registrert i Oslo fylke, og 60 % av disse har fulle lister. En lege har typisk plass til 1 300 pasienter, og det er 816 000 plasser i Oslo fylke totalt. Kun 4,5 % av disse plassene er ledige. Situasjonen er enda verre andre steder i landet. I 2022 var det ille i Rogaland, der kun 3 % av fastlegene hadde ledige plasser. I Bergen og Trondheim var det ikke en eneste ledig plass.

Et stort problem med fulle lister er at dagens ordning ikke tillater bytter mellom pasienter på fulle lister. Dette er ineffektivt fordi det låser markedet. I denne artikkelen skal jeg forklare dette utsagnet og foreslå en løsning på problemet.

Bytter mellom fulle lister

Her er et eksempel på en situasjon der vi kunne kuttet ventetiden drastisk. Ola er på lista til lege A. Kari er på lista til lege B. Ola ønsker å bytte fra lege A til lege B. Kari ønsker å bytte motsatt vei; fra lege B til lege A.

I dag må noen forlate listen til én av legene før Ola, Kari eller Per får innvilget søknad om fastlegebytte. Men Ola og Kari hadde ikke trengt å vente (potensielt i årevis) - de kunne ha byttet plass i dag! Vi trenger bare en algoritme som leter etter sykluser i bytte-grafen og gjennomfører byttene.

Er du bekymret for Per i figuren ovenfor? Den nye algoritmen er bedre for både Ola og Kari, men Per har potensielt stått i kø lengre enn Ola. Kan Per havne i en situasjon der han må vente lengre enn med dagens algoritme? Svaret er ja - det er mulig å konstruere et slikt eksempel:

Anta at pasientene melder ønske om bytte i rekkefølgen Kari, Per og Ola. Vi står i en fastlåst situasjon som vist i figuren ovenfor, der ingen får bytte fordi det ikke er ledige plasser. Om noen dager vil lege A få en ledig plass, men det vet vi ikke enda.

Dagens algoritme. Dersom vi ikke kjører syklus-algoritmen er situasjonen fastlåst frem til lege A får en ledig plass. Da får Kari sitt ønske innvilget ettefulgt av Per, mens Ola står fremdeles i kø.
Foreslått algoritme. Dersom vi kjører syklus-algoritmen bytter Ola og Kari umiddelbart plass. Når lege A får en ledig plass påvirker det ikke Per. Han står fremdeles i kø.

Per må vente til lege B får en ledig plass som ikke umiddelbart fylles fordi den er del av en bytte-syklus. Oppsummert kan det å gjennomføre sykluser-bytter mellom fulle lister fjerne (potensielt lange) stier i grafen som ville innfridd flere ønsker.

Er dette eksempelet et godt argument mot den foreslåtten algoritmen? Jeg mener at det ikke er det. Per er kanskje misfornøyd dersom virkeligheten utspiller seg som ovenfor, men langt flere er misfornøyde med dagens regime. Realiteten er at ingen på venteliste vet om de er en del av en syklus (slik som Ola og Kari) eller ikke (slik som Per), og mange bytter ødelegger ikke for personer som Per. Markedet er i dag helt låst i byer som Bergen, der kun 3 av 274 fastleger har ledige plasser. I slike situasjoner ville syklus-bytter betydd mye for pasientene.

Lengre sykluser

Ovenfor byttet vi plass på to personer: Ola og Kari. Vi kan bytte plass på et vilkårlig antall personer så lenge de lager en syklus i bytte-grafen.

Av og til må vi ta et valg, fordi én persons bytteønske kan brukes i flere sykluser, slik som i figuren nedenfor.

I slike situasjoner kan vi enten velge den lengste syklusen (grønn), eller vi kan velge den syklusen der pasientene har ventet lengst på å få bytte. Det er et verdispørsmål, ikke et matematisk spørsmål - og uansett hva vi velger blir utfallet bedre enn i dag for de aller fleste.

Et annet valg vi må ta er om vi utfører syklus-bytter ofte (f.eks. hver dag) eller sjeldent (f.eks. hver måned). Å bytte ofte gir umiddelbar bevegelse i markedet. Å bytte litt mer sjeldent kan føre til at det bygger seg opp lengre syklusen i grafen som vi kan utnytte. En siste idé er å tillate mer enn ett ønske: hvorfor ikke la pasientene rangere flere leger som de kunne tenke seg å bytte til?

Oppsummering og referanser

På tross av målsetninger om å digitalisere og effektivisere helsesektoren, er det fremdeles noen frukter som henger så lavt at de nesten tar i bakken. Syklus-bytter hjelper ikke de som er villige til å stå i kø i fem år for å være pastient hos den beste legen i byen. Men det hjelper når Ola flytter fra en bydel til en annen og vil bytte lege, mens Kari flytter motsatt vei. Algoritmer som baserer seg på å fjerne sykluser har vært kjent siden 70-tallet, da Top Trading Cycle (TTC) først ble publisert. Selv om TTC er Pareto-effektiv, gir den ikke nødvendigvis de lengste syklusene. Optimal kidney exchange handler om å finne de lengste syklusene.

Boka Who Gets What - and Why: The New Economics of Matchmaking and Market Design av Alvin Roth er en folkelig introduksjon til markeder uten penger og uten delelige varer. Fagfeltet heter mekanismedesign, og er godt kjent gjennom Wikipedia-artikler, bøker og en Nobelpris til Roth i 2012. På tross av dette er denne kompetansen tilsynelatende lav, både i offentlig og privat sektor. Dette skrev jeg også om i artikkelen “Mangelfull evaluering av inntaksmodeller”, der jeg argumenterte mot å bruke politikernes hjemmesnekra modeller til å avgjøre skjebnen til tusenvis at VGS-elever.

De fleste overvurderer betydningen av sitt eget fagfelt. Jeg er neppe noe unntak, fordi jeg tror matematikk kan spille en rolle i en effektiv helsesektor. PS: Les gjerne artikkelen min om turnusplaner for sykepleiere!

Korrespondanse med Norsk helsenett SF

Jeg kontaktet Norsk helsenett SF (Helsenorge) om problemstillingen. Ut fra svaret tolker jeg at denne artikkelen fremstiller dagens system riktig.

For meg virker det som en åpenbar forbedring å tillate bytter, og jeg vet ikke hvorfor Helsenorge ikke gjør det. Det kan hende de har gode grunner til å beholde dagens system, eller at de har andre prioriteringer, eller at de har IT-systemer som er vanskelige å endre, eller mangler kompetanse til å implementere en slik algoritme.

Her er korrespondansen:

Hei,

Dersom to (eller flere) personer innbyrdes ønsker å bytte fastlege, 
skjer byttet umiddelbart?

Eksempel: Ola har A som fastlege. Kari har B som fastlege. 
Både A og B har helt fulle lister. Ola ønsker å bytte fra A til B. 
Kari ønsker å bytte fra B til A. Ingen andre som står til kø til verken 
A eller B må vente lengre dersom Ola og Kari får bytte umiddelbart. 
Er algoritmen smart nok til å gjøre slike bytter, eller må man vente
til en på en ledig plass fordi noen først forlater listene?

// Tommy

--------------------------------

Hei,

Det er dessverre ikke mulig å bytte plass med noen. 
Dersom ønsket lege ikke har ledig plass, må man stå på venteliste.

Vennlig hilsen
Veiledning Helsenorge

Monte Carlo modeling in Python with probabilit

Tommy Odland — Fri, 17 Oct 2025 00:00:00 +0200

probabilit is a Python package for Monte Carlo modeling that I helped create. It’s great for prototyping uncertainty calculations in some types of problems, such as those encountered in undergraduate engineering courses. In addition to a high-level modeling language, it contains algorithms for inducing correlations between variables, as well as computing the nearest correlation matrix.

Here is what probabilit can do:

It has a high-level modeling language that lets us model equations and expressions using probability distributions
It supports quasi Monte Carlo sampling: Latin Hypercube, Sobol, and Halton sequences
It can correlate variables, e.g. induce a correlation between a uniform and a normal distribution

Here is what probabilit does not do:

It’s not suited for modeling complex environments, like multi-agent systems, queues, differential equations, etc.
It’s not as memory efficient nor as fast as writing pure numpy simulations

This article is a tutorial demonstrating the most important features of probabilit. We’ll be using version 0.1.0. Beware that in future versions some things might change slightly. However, the overall ideas will remain the same, and all the Python code in this tutorial is executable (it was created from a jupyter notebook).

In [1]:

import numpy as np
import matplotlib.pyplot as plt

Monte Carlo modeling examples¶

Let us jump into probabilit with some modeling examples.

Braking distance of a car¶

Problem. Suppose we’re driving down the highway at approximately 100 km/h when suddenly a family of ducks appear and we have to slam the brakes. What is the braking distance?

Deterministic answer. It turns out that the relevant equation, obtained by realizing that the braking energy must cancel out the kinetic energy, is d = v^2 / (2 * mu * g), where

d = braking distance (m)
v = velocity (m/s)
mu = coefficient of friction (dimensionless)
g = gravitational acceleration (m/s²)

A deterministic answer is straightforward to compute in Python:

In [2]:

def braking_distance(velocity_kmh, mu, g=9.81):
    velocity_ms = velocity_kmh / 3.6  # Convert to m/s
    return velocity_ms**2 / (2 * mu * g)


velocity_kmh = 100
mu_dry = 0.7  # Coefficient of friction on dry road
distance_dry = braking_distance(velocity_kmh, mu_dry)
print(f"{distance_dry:.1f} meters")

56.2 meters

Probabilistic answer.

With probabilit we can compute braking distance as a distribution. More specifically we’ll be able to draw samples from the braking distance distribution. All we have to do is allow some of the input parameters to be distributions:

We do not trust the speedometer to be exact, so we set velocity to be a triangular distribution.
We do not trust that the coefficient of kinetic friction is exactly 0.7, so we use a normal distribution.

These distributions are somewhat arbitrary. Any distribution in scipy can be used from within probabilit, with the syntax
Distribution("scipy_distr_name", scipy_distr_arg1, scipy_distr_arg2, ...).

In [3]:

from probabilit import Distribution

# Replace velocity and coefficient of friction with distributions
velocity_kmh = Distribution("triang", loc=95, scale=10, c=0.5)
mu_dry = Distribution("norm", loc=0.7, scale=0.03)

# We can use the same function as earlier
distance_dry = braking_distance(velocity_kmh, mu_dry)

# Sample from the answer. This samples the distributions above
# and propagates the samples through the expression defined by the
# 'braking_distance' function. "lhs" = latin hypercube sampling
samples = distance_dry.sample(2500, random_state=42, method="lhs")

plt.figure(figsize=(6, 2.5))
plt.hist(samples, bins="auto", density=True, edgecolor="black")
plt.xlabel("Braking distance")
plt.tight_layout()
plt.show()

In [4]:

import pandas as pd
# We can also get summary statistics
(
    pd.Series(samples, name="braking_distance")
    .describe(percentiles=[0.01, 0.05, 0.95, 0.99])
    .to_frame()
    .T.round(1)
)

Out[4]:

	count	mean	std	min	1%	5%	50%	95%	99%	max
braking_distance	2500.0	56.3	3.3	47.1	49.3	50.9	56.2	62.0	64.3	72.3

With uncertainty added to some of the input parameters, the answer is not nearly as certain as it seemed in the deterministic analysis. We have learned that braking distance could reasonably be anywhere between 51 meters and 62 meters or so. This is the power of (simple) Monte Carlo analysis.

When distance_dry is sampled, the ancestor nodes velocity_kmh and mu_dry are also sampled. In the process, the attribute samples_ is set on all nodes. This is useful if we want to inspect the samples drawn on ancestor variables. For instance, here is a simple sensitivity analysis where we study the effect of the input parameters on the output:

In [5]:

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(6, 3), sharey=True)

ax1.scatter(velocity_kmh.samples_, distance_dry.samples_, s=3, zorder=5)
ax2.scatter(mu_dry.samples_, distance_dry.samples_, s=3, zorder=5)

ax1.set_ylabel("Distance")
ax1.set_xlabel("Velocity")
ax2.set_xlabel("Coefficient of friction")
ax1.grid(True, ls="--", zorder=0, alpha=0.5)
ax2.grid(True, ls="--", zorder=0, alpha=0.5)
fig.tight_layout()
plt.show()

Mutual fund¶

Problem. Suppose we save 100 units of money per month for 10 years and put the money in a mutual fund. The yearly interest rate is around 7 % after accounting for inflation. How much money will we have after 10 years?

Probabilistic solution. The interest rate will not be exactly 7 % every year, so we can add some uncertainty. Notice how probabilit expressions can be used in for-loops without any issues.

In [6]:

from probabilit import Lognormal, plot

YEARS = 10
SAVED_PER_MONTH = 100

money_saved = 0
for year in range(YEARS):
    interest_rate = Lognormal(mean=1.07, std=0.05)
    money_saved = money_saved * interest_rate + SAVED_PER_MONTH * 12

# Sample and plot the result
samples = money_saved.sample(999, random_state=42, method="lhs")
pairgrid = plot(money_saved)  # returns a seaborn pairgrid
pairgrid.axes[0, 0].set_xlabel("Money saved")
pairgrid.axes[0, 0].set_ylabel("")
plt.show()

Exercise. Change the example above so that money saved per month is a distribution. Choose values that are realistic for your own financial situation and study how much money you can save over the next decade.

Bird survival¶

This final example illustrates compound distributions, where one distribution is an argument to another.

Problem. Suppose we have a distribution of eggs per nest for a certain species of bird. We also have a survival percentage for each egg. What is the distribution of the total number of birds per nest that live till adulthood?

Probabilistic solution. We set up separate distributions and pass them as arguments to each other:

In [7]:

# Number of eggs per nest
eggs_per_nest = Distribution("poisson", mu=2)

# Assume we are uncertain of the survival probability too
# According to one source online, around 10-30% of eggs survive
prob_survive = Distribution("beta", a=2, b=2, loc=0.1, scale=0.2)

# Here is the compound distribution, which depends on two distributions
adult_birds_per_nest = Distribution("binom", n=eggs_per_nest, p=prob_survive)

# Calling plot() without sampling will sample a copy of the nodes
plot(
    eggs_per_nest,
    prob_survive,
    adult_birds_per_nest,
    sample_kwargs={"random_state": 42, "method": "lhs"},
    height=1.5,
    aspect=1.2,
);

The discrete nature of the distributions make them hard to visualize without jitter in the default plot call, but hopefully you get the idea.

Exercise. Go to Wikipedia’s examples of compound distributions. Verify computationally that this statement is true: “Compounding a Gaussian distribution with variance distributed according to an exponential distribution yields a Laplace distribution.”

How does probabilit work?¶

Given an expression, probabilit creates a computational graph that is evaluated lazily. For instance, the expression Constant(5) ** 2 is converted to Power(Constant(5), Constant(2)), where Power is a Transform node. Other transform nodes are Exp, Log, Sin, Abs, Add, Divide, etc. There are also Constant nodes and Distribution nodes.

In [8]:

from probabilit import Constant

expression = Constant(5) ** 2 + Distribution("uniform") / Constant(3)
print(expression)

Add(Power(Constant(5), Constant(2)), Divide(Distribution("uniform"), Constant(3)))

The computational graph can be visualized as a tree:

In [9]:

from probabilit import treeprint
treeprint(expression)

Add
   ├──Power
   │  ├──Constant(5)
   │  └──Constant(2)
   └──Divide
      ├──Distribution("uniform")
      └──Constant(3)

Not every probabilit expression is a tree. For instance, if a is a node then b = abs(a) + a creates a graph that is not strictly speaking a tree. All probabilit graphs are DAGs (directed acyclic graphs), since they cannot contain cycles.

A probabilit expression can be converted to a networkx MultiDiGraph.

In [10]:

import networkx as nx

# Create a MultiDiGraph
G = expression.to_graph()

# Draw information
pos = nx.spring_layout(G, seed=0)
labels = {node: type(node).__name__ for node in G.nodes()}

# Draw the graph
plt.figure(figsize=(6, 3))
nx.draw(
    G,
    pos,
    with_labels=True,
    node_color="lightblue",
    node_size=500,
    font_size=6,
    font_weight="bold",
    arrows=True,
    arrowsize=12,
    edge_color="gray",
    labels=labels,
)
plt.show()

It’s important to understand that no computation is performed until .sample() is called on a node. The expression above is just a graph with instructions about how to compute a result. Once .sample() is called, the actual work is done:

In [11]:

expression.sample(10, random_state=42)

Out[11]:

array([25.12484671, 25.31690477, 25.24399798, 25.19955283, 25.05200621,
       25.05199817, 25.0193612 , 25.28872538, 25.20037167, 25.23602419])

When the call asks for the sink node expression to be sampled, all ancestors are sampled in turn and results are propagated through the graph. This has both memory and performance overhead, which probabilit sacrifices in favor of a clean API and a pleasant modeling experience.

Latin HyperCube sampling (LHS)¶

Probabilit supports all major sampling methods in the scipy quasi-monte carlo module.

Below we show the difference between pseudo-random sampling (method=None) and Latin HyperCube (method="lhs").

In [12]:

uniform = Distribution("uniform")
exponential = Distribution("expon")

# Sample a sum to set `.samples_` on distributions for plotting
(uniform + exponential).sample(500, random_state=42, method=None)
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5});

In [13]:

(uniform + exponential).sample(500, random_state=42, method="lhs")
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5});

Notice how the low-discrepancy sequence produced by LHS leads to distributions of samples that looks very much like the theoretical distributions. In many Monte Carlo simulations, using LHS is an excellent choice because we obtain empirical distributions that match the theoretical ones with fewer samples drawn than what we would achieve had we used pseudo-random sequences.

Inducing correlations¶

Probabilit comes with Correlator objects that can be used to correlate variables. They work within the framework of the modeling API, or outside of it (where numpy arrays can be used). This means that these highly performant algorithms can be used outside of the modeling language too.

When all variables are Normal the Cholesky correlator can be a good choice. While it obtains the desired correlation, it does not preserve the marginal distributions. This is a major problem many real-world applications, so be careful when using the Cholesky correlator! Notice in the example below that the exponential distribution contains negative values after inducing correlations.

In [14]:

from probabilit.correlation import Cholesky

# Correlation matrix with desired Pearson correlation of 0.8
corr_mat = np.array([[1.0, 0.8], [0.8, 1.0]])

expression = (uniform + exponential).correlate(uniform, exponential, corr_mat=corr_mat)

expression.sample(500, random_state=42, method="lhs", correlator=Cholesky())
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5})

# Observed Pearson correlation
np.corrcoef(uniform.samples_, exponential.samples_)

Out[14]:

array([[1. , 0.8],
       [0.8, 1. ]])

Above, the marginal of the uniform distribution was preserved. This is a happy accident since we defined that distribution before we defined the exponential distribution in the code. If we switch them, then we will preserve the marginals of the exponential but destroy the marginal of the uniform. In general the Cholesky comes with no guarantees of preserving marginals.

Exercise. Switch the order and define the exponential distribution first, then the uniform distribution. Use the Cholesky correlator and notice how the marginal of the uniform is destroyed.

Iman-Conover¶

The Iman-Conover transformation is almost always a better choice than Cholesky. It leaves the marginal distributions unchanged, while attempting to induce the desired correlation. Iman-Conover is essentially a heuristic where data is transformed to rank-space, then Cholesky is used, then the data is transformed back. The end result is a permutation of the samples in each distribution. The idea is that by choosing a permutation that induces correlation in rank space, the permutation will also get close to the desired correlation in the original space. This typically works pretty well in most cases, but the algorithm comes without any optimality guarantees.

In [15]:

from probabilit.correlation import ImanConover

expression = (uniform + exponential)
expression.correlate(uniform, exponential, corr_mat=corr_mat)

expression.sample(500, random_state=42, method="lhs", correlator=ImanConover())
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5})

np.corrcoef(uniform.samples_, exponential.samples_)

Out[15]:

array([[1.        , 0.70443783],
       [0.70443783, 1.        ]])

Above we do not achieve the exact correlation that we wanted, but we get pretty close. The marginals are unaffected by the transformation.

Inducing correlations by permutation¶

Since Iman-Conover is just a clever and fast heuristic to pick a good permutation of the samples (rows) within a variable (column) in a matrix, perhaps we can do even better if we search the space of all permutations? This space is huge, so we can only afford to search a small part of it. The Permutation correlation does this, and by spending some time on computation we often end up with even better correlations than what Iman-Conover achieves. Notice below that we get extremely close to the desired correlation.

In [16]:

from probabilit.correlation import Permutation

expression = (uniform + exponential).correlate(uniform, exponential, corr_mat=corr_mat)

correlator = Permutation(iterations=5_000, random_state=42)
expression.sample(500, random_state=42, method="lhs", correlator=correlator)
plot(uniform, exponential, height=1.5, aspect=1.2, plot_kws={"s": 5})

np.corrcoef(uniform.samples_, exponential.samples_)

Out[16]:

array([[1.        , 0.78002787],
       [0.78002787, 1.        ]])

Finally, let us compare ImanConover with the Composite algorithm. The composite approach is what probabilit does by default. It first uses Iman-Conover to find a good starting point, then proceeds by running Permutation on that starting point for a number of iterations.

In [17]:

variables = [Distribution("lognorm", s=1) for _ in range(10)]

# Set up correlation matrix
corr_mat = np.ones((10, 10)) * 0.8
np.fill_diagonal(corr_mat, 1.0)

expression = sum(variables).correlate(*variables, corr_mat=corr_mat)

# Iman-Conover
correlator = ImanConover()
expression.sample(999, random_state=42, method="lhs", correlator=correlator)
observed_corr = np.corrcoef(np.array([v.samples_ for v in variables]))
RMSE = np.sqrt(np.mean((corr_mat - observed_corr) ** 2))
print(f"RMSE (observed-desired) for Iman-Conover: {RMSE:.4f}")

RMSE (observed-desired) for Iman-Conover: 0.1011

In [18]:

from probabilit.correlation import Composite

# Composite() => ImanConover(), then Permutation()
correlator = Composite(random_state=0, iterations=5_000, tol=0.01)
expression.sample(999, random_state=42, method="lhs", correlator=correlator)
observed_corr = np.corrcoef(np.array([v.samples_ for v in variables]))
RMSE = np.sqrt(np.mean((corr_mat - observed_corr) ** 2))
print(f"RMSE norm (observed-desired) for Composite: {RMSE:.4f}")

RMSE norm (observed-desired) for Composite: 0.0367

If we are willing to spend some additional time on computation (here 2.5 seconds instead of 15 milliseconds), then we can obtain correlations that are much closer to what we desired. Often this is worth it, as speed is rarely a bottleneck in these types of analyses.

Nearest correlation matrix¶

Just like the ImanConover and Permutation correlator classes can be used without using the modeling API, the nearest_correlation_matrix can also be used without using the modeling API. This algorithm is useful when users (e.g. a domain expert) proposes correlations, but they end up proposing a correlation matrix that is not valid. This happens quite frequently, since specifying a high-dimensional, non-trivial correlation matrix that is positive definite is hard.

In [19]:

from probabilit.correlation import nearest_correlation_matrix

rng = np.random.default_rng(0)

n = 5
proposed_corr = rng.uniform(low=-1, high=1, size=(n, n))

# Make it symmetric
proposed_corr = np.triu(proposed_corr) + np.triu(proposed_corr, 1).T
np.fill_diagonal(proposed_corr, 1.0)

(pd.DataFrame(proposed_corr) * 100).round(1)

Out[19]:

	0	1	2	3	4
0	100.0	-46.0	-91.8	-96.7	62.7
1	-46.0	100.0	45.9	8.7	87.0
2	-91.8	45.9	100.0	-93.3	45.9
3	-96.7	8.7	-93.3	100.0	-15.5
4	62.7	87.0	45.9	-15.5	100.0

In [20]:

np.linalg.eigvals(proposed_corr)  # Negative eigenvalues => not positive definite

Out[20]:

array([-1.05418258, -0.00675076,  1.41936444,  2.20321725,  2.43835165])

In [21]:

nearest_corr = nearest_correlation_matrix(proposed_corr)
(pd.DataFrame(nearest_corr) * 100).round(1)

Out[21]:

	0	1	2	3	4
0	100.0	-35.0	-42.5	-51.7	35.2
1	-35.0	100.0	49.9	11.4	75.0
2	-42.5	49.9	100.0	-51.0	26.8
3	-51.7	11.4	-51.0	100.0	-32.1
4	35.2	75.0	26.8	-32.1	100.0

Summary and references¶

probabilit has two layers:

A high-level modeling language, built on top of Python
Low-level functionality for correlating variables, finding nearest correlation matrices, etc

Depending on what kind of Monte Carlo analysis you want to do, one or both (or none!) of these might be useful.

Probabilit is a bit like Tensorflow in that it uses a lazily evaluated computational graph. The idea of building a Monte Carlo analysis tool in this fashion is due to Feda. Integrating LHS and ImanConover into the tool was also his idea. Knut provided feedback on practical methods and distributions used in some types of Monte Carlo analysis, e.g. the importance of Triangular and PERT distributions. The idea behind the Permutation correlation and how to speed it up by not re-calculating the correlation matrix in every iteration (an idea we did not have time to go into in this tutorial) is my own, but these are not novel ideas.

To read more on these subjects, I recommend:

The probabilit repository. Have a look at the implementations. Create an issue or PR if you find a bug :)
To find the nearest correlation matrix, we use Equation (3) in the paper An augmented Lagrangian dual approach for the H-weighted nearest correlation matrix problem
An excellent explanation of the Iman Conover algorithm is found in The geometry of the Iman-Conover transformation
The book Global Sensitivity Analysis by Saltelli et al. is not directly related to Monte Carlo modeling, but Monte Carlo is often used for sensitivity analysis of models, and therefore this book might be of interest

A Recipe for Curve Fitting

Tommy Odland — Sat, 04 Oct 2025 00:00:00 +0200

Inspired by A Recipe for Training Neural Networks, here are some tips and tricks for curve fitting. Assume we are fitting a non-linear, parametric function $f(x; \boldsymbol{\theta})$ to data $y$. The problem could be anything. Many highly technical problems are illuminated by curve fitting, but it helps to think of everyday examples such as: personal savings vs. time, apartment price vs. square meters, salary vs. experience, strength vs. bodyweight, world population vs. year, etc.

Modeling

Determine the purpose. Do you want to extrapolate, or do you want to interpolate? Is prediction the overall goal, or is it inference of latent variables? In other words: raw predictive power or understanding underlying phenomena? Is uncertainty needed? If yes, then what kind? A posterior conditional distribution $p(x | \hat{\boldsymbol{\theta}})$ or a posterior predictive distribution $p(x | \boldsymbol{\theta}) p(\boldsymbol{\theta} | X)$ that accounts for parameter uncertainty? Is curve fitting a one-time job, or will it every night or on demand? Will it run on a single data set, or perhaps on thousands of datasets?
Read the literature. Before deciding on the function $f$, search the web and review papers. What functions are used on similar problems? Avoid ad-hoc models like unregularized high-degree polynomials or very intricate functions. Apply Occam’s razor.
Know basic functions and how to combine them. Know the behavior of basic building blocks like linear functions, powers, exponentials, logarithms, sigmoids, hyperbolic functions, etc. Sometimes you can transition between them in more natural ways than the convex combination $\alpha f(x) + (1- \alpha) g(x)$. For instance, the power transform $(x^\alpha - 1) / \alpha$ lies between the affine function $x-1$ when $\alpha=1$ and the logarithmic function $\ln x$ when $\alpha \to 0$. Similarly, the generalized pareto distribution lies “between” the exponential $e^{-x}$ and the hyperbola $(1 + x)^{-1}$. Probability density functions and their antiderivatives are a rich source of modeling functions. Know how to combine functions to create new models.
Use an identifiable model. Avoid using non-identifiable models like $f(x; \boldsymbol{\theta}) = \theta_1 \exp(x - \theta_2)$, since they do not have a unique solution. This seems obvious, but I’ve seen papers where non-identifiable models are used without priors.
Choose an appropriate loss. The least squares loss $\mathcal{L}(\boldsymbol{\theta}) = \sum_i (y_i - f(x_i; \boldsymbol{\theta}))^2$ has nice properties, but might not be what you want. Know when to use $p$-norms, huber loss, pseudohuber, geometric loss, harmonic loss and statistical losses. Remember that fitting log-data to log-model with a loss like $(\log y_i - \log f(x_i; \boldsymbol{\theta}))^2$ is not equivalent to fitting data to a model without logs. Determining the appropriate loss function is a big part of the job. Always align the primary metric with the loss. If you care about MAE as the metric, it makes little sense to use RMSE as the loss.
Use a prior, or if the model is not statistical, a regularization term. A prior encodes prior knowledge and helps the solver numerically. It also lets you get a solution when you have little data or no data at all. The 2-norm $\lVert \boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}} \rVert_2$ is a good start and $(\boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}})^T A (\boldsymbol{\theta} - \boldsymbol{\mu}_{\boldsymbol{\theta}})$ can be even better. Often $A$ is diagonal.
Deal with parameter constraints. There are several options: (1) use an optimizer that handles constraints natively, (2) re-parametrize the function as $f(x; t(\boldsymbol{\theta}))$ with $t$ mapping from unbounded real numbers to the desired bounds or (3) add a barrier function to the objective. For instance, if $\theta_1 \in [0, 1]$, then you can (1) use an algorithm like BFGS-B (the B stands for “box constraints”), (2) transform with e.g. a sigmoid like $\theta_1 = t(z) = (1 + e^{-z})^{-1}$ or (3) add barriers such as $-\log(\theta_1) - \log (1 - \theta_1 )$ to the objective. The log-barrier is equivalent to a beta-prior.
Weight the data. Consider a weighted loss like $\sum_i w_i (y_i - f(x_i; \boldsymbol{\theta}))^2$ if (1) some data points are more important than others or (2) you are working on a temporal forecasting problem. Beware of the difference between (a) inverse variance weights and (b) weighting the log-likelihood. They are the same when estimating the mean of a normal, but not in general (see also this post).
Consider partial pooling. Use hierarchical models to automatically regularize with hyperpriors. This is relevant if you have many curve fitting instances that you can conceptualize as being drawn from some population, and you want to learn the distribution of that population. It’s also extremely useful if some instances have little data.

Validation

Cross validate. Split randomly if you’re interpolating. Use a time based split if you’re forecasting. Always match cross validation to how the model will be used. Compute metrics and study them.
Use your eyes. Curve fitting can be visualized! Sample prior curves and plot them, plot the initial guess, curve fit “by hand” a few times, fit on every instance you have in your database. Visually look for errors, edge cases, etc.
Implement the forward model. Implement data generation based on your model if it’s statistical. This is called a prior predictive check. Can you tell a simulation apart from real-world observed data? How? If you can, does it give you ideas for how to improve the model?
Test parameter estimation. Some property-based tests: (1) generate a random $\boldsymbol{\theta}$, use it to generate a dataset, fit on the dataset. Is $\lVert \boldsymbol{\theta} - \hat{\boldsymbol{\theta}} \rVert$ small? It should be, especially in the limit of much data. (2) Test that $\mathcal{L}(\hat{\boldsymbol{\theta}}) \leq \mathcal{L}(\boldsymbol{\theta})$.
Plot parameter-space. You can plot the contours of the loss as a function of two parameters in $\boldsymbol{\theta}$ at a time, while fixing the remaining parameters at $\hat{\boldsymbol{\theta}}$. Sometimes studying the Hessian is worthwhile.
Plot residuals. When are residuals large? Are residuals homoscedastic? Should you model heteroscedasticity?

Optimization

Understand the optimization algorithm. Have at least a rudimentary understanding of the algorithms you use. Read the documentation and perhaps a summary paper. Play with the optimizer on toy problems. Get a feeling for how it works, and more importantly how and when it fails. Study the full verbose output of the algorithm.
Put variables on the same scale. First order methods (gradient methods) depend critically on the condition number. Pure second order methods (Newton’s method) on the other hand are affine invariant and in principle unaffected by the condition number (numerics and quasi-Newton methods make this untrue in reality). Standardize or take logs/exp/sigmoid/etc as needed.
De-correlate variables. Sometimes you can de-correlate variables through re-parametrization of the curve function.
Use a prior. Even if you lack prior knowledge in the modeling sense, weak regularization tends to help the optimizer.
Pick a good starting point. Graphical trial and error often works. You can also solve a simpler problem, e.g. solve a least squares problem first and use that solution as the starting point for your $p=1$ norm problem. The curve function $f$ can sometimes be relaxed by removing non-linearities. You could use (y[-1] - y[0])/(x[-1] - x[0]) as initial guess for slope, mean(y) as initial guess for bias, etc. Stress test the solver on bad initial solutions too.
Implement gradients and/or Hessian. Take the time to work out gradients. Use a CAS like SymPy or WolframAlpha along with pen and paper. Verify gradients numerically with scipy.optimize.check_grad or similar functions. If the gradient is too unwieldy, consider autograd with one of the many available packages: jax, autograd, PyTorch, TensorFlow or Stan.
Use numerically stable implementations. Watch out for expressions like $\ln(1 + e^{\theta})$ that overflow, catastrophic cancellation and other phenomena. See The Right Way to Calculate Stuff for more examples.

Curve fitting seems easy, but this perception can be deceptive. It’s not uncommon that people fail to get the desired results, often because they call standard routines without knowing how they work. I hope the tips and tricks above can help. Experience helps too: work on diverse practical problems and try to solve them. Make notes on what works in practice.

Lønna til norske utviklere i 2025

Tommy Odland — Wed, 17 Sep 2025 00:00:00 +0200

I denne artikkelen skal vi gå gjennom betraktninger og metoder for å lage en lønnskalkulator for IT-utviklere. Utålmodige lesere kan gå rett til Lønnskalkulatoren og se hovedfunnene i figuren nedenfor.

Nettsiden kode24 har lenge hatt årlige spørreundersøkelser om lønn og publisert data i etterkant. Vi kan finne datasett for 2021, 2022, 2023, 2024 og 2025 på nettsidene.

Jeg har tidligere gjort flere analyser av spørreundersøkelsene:

I 2021 gjorde jeg en analyse i det probabilistiske programmeringsspråket Stan
I 2023 brukte jeg generaliserte additive modeller
I 2024 gjentok jeg analysen fra 2023 med det nye datasettet
Basert på 2024-tallene hjalp jeg også til med å lage kodejobb.no’s Lønnskalkulator 2024

Det meste som kan sies har blitt sagt: konsulenter tjener noe mer enn fast ansatte, bosatte i Oslo får en lønnspremie, ledere og arkitekter tjener mest, kjønn er (heldigvis) ubetydelig, og antall år med erfaring dominerer alle andre variabler. Størsteparten av variasjonen i lønn skyldes variabler som spørreundersøkelsene ikke plukker opp. Til sammen forklarer variablene i lønnsundersøkelsene mellom 30% og 50% av variasjonen i lønna; hvor mye avhenger av datasettet og modellen. Les mine tidligere analyser for en nøye gjennomgang.

Årets pro-bono bidrag til IT-miljøet blir å slå sammen alle datasettene og lage en robust lønnskalulator. Den er robust fordi den (1) baserer seg på nesten ti tusen observasjoner og (2) tar automatisk høyde for lønnsvekst i fremtiden.

Datasett

Det er fem datasett for fem ulike år. “Data engineering” er alltid mer jobb enn man håper, og innsatsen for å slå sammen disse datasettene var ikke noe unntak. Jeg måtte:

Sammenstille lønnstall på format som 600 000 med 600.000,00 kr og lignende.
Sammenstille små variasjoner i spørsmålene fra år til år, f.eks. I hvilket fylke ligger jobben din? og I hvilket fylke jobber du?.
Ta hensyn til at det noen år var spurt om lønn med bonus og andre år lønn uten bonus. Dette fikser jeg med en latent variabel i modelleringen, men vi går ikke inn på det i denne artikkelen.
Fjerne tvilsomme observasjoner (omtrent 1.5% av dataene, der lønn var under 400 TNOK eller over 2000 TNOK).
Noen har sannsynligvis tolket spørsmålet “Hvor mange års relevant, formell utdannelse har du?” til å inkludere barneskolen, ungdomsskolen og VGS. Dette må fikses etter beste evne (jeg håper og tror ingen har vært student på et universitet i mer enn femten år).
Ta hensyn til regionreformen og dens delvise omgjørelse.
Slå sammen fagdisipliner, f.eks. var arkitektur et alternativ noen år, mens det het arkitekt andre år.
Slå sammen spørsmål: tidligere var jobb (in-house, konsulent, osv.) og sektor (privat, offentlig) to spørsmål—nå er det slått sammen til ett spørsmål.

Etter en grundig datavask står vi igjen med 9610 observasjoner. En del av disse er nok de samme personene som svarer på nytt, men det tenker jeg at er greit i denne sammenhengen. Som alltid er det masse forbehold: dataene er selvrapporterte, kode24’s lesere er ikke representative for utviklere generelt, osv. Nedenfor er lønn og erfaring visualisert for alle årene.

Medianlønna har økt de siste fem årene, men det har også gjennomsnittlig erfaring blant kode24’s lesere.

år	erfaring	lønn
2021	8.2	690
2022	8.3	725
2023	8.8	790
2024	8.5	850
2025	9.4	880

En regresjon på log-lønna avslører at den gjennomsnittlige lønnsveksten har vært omtrent 6.4% de siste årene, men dette tar ikke hensyn til at erfaringen også har økt. Om vi korrigerer for dette blir lønnsøkningen omtrent 5.1% per år.

Lønnskalkulatorer

En kalkulator implementerer vanligvis en kjent formel. For eksempel implementerer en skattekalkulator formelen for skatt (noe som kan være utfordrende å finne ut av). På samme måte kan en lønnskalkulator implementere “formelen for lønn”, men den vet vi ikke hva er. Derfor må vi først bruke en modell til å lære en formel fra datasettet, og deretter implementere denne. Et alternativ er å ikke implementere noen formel i det hele tatt, men heller bruke en såkalt ikke-parametrisk modell og vise brukeren sammenlignbare personer. Vi skal se på disse to alternativene i denne artikkelen, men det finnes andre muligheter også.

Uansett fremgangsmåte bør en god kalkulator:

Predikere godt. Estimert lønn bør matche faktisk lønn for så mange som mulig. Dette kan kryssvalideres.
Kommunisere usikkerheten. I dette datasettet har hver prediksjon stor usikkerhet, og det må brukeren forstå.

Ikke-parametriske modeller

Anta at en typisk utvikler ønsker å vite hva han eller hun kan forvente å tjene:

{'år': 2025, 'fylke': 'Oslo', 'jobb': 'in-house, privat sektor',
 'erfaring': 6, 'utdannelse': 3, 'fag': 'Fullstack'}

K-nærmeste naboer

Vi kan bruke K-nærmeste naboer med $k=10$ på 2025-data for å vise ti personer som ligner. Disse er

erfaring	utdannelse	fylke	jobb	fag	lønn
5	3	Oslo	in-house, privat sektor	Fullstack	775
7	3	Oslo	in-house, privat sektor	Fullstack	803
6	3	Oslo	in-house, privat sektor	Fullstack	819
5	3	Oslo	in-house, privat sektor	Fullstack	877
6	3	Oslo	in-house, privat sektor	Fullstack	895
6	3	Oslo	in-house, privat sektor	Fullstack	900
6	3	Oslo	in-house, privat sektor	Fullstack	914.375
5	3	Oslo	in-house, privat sektor	Fullstack	950
6	3	Oslo	in-house, privat sektor	Fullstack	1000
6	3	Oslo	in-house, privat sektor	Fullstack	1400

Her var vi heldige, fordi det er mange personer i Oslo med lignende egenskaper. Medianlønna blant disse ti personene er 897 TNOK, og 80% ligger i intervallet mellom 800 og 1040 TNOK.

Beslutningstre

Dersom vi bruker et beslutningstre i stedet, får vi følgende tabell. Modellen er satt til å alltid returnere minst ti personer, og for denne spørringen returnerer den 15. Sammenlignet med K-nærmeste naboer er beslutningstreet mer opptatt av erfaring, og mindre opptatt av fag og jobb.

erfaring	utdannelse	fylke	jobb	fag	lønn
6	3	Oslo	frilans / selvstendig næringsdrivende	DevOps og Automatisering	500
6	3	Oslo	in-house, privat sektor	UX og Design	650
6	3	Oslo	konsulent	Ledelse	735
6	3	Oslo	konsulent	Frontend	780
6	3	Oslo	in-house, privat sektor	Fullstack	819
6	3	Oslo	konsulent	Testing	830
6	3	Oslo	in-house, privat sektor	Fullstack	895
6	3	Oslo	in-house, privat sektor	Frontend	900
6	3	Oslo	in-house, privat sektor	Fullstack	900
6	3	Oslo	in-house, privat sektor	Fullstack	914.375
6	3	Oslo	konsulent	Fullstack	918
6	3	Oslo	konsulent	Frontend	940
6	3	Oslo	in-house, privat sektor	Fullstack	1000
6	3	Oslo	in-house, privat sektor	Fullstack	1400
6	3	Oslo	konsulent	Fullstack	1400

Medianlønna blant disse 15 personene er 900 TNOK, og 80% ligger i intervallet mellom 684 og 1240 TNOK.

Sammenligning: K-nærmeste naboer vs. beslutningstre

Vi kan sammenligne Mean Absolute Error (MAE) på en 10-folds kryssvalidering for å se hvor mange personer modellen bør returnere for å gi gode prediksjoner.

Vi observerer at:

Beslutningstrær er klart bedre enn K-nærmeste naboer, med lavere MAE over hele linja.
Så lenge vi returnerer omtrent 20 personer så får vi gode prediksjoner.
Jo flere personer vi returnerer, desto bedre kan vi estimere et intervall og si noe om lønnspennet.

Grunnen til at beslutningstrær er bedre enn K-nærmeste naboer er fordi førstnevnte automatisk velger hvilke variabler som er viktige (erfaring er helt klart viktigst), mens sistnevnte må gå like langt ut i alle dimensjoner for å finne naboer.

Både K-nærmeste naboer og beslutningstrær har noen felles svakheter i denne problemstillingen:

Nær endepunktene i hver variabel er de nødt til å se innover for å gjøre prediksjoner. For eksempel, dersom man har 45 års erfaring vil modellen ikke finne noen personer som har mer erfaring, kun personer som har mindre erfaring. Det samme gjelder de som har null år med erfaring eller null år med utdannelse. Dette er spesielt kritisk i høye dimensjoner, fordi da er sannsynligvis alle datapunktene nær endepunktene—jo flere ting vi måler, desto mer er alle mennesker ulike på hver sin måte!
Vi brukte 2025-data til nå, men vi ønsker å bruke hele datasettet. Vi vet at tidligere år har lavere lønn på grunn av generell lønnsøkning i samfunnet, men 2025-tallene er nær endepunktet på variabelen “år”. Derfor vil modellene ofte kun hente informasjon fra 2025 og ikke bruke hele datasettet. Begge modellen er lokale heller enn globale, og evner ikke å bruke informasjon fra tidligere år på en god måte.

Parametriske modeller

En parametrisk modell lærer parametrene i en formel som beskriver et datasett. Fordelen er at hele datasettet kan komprimeres ned til én kort formel som beskriver sammenhengene. Ulempen er at man må selv spesifisere en fornuftig formel. Parametriske modeller er globale og ekstrapolerer vanligvis godt, og dette er en fordel i vår sammenheng—spesielt med tanke på årstall inn i fremtiden og lønnsinflasjon.

Første idé er å bruke en Ridge-modell og minimere

\begin{align*} \text{minimize} \, &\lVert \text{lønn} - \boldsymbol{\mu} \rVert_2^2 + \alpha \lVert \boldsymbol{\beta} \rVert_2^2 \\ \mu &= \beta_0 + f(\text{erfaring}) + \beta_{\text{utdannelse}} \text{utdannelse} + \beta_{\text{fylke}} + \beta_{\text{jobb}} + \beta_{\text{fag}} \\ f(z) &= \beta_1 z + \beta_2 z^2 + \beta_3 z^3 + \cdots \end{align*}

og dette er et godt utgangspunkt som gir helt kurante resultater. Høyere-ordens polynomer er vanligvis ingen god idé, fordi de kan overtilpasse og svinge voldsomt. Det er derfor viktig å plotte polynomet (noe som ble gjort men ikke er med i artikkelen).

Hvordan kan vi forbedre denne modellen? For det første vil vi finne medianlønna heller enn gjennomsnittslønna, fordi medianen er mindre påvirket av ekstreme observasjoner og er ofte et bedre valg i slike sammenhenger. Det var derfor vi brukte Mean Absolute Error (MAE) heller enn Mean Squared Error (MSE) tidligere; MAE er minimert av medianen.

For det andre er en tradisjonell regresjonsmodell påvirket av feil i absolutte tall, mens multiplikative feil gir mer mening i lønnssammenheng. Med andre ord, å bomme med 100 TNOK når lønna er 500 TNOK er en stor feil, mens å bomme med 100 TNOK når lønna er 1200 TNOK er ikke like ille, fordi det er en mindre multiplikativ feil.

Vi endrer modellen til å bruke en $\ell_1$-norm på likelihood-leddet og tilpasse log-lønn heller enn lønn:

\begin{align*} \text{minimize} \, &\lVert \log \left( \text{lønn} \right) - \boldsymbol{\mu} \rVert_1 + \alpha \lVert \boldsymbol{\beta} \rVert_2^2 \end{align*}

Dette minner om en Lasso, men med normene byttet om: vi har $\ell_1$-norm på likelihood-leddet og $\ell_2$-norm på regulariseringsleddet. Her er koeffisientene i modellen (klikk på figuren for å se større versjon):

Legg merke til at lønnsveksten estimeres til 5.1% per år.

Å sammenligne mellom kategorier innebærer å konvertere prosentene til relative tall, ikke å legge dem sammen additivt. For eksempel, dersom man går fra en in-house offentlig stilling til å bli konsulent, kan man forvente en lønnsøkning på omtrent

\begin{align*} \frac{1 + 5.9\%}{1 - 5.4\%} = \frac{1.059}{0.946}= 1.119 = 11.9\% . \end{align*}

Her er residualene til modellen:

Dette er relativt pene residualer, fordi det ikke er stor endring i residualene over lønnsintervallet, såkalt heteroskadist… hetroskedasisti…. heteroskedastisitet. Polynomet som ble brukt var et fjerdegradspolynom, som på grunn av regularisering oppfører seg pent når erfaring er mellom null og førti år.

La oss kjøre vår “typiske” utvikler gjennom alle tre modellene:

{'år': 2025, 'fylke': 'Oslo', 'jobb': 'in-house, privat sektor',
 'erfaring': 6, 'utdannelse': 3, 'fag': 'Fullstack'}

Vi observerer noe ulike estimater av medianlønn og intervaller fra de ulike modellene:

KNN: 897 TNOK med 80% intervall mellom 800 og 1040 TNOK.
Tre: 900 TNOK med 80% intervall mellom 684 og 1240 TNOK.
Lineær modell: 918 TNOK med 80% intervall mellom 748 og 1136 TNOK.

Intervallene fra de ikke-parametriske modellene (KNN og tre) er mindre robuste, ettersom de er basert på henholdsvis 10 og 15 observasjoner. Intervallet fra den lineære modellen er basert på alle datapunktene, og har høyere kvalitet.

La oss nå sammenligne MAE for alle modellene, samt en svak dummy-modell som alltid predikerer medianen og en kraftig gradient boosting modell.

Modell / Metrikk	MAE
Dummy	202
KNN	152
Tre	128
Lineær	119
Gradient boosting	114

Gradient boosting har lavest MAE, men den lineære modellen er bare marginalt verre. Likevel har den noen klare fordeler:

Det estimerte intervallet (usikkerheten) er nok mer robust og realistisk.
Den er veldig enkel å implementere i Javascript. Man bruker bare formelen.

Oppsummering

Kode24 har gjort en kjempejobb med å samle inn og publisere data på utvikleres lønn. I denne artikkelen kombinerte vi fem år med lønnsdata til ett datasett med 9610 observasjoner, etter datavask og sammenstilling av variabler på tvers av år.

Vi sammenlignet to ulike tilnærminger til lønnskalkulatorer:

Ikke-parametriske modeller (KNN og beslutningstrær) som viser faktiske, sammenlignbare personer
Parametriske modeller (log-lineær regresjon med $\ell_1$-loss) som bruker en global formel

Den enkle log-lineære modellen (MAE: 119) presterte bedre enn de to ikke-parametriske modellene, fordi MAE var lavere og intervallene var mer robuste. Faktisk hadde modellen nesten like lav MAE som gradient boosting (MAE: 114), men med fordelen av å være tolkbar og enkel å implementere i JavaScript. Beslutningstrær var gode for å finne sammenlignbare personer, betydelig bedre enn K-nærmeste naboer.

Når det gjelder metodikk for lønnskalkulatorer er lærdommen klar: prioriter enkelhet og transparens. Brukere trenger to ting: (1) fornuftige prediksjoner og (2) ærlig kommunikasjon om usikkerheten. Kalkulatoren implementerer den enkle lineære modellen.

Selv om jeg fulgte bud (5) og bud (9) har jeg dårlig samvittighet for å ha brukt et fjerdegradspolynom i modelleringen uten progressiv regularisering—men jeg slapp unna med det på partial dependence plottet. Aldri igjen.

People rating people

Tommy Odland — Mon, 04 Aug 2025 00:00:00 +0200

Ratings and rankings are everywhere in daily life. We rank sports teams, chess players, movies and TV-shows, doctors, local businesses, and sometimes even each other.

This article is about computing an overall rating in a setting where people have individually rated each other. Examples include dating apps like Tinder, e-commerce websites where people act as both sellers and buyers, and other platforms where we want to quantify the trustworthiness or quality of members.

We distinguish between two scenarios:

Symmetry: if person $a$ rates $b$, then $b$ must also rate $a$. Their rating values may differ.
No symmetry: if person $a$ rates $b$, then it does not necessarily follow that $b$ rates $a$ back.

In this article we’ll consider the more general non-symmetric scenario. Given individual ratings that are either positive or negative, the goal is to produce an overall rating of everyone. The figure below shows 5 people, where blue arrows indicate positive ratings and red arrows indicate negative ratings.

In the figure above, we expect $a$ to be highly rated and $c$ to be poorly rated.

Simple aggregation

The simplest thing we can do is to count the number of ratings and create a ratio $r_i$ between $0$ and $1$ for each person.

We define a binary matrix $P$ consisting of positive ratings and a matrix $N$ of negative ratings.

\begin{align*} P_{ij} &= \begin{cases} 1 & \text{if person } i \text{ is rated positively by person } j \\ 0 & \text{otherwise} \end{cases} \\ N_{ij} &= \begin{cases} 1 & \text{if person } i \text{ is rated negatively by person } j \\ 0 & \text{otherwise} \end{cases} \end{align*}

The rating vector $\boldsymbol{r}$ is computed by summing positive ratings and dividing by total ratings. One issue with the naive approach is that someone who has few ratings might end up with a total rating of $0$ or $1$, so we employ the rule of succession by adding two dummy ratings: one positive and one negative. The equation becomes

\begin{align*} r_i &= \frac{1 + \sum_j P_{ij}}{2 + \sum_j P_{ij} + \sum_j N_{ij}} \end{align*}

Or alternatively, in matrix notation, where $\boldsymbol{1}$ and $\boldsymbol{2}$ denote vectors of ones and twos and division is entry-wise:

\begin{align*} \boldsymbol{r} &= \frac{\boldsymbol{1} + P \boldsymbol{1}}{\boldsymbol{2} + P \boldsymbol{1} + N \boldsymbol{1}} \end{align*}

Here’s a piece of Python code that implements this computation on the graph structure shown in the initial figure:

import numpy as np

P = np.array([
     [0, 1, 1, 1, 1], # <- Positive ratings given to A
     [0, 0, 0, 0, 0], # <- Positive ratings given to B
     [0, 0, 0, 0, 0],
     [0, 1, 1, 0, 0],
     [1, 0, 0, 0, 0], # <- Positive ratings given to E
    ])

N = np.array([
     [0, 0, 0, 0, 0], # <- Negative ratings given to A
     [0, 0, 1, 0, 0], # <- Negative ratings given to B
     [0, 1, 0, 1, 1],
     [1, 0, 0, 0, 1],
     [0, 0, 0, 0, 0], # <- Negative ratings given to E
    ])

# Simple aggregation
ratings = (1 + np.sum(P, axis=1)) / (2 + np.sum(P, axis=1) + np.sum(N, axis=1))
print(ratings.round(2))  # [0.83 0.33 0.2  0.5  0.67]

The results are superimposed on the graph in the figure below.

The popular person $a$ gets the highest rating $0.83$ as expected, and unpopular person $c$ gets the lowest rating $0.2$ as expected.

Weighted aggregation

Perhaps all users in a system should not be considered equally trustworthy. In other words, if $a$ rates $b$ positively, then our opinion of $b$ might also depend on $a$.

If $a$ is a long-time user, who is positively rated by others and who has given balanced ratings to others, then we should trust $a$’s ratings.
If $a$ is a new user, or is negatively rated by others, or rates everyone negatively, then we should perhaps not trust $a$’s ratings as much.

We weight each user with a weight $w_j \in [0, 1]$ before we aggregate their individual ratings:

\begin{align*} r_i &= \frac{1 + \sum_j P_{ij} w_j}{2 + \sum_j P_{ij} w_j + \sum_j N_{ij} w_j} \end{align*}

How do we choose the weights $w_j$? There are many possibilities, but the weights $w_j$ must somehow represent the trustworthiness of person $j$.

Weighting by discrimination

One idea is to look at how much discrimination each user applies. Whose ratings do you trust more: someone who rates everyone positively, or someone who rates half the population negatively and the other half positively? If you agree with me that the second person provides more useful information, then one option is to compute a scaled version of the entropy:

\begin{align*} H(p, n) = \frac{1}{\ln 2} \left[ \frac{p}{p+n} \ln \left( \frac{p+n}{p} \right) + \frac{n}{p+n} \ln \left(\frac{p+n}{n}\right) \right], \end{align*}

where $n$ is the number of negative ratings the person has given and $p$ is the number of positive ratings.

Observe that $H(p=k, n=k) = 1$ for all $k \geq 1$ and that $H(p=k, n=0) = H(p=0, n=k) = 0$ for all $k \geq 1$. In other words, this weighting function accomplishes what we set out to do: users with high discriminatory power are given a weight of one, while users who show no discrimination in their ratings are given a weight of zero.

from scipy.stats import entropy

def H(n, p):
    return entropy([n, p]) / np.log(2)

print(H(5, 5))  # 1.0
print(H(0, 5))  # 0.0

weights = entropy(np.stack((np.sum(P, axis=0), np.sum(N, axis=0))), axis=0) / np.log(2)

# Weighted positive and negative sums
P_w = P * weights
N_w = N * weights

# Weighted ratings
ratings = (1 + np.sum(P_w, axis=1)) / (2 + np.sum(P_w, axis=1) + np.sum(N_w, axis=1))
print(ratings.round(2))  # [0.83 0.34 0.21 0.49 0.67]

In this particular problem instance, the results do not differ much from the previous ratings.

Entropy is by no means the only possible quality measure—it depends strongly on the specific use-case. A number of more complicated weighting functions could be used. For instance, we could also weight a user by:

how many ratings they have given in total
how long they have been users in the system/platform
how much information they have on their profile
how well they themselves are rated by others.

All of these signal high levels of trustworthiness. We will now pursue option (4): we’ll weight each person’s ratings by how highly they themselves are rated.

Weighted aggregation by rating

The final model that we’ll consider is the most advanced one. The idea is that the rating person $a$ gives to person $b$ should be weighted by the rating person $a$ has. Building on the matrix notation we established earlier, we write the equation as:

\begin{align*} \boldsymbol{r} &= \frac{\boldsymbol{1} + P \boldsymbol{r}}{\boldsymbol{2} + P \boldsymbol{r} + N \boldsymbol{r}} \end{align*}

In an e-commerce platform, this means that if $a$ is highly rated and gives a positive rating, then that is worth more than a positive rating from a user with mixed or negative ratings.
In a dating app like Tinder where ratings signify attractiveness, the interpretation is that if someone who is attractive rates us favorably, then that rating is worth more than a rating from someone who is not considered attractive by others.

Notice the recursive definition: to weight the ratings given by person $a$ to others, we need to know the overall rating of person $a$—but that quantity is exactly what we’re trying to compute in the first place!

Solution by fixed-point iteration

The simplest and fastest way to compute $\boldsymbol{r}$ is to iterate the equation until it converges. Starting with an initial guess $\boldsymbol{r}_0 = \boldsymbol{1} / 2$ we compute

\begin{align*} \boldsymbol{r}_{k+1} &= \frac{\boldsymbol{1} + P \boldsymbol{r}_{k}}{\boldsymbol{2} + P \boldsymbol{r}_{k} + N \boldsymbol{r}_{k}} \end{align*}

repeatedly until convergence, as measured by $\lVert \boldsymbol{r}_{k+1} - \boldsymbol{r}_{k} \rVert < \epsilon$.

epsilon = 1e-12
ratings = np.ones(N.shape[0]) / 2
for iteration in range(99):
    new_ratings = (1 + P @ ratings) / (2 + P @ ratings + N @ ratings)
    if np.linalg.norm(new_ratings - ratings) < epsilon:
        break
    ratings = new_ratings

print(ratings.round(2))  # [0.74 0.44 0.29 0.42 0.63]

Convergence is linear, and after 13 iterations the procedure terminates.

All three ratings induce rankings on the people, and they are compared in the figure below.

Notice how:

Person $b$ moves up compared to the simple rating. He is given a negative rating by $c$, but $c$ has a low overall rating and this lessens the weight given to $c$’s opinion of $b$.
Person $d$ moves down, because the negative rating from $a$ matters a lot and the positive rating from $c$ matters little.

Summary and references

Rating and ranking is interesting because it is part science and part art. There’s no fixed answer: every scenario is different and must be modeled appropriately.

The problem considered in this article, where people rate people, is special in two ways: (1) there is just one group and (2) ratings are not symmetric. The defining equation is

\begin{align*} \boldsymbol{r} &= \frac{\boldsymbol{1} + P \boldsymbol{r}}{\boldsymbol{2} + P \boldsymbol{r} + N \boldsymbol{r}}, \end{align*}

which is non-linear in the vector $\boldsymbol{r}$ and may contain millions of unknown variables. Despite this, it’s solved by fixed-point iteration in a few seconds on a laptop.

I’ve previously written about how patients can rate doctors. In that problem there were two separate groups: doctors and patients. One group rates the other group. I’ve also written about rating football teams, which is a symmetric problem. If team $a$ plays $b$, then $b$ necessarily plays $a$ too.

The PageRank algorithm ranks websites, and it’s what made Google famous in the late 90s. The paper A Survey on PageRank Computing is an interesting read. Similar algorithms include HITS, SALSA and EigenTrust. The book Who’s #1?: The Science of Rating and Ranking gives an overview of simple methods, with a focus on sports. When users interact with each other, the field is sometimes called reputation systems. The paper The Digitization of Word of Mouth: Promise and Challenges of Online Feedback Mechanisms explores the online reputation mechanisms of the early internet.

The ideas in the paper Propagation of Trust and Distrust are similar to those expressed in this article, but the authors focus more on local propagation while we solve a global problem. Reputation and ranking is closely related to content discovery, recommendation systems and collaborative filtering, and the book Recommender Systems by Aggarwal is a handy reference. This has social implications too, which the paper On Social Machines for Algorithmic Regulation explores.

Appendix A: Empirical results

I created random sparse rating matrices for a million people. I set the density to $p=10^{-5}$, so each matrix has approximately $\left( 10^6 \right)^2 \times p = 10^7$ non-negative entries. Convergence is shown in the figure below.

Below is the computation on the third largest strongly connected component of the Epinions social network dataset. The full dataset has 131,828 nodes and 841,372 and is solved quickly, but it’s too large to visualize in full.

Appendix B: Solution by Newton’s method

Newton’s method and its variants can be an alternative to fixed-point iteration. While Newton’s method exhibits quadratic convergence (fixed-point iteration is linear), the time spent per iteration is so high that it ends up being an inferior choice in practice. Also, if the starting point is not good ($\boldsymbol{r} = 1/2$ tends to work, while $\boldsymbol{r} = 0$ does not) the algorithm will fail. Regardless, here’s how it can be done.

The matrix equation

\begin{align*} \boldsymbol{r} &= \frac{1 + P \boldsymbol{r}}{2 + P \boldsymbol{r} + N \boldsymbol{r}}, \end{align*}

is equivalent to solving

\begin{align*} F(\boldsymbol{r}) = \boldsymbol{r} \odot \left( \boldsymbol{2} + P \boldsymbol{r} + N \boldsymbol{r} \right) - \left( \boldsymbol{1} + P \boldsymbol{r} \right) = \boldsymbol{0} \end{align*}

which we can do with Newton’s method:

\begin{align*} \boldsymbol{r}_{k+1} = \boldsymbol{r}_{k} - J(\boldsymbol{r}_{k})^{-1} F(\boldsymbol{r}_{k}). \end{align*}

To find the Jacobian matrix $J_{ij}(\boldsymbol{r}) = \partial F_i(\boldsymbol{r}) / \partial \boldsymbol{r}_{j}$, we first write out the components of $F(\boldsymbol{r})$.

\begin{align*} F_i(\boldsymbol{r}) = r_i \left( 2 + \sum_k P_{ik} r_k + \sum_k N_{ik} r_k\right) - 1 - \sum_k P_{ik} r_k \end{align*}

Differentiating with respect to $r_j$, then switching back to matrix notation, we obtain:

\begin{align*} J(\boldsymbol{r}) = \partial_{r_j} F_i(\boldsymbol{r}) & = \delta_{ij} \left( 2 + \sum_k P_{ik} r_k + \sum_k N_{ik} r_k\right) + r_i \left( P_{ij} + N_{ij} \right) - 0 - P_{ij} \\ &= 2 I + \operatorname{diag}(P \boldsymbol{r} + N \boldsymbol{r}) + \operatorname{diag}(\boldsymbol{r}) (P + N) - P \end{align*}

This can be implemented in Python as:

def F_and_J(ratings, P, N):
    """Returns a tuple (F, J), with function value F and its jacobian J."""
    Pr = P @ ratings
    Pr_Nr = Pr + N @ ratings
    F = ratings * (2 + Pr_Nr) - 1 - Pr
    J = ratings * (P + N) - P
    np.fill_diagonal(J, J.diagonal() + 2 + Pr_Nr)
    return F, J

epsilon = 1e-12
ratings = np.ones(N.shape[0]) / 2
for iteration in range(99):
    F, J = F_and_J(ratings, P, N)
    new_ratings = ratings - np.linalg.solve(J, F)
    if np.linalg.norm(new_ratings - ratings) < epsilon:
        break
    ratings = new_ratings
print(ratings.round(2))  # [0.74 0.44 0.29 0.42 0.63] <- Same as fixed-point

Note that the code above explicitly stores zeros (it does not use sparse matrices), and would therefore be unsuitable for large matrices.

Appendix C: Properties of the fixed-point iteration

In millions of computational experiments with random matrices $P$ and $N$, the fixed-point iteration seemingly always converges to a unique fixed point. For a long time I conjectured that the fixed point iteration always converges to a unique, attracting fixed point. However, it turns out this conjecture is false.

A counterexample is the following structure: There are three groups and at least 16 people in each group. Within each group, everyone rates everyone else negatively. There are no self-ratings.

With 16 people in each group, the algorithm cycles between $(0.509, 0.366, 0.509)$ and $(0.29, 0.585, 0.29)$ when started at a random $\boldsymbol{r}$. With 15 people in each group, the algorithm does not cycle, but will eventually converge.

Appendix D: Code

Here is some Python 3.12 code for the model.

import numpy as np  # Version: 2.1.3


class RatingProblem:

    def __init__(self, N, P):
        """Initialize a problem given positive binary ratings P
        and negative binary ratings N."""
        assert (N.shape == P.shape) and (N.shape[0] == N.shape[1])
        self.P, self.N = P, N

    @classmethod
    def random(cls, n, seed=None):
        """Create a random problem instance of size n."""
        rng = np.random.default_rng(seed)

        # Create adjacency matrix with random density
        density = rng.random()
        adj = (rng.random(size=(n, n)) < density).astype(int)

        # Zero out diagonal (can't rate yourself)
        np.fill_diagonal(adj, 0)

        # Randomly split into positive and negative ratings
        split = rng.random(size=(n, n)) < rng.random()
        return cls(N=adj * (1 - split), P=adj * split)

    def __call__(self, r):
        assert len(r) == self.N.shape[0]
        P, N = self.P, self.N
        return (1 + P @ r) / (2 + P @ r + N @ r)

    def jacobian(self, r):
        """Compute the Jacobian matrix at r."""
        assert len(r) == self.N.shape[0]

        P, N = self.P, self.N
        Pr, Nr = P @ r, N @ r

        # Vectorized jacobian computation
        numerator = (1 + Nr)[:, None] * P - (1 + Pr)[:, None] * N
        denominator = (2 + Pr + Nr) ** 2
        return numerator / denominator[:, None]

    def solve(self, r_0=None, epsilon=1e-12):
        """Solve using fixed-point iteration."""
        r_0 = np.ones(self.N.shape[0]) / 2 if r_0 is None else r_0

        assert len(r_0) == self.N.shape[0]

        ratings = r_0.copy()
        for iteration in range(99):
            new_ratings = self(ratings)
            if np.linalg.norm(new_ratings - ratings) < epsilon:
                break
            ratings = new_ratings
        return ratings


if __name__ == "__main__":

    rng = np.random.default_rng(0)

    # Test that different starting points converge to the same value
    for test in range(10**4):

        n = rng.integers(4, 20)
        problem = RatingProblem.random(n=n, seed=test)
        r1 = rng.random(size=n) * rng.random()
        r2 = rng.random(size=n) * rng.random()

        s1 = problem.solve(r1)
        s2 = problem.solve(r2)
        assert np.allclose(s1, s2)

        if test % 1_000 == 0:
            print(".", end="")

Hva kan jeg bruke matte til?

Tommy Odland — Thu, 17 Jul 2025 00:00:00 +0200

Elever spør ofte “når får jeg bruk for dette?” eller “hva skal jeg bruke matematikk til?“

Det er et godt spørsmål som ofte går ubesvart.

Det er flere måter å svare på slike spørsmål på. Her en én type svar: en liste med 100 konkrete problemstillinger der matematikk er nyttig.

Noen problemstillinger er relativt enkle, som å bygge en boks med størst mulig volum. Andre er vanskelige, som å spille Go. Å vanne plenen fremstår som et tøyeste problem, men illustrerer hvor dypt man kan gå om man begynner å undersøke en problemstilling. Andre er store og mer seriøse problemer, som å lande en rakett.

Det er på ingen måte ment å være en uttømmende eller spesielt seriøs liste :)

Telle
Spille Go
Spille dart
Sortere tall
Lage kunst
Løse Soduko
Forstå politikk
Vanne plenen
Spille dataspill
Styre en missil
Lagre fiskemat
Lande en rakett
Løse en labyrint
Lage en labyrint
Vedde på hester
Lage en matplan
Føre et regnskap
Regne med flyttall
Gå inn i veddemål
Fargelegge et kart
Teste en hypotese
Lage en søkemotor
Rangere klatreruter
Dele en taxiregning
Bygge en database
Arrangere ekteskap
Designe en auksjon
Simulere stigespillet
Planlegge bussruter
Bygge en søkemotor
Regne på et boliglån
Forstå språkmodeller
Rangere sjakkspillere
Bygge en atomreaktor
Digital bildebehandling
Manipulere valgkretser
Lage en aksjeportefølje
Undersøke dødsårsaker
Bygge elektriske kretser
Anbefale filmer og serier
Lage et treningsprogram
Forstå kunstig intelligens
Matche spillere i dataspill
Løse en differensialligning
Analysere dating-markedet
Lage turnus for sykepleiere
Designe effektive algoritmer
Forstå verden gjennom data
Verdtsettelse av fritidsboliger
Finne minimum av en funksjon
Lage vaktplaner for et sykehus
Designe et demokratisk system
Finne korteste reisevei offshore
Vedde penger på fotballkamper
Designe effektive datastrukturer
Effektivisere robotiserte varehus
Få mest mulig bolig for pengene
Modellere et petroleumsreservoir
Vedlikeholde roterende maskiner
Lagre store mengder informasjon
Lage et program som spiller sjakk
Planlegge sitteplasser i barnehagen
Forsvare infrastruktur mot terrorister
Innholdsbevisst reskalering av bilder
Forstå hvordan et aksjefond fungerer
Bygge en prosessor til en datamaskin
Lage en valgomat for kommunevalget
Bygge skyskrapere som ikke kollapser
Effektivisere Lånekassens bokontroller
Bygge en boks med størst mulig volum
Sette sammen en ungdomsskoleklasse
Predikere hvor billettkontrollører bør stå
Analysere data fra spørreundersøkelser
Estimere kostnadene til et stort prosjekt
Bestemme når man bør pensjonere seg
Predikere hvem som finner fotballkamper
Verdsette fritidsboliger med maskinlæring
Sette sammen det beste Pokémon-partiet
Logge tastetrykk ved hjelp av en mikrofon
Sette best mulig pris på en Airbnb-leilighet
Lage en Bayesiansk sannsynlighetsmodell
Plassere hjertestartere på best mulig steder
Forstå DNA-sekvenser til mennesker og dyr
Finne ut om du blir snytt når du kjøper Twist
Spare 125 millioner på å borre en rett tunnel
Programmere og studere logikken til en heis
Regne ut om det er best å eie eller leie bolig
Sette sammen forsikringer for marin shipping
Projisere høye dimensjoner til få dimensjoner
Lage en inntaksmodell for videregående skole
Bestemme hvor mye et bakeri bør bake hver dag
Kutte tekstilruller slik at man får minst mulig svinn
Designe en kaffekopp som holder godt på varmen
Regne ut sannsynligheten for suksessfull putt i golf
Designe loot-boxes i dataspill for å maksimere profitt
Finne ut hvilke faktorer som er assosiert med høy lønn
Plassere returpunkter for glass- og metallembalasje i byen
Lage timeplan for en skole for best mulig ressursutnyttelse
Teste om man klarer å smake forskjell på brus med og uten sukker
Forstå hvorfor man ikke alltid bør stemme på politikerne man ønsker

Who’s the best bettor?

Tommy Odland — Sat, 07 Jun 2025 00:00:00 +0200

A friend of mine is part of a group that arranges bets at their workplace. Each person proposes bets they believe will come true. The bets have European-style odds $o$ and are from a betting exchange. The money placed on each bet is always the same, so we arbitrarily take it to be one unit. Each bet leads to an outcome $y = 1$ (win) or $y = 0$ (loss).

Given histories of bets for each person, how can we decide who the best bettor is?

For instance, given three people Bob, Alice and Mark with histories:

person    odds (o)             outcomes (y)
bob       (2, 2, 2)            (1, 1, 1)    
alice     (3, 3)               (1, 1)    
mark      (2, 2, 2, 2, 2)      (1, 1, 1, 1, 0)

Who is the best bettor? Both Bob and Alice have won all their bets, but Alice took higher risk. Mark placed more bets that came through, but one of his bets was wrong.

In this article we will define the likelihood of an outcome over a sequence of bets. Then we create a latent variable that encodes the skill of a bettor, relative to the probability given by the odds. Given a sequence of bets and outcomes we can then solve for this latent variable, and the best bettor is the person with the greatest value.

The probability of an outcome

A person performs a sequence of $n$ bets with odds $o_i$ and attains outcomes $y_i$ for each bet $i=1, 2, \ldots, n$. Ignoring the margin of the bookmaker, the odds can be converted to a probabilities by the equation $p_i = 1 / o_i$. Each bet is independent, so the probability of the joint outcome is

\begin{equation*} P(\boldsymbol{p}, \boldsymbol{y}) = \prod_i^n p_i^{y_i} (1 - p_i)^{1 - y_i}. \end{equation*}

Insight

When a bet will be made. Suppose a person believes the true probability of winning a bet is $\tilde{p}$. With European-style odds the net gain is $(o-1)$ if a bet is won and $-1$ if it is lost. A person will only bet if the expected payout is larger than zero, that is, if

\begin{equation*} P(\text{win}) (\text{amount won}) + P(\text{loss}) (\text{amount lost}) = \tilde{p} (o - 1) + (1 - \tilde{p}) (-1) \geq 0. \end{equation*}

Solving for $\tilde{p}$ we learn that a person will only bet if $\tilde{p} \geq 1/o = p$. In other words, they will bet if their own personal probability is greater than the one deduced from the odds.

Introducing the latent insight $\theta$. We will assume that each person has some insight $\theta$, which is a latent (unobserved) variable representing their additional information. We would like to infer how much insight each person has, and rank their betting ability by their insight. One idea is to set $\tilde{p} = p + \theta$, but we prefer $\theta$ to be an unbounded variable and the probabilities $\tilde{p}$ to be restricted to the domain $[0, 1]$. To accomplish this we use the logistic function (sigmoid or expit) and its inverse the logit. The functions are:

\begin{equation*} \operatorname{expit}(z) = \frac{1}{1 + \exp(-z)} \quad \text{and} \quad \operatorname{logit}(z) = \log\left( \frac{z}{1 - z} \right). \quad \end{equation*}

To ensure that $\tilde{p}$ ends up in the domain $[0, 1]$ we transform the probabilities to logits on $\mathbb{R}$, add the insight $\theta$, and transform the result back to $[0, 1]$:

\begin{equation*} \tilde{p}(\theta, p) = \operatorname{expit}\left( \operatorname{logit}(p) + \theta \right) = \frac{p e^{\theta}}{p e^{\theta} - p + 1} \end{equation*}

Substituting $\tilde{p}(\theta, p)$ into the probability of the joint outcome, we obtain the negative log-likelihood function

\begin{align*} - \log P( \boldsymbol{p}, \boldsymbol{y} \mid \theta) &= - \sum_i^n y_i \log \left( \tilde{p}_i(\theta, p_i) \right) + (1 - y_i) \log \left( 1 - \tilde{p}_i(\theta, p_i) \right) \\ &= \sum_i^n - y_i \log{\left(\frac{p_i e^{\theta}}{p_i e^{\theta} - p_i + 1} \right)} + \left(y_i - 1\right) \log{\left(\frac{1 - p_i}{p_i e^{\theta} - p_i + 1} \right)}. \end{align*}

The likelihood is the probability of observing $(\boldsymbol{p}, \boldsymbol{y})$ given a value of $\theta$.

Why a prior is needed. Suppose someone has made three bets with odds $\boldsymbol{o} = (2, 2, 2)$ and outcomes $\boldsymbol{y} = (1, 1, 1)$. Plotting the negative log-likelihood as a function of $\theta$ we get the following figure. Minimizing the negative log-likelihood leads to $\theta \to \infty$. In a sense, the person is able to claim “I’m great at betting! I knew all along that the true probability was one for all these bets!”

This is analogous to estimating the probability of heads for a sequence of coin tosses as the number of heads divided by the total number of coin tosses. If we get three heads, then the estimated probability is one, and that is unreasonable.

Putting a prior on insight

It’s far-fetched to believe that someone who wins a bet or two really has great insight. To fix this issue of $\theta$ going to infinity, we put a prior distribution on it. We center a normal distribution around zero, and since the standard deviation is somewhat arbitrary we choose unity. In other words we choose $\theta \sim \mathcal{N}(1, 0)$ as the prior. A single squared term is added to the negative log-likelihood (now a log-posterior), which becomes

\begin{align*} - \log P(\theta \mid \boldsymbol{p}, \boldsymbol{y}) &\propto - \sum_i^n y_i \log \left( \tilde{p}_i(\theta, p_i) \right) + (1 - y_i) \log \left( 1 - \tilde{p}_i(\theta, p_i) \right) + \theta^2 \\ &\propto \theta^{2} - \sum_i^n \theta y_i + \log{\left(p_i e^{\theta} - p_i + 1 \right)}. \end{align*}

As a result, the negative log-posterior now has a finite minimizer: We now have a reasonable statistical model.

To summarize, we assumed that each person had some insight $\theta$ that led them to believe that the true probabilities $\tilde{p}_i$ are higher than the $p_i$ deduced from the odds. Then we compute the maximum a posteriori (MAP) estimate of $\theta$, which is the most probable value of $\theta$ given the observed sequence of bets. A higher value of $\theta$ means the person has insight and beats the bookmaker, $\theta$ around zero means no additional insight is available apart from the one deduced by the odds, and a negative value of $\theta$ means the person consistently underperforms.

Cases

We will now go through some concrete cases and check if the model is sensible.

Placing more and more bets. Suppose the winning streak is increased by one bet at a time. The table below shows odds, outcomes and MAP estimates of the insight $\theta$.

case  odds (o)       outcomes (y)  insight (theta)
1     (,)            (,)            0.0000
2     (2,)           (1,)           0.2223
3     (2, 2)         (1, 1)         0.4011
4     (2, 2, 2)      (1, 1, 1)      0.5491
5     (2, 2, 2, 2)   (1, 1, 1, 1)   0.6748

No bets placed implies zero insight, as there is no reason to believe the bettor is any better than randomly guessing. As the number of consecutive wins increases, the posterior mode moves away from the prior and we believe more and more that the person really has insight and outperforms the bookmaker.

Equal gains. Suppose earlier wins can be used to be more money on the next bets. Here are three bettors who all multiplied their money by $16$.

case  odds (o)       outcomes (y)  insight (theta)
1     (16,)          (1,)           0.4526
2     (4, 4)         (1, 1)         0.6179
3     (2, 2, 2, 2)   (1, 1, 1, 1)   0.6748

The model assigns the most insight to the person who placed four bets. Suppose each person has some small insight $\epsilon$. Then the probability $(1/16 + \epsilon) \geq (1/2 + \epsilon)^4 = (1/2)^4 + (1/2)^3 \epsilon + \mathcal{O}(\epsilon^2)$ when $\epsilon$ is small. The probability that the person who places four bets ends up with four wins is lower, and it is therefore more impressive. In other words, the model favors winning on many smaller bets rather than one big bet, even though the final monetary gain is identical.

Although the amount of money placed on each bet is the same in our setup, the result above makes even more sense if we were betting on a sequence and re-investing our wins. A longer sequence of smaller wins is more beneficial than one large win, since we can re-invest our previous earnings along the way.

A win and a loss. Now consider four bettors who all win one bet and lose one bet.

case  odds (o)       outcomes (y)  insight (theta)
1     (2, 2)         (1, 0)         0.0
2     (3, 3)         (1, 0)         0.1358
3     (4, 4)         (1, 0)         0.2088
4     (5, 5)         (1, 0)         0.2559

The model assigns the most insight to the person who took the most risk. This also makes sense.

Case 1 represents winning once with odds two (doubling the money), then losing one unit of money. This bettor is back to where he started.
Case 5 represents winning once with odds five (returning five times the original amount of money), then losing one unit of money. This bettor is still up by four units of money.

Some random cases. Below is a table with nine cases, sorted by the inferred insight $\theta$ of each bettor. These cases are all pretty random, and are shown so the reader can study how different odds and outcomes measure up against each other.

case  odds (o)       outcomes (y)  insight (theta)
1     (2, 2, 2)      (0, 0, 1)     -0.182
2     (2, 2)         (1, 0)         0.0
3     (2, 2, 2)      (0, 1, 1)      0.182
4     (2, 2, 2)      (1, 0, 1)      0.182
5     (5, 5)         (1, 0)         0.2559
6     (8,)           (1,)           0.4113
7     (16,)          (1,)           0.4526
8     (3, 3)         (1, 1)         0.5386
9     (2, 2, 2)      (1, 1, 1)      0.5491

Conclusion

To determine who the best bettor is, we modeled a bettor’s insight $\theta$ as a latent variable. The insight is added to the probability deduced from the odds, and the best bettor is the one with the greatest insight. In order to avoid $\theta$ becoming very large or very small in small sequences with only wins or losses, we put a prior on it. The skill of a bettor is the maximum a posteriori estimate of $\theta$.

There might be other interesting ways of solving this problem, but I think this approach works well. It’s conceptually simple, all test cases look good, and it’s simple to implement and compute. Further work that might be interesting includes: making the optimization routine faster, studying the equations in greater depth, or computing the entire posterior distribution instead of finding the value that maximizes it.

Code

Here’s a Python 3.11 code snippet if you want to run the model.

import numpy as np  # 1.26.4
import scipy as sp  # 1.13.0
import functools


def log_posterior(theta, odds, y):
    """Compute the log-posterior of theta, given `odds` and `y`."""

    # Convert from odds to base probabilities
    p = 1 / odds
    # Convert to logit scale using logit(z) = log(z / (1-z))
    logits = sp.special.logit(p)
    # Add `theta` (insight) and convet back using expit(z) = 1/(1+exp(-z))
    p_tilde = sp.special.expit(logits + theta)
    # Compute the log-likelihood
    log_likelihood = y * np.log(p_tilde) + (1 - y) * np.log(1 - p_tilde)

    # A more numerically stable version as theta -> oo is:
    # log_likelihood = -theta * (1 - y) - np.logaddexp(
    #     np.log(p), -theta + np.log1p(-p))

    # Add prior and return the negative log-posterior
    return -np.sum(log_likelihood) + theta**2


# Example case
odds = np.array([2, 2, 2])
y = np.array([1, 1, 1])

# Optimize the log-posterior for optimal theta
fun = functools.partial(log_posterior, odds=odds, y=y)
result = sp.optimize.minimize_scalar(fun=fun)
print(result.x)  # 0.549107...

Fraværsgrensa

Tommy Odland — Thu, 22 May 2025 00:00:00 +0200

Fraværsgrensa i videregående skole er på 10%. Det betyr at dersom en elev har udokumentert fravær på 10% eller mer i et fag, så mister eleven karakter i faget. I Reddit-tråden “Videregående skole er no bullshit” har en elev fått varsel fordi han eller hun var borte én dag, som utløste 3 timers fravær i kroppsøving og oversteg 5%.

Kroppsøving inneholder 56 undervisningstimer totalt, mens f.eks. matematikk har 140 timer. Det betyr at fraværsgrensen har ulik effekt avhengig av hvor mange skoledager som inneholder faget:

Om kroppsøving er på 3 timer per dag, så vil 2 dager med fravær betyr at man mister karakteren, fordi $6 / 56 \approx 0.107 \geq 0.1$.
Om matematikk er på 2 timer per dag, så vil 7 dager med fravær betyr at man mister karakteren, fordi $14 / 140 = 0.1 \geq 0.1$.

Selv om det er mye fravær som kan unntas fraværsgrensen, er det rimelig å anta at noe udokumentert fravær vil forekomme. Denne artikkelen svarer på spørsmålet: “gitt at udokumentert fravær inntreffer tilfeldig, hva er sannsynligheten for at en elev mister karakteren i et fag?”

Sannsynligheten for å miste karaktergrunnlaget i et fag

La oss gjøre to forenklinger før vi fortsetter:

Vi antar at hver dag med et fag inneholder like mange timer. Dette er tilfellet om man har 3 timer gym hver onsdag. Det er ikke tilfellet om man for eksempel har 2 timer matte på mandager og 1 time på torsdager. Denne antagelsen betyr at vi slipper å telle opp antall timer innad i hver dag og antall timer i faget totalt. Dette gjør utregningene litt enklere fordi vi slipper å forholde oss til faktiske timeplaner, som uansett vil variere fra skole til skole.
Vi antar at udokumentert fravær inntreffer tilfeldig med en viss sannsynlighet, og at fravær er uavhengig. Med andre ord: om en elev hadde fravær i går påvirker ikke sannsynligheten for at eleven har fravær i dag.

Figuren nedenfor viser sannsynligheten for å miste karakter et fag, som funksjon av antall dager med faget. $P(\text{fravær})$ er sannsynligheten for fravær på en gitt dag, og utregningen er visualisert for 1%, 2% og 3%.

Den overordnede trenden er at sannsynligheten for å miste karakter er høyere desto mindre antall dager man har med faget. Det stemmer med de innledende observasjonene om kroppsøving og matematikk.

Vi skal nå forklare knekkene i grafen: før man har 11 dager med et fag, vil én fraværsdag føre til at man overstiger grensa på 10%. På 11 dager har man litt rom for fravær ettersom én dag ikke lengre gjør at man overstiger grensa. Situasjonen blir likevel verre fra 11 dager til og med 20 dager. I dette intervallet vil to eller flere fraværsdager utløse grensa, og med flere dager er det større sannsynlighet for at det skjer—derfor stiger grafen. På 21 dager har man igjen litt mer slingring, og slik fortsetter mønsteret.

Utregning

La sannsynligheten for fravær være $p = P(\text{fravær})$, la fraværsgrensa være $g=0.1$ og la antall dager med faget være $d$. Antall fraværsdager $X \sim \text{Binom}(p, d)$ er da en stokastisk variabel med binomisk sannsynlighetsfordeling.

Sannsynligheten for at man mister karakter er $P(X \geq \lceil g d \rceil)$, der $\lceil \cdot \rceil$ er matematisk notasjon for å runde opp, f.eks. så er $\lceil 1.7 \rceil = 2$. Vi regner oss frem til sannsynligheten ved hjelp av den kumulative sannsynlighetsfordelingen.

\begin{equation*} P(X \geq \lceil g d \rceil) = 1 - P(X < \lceil g d \rceil) = 1 - (P(X \leq \lceil g d \rceil) - P(X = \lceil g d \rceil)). \end{equation*}

Eksempel
Om vi setter $p=0.02$ og antall dager lik $d=13$, så er

\begin{align*} P(X \geq \lceil (0.1) 13 \rceil) &= P(X \geq 2 ) \\ &= 1 - P(X < 2) \\ &= 1 - \left[ P(X = 0) + P(X = 1) \right] \\ &= 1 - \left[ \binom{13}{0} 0.02^{0} (1 - 0.02)^{13} + \binom{12}{1} 0.02^{1} (1 - 0.02)^{12} \right] \\ &= 1 - \left[ 1 (0.98)^{13} + 12 0.02 (0.98)^{12} \right] \\ &\approx 1 - \left[ 0.769 + 0.188 \right] \approx 0.0426455. \end{align*}

Hva bør fraværsgrensa være?

Dette er et verdispørsmål som matematikk alene ikke kan svare på, men la oss gjøre noen antagelser og undersøke hva en rimelig fraværsgrense som funksjon av antall dager $d$ bør være:

Anta at vi ønsker at $P(\text{ingen karakter}) \leq 0.001$, gitt et normalt udokumentert fravær.
Anta at noe udokumentert fravær vil forekomme, f.eks. at $p = 0.02$ eller lignende er normalnivå.

Da kan vi spørre “Hva er minste (strengeste) fraværsgrense $g$ vi kan ha som oppfyller $P(\text{ingen karakter}) \leq 0.001$?”

Svaret er gitt grafisk i figuren nedenfor, for tre ulike verdier av $p$ og ulike verdier av dager $d$ med et gitt fag.

Å velge $P(\text{ingen karakter})$ og $p$ er en skjønnsmessig vurdering. Uansett er konklusjonen at antall dager med faget spiller en viktig rolle. Desto lavere (strengere) fraværsgrense, desto høyere sannsynlighet for at man har uflaks og mister karaktergrunnlaget, selv når sannsynligheten for fravær $p$ i utgangspunktet er lav.

Personlig ville jeg heller straffet elever med nedsatt karakter dersom fraværet blir stort, heller enn å ta bort hele karaktergrunnlaget. Dette gir en mer glidende overgang, i motsetning til en hard statisk grense på 10%. Det er litt rart at politikerne ikke har valgt en annen mekanisme enn en statisk grense på 10% for alle fagene. Kanskje de selv hadde fravær? All matematikken vi har brukt i denne artikkelen er i hvert fall VGS-pensum.

Kode

# Python 3.12.9,
import numpy as np  # numpy: 2.1.3
import scipy as sp  # scipy: 1.15.1


def prob_lose_grade(num_days, prob_absence=0.01, limit_absence=1 / 10):
    """Probability of losing your grade, given a course with `num_days` days."""
    # Limit for losing grades, in days. Hitting this means losing the grade.
    days_limit = int(np.ceil(num_days * limit_absence))
    # Probability of absence is Binomial
    distr = sp.stats.binom(n=num_days, p=prob_absence)
    # P(X >= limit) = 1 - P(X < limit) = 1 - (CDF(limit) - P(X=limit))
    return float(1 - distr.cdf(days_limit) + distr.pmf(days_limit))


def find_limit(num_days, prob_absence=0.01, max_prob_lose_grade=0.001):
    """Find largest fraværsgrense (limit), such that the probability of
    losing grade is <= target_value."""

    def objective(limit):
        return prob_lose_grade(num_days, prob_absence, limit) - max_prob_lose_grade

    # The goal is to find the largest possible limit that ensures that the
    # return value of prob_lose_grade is smaller than or equal to target_value.
    optimal_limit = sp.optimize.bisect(objective, a=0, b=1) + 1e-8
    # print(f"The absence limit should be set to {optimal_limit:.4f}")

    prob_lose = prob_lose_grade(num_days, prob_absence, optimal_limit)
    # print(f"Then the prob of losing the grade is {prob_lose:.4f}")
    assert prob_lose < max_prob_lose_grade
    return float(optimal_limit)


if __name__ == "__main__":
    # Test case - single day
    assert np.isclose(prob_lose_grade(1, prob_absence=0.01, limit_absence=1 / 10), 0.01)

    # Test case - two days
    prob = 0.01 * 0.01 + 2 * 0.01 * (1 - 0.01)
    assert np.isclose(prob_lose_grade(2, prob_absence=0.01, limit_absence=1 / 10), prob)

    # Test case - tree days
    p = 0.01
    prob = p * p * p + 3 * p**2 * (1 - p) + 3 * p * (1 - p) ** 2
    assert np.isclose(prob_lose_grade(3, prob_absence=0.01, limit_absence=1 / 10), prob)

How do Voting Advice Applications work?

Tommy Odland — Mon, 07 Apr 2025 00:00:00 +0200

This is a translation of my Norwegian article about VAAs. The figures have not been translated, but are hopefully understandable from the context.

This article presents the mathematics behind voting advice applications (VAAs). VAAs are essentially questionnaires: a user expresses agreement or disagreement with statements about politics, and the VAAs tells the user what parties they align with.

This article consists of three main parts:

Part 1: VAAs and geometry
Part 2: How does a VAA explain the results?
Part 3: How should a VAA present questions to the user?

The article begins at a basic mathematical level and becomes progressively more challenging. The advantage of this approach is that there is something to learn for everyone regardless of level. The disadvantage is that the article does not have a single clear target audience.

PS: In 2023, I visualized the parties in the Norwegian municipal election. The focus of that article was insight into party politics. This article focuses more on the design of VAAs.

Part 1: VAAs and geometry

Opinions and matrices

To create a VAA, we ask political parties to which degree they agree with various statements. The results are quantified by translating responses like “Strongly agree” into numbers. The answers can then be visualized in a table with one row per statement and one column per party. Below, “Strongly agree” is translated to $2$ and “Strongly disagree” to $-2$.

The table forms a matrix $X$. The rows and columns have a geometric interpretation as vectors. A vector is a list of numbers, e.g., $(1, 2)$, and can be interpreted geometrically as an arrow or as a point.

Statements and parties

Statements as vectors

To visualize statements and parties as vectors, we must limit ourselves to two or three dimensions. Higher dimensions cannot be visualized.

Let’s study geometrically what the parties SV and Sp think about a selection of statements.

In the upper right of the figure, we find statements that both parties agree with. In the lower right, we find statements that SV agrees with, but Sp disagrees with. We can make similar observations for the other two quadrants.

Which statements are good and which are bad?

A good statement allows us to distinguish parties from each other:

In the figure above, “Build railway northward to Tromsø” (Norwegian: “Bygg jernbane nordover til Tromsø.”) is a poor statement. Regardless of whether you agree or disagree, the statement does not clarify which of the two parties you are politically closest to. Trivial statements, such as “A just society is good,” are agreed upon by all parties. Therefore, they provide no information. The same applies to trivial statements that all parties disagree with, such as “The county municipality should be abolished” (Norwegian: “Fylkeskommunen bør avskaffes.”) in the figure above.
“Drop the ferry-free E39 along the west coast” (Norwegian: “Dropp ferjefri E39 langs Vestlandet.”) is a good statement because it discriminates between the parties. If you agree, you are politically closer to SV. If you disagree, you are closer to Sp.

In two dimensions (two parties), statements are good if they are far from the subspace spanned by the vector $\boldsymbol{e} = (1, 1)$, i.e., the diagonal line $y=x$. At the end of this article, we will generalize this idea to construct an algorithm that finds a good ordering of questions.

Parties as vectors

Above, we considered the rows (statements) in the table as vectors. Let’s now look at the complementary vector space where the columns (parties) are vectors. We choose two random statements: one about wolves (“Det skytes for mye ulv her i landet.” / “Too many wolves are shot in this country.”) and one about state ownership in companies (“Staten bør eie flere bedrifter.” / “The state should own more companies.”).

Two parties are similar if they are close to each other in the space of opinions in the figure above. For example, we can measure distance as the distance on the horizontal axis plus the distance on the vertical axis (a mathematician might call this “Manhattan distance” or $p=1$ norm).

Parties R and AP are close to each other. The distance is $0 + 1 = 1$.
Parties R and V are far from each other. The distance is $3 + 4 = 7$.

The distance in this space is the key to how a VAA determines which parties you most agree with when you take the test. This is what we will cover in part 2.

Part 2: How does a VAA explain the results?

Explanation: ranking parties by distance

Above, we saw that party similarity can be quantified by looking at the distance between parties. We visualized this in two dimensions, but the calculation is the same in any number of dimensions.

Here is a matrix with the numbers from the first figure in this article:

\begin{equation*} X = \begin{bmatrix} \text{AP} & \text{Sp} & \text{Frp} & \text{H}\\ -1 & 1 & -2 & -1\\ 2 & 1 & -2 & -2\\ -2 & -2 & 2 & 2\\ 1 & -1 & -2 & -2\\ 1 & 2 & 2 & -2\\ -1 & -2 & -2 & 2 \end{bmatrix} \end{equation*}

If you take the VAA and your opinions are the vector $\boldsymbol{y} = (-1, -2, 2, -2, 0, 0)$, then your distance to AP is $13$, to Sp $14$, to Frp $5$, and to H $4$. In other words, you are likely on the political right and should consider voting for Frp or H.

The distance $d_j$ between your opinions $\boldsymbol{y}$ and a party $j$’s opinions is given by the sum of the distances for each statement $i$:

\begin{equation*} d_j = \sum_i | X_{ij} - y_i | \end{equation*}

Distances, $p$-norms, and weighting

We can generalize to an arbitrary $p$-norm and add a weighting $w_i$:

\begin{equation*} d_j = \left( \sum_i w_i | X_{ij} - y_i |^p \right)^{(1/p)} \end{equation*}

The weights $w_i$ are typically equal to $1$ for most statements, while $w_i = 2$ for core issues or issues that are particularly important to you. For issues where you have no opinion, we can set $w_i = 0$. The value chosen for $p$ depends on how you want to measure distance.

If you believe that $\boldsymbol{y}$ is closer to party $A$ in the example below, you should choose $p=2$. If you think $\boldsymbol{y}$ is closer to $B$, you should choose $p=1$.

\begin{align*} \boldsymbol{y} &= (0, 0, 0, 0) \\ A &= (1, 1, 1, 1) \\ B &= (0, 0, 0, 3) \end{align*}

This is a subjective assessment, and personally, I lean toward $p=1$. Both of NRK and VG do the same; they both use $p=1$ in their VAAs.

The calculations ultimately result in a list of parties, sorted by distance:

1. H   (distance 4 )
2. Frp (distance 5 )
3. AP  (distance 13)
4. Sp  (distance 14)

Such a list, sorted by distance, is one possible way to explain the user’s results. But it’s not the only option; the next section covers another type of explanation.

Explanation: Your position as a combination of party positions

The parties’ vectors set directions in the space of opinions. After taking a VAA, you also get a position in this space. (“Ditt standpunkt” means “Your standpoint” or “Your position” in English.)

In the figure above, your position is $\boldsymbol{y} = (1, -2)$.

One way to reach your position with vector arithmetic is to take MDG’s position and add KrF’s. This can be observed geometrically in the figure above, or algebraically:

\begin{align} \text{MDG} + \text{KrF} = \boldsymbol{y} \\ (2, -1) + (-1, -1) &= (1, -2) \end{align}

Drafting an optimization model

Here is an alternative idea to a distance-based explanation: we express your position as a combination of party positions. We are looking for a combination of parties that describes you by taking us as close to your position as possible. In other words, we are search for a vector $\boldsymbol{\beta}$ that solves the optimization problem

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \|. \end{align}

This model has some practical challenges:

It allows complicated explanations in the form of long vectors $\boldsymbol{\beta}$. For example, we can reach $\boldsymbol{y}$ through the linear combination
\begin{align} \text{V} + \text{R} + \text{H} + \text{Ap} + \text{MDG} = \boldsymbol{y}. \end{align}
The model allows negative directions because $\boldsymbol{\beta}$ can consist of negative numbers. This is difficult to interpret. For example, $-\text{R} = \boldsymbol{y}$ in the example above, but you naturally want to know who you agree with, rather than being told that you disagree with the party $\text{R}$.
The model allows explanations where the sum of the parties’ contributions $\boldsymbol{\beta}$ exceeds $1$. This is the case in $\text{MDG} + \text{KrF} = \boldsymbol{y}$ and in the long explanation above. Again, it is difficult to interpret such an answer.

All these problems can be solved if we limit ourselves to convex combinations.

Final optimization model

We fix the problems in the above model by adding some constraints. Rather than look for any linear combination of parties, we look for the shortest convex combination:

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \| + \epsilon \| \boldsymbol{\beta} \| \\ & \text{subject to} && \beta_j \geq 0 && \\ & && \sum_j \beta_j = 1. && \end{align}

The regularization term $\epsilon \| \boldsymbol{\beta} \|$ means that we prefer simple (short) explanations. The two constraints on $\boldsymbol{\beta}$ are the definition of a convex combination, and ensure that we avoid negative directions and sums that exceed $1$. The sum $\sum_j \beta_j$ must equal $1$, rather than be less than $1$, because if $\boldsymbol{y} = (0, 1)$, $A = (0, 1)$, and $B = (0, 2)$, we expect $\boldsymbol{\beta} = (1, 0)$ as the answer, not $\boldsymbol{\beta} = (0, 0.5)$.

With the $2$-norm on both terms in the objective function, the solution to the optimization problem becomes:

\begin{align} 0.69 \, \text{V} + 0.23 \, \text{H} + 0.08 \, \text{Frp} = \boldsymbol{y}. \end{align}

If you go back to the figure and compare the geometry with this solution, you’ll see that the solution makes sense. That said, there are often no single correct approach in mathematical modeling, and other variations of such a model might be appropriate.

The optimization model on a larger example

Let’s run the optimization model on the same example as before:

\begin{align*} [X \mid \boldsymbol{y}] = \begin{bmatrix} \text{AP} & \text{Sp} & \text{Frp} & \text{H} & \boldsymbol{y} \\ -1 & 1 & -2 & -1 & -1\\ 2 & 1 & -2 & -2 & -2\\ -2 & -2 & 2 & 2 & 2\\ 1 & -1 & -2 & -2 & -2 \\ 1 & 2 & 2 & -2 & 0\\ -1 & -2 & -2 & 2 & 0 \end{bmatrix} \end{align*}

The answer is given below and corresponds well with the ranking we obtained earlier.

1. H   (51%)
2. Frp (45%)
4. Sp  (4%)
3. AP  (0%)

The difference between distance-based and combination-based explanation

We have seen two ways to explain a user’s results in a VAA:

Distance-based explanation - ranks parties by distance from the user
Combination-based explanation - the user is expressed as the nearest convex combination of parties

The difference between the explanations is illustrated in the figure below.

Distance-based explanation says that $\boldsymbol{y}$ is closest to $B$, then $A$, and finally $C$. Distance-based explanation is mutually independent; the distance to $B$ does not affect the distance to $A$. If $B$ had disappeared, the distance to $A$ would remain unchanged.
Combination-based explanation expresses the position $\boldsymbol{y}$ as a convex combination. The answer in the figure above becomes $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0, 0.7, 0.3)$. The optimization model’s explanation is mutually dependent. If $B$ had disappeared, the explanation would have relied on $A$, but when $B$ exists, $A$ is not needed at all to explain $\boldsymbol{y}$.

The figure below shows an example where the explanations become completely different.

Measured by distance, $C$ is closer to $\boldsymbol{y}$ than both $A$ and $B$. Yet $\boldsymbol{y}$ is explained using $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0.5, 0.5, 0)$ without using $C$. Shortest distance tells you which single party is closest. The optimization model tells you which coalition of parties is closest to you.

Part 3: How should a VAA present questions to the user?

In this section, we present an algorithm that selects statements that discriminate well between parties. There may be two reasons for wanting this:

We want to use a selection of statements. If parties have answered many questions, we may want to limit ourselves to a smaller selection in the VAA. How can we choose a small subset of good statements that differentiate between parties?
Early termination. We want the user to be able to end the VAA after any number of questions. How do we ensure that the user has been presented with a diverse set of statements even if they do not answer every question?

To motivate the algorithm, we start by looking at the same example as earlier in the article.

If we ask about a railway to Tromsø or whether the county municipality should be abolished, we gather no information because the user’s answer does not discriminate between parties. The subspace spanned by the vector $\boldsymbol{e} = (1, 1)$ consists of such trivial statements. The best statement to present to the user is therefore “Drop the ferry-free E39 along the west coast,” since this statement lies furthest from $\operatorname{span} (\{ \boldsymbol{e} \} )$.

Algorithm for selecting statements in sequence

Let’s generalize from two dimensions to an arbitrary number of dimensions:

Start with the trivial subspace spanned by the statement $\boldsymbol{e} = (1, 1, \ldots, 1)$.
The first statement selected is the one that is furthest from $\operatorname{span} (\{ \boldsymbol{e} \} )$. This is $\boldsymbol{v}_1 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, where $P_S$ is a projection matrix that projects a vector onto $S$. The formula for such a projection matrix is $P_S = S S^{+} = S (S^T S)^{-1} S^T$, where the columns of the matrix $S$ are the vectors that span the space we want to project onto (here just $\boldsymbol{e}$).
The next statement selected should be far from both the trivial subspace $\boldsymbol{e}$ and far from the previously selected statement $\boldsymbol{v}_1$. So we choose the statement that is furthest from $\operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1 \} )$. In mathematical notation, this becomes $\boldsymbol{v}_2 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, where $S$ in this iteration is a matrix with columns $\boldsymbol{e}$ and $\boldsymbol{v}_1$.
We continue like this, selecting statements that are furthest from those we have already chosen. If $\operatorname{rank} \operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1, \ldots, \boldsymbol{v}_n \} )$ becomes equal to the dimensionality of the space (number of parties), then all remaining statements are in the space. Then the distance $\| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2 \approx 0$ for all remaining statements $\boldsymbol{v}_j$. The solution is to treat the selected statements as a FIFO queue: we remove the first selected statement $\boldsymbol{v}_1$ and try again. Then the next statement we select is always far from the most recently selected statements.

Python code for the algorithm is included at the end of the article. The figure below illustrates the concept in three dimensions. The next statement selected is always the one that is furthest from the space spanned by the trivial direction and previously selected statements. (In english, the first figure says “Start with (1, 1, 1)”, the next says “Choose first statement” and the third says “Choose second statement”.)

Results - bad statements

Let’s look at an example with statements that are similar to each other.

The first two statements are very similar. The third statement about Erna is on the opposite side of the vector space but still does not provide much new information because it is close to the space spanned by the first two statements. In other words: the question about Erna thus reveals little new information, despite the statement being on the opposite side of the vector space.

Results - good statements

Here is a selection of statements that are different from each other, identified by the algorithm above.

The first statement is far from trivial because half of the parties disagree and the other half agree. The second statement is answered differently by six parties and differs from both the trivial statement and the first statement. The algorithm continues to select statements that are consistently different from what the VAA user has already been presented with.

Summary and references

Voting advice applications quantify political opinions. Party responses can be represented as a matrix, where the rows (statements) and columns (parties) can be interpreted as vectors. Such a geometric interpretation gives us valuable insight: we can define what it means for a person to be close to a party, and we can formalize what it means for a statement to discriminate between parties.

The idea of viewing parties as vectors and ranking users by proximity is obvious, and for the 2019 municipal election, NRK used $p=1$ as a metric. (The fact that NRKbeta’s article in the link above describes middle school mathematics in phrases like “explain the calculation to me who doesn’t have math anxiety” and “I get a headache!” is sad.) The idea of describing a user as a convex combination of parties is my own. In practice, it is more difficult to implement in code and more challenging for an average user to understand. Simple algorithms have inherent value, and distance calculation provides a simple and understandable explanation.

The idea of viewing statements as vectors and selecting statements that are far from each other is my own. The inspiration comes from QR decomposition with pivoting, which is an algorithm for orthogonalizing a matrix. The pivot columns in QR can be seen as a greedy approximation of D-optimality in optimal experimental design, which maximizes the volume of a high-dimensional parallelepiped. The differences between my algorithm and QR decomposition are that (1) we always stay away from the trivial statement $\boldsymbol{e} = \boldsymbol{1}$ and that (2) we use a FIFO queue. Again I believe simplicitiy has inherent value: I like that this algorithm does not adapt the sequence of questions to what the user responded, since I do not like the idea of a personalized VAA.

This article focused mostly on mathematics, but a VAA must of course contain politically relevant statements, good graphic design, potentially the ability to skip statements or weight statements up, subjective analyses of the results, and so on.

Code

Here is a simple implementation of the algorithm that finds statements that are far from each other. I have not emphasized runtime, numerical stability, caching of calculations, etc. The code is certainly good enough to showcase the idea, but for larger datasets, it should be rewritten.

import collections  # Python 3.12
import numpy as np  # NumPy 2.1.3


def away_from_last_k(A, k=None, verbose=False):
    """Yields the column indices of A that are as far away from a continually
    updated subspace spanned by (1, 1, 1, ...) and a First-In-First-Out (FIFO)
    queue of length k. When a column index is yielded, it is added to the queue.

    Parameters
    ----------
    A : np.ndarray
        A matrix (2D array) with columns [v1 | v2 | ... ].
    k : int, optional
        Numbers of elements to keep in the FIFO queue. None means no limit.
    verbose : bool, optional
        Whether to print information. The default is False.

    Examples
    --------
    >>> A = np.array([[3, 0, 0], [0, 2, 0], [0, 0, 1], [2.5, 0, 0]]).T
    >>> list(away_from_last_k(A, k=None))
    [0, 1, 3, 2]
    >>> list(away_from_last_k(A, k=0)) #  Distance from span({(1, 1, 1)}) only
    [0, 3, 1, 2]
    >>> list(away_from_last_k(A, k=1))
    [0, 1, 3, 2]
    """
    subspace = collections.deque([], maxlen=k)  # Vector in FIFO subspace
    ones = np.ones(A.shape[0])  # Vector of ones, always in subspace
    remaining = dict(enumerate(A.T))  # Remaining column indices of A

    def distance_from_subspace(tuple_):
        idx, v = tuple_
        S = np.vstack([ones] + list(subspace)).T  # Full space as col. mat.
        P_S = S @ np.linalg.pinv(S)  # Projection matrix onto span(S)
        return np.linalg.norm(P_S @ v - v)  # Distance between v and span(S)

    while remaining:
        if verbose:
            inds = list(remaining.keys())
            print(f"\nCol idx of {len(remaining)} remaining vectors: {inds}")
            print(f"Vectors in subspace (excl. ones): {len(subspace)}")

        # Get idx of col vector furthest from S = [ones | subspace]
        idx, _ = max(remaining.items(), key=distance_from_subspace)
        maxdist = distance_from_subspace((idx, remaining[idx]))
        if np.isclose(maxdist, 0):
            if verbose:
                print("Distance was 0. Popping from subspace.")
            subspace.popleft()
            continue

        if verbose:
            print(f"Col idx {idx} furthest from subspace. Dist: {maxdist:.6f}")
        subspace.append(remaining.pop(idx))
        yield idx

Hvordan fungerer en valgomat?

Tommy Odland — Mon, 07 Apr 2025 00:00:00 +0200

Denne artikkelen presenterer matematikken bak valgomater. Den består av tre hoveddeler:

Del 1: Valgomater og geometri
Del 2: Hvordan forklarer en valgomat resultatene?
Del 3: Hvordan bør en valgomat presentere spørsmål for brukeren?

Artikkelen begynner på lavt matematisk nivå og blir progressivt mer krevende. En fordel med dette er at det er noe å lære for alle, uansett nivå. Ulempen er at artikkelen ikke har én tydelig målgruppe.

PS: I 2023 visualiserte jeg partiene i kommunevalget. Fokuset i den artikkelen var innsikt i partienes politikk. Denne artikkelen retter seg mer mot design av valgomater.

PSS: Idéene i denne artikkelen er implementert i min egen Valgomat for Stortingsvalget 2025.

Del 1: Valgomater og geometri

Meninger og matriser

For å lage en valgomat spør man politiske partier hvor enige de er i ulike påstander. Resultatet kvantifiseres ved at svar som “Helt enig!” blir oversatt til tall. Svarene kan deretter visualiseres i en tabell med en rad per påstand og en kolonne per parti. Nedenfor er “Helt enig” oversatt til $2$ og “Helt uenig” oversatt til $-2$.

Tabellen utgjør en matrise $X$. Radene og kolonnene har en geometrisk tolkning som vektorer. En vektor er en liste med tall, e.g. $(1, 2)$, og kan tolkes geometrisk som en pil eller som et punkt.

Påstander og partier

Påstander som vektorer

For å visualisere påstander og partier som vektorer må vi holde oss til to eller tre dimensjoner. Høyere dimensjoner lar seg ikke visualisere.

La oss studere geometrisk hva partiene SV og Sp mener om et utvalg påstander.

Oppe til høyre i figuren finner vi påstander som begge partiene er enige i. Nede til høyre finner vi påstander som SV er enige i, men som Sp er uenige i. Vi kan gjøre tilsvarende observasjoner for de to andre kvadrantene.

Hvilke påstander er gode og dårlige?

En god påstand lar oss skille partiene fra hverandre:

I figuren ovenfor er “Bygg jernbane nordover til Tromsø” en dårlig påstand. Uansett om du er enig eller uenig, avklarer ikke påstanden hvilket av de to partiene du er nærmest politisk. Trivielle påstander, som “Et rettferdig samfunn er bra”, er alle partiene enige i. Derfor gir de heller ingen informasjon. Det samme gjelder trivielle påstander som alle partiene er uenige i, for eksempel “Fylkeskommunen bør avskaffes” i figuren ovenfor.
“Dropp ferjefri E39 langs Vestlandet” er en god påstand fordi den diskriminerer mellom partiene. Om du er enig ligger du nærmere SV politisk. Er du derimot uenig ligger du nær Sp.

I to dimensjoner (to partier) er påstander gode dersom de ligger langt fra underrommet som spennes av vektoren $\boldsymbol{e} = (1, 1)$, altså den diagonale linja $y=x$. På slutten av denne artikkelen skal vi generalisere denne idéen for å konstruere en algoritme som finner en god rekkefølge å stille spørsmål på.

Partier som vektorer

Ovenfor betraktet vi radene (påstandene) i tabellen som vektorer. La oss nå se på det komplementære vektorrommet der kolonnene (partiene) er vektorer. Vi velger to tilfeldige påstander: en om ulv og en om statens eierskap i bedrifter.

To partier ligner hverandre dersom de ligger nær hverandre i meningsrommet. Vi kan for eksempel måle avstand som distanse på den horisontale aksen pluss distanse på den vertikale aksen.

Partiene R og AP er nær hverandre. Distansen er $0 + 1 = 1$.
Partiene R og V er langt fra hverandre. Distansen er $3 + 4 = 7$.

Distansen i dette rommet er nøkkelen til hvordan en valgomat avgjør hvilke partier du er mest enig med når du tar valgomaten. Det er dette vi nå skal ta for oss i del 2.

Del 2: Hvordan forklarer en valgomat resultatene?

Forklaring: rangering av partiene etter avstand

Ovenfor så vi at partienes likhet kan kvantifiseres ved å se på avstanden mellom partiene. Vi visualiserte dette i to dimensjoner, men beregningen er den samme i et vilkårlig antall dimensjoner.

Her er en matrise med tallene fra den første figuren i denne artikkelen:

Hvis du tar valgomaten og dine meninger er vektoren $\boldsymbol{y} = (-1, -2, 2, -2, 0, 0)$, så blir din avstand til AP $13$, Sp $14$, Frp $5$ og H $4$. Du ligger med andre ord sannsynligvis på høyresiden politisk, og bør vurdere å stemme Frp eller H.

Distansen $d_j$ mellom dine meninger $\boldsymbol{y}$ og et parti $j$ sine meninger er gitt av summen av avstandene mellom hver påstand $i$:

\begin{equation*} d_j = \sum_i | X_{ij} - y_i | \end{equation*}

Avstander, $p$-normer og vekting

Vi kan generalisere til en vilkårlig $p$-norm og legge til en vekting $w_i$:

\begin{equation*} d_j = \left( \sum_i w_i | X_{ij} - y_i |^p \right)^{(1/p)} \end{equation*}

Vektene $w_i$ er gjerne lik $1$ for de fleste påstander, mens $w_i = 2$ for kjernesaker eller saker som er spesielt viktige for deg. For saker der du ikke har noen mening kan vi sette $w_i = 0$. Verdien som velges for $p$ er avhengig av hvordan man vil måle distanse.

Dersom du mener at $\boldsymbol{y}$ er nærmere partiet $A$ i eksempelet nedenfor bør du velge $p=2$. Mener du derimot at $\boldsymbol{y}$ er nærmere $B$ bør du velge $p=1$.

\begin{align*} \boldsymbol{y} &= (0, 0, 0, 0) \\ A &= (1, 1, 1, 1) \\ B &= (0, 0, 0, 3) \end{align*}

Dette er en subjektiv vurdering, og personlig lener jeg mot $p=1$. Det samme gjør både NRK og VG; begge bruker $p=1$ i sine valgomater.

Beregningene resulterer til slutt i en liste av partier, sortert etter distanse:

1. H   (avstand 4 )
2. Frp (avstand 5 )
3. AP  (avstand 13)
4. Sp  (avstand 14)

En slik liste, sortert etter avstand, er én mulig måte å forklare brukerens resultater på. Men det er ikke den eneste; neste seksjon tar for seg en annen type forklaring.

Forklaring: Ditt standpunkt som en kombinasjon av partienes standpunkter

Partienes vektorer setter retning i rommet av meninger. Etter å ha tatt en valgomat får du også en plassering i dette rommet.

I figuren ovenfor er ditt standpunkt $\boldsymbol{y} = (1, -2)$.

Én måte å nå ditt standpunkt på med vektorregning er å ta MDGs standpunkt og legge på KrFs. Dette kan observeres geometrisk i figuren ovenfor, eller algebraisk:

\begin{align} \text{MDG} + \text{KrF} = \boldsymbol{y} \\ (2, -1) + (-1, -1) &= (1, -2) \end{align}

Utkast til optimeringsmodell

Her er en alternativ idé til distansebasert forklaring: vi uttrykker ditt standpunkt som en kombinasjon av partienes standpunkter. Vi leter da etter en kombinasjon av partier som beskriver deg ved at de tar oss så nært ditt standpunkt som mulig. Med andre ord er vi på jakt etter en vektor $\boldsymbol{\beta}$ som løser optimeringsproblemet

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \|. \end{align}

Denne modellen har noen praktiske utfordringer:

Den tillater kompliserte forklaringer i form av lange vektorer $\boldsymbol{\beta}$. For eksempel kan vi nå frem til $\boldsymbol{y}$ ved den lineære kombinasjonen
\begin{align} \text{V} + \text{R} + \text{H} + \text{Ap} + \text{MDG} = \boldsymbol{y}. \end{align}
Modellen tillater negative retninger fordi $\boldsymbol{\beta}$ kan bestå av negative tall. Dette er vanskelig å tolke. For eksempel er $-R = \boldsymbol{y}$ i eksempelet ovenfor, men du ønsker naturligvis å vite hvem du er enig med, heller enn å få vite at du er uenig med Rødt.
Modellen tillater forklaringer der summen av partienes bidrag $\boldsymbol{\beta}$ overskrider $1$. Dette er tilfellet i $\text{MDG} + \text{KrF} = \boldsymbol{y}$ og i den lange forklaringen ovenfor. Igjen er det vanskelig å tolke et slikt svar.

Alle disse problemene kan løses om vi begrenser oss til konvekse kombinasjoner.

Endelig optimeringsmodell

Vi fikser problemene til modellen ovenfor ved å utvide den. Heller enn å leter etter en hvilken som helst lineær kombinasjon av partier, leter vi etter den korteste konvekse kombinasjonen:

\begin{align} & \underset{\boldsymbol{\beta}}{\text{minimize}} && \| X \boldsymbol{\beta} - \boldsymbol{y} \| + \alpha \| \boldsymbol{\beta} \| \\ & \text{subject to} && \beta_j \geq 0 && \\ & && \sum_j \beta_j = 1. && \end{align}

Begge noremen ovenfor er $2$-normer. Setter vi $\alpha = \epsilon$ (et lite positivt tall) i regulariseringsleddet, så vil modellen foretrekke løsninger der elementene i $\boldsymbol{\beta}$ jevnt over er små. Dersom $\alpha \to \infty$ så velges $\boldsymbol{\beta} \propto \boldsymbol{1}$, dersom $\alpha = -\epsilon$ vil modellen foretrekke løsninger som bruker så få partier som mulig, og dersom $\alpha \to -\infty$ er vi tilbake til en typisk valgomat som velger ett parti. Merk at når $\alpha < 0$ så er ikke problemet konvekst. Vi setter $\alpha = \epsilon$ i resten av artikkelen.

De to bibetingelsene på $\boldsymbol{\beta}$ er definisjonen på en konveks kombinasjon, og gjør at vi unngår negative retninger og summer som overskrider $1$. Summen $\sum_j \beta_j$ må være lik $1$, heller enn mindre enn $1$, fordi dersom $\boldsymbol{y} = (0, 1)$, $A = (0, 1)$ og $B = (0, 2)$ så forventer vi $\boldsymbol{\beta} = (1, 0)$ som svar, ikke $\boldsymbol{\beta} = (0, 0.5)$.

Med $2$-norm på begge leddene i objektivfunksjonen blir løsningen på optimeringsproblemet:

\begin{align} 0.69 \, \text{V} + 0.23 \, \text{H} + 0.08 \, \text{Frp} = \boldsymbol{y}. \end{align}

Om du går tilbake til figuren og sammenligner geometrien med denne løsningen, ser du at løsningen er fornuftig. Når det er sagt er det ingen absolutt fasit i matematisk modellering og andre variasjoner av en slik modell kan være hensiktsmessige.

Optimeringsmodellen på et større eksempel

La oss kjøre optimeringsmodellen på samme eksempel som tidligere:

Svaret er gitt nedenfor, og samsvarer godt med rangeringen som vi fikk tidligere.

1. H   (51 %)
2. Frp (45 %)
4. Sp  ( 4 %)
3. AP  ( 0 %)

Forskjellen på distansebasert og kombinasjonsbasert forklaring

Vi har sett to måter å forklare en brukers resultater på i en valgomat:

Distansebasert forklaring - rangerer partiene etter distanse fra brukeren. Svarer på spørsmålet “Hvilket enkeltparti er nærmest meg?”
Kombinasjonsbasert forklaring - brukeren uttrykkes som den nærmeste konvekse kombinasjonen av partiene. Svarer på spørsmålet “Hvilken koalisjon av partier er nærmest meg?”

Forskjellen mellom forklaringene er illustrert i figuren nedenfor.

Distansebasert forklaring sier at $\boldsymbol{y}$ er nærmest $B$, deretter $A$ og til slutt $C$. Distansebasert forklaring er innbyrdes uavhengig; avstanden til $B$ påvirker ikke avstanden til $A$. Dersom $B$ hadde forsvunnet ville avstanden til $A$ vært uendret.
Kombinasjonsbasert forklaring uttrykker standpunktet $\boldsymbol{y}$ som en konveks kombinasjon. Svaret i figuren ovenfor blir $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0, 0.7, 0.3)$. Optimeringsmodellens forklaring er innbyrdes avhengig. Dersom $B$ hadde forsvunnet ville forklaringen lent seg på $A$, men når $B$ finnes trengs ikke $A$ i det hele tatt for å forklare $\boldsymbol{y}$.

Figuren nedenfor viser et eksempel der forklaringene blir helt ulike.

Målt i distanse er $C$ nærmere $\boldsymbol{y}$ enn både $A$ og $B$. Likevel forklares $\boldsymbol{y}$ ved hjelp av $\boldsymbol{\beta} = (\beta_A, \beta_B, \beta_C) = (0.5, 0.5, 0)$ uten å bruke $C$. Korteste distanse forteller deg hvilket enkeltparti som er nærmest. Optimeringsmodellen forteller deg hvilken kombinasjon av partier som står deg nærmest.

Del 3: Hvordan bør en valgomat presentere spørsmål for brukeren?

I denne seksjonen presenterer vi en algoritme som velger påstander som diskriminerer godt mellom partiene. Det kan være to grunner til at man ønsker dette:

Man skal bruke et utvalg påstander. Om partiene har svart på mange spørsmål, ønsker vi gjerne å begrense oss til et utvalg. Hvordan velger vi gode påstander som differensierer mellom partiene?
Tidlig avslutning. Man ønsker at brukeren skal kunne avslutte valgomaten etter et vilkårlig antall spørsmål. Hvordan garanterer vi at brukeren har blitt presentert for et allsidig sett med påstander?

For å motivere algoritmen begynner vi med å se på samme eksempel som tidligere i artikkelen.

Spør vi om jernbane til Tromsø eller om fylkeskommunen bør avskaffes, innhenter vi ingen informasjon fordi brukerens svar ikke diskriminerer mellom partiene. Underrommet som spennes av vektoren $\boldsymbol{e} = (1, 1)$ består av slike trivielle påstander. Den beste påstanden å presentere for brukeren er derimot “Dropp ferjefri E39 langs Vestlandet”, ettersom denne påstanden ligger lengst i fra $\operatorname{span} (\{ \boldsymbol{e} \} )$.

Algoritme for å velge påstander i rekkefølge

La oss generalisere fra to dimensjoner til et vilkårlig antall dimensjoner:

Start med det trivielle underrommet som spennes av påstanden $\boldsymbol{e} = (1, 1, \ldots, 1)$.
Den første påstanden som velges er den som er lengst fra $\operatorname{span} (\{ \boldsymbol{e} \} )$. Dette er $\boldsymbol{v}_1 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, der $P_S$ er en projeksjonsmatrise som projiserer en vektor på $S$. Formelen for en slik projeksjonsmatrise er $P_S = S S^{+} = S (S^T S)^{-1} S^T$, der kolonnene i matrisen $S$ er vektorene som spenner rommet vi ønsker å projisere på (her bare $\boldsymbol{e}$).
Den neste påstanden som velges skal være langt fra både det trivielle underrommet $\boldsymbol{e}$, og langt fra den forrige valgte påstanden $\boldsymbol{v}_1$. Vi velger altså påstanden som er lengst fra $\operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1 \} )$. I matematisk notasjon blir dette $\boldsymbol{v}_2 = \arg \max_j \| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2$, der $S$ i denne iterasjonen er en matrise med kolonner $\boldsymbol{e}$ og $\boldsymbol{v}_1$.
Slik fortsetter vi og velger påstander som er lengst fra de vi allerede har valgt. Dersom $\operatorname{rank} \operatorname{span} (\{ \boldsymbol{e}, \boldsymbol{v}_1, \ldots, \boldsymbol{v}_n \} )$ blir lik dimensjonaliteten til rommet (antall partier), så er alle gjenværende påstander i rommet. Da blir distansen $\| P_S \boldsymbol{v}_j - \boldsymbol{v}_j \|_2 \approx 0$ for alle gjenværende påstander $\boldsymbol{v}_j$. Løsningen er å behandle de valgte påstandene som en FIFO-kø: vi fjerner den første valgte påstanden $\boldsymbol{v}_1$ og prøver på nytt. Da er neste påstand vi velger alltid langt fra de siste valgte påstandene.

Python-kode for algoritmen er vedlagt nederst i artikkelen. Figuren nedenfor illustrerer konseptet i tre dimensjoner. Den neste påstanden som velges er alltid den som er lengst i fra rommet som spennes av den trivielle retningen og tidligere valgte påstander.

Resultater - dårlige påstander

La oss se på et eksempel med påstander som innbyrdes ligner på hverandre.

De første to påstandene er veldig like. Den tredje påstanden om Erna er på motsatt side av vektorrommet, men gir likevel ikke mye ny informasjon fordi det er nær rommet som spennes av de første to spørsmålene. Spørsmålet om Erna avdekker altså lite ny informasjon, på tross av at påstanden er på motsatt side av vektorrommet.

Resultater - gode påstander

Her er et utvalg påstander som er ulike hverandre, avdekket av algoritmen ovenfor.

Den første påstanden er langt fra triviell, fordi halvparten av partiene er uenige og den andre halvparten er enige. Den andre påstanden er ulikt besvart av seks partier, og skiller seg både fra den trivielle påstanden og den første påstanden. Slik fortsetter algoritmen med å velge påstander som hele tiden er annerledes enn det brukeren av valgomaten har blitt presentert for.

Oppsummering og referanser

Valgomater kvantifiserer politiske meninger. Partienes svar kan representeres som en matrise, der radene (påstander) og kolonnene (partier) kan tolkes vektorer. En slik geometrisk tolkning gir oss verdifull innsikt: vi kan definere hva det vil si at en person er nær et parti og vi kan formalisere hva det vil si at en påstand diskriminerer mellom partier.

Idéen som å se på partier som vektorer og rangere brukeren etter nærhet er rimelig åpenbar, og for kommunevalget 2019 brukte NRK $p=1$ som metrikk. (At NRKbetas artikkel pakker ungdomsskolematematikk inn i fraser som “forklar utregningen for meg som ikke har matteskrekk” og “Jeg får vondt i hodet!” er sørgelig.) Idéen om å beskrive en bruker som en konveks kombinasjon av partier er min egen. I praksis er det vanskeligere å implementere i kode, samt mer krevende å forstå for en gjennomsnittlig bruker. Enkle algoritmer har iboende verdi og distanse-beregning gir en enkel og forståelig forklaring.

Idéen om å se på påstander som vektorer og velge ut påstander som er langt fra hverandre er min egen. Inspirasjonen kommer fra QR-dekomposisjon med pivotering, som er en algoritme for å ortogonalisere en matrise. Pivot-kolonnene i QR kan sees på som en grådig approksimasjon av D-optimalitet i optimal eksperimentdesign, som maksimerer volumet av en høydimensjonell parallellepiped. Forskjellene mellom min algoritme og QR-dekomposisjon er at (1) vi alltid holder oss borte fra den trivielle påstanden $\boldsymbol{e} = \boldsymbol{1}$ og at (2) vi bruker en FIFO-kø.

Denne artikkelen fokuserte mest på matematikk, men en valgomat må selvsagt ha politisk relevante påstander, godt grafisk design, potensielt mulighet for å hoppe over påstander eller vekte påstander opp, subjektive analyser av resultatene, og så videre.

Kode

Her er en enkel implementasjon av algoritmen som finner påstander som er langt fra hverandre. Jeg har ikke vektlagt kjøretid, numerisk stabilitet, caching av beregninger, etc. Koden er absolutt god nok for bruksområdet, men for større datasett bør man skrive om.

import collections  # Python 3.12
import numpy as np  # NumPy 2.1.3


def away_from_last_k(A, k=None, verbose=False):
    """Yields the column indices of A that are as far away from a continually
    updated subspace spanned by (1, 1, 1, ...) and a First-In-First-Out (FIFO)
    queue of length k. When a column index is yielded, it is added to the queue.

    Parameters
    ----------
    A : np.ndarray
        A matrix (2D array) with columns [v1 | v2 | ... ].
    k : int, optional
        Numbers of elements to keep in the FIFO queue. None means no limit.
    verbose : bool, optional
        Whether to print information. The default is False.

    Examples
    --------
    >>> A = np.array([[3, 0, 0], [0, 2, 0], [0, 0, 1], [2.5, 0, 0]]).T
    >>> list(away_from_last_k(A, k=None))
    [0, 1, 3, 2]
    >>> list(away_from_last_k(A, k=0)) #  Distance from span({(1, 1, 1)}) only
    [0, 3, 1, 2]
    >>> list(away_from_last_k(A, k=1))
    [0, 1, 3, 2]
    """
    subspace = collections.deque([], maxlen=k)  # Vectors in FIFO subspace
    ones = np.ones(A.shape[0])  # Vector of ones, always in subspace
    remaining = dict(enumerate(A.T))  # Remaining column indices of A

    def distance_from_subspace(tuple_):
        idx, v = tuple_
        S = np.vstack([ones] + list(subspace)).T  # Full space as col. mat.
        P_S = S @ np.linalg.pinv(S)  # Projection matrix onto span(S)
        return np.linalg.norm(P_S @ v - v)  # Distance between v and span(S)

    while remaining:
        if verbose:
            inds = list(remaining.keys())
            print(f"\nCol idx of {len(remaining)} remaining vectors: {inds}")
            print(f"Vectors in subspace (excl. ones): {len(subspace)}")

        # Get idx of col vector furthest from S = [ones | subspace]
        idx, _ = max(remaining.items(), key=distance_from_subspace)
        maxdist = distance_from_subspace((idx, remaining[idx]))
        if np.isclose(maxdist, 0):
            if verbose:
                print("Distance was 0. Popping from subspace.")
            subspace.popleft()
            continue

        if verbose:
            print(f"Col idx {idx} furthest from subspace. Dist: {maxdist:.6f}")
        subspace.append(remaining.pop(idx))
        yield idx

Rettferdig fordeling av boliglån med ulik eierandel

Tommy Odland — Fri, 07 Mar 2025 00:00:00 +0100

Anta at to personer skal kjøpe en bolig sammen, men har ulik egenkapital og/eller ulik betalingsevne på boliglånet. Hva er en økonomisk rettferdig fordeling?

Ulik egenkapital, lik fordeling av lånet

I DNBs artikkel “del boliglånet rettferdig” blir følgende fordeling foreslått som rettferdig dersom partene ikke har lik egenkapital. Alle tall er i tusen kroner.

	Johan	Andrea
Andel av lån	1 500	1 500
Egenkapital	800	200
Total andel av kjøpesum	2 300	1 700
Andel ift. totalsum	2 300 / 4 000	1 700 / 4 000
Eierandel	57,5 %	42,5 %

Men dette kan oppleves som urettferdig. For å forstå hvorfor gjør vi eksempelet mer ekstremt—hva om Johan har mye mer penger enn Andrea?

Da kan vi få en fordeling av lån og egenkapital som for eksempel ser slik ut:

	Johan	Andrea
Andel av lån	100	100
Egenkapital	3 800	0
Total andel av kjøpesum	3 900	100
Andel ift. totalsum	3 900 / 4 000	100 / 4 000
Eierandel	97,5 %	2,5 %

Johan har gått inn med 97,5 %, og skal få 97,5 % av salget. Men Andrea har likevel gjort et kupp! Selv om hun får lite når boligen selges, så hun bor i praksis nesten gratis. Hun bruker tross alt halve boligen i hverdagen, men betaler nesten ikke noe for det.

En mer rettferdig fordeling?

For å korrigere for skjevheten som oppstår når Andrea disponerer halve boligen kan vi anse henne som delvis å være leietaker. Hun betaler da leie for den andelen av hennes 50 % bruksrett som ikke er dekket av eierandelen. Utregningen ser slik ut i Python:

# Egenkapital for begge personene
egenkapital_A = 3_800
egenkapital_B = 0
egenkapital = egenkapital_A + egenkapital_B

# Andel av lån for begge personene
lån_A = 100
lån_B = 100
lån = lån_A + lån_B

# Utregninger
boligverdi = egenkapital + lån
eierskap_A = egenkapital_A + lån_A
eierskap_B = egenkapital_B + lån_B

print(f"Boligens verdi: {boligverdi:.2f}")
print(f"Belåningsgrad: {lån / boligverdi:.2%}\n")
print(f"Person A eier {eierskap_A / boligverdi:.1%}")
print(f"Person B eier {eierskap_B / boligverdi:.1%}\n")
print(f"Person A skal betale {lån_A / lån:.1%} av lånet hver måned")
print(f"Person B skal betale {lån_B / lån:.1%} av lånet hver måned\n")

# Total leiepris per måned for tilsvarende bolig
leiepris = 15
leieprosent_A = max(1 - 2 * (eierskap_A / boligverdi), 0)
leieprosent_B = max(1 - 2 * (eierskap_B / boligverdi), 0)

if leieprosent_A > 0:
    print(f"Person A er {leieprosent_A:.1%} leietaker")
    print(f"Person A betaler {leieprosent_A * leiepris / 2:.2f} til person B i leie")
if leieprosent_B > 0:
    print(f"Person B er {leieprosent_B:.1%} leietaker")
    print(f"Person B betaler {leieprosent_B * leiepris / 2:.2f} til person A i leie")

Kjører man koden blir resultatet:

Boligens verdi: 4000.00
Belåningsgrad: 5.00%

Person A eier 97.5%
Person B eier 2.5%

Person A skal betale 50.0% av lånet hver måned
Person B skal betale 50.0% av lånet hver måned

Person B er 95.0% leietaker
Person B betaler 7.12 til person A i leie

I dette eksempelet ville en tilsvarende bolig kostet 15 tusen å leie per måned. Andrea disponerer halve boligen, og av denne halvdelen leier hun i praksis 95 % og eier 5 %. Hun eier altså 5 % av sin 50 % bruksandel, som blir 0,05 * 0,5 = 2,5 % av boligens totalverdi. Hun må betale 7,12 tusen per måned i leie til Johan.

Det originale eksempelet

La oss gå tilbake til det originale eksempelet der Andrea går inn med 200 tusen i egenkapital og Johan går inn med 800 tusen. Tallene ser da slik ut:

egenkapital_A = 800
egenkapital_B = 200
lån_A = 1500
lån_B = 1500

Utregningen blir nå at Andrea eier 42,5 % av boligen. Men med en antatt månedlig leiekostnad på 15 tusen må hun både (1) betale ned halvparten av lånet per måned og (2) betale 1,13 tusen kroner i leie til Johan ettersom han gikk inn med en større andel egenkapital.

Et ekstremt eksempel

Hva om Andrea ikke går inn med noe penger i det hele tatt?

egenkapital_A = 1_000
egenkapital_B = 0
lån_A = 3_000
lån_B = 0

Da må Andrea betale 7,5 tusen i leie til Johan hver måned, fordi det er halvparten av boligens antatte leiekostnad. Hadde vi fulgt DNBs logikk ville hun bodd gratis.

Konklusjon

DNBs eksempel tar ikke hensyn til at parten som eier den minste andelen i boligen disponerer halve boligen. For å korrigere for dette kan vi anse parten som eier minst som å delvis være leietaker. Om leietakeren eier 42,5 % av boligen men 50 % av lånet, slik som i DNBs eksempel, så får hun 42,5 % av salgssummen og betaler 42,5 % av vedlikeholdskostnadene. Hun betaler også 50 % av lånet, pluss leie til den andre parten.

Dersom egenkapital er begrensningen, er en annen mulighet at den ene parten låner penger av den andre parten. Da kan begge gå inn med 50 % i både egenkapital og lån, men det er tatt opp et privat lån på siden. I praksis kan det oppleves kjipt å være delvis leietaker, spesielt om eierbrøken uansett er tilnærmet lik 50 %. Da er det nok mer praktisk å følge DNBs utregning.

Rent matematisk er lærdommen at man bør konstruere ekstreme eksempler for å sjekke om en utregning gir mening. DNBs utregning feiler på denne testen og er åpenbart utrettferdig når eierbrøken blir for skjev.

Se også “Slik deler dere boliglånet og utgiftene riktig” hos Pengeverkstedet. Artikkelen sier at totalen er lik egenkapital pluss lån, og korrigerer ikke for at hver part disponerer halvparten av boligen.
Om du vil ha en utfordring kan du forsøke å generalisere logikken ovenfor. Hva blir utregningen om mer enn to personer går sammen med ulik egenkapital og/eller ulik betalingsevne på lån?

Strømstøtte, fastpris, makspris og insentiver

Tommy Odland — Tue, 04 Feb 2025 00:00:00 +0100

Strømprisen har tatt stor plass i media de siste årene. Arbeiderpartiet foreslår nå å bytte ut strømstøtte med fastpris, mens Fremskrittspartiet og Kristelig Folkeparti vil ha makspris.

I denne artikkelen skal vi visualisere disse ordningene, fikse de svake insentivene ved hjelp av litt enkel matematikk på VGS-nivå, og se at politikernes tenkemåte rundt strømpriser har fellestrekk med hvordan de tenker på f.eks. fraværsgrensa.

Konstant pris, lineær pris og insentiver

I Oslo kommune er det to måter å fastsette vann- og avløpsgebyrer på:

Konstant pris basert på en arealberegning, som er uavhengig av forbruk
Lineær pris proporsjonal med forbruk (pluss en fastpris)

I motsetning til vannpris, er strømpris alltid basert på kundens forbruk. Om prisen var konstant, ville det ikke vært noe økonomisk insentiv for å isolere, holde temperaturen nede, kjøpe varmepumpe, eller spare strøm på andre måter.

Strømprisen har en annen egenskap som vannprisen ikke har; den er ustabil over tid. Prisen varierer innad i døgnet, innad året, og fra år til år.

Dagens strømstøtte

Figuren nedenfor viser prisen per kvartal de siste ti årene. Dataene er fra NRKs.no / SSB.

Dagens strømstøtte-ordning er også vist i figuren. Opp til 73 øre betaler man alt selv, deretter betaler man 10% av overskytende. La oss kalle spotprisen for $x$ og nettoprisen som kunden betaler for $y$. Da kan formelen for nettopris skrives som

\begin{equation*} y(x) = \min\left( x, 73 + 0.1 (x - 73) \right). \end{equation*}

Edit: Fra og med 1. januar 2025 er strømstøtten 75 øre, ikke 73 øre.

Fastpris

Arbeidetpartiet lanserte januar 2025 forslag om fastprisavtaler på 40 øre/kWh. En slik avtale er visualisert i denne artikkelen og gjenskapt i figuren nedenfor. Formelen er enkel; uansett hva spotprisen $x$ er, blir nettoprisen $y$ lik $40$ øre/kWh.

\begin{equation*} y(x) = 40 \end{equation*}

Makspris

Frp og KrF ønsker 100 prosent kompensasjon for priser over 50 øre/kWh, altså en makspris. Formelen blir

\begin{equation*} y(x) = \min\left(x, 50 \right), \end{equation*}

og resultatet av en slik regel er vist i figuren nedenfor.

Insentiver og den deriverte

Fastprisforslaget får kritikk for å ta bort insentiver. Kritikken kommer fra professorer, økonomer, og fra Støre selv, som tidligere uttalte at:

“Makspris på strøm løser ikke dette problemet, makspris gjør ikke noe med gassprisene og det fyller heller ikke vannmagasinene.”

Dette er riktig, men vi må huske at det er snakk om en fastpris per kilowattime, ikke en fastpris per måned. En fastpris per kilowattime insentiverer til lavt forbruk; kutter man forbruket til halvparten, betaler man også halvparten. Derimot forsvinner insentiver for å bruke strøm når den er billig, heller enn når den er dyr, under en slik ordning.

Politikere klamrer seg ofte til harde grenser som fastpriser og makspriser når de vil regulere et marked. Ofte tenker de at det må gå en hard grense mellom regulering og frie markeder i seg selv. Her er noen eksempler der harde grenser kunne vært myknet opp:

Skoleopptak. Politikerne i Oslo har i mange år ment at skoleopptak enten må være helt karakterbasert, eller basert på ren loddtrekning. En mellomting er mulig med vektet loddtrekning, som jeg skrev om i 2020.
Fraværsgrensa. En fraværsprosent på 9% er uproblematisk, men med 11% fravær mister man hele standpunktkarakteren. En mykere overgang ville vært mulig ved å redusere karakteren noe før man mister hele karakteren.
Arbeidsavklaringspenger (AAP). Du kvalifiserer om du er 50% syk, men ikke om du er 49% syk.
Toll. Frem til 1. januar 2024 var det 350-kronersgrense for toll- og avgiftsfritak på varer kjøpt fra utlandet.

Ofte er harde grenser både nødvendig, enkelt og praktisk; for eksempel 18-års aldersgrense på alkohol. Andre ganger kan harde grenser skape perverse insentiver og uheldige tilpasninger, som når elever strategisk planlegger fravær rett under grensen. Da bør vi vurdere mer gradvise overganger som bedre reflekterer de underliggende realitetene. Dette er en enkel form for mekanismedesign.

I våre modeller $y(x)$ er insentiver analogt med den deriverte, som er stigningen til funksjonen. En flat funksjon dreper koblingen mellom input og output, som her er bruttopris og nettopris. For å insentivere strømforbruk når det er billig, kan vi glatte ut Frp og KrFs makspris.

Glatt makspris

Vi kan glatte ut makspris-forslaget til Frp og KrF for å beholde insentiver, og samtidig garantere at prisen aldri overstiger 50 øre/kWh. Insentivene blir svake, men det er bedre enn ingen ting.

Vi starter med softplus-funksjonen $s(x; \alpha) = \log \left(1 + \exp\left( \alpha x \right) \right) / \alpha$, skifter og skalerer til

\begin{equation*} y(x) = 50 - s(50 - x; \alpha) \end{equation*}

og velger en fornuftig $\alpha$-verdi for å kontrollere hvor glatt funksjonen blir. Resultatet er vist i figuren nedenfor.

Logaritmisk pris

Enda sterkere insentiver får man om man ikke begrenser seg til en makspris på 50 øre/kWh. Det politikerne egentlig ønsker å gjøre er å skjerme forbrukerne mot den enorme prisvariasjonen. Grunnen til at de strekker seg mot fastpris og makspris heller enn en logaritme, kan være at de ikke vet hva en logaritme er, eller tror at befolkningen ikke vet hva en logaritme er.

For å begrense svingningene, men samtidig beholde insentiver i større grad enn med makspris, kan vi bruke en logaritmisk pris. Formelen kan skrives som

\begin{equation*} y(x) = \log \left( 1 + \alpha x \right) / \alpha , \end{equation*}

der $\alpha$-verdien kontrollerer styrken. En logaritmisk funksjon vil begrense prisen kraftig, men la kunden føle toppene mer enn med makspris. Logaritmen er en enkel funksjonell form som oppnår ønsket effekt, men mange andre funksjoner kan også oppnå lignende effekter og være vel så gode.

Oppsummering

Hvordan strømmarkedet skal fungere er en politisk beslutning, og de fleste problemene er uten tvil politisk skapt heller enn noe som kan fikses med matematikk. Når det er sagt er det fremdeles interessant å analysere politikernes tankemåte matematisk, og det er nok lurt å lytte til økonomenes betraktninger. Politikere tyr ofte til harde grenser som fastpris og makspris, men vi har sett at enkel matematikk kan gi oss mer nyanserte løsninger.

Vi skisserte to prismekanismer som både beskytter forbrukerne mot ekstreme prissvingninger og samtidig bevarer insentiver for fornuftig strømbruk: en myk makspris og en logaritmisk pris. Samme tankegang kunne forbedret andre områder der harde grenser skaper uheldige virkninger.

Beer tasting

Tommy Odland — Fri, 10 Jan 2025 00:00:00 +0100

In this article we’ll analyze data from a blind test. Blind tests are fun experiments that can determine, for instance, if family members or friends can distinguish between:

Pepsi and Cola
Red and white wines
Brand-name and generic foods

Analysis of such data is interesting, especially when you’ve designed the experiment and collected the data yourself. You might learn some statistics along the way, and that knowledge could prove useful in industry settings as well as in everyday situations.

The beer tasting experiment

A friend wanted to determine her true beer preferences, so we decided to run a blind test:

We bought 6 different brands of beer.
She tasted each brand on 3 different occasions.
The ordering of the 18 tastings was randomized.
Each of the 18 tastings was conducted on a separate day.

In every tasting, the unknown beer brand was given a rating on a scale from 1 to 10. Knowledge of which brands were given which rating was not disclosed until the experiment was finished. The results of the full experiment are shown in the figure below, and the full dataset is available at the end of this article.

The Oslo-based beer brand “Schous” is the winner, with an average rating of

\begin{align*} (7 + 7 + 8)/3 = 22/3 \approx 7.33 \end{align*}

Can my friend tell beers apart?

Let us examine the figure above a bit more critically. The ratings are somewhat consistent, in the sense that they do not look like random integers between 1 and 10. However, it is surprising that the very same beer is awarded both the lowest observed rating (3) and the highest observed rating (8).

This raises the question: can my friend consistently tell beers apart? Let’s test this with a permutation hypothesis test. We’ll test if the explained variance $r^2$ in the dataset is higher than what could expect if knowledge of beer brands contained no information.

Let $y_i$ be the rating at tasting $i$. Defint the variable $x_{ij}$ to be a one-hot encoding of the brands: it is equal to $1$ if beer brand $j$ was tasted in tasting $i$, otherwise it is equal to $0$. We’ll assume a linear model with structure

\begin{align*} y_i = \sum_j \beta_j x_{ij} + \epsilon_i, \end{align*}

where $\epsilon_i$ is normally distributed noise. The coefficients $\beta_j$ have a natural interpretation: they are the average ratings of each brand $j$.

Fitting this model to the data gives an explained variance $r^2 = 0.375$. The beer brand explains around one third of the total variane in the ratings. To put this number into perspective, consider that:

A perfectly consistent beer-rater would achieve $r^2 = 1$ on a training set (and on a test set), since the beer brand would explain their ratings perfectly and account for all variation.
A completely random beer-rater would on average achieve $r^2 \approx 0$ on a test set, since the beer brand would not explain the ratings at all. However, on a training set they would achieve $r^2 > 0$ since there are six degrees of freedom in the brand coefficients $\beta_j$, and these will overfit and capture some of the variance.

Let the null hypothesis $H_0$ be that the beer brands have no effect on the ratings at all. Permuting the rows of the data set randomly (so there is no relationship between brands and ratings) produces the following distribution of $r^2$ under the null hypothesis.

If $H_0$ is true (brands have no effect), we will on average observe an $r^2$ of around $0.292$. The variation in $r^2$-values in the permuted data sets is large, and anything between $0.08$ and $0.57$ is relatively common (coverage of 90%).
The observed $r^2$ is $0.375$, which is certainly higher than the average, but by no means extreme.
The $p$-value, which is the probability of observing a value of $0.375$ or higher, given that $H_0$ is true, is around $0.289$.

Scientists typically reject $H_0$ if the $p$-value is lower than $0.05$. Here the $p$-value is much larger; this level of consistency in the ratings is well within the realm of what could happen even if my friend cannot tell beers apart. It might be that any apparent consistency is attributable to chance rather than genuine tasting ability.

Summary

The brand “Schous” is the winner, but there is a concerning amount of variation in the ratings within each brand. It’s not clear that my friend can consistently tell beers apart in this experiment. The $p$-value is so high that any scientific paper would report a “no effect” conclusion, failing to discard $H_0$. That being said, I am also reluctant to decisively claim that my friend cannot tell beers apart. While $H_0$ is mathematically convenient, it imposes a somewhat one-sided burden of proof.

People’s tasting abilities may be less consistent than they believe. The paper An Examination of Judge Reliability at a major U.S. Wine Competition investigates this to some extent. A larger, more involved experiment might be needed to figure out if my friend can tell beers apart. For now, the data suggests Schous remains the preferred choice.

Dataset

Here is the dataset used in this article in .csv format:

order,beer_brand,rating
1,Frydenlund,3
2,CB,7
3,Pokal,5
4,Tuborg,3
5,Frydenlund,5
6,Tuborg,8
7,Frydenlund,5
8,Hansa,6
9,CB,7
10,CB,5
11,Pokal,3
12,Schous,7
13,Hansa,3
14,Schous,8
15,Hansa,8
16,Tuborg,4
17,Schous,7
18,Pokal,5

Python code

Here’s a sketch of Python code to perform the statistical test. Python 3.12.8 was used with Pandas 2.2.3, NumPy 1.26.4 and scikit-learn 1.5.2.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
import numpy as np

# Read dataset
df = pd.read_csv("dataset.csv")

# Create X and y
encoder = OneHotEncoder(drop=None)
X = encoder.fit_transform(df[["beer_brand"]])
y = df["rating"].values

# Fit linear model, compute r2
linreg = LinearRegression(fit_intercept=False)
linreg.fit(X, y)
r2 = linreg.score(X, y)
print(f"r^2={r2:.3f}")  # r^2=0.375

# Randomly permute data and fit models
rng = np.random.default_rng(42)
r2_scores = []
for simulation in range(999):
    y_permuted = rng.permutation(y)
    linreg.fit(X, y_permuted)
    r2_scores.append(linreg.score(X, y_permuted))

p_value = np.mean(np.array(r2_scores) >= r2)
print(f"p_value={p_value:.3f}")  # p_value=0.311

Papers read in 2024

Tommy Odland — Sat, 14 Dec 2024 00:00:00 +0100

I try to read a few math books every year, but the more I learn, the more overlap I see in their content. In 2024 I decided to organize with a group of friends and read papers instead.

We read 17 papers together, and I skimmed many more to decide on which 17 to read in the group. Below is the list of papers, which might be useful if you want to learn some new data science related conceps, or if you want you organize a reading club of your own.

Lessons learned

Here’s what I learned:

Most papers are not ideal for learning; the authors are not great writers and they have limited space to convey their ideas.
Most practitioners should read books rather than papers to gain general knowledge.
Finding papers that are on an appropriate technical level and of interest to everyone in a group is difficult.

Before reading papers, it might be useful to read some guides on reading papers:

How to Read a Paper (2007)
How to Read a Technical Paper (2009)
A guide to reading scientific papers (2016)
How to read Machine Learning and Deep Learning Research papers (2021)

Here is the list of papers that we read. Honorable mentions are papers that I was interested in, but ended up not choosing for the group. Papers that I for some reason enjoyed more than others get a star (⭐).

Data Science

We chose to start with two papers on the culture of data science and practical tips for data science projects.

50 Years of Data Science. David Donoho (2017)
⭐ Good enough practices in scientific computing. Wilson et al. (2017)

Honorable mentions

A few useful things to know about machine learning. Pedro Domingos (2012)
⭐ Statistical Modeling: The Two Cultures. Leo Breiman (2001)

Recommendation engines

We looked into recommendation engines: the Netflix price, matrix factorization and extensions to neural networks.

Matrix Factorization Techniques for Recommender Systems. Koren et al. (2009)
⭐ Factorization Machines. Steffen Rendle (2010)
Neural Collaborative Filtering. He et al. (2017) [arXiv]

Honorable mentions

Collaborative Filtering for Implicit Feedback Datasets. Hu et al. (2008)
Neural Factorization Machines for Sparse Predictive Analytics. He et al. (2017)

Continuous optimization

These articles cover practical methods for solving optimization problems.

An overview of gradient descent optimization algorithms. Sebastian Ruder (2016)
A comparison of numerical optimizers for logistic regression. Thomas P. Minka (2003)
An Interior-Point Method for Large-Scale l1-Regularized Least Squares. Kim et al. (2007)

Honorable mentions

Enhancing Sparsity by Reweighted L1 Minimization. Candes et al. (2007)

Operations research

In the age of data science, operations research is underrated.

UPS Optimizes Delivery Routes. Holland et al. (2017)
Increasing the Responsiveness of Firefighter Services …. van den Berg et al. (2017)
Data analysis and optimization for (citi)bike sharing. O’Mahony et al. (2015)

Honorable mentions

Defending Critical Infrastructure. Brown et al. (2006)

Information and compression

Information theory has a rich history and is the foundation of many modern techniques.

A Mathematical Theory of Communication. Claude E. Shannon (1948)
Arithmetic coding for data compression. Witten et al. (1987)

Statistics

More statistics never hurts, and seeing how practitioners attack problems is always interesting.

Should the Olympic sprint skaters run 500 meter twice?. Nils Lid Hjort (1994)
⭐ A Conceptual Introduction to Hamiltonian Monte Carlo. Michael Betancourt (2017)
MCMC using Hamiltonian dynamics. Radford M. Neal (2012)
⭐ Hierarchical generalized additive models in ecology. Pedersen et al. (2019)

Honorable mentions

Mindless statistics. Gerd Gigerenzer (2004)
Prediction, Estimation, and Attribution. Bradley Efron (2020)
Hyperparameters: Optimize, or Integrate Out? David J.C. MacKay (1996)

Solving NRK’s game “Former”

Tommy Odland — Tue, 26 Nov 2024 00:00:00 +0100

NRK published a fun game on their website called “Former” (English: “Shapes”). The gameplay is straightforward: the player clicks on a cell on the board, and as a result every connected cell (up, down, left and right) with the same color vanishes. The surrounding shapes fall down and the player is ready for the next move.

The goal is to make all shapes vanish in as few clicks as possible.

In this article we formulate the game as a graph search problem, show how to reduce the size of the graph and attempt to solve it using various graph search algorithms. The full code is available at github.com/tommyod/NRK-former-game.

Games as directed graphs

Games like these can be represented as directed graphs: nodes represent game states and directed edges represent moves (clicks). The initial node is the first board we see and the terminal node is the empty board.

Any path from the initial node to the terminal node is a solution.
The optimal solution is the shortest solution path.

The original game has 9 rows and 7 columns, which leads to a huge graph and therefore a huge search space. In the game instance shown in the animation above:

There are 39 possible first moves
There are 1,446 possible sequences of two moves
There are 51,162 possible sequences of three moves
There are 1,730,312 possible sequences of four moves
There are 55,950,299 possible sequences of five moves

We count one move as one click on any cell in a connected group of cells of the same color. The search space continues to grow as we search for sequences of moves. The branching factor is roughly 35, so at depth 10 we can expect there to be around $35^{10} \approx 10^{15}$ nodes.

A simpler instance

Let’s examine a simple 2-color instance on a board with 2 rows and 2 columns. Below is the full search space for such a problem instance.

The initial node is at the top and the terminal node is at the bottom of the figure. There are 24 nodes in total. The game can be solved in 3 or 4 moves, depending on which path we choose. The original game instance has billions of such nodes, and solving it comes down to searching the graph efficiently.

Reducing the search space

The first observation we make is that a click on any group of connected colors leads to the same node. It does not matter which specific cell we click as long as it’s connected to a group of the same color, so we might as well only consider the top-left cell in any group of connected colors.

The second observation is that any permutation of colors represents the same state. If there are $4$ colors, there are $4! = 4 \times 3 \times 2 \times 1 = 24$ ways to permute them. Applying this observation we can reduce the search space from 24 to 12 nodes in this specific problem instance.

The third observation is that horizontal flips represent the same state. If we’ve solved the game at some node $n$, then we’ve also solved the node $n'$ with flipped columns. There are two such states (flipped and not flipped), so this can only reduce the search space by a factor of 2. By applying this observation we can reduce the search space down to 8 nodes.

Based on these observations we can define a canonical representation of each game state (node) where we re-label the colors and flip the game. There is a tradeoff between:

the computation time spent transforming a game state into canonical form
the storage of game states as we search the graph

Re-coloring reduces the size of the search space by quite a bit, so we’ll apply this transformation. We’ll skip the flipping and full canonicalization.

Optimal solutions

The simplest algorithm that is guaranteed to produce an optimal solution is breadth-first search (BFS). It first explores all nodes one move away from the initial node, then all nodes two moves away, then all nodes three nodes away, and so forth. When we get to an empty board we know that no shorter path exists.

Unfortunately, BFS holds too many nodes in memory and explores every path. A smarter approach is to use A* search. All nodes are put in a priority queue, and we continually pop off the node with the smallest value of

\begin{equation*} f(n) = g(n) + h(n), \end{equation*}

where $n$ is a node, $g(n)$ is the length of the path so far and $h(n)$ is an admissible heuristic that estimates the total remaining path.

Admissible heuristics

If the heuristic $h$ never overestimates the length of the remaining path, then A* is guaranteed to return an optimal solution. Such a heuristic is called admissible.

An obvious admissible $h$ is the unique number of colors left on the board. This never overestimates the number of remaining moves needed, because at least one click is required to remove all cells of a given color.
An even better $h$ is described in the code below. It’s based on the following idea: if any given color appears on both sides of a column not containing that color, then at least two moves are needed to get rid of that color.

def consecutive_groups(iterable):
    """Count how many consecutive groups of True there are.

    Examples
    --------
    >>> consecutive_groups([False, True, True, False, False, True, True])
    2
    """
    return sum(1 for key, group in itertools.groupby(iterable) if key)


def a_star_heuristic(board: Board) -> int:
    """A lower bound on how many moves are needed to solve.

    This function examines each color in turn. For each color, count whether
    it appears in each column. For each group of columns, separated by
    columns where the color does not appear, we need at least one click.

    Examples
    --------
    >>> board = Board([[1, 2, 1],
    ...                [2, 2, 2],
    ...                [1, 2, 1]])
    >>> a_star_heuristic(board)
    3
    >>> board = Board([[1, 3, 1],
    ...                [2, 3, 1],
    ...                [1, 3, 1]])
    >>> a_star_heuristic(board)
    4
    """
    # Transpose the grid
    matrix = list(map(list, zip(*board.grid)))

    # Get the unique integers larger than zero in the matrix
    unique_integers = set(c for col in matrix for c in col if c > 0)

    # For every integer, see if it exists in each column
    integer_in_col = (((c in col) for col in matrix) for c in unique_integers)

    # Count groups of integers separated by other integers
    return sum(consecutive_groups(integer) for integer in integer_in_col)

Unfortunately, even with this heuristic A* is too slow for boards of a decent size. The figure below shows solution times for A* and BFS as a function of board size. Solution times grow exponentially. Whether we re-label colors or not seems to have little practical impact on solution times.

Good-enough solutions in limited time

A heuristic graph search algorithm (not to be confused with the heuristic function $h$ in A* search) is an algorithm that attempts to find a good solution. With heuristics, there are no guarantees that the optimal solution is found.

Some simple heuristics are:

Random search. Always choose a random move.
Greedy best-first search. Always choose the move that clears the most cells.
Randomized best-first search. Choose a move at random, but use weighted sampling and weigh the moves by the number of cells they remove. Setting weight = cells_removed**power for some power >= 0 is one way to parametrize this graph search. If power = 0 then the search behaves like random search, and if power is very large then it behaves like greedy best-first search. Setting power=None means using no randomness.

The figure below shows what happens when we solve the large $9 \times 7$ board 100 times using different values of power.

Setting power=5 and power=10 both yield solution solution paths with 17 moves.

We implement three algorithms that yield good-enough solutions quickly. They are called anytime beam search, heuristic search and Monte Carlo search, and all are more clever than simply repeating randomized best-first search many times.

Anytime beam search

Beam search is a simple algorithm:

Start at the root, expand all children. Keep only the beam_width children that are promising, for instance those that have the largest values of cells_cleared / num_moves.
From the beam_width nodes under consideration, expand all children. Filter them again and keep only the beam_width most promising ones.
Repeat until a solution is found.

One disadvantage of beam search is that for large values of beam_width we have to wait a while until we get a solution. An anytime algorithm is one that yields good-enough solutions quickly, and if it runs for longer it improves on the solution. To make beam search into an anytime algorithm we run it for beam_width=1, 2, 4, 8, ..., keeping track of the best solution seen so far and yielding as better solutions are found.

Heuristic search

This algorithm yields better and better solutions as they are found. If it runs long enough, it will eventually find an optimal solution. In practice it will run out of memory before it’s guaranteed to explore every path, so there are no guarantees. It works roughly like this:

Start by running a single deterministic best-first search. This gives a quick upper bound best_solution_length_so_far on the optimal solution that we can use to prune solutions with later.
Sort the priority queue by a non-admissible heuristic function, for instance cells_cleared / num_moves. Always consider moves that maximize this function, hoping they lead to a good solution.
Let h(n) be the admissible heuristic function discussed earlier. If num_moves + h(n) >= best_solution_length_so_far, then we can prune node n.

Monte Carlo tree search

Monte Carlo tree search also yields better and better solutions as they are discovered. Like the algorithm above, it also eventually finds an optimal solution but provides no guarantees when run for a short time. It works roughly like this:

Start with a deterministic best-first search to establish an upper bound used for pruning.
Randomly sample nodes in the explored tree, going down guided by the so-called upper confidence bound, which trades of exploration and exploitation.
When a new unexplored node is reached, run a randomized best-first search from the node down to the bottom of the tree to score the node. If this results in a new best solution, update the upper bound. Propagate the score cells_cleared / num_moves up in the explored tree to the root. This score affects the upper confidence bound used in the next iteration.
Prune by using the admissible heuristic h(n) as the tree is explored.

Running all algorithms for around four minutes on the game instance of size $9 \times 7$, we get the following results.

Anytime beam search finds the best-known solution from NRK in two seconds.
Heuristic search finds a solution of length 17 in around two minutes, but is unable to discover any better solutions in the remaining time.
Monte Carlo search finds a solution of length 14 in around twenty seconds.

Here is the solution sequence for the 12 move solution (click to enlarge).

The NRK record was 12 moves, and we managed to solve the board in 12 moves. On this particular instance, running for longer does not lead to further improvements. On other instances we are not always so lucky; sometimes we are unable to find the best known solution, at least in a few minutes of compute time.

Summary, notes and references

The NRK game “Former” can be formulated as a graph search problem, but the graph is huge. There are billions and billions of nodes in the graph, so complete enumeration is out of the question. While A* search is effective similar puzzles (e.g. the 15 puzzle), here it fails because we were unable to design a great admissible heuristic function.

Heuristic search algorithms are still able to produce good-enough solutions in a short time. We implemented three such algorithms with different flavors: anytime beam search, heuristic search and Monte Carlo search. In the end we managed to find the record solution of 12 moves on this instance. These algorithms can be combined: heuristic search and Monte Carlo search perform better if an upper bound is established quickly, so running beam search as a first step could help.

The game is similar to SameGame, and the paper Single-Player Monte-Carlo Tree Search proposes Monte Carlo search for this game. An implementation by Odin at github.com/odinhg/nrk-former-sp-mcts implements this algorithm. Another paper is Solving SameGame and its Chessboard Variant, which also uses Monte Carlo search. The sgbust program is a SameGame solver written in C++ that uses beam search. A good introductory reference for game solving is Chapter 3 “Solving Problems by Searching” in Artificial Intelligence: A Modern Approach. Another avenue to go down is reinforcement learning, see for instance the book Reinforcement Learning: An Introduction.

Thanks to Andreas for introducing me to the game, and to Eivind and Gunvor who independently discovered the admissible heuristic.

Appendix: more boards

Above we considered the NRK game instance for the 16th of November. Below we show performance on more game instances from four other days.

The average value

Tommy Odland — Sun, 10 Nov 2024 00:00:00 +0100

Presentation slides and video of a talk I gave about notions of averages (one-parameter models), loss functions and simple machine learning problems.

Below is a transcript with slides, if you prefer reading over listening.

The average value
Table of contents
The Pythagorean averages
The $p$-norm
Huber, mellowmax and quantiles
ETA for file downloads
Ranking products by ratings
Summary

The average value

Hello, my name is Tommy and this is a recording of a talk I gave at an IT conference. The title is “The average value” and the intended audience is intermediate-level data scientists. In this recording, I’ll go through the presentation informally. I’ll assume you have some mathematical background and are interested in data science. If you want to look at these slides or check the references, you can download them from this URL.

The average value—this is the equation for it, and if that was all there was to say, the talk would end here. This slide introduces some notation, but obviously to make a presentation out of this, we need to generalize away from this equation.

We’ll do that by looking at the optimization problem that this equation—the arithmetic average—solves. That’s the least squares optimization problem. In general, in machine learning, when we do regression tasks, it’s very common that we have squared loss and some kind of function $f$ where we want to compute the optimal parameters for $f$. Here, $f$ could be a tree-based model, linear regression, or neural network, and somehow we find the best parameters $\theta$.

However, there is typically no closed-form solution. For the arithmetic average there is a closed-form equation, but for many more complicated models, you use something like gradient descent or Newton’s method. You end up with a result, but there’s no equation you can use to just compute the results—it’s an iterative procedure. In many problems, there’s no guarantee that you have a global minimum or that you always end up with the same one.

Basically, this talk isn’t really about the average value—it’s about setting the model $f$ to be a single parameter model. We set $f(x,\theta) = \theta$ and ask ourselves “what else can we do?”. If we fix the model to be the simplest possible model we can think of, what else can we do? We can tweak the loss function!

Here’s the table of contents for the presentation:

First, we’ll look at the Pythagorean averages—the arithmetic average, geometric average, and harmonic average—and see that there are loss functions that induce these averages in a very natural way
Then we’ll look at the $p$-norm, which you might know about—it’s a way to measure distance, and we can use that to get the notion of the median and the midpoint
We’ll examine Huber, Mellox, and quantiles, which are one-parameter models that go further away from the Pythagorean averages and $p$-norm
Finally, we’ll look at some real-world examples aimed at developers: (1) ETA for file downloads and (2) Ranking products by ratings on websites

The Pythagorean averages

Let’s start with the Pythagorean averages.

The arithmetic average is the most common average, and it minimizes squared loss. There’s an assumption that if you’re twice as far off, it’s four times worse because you’re squaring the residuals. It has really nice mathematical properties—basically, if you have squared loss, you have a quadratic, so when you differentiate it, you get something linear that you can solve immediately. Newton’s method can solve this in one iteration—you can one-shot these squared problems!

This figure is important to understand—it shows for each data point (1, 3, and 6) the loss, and then shows the sum of these three losses over the three data points. That’s the solid blue line, and this is the loss function. If you minimize it, you get the arithmetic average, shown in the dashed line.

One final comment in the footnote: how do we get this squared loss in the first place? One way is to look at the normal distribution, which is essentially an exponential to the power of squares. But you also end up with averages in Poisson models and binomial models, so there are many types of assumptions about data that lead to arithmetic averages.

The geometric average minimizes multiplicative loss. If you take the logarithm of the data and the logarithm of the model and compare those, you get the multiplicative loss. Here, being one order of magnitude below is equally bad as being one order of magnitude above. The geometric average is like the arithmetic average but with multiplication—it’s the number that you have to multiply with itself $n$ times to get the product of the data.

The harmonic average minimizes relative loss, where you measure errors relative to the true value. This is equivalent to doing weighted least squares where the weights are given by $1/y_i$. This particular loss—the relative loss—is not the only loss function that induces this notion of average. The loss in the footnotes will also lead to the same harmonic average, but the relative loss one is more natural and commonly used.

Let’s look at these three loss functions in a different problem setting. Suppose you have resources to distribute—maybe you have bakeries and predictions for tomorrow’s bread sales. You have a central bakery where you bake bread and want to distribute it to cover every store.

Unfortunately, you don’t have enough bread to cover every store, so you need to distribute according to some rule. You would like your distribution $\theta$ to be as close as possible to the demand of each store $y$, so you use a loss function. Depending on which loss function you use, you get three different distributions.

I’m not sure what the correct answer would be—it would probably depend on the bakeries’ goals. Would you use squared loss, multiplicative loss, or relative loss in the real world? I’ll leave that up to you to think about.

We’ve seen several loss functions, and it might be interesting to discuss how you would choose between them. Every loss function has the property that if your prediction is very close to the target, it’s good, and if it’s far off, it’s bad.

How do you choose between loss functions? One thing I like to do is consider the null space—by that, I mean considering which types of errors are equally bad and whether that makes sense in your application. For instance, if the true values $y$ are 2 and 8:

Squared errors would imply that going two up in the positive direction on either prediction is equally bad
Multiplicative errors mean that predicting 4 (8/2) or 16 (8*2) is equally bad
Relative errors mean that subtracting 2 from the first prediction or adding 8 to the second one is equally bad, because those represent equal units of true values

Here are some practical examples:

If you’re trying to predict medicine dosages, you probably want to use relative or multiplicative errors, since different drugs work on different dosage scales
If you’re predicting economic outcomes or investments, you probably don’t want to use multiplicative or relative errors because money isn’t relative—one unit of money is the same whether it comes from a small project or a huge project
For financial predictions, you might want to use absolute errors rather than squared errors, because being twice as far off means losing twice as much, not four times as much (unless you want to consider the utility of money, but let’s skip that for now)
When people look at interest rates or summarize computer performance benchmarks (CPUs, etc.), they almost always use the geometric average

The $p$-norm

Having finished the first part about Pythagorean averages, let’s talk about the $p$-norm.

We’re going to generalize this equation in a different direction by solving this optimization problem for different values of $p$ instead of $p=2$ (which gives us the arithmetic average).

If we set $p$ equal to one, we get the absolute error. This is an absolute value function placed on each data point, and when you sum it up, it’s piecewise linear. If there’s an odd number of data points, there’s a unique maximum; if there’s an even number of data points, it’s flat between the two middle points—there’s no unique minimum.

The median has an interesting property: if I were to change the data point 6 and move it up to 8 or down to 4, as long as it doesn’t cross 3, the median stays the same. The median as a function completely ignores the location of every data point except for the middle one. Obviously, if 6 crosses 3, then something happens with the function, but it’s essentially blind to many of the data points.

Choosing $p=1.5$ is one of many ways to create a function between a median and an average. It’s also the first optimization problem we’ve seen where there is no closed-form solution—here you would use an iterative optimization routine to find the minimum.

Choosing $p=2$ is the squared error, which we’ve seen before.

As you increase $p$ further, the functions become more and more sharp.

You get to an interesting property in the limit as $p$ becomes very large: minimizing the $p$-norm becomes equivalent to minimizing the maximum. If you want to minimize the maximum distance, you want to make the worst case as good as possible, and you do that by placing yourself at the midpoint. This is one of the measures of centrality you might have learned in high school—the midpoint.

How can we combine these Pythagorean losses and $p$-norms? Let’s put them in a table:

Mean absolute error
Mean squared error
Maximum error (not used much in machine learning but used in engineering where you want to optimize the worst case, like in airplane design)
Mean squared logarithmic error
Mean absolute percentage error
Weighted mean squared error

Some combinations aren’t listed but could be imagined. For instance, with $p=1$ and multiplicative error, you’d get something like the mean absolute logarithmic error. The point is that you can construct and combine these functions.

How would you do that? You can first look at an individual data point and consider what kind of loss function you’re interested in, then use a $p$-norm to combine them. The $p$-norm is a way to combine the individual losses.

Think about what properties you want in your application:

With absolute error, residuals of $(4,0)$ are as bad as $(3,1)$ and $(2,2)$. With money, this makes sense—if you’re off by four, you probably don’t care how it’s distributed if it’s money lost
It also means that $(9,0)$ is better than $(5,5)$, even though errors in $(5,5)$ are more balanced
Maximum error just looks at the largest value, so $(9,0)$ would be as bad as $(9,5)$ and $(9,9)$, while $(8,8)$ would be better than $(9,0)$

Huber, mellowmax and quantiles

Now let’s look at even more esoteric notions of averages, maxima, minima, and everything in between.

One issue with squared errors is that they can be overly sensitive to outliers. One remedy for this is to use the smooth Huber loss, which acts like a square when you’re close to zero but like the absolute value when you zoom out. This makes it less sensitive to outliers while still maintaining a unique minimum even with an even number of data points.

There’s an opposite problem that might occur in some applications: squared errors are very flat around zero, so if you have a small residual and you square it, it becomes even smaller. If you’re regularizing coefficients (like the LASSO does) and want to drive them all the way to zero, it’s very nice to have something that acts like the absolute value close to zero. The reverse Huber has this effect—it’s like the absolute value when you’re close to zero and acts like a square if you’re far away.

Mellowmax and mellowmin are smooth generalizations of maximum and minimum. They are functions that:

As $\alpha$ goes to infinity, the function approaches the maximum
As $\alpha$ goes to negative infinity, the function approaches the minimum
Around zero, the function approaches the arithmetic average

If you want something that acts in between the arithmetic average and the maximum, mellowmax is a nice way to do it. The maximum only depends on the maximum value in the dataset, but sometimes when you want to minimize the maximum, you want to bring all values down. One way to do that is to drive down the maximum value, but as a function, the maximum is blind to every other value apart from the largest one. If you use mellowmax instead, it’s actually a bit sensitive to other values, which can better inform the optimization routine.

Sometimes it’s very nice to relax these functions that are piecewise linear or have derivatives that are zero for most inputs. The median only depends on one or two values for most inputs, and if you want something more smooth, this is a nice option.

Let’s quickly cover quantiles—I assume you know about them. They’re a way to be somewhere between the median and the minimum/maximum. They put an absolute value function on every data point but scale it up and down on the left and right.

A small generalization: there’s no reason we have to use the absolute value—we could use squared residuals too. If $p=2$, you would call it expectile, but in general, you don’t have to use $p=1$ or $p=2$. You could in principle use any value for $p$.

ETA for file downloads

Let’s look at our first practical example: ETA for file downloads.

Suppose you’re a developer writing a routine that will download files, copy files, or install software. The computer is doing chunks of work, and you have the time that each chunk takes. You know the amount of work done and how much work will be done in total (like the total megabytes for all files to download). You want to estimate how long this will take.

To solve this, we need to handle:

Non-equidistant arrivals (times or pieces of work)
We’ll assume a linear model is reasonable (parameters can change—if someone else starts downloading, your speed might drop, but after it drops we assume it’s still linear with a different rate)
Very importantly, when a new data point arrives, we must be able to update our model in constant time—we can’t iterate over all previous data points, as there could be millions

Here’s one approach: we can compute the average rate of work. With two pieces of work (two files downloaded), we can compute the slope for each piece (work done over time elapsed). By minimizing this function, we get $\theta$, which is the average amount of work done per time.

I weight this by $\Delta t$ (time) because when I solve that optimization problem, it leads to an equation with nice symmetry in $\theta$. It should make intuitive sense that downloading a file in 1 second contains some information, but downloading the next file in 30 seconds contains much more information—so large $\Delta t$ should be weighted more heavily.

We use $w_i$ to weigh the most recent data more heavily. We can split it up like this: the most recent data point is attached at the end, so we have the previous estimate and some new data. We can quickly update by keeping track of our previous computation and adding to it.

We then take a linear model constrained to go through the last data point seen. We have one degree of freedom left, which we use on the slope with this estimate. We set the weights proportional to some number to the power of $i$ over $\gamma$, where $\gamma$ is the half-life. This parameterized the weights so the most recent data is weighted more heavily.

Here’s how it performs:

If the speed changes when we see more data, obviously it’s not a very good model
If it’s linear all the way through, it’s quite accurate
With some deviation from linearity, it picks up an average rate somewhere in between
With noise, it goes through the last data point and picks up an average rate that’s quite close

As more data arrives, it will actually improve because it’ll pick up on new regimes with different rates of change. This is a simple algorithm—very fast and accurate in many real-world cases. It depends on the application, but it’s probably more advanced than many file ETA estimates I’ve seen, which tend to be jittery and jump around too much.

Ranking products by ratings

Our final example is ranking products by ratings.

This is a fascinating real-world example of how you can use averages effectively. I looked at Norwegian e-commerce websites where users can give ratings. All these websites use a naive average rating, which produces some strange results:

The average of a single 5 is 5, but the average of many 5s and a few 4s would be slightly less than 5
This means a product with one 5-star rating would rank higher than a product with hundreds of 5-star ratings and a few 4-star ratings
I would trust the product with more ratings to be better, as one rating doesn’t give much certainty about true quality
Many websites also place products with no ratings at the very bottom, which I think is wrong—a new product with no ratings is likely average, not the worst product

I didn’t find any Norwegian e-commerce websites that do this correctly (I didn’t look at international ones). One of the few sites that gets this right is the IMDb Top 250 list—they account for the number of ratings and use an equation similar to what we’re going to deduce.

We want our average rating to have two properties:

It should take into account the number of ratings
A product without any ratings is likely typical, not really bad

Here’s a crucial idea: what are we really doing when we’re trying to rank products? We’re going to frame this as a prediction problem—we’re predicting the next rating. If you were to buy this product and rate it, what would your rating be? That would be the next rating for this product.

So we’re casting it as a prediction problem and saying that ranking products is equivalent to predicting the next rating and sorting by that prediction. We’ll do this by:

Introducing a prior average $\bar{y}$ and a pseudo count $\alpha$
Solving this optimization problem
The solution shows the meaning of the pseudo count—it’s added to $n$ as dummy ratings, and the value of these dummy ratings is $\bar{y}$ (the prior average rating)

To determine the hyperparameters $\bar{y}$ and $\alpha$, we need to do cross-validation. I assume you know about leave-one-out and k-fold cross-validation. We’re going to do “leave-next-out” cross-validation.

It’s very important that your cross-validation procedure be as similar as possible to how you want to use the model. Since we defined our notion of ranking as predicting the next value, we’ll simulate that exactly:

Start with no ratings, try to predict the first one
Get to know the first rating, look at the squared residual
Use that first data point to predict the second one
Get to know the second data point and compare
Continue this process

This is essentially a time series split where you learn one more data point at a time. Both of these can be computed in linear time for these simple models, so we can do this very quickly.

Let’s apply it to a real dataset. There’s a Norwegian website called Legelisten that ranks doctors.

I found 17 doctors with 212 total ratings. I used our average with some weights, parameterizing those weights by the half-life $\bar{y}$. Using leave-next-out cross-validation, I found:

Optimal number of pseudo-ratings is around $\alpha=2$
Optimal half-life is around $\gamma =7$ years
Prior rating is $\bar{y}=3.8$ (on a scale from 1 to 5)

The loss when using the arithmetic average as our predictor was 1.77. With this regularized temporal average, it’s 1.55—a very nice reduction in loss that leads to better predictions of the next rating.

You might ask why not make this into a big machine learning project using all kinds of covariates, but I don’t think people would like that. If you rank doctors using information about their gender and age, it’s probably not good. This solution is something people will understand and won’t feel discriminates unfairly based on age, gender, or other factors. It’s still super simple, just more nuanced and created from thinking through the problem more carefully.

Looking at the results (blue is the old website rating, green is my proposed rating):

The third doctor: One person rated this doctor with a 5. The old ranking puts them first; we say no—one rating isn’t much, so we pull them down. The doctor with many 5s deserves to be ranked higher.
The tenth doctor is interesting because they’re pulled away from the grand average. Typically without temporal weighting, you pull towards the grand average, but this doctor has gotten more positive reviews lately—lots of 5s and 4s, 44 ratings in total. This doctor has improved over the years, so the half-life is forgiving the older low ratings and pulling up towards the more recent 5s.
The seventeenth doctor hasn’t been rated once. The old ranking would place them last; we say no—with no ratings, we believe you’re in the middle of the pack. We place them there, certainly better than someone with just many 1-star ratings.

Summary

Let’s summarize the key points.

Only simple problems have closed-form solutions. Don’t worry so much about how to solve things when you start modeling—look at what properties you want to model instead. Solving is left to an optimization algorithm.
There’s a good reason why people like squared errors—they have nice mathematical properties. Not everything I showed today can be solved efficiently for bigger models.
In machine learning, the solution depends on both the model and the loss function. Many data scientists focus heavily on the model, discussing which is better in detail. While the model is important, choosing the wrong loss function can be just as problematic. For instance, if errors are truly relative in your application but you choose RMSE, or you have an economic application but use the wrong loss function, you’re probably modeling the whole thing incorrectly.
Adapt your approach to the problem. Combine existing ideas but don’t reinvent—there are many good books and sources. People have thought about amazing things, so read up on that. If you find yourself inventing something you believe is totally new to solve a practical problem, you’re probably reinventing something—do a literature search.
Always start simple. Never use models you don’t understand—you’ll have a hard time explaining them to others, and they might behave in ways you don’t fully understand. There’s still plenty of room to have fun with simple models. Even today, some e-commerce sites are billion-dollar businesses and don’t implement averages correctly.

The final comment I want to make is that if you go back to the first slide and remove the restriction of the one-parameter model, everything we talked about still holds. All the loss functions and modeling approaches are still relevant, but by choosing one-parameter models, I was able to create nice plots and talk about things simply. Many of these concepts are still relevant for other models, even though computationally it’s not always easy to optimize.

I really hope you enjoyed my presentation. Thanks a lot for watching, and again, you can download the slides and look at the references if you want to. Thank you.

Smooth taxes without brackets

Tommy Odland — Wed, 16 Oct 2024 00:00:00 +0200

I recently implemented a calculator for Norwegian income tax, and it was harder than I anticipated. The implementation required around 30 lines of code and 20 constants.

In this article we use the term “income tax” to refer to everything the government deducts from gross income. To compute income tax, one must take into account the brackets, the national insurance contributions, the minimum standard deduction, the personal allowance, and so forth.

In this article we’ll simplify the income tax calculation to one line of code and two constants.

We’ll start by looking empirically at the Norwegian income tax, then we’ll study properties that a tax scheme should have. In the end we’ll construct a smooth tax function with two adjustable parameters $H$ and $k$:

\begin{align*} T(x) = \frac{H(e^{-kx} + kx - 1)}{k}. \end{align*}

We’ll show that this function has good properties and that for suitable values of $H$ and $k$ it approximates the current income tax very well. Below is a figure showing the average tax rate and the marginal tax rate induced by the function $T(x)$ above. You’ll understand this figure better as you read the next sections.

Introduction

The tax function $T(x)$

Let $T: \mathbb{R}_+ \to \mathbb{R}_+$ be the income tax as a function of gross income $x$. For instance, if Alice earns a gross income of $x=100$ units of money, then she might have to pay $T(100) = 20$ in taxes, leaving her with a net income of $x - T(x) = 100 - 20 = 80$.

If we have such a function $T(x)$, then we can deduce three other interesting functions that answer specific questions:

The net income $x - T(x)$: “How much money do I have left after paying taxes?”
The average tax rate $T(x) / x$: “What percentage of my total income do I pay in taxes?”
The marginal tax rate $T'(x) \approx T(x+1) - T(x)$: “If I earn one more unit of money, what fraction of that unit of money goes to taxes?”

Proportional and progressive tax

A tax scheme is called proportional if the tax rate is a fixed percentage. In our notation, this would mean that $T(x) = cx$ for some constant $c$.

A tax scheme is called progressive if those who earn more pay a higher percentage in taxes. In addition to progressive tax schemes, we can define non-regressive and strictly progressive tax schemes.

Non-regressive: the average tax rate $T(x)/x$ never decreases.
Progessive: the average tax rate $T(x)/x$ never decreases, and it increases at least once in the domain.
Strictly progressive: the average tax rate $T(x)/x$ always increases.

Mathematically, these requirements correspond to conditions on the derivatives of the average tax rate. For instance, a non-regressive tax function $T(x)$ has the constraint that

\begin{align} \label{progressive} \tag{1} \left( \frac{T(x)}{x} \right)' \geq 0 \Rightarrow T'(x) - \frac{T(x)}{x} \geq 0. \end{align}

The second inequality comes from applying the product rule of differentiation. Having established some notation and concepts that we’ll use later, we will now empirically examine the Norwegian income tax.

The Norwegian income tax

The Norwegian income tax has several deductions, a few if-then-else rules and a (mostly) increasing marginal tax rate. Once all the rules are implemented and combined, the result can be visualized as a function. The implementation was tested against the calculator Skatteberegning 2024, and in the background the income distribution from 2022 is shown.

The jumps on the marginal tax rate $T'(x)$ make the figure a little disorderly, but notice that:

It’s progressive, since the average tax rate $T(x)/x$ never decreases and it does increase.
It’s not strictly progressive because $T(x)/x$ has two flat regions.
The limit evaluates to $\lim_{x \to \infty} T(x) / x = 0.474$.
The marginal tax rate is discontinuous, and actually jumps down once.

The government often changes the effective tax rates: they adjust deductions, the lower and upper limits on the brackets and rates within each bracket.

To get an idea of how erratic these changes are, consider the highest bracket that extends to infinity. In 2020 the lower limit of the highest bracket was 999 550, then in 2021 it was raised to 1 021 550. It doubled to 2 000 000 in 2022, was reduced by half a million to 1 500 000 in 2023 and cut further to 1 350 001 in 2024.

Properties that smooth taxes should have

To implement the Norwegian income tax I had to read up on tax rules from several sources online. I also had to test the implementation against an online calculator multiple times before I got it right. Wouldn’t it be nice if $T(x)$ was just a simple formula? No jumps and kinks—just a smooth, simple function.

For instance, given a constant $k >0$, what if $T(x) = x(1 - e^{-xk})$? That way the average tax rate becomes $T(x)/x = 1 - e^{-xk}$, which seems reasonable… right?

It turns out that this is a seriously bad idea, as we can see in this figure:

The net income $x - T(x) = x - x(1 - e^{-xk})$ is maximized at $x^{\star} = 1/k$. There is no incentive to earn more money than this, since you lose money by earning more. This is a terrible property for a tax system to have—back to the drawing board!

Property 1: Good incentives

Let’s set a simple rule: earning more should always mean keeping more after taxes. In other words, if we increase our gross income from $x$ to $x + \epsilon$, then we should end up with more net income after paying taxes:

\begin{align*} \underbrace{(x + \epsilon) - T(x + \epsilon)}_{\text{net income with $\epsilon$ more}} > \underbrace{x - T(x)}_{\text{net income}} \end{align*}

Rearranging this expression we find that the condition is equivalent to

\begin{align*} \frac{T(x + \epsilon) - T(x)}{\epsilon} < 1, \end{align*}

and in the limit $\epsilon \to 0$ the requirement says that the marginal tax rate $T'(x)$ must obey $T'(x) < 1$. If this inequality is satisfied, then higher pre-tax income always leads to higher post-tax income. In other words, there’s always an incentive to increase one’s gross income $x$.

Property 2: Progressive taxation

Another natural constraint to impose is that $T'(x)$ should be non-decreasing, since then the tax scheme must be non-regressive. To see why this is true, note that if $T'(x)$ is non-decreasing and non-negative, then $T(x) = \int_0^{x} T'(\tau) \, d \tau \leq x T'(x)$. Rearranging this inequality gives $T'(x) \geq T(x)/ x$, which is an inequality that we have seen before in Equation \eqref{progressive} in the introduction. It is exactly what is needed for a tax scheme to be non-regressive.

Note that the converse is not true: a tax scheme can be non-regressive even if $T'(x)$ decreases. For instance, the Norwegian marginal tax does decrease at one point, but the tax is still non-regressive overall. If $T'(x)$ is not only non-decreasing, but also increases at some point, then the resulting tax scheme is guaranteed to be progressive.

Due to these constraints on the marginal tax rate $T'(x)$, it’s easier to find good functions $T(x)$ if we start with $T'(x)$ and integrate to obtain $T(x) = \int_0^{x} T'(\tau) \, d \tau$.

Constructing tax functions

Recall that the marginal tax rate $T'(x)$ says how much we’ll be taxed on the next unit of money earned. For instance, $T'(x=100 \, 000) = 0.25$ in the Norwegian income tax, so if we earn $100\, 000$ NOK the next unit of money will be taxed by $0.25$.

Proposing a function. Let’s again consider the function $1 - e^{-xk}$, but this time as the marginal tax rate $T'(x)$ instead of as an average tax rate:

\begin{align*} T'(x) = 1 - e^{-xk}. \end{align*}

This function satisfies both our properties: it’s always less than one, and it never decreases. If we integrate it we obtain

\begin{align*} T(x) = \int_0^{x} T'(\tau) \, d \tau = \int_0^{x} (1 - e^{-\tau k}) \, d \tau = \frac{e^{-kx} - 1}{k} + x. \end{align*}

Notice that our proposed marginal tax rate $1 - e^{-xk}$ is zero when $x=0$ and approaches one as $x\to \infty$. The interpretation is that there is no tax initially as one starts to earn income, but if one approaches infinite income then units of money earned toward infinity will be taxed more and more.

Generalizing ever so slightly, we can change the marginal tax rate to $T'(x) = H - (H-L)e^{-k x}$. This way the marginal tax starts at a low value $L$ and converges to a high value $H$. Integrating, we obtain the tax function

\begin{align*} T(x) = \int_0^{x} T'(\tau) \, d \tau = \int_0^{x} H - (H-L)e^{-k \tau} \, d \tau = \frac{(H-L) e^{-kx} + H(kx - 1) + L}{k}. \end{align*}

The current policy today in Norway is that the first units of money earned are not taxed. Therefore, let us set $L=0$ and optimize for $H$ and $k$ by curve fitting to the Norwegian tax rate:

The smooth model based on $T(x)$ is shown in dashed lines. The error is never more than 5300 NOK, so this approximation is pretty good. It’s easy to implement if you want to play around with it in Excel or Python:

import numpy as np


def T(x):
    """Norwegian income tax for 2024. 
    Max error < 5300 NOK, mean absolute error ~1400."""
    H, k = 0.4772, 2.8745e-06
    return H * (np.exp(-k * x) + k * x - 1) / k

Summary

Income tax computation is quite complex. To implement it one needs around 30 lines of code with deductions, rules and tax brackets, all defined using around 20 constants.

In this post we examined smooth tax functions $T(x)$. One such function is

\begin{align*} T(x; H, k) = \frac{H(e^{-kx} + kx - 1)}{k}, \end{align*}

which mimics the overall behavoir of the current income tax calculation with good accuracy.

Maybe it could be an alternative to today’s system? After all, a single formula is easy to implement compared to the current rule set with various deductions, constants and tax brackets. Everyone has access to the exponential function on their phones or computers. A simple function makes it harder for politicians to obfuscate the tax system by introducing new rules and conditions. They could still modify the tax rate by modifying constants such as $H$ and $k$.

One argument against a smooth function is that the average person does not understand the mathematics of it. This might be true, but then again: the average person probably does not understand what a bracketed marginal tax system is either.

Keep it simple!

If I was a benevolent dictator, I would:

Simplify tax formulas and other computations performed by the government as much as possible
Favor simple, smooth functions (2-3 parameters) over discretizations and bracketing
Require government agencies that use formulas to publish them in math and code, with tests
Ensure that properties such as good incentives are baked into the models
Run an exhaustive test suite to guarantee good incentives across all combinations of gross income and all manner of government payouts, ensuring that citizens are never punished for earning more and working more (this does happen, and it might be a sign of overly complex systems that have not been thought through carefully)

Notes and references

Continuous tax brackets were discussed on the Personal Finance & Money Stack Exchange back in 2020.
The paper Smooth Income Tax Schedules: Derivation and Consequences by Schwarz et al discusses how various countries approach taxation and propose using smooth functions.
A Quora post with the fantastic title The US Federal Tax Code is More Complicated Than Calculus. So Let’s Use Calculus also looks at smooth functions. The author proposes shifting and scaling a function of the form $T'(x) = 1 - 1/(x + 1)$, whereas we shifted and scaled $T'(x) = 1 - e^{-x}$.
Any sensible cumulative probability density function can be used to construct $T'(x)$, offering a flexible approach to designing smooth tax functions. I chose the exponential function because it’s relatively simple to work with.

Arithmetic coding in Python

Tommy Odland — Thu, 19 Sep 2024 00:00:00 +0200

Arithmetic coding is a lossless data compression algorithm. It encodes a sequence of symbols into a sequence of bits. After reading the 1987 paper “Arithmetic coding for data compression” I decided to implement the algorithm in clean, modern Python.

In this article I’ll demonstrate how to use my Python implementation, how well arithmetic coding works when the symbol frequency is skewed, and how to encode “Crime and Punishment.” We also compare arithmetic coding with the more well-known Huffman coding. The implementation is available at: github.com/tommyod/arithmetic-coding

A simple example

Here we create a message to encode and put an End Of Message (EOM) symbol "<EOM>" at the end of it. We also need to set up a table of symbol frequencies. Arithmetic coding exploits the frequent symbols and assigns fewer bits to them, though unlike Huffman encoding arithmetic encoding does not actually use a fixed bit sequence per symbol.

>>> import collections
>>> from arithmetic_coding import ArithmeticEncoder
>>> message = ["A", "B", "A", "A", "A", "<EOM>"]
>>> frequencies = collections.Counter(message)
>>> encoder = ArithmeticEncoder(frequencies=frequencies, bits=12)
>>> bits = list(encoder.encode(message))
>>> bits
[1, 1, 0, 0, 0, 0, 1, 1]

We can decode these bits back to symbols and we get the original message back.

>>> list(encoder.decode(bits))
['A', 'B', 'A', 'A', 'A', '<EOM>']

This examples encodes the message to 8 bits. A naive approach would be to assign a unique 2-bit sequence to each symbol, e.g., {"A": 0b00, "B": 0b01, "<EOM>": 0b10}. Since the message contains 6 symbols, we would need 6 * 2 = 12 bits with this naive approach.

Infrequent symbols

To see how the encoder exploits the frequency of symbols, consider this example. We initialize ArithmeticEncoder(bits=32) because we need more internal bits in the buffer of the encoder to represent low-frequency symbols.

>>> message = ["A"] + ["B"] * 1_000_000 + ["<EOM>"]
>>> frequencies = collections.Counter(message)
>>> encoder = ArithmeticEncoder(frequencies=frequencies, bits=32)
>>> bits = list(encoder.encode(message))
>>> len(bits)
43

Amazingly, a message with a million symbols is encoded to only 43 bits! Having seen how efficient arithmetic coding is with skewed frequencies, let’s apply it to a real-world example.

Encoding “Crime and Punishment”

Let’s download the English translation of “Crime and Punishment” by Fyodor Dostoevsky.

>>> import requests
>>> import string
>>> url = "https://www.gutenberg.org/cache/epub/2554/pg2554.txt"
>>> response = requests.get(url)
>>> symbols = (s for s in response.text if s in set(string.printable))
>>> message = list(symbols) + ["<EOM>"]
>>> len(message)
1164281

For this example, we’ll use the actual frequencies from the text, though in practice one might use a more general frequency table for the English language.

>>> frequencies = collections.Counter(message)
>>> len(frequencies)
83
>>> frequencies["e"], frequencies["q"]
(104735, 766)

Naive encoding

The Python code above shows that there are 83 distinct printable symbols in the book. Since $2^7 = 128$, a naive approach would be to use seven bits for each symbol, regardless of its frequency. We would then encode "A" to 0b0000000, "B" to 0b0000001, "C" to 0b0000010 and so forth. This approach would encode the book using 1164281 * 7 = 8149967 bits.

Arithmetic encoding

Instead of using the naive approach, the arithmetic encoder exploits that symbols like "e" are much more frequent than symbols like "q". In around three seconds we’re able to encode all the 1.05 million symbols in the book:

>>> encoder = ArithmeticEncoder(frequencies=frequencies, bits=32)
>>> bits = list(encoder.encode(message))
>>> len(bits)
5259420

The result has around 5.26 million bits, and the bits/symbol ratio is 4.52.

Shannon entropy as a lower bound on bits

The Shannon entropy, which is a lower bound on how many bits we need on average to encode a message of this length with these frequencies, is

\begin{equation*} -\sum_i f_i \log_2 \left( p_i \right), \end{equation*}

where $f_i$ is the frequency (count) of symbol $i$ and $p_i = f_i / \sum_j f_j$ is the probability of symbol $i$. The bound only applies if we assume that each symbol is independent of the others—if we assume higher-order statistics, higher compression ratios might be possible.

Computing this value for “Crime and Punishment”, we find that the best we can hope for is

>>> import math
>>> total = sum(frequencies.values())
>>> probs = {s: c / total for (s, c) in frequencies.items()}
>>> shannon_bound = -sum(frequencies[s] * math.log2(probs[s]) 
...                     for s in frequencies.keys())
>>> shannon_bound
5259418.898940693

Huffman encoding

We also use the dahuffman Python library to produce a Huffman encoding, which maps each symbol to a unique code value of variable length. Notice how common symbols like "e" are mapped to short code values, while rare symbols like "B" correspond to long code values.

>>> from dahuffman import HuffmanCodec
>>> codec = HuffmanCodec.from_frequencies(frequencies)
>>> codec.print_code_table()
Bits Code                    Value Symbol
   3 000                         0 'e'
   4 0010                        2 'h'
   4 0011                        3 'i'
   5 01000                       8 'u'
   9 010010000                 144 'R'
  10 0100100010                290 'Y'
  10 0100100011                291 'B'
  ...
>>> symbol_to_bitsize = {code:bitsize for (code, (bitsize, _))
...                      in codec.get_code_table().items()}
>>> sum(symbol_to_bitsize[symbol] for symbol in message)
5298110

Huffman encodes the book to around 5.3 million bits.

Arithmetic coder with an adaptive model

If we do not want to assume known frequencies, we can use a simple adaptive model. The frequency count for each symbol is initialized to one. As the encoder processes symbols it increments the counts to update the probability model. The decoder mimics this process exactly, but in reverse. We still need to know all distinct symbols that the encoder and decoder expects to see beforehand though (unless we iterate over the data twice).

>>> symbols = list(set(message))
>>> encoder = ArithmeticEncoder(symbols, bits=32)
>>> bits = list(encoder.encode(message))
>>> len(bits)
5260166
>>> decoded = list(encoder.decode(bits))
>>> decoded == message
True

A comparison table

Comparing all the methods we’ve mentioned, we get the following table.

Compression Method	Bits Used
Naive encoding (7 bits/symbol)	8,149,967
Huffman encoding	5,298,110
Arithmetic encoding (adaptive)	5,260,166
Arithmetic encoding (fixed)	5,259,420
Shannon entropy bound	5,259,418.9

Naively encoding each symbol using a fixed bit length is wasteful, since it does not account for the fact that some symbols are much more frequent than others. Huffman encoding improves greatly upon this, but arithmetic encoding is even better and is able to get within two bits of the Shannon bound.

Summary

Arithmetic coding maps a sequence of symbols into a sequence of bits. Unlike Huffman coding, there is no one-to-one correspondence between a single symbol and a single bit sequence. Both methods exploit the fact that some symbols are typically more common than others.

With arithmetic coding there are two types of probability models: fixed and adaptive. A fixed model uses a static frequency count, computed by either looping the data in advance or by observing a larger corpus of text. A dynamic model starts out with equal probabilities for all symbols, and adapts to the specific message as it iterates through the symbols. Regardless of which model is used, the logic of the decoder must exactly match the logic of the encoder.

Turnusplaner for sykepleiere

Tommy Odland — Thu, 08 Aug 2024 00:00:00 +0200

Det er mange triste saker om dårlige turnusplaner i media. Her er et utvalg fra NRK:

Men det er også håp. I en artikkel fra sykepleien.no forteller Marita om hennes “matematiske turnus”. Hun fordeler ubekvemme vakter tilnærmet likt på sykepleierne ut fra stillingsprosent. Dette virker som en helt åpenbar og fornuftig tilnærming.

Vi kan ta Maritas idé enda lengre og faktisk bruke matematikk til å lage en turnusplan. I denne artikkelen lager vi fem matematiske optimeringsmodeller med økende kompleksitet og antall krav til turnusplanen.

Nok sykepleiere på jobb

Det første kravet er å ha nok folk på jobb. I vårt fiktive eksempel tar vi utgangspunkt i 22 sykepleiere og tre typer vakter: dagvakt, aftenvakt og nattevakt. Ifølge Norsk Sykepleierforbund har en sykepleier rett til å se turnusen senest 14 dager i forkant. Vi lager derfor en 28 dagers turnus for å være på den sikre siden. Grunnbemanningen vi ønsker å oppnå er 8 personer på dagvakt, 4 på aftenvakt og 2 på nattevakt. I helgene er det lavere bemanning.

Algoritmen bygger opp fra en tom turnusplan og evaluerer 100 000 forslag. Dette tar noen sekunder, og animasjonen nedenfor viser resultatet. Vi forsøker kun å dekke behovet for antall ansatte på jobb. Hver rad er en sykepleier og hver kolonne er en vakt. Tre kolonner utgjør et døgn, og det er totalt 28 døgn.

Vi oppnår perfekt dekning: vår bemanning matcher ønsket bemanning på hver eneste vakt.

Fylle opp stillingsprosenter

Vaktplanen ovenfor innfrir grunnbemanningen, men ignorerer stillingsprosentene. Av våre 22 sykepleiere skal 6 jobbe 100%, 11 jobbe 80% og 5 jobbe 60%. Fordelingen av antall timer er vist til venstre i figuren.

Vi skal nå forsøke å innfri to målsetninger samtidig:

Det skal være nok sykepleiere på jobb hver dag.
Over perioden på 28 dager skal hver sykepleier jobbe riktig antall vakter.

Disse to målsetningene er ofte i konflikt, og det er også tilfellet her. Det er umulig å både (1) alltid ha riktig antall folk på jobb og (2) innfri alle stillingsprosenter helt nøyaktig.

Disse konflikterende målene kan skape gnisninger i arbeidslivet, som da sykepleierne i Nesseby følte seg presset inn i en turnus med korte arbeidsdager og mange helgevakter. Hovedtillitsvalgt uttalte at det er “flere sykepleiere som ser etter andre jobber, og det blir vanskelig å rekruttere sykepleiere hit i en sånn type turnus”.

Gode turnuser er viktig for arbeidsmiljøet til sykepleierne og for samfunnet generelt. Matematikk kan ikke fjerne den iboende konflikten i målsetningene, men den kan finne beste trade-off. Med andre ord: om man må ofre stillingsprosentene noe, bør man kunne garantere at man ofrer minst mulig!

Friperioder mellom vakter

Ser vi nøye på turnusen ovenfor oppdager vi at den ikke er særlig god. Den har eksempelvis 48 doble vakter. Norsk Sykepleierforbund skriver om turnus at:

Det bør ikke være mindre enn 11 timer fri mellom arbeidsperiodene.

Dette har vi på ingen måte innfridd til nå. Det er ikke noe i vår matematiske modell som forhindrer doble vakter, eller tre vakter på rad for den saks skyld.

Vi kan modellere dette matematisk ved å straffe doble vakter og vakter med bare én friperiode på 8 timer mellom. Resultatene blir umiddelbart bedre:

Forrige turnus hadde 48 doble vakter og 54 vakter med bare én friperiode. Denne turnusen har null doble vakter og null vakter med bare én friperiode! Visuelt får vi mer “luft” i turnusen.

Færrest mulig nattevakter på rad

I en sak om helsefremmende turnuser fra sykepleien.no kan vi lese at et av kriteriene for en god turnus er:

Færrest mulig nattevakter på rad (maks to i uken og maks tre i en helg), og lengst mulig fri etter nattevakten.

Vi forenkler dette kravet litt og legger inn en straff for to nattevakter på rad.

Ovenfor er det null doble nattevakter, mens i forrige turnus var det 6 doble nattevakter.

Lik fordeling av antall aftenvakter og nattevakter

Til sist innfrir vi et av Maritas krav til en matematisk turnus. Hun sier at:

Vi har lagt vekt på å fordele de ubekvemme vaktene tilnærmet likt på alle, ut fra stillingsprosent. Det vil for eksempel si at alle med 100 prosent stilling har likt antall seinvakter og likt antall nattevakter.

Det er ikke enkelt å se i figuren, men ovenfor jobber en av sykepleierne 1 natt og en annen hele 5 netter, selv om de har samme stillingsprosent. Turnusen nedenfor forsøker å redusere forskjeller mellom sykepleierne, og resultatet er at det maksimalt er én nattevakt i forskjell mellom sykepleierne som har samme stillingsprosent.

Teller vi antall nattevakter for de 11 sykepleierne med 80% stilling får vi:

Fordeling i forrige turnus: [1 1 2 2 3 3 3 3 4 5 5]
Fordeling i denne turnusen: [2 2 2 2 2 2 3 3 3 3 3]

Denne turnusen er mye mer balansert. Dette gjelder for alle stillingsprosenter, og for både nattevakter og aftenvakter.

Oppsummert har turnusen ovenfor seks gode egenskaper

Faktisk bemanning matcher perfekt med ønsket bemanning
Sykepleierne jobber svært nær sin stillingsprosent
Det er ingen doble vakter
Alle vakter har minst to friperioder mellom seg
Det er ingen doble nattevakter
Nattevaktene og kveldsvaktene er jevnt fordelt

Det tar et par sekunder å finne en slik turnus på min laptop. Resultatene blir litt bedre om vi lar algoritmen kjøre lengre. En større turnus (flere sykepleiere, eller lengre enn 28 dager) vil kreve mer regnekraft.

Vi kunne ha fortsatt å legge til flere egenskaper. For eksempel kan vi fordele helgevaktene likt mellom sykepleierne. Det er også tenkelig at ulike sykepleiere har ulike ønsker. Noen misliker kanskje doble nattevakter, men andre synes det er greit. Enkelte foretrekker å jobbe kveldsvakter, andre dagvakter. Slike personlige preferanser kunne ha vært lagt til i modellen.

Oppsummering og referanser

Vaktplaner kan være en kilde til konflikt og frustrasjon, men problemstillingen er utrolig tilgivelig rent matematisk. Datastrukturene er enkle og grunnleggende algoritmer (her simulert størkning) gir gode resultater på få sekunder.

Målet var å undersøke noenlunde realistiske problemer med fiktive data. Det ville vært umulig å løse alle bemanningsproblemer med én modell. Praktisk turnusplanlegging varierer mellom ulike avdelinger innad i samme sykehus, mellom ulike sykehus, og mellom f.eks. et eldrehjem, en legevakt og et sykehus. Hver for seg kan disse problemstillingene løses med matematikk, men slike modeller må ofte spesialtilpasses.

Det har blitt skrevet forskningsartikler om matematisk optimering av sykepleieres vaktplaner siden 1960-tallet. Artikkelen The State of the Art of Nurse Rostering gir en oversikt. Det trengs neppe mer forskning, men heller at man i større grad tar kjente metoder i bruk.

En av de store fordelene med modeller er at de danner grunnlag for diskusjon. Hovedformålet er innsikt, ikke nødvendigvis et helt riktig svar. I praksis vil ofte en ekspert se over resultatene, gjøre endringer, og kjøre modellen flere ganger.

Her er en video som viser hvordan simulert størkning finner en løsning på et større problem:

The reverse pseudo-Huber loss function

Tommy Odland — Fri, 05 Jul 2024 00:00:00 +0200

In this article we introduce the reverse pseudo-Huber function

\begin{align*} R(x) &= \frac{1}{2} \left( \lvert x \rvert \sqrt{x^2 + 1} + \ln \left( \lvert x \rvert + \sqrt{x^2 + 1} \right) \right), \end{align*}

which acts like the absolute value when $|x|$ is around zero and like a square as $|x|$ becomes large.

Introduction to pseudo-Huber loss

The pseudo-Huber function $H(x) = \sqrt{x^2 + 1} - 1$ is a smooth approximation to the Huber loss. It acts like $x^2/2$ near zero and like $\lvert x \rvert$ near infinity. A typical use-case is robust regression, where we want to employ a squared loss but simultaneously limit the influence of outliers. Below we plot the pseudo-Huber function along with the two functions it approximates:

The equation for the pseudo-Huber and its derivative is

\begin{equation*} H(x) = \sqrt{x^2 + 1} - 1 \qquad H'(x) = \frac{x}{\sqrt{x^2 + 1}}. \end{equation*}

The reverse pseudo-Huber

What about the reverse case? What if we want a function that acts like $\lvert x \rvert$ near $x=0$ and like $x^2/2$ near $x = \pm \infty$? I found no such function in the literature when searching, though it would not surprise me if someone has deduced it already.

I propose the following equation for the reverse pseudo-Huber and its derivative:

\begin{align*} R(x) &= \frac{1}{2} \left( \lvert x \rvert \sqrt{x^2 + 1} + \ln \left( \lvert x \rvert + \sqrt{x^2 + 1} \right) \right) \\ R'(x) &= \operatorname{sign}(x) \sqrt{x^2 + 1} \end{align*}

Here’s a plot of the reverse pseudo-Huber:

Deriving the reverse pseudo-Huber

To derive the reverse pseudo-Huber, I plotted the derivative $H'(x)$ for $x>0$. The function $H'(x)$ acts like $\lvert x \rvert$ near $x=0$ and like $1$ near $x = \infty$.

Our goal is to find a function with the opposite behavoir—it should act like $1$ near $x=0$ and like $\lvert x \rvert$ near $x = \infty$. But this is exactly the behavoir of $H(x)$ if we simply add one to it!

Therefore we guess that $R(x) = \int_0^x \sqrt{\tau^2 + 1} \, d\tau$ would be a proposal for a function when $x>0$. We mirror it across the vertical axis to obtain symmetry. It turns out that this guess is very good indeed!

It has exactly the desired behavoir at $x=0$ and $x = \pm \infty$.
The functions $H(x)$ and $R(x)$ act as tight lower and upper bounds on both the functions $\lvert x \rvert$ and $x^2/2$.
The proposed $R(x)$ has a similar functional form as $H(x)$, since it’s more or less the integral.

Just like we can scale the pseudo-Huber as $c H(x / c)$ for $c > 0$, we can scale the reverse pseudo-Huber as $c^2 R(x / c)$. The reason we scale with the square is that $H(x)$ acts like $\lvert x \rvert$, while $R(x)$ acts like $x^2 / 2$ as $x \to \pm \infty$. To undo the scaling as $x \to \pm \infty$ we need to multiply $H(x)$ and $R(x)$ by $c$ and $c^2$ respectively.

Other approximations to Huber loss

Two alternative loss functions that act like $x^2/2$ near $x=0$ and like $\lvert x \rvert$ near $x = \pm \infty$ are

\begin{equation*} \ln \cosh x \quad \text{and} \quad x \tanh \left( x/2 \right). \end{equation*}

I was unable to construct reverse functions for these two. Applying the same trick as above does not work if we want to avoid special functions, since the integral depends on the Dilogarithm function.

Thanks to my friend Floris for discussions on the contents of this article.

Lønna til norske utviklere i 2024

Tommy Odland — Sun, 09 Jun 2024 00:00:00 +0200

Denne artikkelen ble også publisert som et leserinnlegg på kode24.no.

Kode24 har publisert data fra sin spørreundersøkelse om lønn. Her kommer en analyse av tallene. Resultatet er ganske likt som i min analyse av 2023-tallene.

Det er ti variabler i datasettet. Variablene er ikke bare korrelerte med lønna, de er også korrelerte med hverandre. Eksempelvis har arkitekter høy lønn, men de har også lang erfaring. Hvor mye av lønna bør tilskrives fagfeltet heller enn erfaringen? For å svare på dette skal vi dekomponere variablene ved hjelp av en Generalisert Additiv Modell (GAM).

Vi begynner med konklusjonen av analysen, som er oppsummert i figuren nedenfor. Seks variabler er kategoriske og fire er numeriske. Effekten av hver variabel er vist som en differanse fra modellens konstantledd, som er en lønn på 970 tusen kroner (TNOK). Usikkerheten i effektene er også vist i figuren.

Antall personer i hver kategori er vist i parentes. Frilans har en koeffisient på 250 TNOK, men ble kuttet i figuren for å vise helheten bedre.

Vi skal være forsiktige med å tolke et slikt datasett og tilhørende analyser med stor sikkerhet. Det er mange feilkilder: dataene er selvrapporterte, kode24-lesere er ikke nødvendigvis representative for IT-folk generelt, den statistiske modellen gjør visse antagelser og det er usikkerhet i effektene som modellen plukker opp.

La oss ta noen skritt tilbake og jobbe oss frem til resultatet ovenfor steg for steg, samtidig som vi gjør syv observasjoner.

(1) Medianlønna er 850 TNOK

Vi begynner med å plotte fordelingen av lønna. Tvilsomme datapunkter som sannsynligvis var feilrapporterte har blitt fjernet. Figuren nedenfor viser at medianlønna er 850 TNOK og at halvparten har lønn mellom 700 og 1010 TNOK. På bunnen av figuren er 200 tilfeldige personer vist.

(2) En additiv struktur forklarer datasettet godt

Vi skal nå bruke et utvalg fra datasettet til å forklare hvordan den additive modellen fungerer. Figuren til venstre nedenfor viser utviklere fordelt i 15 grupper basert på fag og sted. Innad i hver gruppe er medianlønna vist. Totalt sett er medianlønna 850 TNOK.

Arkitekter i Oslo har en medianlønn på 1150 TNOK. Både stedet (Oslo) og faget (arkitekt) tilsier at gruppen skal tjene godt. Hvor mye av den høye lønna bør da tilskrives hver variabel?

For å svare på dette bruker vi en additiv modell. Vi assosierer en numerisk koeffisient med hvert sted og fagfelt, som deretter legges på medianen 850 TNOK for å estimere lønna. Stedet Oslo får koeffisienten 90 TNOK og fagfeltet arkitektur får koeffisienten 220 TNOK. Modellen estimerer derfor at arkitekter i Oslo tjener 850 + 90 + 220 = 1160 TNOK.

Arkitekter i Oslo tjener i størst grad godt fordi de er arkitekter, ikke fordi de bor i Oslo. Modellen lar oss estimere hva en typisk arkitekt i Troms og Finnmark ville tjent, selv om ingen slike personer finnes i datasettet. Den additive modellstrukturen er enkel, forståelig og beskriver datasettet godt.

(3) Vi bør være forsiktige med årsakssammenhenger

Ovenfor sa vi at lønna “skyldes” en variabel og at noen tjente godt “fordi” de er arkitekter. Denne språkbruken er i kontekst av en dekomposisjon av variabler i et datasett, og er ikke ment som en kausal tolkning av virkeligheten.

Vi vet at korrelasjon ikke er det samme som kausalitet, men betyr det at vi ikke kan si noe om årsakssammenhenger? Statistikere har lenge drøftet nyansene i dette spørsmålet, og det finnes ingen enkel fasit.

Virkeligheten er kompleks og effektene som en modell finner kan skyldes både korrelasjon og kausalitet. Det er ikke nødvendigvis galt å spekulere i årsakssammenhenger, men husk at en statistisk modell alene ikke sier noe om årsak—bruk sunn fornuft og vær forsiktig med bastante konklusjoner.

(4) Variabler er korrelerte

La oss undersøke hvordan en additiv modell kan dekomponere en numerisk variabel (erfaring) og en kategorisk variabel (stilling). Her er et eksempel der variablene er sterkt korrelerte:

En typisk person med “junior” i stillingstittelen har ett års erfaring og tjener 630 TNOK.
En typisk person med “senior” i stillingstittelen har åtte års erfaring og tjener 945 TNOK.

I hvor stor grad skyldes lønnsforskjellen stillingstittelen kontra erfaringen? Modellen antar en numerisk koeffisient per stillingstittel og en glatt funksjon for erfaring.

Modellantagelsen er at lønnsøkningen som funksjon av erfaring er den samme, og at stillingstittelen gir et påslag i form av en koeffisient. Gitt lik erfaring anslår modellen at en typisk senior tjener 80 - (-80) = 160 TNOK mer enn en junior. Om vi ikke korrigerer for erfaring er differansen mellom en senior og en junior lik 945 - 630 = 315 TNOK.

Tidligere så vi hvordan modellen dekomponerte to kategoriske variabler (sted og fag). Her ser vi hvordan modellen dekomponerer en kategorisk variabel (stilling) fra en numerisk variabel (erfaring). Den endelige modellen dekomponerer alle de ti variablene samtidig.

(5) Estimater er usikre

Modellen estimerer effekten av alle variablene samtidig. Usikkerheten i estimatene påvirkes av spredningen i lønna, antall svar i spørreundersøkelsen og hvor mange koeffisienter som estimeres samtidig.

Figuren nedenfor viser usikkerheten i de glatte funksjonene som estimerer effekten av numeriske variabler på lønna. Når erfaringen øker blir usikkerheten større, fordi det er få personer med lang erfaring i datasettet. Denne typen usikkerhet forekommer i alle estimater, også i effekten av kategoriske variabler som sted og fagfelt.

At det er lite usikkerhet forbundet med effekten av erfaring de første ti årene betyr at vi er sikre på den generelle trenden. Det betyr ikke nødvendigvis at vi med sikkerhet kan predikere lønna til et enkeltindivid.

(6) Noen få variabler gir gode prediksjoner

La oss bruke en modell til å predikere lønna til enkeltindivider. Vi bruker Mean Absolute Error (MAE) som mål på prediksjonsfeil og undersøker tre ulike modeller:

En enkel modell som bare har et konstantledd og alltid gjetter medianen
Vår forklarbare additive modell (GAM), med ulike antall variabler
En Gradient Boosting (GB) maskinlæringsmodell

En modell som bare bruker et konstantledd (medianen) har en typisk feil på 213.7 TNOK.

Om vi får bruke én variabel i en GAM bør vi velge erfaring, som reduserer feilen til 148.2 TNOK. Får vi legge til enda en variabel bør vi velge jobb, som tar feilen ned til 140.5 TNOK. Slik fortsetter vi til alle variablene er med og feilen er 129.1 TNOK. Å ta med kjønn forverrer modellen litt ettersom den overtilpasser og generaliserer dårligere.

GB har en typisk feil på 119.3 TNOK, som er litt bedre enn GAM, ettersom GB ikke er begrenset til en additiv struktur. Formålet med analysen bør diktere modellvalget, og her er hovedformålet innsikt og forståelse. Vi bruker derfor GAM heller enn GB, selv om det også er mulig å trekke mye innsikt ut av black-box modeller som GB.

Noen få variabler er alt vi trenger for å predikere rimelig godt. Modellene halverer prediksjonsfeilen sammenlignet med å bruke medianen. Den resterende halvparten av lønnsforskjellene ikke er forklart av variablene i datasettet.

(7) Erfaring er viktigst

Det finnes mange mål på hvor viktige variabler er i en modell. En metode er å stille følgende spørsmål: gitt at vi kan bygge en modell med kun én variabel, hvilken bør vi velge? Dette gjorde vi i forrige seksjon, og svaret var erfaring.

En annen metode er permutation importance. Litt forenklet er idéen å først trene en modell på alle variablene. Deretter bruker vi den til å predikere, men vi lar folk lyve om én variabel. Hvor mye prediksjonskraft taper vi dersom folk er uærlige?

Resultatene er vist i figuren nedenfor. Om du lyver om erfaring vil det gå kraftig ut over modellens evne til å predikere lønna di. Om du lyver om de andre variablene har derimot mindre å si, og disse variablene kan derfor anses som mindre viktige.

Det enkleste argumentet for at erfaring er viktig er å se på figuren som oppsummerer modellen. Forskjellen mellom kort og lang erfaring er omtrent en halv million kroner i årslønn, som er langt mer enn effekten av noen annen variabel.

Oppsummering og referanser

Datasett som dette bør tolkes forsiktig, men vi kan likevel lære mye av å analysere dataene. Nåtidens fokus på maskinlæring og AI kan gi inntrykk av at formålet med alle modeller er prediksjon. Her var derimot hovedfokus innsikt, og vi forklarte halvparten av lønnsforskjellene ved hjelp av ti variabler. Jeg brukte min egen Python-pakke generalized-additive-models til analysen.

Jeg håper du fikk forståelse for hvilke faktorer som er med på å påvirke lønn, samtidig som du forhåpentligvis lærte litt om statistisk modellering. Denne gangen var problemstillingen lønn, men metodene og modellene er generelle. I dag har vi mer data, kraftigere maskiner og flere mennesker som analyserer data enn noen gang—statistikk er derfor viktigere enn noensinne.

Om man vil lese populærvitenskapelige introduksjoner til statistikk kan jeg anbefale:

Flaws and Fallacies in Statistical Thinking, av Campbell
The Book of Why, av Pearl
The Lady Tasting Tea, av Salsburg

Ønsker man faglitteratur for modelling ville jeg startet med:

Regression and Other Stories, av Gelman
Statistical Rethinking, av McElreath

Lønnskalkulator

Jeg hjalp også kode24 med å lage en lønnskalkulator. Modellen ligner på den ovenfor, men er ikke helt lik. Man trenger en enkel formel som kan implementeres i frontend, så jeg satte opp en lineær regresjonsmodell på et utvalg variabler:

\begin{align} \text{lønn} &= w_0 + w_1 \log\left( \alpha + \text{erfaring} \right) + w_2 \text{utdanning} \\ &+ w_\text{jobb} + w_\text{region} + w_\text{fag} + w_\text{tittel} \end{align}

Modellen ble trent for å minimere $\ell_1$-norm (absoluttverdi-feil) heller enn $\ell_2$-norm (kvadratsum-feil), som betyr at det er en medianregresjon. Kryssvalidering ble brukt for å regularisere koeffisientene $\boldsymbol{w}$ med $\ell_1$-norm og til å velge hyperparameter $\alpha$. For å få et $90\%$ intervall rundt prediksjonen trente jeg to kvantilregresjoner på residualene: én på $5\%$ og én på $95\%$.

Tilnærmingen er noe enkel, men fungerer godt. Resultatet er en formel som gir typisk lønn og et $90\%$-intervall som beskriver spredningen. Denne lineære modellen har en MAE på 130.7 TNOK, som kan sammenlignes med Gradient Boosting (119.3 TNOK) og et modell med bare konstantledd (213.7 TNOK). Et utvalg variabler i en lineær modell oppnår altså $(213.7 - 130.7) / (213.7 - 119.3) = 88\%$ av hva som er mulig med en kompleks maskinlæringsmodell. Det er ikke ille med tanke på hvor strukturelt enkel modellen er.

Tags for optimal information retrieval - part 1: motivation

Tommy Odland — Fri, 31 May 2024 00:00:00 +0200

This is part one of a two-article series. I recommend reading them in order:

In information management systems tags are one of several ways to arrange and filter items. The goal of using tags is to help users navigate and retrieve information efficiently. Here are two examples of real-world datasets with tags:

Persons and skills. The items are persons and the tags are their skills. Tag $i$ is applied to person $j$ if the person has the skill. The tag “data science” might apply to me.
Movies and keywords. The items are movies and the tags are keywords that describe the movies. Tag $i$ is applied to movie $j$ if the keyword is relevant for the movie. As an example, the tag “superhero” could be applied to the movie “Ant-Man.”

The figure below shows an example of a tag structure. We visualize tags in rows and items in columns. If you had to pick two out of the four tags to describe the items, which two would you choose to help your users navigate the items? The answer is not obvious.

The plan for this article is to discuss what a descriptive set of tags looks like and create an objective function that can be used to mathematically evaluate and choose descriptive tags. We’ll also apply the proposed method to the two real-world datasets introduced above and study the results.

What does a descriptive set of tags look like?

Let’s investigate which properties a good tag structure might have. It’s always a good idea to fetch a pen and paper and play with a few simple ideas when attacking problems like these. Here are my own initial thoughts about the problem.

No tag should cover very few or very many items. A tag that covers all or almost all items is hardly descriptive. For instance a movie tag “color-tv” does not appear worthwhile, since almost all films are shot in color. Conversely, extremely specific tags are also quite worthless, for instance a tag like “romantic horror filmed outdoors.”

Equal group size. Tags group the items, and in a descriptive tag structure each group should have roughly the same size. If we had to choose between the two sets of tags below, we would prefer the more balanced set of tags on the right. Notice that equal group size and equal tag size are not identical concepts—two tags could each apply to the same number of items but also overlap on some items, inducing a third group of a different size.

Balanced coverage. We define the coverage of an item as the number of tags that are applied to it. The figure below shows full one-coverage, two-coverage and three-coverage. In general some items might have large coverage while others have little or no coverage. We might wish for balanced coverage—there is no good reason for some items to have no tags while others have very many tags.

Binary trees are in a sense optimal. To compress as much information as possible into a set of tags, a binary tree structure is optimal. Using a binary tree, $m$ tags induce $2^m$ distinct groups onto the items. However, the groups induced by binary trees have very unbalanced coverage. Some items are covered by zero tags, while others will be covered by all $m$ tags.

Binary trees are optimal if the information management system shows exactly the set of items corresponding to the tags applied and not applied. But information systems rarely do—if you go to a website with tags and you apply no tags, you’ll see every item, not the items with no tags. For our purposes, binary trees are not the answer—their optimality does not correspond to how tags are typically used in information systems.

The objective function: minimize number of items viewed

It is tempting to start working with the properties above directly. We could search for a set of tags that (1) induces groups of roughly equal size and (2) has roughly equal coverage of all items. But going down this path would be a modeling mistake.

Instead of trying to satisfy several ad-hoc properties at once in a messy multi-objective problem, we should pause and think—what is it that we’re trying to accomplish here, really?

The goal of using tags is to help users retrieve information as efficiently as possible. Let’s pursue this thought and create a unified objective function. To do so, we must first understand how tags are used in more detail.

Tags and filters

Go to a few websites that use tags and look for an item: an apartment, a movie, or a new dishwasher. You’ll quickly realize that tags are used differently depending on the website and context:

Tags as one-of-many filters over mutually exclusive outcomes (“cat” or “dog”)
Tags as OR-filters (selecting tags “a” and “b” shows the union “a OR b”)
Tags as AND-filters (selecting tags “a” and “b” shows the intersection “a AND b”)

The UC Irvine Machine Learning Repository uses OR-filters for tags related to the “data types” filter on their datasets. There are 665 datasets in total, and if we activate the “Tabular” tag we filter down to 48 datasets. If we add the “Text” tag we increase the search result to 121 datasets. Adding more tags enlarges the result set by widening the selection.

The Norwegian marketplace FINN.no uses AND-filters for real estate properties. When we activate the “Elevator” tag and the “Fireplace” tag, we are presented with housing that has both an elevator and a fireplace. Adding more tags narrows the selection and decreases the size of the result set.

The objective function: We assume that we’re using tags with AND-filters, and that we’re looking for a randomly selected, but specific item. The goal is to minimize the number of items viewed on average before we find the item we’re looking for.

A view means an item considered; like going through a deck of cards one-by-one and asking “is this my card?”. Each item considered this way counts as a view, and each view costs time for the person searching through the items.

Example computation

The equation for the objective function, which we will present shortly, is a bit opaque. To make the equation more understandable we introduce it with an example.

Consider the following example with $m=3$ tags and $n=6$ items. The tag structure is described by a binary matrix, with entry $1$ if tag $i$ is applied to item $j$. Zeros are omitted for readability.

\begin{align*} \begin{bmatrix} 1 & 1 & & & & 1 \\ 1 & & 1 & 1 & 1 & 1 \\ & & & & 1 & 1 \\ \end{bmatrix} \end{align*}

Suppose we’re looking for the first item with tags $[1, 1, 0]$. We know exactly what we’re looking for, so we apply tags one and two to our AND-filter. The result of filtering all items on tags one and two is the result set $\{[1, 1, 0], [1, 1, 1]\}$. It is coincidental that we happen to be looking for the first item in this set. We want the objective to be invariant to permutations of the columns, and we assume that we have no control of the order of the filtered results. If we’re looking for the first item we’ll view one item, which is the item we wanted to find. If we’re looking for the second item we’ll view two items before we find what we’re looking for. On average we view $(1 + 2)/2 = 3/2$ items.

Now suppose we’re looking for the second item, with tags $[1, 0, 0]$. Again we know exactly what we’re looking for, so we apply the first tag to filter the items. The result set is $\{[1, 0, 0], [1, 0, 0], [1, 1, 1]\}$, with three items. The ordering is arbitrary and on average we view $(1+2+3)/3 = 2$ items before finding what we’re looking for.

We continue this way for every item. If we’re looking for the third item we’ll view $3$ items on average before we find it, $3$ for the fourth, $3/2$ for the fifth and $1$ for the sixth.

Averaging over all the six items which we might be looking for, we’ll view

\begin{align*} \frac{3/2 + 2 + 3 + 3 + 3/2 + 1}{6} = 2 \end{align*}

items on average before we find a specific item, if the item we’re looking for was drawn uniformly at random.

Equation and code

We now formalize the logic above into an equation and into code. Consider a binary matrix $T \in \{0,1\}^{m \times n}$ consisting of $m$ tags and $n$ items. An equation that implements the computation above is

\begin{align*} f(T) = \frac{1}{2} + \frac{1}{2n} \sum_{j=1}^{n} \sum_{k=1}^{n} \prod_{i=1}^{m} \left[ T_{ik} \geq T_{ij} \right]. \end{align*}

The square bracket is the Iverson bracket, which evaluates to $1$ if the argument inside it is true and $0$ otherwise.

Let’s untangle the equation a little. Given items $j$ and $k$, the expression $\prod_{i=1}^{m} \left[ T_{ik} \geq T_{ij} \right]$ equals $1$ if applying filters for tags associated with item $j$ retains item $k$. If item $k$ is excluded by the filter, then the expression becomes zero. We can think of the product as the AND function applied to a set of logical implications computed using the “greater than or equal to” function. The total number of items in the result set when we apply tags corresponding to item $j$ is therefore $\sum_{k=1}^{n} \prod_{i=1}^{m} \left[ T_{ik} \geq T_{ij} \right]$.

Assume that applying filters for item $j$ leads to a result set with $\ell$ items. On average we then have to view

\begin{align*} \frac{1 + 2 + 3 + \cdots + \ell}{\ell} = \frac{\ell (\ell + 1)/2}{\ell} = \frac{\ell + 1}{2} \end{align*}

items before finding the specific item we’re looking for.

Combining these observations together and re-arranging a little leads to the objective function $f$ above. The idea is perhaps more easily understood through code. Below is a Python snippet that implements the objective function above.

import numpy as np


def items_viewed(T):
    """Assume we're looking for a random item and we apply
    the correct AND-filters. How many items must we look at
    at on average before finding it?"""

    views = []  # Average items viewed before finding each item

    # Loop over each item (column) in T
    for j in range(T.shape[1]):
        # Applying the AND-filter, this is the number of items found
        num_found = np.all(T >= T[:, [j]], axis=0).sum()

        # On average we look at (1 + 2 + ... + num_found) / num_found
        # items, and (1 + 2 + ... + k) / k = (k + 1)/2
        views.append((num_found + 1) / 2)

    return np.mean(views)  # Average over every item


# An example where T has shape (tags, items)
T = np.array([[1, 1, 0, 0, 0, 1], 
              [1, 0, 1, 1, 1, 1], 
              [0, 0, 0, 0, 1, 1]])
items_viewed(T)  # 2.0

The function runs in $\mathcal{O}(mn^2)$ time.

A penalty for the number of tags

The objective function $f$ never penalizes adding more tags to the system (rows to the matrix $T$). A thousand tags for a thousand items is perfectly OK. We will now introduce a penalty for the number of tags. This makes the model more realistic and helps us choose an optimal subset of tags without having to specify the size of the subset beforehand.

Recall that $f(T)$ is the expected number of items we have to view before we find the item we’re looking for, averaged over all possible items that we could be looking for in the first place. We can also penalize viewing the tags themselves. After all, in a database we have to view all the $m$ tags to decide whether or not to apply each tag. Just like viewing items, this represents a cost of time wasted.

The simplest way to modify the objective to account for the number of tags is to penalize viewing a tag roughly equally to viewing an item:

\begin{align*} g(T; \alpha) = f(T) + \alpha m = \frac{1}{2} + \frac{1}{2n} \sum_{j=1}^{n} \sum_{k=1}^{n} \prod_{i=1}^{m} \left[ T_{ij} \geq T_{ik} \right] + \alpha m \end{align*}

The number $\alpha \geq 0$ is the relative cost of viewing a tag compared to an item. We set $\alpha=1$ unless we explicitly state otherwise.

While $f$ can answer questions like “which ten tags out of these $m$ are the best ones?”, the function $g$ does not require choosing the number of tags beforehand. Therefore $g$ can answer “which subset of these $m$ tags is the best one?”

Example gallery

To get a feeling for the objective function, here are a few matrices $T$ and the objectives $f(T)$.

With no tags the objective is $1/2 + n/2 = 1/2 + 12/2 = 6.5$. If one tag is available, the optimal structure is to cover half the items with it. If two tags are available, the optimal structure is that each tag covers half the items with no overlap. If three tags are available, there are several minima. One of them is a structure where each tag covers a third of the items with no overlap. Another allows symmetric overlaps between pairs of two tags, but no overlap between all three tags.

We discuss theoretical properties of the optimal tag structure in part two of this article series.

Computational solution: finding an optimal subset

Given a dataset as a matrix $T \in \{0,1\}^{m \times n}$, we want to find a subset of the $m$ tags that minimize $g$. Typically $m$ is large and we wish to eliminate tags that are redundant.

We encode the solution as a binary vector $\mathbf{x} \in \{0,1\}^{m}$. If $x_i = 1$, then tag $i$ is chosen. In Python we use $\mathbf{x}$ to subset the rows of $T$, and penalize the number of tags chosen $\sum_i x_i$.

def evaluate(x, alpha=1):
    """Given a binary vector x, evaluate the solution."""
    return items_viewed(T[x, :]) + alpha * np.sum(x)

Finding the optimal $\mathbf{x}$ is a combinatorial optimization problem with a large solution space. There are $2^m$ possible solutions, but even in such unfathomably large search spaces we can often find high-quality solutions. A number of heuristic algorithms can be used: greedy algorithms, simulated annealing, tabu search, genetic algorithms, variable neighborhood search, and so forth.

Greedy forward selection

Greedy forward selection starts with an all-zero solution vector $\mathbf{x} = \mathbf{0}$. The first iteration considers every tag, and picks the one that leads to the lowest objective. Subsequent iterations consider every tag not already chosen, and after each iteration one new tag is added to the solution. The algorithm continues to add tags in this manner until the objective no longer improves.

Each of the outer iterations requires evaluating $\mathcal{O}(m)$ tags, and each evaluation of the objective function takes $\mathcal{O}(mn^2)$ time. The number of outer iterations is at most $\mathcal{O}(m)$, so the worst case complexity is $\mathcal{O}(m^2 n^2)$.

Greedy forward selection performs well in practice, but it’s not guaranteed to find an optimal solution to this problem. An alternative approach would be a greedy backward algorithm—starting with $\mathbf{x} = \mathbf{1}$ and iteratively removing tags.

Simulated annealing

Simulated annealing starts with an all-zero vector $\mathbf{x} = \mathbf{0}$. In each iteration we choose a random element in $\mathbf{x}$ and flip it from zero to one or vice versa. If this new candidate solution is better, or if the temperature criterion kicks in, we keep the candidate solution and try to improve upon it.

Results on real-world datasets

We now consider a few real-world datasets. For each dataset we run (1) greedy forward selection and (2) simulated annealing with $\sqrt{n}m$ iterations.

Dataset: consultants and roles

Folq is a Norwegian company that matches independent software consultants with projects. One of the ways consultants describe themselves is by using binary tags for roles. Examples of roles are “Tech Lead”, “Scrummaster” and “Data Scientist.” Most roles are Norwegian words, but many of them are not too far from their English counterparts.

The dataset contains $m=48$ roles (tags) and $n=2255$ consultants (items). Are all $48$ roles needed, or could some of them be removed? Let’s see what the model has to say about the matter.

Greedy. The best objective achieved was $99.57$, and $29$ out of $48$ roles were chosen.

Simulated annealing. The best objective achieved was $100.43$, and $30$ out of $48$ roles were chosen.

The figure below shows how roles were chosen in each iteration of the forward greedy algorithm. A subset of approximately $30$ roles is all we need to efficiently find consultants in this dataset. Notice the diminishing returns adding more and more roles. It seems like $15$ roles or so does a perfectly okay job.

It should go without saying that mathematical models must be interpreted with care. Highly relevant tags could exist but not be in this dataset, because no one created them in the first place. Conversely, tags could be chosen by the model but still be nonsensical in the real world (for instance a tag for “blonde hair” for consultants). Models are often helpful, but work best alongside humans who understand the domain at hand.

Below is the full tag structure. The tags on the top were added first by the greedy solution. The items are sorted as binary vectors. Beware that even though the tag “Fullstackutvikler” (full stack developer) was added first by the greedy algorithm, it is not necessarily the most important tag in the context of all other chosen tags.

The following roles were discarded: AI-rådgiver, AI-utvikler, CISO, Content Manager, Endringsleder, Informasjonsarkitekt, Organisasjonsutvikler, Penetrasjonstester, Personvernrådgiver, Produkteier, Produktleder, Prosessleder, Security governance manager, Sikkerhetsanalytiker, Sikkerhetsarkitekt, Sikkerhetsrevisor, Sikkerhetsutvikler, Smidig coach and Virksomhetsarkitekt.

Dataset: consultants and skills

Folq also allows consultants to describe themselves using skills, such as “SQL”, “Java”, “Agile” and “UI-design.” This dataset contains $m=291$ skills (tags) and $n=2255$ consultants (items).

Greedy. The best objective achieved was $58.02$, and $38$ out of $291$ skills were chosen.

Simulated annealing. The best objective achieved was $57.01$, and $38$ out of $291$ skills were chosen.

Simulated annealing found a solution that was slightly better than the greedy solution. The figure below shows which roles were chosen in each iteration of the forward greedy algorithm.

Below is the full tag structure. The tags on the top were added first by the greedy algorithm.

Dataset: movies and keywords

We will now consider a dataset from the GitHub repository IMDB-Movie-Dataset-Analysis. It’s quite large, so we select a random subset of movies. After filtering, the dataset has $m=3341$ keywords (tags) and $n=2000$ movies (items).

We set $\alpha=10$ on this dataset, since very many tags are chosen if $\alpha=1$. This dataset has a sub-optimal structure with little overlap between tags.

Greedy. The best objective achieved was $859.39$, and $17$ out of $3341$ keywords were chosen.

Simulated annealing. We do not bother with simulated annealing on this dataset, since there is so little overlap between the tags. The greedy algorithm does a much better job in reasonable time.

The objective decreases very slowly as keywords are added. The keywords are not very descriptive, in the sense that they won’t help a user find a movie quickly.

The tag structure reveals why the objective decreases so slowly. A tag structure such as this is sub-optimal—each tag covers a small set of movies, and there is little overlap between the tags.

The same movies are also tagged with genres. There are $m=20$ movie genres, and $11$ are selected by our greedy selection if we run it with $\alpha=10$. The objective function is $234.12$ if we use genres—much better than the $859.39$ achieved with keywords. Genres result in a lower objective function value while also using fewer tags.

Summary and references

Tags should help users retrieve information efficiently. We assumed that a user is looking for a particular item, and that he knows which tags to apply. Then we asked: “which tags will, on average, minimize time spent searching for the item?” We formulated this as a mathematical optimization problem.

For concrete datasets we solve the problem using greedy forward selection or simulated annealing. This works reasonably well for small-to-medium sized datasets. It does not scale to large datasets, since evaluating the objective function takes $\mathcal{O}(mn^2)$ time, where $m$ is the number of chosen tags and $n$ is the number of items.

Minimizing the worst case–the maximum number of views—is also an option, but generally harder due to discontinuities in the objective function. An interesting question is whether the objective function can be evaluated more efficiently, either by using previous computations or by exploiting the subset structure of the tags. A more in-depth analysis of optimal tagging structures is presented in a follow-up article.

Literature review

Wikipedia has entries on subject indexing, tagging, taxonomies and information retrieval—but I found no research into optimal tag structures.

This and this question on Stack Overflow ask about the minimum number of tags that cover all items. This is the NP-hard set cover problem, but solving it does not lead to efficient information retrieval—the number of items covered by each tag is not accounted for.

In the paper “Limiting Tags Fosters Efficiency,” the authors use the conditional entropy of items given tags as a proxy for retrieval efficiency. The optimal solution is then one unique tag per item. A system with that many tags is obviously only “efficient” in a very specific and somewhat theoretical sense.

The idea in the paper “Mining Association Rules in Folksonomies” could be used on tags to create a hierarchical structure out of non-hierarchical tags, probably with varying results. The papers “Improved Annotation of the Blogosphere via Autotagging and Hierarchical Clustering” appears to be in the same vein.

Tags for optimal information retrieval - part 2: theory

Tommy Odland — Fri, 31 May 2024 00:00:00 +0200

This is part two of a two-article series. I recommend reading them in order:

In this article we investigate optimal tag structures. In the previous article we answered “which subset of a given set of $m$ tags is best?”, and we will now focus on the continuous problem “given $m$ tags, how should they be structured if we have freedom to arrange them as we wish?”

Consider again the case where we’re looking for a specific item. The item is uniformly distributed on the interval $[0, 1]$. Assume that we have $m$ tags that we can cover the interval with, to help us find the item as quickly as possible.

Each combination of tags induces a sub-interval. In the figure above, the two tags $1$ and $2$ induce intervals with lengths $p_1$, $p_{12}$ and $p_2$. The goal is to choose the tag structure (interval lengths) to minimize the total expected length of interval searched as we look for an item. As in the previous article, we assume that we are searching using an AND-filter.

Small problems

To get a feeling for the problem, we’ll consider the cases $m=1, 2, 3$ in detail before generalizing.

A single tag

Consider first a single tag. The tag covers an interval of length $0 \leq p_1 \leq 1$, as shown in the figure below. Our goal is to find the optimal interval length $p_1$.

There are two outcomes: either the item we’re looking for is in the interval covered by $p_1$ or it is not. The probability that the item we’re looking for is in the interval is $p_1$, since it is drawn uniformly from $[0, 1]$. If the item is in the interval $p_1$, then on average we will look through half the interval length $p_1/2$ before we find it. On the other hand, the probability that the item is not in the interval is $1 - p_1$. In that case, we have to look through the entire interval $[0, 1]$ since we cannot apply any tags. On average we find it after searching half the interval, so $1/2$ is the expected length of interval searched.

Combining these two cases, the objective function that measures the expected total length of interval searched is

\begin{align*} f(p_1) = p_1 \times (p_1)/2 + (1 - p_1) \times (1)/2. \end{align*}

We minimize this function subject to the constraint $0 \leq p_1 \leq 1$. The minimizer is $p_1^\star = 1/2$, and $f(p_1^\star) = 3/8$.

Two tags

Now consider two tags. There are three intervals covered by at least one tag: $p_1$ is the interval covered by only tag $1$, $p_2$ is the interval covered by only tag $2$ and $p_{12}$ is the interval covered by both tags. The figure below shows the situation:

Again we sum probabilities times outcomes. Each outcome is the expected interval searched before we find the item. Notice that if the item is in the interval covered by $p_1$ and we apply the filter for tag $1$, then an AND-filter will present us with all items covered by tag $1$, irrespective of whether or not they are also covered by tag $2$. This means that we have to search through both intervals $p_1$ and $p_{12}$.

The objective function becomes:

\begin{align*} f(p_1, p_2, p_{12}) &= p_1 \times (p_1 + p_{12})/2 \\ &+ p_2 \times (p_2 + p_{12})/2 \\ &+ p_{12} \times (p_{12})/2 \\ &+ (1 - p_1 - p_2 - p_{12}) \times (1)/2 \end{align*}

We must minimize this function, subject to $p_1 + p_2 + p_{12} \leq 1$ and $p_1, p_2, p_{12} \geq 0$. The minimizer is $(p_1^\star, p_2^\star, p_{12}^\star) = (1/2, 1/2, 0)$, which leads to the objective function value $1/4$.

Three tags

A pattern starts to emerge. The case of three tags can be visualized as follows.

There are $2^m-1$ intervals covered by at least one tag. The objective function becomes:

\begin{align*} f(p_1, p_2, \ldots, p_{123}) &= p_1 \times (p_1 + p_{12} + p_{13} + p_{123})/2 \\ &+ p_2 \times (p_2 + p_{12} + p_{23} + p_{123})/2 \\ &+ p_3 \times (p_3 + p_{13} + p_{23} + p_{123})/2 \\ &+ p_{12} \times (p_{12} + p_{123})/2 \\ &+ p_{13} \times (p_{13} + p_{123})/2 \\ &+ p_{23} \times (p_{23} + p_{123})/2 \\ &+ p_{123} \times ( p_{123})/2 \\ &+ (1 - p_1 - p_2- p_3- p_{12}- p_{13}- p_{23}- p_{123}) \times (1)/2 \end{align*}

The constraints for the minimization problem are $\sum_i p_i \leq 1$ and $p_i \geq 0$ for all tags $i$. While the $m=1$ and $m=2$ cases have unique minimizers, here the solution is a convex combination

\begin{align*} (1-\beta) \times (1/3, 1/3, 1/3, 0, 0, 0, 0) + \beta \times(0, 0, 0, 1/3, 1/3, 1/3, 0) \end{align*}

for any $0 \leq \beta \leq 1$.

General problem with $m$ tags

In the general problem, we sum over all children in the powerset of tags. The powerset can be visualized as a Hasse diagram, shown below for $m=4$ tags. The general function for $m$ tags takes $2^m-1$ variables as inputs. Let $P(m)$ be the intervals corresponding to the powerset of $m$ tags, for instance $P(3) = \{p_{1}, p_{12}, p_{2}, p_{13}, p_{123}, p_{23}, p_{3}\}$.

The general function considers each interval in the powerset in turn, and loops down through the Hasse diagram to all children (supersets). The function is

\begin{align*} f(\boldsymbol{p}) &= \sum_{p_i \in P(m)} p_i \times (p_i + \operatorname{children}(p_i))/2 \\ &+ (1 - \sum_{p_i \in P(m)} p_i)/2. \end{align*}

As always, the function must be minimized subject to constraints $p_i \geq 0$ and $\sum_{p_j \in P(m)} p_i \leq 1$.

Another way to construct this function is to use a recursive definition of the power sets, ordering the variables as follows:

$m=1$ yields the ordered set $P(1) = \{p_{1}\}$.
$m=2$ yields the ordered set $P(2) = \{p_{1}, p_{12}, p_{2}\}$.
$m=3$ yields the ordered set $P(3) = \{p_{1}, p_{12}, p_{2}, p_{13}, p_{123}, p_{23}, p_{3}\}$.
In general $P(m) = S_{m-1} \cup \{ p_i \cup m \mid p_i \in S_{m-1} \} \cup \{ p_m \}$.

Given this ordering of the variables, we can write the objective function as a quadratic form

\begin{align*} f(\boldsymbol{p}) = \frac{1}{2} \boldsymbol{p}^T N_{k} \boldsymbol{p} - \frac{1}{2} \boldsymbol{1}^T\boldsymbol{p} + \frac{1}{2}, \end{align*}

where the non-symmetric matrix $N_{k} \in \{0, 1\}^{2^m-1}$ is recursively defined as

\begin{align*} N_1 = \begin{bmatrix}1\end{bmatrix} \quad N_{k+1}= \begin{bmatrix} N_k & N_k & \\ & N_k & \\ & \mathbf{1}^T & 1 \\ \end{bmatrix}. \end{align*}

Reducing the number of variables with symmetry

We make three observations about the objective function:

The function $f(\boldsymbol{p})$ is invariant to permutations of values of variables that represent the same cardinality elements in the power set (the same cover of the items). For instance, when $m=2$ the solutions $(p_1, p_2, p_{12}) = (1/3, 1/2, 0)$ and $(p_1, p_2, p_{12}) = (1/2, 1/3, 0)$ yield identical objective function values.
If only the variables that represent a given cardinarily in the power set are non-zero, then the optimal solution is to let them be equal to each other. In other words, if we consider only one level in the Hasse diagram, then the optimal solution is to spread out the coverage equally across all variables associated with that level. For instance, if we only consider variables that represent cardinality three in the $m=4$ problem, then $p_{123} = p_{124} = p_{134} = p_{234} = 1/4$ is optimal.
If we consider sets of variables with a given cardinality, i.e., a single level in the Hasse diagram, then the level with the most variables is the optimal level. With $m\geq2$ tags this is level $\lfloor m/2 \rfloor$, which has $\binom{m}{\lfloor m/2 \rfloor}$ variables representing unique intervals all having $\lfloor m/2 \rfloor$-coverage. This will be explained in more detail later in the article.

We conjecture that it suffices to check solutions where each variable that represents the same cardinality elements in the power set is equal, for instance if $m=3$ then we assume $p_{1} = p_{2} = p_{3}$ and $p_{12} = p_{13} = p_{23}$. By this symmetry argument, we reduce the number of variables from $2^m$ to $m$.

Let $q_i$ be the value of every variable that represents $i$ cardinality variables, i.e., intervals covered by exactly $i$ tags. For instance, if $m=3$ then the $q_1$ is the value of variables $p_1, p_2, p_3$ and $q_2$ is the value of $p_{12}, p_{13}, p_{23}$ and $q_3$ is the value of $p_{123}$.

With respect to the new variable vector $\boldsymbol{q}$, the problem becomes:

\begin{align*} f(\boldsymbol{q}) &= \frac{1}{2} \sum_{i=1}^m \binom{m}{i} q_i \sum_{k=i}^m q_k \binom{m-i}{k-i} - \frac{1}{2} \sum_{i=1}^m \binom{m}{i} q_i + \frac{1}{2} \\ &= \frac{1}{2} \sum_{i=1}^m \sum_{k=i}^m \binom{m}{k} \binom{k}{i} q_i q_k - \frac{1}{2} \sum_{i=1}^m \binom{m}{i} q_i + \frac{1}{2} \\ &= \frac{1}{2} \boldsymbol{q}^T C_m \boldsymbol{q} - \frac{1}{2} \boldsymbol{c}_m^T \boldsymbol{q} + \frac{1}{2} \end{align*}

We minimize this function with the constraint $\boldsymbol{c}_m^T \boldsymbol{q} \leq 1$ and $\boldsymbol{q} \geq \boldsymbol{0}$.

The problem is not convex

With one and two tags the problem is strictly convex. With three tags the problem is convex, but not strictly convex. With four or more tags the problem is neither convex nor concave.

Here is an example with $m=4$ showing the non-convexity of $f(\boldsymbol{q})$. Along one search direction in four dimensional space the function is convex, but along another search direction it is concave.

For each $m$ we will now search along a path that represents convex combinations of neighboring levels in the Hasse diagram. In the figure below $0$ maps to $(0, 0, 0, \ldots)$, $1$ maps to $(1/\binom{m}{1}, 0, 0, \ldots)$, $2$ maps to $(0, 1/\binom{m}{2}, 0, \ldots)$ and so forth. A decimal value like $1.3$ maps to a convex combination of the vectors that $1$ and $2$ map to.

We observe that setting $q_i = 1/\binom{m}{i}$ at level $i = \lfloor m/2 \rfloor$ minimizes the function along the path searched in the figure. A formal proof that this is the best achievable solution escapes me. However, in millions of computer simulations no better solution was found for $m\geq 2$, so I am quite confident that it’s the optimal solution.

Two bounds on the optimal solution

We present two upper bounds on the minimization problem—the objective is at least as low as these.

One-coverage. A bound for $m \geq 2$ can be established by covering each item with a single tag, where each tag covers an interval of length $1/m$. This is represented by $\boldsymbol{q} = (1/m, 0, 0, \ldots)$, which is clearly a feasible solution. The objective function value is $f(\boldsymbol{q}) = 1/(2m)$. In the discrete case of $n$ items, the objective function $g$, which penalizes the number of tags too, becomes

\begin{align*} g(m) = \frac{n}{2m} + \alpha m. \end{align*}

Differentiating with respect to $m$ and solving $g'(m) = 0$ shows us that the optimal number of tags in this situation is $m^\star = \sqrt{n/2 \alpha }$. Furthermore $g(m^\star) = \sqrt{2n\alpha}$. To summarize, if we construct a set of $\sqrt{n/2 \alpha}$ mutually exclusive tags we have a reasonable starting point in a real-world system.

The conjectured optimal coverage. We conjecture that the following is an optimal solution for $m \geq 2$ tags. Choose a full, balanced $\lfloor m/2 \rfloor$ coverage. In other words, choose the level in the Hasse diagram with the largest number of variables. This is represented by setting $q_i = 1/\binom{m}{i}$ at level $i = \lfloor m/2 \rfloor$, and the remaining entries in $\boldsymbol{q}$ are set to zero. The objective function value can be shown to be $f(\boldsymbol{q}) = \frac{1}{2 \binom{m}{\lfloor m/2 \rfloor}}$.

Solving

\begin{align*} g(m) = \frac{n}{2 \binom{m}{\lfloor m/2 \rfloor}} + \alpha m \end{align*}

for the optimal $m$ must be done numerically, as no simple analytical expression is available. For instance, if we set $\alpha=1$ and have $n=10^2$ items, then $m=7$ tags is the optimal number, given that those tags obey the conjectured optimal structure. With $n=10^3$ items $m=10$ tags is all we need, $n=10^4$ items require $m=14$ tags and $n=10^6$ items require only $m=21$ tags.

Here are two examples showing optimal tag structures:

Summary and references

In the article leading up to this one we solved a combinatorial optimization problem using real-world data. That led us to wonder more about the theoretically optimal tag structure, which we investigated in this article.

We began by solving small cases with one, two and three tags. Only after understanding the objective function for these small cases did we generalize. This is how I typically work with mathematical problems—mastering the small concrete cases first, then attempting to generalize.

After writing down the objective function for $m$ tags, we made observations regarding the symmetry of the function. We reduced it from a function taking $2^m$ arguments to one taking $m$ arguments. Then we determined that the function was not convex and established two upper bounds, one of which we conjecture to be the optimal solution. I was unable to prove this, but ran millions of computer simulations trying out random tag structures, and not a single one beat the bound—so I believe the conjecture is true.

For me this was an interesting problem to look at, since I found no literature on this subject. Wikipedia has entries on subject indexing, tagging, taxonomies and information retrieval—but I found no research into how tags can be used for maximally efficient information retrieval.

tommyodland.com

Ten lessons from twenty years of strength training

Start easy

Stay consistent

Push yourself

Two steps forward, one step back

Switch it up

Movements, not muscles

Compete or cooperate

Food makes you look good

Mute the marketers

Keep learning

Optimal bus stop spacing

Simulation

Analytical solution

Deltidsstillinger

Dagens praksis med merarbeid og overtid

EU-domstolens konklusjon

Problemstillingen

Et kompromiss

Continuous interest with continuous deposits

A simple savings calculator

The discretization error

Continuous deposits

Solution by integration

Solution by differential equations

Summary and references

Skatter og bidrag

Inntekt

Matematiske egenskaper

Hva kan vi forenkle?

Vanlige folks tur

Oppsummering: Inntekt

Bidrag og støtteordninger

Å fange de fattige

Bidrag \(B(x)\) med gode egenskaper

Hva kan vi forenkle?

En realistisk beregning

Oppsummering: Bidrag og støtteordninger

Formuesskatt

Oppsummering

Referanser og videre lesing

Appendiks A: Bevis for at bidragsfunksjonen har gode insentiver

Appendiks B: Formuesskatten

Appendiks C: Kode

Practical Linear Algebra with NumPy

Don’t compare floats

Don’t multiply diagonal matrices

Don’t add diagonal matrices

Don’t multiply matrices in arbitrary order

Don’t compute a product if you only need the diagonal

Don’t form the inverse matrix

Don’t use memory you don’t need

Don’t use pure Python loops

Do use the Woodbury identity

Do think about what the meaning is

Do time everything

Do read the documentation

Do consider approximations

Resources and further reading

Books

Mathematics

Scientific computing

Python and programming

Machine learning

Deep learning etc.

Statistics

Optimization

Algorithms

Popular science

Miscellaneous

Fastlegebytter

Bytter mellom fulle lister

Lengre sykluser

Oppsummering og referanser

Korrespondanse med Norsk helsenett SF

Monte Carlo modeling in Python with probabilit

Monte Carlo modeling examples¶

Braking distance of a car¶

Mutual fund¶