What is the score?

The most common sentence at my padel club is “What is the score?” Usually I am the one asking.

It should not be that hard. Each side only counts up to three points: 15, 30, 40. Four adults with jobs still lose count before we get there, so we stop and settle on a number nobody believes.

Professionals do not have this problem. They have a person in a tall chair whose whole job is to watch the ball and say the score out loud. At my club the closest thing we have to that is whoever is the loudest.

So I asked what keeping score would become if it were built AI first. My initial idea was a phone at the back of the court that watches the game, calls each point out loud and sends the score to the players’ watches. Nobody touches anything.

Teaching it to watch

I collected videos of real matches and set out to train a model to find the ball and say who won each point.

I thought training a vision model would require me to subscribe to a GPU farm or build my own server room. While I was debating that, my laptop started crawling. I was sure it was Chrome. It turns out that while I was wondering where on earth I was going to find the resources to train a model, Claude had already taken over my laptop’s GPU and was almost done with the first training run. I owe Chrome an apology.

I pressed play expecting the green line to wander around the court the way I do. It stayed on the ball for the whole rally and then called the point correctly. I kept staring at the screen for a fairly long time. This is that playback.

In those videos the ball is a few yellow pixels, and a padel court is full of things that are a few yellow pixels. For a while the model followed a spectator’s shoe. On another point it locked onto a player’s shirt logo while the ball flew past him. In fairness, I have lost points the same way.

When it does see the point, it is a pleasure to watch. It follows the last ball to the first thing it touches, then says who won and why.

Two ways to keep score

The first is the one I want. The phone watches, announces the point, the watches update, and nobody does anything. The model is not there yet. It was pretty accurate, but not foolproof, and at five games all a scorekeeper has to be foolproof. Pretty accurate we already have. There are four of us on the court, which on a good day add up to one almost accurate umpire.

The second asks one player to press a button after every point. I built that one while I keep training the first. It is a watch app with two buttons, US and THEM. You tap the side that won the point, and it knows the rules, so deuce and tie-breaks take care of themselves.

The watch also saves every tap with its time, so each match I play with it becomes an answer key for the camera.

I wore it in a real game today. I still lost, so now even my watch knows how badly I play.

The takeaway

When the version you dream of is not ready, ship the one that still needs you, and make it take notes. Every point I tap on my watch is one more labeled point for the model. I am training my watch’s replacement, one losing game at a time.