Let me make this concrete. Schools care about test scores, but only because they measure learning. It’s only a measure, until you use it to determine graduation requirements. Investors care about bond ratings, but only because they measure risk of default. It’s only a measure, until you use it to determine capital reserves. Bank regulators care about capital reserves, but only because it is a measure of solvency. It’s only a measure, until you use it to set bank reserve requirements.
Metrics and measures are used to bias technical systems:
Specify a metric for user engagement, and as Zeynep Tufekci pointed out in a very worthwhile analysis, Facebook starts to select for sensationalism and garbage. In the article, she says this is because algorithms are not neutral — but I think she’s wrong. Tools themselves are neutral, but how they are used are not. Once we use a neutral algorithmic tool to pursue a goal using a metric, the system is no longer neutral — it’s biased by the metric. So we see that once you specify a metric for reducing recidivism in convicts, you create racial bias. The measure used collapsed the multidimensional goals into a metric that didn’t include fairness, so the system doesn’t make itself fair.
The authors behind ribbonfarm also created Breaking Smart.