Of tires and small sample sizes
 

The other thread about reliable tubulars got me thinking. So much of what we hear about tires is anecdotal information.

I'd argue that there is simply no way for a single person to evaluate puncture resistance of a tire in any meaningful way. For one, even a person who is riding 10,000 miles a year can only be going through a dozen or so tires, and that does not even begin to scratch the surface of what is available in terms of brands, models and sizes. Second, there is no way to escape the cognitive biases that humans have (specifically availability bias -- more likely to remember the high stress punctures compared to the one you find at home -- and confirmation bias -- the idea that once you have a hypothesis (this tire is prone to punctures) you're more likely to find supporting evidence, and disregard non-confirming evidence). And third, even if you're riding on the same roads, punctures are rare enough that it is hard to say whether 2 or 3 extra punctures over a few thousand miles is the result of a real difference in tires, or just bad luck.

Is there any large scale study of tire durability that tries to eliminate these issues and actually measure a statistically reliable difference in durability and puncture resistance?

I agree with this completely. For me anyway, flat tires happen very infrequently, and when they do happen, under a very wide range of circumstances. I think most of the time it happens under a confluence of a number of random and unpredictable events, so as to not be able to discern any pattern.

Some studies attempt to measure puncture resistance by pressing tires against sharp objects, to determine the force/time required to puncture (or if they withstand a particular threshold). I'm not sure this is a realistic scenario, but I'm not sure what other test they could do that doesn't require a long time and large sample size.

not sure about the efficacy of their methods, but...
 

these guys seem serious:

https://www.bicyclerollingresistance...d-bike-reviews

If it harder to stick a needle through one tire than it is stick it through another tire, I'd say the first one is more puncture resistant :).

Yes, this is a good approach to having a baseline that is consistent and systematic. Here's their testing protocol.

The logical jump that is easy to make, but that I don't see any evidence of, is that real world punctures are related to punctures in the lab with a 1mm thick steel needle perpendicular to the tire.

How does a broken shard of glass compare? That is more of a slicing action than a piercing action. Also, we have arguments from Jan Heine that pressure can also make a difference in puncture protection.

Another point about that test is what is the tread depth when they do the test? Is it a new tire? 1000 miles on it? threads starting to show? And how does that compare to the average tire on the road?

Also, what is the difference in actual punctures between a 5 and 10 rating? If I understand their measurements, a 240 pound combined bike/rider system would exert twice the force on tires than a 120 pound bike/rider system. Does the real world evidence bear out that the heavier rider gets more punctures than the lighter one, and is that relationship 2 to 1?