• chiliedogg@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    1 day ago

    Yes, but how do you record that precision without needing a qualifying statement. When precision matters, “0.25” represents a measurement that is known to be closer to 0.25 than it is to either 0.24 or 0.26. Something that is only precise to 1/4 of a unit isn’t that precise. The decimal way to record a precision of 1/4 is “0.25 +/- 0.125”.

    The thing to understand about decimals and precision is that you’re still recording a fractional measurement, but your denominator is fixed to powers of 10. 0.1 is 1/10. 0.01 is 1/100. So when increasing precision by less than a factor of 10 is difficult to represent.

    This matters a lot for things like digital calipers, where a cheap set will show the same measurement as a nice set that’s more precise because the good ones aren’t 10 times as precise. But if they have a fractional setting, the nicer ones will read more precisely because that increased precision can be represented on the display.