Tag: precision

The precision of a value describes the number of digits that are used to express that value. In a scientific setting this would be the total number of digits (sometimes called the significant figures or significant digits) or, less commonly, the number of fractional digits or decimal places (the number of digits following the point). This second definition is useful in financial and engineering applications where the number of digits in the fractional part has particular importance.
In both cases, the term precision can be used to describe the position at which an inexact result will be rounded. For example, in floating point arithmetic, a result is rounded to a given or fixed precision, which is the length of the resulting significand. In financial calculations, a number is often rounded to a given number of places (for example, to two places after the decimal separator for many world currencies).
As an illustration, the decimal quantity 12.345 can be expressed with various numbers…