Describing the uncertainties in experimental results

Describing the Uncertainties in Experimental Results Robert J. Moffat Professor of Mechanical Engineering, Stanford University, Stanford, California I I It is no longer acceptable, in most circles, to present experimental results without describing the uncertainties involved. Besides its obvious role in pub- lishing, uncertainty analysis provides the experimenter a rational way of eval- uating the significance of the scatter on repeated trials. This can be a powerful tool in locating the source of trouble in a misbehaving experiment. To the user of the data, a statement (by the experimenter) of the range within which the results of the present experiment might have fallen by chance alone is of great help in deciding whether the present data agree with past results or dif- fer from them. These benefits can be realized only if both the experimenter and the reader understand what an uncertainty analysis is, what it can do (and cannot do), and how to interpret its results. This paper begins with a general description of the sources of errors in en- gineering measurements and the relationship between error and uncertainty. Then the path of an uncertainty analysis is traced from its first step, identify- ing the intended true value of a measurement, through the quantitative estima- tion of the individual errors, to the end objective--the interpretation and re- porting of the results. The basic mathematics of both single-sample and multiple-sample analysis are presented, as well as a technique for numerically executing uncertainty analyses when computerized data interpretation is in- volved. The material presented in this paper covers the method of describing the uncertainties in an engineering experiment and the necessary background ma- terial. Keywords: experimental uncertainty, error analysis, single-sample analysis, multiple-sample analysis, system errors INTRODUCTION The error in a measurement is usually defined as the difference between its true value and the measured value. This definition is clear but not very helpful: the only real situations in which we even claim to know both the true value and the measured value are those in which we are calibrating or "qualifying" an experiment against baseline data or against one of the basic conservation laws of engineering, In most situations, we cannot talk very confidently about what the error in a measurement is, we can only talk about what it might be--about the limits that we feel bound the possible error. The term "uncertainty" is used to refer to "a possible value that an error may have." Kline and McClintock [1] attribute this definition to Airy [2], and it still seems an appropriate and valuable concept. The terms "uncertainty interval" and "uncer- tainty" are commonly used interchangeably, and they will be so used in this discussion, both referring to the interval around the measured value within which the true value is believed to lie. The term "uncertainty analysis" refers to the process of estimating how great an effect the uncertainties in the individual measure- ments have on the calculated result. There is more to uncertainty analysis than just drawing error bars on a plot of the data or calculating the root-mean-square deviation of data from a curve fit, or (worse yet) calculating the mean absolute value of that deviation. Those are simply tech- niques for providing some measure of the scatter in the results and provide no way of judging whether or not the observed scatter was "reasonable"; that is where uncertainty analysis comes in. Uncertainty analysis began as the statistical interpretation of the errors in well-replicated experimental results. It became apparent during the early 1950s that many important engineering experi- ments could not be repeated enough times to provide useful statistical information, for reasons of economy or pressure of time. A rational way to use the framework of statistical inference to estimate the uncertainty in these single-sample experiments was described by Kline and McClintock [1] and still forms the basis for this branch of the an. Over the years, single-sample uncertainty analysis has been used more in research experiments than in production, and that association has led to its evolution as a diagnostic tool for the development of experiments. As a consequence, single-sample uncertainty analysis presents two uncertainty measures that are particularly useful during the planning and debugging stages of experiments, in addition to the usual "overall uncertainty." The distinction between single-sample and multiple sample Address correspondence to Professor Robert J. Moffat, Mechanical Engineering Department, Stanford University, Stanford, CA 94305. Experimental Thermal and Fluid Science 1988; 1:3-17 ©1988 by Elsevier Science Publishing Co., Inc., 52 Vanderbilt Avenue, New York, NY 10017 0894-1777/88/$3.50 4 R. J . Moffat analysis hinges on whether or not a " large" or a "smal l" number of independent data points are taken at each test point and on how the data are handled. In this era of high-speed digital data acquisition, the issue of independence takes on more subtle overtones than it had in the early 1950s when the term was coined. Consider, for example, a measuring system capable of acquiring data at 100 kHz, a system readily available under current technology. Applied to an experiment whose output varied at approximately 10 mHz, this 100 kHz sample rate would produce about 100 readings in a 1 ms period. These can each be regarded as independent measures of the process, assuming the time between consecutive readings to be large compared with the autocorrelation time of the signal. This set of 100 readings would be, then, a multiple-sample set of observations. The same equipment, applied to a system whose output varied at 1 Hz, would produce a single-sample measure of the process, tainted, perhaps, by a multiple-sample contribution from the high- frequency random errors present in the measuring system. Classification of a given experiment as single-sample or multiple-sample must take into account not only the number of observations made at each test point but also the data-sampling rate and the spectrum of significant frequencies in the process being studied and whether the data are averaged before processing or processed before averaging. Single-sample uncertainty analysis has been described in the engineering literature by the works of Kline and McClintock [ 1] and Moffat [3, 4]. The techniques of multiple-sample analysis are described by Abernethy and Thompson [5] and summarized by Abernethy et al [6] and by ANSI/ASME PTC 19.1-1985 |7]. Both have the same final objective (to estimate the effect of the accumulated measurement uncertainties on the accuracy of the result), even though somewhat different procedures are required. Either the Nth-order uncertainty interval of single-sample analysis or the U0.95 interval of multiple-sample analysis answers the question: How close to this result does the true value probably lie. assuming that these are the right equations to use, that all the important variables have been included, and that the test situation is representative? In addition to this final result, each method produces auxiliary information about the experiment, mainly useful as diagnostics, during the developmental phase of an experiment or in monitoring its "health" during a long series of runs. Single-sample uncer- tainty analysis generates two diagnostics: the zeroth-order uncer- tainty, which evaluates the contribution to the total uncertainty introduced by the measuring system, and the first-order uncer- tainty, which predicts the scatter that should be observed on repeated trials with the same equipment and the same instruments. Multiple-sample uncertainty analysis produces estimates of the total fixed error and the total random error in the result. An uncertainty estimate of either type is only as good as the equation(s) it is based on. If those equations are incomplete and do not acknowledge all the significant factors that affect the result, o, if falsely low values are used for the component uncertainties, then the analysis will underestimate the uncertainty in the result. On the other hand, if the component uncertainties are exagger- ated, then the analysis will overestimate the uncertainty. The most common, and most visible, use of uncertainty analysis is in reporting results to the technical community through publications, but it must be noted here that it has far broader uses. During the shakedown period of an experiment, it is a powerful diagnostic tool in seeking out the sources of residual error. In the early stages of an experiment, for example, when comparing the first results from a new test rig with those from an existing baseline set, the most frequent question is: Does the difference 1 see mean that the new results are really different, or is this difference just a consequence of the uncertainties in my measure. ments? Uncertainty analysis provides clear, unambiguous guid- ance: If the observed difference exceeds zero by more than the expected uncertainty interval for the difference, then the observed difference is probably significant. Even earlier in an experiment, uncertainty analysis can be used to help choose the most reliable technique for a given measurement or to identify the critically important instruments in a system (and thereby determine where expensive instruments are needed!). THE BAS IC MATHEMATICS This section introduces the root-sum-square (RSS) combination, the basic form used for combining uncertainty contributions in both single-sample and multiple-sample analyses. In this section, the term 3Xi refers to the uncertainty in Xi in a general and nonspecific way: whatever is being dealt with at the moment (tor example, fixed errors, random errors, or uncertainties). Describing One Variable Consider a variable Xi, which has a known uncertainty 6X,. The form for representing this variable and its uncertainty is X,=Xi(measured)+_6Xi (20:1) (1~ This statement should be interpreted to mean the lollowing: • The best estimate of X, is Xi (measured) • There is an uncertainty in Xi that may be as large as +_ bX~_ • The odds are 20 to 1 against the uncertainty of Xi being larger than +_ 8X,. The value of Xi (measured) represents the observation in a single-sample experiment or the mean of a set of N observations in a multiple-sample experiment. The value of 6Xi represents 20 for a single-sample analysis, where o is the standard deviation of the population of possible measurements from which the single sample Xi was taken. For multiple-sample experiments, 6Xi can have three meanings. It may represent tS~N)/q-N for random error components, where ScN I is the standard deviation of the set of N observations used to calculate the mean value Xi and t is the Student's t statistic appropriate for the number of samples N and the confidence level desired. It may represent the bias limit for fixed errors (this interpretation implicitly requires that the bias limit be estimated at 20:1 odds). Finally, 8Xi may represent U95, the overall uncer- tainty in X,. The Student's t multiplier is a number, always larger than 2.0, that allows one to use S~.~ rather than o in estimating the uncertainty in the mean of a set. The Student's t statistic is tabulated in most statistical reference books under that name The result R of the experiment is assumed to be calculated from a set of measurements using a data interpretation program (b~ hand or by computer) represented by R=R(XI, X2, A'~, "- - , Xx) ~2i The objective is to express the uncertainty in the calculated result at the same odds as were used in estimating the uncertainties in the measurements. This issue was taken up by Kline and McClintock [1], who showed that the uncertainty in a computed result could be estimated with good accuracy using a root-sum square combination of the effects of each of the individual inputs and that the RSS operation preserved the odds. The effect of the uncertainty in a single measurement on the Uncertainties in Experimental Results 5 calculated result, if only that one measurement were in error would be fR _ OR Xi- -~i fxi (3) The partial derivative of R with respect to Xi is the sensitivity coefficient for the result R with respect to the measurement Xi. When several independent variables are used in the function R, the individual terms are combined by a root-sum-square method. fR = 6Xi (4) This is the basic equation of uncertainty analysis. Each term represents the contribution made by the uncertainty in one variable, fXi, to the overall uncertainty in the result, fiR. Each term has the same form: the partial derivative of R with respect to Xi multiplied by the uncertainty interval for that variable. The estimated uncertainty in the result has the same probability of encompassing the true value of the result as the uncertainties in the individual measurements have of encompassing their true values. Equation (4) applies as long as [1] 1. Each of the measurements was independent 2. Repeated observations of each measurement, if made, would display Gaussian distributions 3. The uncertainty in each measurement was initially expressed at the same odds The data interpretation program may be simple enough that all the partial derivatives can be evaluated analytically, or it may be too complex for that and require direct computer analysis; the procedures are the same in either case. A general technique for computerized uncertainty analysis is given in the next section. The following comments might be helpful in "hand" analysis; they are mainly aimed at simplifying the task. In most situations, the overall uncertainty in a given result is dominated by only a few of its terms. Terms in the uncertainty equation that are smaller than the largest term by a factor of 3 or more can usually be ignored. This is a natural consequence of the RSS combination: Small terms have very small effects. There are exceptions, of course, where there are many terms of approxi- mately the same size, but in general that is not the case. In many applications, the uncertainty estimate is wanted as a fraction of reading, rather than in engineering units. While this can always be calculated, starting from the results of the general form in Eq. (4), it is sometimes possible to do the calculation of relative uncertainty directly. In particular, whenever the equation describing the result is a pure "product form," such as Eq. (5), or can be put into that form, then the relative uncertainty can be found directly. That is, if , b ~. ~ (5) R=XlX2X 3 .. X M then fR [( fXl~2 ( fX2~2 _ ( fXm' l l /2 -~= a--~-i ; + b X2 /i " "+ m Xm ) (6) This is a natural and convenient approach in situations where the uncertainties of the component measurements are described in terms of percent of reading and the result is needed in the same terms. The exponent of Xi becomes its sensitivity coefficient. COMPUTERIZED UNCERTAINTY ANALYS IS When R is calculated using a large-scale computer program or involves operations that are difficult to differentiate (e.g., table look-ups or numerical integrations), the operations represented by Eqs. (3) and (4) either cannot or will not be done by hand. In most cases, it is not practical to write a separate computer program for the evaluation of uncertainties, both from the standpoint of complexity and because of the difficulty of ensuring that the uncertainty code is updated each time the main data interpretation code is revised. For these more complex experiments, the data interpretation program itself can be used to generate the uncer- tainty analysis, by sequentially perturbing the input values and accumulating the individual uncertainty contributions. This direct computer-executed uncertainty analysis can be accomplished by sequentially perturbing the inputs according to the following procedure [4]: 1. Calculate the result R for the recorded data. Identify the value as R0 and store it. 2. For i = 1 to N, where N is the number of variables in R: Increase the value of the ith variable, Xi, by its uncertainty interval, fXi, and calculate the result, Ri+ , using the aug- mented value of the ith variable with all other variables at their recorded (nominal) values. Find the difference Ri+ - Ro and store it as Ci+, the contribution to the uncertainty of R caused by the ith variable, assuming a positive excursion. If the result R is likely to be a strongly nonlinear function of Xi, including consideration of the size of its uncertainty interval, then also calculate Ci- using Ri- - Ro. The present recommendation would be to use the average of the absolute values of Ci+ and Ci- as the working value of Ci, but that recommendation must be regarded as tentative, since no definitive analysis has been done to investigate the issue. Next i. 3. The uncertainty in the result is the root-sum-square of the Ci. A primary advantage of this method is that the actual working data interpretation program itself is used in the assessment of uncertainties. Thus, each time the program is modified, the modifications are automatically incorporated into the uncertainty calculations. The process of determining the uncertainty in the computed heat transfer coefficient from its basic measurements is illustrated in Fig. 1. The method of sequential perturbations can also be applied to data interpretation programs of intermediate difficulty (too large to do by hand but not large enough to warrant a dedicated program) using spreadsheets on a personal computer, as pointed out by Catz [8]. THE SOURCES OF UNCERTAINT IES The uncertainty attributed to a measurement is an estimate of the possible residual error in that measurement after all proposed corrections have been made. Although this discussion of the sources of uncertainties will begin with a discussion of errors, the distinction between the original error and the residual error must be clearly maintained. The uncertainty is determined by the residual error after correction, not the original error. An error source is usually categorized as "fixed" or "random" depending on whether the error it introduces is steady or changes during the time of one complete experiment. The errors themsel- ves are called bias or precision errors, and the precision error is presumed to behave randomly, with a zero mean. Both the bias and precision are presumed to represent stationary statistical properties of a Gaussian distributed data set. Such a description tacitly assumes that each observation of the error is independent of its preceding observation--that the er ror source in question has no systematic variation. This is true only 6 R. J . Moffat Figure 1. The method of sequen- tial perturbation for calculating uncertainty intervals from the data reduction program. INPUT - Q",T~,T2,&(~",6T~,&T2 ho= i Run "O" Run 1 Run 2 Run 3 (Base Case) (Q"-Loop) (T~-Loop) (T2-Loop) Q".T~.T2 ~-~ (Q"+aQ"),T,,T2 ~ Q",(T,~T~).T2 ~. .~ Q".T,.(T2+6T2) 1 Q,, T~ - T~ il I h d Q"+bQ" T~ - "I"2 a,, h,,~ - (TI +6T,) T2 a,, h'T~- TI (T2+&T2) 2 2 2 6h__ { ( 0h &Q" + ~ 0T2 6TI ) ( ah ' TTT ) ,,=, when the interval between observations is longer than the autocorrelation time of the error source. For most engineering experiments, a third category of error is necessary: a category for errors that change during an experiment, but not randomly. These will be described here as "variable but deterministic." For example, the radiation error of