2008amstat_univariate distribution relationships

Univariate Distribution Relationships Lawrence M. LEEMIS and Jacquelyn T. MCQUESTON Probability distributions are traditionally treated separately in introductory mathematical statistics textbooks. A figure is pre- sented here that shows properties that individual distributions possess and many of the relationships between these distribu- tions. KEY WORDS: Asymptotic relationships; Distribution proper- ties; Limiting distributions; Stochastic parameters; Transforma- tions. 1. INTRODUCTION Introductory probability and statistics textbooks typically in- troduce common univariate distributions individually, and sel- dom report all of the relationships between these distributions. This article contains an update of a figure presented by Leemis (1986) that shows the properties of and relationships between several common univariate distributions. More detail concern- ing these distributions is given by Johnson, Kotz, and Balakrish- nan (1994, 1995) and Johnson, Kemp, and Kotz (2005). More concise treatments are given by Balakrishnan and Nevzorov (2003), Evans, Hastings, and Peacock (2000), Ord (1972), Pa- tel, Kapadia, and Owen (1976), Patil, Boswell, Joshi, and Rat- naparkhi (1985), Patil, Boswell, and Ratnaparkhi (1985), and Shapiro and Gross (1981). Figures similar to the one presented here have appeared in Casella and Berger (2002), Marshall and Olkin (1985), Nakagawa and Yoda (1977), Song (2005), and Taha (1982). Figure 1 contains 76 univariate probability distributions. There are 19 discrete and 57 continuous models. Discrete distri- butions are displayed in rectangular boxes; continuous distribu- tions are displayed in rounded boxes. The discrete distributions are at the top of the figure, with the exception of the Benford Lawrence M. Leemis is a Professor, Department of Mathematics, The College of William & Mary, Williamsburg, VA 23187–8795 (E-mail: leemis@math.wm.edu). Jacquelyn T. McQueston is an Operations Researcher, Northrop Grumman Corporation, Chantilly, VA 20151. The authors are grate- ful for the support from The College of William & Mary through a summer research grant, a faculty research assignment, and a NSF CSEMS grant DUE– 0123022. They also express their gratitude to the students in CS 688 at William & Mary, the editor, a referee, Bruce Schmeiser, John Drew, and Diane Evans for their careful proofreading of this article. Please e-mail the first author with updates and corrections to the chart given in this article, which will be posted at www.math.wm.edu/∼leemis. distribution. A distribution is described by two lines of text in each box. The first line gives the name of the distribution and its parameters. The second line contains the properties (described in the next section) that the distribution assumes. The parameterizations for the distributions are given in the Appendix. If the distribution is known by several names (e.g., the normal distribution is often called the Gaussian distribu- tion), this is indicated in the Appendix following the name of the distribution. The parameters typically satisfy the following conditions: • n, with or without subscripts, is a positive integer; • p is a parameter satisfying 0 < p < 1; • α and σ , with or without subscripts, are positive scale pa- rameters; • β, γ , and κ are positive shape parameters; • μ, a, and b are location parameters; • λ and δ are positive parameters. Exceptions to these rules, such as the rectangular parameter n, are given in the Appendix after any aliases for the distribution. Additionally, any parameters not described above are explic- itly listed in the Appendix. Many of the distributions have sev- eral mathematical forms, only one of which is presented here (e.g., the extreme value and discrete Weibull distributions) for the sake of brevity. There are numerous distributions that have not been in- cluded in the chart due to space limitations or that the dis- tribution is not related to one of the distributions currently on the chart. These include Be´zier curves (Flanigan–Wagner and Wilson 1993); the Burr distribution (Crowder et al. 1991, p. 33 and Johnson, Kotz, and Balakrishnan 1994, pp. 15– 63); the generalized beta distribution (McDonald 1984); the generalized exponential distribution (Gupta and Kundu 2007); the generalized F distribution (Prentice 1975); Johnson curves (Johnson, Kotz, and Balakrishnan 1994, pp. 15–63); the kappa distribution (Hosking 1994); the Kolmogorov–Smirnov one- sample distribution (parameters estimated from data), the Kolmogorov–Smirnov two-sample distribution (Boomsma and Molenaar 1994); the generalized lambda distribution (Ramberg and Schmeiser 1974); the Maxwell distribution (Balakrishnan and Nevzorov 2003, p. 232); Pearson systems (Johnson, Kotz, and Balakrishnan 1994, pp. 15–63); the generalized Waring dis- tribution (Hogg, McKean, and Craig 2005, p. 195). Likewise, c©2008 American Statistical Association DOI: 10.1198/000313008X270448 The American Statistician, February 2008, Vol. 62, No. 1 45 Devroye (2006) refers to Dickman’s, Kolmogorov–Smirnov, Kummer’s, Linnik–Laha, theta, and de la Valle´e–Poussin dis- tributions in his chapter on variate generation. 2. DISTRIBUTION PROPERTIES There are several properties that apply to individual distribu- tions listed in Figure 1. • The linear combination property (L) indicates that lin- ear combinations of independent random variables having this particular distribution come from the same distribution family. Example: If Xi ∼ N ( μi , σ 2 i ) for i = 1, 2, . . . , n; a1, a2, . . . , an are real constants, and X1, X2, . . . , Xn are independent, then n∑ i=1 ai Xi ∼ N ( n∑ i=1 aiμi , n∑ i=1 a2i σ 2 i ) . • The convolution property (C) indicates that sums of inde- pendent random variables having this particular distribu- tion come from the same distribution family. Example: If Xi ∼ χ2(ni ) for i = 1, 2, . . . , n, and X1, X2, . . . , Xn are independent, then n∑ i=1 Xi ∼ χ2 ( n∑ i=1 ni ) . • The scaling property (S) implies that any positive real constant times a random variable having this distribution comes from the same distribution family. Example: If X ∼ Weibull(α, β) and k is a positive, real constant, then k X ∼ Weibull(αkβ, β). • The product property (P) indicates that products of inde- pendent random variables having this particular distribu- tion come from the same distribution family. Example: If Xi ∼ lognormal(μi , σ 2i ) for i = 1, 2, . . . , n, and X1, X2, . . . , Xn are independent, then n∏ i=1 Xi ∼ lognormal ( n∑ i=1 μi , n∑ i=1 σ 2i ) . • The inverse property (I) indicates that the reciprocal of a random variable of this type comes from the same distri- bution family. Example: If X ∼ F(n1, n2), then 1 X ∼ F(n2, n1). • The minimum property (M) indicates that the smallest of independent and identically distributed random variables from a distribution comes from the same distribution fam- ily. Example: If Xi ∼ exponential(αi ) for i = 1, 2, . . . , n, and X1, X2, . . . , Xn are independent, then min{X1, X2, . . . , Xn} ∼ exponential ( 1 / n∑ i=1 (1/αi ) ) . • The maximum property (X) indicates that the largest of independent and identically distributed random variables from a distribution comes from the same distribution fam- ily. Example: If Xi ∼ standard power (βi ) for i = 1, 2, . . . , n, and X1, X2, . . . , Xn are independent, then max{X1, X2, . . . , Xn} ∼ standard power ( n∑ i=1 βi ) . • The forgetfulness property (F), more commonly known as the memoryless property, indicates that the conditional dis- tribution of a random variable is identical to the uncondi- tional distribution. The geometric and exponential distri- butions are the only two distributions with this property. This property is a special case of the residual property. • The residual property (R) indicates that the conditional distribution of a random variable left-truncated at a value in its support belongs to the same distribution family as the unconditional distribution. Example: If X ∼ Uniform(a, b), and k is a real constant satisfying a < k < b, then the conditional distribution of X given X > k belongs to the uniform family. • The variate generation property (V) indicates that the in- verse cumulative distribution function of a continuous ran- dom variable can be expressed in closed form. For a dis- crete random variable, this property indicates that a variate can be generated in an O(1) algorithm that does not cycle through the support values or rely on a special property. Example: If X ∼ exponential(α), then F−1(u) = −α log(1− u), 0 < u < 1. Since property L implies properties C and S, the C and S properties are not listed on a distribution having the L property. Similarly, property F ⇒ property R. Some of the properties apply only in restricted cases. The minimum property applies to the Weibull distribution, for ex- ample, only when the shape parameter is fixed. The Weibull distribution has Mβ on the second line in Figure 1 to indicate that the property is valid only in this restricted case. 46 Teacher’s Corner Figure 1. Univariate distribution relationships. The American Statistician, February 2008, Vol. 62, No. 1 47 3. RELATIONSHIPS AMONG THE DISTRIBUTIONS There are three types of lines used to connect the distribu- tions to one another. The solid line is used for special cases and transformations. Transformations typically have an X on their label to distinguish them from special cases. The term “transfor- mation” is used rather loosely here, to include the distribution of an order statistic, truncating a random variable, or taking a mix- ture of random variables. The dashed line is used for asymp- totic relationships, which are typically in the limit as one or more parameters approach the boundary of the parameter space. The dotted line is used for Bayesian relationships (e.g., Beta– binomial, Beta–Pascal, Gamma–normal, and Gamma–Poisson). The binomial, chi-square, exponential, gamma, normal, and U (0, 1) distributions emerge as hubs, highlighting their central- ity in applied statistics. Summation limits run from i = 1 to n. The notation X(r) denotes the r th order statistic drawn from a random sample of size n. There are certain special cases where distributions overlap for just a single setting of their parameters. Examples include (a) the exponential distribution with a mean of two and the chi- square distribution with two degrees of freedom, (b) the chi- square distribution with an even number of degrees of freedom and the Erlang distribution with scale parameter two, and (c) the Kolmogorov–Smirnov distribution (all parameters known case) for a sample of size n = 1 and the U (1/2, 1) distribution. Each of these cases is indicated by a double-headed arrow. The probability integral transformation allows a line to be drawn, in theory, between the standard uniform and all others since F(X) ∼ U (0, 1). Similarly, a line could be drawn between the unit exponential distribution and all others since H(X) ∼ exponential(1), where H(x) = ∫ x−∞ f (t)/(1−F(t))dt is the cumulative hazard function. All random variables that can be expressed as sums (e.g., the Erlang as the sum of independent and identically distributed ex- ponential random variables) converge asymptotically in a pa- rameter to the normal distribution by the central limit theo- rem. These distributions include the binomial, chi-square, Er- lang, gamma, hypoexponential, and Pascal distributions. Fur- thermore, all distributions have an asymptotic relationship with the normal distribution (by the central limit theorem if sums of random variables are considered). Many of the transformations can be inverted, and this is indi- cated on the chart by a double-headed arrow between two dis- tributions. Consider the relationship between the normal distri- bution and the standard normal distribution. If X ∼ N (μ, σ 2), then X−μσ ∼ N (0, 1) as indicated on the chart. Conversely, if X ∼ N (0, 1), then μ + σ X ∼ N (μ, σ 2). The first direction of the transformation is useful for standardizing random vari- ables to be used for table lookup, while the second direction is useful for variate generation. In most cases, though, an in- verse transformation is implicit and is not listed on the chart for brevity (e.g., extreme value random variable as the logarithm of a Weibull random variable and Weibull random variable as the exponential of an extreme value random variable). Several of these relationships hint at further distributions that have not yet been developed. First, the extreme value and log gamma distributions indicate that the logarithm of any survival distribution results in a distribution with support over the en- tire real axis. Second, the inverted gamma distribution indicates that the reciprocal of any survival distribution results in another survival distribution. Third, switching the roles of F(x) and F−1(u) for a random variable with support on (0, 1) results in a complementary distribution (e.g., Jones 2002). Additionally, the transformations in Figure 1 can be used to give intuition to some random variate generation routines. The Box–Muller algorithm, for example, converts a U (0, 1) to an exponential to a chi-square to a standard normal to a normal random variable. Redundant arrows have typically not been drawn. An arrow between the minimax distribution and the standard uniform dis- tribution has not been drawn because of the two arrows connect- ing the minimax distribution to the standard power distribution and the standard power distribution to the standard uniform dis- tribution. Likewise, although the exponential distribution is a special case of the gamma distribution when the shape parame- ter equals 1, this is not explicitly indicated because of the special case involving the Erlang distribution. In order to preserve a planar graph, several relationships are not included, such as those that would not fit on the chart or involved distributions that were too far apart. Examples include: • A geometric random variable is the floor of an exponential random variable. • A rectangular random variable is the floor of a uniform random variable. • An exponential random variable is a special case of a Makeham random variable with δ = 0. • A standard power random variable is a special case of a beta random variable with δ = 1. • If X has the F distribution with parameters n1 and n2, then 1 1+(n1/n2)X has the beta distribution (Hogg, McKean, and Craig 2005, p. 189). • The doubly noncentral F distribution with n1, n2 degrees of freedom and noncentrality parameters δ, γ is defined as the distribution of( X1(δ) n1 )( X2(γ ) n2 )−1 , where X1(δ), X2(γ ) are noncentral chi-square random variables with n1, n2 degrees of freedom, respectively, (Johnson, Kotz, and Balakrishnan 1995, p. 480). • A normal and uniform random variable are special and lim- iting cases of an error random variable (Evans, Hastings, and Peacock 2000, p. 76). • A binomial random variable is a special case of a power se- ries random variable (Evans, Hastings, and Peacock 2000, p. 166). • The limit of a von Mises random variable is a normal ran- dom variable as κ → ∞ (Evans, Hastings, and Peacock 2000, p. 191). 48 Teacher’s Corner • The half-normal, Rayleigh, and Maxwell–Boltzmann dis- tributions are special cases of the chi distribution with n = 1, 2, and 3 degrees of freedom (Johnson, Balakrish- nan, and Kotz 1994, p. 417). • A function of the ratio of two independent generalized gamma random variables has the beta distribution (Stacy 1962). Additionally, there are transformations where two distribu- tions are combined to obtain a third, which were also omitted to maintain a planar graph. Two such examples are: • The t distribution with n degrees of freedom is defined as the distribution of Z√ χ2(n)/n , where Z is a standard normal random variable and χ2(n) is a chi-square random variable with n degrees of freedom, independent of Z (Evans, Hastings, and Peacock 2000, p. 180). • The noncentral beta distribution with noncentrality param- eter δ is defined as the distribution of X X + Y , where X is a noncentral chi-square random variable with parameters (β, δ) and Y is a central chi-square random variable with γ degrees of freedom (Evans, Hastings, and Peacock 2000, p. 42). References for distributions not typically covered in introduc- tory probability and statistics textbooks include: • arctan distribution: Glen and Leemis (1997) • Benford distribution: Benford (1938) • exponential power distribution: Smith and Bain (1975) • extreme value distribution: de Haan and Ferreira (2006) • generalized gamma distribution: Stacy (1962) • generalized Pareto distribution: Davis and Feldstein (1979) • Gompertz distribution: Jordan (1967) • hyperexponential and hypoexponential distributions: Ross (2007) • IDB distribution: Hjorth (1980) • inverse Gaussian distribution: Chhikara and Folks (1989), Seshadri (1993) • inverted gamma distribution: Casella and Berger (2002) • logarithm distribution: Johnson, Kemp, and Kotz (2005) • logistic–exponential distribution: Lan and Leemis (2007) • Makeham distribution: Jordan (1967) • Muth’s distribution: Muth (1977) • negative hypergeometric distribution: Balakrishnan and Nevzorov (2003), Miller and Fridell (2007) • power distribution: Balakrishnan and Nevzorov (2003) • TSP distribution: Kotz and van Dorp (2004) • Zipf distribution: Ross (2006). A. APPENDIX: PARAMETERIZATIONS A.1 Discrete Distributions Benford: f (x) = log10 ( 1+ 1 x ) , x = 1, 2, . . . , 9 Bernoulli: f (x) = px (1− p)1−x , x = 0, 1 Beta–binomial: f (x) = 0(x + a)0(n − x + b)0(a + b)0(n + 2) (n + 1)0(a + b + n)0(a)0(b)0(x + 1)0(n − x + 1) , x = 0, 1, . . . , n Beta–Pascal (factorial): f (x) = ( n − 1+ x x ) B(n + a, b + x) B(a, b) , x = 0, 1, . . . Binomial: f (x) = ( n x ) px (1− p)n−x , x = 0, 1, . . . , n Discrete uniform: f (x) = 1 b − a + 1 , x = a, a + 1, . . . , b Discrete Weibull: f (x) = (1− p)xβ − (1− p)(x+1)β , x = 0, 1, . . . Gamma–Poisson: f (x) = 0(x + β)α x 0(β)(1+ α)β+x x! , x = 0, 1, . . . Geometric: f (x) = p(1− p)x , x = 0, 1, . . . Hypergeometric: f (x) = ( n1 x )( n3 − n1 n2 − x )/( n3 n2 ) , x = max(0, n1 + n2 − n3), . . . ,min(n1, n2) The American Statistician, February 2008, Vol. 62, No. 1 49 Logarithm (logarithmic series, 0 < c < 1): f (x) = −(1− c) x x log c , x = 1, 2, . . . Negative hypergeometric: f (x) = ( n1 + x − 1 x )( n3 − n1 + n2 − x − 1 n2 − x ) /( n3 + n2 − 1 n2 ) , x = max(0, n1 + n2 − n3), . . . , n2 Pascal (negative binomial): f (x) = ( n − 1+ x x ) pn(1− p)x , x = 0, 1, . . . Poisson (μ > 0): f (x) = μ x e−μ x! , x = 0, 1, . . . Polya: f (x) = ( n x ) x−1∏ j=0 (p + jβ) n−x−1∏ k=0 (1− p + kβ) / n−1∏ i=0 (1+ iβ), x = 0, 1, . . . , n Power series (c > 0; A(c) =∑x ax cx ): f (x) = ax c x A(c) , x = 0, 1, . . . Rectangular (discrete uniform, n = 0, 1, . . .): f (x) = 1 n + 1 , x = 0, 1, . . . , n Zeta: f (x) = 1 xα ∑∞ i=1(1/ i)α , x = 1, 2, . . . Zipf (α ≥ 0): f (x) = 1 xα ∑n i=1(1/ i)α , x = 1, 2, . . . , n A.2 Continuous Distributions Arcsin: f (x) = 1 π √ x(1− x) , 0 < x < 1 Arctangent (−∞ < φ <∞): f (x) = λ[ arctan(λφ)+ π2 ][ 1+ λ2(x − φ)2 ] , x ≥ 0 Beta: f (x) = [ 0(β + γ ) 0(β)0(γ ) ] xβ−1(1− x)γ−1, 0 < x < 1 Cauchy (Lorentz, Breit–Wigner, −∞ < a <∞): f (x) = 1 απ [1+ ((x − a)/α)2] , −∞ < x <∞ Chi: f (x) = 1 2n/2−10(n/2) xn−1e−x2/2, x > 0 Chi-square: f (x) = 1 2n/20(n/2) xn/2−1e−x/2, x > 0 Doubly noncentral F : f (x) = ∞∑ j=0 ∞∑ k=0 e−δ/2 ( 1 2δ ) j j!  e−γ /2 ( 1 2γ )k k!  ×n(n1/2)+ j1 n(n2/2)+k2 x (n1/2)+ j−1 ×(n2 + n1x)− 12 (n1+n2)− j−k × [ B ( 1 2 n1 + j, 12n2 + k )]−1 , x > 0 Doubly noncentral t : See Johnson, Kotz, and Balakrishnan (1995, p. 533) Erlang: f (x) = 1 αn(n − 1)! x n−1e−x/α, x > 0 Error (exponential power, general error; −∞ < a < ∞, b > 0, c > 0): f (x) = e