![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to chap2.tex CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [Research Unix] / researchv10dc / vol2 / index|
Return to chap2.tex CVS log | Up to [Research Unix] / researchv10dc / vol2 / index |
1.1 root 1: \ifnum\pageno=1 \input macros.tex \pageno=200\immediate\openout\inx=chap2.index\makecontents\draft{\date (Rob)} \fi
2: %\proofmodefalse
3: \def\gsmooth{{s}}
4: \def\vgsmooth{{\vec s}}
5: \def\symNN{N^S}
6: \def\NN{N}
7: \def\pepscat{1.1}
8: \def\pepadd{1.2}
9: \def\X{\bf X}
10: \chapter 2 {Smoothing}
11: \medskip
12: \Section{What is a smoother?}
13: A smoother is a tool for summarizing the dependence of a response
14: measurement $Y$ on one or more predictor measurements $X_1,\ldots,
15: X_p$.
16: It produces an estimate of this dependence that is less variable than $Y$
17: itself; hence the name {\sl smoother}.
18: An important property of a smoother is its {\sl nonparametric} nature; that is,
19: it doesn't assume a rigid
20: form for the dependence.
21: For this reason, a smoother is
22: often referred to as a
23: tool for nonparametric regression.
24: The running mean (or moving average) is a simple example of a smoother,
25: while a regression line is not thought of as a smoother because of its rigid
26: parametric form.
27: \input spansizes.tex
28: \par\figname{\allsmooths}
29: We call the estimate produced by a smoother a {\sl smooth}.
30: The most important example of smoothing involves a single predictor
31: setting, usually referred to as {\sl scatterplot
32: smoothing}.
33:
34: Smoothers have two main uses.
35: The first use is for description.
36: For example, a scatterplot smoother can be used to enhance the
37: visual appearance of the scatterplot of $Y$ vs $X$,
38: to help our eyes pick out the trend in the plot.
39: Figure~\pepscat\ in Chapter~1 shows a plot
40: of \name{log(C-peptide)} versus \name{age} for the diabetes data.
41: It seems that \name{log(C-peptide)} has a strong dependence on \name{age} and a scatterplot smoother
42: will be helpful in describing this relationship.
43: Figure~\allsmooths\ shows a number of scatterplot smooths of these data; each described in turn in this chapter.
44: The second use of a smoother is as an estimator of the dependence of $\ev(Y)$ on
45: the predictors, and thus as a building block for the estimation of additive
46: models, discussed in the remainder of the book.
47:
48:
49: In this chapter we give a brief overview of some useful smoothers.
50: Most of our discussion concerns scatterplot smoothers; at the end of the
51: chapter we give a brief description of multiple-predictor smoothers.
52:
53: The simplest smoother occurs in the case of a {\sl categorical}
54: predictor,
55: for example, \name{sex} (male, female) or \name{colour} (red, blue, green etc.)
56: To smooth $Y$ we can simply average the values of $Y$ in each category.
57: This satisfies our requirements for a scatterplot smooth: it captures the
58: trend of $Y$ on $X$ and is smoother than the $Y$-values themselves.
59: While the reader might not normally think of this as {\sl smoothing}, this simple
60: averaging process is the conceptual basis for
61: smoothing in the most general setting, that of an ordered (noncategorical)
62: predictor.
63: The problem here is often the lack of replicates at each
64: predictor value.
65: Most smoothers attempt to mimic category averaging through
66: {\sl local averaging}, that is,
67: averaging the
68: $Y$-values of observations having predictor values close to a target
69: value.
70: The averaging is done in {\sl neighbourhoods} around the target value.
71: There are two main decisions to be made in scatterplot
72: smoothing:\smallskip
73: {\parindent 20pt
74: \item{(i)}how to average the
75: response values in each neighbourhood, and
76: \item{(ii)} how big to take the neighbourhoods.
77:
78: }\smallskip
79: \input allsmooths.tex
80: \par
81: The question of how to average within a neighbourhood is really the
82: question of which {\sl brand} of smoother to use, because smoothers
83: differ mainly in their method of averaging.
84: In this chapter we describe a number of different smoothers and
85: compare them informally.
86: Formal recommendations on how to choose among smoothers are
87: difficult to make because
88: few systematic comparisons have been made so far in the
89: literature.
90:
91: The question of how big to make the neighbourhoods is discussed in
92: the next chapter.
93: The issue underlying this question is very important, however, so we briefly
94: discuss it here.
95: The size of the neighbourhood is typically expressed in terms of an
96: adjustable {\sl smoothing parameter}.
97: Intuitively, large neighbourhoods will produce an estimate with low
98: variance but potentially high bias, and conversely for small
99: neighbourhoods (Fig.~\spansizes).
100: Thus there is a {\sl fundamental tradeoff between bias and variance},
101: governed by the smoothing parameter.
102: This issue is exactly analogous to the question of how many predictors to put
103: in a regression equation.
104: In the
105: next
106: chapter we discuss this important issue and the practical
107: question of how to choose the smoothing parameter, based on the data,
108: to trade bias against variance in an optimal way.
109: We also discuss some other important topics, such as
110: linear and nonlinear smoothers, the
111: incorporation of observation weights into --- and inference
112: for --- the
113: fitted smooth.
114:
115: All the smoothers we shall describe can easily be {\em robustified} by replacing the averaging or least-squares operation by a more robust procedure.
116: Details can be found in Chapters~3 and 9.
117:
118:
119: Bibliographic notes on smoothers are given at the end
120: of the next chapter.
121:
122: \Sectionskip
123: \Section{Scatterplot smoothing: definition}
124: \Mark{SCATTERPLOT SMOOTHING}
125: Suppose we have response measurements $\vec y=(y_1,\ldots,y_n)^T$ at design
126: points $\vec x=(x_1,\ldots,
127: x_n)^T$. We assume that each of $\vec y$ and $\vec x$ represent
128: measurements of variables $Y$ and $X$.
129: In most cases it is useful to think of
130: $Y$, and sometimes $X$, as having been generated by some random mechanism, but this is not
131: necessary for the discussion here.
132: In particular we don't need to assume
133: that the pairs $(x_i,y_i)$ are a random sample from some joint distribution.
134: For
135: example, the $X$-values might be preset dose levels of a drug.
136:
137:
138:
139: Since $Y$ and $X$ are noncategorical we don't expect to find
140: many replicates at any given value of $X$.
141: For convenience we assume
142: that the data are sorted by $X$
143: and for the present discussion that there are no tied $X$-values, so that
144: $x_1 < \cdots < x_n$.
145: A trivial remedy when there are ties is to use weighted smoothers, which we
146: discuss in Chapter~3.
147:
148:
149: We denote a scatterplot smooth of $\vec y$ against $\vec x$ by $\gsmooth=\smooth(\vec y\given \vec x)$, where $\gsmooth=\gsmooth(x)$ is a function of $X$.
150: Usually the recipe that defines $\gsmooth(x_0)$, which is the function $\smooth(\vec y\given \vec x)$ evaluated at $x_0$, will be defined for all $x_0$, but at other times is defined only at $x_1,\ldots,x_n$, the sample values of $X$.
151: In this latter case some kind of interpolation is necessary in order to
152: obtain estimates at other $X$-values.
153:
154: \Sectionskip
155: \Section{Parametric Regression}
156: A regression line, estimated for example by least-squares,
157: provides an estimate of the dependence of $\ev(Y)$ on $X$.
158: It does so, however,
159: by assuming a rigid form for this dependence,
160: and thus it
161: may or may not be appropriate for a given set of data.
162: If the dependence is linear or close to it, the regression line provides a
163: concise and useful summary.
164: For the data in Fig.~\allsmooths, however, the regression line is clearly inappropriate and creates a misleading impression.
165:
166: In a sense the regression line is an {\sl infinitely smooth} function, and
167: not surprisingly many of the scatterplot smoothers that we
168: discuss approach the linear regression line as one extreme.
169: The other extreme
170: is usually some function that interpolates the data.
171:
172:
173: Other nonlocal parametric fits, such as polynomial regression estimates, share
174: the same pros and cons as the regression line.
175: They are useful
176: if they are appropriate for the data at hand but potentially misleading otherwise.
177: Although parametric fitting certainly isn't the solution to the scatterplot
178: smoothing problem,
179: we will still make use of it for
180: comparative purposes and for possible summaries of
181: smooths estimated by other methods.
182: Regression splines are a less rigid
183: form of parametric fitting which are closer in spirit to a smoother; we discuss them later in this chapter.
184: \Sectionskip
185: \Section{Bin smoothers}
186: A bin smoother mimics a categorical smoother by
187: partitioning the predictor values into a number of disjoint and exhaustive regions,
188: then averaging the response in each region.
189: Formally, we choose cutpoints
190: $c_0 < \cdots < c_K$
191: where $c_0=-\infinity$ and $c_K=\infinity$,
192: and define
193: $$R_i=\{j;c_i\leq x_j < c_{i+1}\};\qquad i=1,\ldots, K$$
194: the indices of
195: the data points in each region.
196: Then $\gsmooth=\smooth(\vec y\given \vec x)$ is given by $\gsmooth(x_0)=\ave_{j\in R_i} (y_j)$ if
197: $x_0 \in R_i$.
198: Typically one chooses five regions, for example, and picks the cutoff points
199: so that
200: there is approximately an equal number of points in each region.
201: The bin smooth shown in Fig.~\allsmooths\ was constructed in this way and illustrates
202: the limitation of the method.
203: The estimate is not very smooth because it jumps at each cut point.
204: Unless we have a prior reason to believe such discontinuities exist, this
205: estimate is not very appealing.
206: One way to make it more smooth is through the use of
207: overlapping regions, as we'll see in the next section.
208: \Sectionskip
209: \Section{Running-mean and running-lines smoothers}
210: Assume that our target value $x_0$ equals one of the $x_j$s, say $x_i$. If we had replicates at $x_i$,
211: we could simply use the average of the $Y$-values at $x_i$ as our estimate
212: $\gsmooth(x_i)$.
213: Now
214: we are assuming that we don't have replicates, so instead we can average $Y$-values corresponding to $X$-values close to $x_i$.
215: How do we pick points that
216: are {\sl close} to $x_i$?
217: A simple way is to choose $x_i$ itself, as well as the $k$ points to the left and $k$ points to the
218: right of $x_i$ that are closest in $X$-value to $x_i$.
219: This is called a {\sl symmetric nearest neighbourhood} and
220: the indices of these points
221: will be denoted by
222: $\symNN(x_i)$. Then we could define
223: the {\sl running mean}
224: $$\gsmooth(x_i)=\ave_{j\in
225: \symNN(x_i)} (y_j).\eqn{\runm}$$
226: If it is not possible to take $k$ points to the left or right of
227: $x_i$, we take as many as we can.
228: A formal definition of a symmetric nearest neighbourhood is
229: $$\symNN(x_i)= \{ \max(i-k ,1),\ldots,i-1,i,i+1,\ldots,\min(i+ k
230: ,n)\}.\eqn{\neigh}$$
231:
232: It is not obvious how to define the symmetric nearest neighbours at target points $x_0$ other than the $x_i$ in the sample.
233: We could simply interpolate linearly
234: between the fit of the two values of $X$ in the sample adjacent to $x_0$.
235: Alternatively we could ignore symmetry and take the $r$ closest points to $x_0$, regardless
236: of which side they are on; this is called a {\sl nearest neighbourhood}.
237: This handles arbitrary $x_0$ in a simple and clean way.
238: The pros and cons of these two types of neighbourhoods are discussed later.
239: \input rlsmooth.tex
240: \par
241: This simple smoother
242: is
243: also called a {\sl moving average}, and is popular for evenly-spaced time-series data.
244: Although it is valuable for theoretical calculation because of its simplicity,
245: in practice it does not work very well. It tends to be so wiggly that it hardly deserves the name {\sl smoother.}
246: Apart from looking unpleasant, it tends to flatten
247: out trends near the endpoints and hence can be severely biased.
248: For example, Fig.~\allsmooths\ shows a running mean smooth with $2k+1=11$
249: or 25\% of the 43 observations.
250:
251:
252: A simple generalization of the running mean alleviates the bias problem: we compute
253: a least-squares line instead of a mean in each neighbourhood.
254: The {\sl running
255: lines smoother} is defined by
256: $$\gsmooth(x_0)=\hat\alpha(x_0) +\hat
257: \beta(x_0) x_0\eqn{\runl}$$
258: where $\hat\alpha(x_0)$ and $ \hat \beta(x_0) $ are the least-squares
259: estimates for the data points in $\symNN(x_0)$.
260: Figure~\rlsmooth\ shows the local line computed in two different
261: neighbourhoods, one in the interior, the other near the boundary. As we
262: would expect, the fit in the interior is dominated largely by the mean and the slope
263: plays a small role, whereas near the boundary the slope is important for
264: picking up the trend in the asymmetric neighbourhood.
265: A running-lines smooth with a neighbourhood size of $2k+1=13$ or 30\% for the diabetes data, is shown in Fig.~\allsmooths.
266: It seems to capture the trend in the data quite nicely
267: but is still somewhat jagged.
268:
269:
270: The parameter $k$ controls the appearance of the running-lines smooth.
271: Large values of $k$ tend to produce smoother curves while small values
272: tend to produce more jagged curves.
273: It is more convenient to think not in terms of $k$ but instead in terms of
274: $w=(2k+1)/n$, the proportion of points in each neighbourhood, called the {\it
275: span}.
276: We denote by $[\symNN(x_i)]$ the number of points in $\symNN(x_i)$.
277: In the extreme case, if $w= 2$, so that each neighbourhood contains all of
278: the data (note that $w=1$ won't work because of endpoint effects), the running-lines smooth is the least-squares line.
279: On the other hand, if $w=1/n$, each neighbourhood consists only of one data point
280: and hence the smoother interpolates the data.
281: In the next chapter we discuss the quantitative effects of varying the span and a
282: data-based criterion for choosing it.
283:
284:
285: The running-lines smoother can be computed in O(n) operations (Exercise~2.2),
286: and there is a simple, effective method for span selection.
287: These can be important advantages when it is used as a building block in the
288: iterative algorithms discussed later in this book.
289: On the other hand, it tends to produce curves that are quite jagged so that
290: a second stage of smoothing might be necessary.
291:
292:
293:
294: One way to improve the appearance of the running-lines smooth is through
295: the use of a {\sl weighted} least-squares fit in each
296: neighbourhood.
297: The running-lines smoother can produce jagged output because points in
298: a given neighbourhood are given equal (nonzero) weight while points
299: outside of the neighbourhood are given zero weight.
300: Thus as the neighbourhoods move from left to right, there are discrete
301: changes in the weight given to the leftmost and rightmost points.
302: We can alleviate this problem by
303: giving the highest weight to $x_i$, and weights smoothly decreasing as
304: we move further away from $x_i$.
305: ^{Cleveland's (1979)} implementation of a locally-weighted running-lines smoother, {\sl loess},
306: is popular and (essentially) works in this way.
307: A full description of locally-weighted running-line smoothers is given later in this Chapter; Fig.~\allsmooths\ shows one applied to the diabetes data.
308: It is smoother than the unweighted running-lines smoother.
309: The cost of using smooth weights is a computational one, as the
310: shortcuts available for the unweighted running-lines
311: smoother (Exercise~2.2) do not work for locally
312: weighted running-lines smoothers.
313: Henceforth we will refer to the unweighted running-lines smoother
314: simply as the running-lines smoother.
315: \Sectionskip
316: \Section{Kernel smoothers}
317: A kernel smoother uses an explicitly defined set of local weights, defined by the {\sl kernel}, to
318: produce the estimate at each target value.
319: Usually a kernel smoother uses weights that decrease in a smooth fashion
320: as one moves away from the target point.
321: \input kernelsm.tex
322: \par
323: The weight given to the $j$th point in producing the estimate at $x_0$
324: is defined by
325: $$S_{0j}={c_0\over \lambda}
326: d\left(\big\vert {{x_0-x_j}\over{\lambda}} \big\vert\right)\eqn{\kernel}$$
327: where $d(t)$ is an even function defined for $\abs{t}>0$.
328: The parameter $\lambda$ is the window width,
329: and the constant
330: $c_0$ is usually chosen so that the weights sum to unity, although
331: there are slight variations on this.
332: A natural candidate for $d$ is the standard Gaussian density:
333: this gives the so-called { Gaussian kernel} smoother.
334: Other popular kernels, with some theoretical justification (^{Eubank, 1988}), are the Epanechnikov kernel
335: $$d(t)=\cases{ {3\over 4} (1-t^2),& for $\abs{t}\leq 1$;\cr 0& otherwise\cr}$$
336: which minimizes (asymptotic) mean squared error, and the minimum
337: variance kernel
338: $$d(t)=\cases{ {3\over 8} (3-5t^2),& for $\abs{t}\leq 1$;\cr 0& otherwise\cr}$$
339: which minimizes the asymptotic variance of the estimate.
340: Note that the weight given to an observation is a
341: function only of its {\sl metric} distance from $x_0$, while the weights used by the nearest-neighbour smoothers are typically a function of both {\sl metric} and {\sl rank} distance.
342:
343: Kernel smoothers also exhibit biased endpoint behaviour.
344: Special kernels have been developed to overcome this bias; a simple approach is to use kernel weights in a locally-weighted straight-line fit.
345:
346: A Gaussian-kernel smooth for the diabetes data is shown in
347: Fig.~\allsmooths, where once again
348: the parameter $\lambda$ was chosen so that the approximate degrees of freedom of the
349: fitted smooth was five, the same as that of the other smooths in
350: Fig.~\allsmooths.
351:
352: \sectionskip
353: \section{Computational issues}
354: One can visualize the action of the kernel smooth as sliding the weight function along the $x$-axis in short steps, each time computing the weighted mean of $y$.
355: The smooth is thus similar to a convolution between the kernel and an empirical step function defined on the data.
356: This is indeed the case, although the practical details obscure the resemblance.
357: For example, the kernel is usually truncated at the ends of the data, unless the data itself is periodic.
358: Typically the kernel smooth is computed as
359: $$\gsmooth(x_0)={\sum_{i=1}^n
360: d({x_0-x_i \over \lambda})y_i\over \sum_{i=1}^n d({x_0-x_i \over \lambda})},
361: \eqn{\NadWat}$$
362: and so both the numerator and denominator are convolutions.
363: If the $x$-values are evenly spaced (and preferably a power of two in number), great savings can be made by using the FFT (fast Fourier transform) in performing the calculations.
364: Although evenly spaced data are rarely the case in general regression contexts, reasonable approximations can often reduce the data to a fine grid, where the FFT can be put to work.
365: The details are laid out in Exercise~2.9.
366:
367: \Sectionskip\Section{Running medians and enhancements}
368: A somewhat different approach to smoothing is based on
369: improvements to the simple
370: running mean smoother.
371: First,
372: the running mean is replaced by a running median to make the smoother
373: resistant to outliers in the data.
374: Then the appearance of the estimate is enhanced by applying compound operations
375: known as {\em Hanning}, {em splitting} and {\em twicing}, in various combinations.
376: We will not discuss these smoothers here because they are mainly
377: useful for evenly spaced data (sequences), especially time series, and they don't
378: typically provide an adequate amount of smoothing
379: for our purposes.
380: The interested reader can pursue the references given in the Bibliographic notes.
381: We focus on smoothers for unevenly spaced data (scatterplots), sometimes
382: called {\em regression smoothers}.
383: In our view,
384: another disadvantage of the enhanced running median smoothers is that they are
385: highly nonlinear functions of the response, and thus it is difficult to assess
386: the amount of fitting that they do.
387: This is an important consideration for the inferential stage of a data analysis.
388: Many of the smoothers that we discuss are {\em linear} (section~3.4.2), and this facilitates
389: an approximate assessment of their {\em degrees of freedom} (section~3.5).
390:
391: We return to the time series setting for our discussion of the Fourier analysis
392: of smoothers in section~3.7, and a seasonal decomposition procedure in
393: section~8.5.
394: \input hatplot.tex
395: \Sectionskip
396: \Section{Equivalent kernels}
397: The smoothers described in this chapter are defined in quite different ways, with
398: more different ways to come.
399: Their {\sl equivalent kernels} are one way to compare
400: them on common ground.
401: All the smoothers studied in this chapter are {\sl linear} in $Y$, which means that the fit at a point $x_0$ can be written as $\gsmooth(x_0)=\sum_{j=1}^n S_{0j}y_j$, and the $S_{0j}$ depend on all the $x_i$ and on the smoothing parameter $\lambda$.
402: Thus the weight in the fit at $x_0$ which is associated with the point $x_j$ is $S_{0j}$, and this sequence of weights is known as the {\em equivalent kernel} at $x_0$.
403: A plot of $S_{0j}$ against $x_j$ shows which observations have an influence on the fit at $x_0$.
404: Looking at Fig.~\hatplot, we see that the running mean smoother has weights
405: either zero or $1/m$, where $m$ is the number of observations in the neighbourhood.
406: We have used the same diabetes data as was used in Fig.~\allsmooths.
407:
408: For each smoother we have computed the equivalent kernel centered at two points: one near the boundary and one at the centre.
409: The weights at the boundary are larger and warn us that end effects may be a problem (as indeed they are).
410:
411: For the running-line smoother the weights change linearly within the window (Exercise~2.3).
412: For both the running mean and line smoothers the weights drop off
413: abruptly to zero outside the neighbourhood, and account for their jagged appearance.
414: The {\em loess} smooth on the other hand has a strictly local neighbourhood yet the weights die down smoothly to zero.
415:
416: We need to calibrate the smoothers so that they are doing approximately the same amount of smoothing, since the value of the smoothing parameter will clearly widen or narrow the equivalent kernels.
417: We do this using the {\sl equivalent degrees of freedom}, which we describe in the next chapter.
418: All the smoothers in Fig.~\hatplot\ have been calibrated to do about five degrees of freedom worth of smoothing.
419:
420: In the top panel we see the equivalent kernel for the quartic polynomial fit.
421: As we might expect it spreads its influence everywhere.
422:
423: We will return to Fig.~\hatplot\ as we encounter the remaining smoothers.
424:
425: \Sectionskip
426: \Section {Regression splines}
427: Polynomial regression has limited appeal due to the global nature of the
428: fit, while in contrast the smoothers we have seen so far have an explicit local nature.
429: Regression splines offer a compromise by representing the fit as a {\sl piecewise}
430: polynomial.
431: The regions that define the pieces are separated by a sequence
432: of {\sl knots} or breakpoints, $\xi_1,\ldots,\xi_K$.
433: In addition, it is customary to enforce the piecewise polynomials to join smoothly at these knots.
434: Although many different configurations are possible (the bin smoother was one), a popular choice
435: consists of piecewise cubic polynomials constrained to be continuous and have continuous first and second derivatives at the knots.
436: Apparently our eyes are skilled at picking up second order and lower discontinuities, but not higher.
437: \input BS.tex
438: \par
439: To illustrate this point, Fig.~\BS\ shows four piecewise cubic fits to some simulated data, with 2
440: interior knots and different orders of continuity.
441: Apart from being smoother, the fit in panel (d) required fewer parameters (6 altogether) than the others.
442:
443: By allowing more knots, the family of curves becomes more flexible.
444: For any given set of knots, the smooth is computed by multiple regression on an appropriate set of basis vectors.
445: These vectors are the basis functions representing the particular family of piecewise cubic polynomials evaluated at the observed values of the predictor $X$.
446: This approach is attractive because
447: of its computational and statistical simplicity; for example standard
448: parametric inferential methods can be used to test the importance
449: of any of the parameters.
450:
451: A variant of polynomial splines are the natural splines; although they are defined for all piecewise polynomials of odd degree, we discuss the natural {\sl cubic} splines.
452: These are cubic splines with the additional constraint that the function is linear beyond the boundary knots.
453: To enforce this condition we have to impose the two constraints in each of the boundary regions: $f'''=f''=0$, which reduces the dimension of the space from $K+4$ to $K$ if there are $K$ knots.
454: In practice it is common to supply an additional knot at each extreme of the data, and impose the linearity beyond them.
455: Then with $K$ interior knots (and two boundary knots), the dimension of the space of fits is $K+2$.
456: Natural splines have less flexibility at the boundaries, but this tends to be a plus since the fitted values of regular regression splines have high variance near the boundary.
457: Furthermore, for the same number of parameters, we get two more interior knots.
458:
459: \sectionskip
460: \section{Computational aspects}
461: The main difficulty when working with regression splines is
462: selecting the number and position of the knots.
463: A very simple approach (referred to as cardinal splines) requires a single parameter, the {\sl number} of interior knots.
464: The positions are then chosen uniformly over the range of the data.
465: A slightly more adaptive version will place the knots at appropriate quantiles of the predictor variable; e.g., three interior knots would be placed at the three quartiles.
466: More adventurous schemes use data driven criteria to select the number and positions of the knots.
467: The main challenge is to come up with a sensible procedure, while avoiding a combinatorial nightmare.
468: We describe some specific proposals in Chapter~9.
469:
470:
471: Another computational aspect concerns the choice of basis functions for representing the splines (a vector space) for a given set of knots.
472: Suppose these interior knots are denoted by $\xi_1<\cdots<\xi_K$, and for notational simplicity we augment the set with two boundary knots $\xi_0$ and $\xi_{K+1}$.
473: A simple choice of basis functions for piecewise-cubic splines, known as the truncated power series basis, derives from the parametric expression for the smooth
474: $$\gsmooth(x)= \beta_0+\beta_1x+\beta_2x^2+\beta_3x^3+\sum_{j=1}^K\theta_j(x-\xi_j)_+^3.\eqn{\tpseries}$$ where the $a_+$ denotes the positive part of $a$.
475: Evidently (Exercise~2.4) \tpseries\ has the required properties:\smallskip
476: {\parindent 20pt
477: \item{(i)} $\gsmooth$ is cubic polynomial in any subinterval $[\xi_j,\xi_{j+1})$,
478: \item{(ii)} $\gsmooth$ has two continuous derivatives, and
479: \item{(iii)} $\gsmooth$ has a third derivative that is a step function with jumps at $\xi_1,\ldots,\xi_K$.
480:
481: }\smallskip
482:
483: We can write \tpseries\ as a linear combination of $K+4$ basis functions $P_j(x)$: $P_1(x)=1$, $P_2(x)=x$, and so on.
484: Each of these functions must also satisfy the three conditions and be linearly independent in order to qualify as a basis.
485: Even without \tpseries\ it is clear that we require $K+4$ parameters to represent piecewise cubics; four for each of $(K+1)$ cubics, less three per interior knot due to the constraints.
486: To actually smooth some data pairs $\{x_i,y_i\}$, we would construct a regression matrix with $K+4$ columns, each column corresponding to a function $P_j$ evaluated at the $n$ values of $x$.
487:
488:
489: Although \tpseries\ has algebraic appeal, it is not the recommended form for computing the regression spline.
490: Even though a particular cubic piece is designed to accommodate one interval, it is evaluated at all points to the right of its knot and the numbers usually get large.
491: \input BSbasis.tex
492: \par
493: The $B$-spline basis functions provide a numerically superior alternative basis to the truncated power series.
494: Their main feature is that any given basis function $B_j(x)$ is nonzero over a span of at most five distinct knots.
495: In practice this means that their evaluation rarely gets out of hand, and the resulting regression matrix is banded.
496: Of course the $B_j$ are themselves piecewise cubics, and we need $K+4$ of them if we want to span the space.
497: Their definition is rather simple (in terms of divided differences), but unhelpful for fully understanding their properties, so we omit it here and refer the reader to ^{de~Boor~(1978)}.
498: Figure~\BSbasis\ shows the evaluated $B$-splines used to compute the fits in Fig.~\BS(d).
499:
500: In summary, regression splines are attractive because of their computational neatness, {\sl when the knots are given}.
501: Standard linear model estimation can be applied, which will be found to be very convenient in later chapters when we use regression splines in additive models.
502:
503: However the difficulty of choosing the number and position of the knots
504: tends to be a drawback of this approach.
505: When a small number of knots is used, the smoother can show some disturbing nonlocal behaviour.
506: Figure~\hatplot\ shows the equivalent kernels for a natural cubic regression spline with three interior knots.
507: The right-hand kernel does not look much different from that for the quartic polynomial! With more knots this global influence would be dampened, but we often don't have that many \df\ to spare.
508:
509: Another problem with regression splines is that the smoothness of the
510: estimate can not easily be varied continuously as a function of a single smoothing
511: parameter; this is an attractive, if only approximate, property of most of the
512: other smoothers that we discuss.
513:
514: \Sectionskip
515: \Section{Cubic smoothing splines}
516: This smoother is not constructed explicitly like those described so far, but instead emerges as the solution to an optimization
517: problem.
518:
519:
520: Consider the following problem: among all functions
521: $f(x)$ with absolutely continuous first derivatives and integrable second derivatives, find the one that minimizes
522: $$\sum_{i=1}^n\{y_i-f(x_i)\}^2 +\lambda\int_{-\infty}^\infty \{f''(t)\}^2 \, dt\eqn{\qb}$$ where
523: $\lambda$ is a fixed constant.
524: This criterion satisfies the requirements for a scatterplot smoother that we
525: mentioned earlier.
526: The first term measures closeness to the data while the
527: second term penalizes curvature in the function.
528: Now it is one thing to state
529: a criterion like $\qb$ and quite another to find the optimizing function.
530: Remarkably, it can be shown that $\qb$ has an explicit, unique minimizer and
531: that minimizer is a {\sl natural cubic spline} with knots at the unique values of $x_i$
532: (Exercise~2.6).
533: At face value it seems that the family is overparametrized, since there are as many as $n-2$ interior knots.
534: This would result in $n+2$ parameters, although the constraints on each end bring it down to $n$. We'll see however that the coefficients are estimated in a constrained way as well, and this can bring the {\sl effective} dimension down dramatically.
535:
536:
537: A cubic
538: smoothing spline fitted to the diabetes data is shown in Fig.~\allsmooths.
539: It looks
540: much like the
541: running-lines smooth
542: but is less jagged.
543:
544:
545: The parameter $\lambda$ plays the same role as the span in the running-lines smooth.
546: Large values of $\lambda$ produce smoother curves while
547: smaller values produce more wiggly curves. At the one extreme, as
548: $\lambda\rightarrow \infinity$, the penalty term dominates, forcing
549: $f''(x)=0$, and thus the solution is the least-squares line.
550: At the other extreme, as $\lambda\rightarrow 0$, the penalty term becomes unimportant and the
551: solution tends to an interpolating twice-differentiable function.
552: We discuss methods for choosing $\lambda$ in Chapter~3.
553:
554: \sectionskip
555: \section{Computational aspects}
556: Using the fact that the solution to $\qb$ is a natural cubic spline with $n-2$ interior knots, we can represent it in terms of a basis for this space of fits.
557: For computational convenience, we will use the unconstrained $B$-spline basis, and write
558: $\gsmooth(x)=\sum_1^{n+2} \gamma_i B_i(x)$, where $\gamma_i$ are
559: coefficients and the $B_i$ are the cubic $B$-spline basis functions.
560: As written here, $s$ lies in an $(n+2)$-dimensional space; however, the natural splines are a subspace.
561: We can now simply replace $f$ in \qb\ by $s$ and perform the integration.
562: Then defining the $n\times (n+2)$ matrix $ \bB$ and $(n+2)\times (n+2)$ matrix $\fat{\Omega}$ by
563: $$
564: B_{ij}=B_j(x_i)$$
565: and
566: $$\fat{\Omega}_{ij}=\int_{-\infty}^\infty B_i''(x)B_j''(x) \, dx,$$
567: we can rewrite the criterion $\qb$ as
568: $$(\vec y-\bB\fat\gamma)^T(\vec y-\bB\fat\gamma) +\lambda\fat \gamma^T\fat{\Omega}
569: \fat \gamma\eqn{\qd}$$
570: Although at face value it seems that there are no boundary derivative constraints, it turns out that the penalty term automatically imposes them.
571:
572: Setting the derivative with respect to $\gamma$ equal to zero gives
573: $$(\bB^T\bB+\lambda\fat\Omega)\hatfat\gamma=\bB^T\vec y\eqn{\eqq}$$
574: Since the columns of $\bB$ are the evaluated $B$-splines, in order from left to right and evaluated at the {\sl sorted} values of $X$, and the cubic $B$-splines have local support, $\bB$ is lower 4-banded.
575: Consequently the matrix $\vec M=(\bB^T\bB+\lambda\fat{\Omega})$ is
576: 4-banded and hence its Cholesky decomposition $\vec M=\bL\bL^T$ can be computed easily.
577: One
578: then solves $\bL\bL^T\hatfat\gamma=\bB^T\vec y$ by back-substitution to give $\hatfat\gamma$
579: and hence the solution $\hat\gsmooth$ in O(n) operations.
580:
581:
582: For theoretical purposes, it is convenient to rewrite the solution
583: vector $\vgsmooth$ ($\gsmooth$ evaluated at each of the $n$ values of $X$ in the sample) in another form.
584: Let $\bN$ be an $n\times n$ nonsingular {\sl natural-spline} basis matrix for representing the solution (Exercise~2.5).
585: Denote by $\hatfat{\beta}$ the transformed version of $\hatfat\gamma$ corresponding to the change in basis. Then we can write
586: $$\vgsmooth=\bN\hatfat{\beta}=\bN(\bN^T\bN+\lambda\fat{\Omega})^{-1}\bN^T\vec y=
587: (\bI+\lambda \bK)^{-1}\vec y\eqn{\qf}$$
588: where $\bK={\bN^{-T}}\fat{\Omega} \bN^{-1}$.
589: In terms of the candidate fitted vector $\vec f$ and $\bK$, the cubic smoothing spline $\vgsmooth $ minimizes
590: $$(\vec y-\vec f)^T(\vec y-\vec f) +\lambda\vec f^T\bK
591: \vec f\eqn{\nqd}$$ over all vectors $\vec f$.
592: It is appropriate to call the term $\vec f^T\bK\vec f$ a roughness penalty, since it can be shown to be a quadratic form in second differences.
593: The cubic smoothing spline, because of the implicit way it is defined, doesn't appear
594: to use {\sl local averaging}.
595: However, the equivalent-kernel weights in Fig.~\hatplot\ show that it does possess local behaviour quite similar to kernels or locally-weighted lines.
596: The equivalent kernel is nowhere nonzero, but is close to zero far from the target point.
597: Other examples of equivalent kernels are given in ^{Buja, Hastie and Tibshirani (1989)}, and
598: an asymptotic form for the equivalent kernel of smoothing splines is derived by ^{Silverman (1984)}.
599: One might say, then, that a cubic smoothing spline is approximately a {\sl kernel}
600: smoother.
601:
602: The smoothing parameter $\lambda$ controls the shape of the kernel
603: or weight function.
604: As one would expect,
605: when $\lambda$ is decreased
606: the weights tend to concentrate more around the target point, while as
607: $\lambda$ is increased, they spread out more.
608: We discuss the properties of smoothing splines in more detail in Chapters~5 and section~9.3.6.
609:
610: \Sectionskip
611: \Section{Locally weighted running-line smoothers}
612: Here we describe in more detail the locally-weighted smoother of ^{Cleveland (1979)}, currently called {\sl loess} in the S statistical-computing language.
613: Although any polynomial can be fitted locally, we discuss local lines.
614:
615: A locally-weighted straight-line smooth $\gsmooth(x_0)$ using $k$ nearest neighbours is computed in a number of steps:\smallskip
616: {\parindent 20pt
617: \item{(i)} The $k$ nearest neighbours of $x_0$ are identified, denoted by
618: $\NN(x_0)$.
619: \item{(ii)}$\Delta(x_0)=\max_{N(x_0)}\abs{x_0-x_i}$ is computed, the distance of the furthest near-neighbour from $x_0$.
620: \item{(iii)}Weights $w_i$ are assigned to each point in $\NN(x_0)$, using the {\sl tri-cube} weight function:
621: $$W\biggl({\abs{x_0-x_i}\over {\Delta(x_0)}}\biggr)
622: \eqn{\clev}$$
623: where $$W(u)=(1-u^3)^3 \quad\for\quad 0\leq u < 1 \eqn{\tri}$$
624: and zero otherwise.
625: \item{(iv)} $\gsmooth(x_0)$ is the fitted value at $x_0$ from the weighted least-squares fit of $y$ to $x$ confined to $\NN(x_0)$ using the weights computed in (iii).
626:
627: }\smallskip
628: \input loessm.tex
629: \par
630: Figure~\loessm\ depicts the situation, using the diabetes data once again.
631: Notice that the kernel is truncated at one end due to the metric asymmetry of the neighbourhood.
632:
633: Cleveland also discusses the use of a robust regression within each
634: neighbourhood, to protect against outliers.
635: This effectively works by repeatedly smoothing the data, and at each iteration down-weighting points with large residuals.
636: See Chapters~3 and 9 for more details.
637:
638: A locally-weighted running-line smooth
639: is shown in Fig.~\allsmooths.
640: The number of nearest-neighbours, usually expressed
641: as a percentage or {\sl span} of the data points, is the smoothing parameter.
642: In Figs~\allsmooths\ and \loessm\ a span of $21/43$ was used, chosen to make
643: the degrees of freedom approximately equal to that of the other smooths.
644:
645: In principle, nearest neighbourhoods
646: are preferable to
647: symmetric nearest neighbourhoods because in a neighbourhood with
648: a fixed number of points, the average distance of the points to the target point
649: is less in the nearest neighbourhood (unless the predictors are evenly spaced).
650: In general this should result in less bias.
651: We use symmetric nearest neighbourhoods in a running-lines smoother, however,
652: because of their superior performance at the left and right endpoints.
653: A nearest neighbourhood at the endpoint contains the same number of data
654: points
655: as a neighbourhood in the middle of the data,
656: so that when a least-squares fit is applied there, too much
657: weight is given to points far away from the target point.
658: A symmetric nearest neighbourhood, on the other hand, contains only about
659: half of the number of points as a full neighbourhood, and thus
660: automatically reduces the weight given to far way points.
661: Nearest neighbourhoods work satisfactorily with {\sl loess} at the endpoints, however,
662: because the tri-cube function does the job of down-weighting far away points.
663:
664: The locally-weighted smoothers are popular, since they enjoy the best of both worlds.
665: They share the ability of near-neighbour smoothers to adapt their bandwidth
666: to the local density of the predictors, while they have the smoothness features
667: of kernel smoothers.
668:
669: Although in principal it requires $O(n^2)$ operations to compute a locally-weighted smooth, the current implementations compute the fit over a grid of values of $x$, and use interpolation elsewhere.
670:
671: \Sectionskip
672:
673: \Section{Smoothers for multiple predictors}
674: So far we have talked about smoothers for a single predictor, that is,
675: scatterplot smoothers.
676: What if we have more than one predictor, say $X_1,\ldots, X_p?$.
677: Then our problem is one of fitting a $p$-dimensional surface to $Y$.
678: The multiple regression of $Y$ on $X_1,\ldots, X_p$ provides a
679: simple, but very limited, estimate of the surface.
680: On the other hand,
681: it is easy conceptually to generalize the running mean, locally-weighted
682: running-lines, and kernel smoothers to this setting.
683: The first two smoothers require a definition of {\sl nearest neighbours}
684: of a point in $p$-space.
685: {\sl Nearest} is determined by a distance measure and for this
686: the most obvious choice is Euclidean distance.
687: Note that the notion of symmetric nearest neighbours is no longer meaningful
688: when $p > 1$.
689: Having defined a neighbourhood,
690: the generalization of the running mean estimates
691: the surface at the target point by averaging the
692: response values in the neighbourhood.
693:
694: This
695: highlights an important detail: what shape neighbourhood do we use?
696: Nearest-neighbourhoods defined in terms of Euclidean distance are typically
697: spherical, but one could imagine more general
698: neighbourhoods defined in terms of a covariance matrix $\Sigma$ of the predictors.
699: Such a metric would consider points lying on an ellipse centered at the target
700: point to be equidistant from the target.
701: This generalization is not a purely
702: academic issue --- it may be important if the covariates are measured in different
703: units or are correlated. In linear
704: regression, the coefficients automatically scale each covariate
705: correctly.
706: Various strategies are used in practice, an obvious one being to standardize the individual variables prior to smoothing.
707: This might not always be the best choice, however.
708: For example, if the underlying surface changes more rapidly with one variable
709: than the other, one may wish to have the neighbourhoods thinner in the direction of the first variable.
710: This can be achieved by differential scaling of the variables.
711: \figname{\bivar}\par
712: The kernel smoother is generalized in an analogous way
713: to the running mean.
714: Given two $p$-vectors
715: $\vec x^0$ and $\vec x^i$
716: in predictor space,
717: the weight given to this $i$th point for the fit at $\vec x^0$
718: is $S_{0i}=(c_0/\lambda)
719: d({{\norm{\vec x^0-\vec x^i}}/\lambda})$
720: where $d(t)$ is an inverse distance measure
721: and $\norm{\cdot}$ is a norm, for example squared distance.
722: The constant $c_0$ is usually chosen to make the weights sum to unity, so that the smoother
723: reproduces the constant function.
724: The multi-predictor Gaussian kernel uses
725: $d(t)$ equal to the standard Gaussian density.
726: The resulting weight function is depicted in Fig.~\bivar.
727: \midinsert
728: \vbox{
729: \centerline{\psfig{file=figures/bivar.p,width=3in}}
730: \figurecaption{\bivar}{ The weight function for a bivariate Gaussian kernel smoother.
731: }*
732: }
733: \endinsert
734: The choice of norm is important in this setting, and determines the shape of
735: the neighbourhoods in predictor space.
736: One might argue, for example, that the norm should incorporate
737: covariance information by replacing the
738: squared error norm with the ``Mahalanobis'' distance
739: $$\norm{\vec x^0-\vec x^i}=(\vec x^0-\vec x^i)^T\Sigma^{-1}(\vec x^0-\vec x^i)$$
740: where
741: $\Sigma$ is the covariance matrix of the predictors.
742:
743:
744: Locally-weighted lines generalize equally easily to two or more dimensions.
745: The simplest generalization hardly needs any different wording than that used to describe the univariate situation.
746: Once a multivariate distance is defined,
747: nearest-neighbourhoods and tri-cube weights get assigned in exactly the same way.
748: One computes a local plane rather than a line.
749:
750:
751: The cubic smoothing spline is more difficult to generalize to two or higher
752: dimensions: the so-called {\sl thin-plate spline} is one such
753: generalization.
754: It derives from generalizing the second derivative penalty for smoothness to a two dimensional Laplacian penalty of the form
755: $$\int\int\left\{\left({\partial^2 f\over \partial x_1^2}\right)^2 + \left({\partial^2 f\over \partial x_1\partial x_2}\right)^2+\left({\partial^2 f\over \partial x_2^2}\right)^2\right\}\, dx_1dx_2.$$
756: The computations are $O(n^3)$ in number compared to the $O(n)$ for univariate splines.
757: Details are not given here; the interested reader can find references in the
758: bibliographic notes at the end of Chapter~3.
759:
760: Another generalization is known as multivariate {\sl tensor product} splines.
761: These are also useful for generalizing univariate regression splines.
762: The basic idea is to construct two-dimensional basis functions by multiplying together one-dimensional basis functions in the respective predictors.
763: We discuss these in more detail for multivariate regression splines in Chapter~9.
764: For the smoothing version of tensor product splines, the effective basis set has dimension $n^2$ formed by the tensor product of the univariate bases.
765: The fit will be a damped regression onto this space of basis functions.
766: See the references at the end of Chapter~3 for details.
767:
768: We are not devoting much space to multi-predictor smoothers because we don't
769: feel that they are very useful for more than two or three predictors.
770: Indeed, their many shortcomings (e.g.~difficulty of interpretation and computation)
771: provide an impetus for studying additive models, the central
772: topic of this book.
773: These comments refer to the generic multivariate smoothers as described here.
774: We are not referring to some of the adaptive multivariate nonparametric regression methods, which might also be termed surface-smoothers, and which were designed to overcome some of these objectional aspects.
775: We discuss some of the problems associated with multi-predictor smoothing in detail at the
776: beginning of Chapter~4.
777: Despite these criticisms, multi-predictor smoothers (especially the bi-predictor) are useful
778: in certain settings, as later examples show.
779:
780: \Sectionskip
781: \Section{Discussion and bibliography}
782: In this chapter we have described a number of scatterplot smoothers,
783: including the running-mean and running-lines smoothers, cubic regression and smoothing
784: splines, locally-weighted running-lines and kernel smoothers.
785: We have also touched on surface smoothers.
786: Our discussion has focused on smoothers with which we have some experience, and any imbalance in our introduction simply reflects this.
787: In addition, the emphasis in the book is not on smoothers {\sl per~se}, but
788: on
789: smoothers as building blocks in the estimation of additive models.
790:
791:
792: Some recent theoretical results suggest that for appropriately chosen
793: smoothing parameters, there are not likely to be large differences between
794: locally-weighted running-lines, cubic smoothing-splines and kernel smoothers
795: (^{Silverman, 1984}; ^{Muller, 1987}).
796: In order to assess the finite-sample operating characteristics of these
797: and other smoothers, a comprehensive Monte Carlo study, akin to the
798: Princeton robustness study, would be very useful.
799: A recent report by ^{Breiman}
800: and Peters (1988) has this flavour.
801:
802: For a detailed bibliography, see the notes at the end of Chapter~3.
803: Here we list some introductory references we have found useful.
804: ^^{Eubank, R.L.} Eubank (1988) covers most of the material in various levels of detail, especially smoothing splines.
805: ^^{Silverman, B.W.} Silverman's (1985) paper on smoothing splines not only gives an easy to read introduction but also touches on a variety of applications.
806: ^^{Cleveland, W. S.} Cleveland (1979) introduces the locally-weighted running-line smoother for univariate problems, and
807: ^^{Devlin, S.}^^{Grosse, E. H.} ^{Cleveland, Devlin and Grosse (1988)} describe the multivariate version.
808:
809: \Sectionskip
810: \Section{Further results and exercises 2}
811: \Mark{EXERCISES \ 2}%
812: \beginexercises%
813: \exercise For the diabetes data, compute one of each of the smooths described
814: in this Chapter
815: for the variable \name{base deficit}.
816: Compare qualitatively the resulting estimates in terms of notable features
817: and smoothness.
818:
819: \exercise {\sl Updating formula for running-line smooth.}
820: Suppose we add a point $(x_{j+1},y_{j+1})$ to a neighbourhood
821: containing $j$ points.
822: If the means, variance of $X$, and covariance of $X$ and $Y$ for the first $j$ points are
823: $\bar x_j$, $\bar y_j$, $S^x_j$, and $S^{xy}_j$, show that
824: $$\bar x_{j+1}=(j\bar x_j+x_{j+1})/(j+1)$$
825: $$\bar y_{j+1}=(j\bar y_j+y_{j+1})/(j+1)$$
826: $$(j+1)S^x_{j+1}=jS^x_j+{{j+1}\over j}(x_{j+1}-\bar x_{j+1})^2$$
827: $$(j+1)S^{xy}_{j+1}=jS^{xy}_j+{{j+1}\over j}(x_{j+1}-\bar x_{j+1})(y_{j+1}-
828: \bar y_{j+1}).\eqn{\upd}$$
829:
830: What are the equivalent equations for deleting a point?
831: Together these
832: can be used to update the least-squares slope and intercept, and hence the entire fit can be computed in $O(n)$ operations.
833:
834: \exercise Derive an expression for $S_{ij}$, the coefficient of $y_j$ in the
835: expression $\gsmooth(x_i)=\sum_{j=1}^n S_{ij}y_j$ for the running-line fit at $x_i$,
836: using $\symNN(x_i)$ to denote the set of indices of the $k$ symmetric nearest-neighbours to $x_i$.
837:
838:
839: \exercise Show that the truncated power series representation $$\gsmooth(x)= \beta_0+\beta_1x+\beta_2x^2+\beta_3x^3+\sum_{j=1}^k\theta_j(x-\xi_j)_+^3$$ satisfies the three conditions of a cubic spline given in the text.
840:
841: \exercise {\sl Basis for natural splines.} Suppose $B$ is an $n\times (K+4)$ matrix containing the evaluations of the cubic $B$-spline basis functions with $K$ interior knots evaluated at the $n$ values
842: of $X$.
843: Let $C$ be the $2\times (K+4)$ matrix containing the second derivatives
844: of the basis functions at the boundary points $x_1$ and $x_n$.
845: Show how
846: to derive $N$ from $B$, an $n\times (K+2)$ basis matrix for the natural cubic splines with the same interior knots and boundary knots at the extremes of the $X$.
847:
848: \exercise {\sl Derivation of smoothing splines; ^{Reinsch (1967)}.} Consider the following optimization problem: minimize
849: $$\int_{-\infty}^\infty \{f''(x)\}^2 \, dx
850: \quad \hbox{subject to}\quad \sum_{i=1}^n \{y_i-f(x_i)\}^2 \leq \sigma
851: \eqn{\sopt}$$
852: over all twice differentiable
853: functions $f(x)$.
854: To solve this problem, consider the Lagrangian functional
855: $$F(f)=\int_{-\infty}^\infty\{f''(x)\}^2 \, dx + \rho \left[ \sum_{i=1}^n \{y_i-f(x_i)\}^2-\sigma+z^2\right]\eqn{\lageq}$$
856: where $z$ is an auxiliary variable.
857: \smallskip
858: {\parindent 20pt
859: \item{(i)} To find the minimum of $F(f)$, compute $d F\{f+\delta h(x)\}\delta$ where
860: $h(x)$ is any twice differentiable function.
861: (This is a standard technique in the calculus of variations).
862: Integrate by parts the resulting expression (twice) and conclude that the solution
863: is a piecewise cubic polynomial with knots at the $x_i$.
864: Show that the solution and its first two derivatives are continuous at the
865: knots, but that the third derivative may be discontinuous there.
866: Show also that the second derivative is zero outside of the range of the
867: $x_i$.
868:
869: \item{(ii)} Compute $d F(f)\rho$ and $\partial F(f)/\partial \sigma$ and conclude that either $\hat \rho=0$
870: implying that the least-squares line is a solution to $\sopt$, or
871: $\hat\rho\neq 0$, in which case the solution is nonlinear.
872:
873: \item{(iii)} Establish an equivalence between the $\sopt$ and the penalized least-squares
874: problem $\qb$ given in the text.
875: }
876:
877: \exercise Suppose each observation has associated with it a weight $w_i$.
878: \smallskip
879: {\parindent 20pt
880: \item{(i)} Derive the appropriate cubic smoothing spline using a weighted residual sum of squares.
881: \item{(ii)} Suppose there are ties in the $x$-values.
882: Suggest a way to overcome this for smoothing splines.
883:
884: }
885:
886: \exercise {\sl Semi-parametric regression; ^^{Green, P.J.}^^{Jennison, C. }^^{Seheult, A.} Green, Jennison, and Seheult (1985).} Suppose we have a set of $n$ observations of $p$ predictors arranged in
887: the $n \times p$ matrix $\bf X$, an additional covariate $\vec z$, and a response vector $\vec y$.
888: We wish to fit the model $y_i=\vec x^i{\fat\beta} +f(z_i) +\varepsilon_i$ by penalized least-squares.
889: Construct an appropriate penalized residual sum of squares, and show that the minimizers must satisfy the following pair of {\sl estimating} equations:
890: $$\eqalign{ {\fat\beta}&=(\X^T\X)^{-1}\X^T(\vec y-\vec f)\cr
891: f &= \smooth(\vec y -\X{\fat\beta}\given \vec z)\cr}$$
892: where $\smooth$ is an appropriate smoothing-spline operator, and $\vec f$ represents the function $f$ evaluated at the $n$ values $z_i$.
893: \exercise {\sl Efficient kernel smoothing; ^{Silverman (1982)}, ^{H\"ardle (1986)}}
894: Consider the ^^{Nadaraya, E.A.}^^{Watson, G.S.}Nadaraya-Watson form of the kernel smooth given by equation \NadWat:
895: $$\gsmooth(x_0)={\sum_{i=1}^n\rho({x_0-x_i \over \lambda})y_i\over \sum_{i=1}^n\rho({x_0-x_i \over \lambda})}.$$
896: We wish to use the FFT to compute both the numerator and denominator.
897: With that end in mind we define a fine grid that covers the range of $X$.
898: Describe how you would approximate the data on the grid and use the FFT to perform the computations.
899: Pay particular attention to:\smallskip
900: {\parindent 20pt
901: \item{(i)} what happens at the boundaries, and
902: \item{(ii)} a method for overcoming discontinuities due to the approximation.
903:
904: }
905: \exercise Extend the methods described in the previous exercise in order to compute a kernel-weighted running-line fit.
906: \exercise Suppose $\bS$ is an $n\times n$ smoother matrix for
907: smoothing against the $n$ unique predictor values $\vec x=(x_1,\ldots, x_n)^T$.
908: That is, if $s(x_i)=\sum_{j=1}^n S_{ij}y_j$, then $\bS$ has $ij$th entry $S_{ij}$.
909: For all of the
910: smoothers that we have described, $\bS\vec 1=\vec 1$, and for most $\bS\vec
911: x=\vec x$. For which is the second statement true, and what are the
912: implications of these two results? For smoothing splines the symmetric
913: statements are also true: $\bS^T\vec 1=\vec 1$ and $\bS^T\vec x=\vec
914: x$, since $\bS$ is symmetric in this case. Is this true for any of the
915: other smoothers? What are the implications of these conditions? How
916: would you modify $\bS$ for the other smoothers in order that the first
917: or both of these latter conditions are satisfied. Try some examples
918: and see what happens to the local support properties of the equivalent
919: kernels.
920: \endexercises
921: \vfill\supereject
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.