By John Sall, Co-found and Executive VP of SAS and head of the JMP software division
Wednesday, June 18, 2008
Computing has revolutionized our testing capacity, but efficient design of experimentation remains a valuable tool for success.
 John Sall Co-founder and Executive VP of SAS and head of the JMP software division. |
We have come a long way in harnessing the power of scientific experimentation, and many of the most important advances have come along in the last few years as computing power has enabled new ideas to emerge and become practical. Here is a brief history of the state of the art in experimentation.
Experimentation is learning
Experimentation is trial and error, something we have always done. Try something and see what happens. Experiment has the same word root as experience, and actually is the same word in French. You learn from experience. You learn from varying factors and observing the results. This is the inductive method that encompasses most learning.
The controlled experiment
However, we are sometimes fooled unless we take care to control the experimental conditions. Controlling them is the first critical step in doing scientific experiments. This is the step that distinguishes experimental results from happenstance data. You might obtain clues about how the world works from non-experimental data, but you cannot put full trust into learning from happenstance phenomena.
The statistical framework
Experiments are not exactly repeatable, because the response has a random component that is not controlled. Understanding how to model experiments was first mastered by R. A. Fisher and others in the 1920s. They understood the importance of randomization and blocking and developed standard statistical distributions and tests.
Multifactor experimentation
Experimenting with only one factor at a time is both inefficient and blind to interactions.
If it takes a certain number of experimental runs to obtain a precise enough estimate for one factor, then you can use those same experimental runs to test another factor if you arrange the factor settings efficiently. For example, suppose that you have one factor with two levels, and you need eight runs to get the standard error of the means difference estimate to be small enough to meet your requirements. If you have seven factors, running each one as a separate experiment takes 7*8 = 56 runs. You can use an experimental design to run all seven factors in eight runs and have the same standard error of the difference, thus saving you 7/8 of your runs. This is the true magic of experimental design—using the same runs to answer more questions without spending anything more.
Adding more factors into a design does add cost to your estimate of the variances and the significance tests that use these variances. However, multifactor designs also introduce opportunities for interactions. It becomes important to design carefully so that lower-order interactions do not distort main effects, and if there are enough runs, to estimate those interactions. Multifactor designs also allow you to adjust for variation from external factors and better focus on the effects of interest.
Orthogonal designs
An algebra of orthogonality was developed to construct designs in which the effects are orthogonal. Orthogonal designs are efficient, produce tests that are independent, project well to make higher-order interactions estimable, and most of all, make the calculations so simple that you could do it all without a computer. An initial drawback was that only certain kinds of rigid layouts were orthogonal and you had to pick them out of a design catalog or from algebraic generators, like Hadamard matrices (Plackett-Burman) or 2(n-k) confounding generators (fractional factorial). With the advent of computers, orthogonality is no longer a compelling need. If you can budget 17 runs, but there are no orthogonal designs that have 17 runs, you don’t throw away the 17th run just to make the design orthogonal. That 17th run only increases what you can learn from the experiment.
Response surface designs
Response surface experimentation allows researchers to estimate curved surfaces and then find settings of the continuous factors that produce the most desirable response. Classical response surface designs use quadratic polynomials to model the curved surface.
Optimal designs
The idea for optimal designs moved design from algebra to calculus and from pencil and paper to computers. Optimal designs produce the most efficient designs without the restrictions imposed by earlier designs that the number of runs be a power of two or certain other numbers. Instead of picking a design from a catalog, you specify what you want to model and let the computer create the optimal experiment for that model. Optimal designs also support restrictions on factor settings, as well as optimal design augmentation to add runs that can answer further questions on interactions or refine estimates. D-Optimal designs maximize the efficiency on evaluating the significance of the effects. I-Optimal designs minimize the average error of prediction over a response surface. In situations where you want the design to prefer high-order effects to be estimable when that is possible, the Bayesian D-Optimal method is available.
Screening designs for a large number of factors
Initial experiments are often used for screening many factors to see which seem to have a large effect. You expect only a few factors to have a sizable effect. This situation is called effect sparsity. It turns out that in this situation you can find the large effects with an extremely lightweight design—one with fewer runs than factors—called a supersaturated design. Design of Experiments (DOE) software with Bayesian D-Optimal features can produce these designs by making the main effect estimability if possible instead of required.
Response surface designs for a large number of factors
If you have a great many factors and the experiment is allowed to pick a high-dimensional corner of the design space, then it uses a combination of factor settings that, used together, become extreme and beyond realistic settings. Spherical designs prevent the design from having these high-dimensional far-out corners, keeping the radius of the experimental design space within bounds.
Flexible response surfaces
While quadratic polynomial response surface models approximate local behavior well, they are incapable of modeling complex wide-scale response surfaces. Higher-order polynomials are sometimes effective, but there are better, more flexible models that can handle any local behavior. To model flexible response surfaces, you need a space-filling design that spreads the points more evenly across the space. Then you use a smoother, such as a multivariate smoothing spline, a neural net, or a Gaussian process model, to fit the surface. These kinds of designs and models are especially valuable for computer simulation experiments.
Exploitation of response surface fits
Modern software does more than just estimate the models. Effective design software provides different visualizations of the response surface together with tools for optimization and simulation. This can include features to better tune processes and make them more robust with respect to variation in the factors.
Designing with hard-to-change factors
Often, in a typical industrial experiment, some factors are easier to reset than others. If you need an experiment that is not too costly, i.e. one that doesn’t often change the hard-to-change factors, then you might be able to use a split-plot design.
Only recently has it become possible to optimally construct split-plot designs. Many industrial experiments are actually inadvertent split-plot designs, where the runs are sorted and the hard-to-change factors are not randomized; if the experiment is not analyzed with respect to this structure, the results are invalid and possibly misleading, depending on the strength of the whole-plot-to-split-plot variance ratio.
Recent advances in fitting from Kenward and Roger produce precisely sized tests of effects in small samples.
This approach works even in situations where the within-block covariance estimates are negative, which sometimes happens with small designs.
Designs for nonlinear models
A difficult problem with models that are nonlinear in the parameters is that the optimal design depends on the values of the parameters themselves. Current experimental design software for nonlinear models allows you to specify a distribution on the parameters, and the resulting design has good learning behavior with respect to that distribution. Recent computational advances using Mysovskikh quadrature make designs for nonlinear models practical.
Choice designs
In our world of engineering, we tend to focus on making products well (process Six Sigma) and making good products (design for Six Sigma), but for consumer products, we sometimes neglect to optimize for product features that people want (market research). Choice experiments are the tool to model the value system of consumers with respect to product features, and recent research has been producing optimal experimental designs for choice experiments. Methods due to Firth have led to improved estimates and tests.
Conclusion
When you consider your enterprise experimentation policies and procedures, you should take a look at some of the more recent advances in experimental design.
You don’t have to be an expert statistician to use these latest advances. Fast computers and powerful software have made experimental design fairly straightforward. It has never been easier or more effective to put experimentation to work in improving your products and processes. Software has unlocked the opportunities and value available in experimentation, and everyone should consider doing more experiments. Efficient learning lowers the cost of learning and yields more value.
Published in R & D magazine: Vol. 50, No. 3, June, 2008, p.13-15.