When is a Model a Model?

I recently gave a short talk to a mixed group of scientists and engineers (including a small body of statisticians). The purpose of the talk was to briefly describe a rather large simulation study that I had completed.  The study sought to understand how various departures from our model assumptions affect quantities that we were interested in calculating. 

So suppose that our model is a very simple linear model:

Many of you will recognize this as a straight line. For this model I will assume the typical distributional properties for  , i.e., iid and normal distributed, let's say standard normal for simplicity.  We also assume that the model parameters and    (my apologies for the misaligned math symbols, I am figuring out the best way to input math symbols into this blog) are unknown. Assume this is the model that I was describing in my talk.

After I described the model given above, I then discussed how when we calculate our various statistics, we just assume that we know the values of the model parameters given above and evaluate them into our carefully derived statistical formulas.  In the simulation study, we simulated data from the model form above, calculated parameter estimates for the unknown parameters (given above) from the simulated data, and then assess the variability in the test statistics (I was being a proper frequentist).   Here is where my talk was criticized.

The main criticism that I received was that the "head" statistician in the room claimed that my linear model given above was not a model.  She then issued her definition of a model, which is not only the linear form of the model and distributional assumptions, but we must also specify values of the unknown model parameters.  My definition of the model as presented was the distributional assumptions and the linear form whereas her's would have also included values of the model parameters.  Do you see the slight difference?

Unfortunately, when this criticism was thrown at me, I stood there without a response.  In all honesty, I felt that I did not have a good response to her criticism because I realized that as an early career statistician, I have not personally developed my own philosophy/definition of what I think a model is.  Or, to put it another way, at what point has our mathematical object been specified enough that we may declare it as a model?

 Currently I do not have a good answer to my own question "At what point have we listed enough things that we may declare our list a model?"  But I am curious, what do you think a list of objects must include in order for us to have a proper, statistical model?

The Challenger O-ring data

The Challenger disaster data is one that I have seen numerous times.  The first time that I have seen this data (given below) was when I was a temporary laboratory instructor for an introductory statistics course. Typically, this data set is used to argue how important it is for someone to have basic data analysis skills. Another Challenger related data set can be found in the Journal of the American Statistical Association article that contains other variables and analysis.

The data set given below consists of observations made on various o-rings used in solid rocket boosters.  Each o-ring was tested at a different temperature and the erosion depth of the o-ring was measured.  The trend that one can see is that as temperatures decrease, the erosion depth of the o-rings increases, leading to a failure of the o-ring.  The general conclusion that was reached from this data set is that these o-rings should not have been used at cold temperatures.

Below is the data.  For more on the disaster, see wikipedia's page.

O-ring temp in °F	Erosion depth,δ mils *
66.0	0.0
70.0	53.0
69.0	0.0
68.0	0.0
67.0	0.0
72.0	0.0
73.0	0.0
70.0	0.0
57.0	40.0
63.0	0.0
70.0	28.0
78.0	0.0
67.0	0.0
53.0	48.0
67.0	0.0
75.0	0.0
70.0	0.0
81.0	0.0
76.0	0.0
79.0	0.0
75.0	0.0
76.0	0.0

 

A New Section

Loyal Readers,

I have decided to add a new section to this blog.  Because I can't seem to get new datasets quick enough to discuss on this blog, I thought it might be fascinating to seek out some of the most historical datasets and discuss them on this blog.  For example, the O-ring data for the Challenger disaster.

I believe it is time that we (i.e., statisticians) began to collect all these well known and well studied datasets and put them in a single place.  This way, future statisticians can study their statistical roots through these data sets.

More to come and if you have any ideas for historical datasets, please leave me a comment.

Brian

When you insult something how will it respond

In early October, I, along with a colleague of mine, gave a talk at the joint American Statistical Association and American Society for Quality Fall Technical Conference in Kansas City.  The purpose of this conference is to share ideas among Statisticians and researchers in the engineering and physical sciences, though there is a high concentration towards industrial engineering (which I think is a drawback to this conference and would like to see other areas of science represented).

The talk that my colleague and I gave, I felt, totally stood out at this conference.  Most talks focused on how to improve the quality of items (quality being a broad term) and designing experiments in an industrial setting.   Ours, on the other hand, focused more on incorporating the current scientific knowledge about a nonstandard experiment (which isn't much) into some parsimonious (simplest equation that describes the science) model (though it is hard to say if it is).  I am not trying to discount the other talks, I would have just liked to see more nonstandard problems and the researchers approach to solving them.

I will spend the rest of this entry describing our talk which I hope you find interesting and clear if you are not a statistician (and if all goes well, this work will be submitted to the Journal of the American Statistical Association...boo yeah).

There is a safety group at my "place of business" whose job is to think up different scenarios that would cause a detonator (det) to "Go" (explode) when they do not want it to.  Typically, these scenarios involve either the environment that the det resides in or the workers who "tinker" with these dets accidentally insulting the det (oops I dropped my sandwich on the det. It always amazes me how calm people are who work around explosives.  One guy was even having me smell some them...which was pretty AWESOME.  They remind me of Coroners eating around corpses.  Sorry random tangent haha).  The researchers are interested in how the det will respond to such insults.  Below is a data set from such an experiment.  

The data represents an experiment where the researchers insulted a det with what they called electrostatic discharge.  They controlled this discharge by varying different levels of voltage and capacitance (i.e., they paid grad students to rub their feet on carpet and touch the det HAHA).  They then continued to insult the det until it either responded with a Go or degraded past a point that it will never Go. 

The researchers are interested in Go probabilities for a certain number of insults (usually 1).  They are then really happy if the Go probabilities are extremely small (there are issues with this type of estimation and if you are interested leave me a comment).  My colleague and I then needed to develop a class of models (equations that we pretend produced the data) that we think approximately is the "machine" that created the data and captures the current scientific knowledge for detonators.  

The current believe in the scientific community is that for a det to Go (probability of a Go is very large), the voltage and the energy (energy = .5 times capacitance times voltage^2) that the det was exposed to needs to be larger than some threshold value (which of course are unknown, the researchers didn't want to make this too easy for us).  Our job as statisticians is to take this scientific information and develop models that capture this information.     In other words, when I look at probabilities as a function of voltage and energy, the proposed model should somehow capture this notion of voltage and energy thresholds.

If we can find or develop a class of models that capture the already mentioned information then we can use the data to fit the models to the data.  Fortunately, my colleague and I developed a models which we think does a decent job of explaining the data and the science.  The following figure shows our estimates of one-insult probabilities (using our models) as a function of energy and voltage (for you stats nerds the left plot represents the lower credible bound, the middle plot is the median (posterior) estimate of one-insult probabilities and the upper plot represents the upper credible bound.  The contours represent log 10 probabilities).  Notice that the middle plot has an "L" shape in the contours.  This shape represents the notion of thresholds values.  

How you say does this plot capture the current science?  Great question, please let me explain.  Notice that for a fixed voltage, the probability of Go (for a single insult) increases as energy increases.  As the energy continues to increase, however, notice that the probability quits increasing (the vertical and roughly parallel lines towards the top of the plot).  In other words, it is not enough to keep pumping energy into the det, but we also need to expose it to a large voltage in order to get the probability to approach one.   


The left and right plots are used to represent our uncertainty in the shape of Go probabilities.  This statement means that we never really know what the true shape is, the best we can do is estimate this shape using the data and current scientific knowledge.  These estimates will never be correct so we use the other plots to give a range of "plausible shapes" and hope that this range covers the truth.


I will eventually provide a link providing the statistical mechanics that produced all these results.  But that will have to wait until the work is published.  Leave me a comment if you are interested in the mechanics and I can at least send the talk which gives a little more detail.

The Lorenz Attractor

Chaos theory is an odd thing.  One would think it implies randomness...but it doesn't.  It is actually a deterministic system (i.e., no randomness).  Below is something called the Lorenz attractor.  This beautiful image was named after the MIT meteorologist Edward Lorenz  who I think is credited with the discovery of chaos theory (or at least one of the discoverers).  This plot is where the term "butterfly effect" came from.  Unfortunately, if you google butterfly effect, the main search results are those weird movies.  Enjoy the beauty.

My First Research Poster

I was recently challenged to submit a poster for the upcoming conference CODA (which is being put together by the group that I work for).  This is the first poster I have ever done.  Fortunately, there is this website by a scientist who provides tips on making great posters.  This site helped me extensively, especially since I have no experience making posters.  If you are interested, the poster is below.  For some of the background, read the bit on calorimeters in the "Drifting Bias" post.  Enjoy!!