Nice post. I just read that article in CIO Insight after reading Jim Novo’s blog. I think you illustrate well how false dependencies can be made.

Your point:

1. is right on: I have been an advocate for adding attitudinal analysis to the behavioral one for a long time. But wouldn’t “reasons” for visits add to the extra variables? Isn’t there a risk to add to false dependencies? This being said, I am a big proponent of free will, and believe there is still some left in us, the consumers. This means I find a lot of explanation in the “why” people say they do what they do.

3. Nice advise. Too much friction definitely don’t help the whole process.

4. “variables that make sense”, yes, but I think this is the whole question here: how does one recognize what makes sense, I mean, with the stuff that’s not obvious (i.e. the wheather, etc.)? Isn’t there an element of discovery and learning?

5. Could you explain a little more?

6. ?

4. When building models you don’t want to include variables that may not make sense when you try to explain it. In some cases you have to be cognoscent of protected classes in law. You can’t build a credit score with demographics like age, race, etc…. Also, I am thinking of this from a practicality standpoint. Try to make the model as simple as possible. That makes it easier to implement to future events. But you are right in it does take away some of the surprise element.

5. When you use data mining techniques to build models you need to test the predictive accuracy, computational speed, robustness, scalability, and interpretability (point #4 above). Think of this as taking a test twice with the same questions. You’d expect to get a better score the second time around on the same set of questions but if you take a test on the same topic but with two sets of questions you may or may not do better the second time around. So many people don’t hold back enough data to conduct a sound validation of the model. The validation process helps identify the accuracy of the overall model.

6. There is a DOE (design of experiment) that I think applies better to transactional data or data that has an order/sequence which is called Repeated Measures. This technique is not used enough (in my opinion). So many people just use aggregate data which may be enough but the strength in knowing when something will happen with better precision is golden. Peter Fader touches on this in his interview.

Hope this helps!
Wendi

If the task at hand is to make predictions why would you worry about covariance of your independent variables (assuming they are not perfectly collinear, in which case you are not adding any additional information)? Won’t this just increase the variance around the parameter estimates? Does that matter if they are not policy (control) variables? Also is this assuming that you are using regression as a data mining method?
There are various non OLS methods that work in very large/infinite feature spaces that might be better for classification and prediction problems than OLS.
I agree with you about rule 5 with one caveat - rather than making this a hard and fast rule the analyst should estimate the marginal benefit of the data vs the cost to acquire - and in fact that should be done for all of the data. If data is really expensive (time, hassle, $$$) but improves predictions so that the incremental return is greater than the cost of the data then by all means get that extra data.