Comments for Coremark Analytics http://coremarkanalytics.com/blog Web Analytics Blog - Paving the way to understanding web data as it relates to statistics and other methodologies. Sun, 05 Feb 2012 01:53:48 +0000 http://wordpress.org/?v=2.1.3 Comment on Comparing Population Proportions – A/B Testing by tazman http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-15340 tazman Mon, 30 Mar 2009 22:07:57 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-15340 Wendi: Another little 'gotcha' I discovered is that you had to put the higher performing number in the LP1 Conversion cell, otherwise you would always fail to reject! It took me a bit of playing around to figure that out as I couldn't understand how I wasn't getting significant results (and I had sample sizes in the 750k+ range)! A quick fix so you don't have to remember to do that is add an if caluse in the p1-p2 calculation like this: =IF(C7>D7,C7-D7,D7-C7) that way you don't have to worry about the order of entry... Wendi:

Another little ‘gotcha’ I discovered is that you had to put the higher performing number in the LP1 Conversion cell, otherwise you would always fail to reject! It took me a bit of playing around to figure that out as I couldn’t understand how I wasn’t getting significant results (and I had sample sizes in the 750k+ range)! A quick fix so you don’t have to remember to do that is add an if caluse in the p1-p2 calculation like this:

=IF(C7>D7,C7-D7,D7-C7)

that way you don’t have to worry about the order of entry…

]]> Comment on Stop Collecting So Much Data… by Matt Gershoff http://coremarkanalytics.com/blog/2007/07/02/stop-collecting-so-much-data%e2%80%a6/#comment-4811 Matt Gershoff Thu, 24 Jul 2008 03:38:17 +0000 http://coremarkanalytics.com/blog/2007/07/02/stop-collecting-so-much-data%e2%80%a6/#comment-4811 Wendi, If the task at hand is to make predictions why would you worry about covariance of your independent variables (assuming they are not perfectly collinear, in which case you are not adding any additional information)? Won’t this just increase the variance around the parameter estimates? Does that matter if they are not policy (control) variables? Also is this assuming that you are using regression as a data mining method? There are various non OLS methods that work in very large/infinite feature spaces that might be better for classification and prediction problems than OLS. I agree with you about rule 5 with one caveat - rather than making this a hard and fast rule the analyst should estimate the marginal benefit of the data vs the cost to acquire - and in fact that should be done for all of the data. If data is really expensive (time, hassle, $$$) but improves predictions so that the incremental return is greater than the cost of the data then by all means get that extra data. Wendi,

If the task at hand is to make predictions why would you worry about covariance of your independent variables (assuming they are not perfectly collinear, in which case you are not adding any additional information)? Won’t this just increase the variance around the parameter estimates? Does that matter if they are not policy (control) variables? Also is this assuming that you are using regression as a data mining method?
There are various non OLS methods that work in very large/infinite feature spaces that might be better for classification and prediction problems than OLS.
I agree with you about rule 5 with one caveat - rather than making this a hard and fast rule the analyst should estimate the marginal benefit of the data vs the cost to acquire - and in fact that should be done for all of the data. If data is really expensive (time, hassle, $$$) but improves predictions so that the incremental return is greater than the cost of the data then by all means get that extra data.

]]> Comment on Predicting their Next Move by Great Blog Posts From 2007 | Digital Alex - Marketing Strategy Blog by Alex Cohen http://coremarkanalytics.com/blog/2007/05/25/predicting-their-next-move/#comment-2067 Great Blog Posts From 2007 | Digital Alex - Marketing Strategy Blog by Alex Cohen Tue, 08 Jan 2008 05:41:34 +0000 http://coremarkanalytics.com/blog/2007/05/25/predicting-their-next-move/#comment-2067 [...] Predicting Their Next Move [...] […] Predicting Their Next Move […]

]]> Comment on Determining your Sample Size by seby kallarakkal http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-1529 seby kallarakkal Tue, 27 Nov 2007 02:49:41 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-1529 Hi Wendi, Great post. Thanks for sharing. I came across your blog few days back and have been catching up on the older posts. Your perspective on using stats for analytics is proving to be really useful. Hi Wendi,
Great post. Thanks for sharing. I came across your blog few days back and have been catching up on the older posts. Your perspective on using stats for analytics is proving to be really useful.

]]> Comment on Comparing Population Proportions – A/B Testing by Sylvester http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-866 Sylvester Wed, 31 Oct 2007 15:36:44 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-866 Wendi, what happens when you've got three tests running: a control and two variations. I can guess what to do with the denominator in your z1 formula, but not the numerator Wendi,

what happens when you’ve got three tests running: a control and two variations. I can guess what to do with the denominator in your z1 formula, but not the numerator

]]> Comment on Determining your Sample Size by Rachel http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-405 Rachel Wed, 26 Sep 2007 13:32:27 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-405 I am still a bit confused on the sample size. Does she need 1700 visitors for each experiment or in total for all three? Right now in total she is at 783 and I'm trying to figure out if she needs ~1000 more or 4300 more. Thanks. I am still a bit confused on the sample size. Does she need 1700 visitors for each experiment or in total for all three? Right now in total she is at 783 and I’m trying to figure out if she needs ~1000 more or 4300 more. Thanks.

]]> Comment on Determining your Sample Size by Erik http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-279 Erik Tue, 28 Aug 2007 23:49:41 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-279 Thanks Wendi, This has been a really helpful article as I get prepared for a testing program. I have a question regarding the number of variants (or 'combinations' as google calls them). I read the above N as 1728 total views, or an average of 576 views for each of the three variants. What happens if we are testing 6 different variants or even 9 (as in some multivariate testing)? If your N is fixed at 1728, the required views per variant would decrease as variants are added. At some point it seems that you would get too few views per variant to have a statistically significant test. It seems to me the number of variants should be factored into the total sample size. In reading about the Power formula, it seems it is based on a 2 sample test. If that's the case should the above formula be multiplied by (r / 2), where 'r' is the number of variants? Hope I'm on the right track with this, if not please let me know, Thanks! Thanks Wendi, This has been a really helpful article as I get prepared for a testing program. I have a question regarding the number of variants (or ‘combinations’ as google calls them).

I read the above N as 1728 total views, or an average of 576 views for each of the three variants. What happens if we are testing 6 different variants or even 9 (as in some multivariate testing)? If your N is fixed at 1728, the required views per variant would decrease as variants are added. At some point it seems that you would get too few views per variant to have a statistically significant test.

It seems to me the number of variants should be factored into the total sample size. In reading about the Power formula, it seems it is based on a 2 sample test. If that’s the case should the above formula be multiplied by (r / 2), where ‘r’ is the number of variants? Hope I’m on the right track with this, if not please let me know,
Thanks!

]]> Comment on Google Analytics – Ups & Downs from an Analyst’s Perspective by Wendi http://coremarkanalytics.com/blog/2007/08/17/google-analytics-%e2%80%93-ups-downs-from-an-analyst%e2%80%99s-perspective/#comment-270 Wendi Mon, 20 Aug 2007 15:43:43 +0000 http://coremarkanalytics.com/blog/2007/08/17/google-analytics-%e2%80%93-ups-downs-from-an-analyst%e2%80%99s-perspective/#comment-270 Marshall - Thanks for all your great feedback and insight. I have felt some of your same frustrations and I think that some companies are not investing into the methodology and they have a false sense of reality in believing they get the package deal when they sign up for GA. Jacques - Thanks for the comment. I would certainly recommend; and have in the past, GA for companies with limited budgets. And as Marshall stated previously, nothing is for free so I like your "price competitiveness" point. Makes sense. Sebastien - I'll need to dig a little deeper and learn how the auto-delivery process works. Thanks for the heads up. However I did know that you can setup various profiles; especially duplicate profiles that don't require any additional coding so that it would open up more places for conversion tracking but I'd like to have it all in one place so I am not having to bounce back and forth between numerous profiles. But great comment. Thanks! Wendi Marshall - Thanks for all your great feedback and insight. I have felt some of your same frustrations and I think that some companies are not investing into the methodology and they have a false sense of reality in believing they get the package deal when they sign up for GA.

Jacques - Thanks for the comment. I would certainly recommend; and have in the past, GA for companies with limited budgets. And as Marshall stated previously, nothing is for free so I like your “price competitiveness” point. Makes sense.

Sebastien - I’ll need to dig a little deeper and learn how the auto-delivery process works. Thanks for the heads up. However I did know that you can setup various profiles; especially duplicate profiles that don’t require any additional coding so that it would open up more places for conversion tracking but I’d like to have it all in one place so I am not having to bounce back and forth between numerous profiles. But great comment.

Thanks!
Wendi

]]> Comment on Google Analytics – Ups & Downs from an Analyst’s Perspective by Sébastien Brodeur http://coremarkanalytics.com/blog/2007/08/17/google-analytics-%e2%80%93-ups-downs-from-an-analyst%e2%80%99s-perspective/#comment-269 Sébastien Brodeur Mon, 20 Aug 2007 15:23:33 +0000 http://coremarkanalytics.com/blog/2007/08/17/google-analytics-%e2%80%93-ups-downs-from-an-analyst%e2%80%99s-perspective/#comment-269 Some of the point in the Down side can be argue. For example: - Setup auto-delivery Auto-delivery can be set. - Limited commerce tracking (only 4 success events can be tracked) Maybe, but you can use multiple profiles to get more than 4 success events. You can have 50 profiles, each having 4 events trackable (4x50 = 200 events). I'm not defending Google Analytics versus a paid solution. Having see what other solutions like Coremetrics and Omniture can offer, I believe that GA is not as "good". But for small site with low budget, GA can be a very good introduction to Web Analytics. I still believe a good analyst using GA is better than no analyst using Visual Science, Coremetrics, Omniture AND WebTrends. Some of the point in the Down side can be argue.

For example:
- Setup auto-delivery
Auto-delivery can be set.

- Limited commerce tracking (only 4 success events can be tracked)
Maybe, but you can use multiple profiles to get more than 4 success events. You can have 50 profiles, each having 4 events trackable (4×50 = 200 events).

I’m not defending Google Analytics versus a paid solution.

Having see what other solutions like Coremetrics and Omniture can offer, I believe that GA is not as “good”. But for small site with low budget, GA can be a very good introduction to Web Analytics.

I still believe a good analyst using GA is better than no analyst using Visual Science, Coremetrics, Omniture AND WebTrends.

]]> Comment on Google Analytics – Ups & Downs from an Analyst’s Perspective by Jacques Warren http://coremarkanalytics.com/blog/2007/08/17/google-analytics-%e2%80%93-ups-downs-from-an-analyst%e2%80%99s-perspective/#comment-243 Jacques Warren Sat, 18 Aug 2007 11:37:55 +0000 http://coremarkanalytics.com/blog/2007/08/17/google-analytics-%e2%80%93-ups-downs-from-an-analyst%e2%80%99s-perspective/#comment-243 Robbin, I don't see what the client base volume has to do with anything, except to say that people rushed to some free stuff from Google. I am not even sure those were all people longing for web analytics, and hoping someone somewhere would finally bring them its blessings for free. It was Google, and we all tend to react in a very pavlovian way when the guys from Mountain View release anything. But, humour aside, you are absolutely right when you say GA ended up changing the Web Analytics world, even if it's not a revolution all those companies were waiting for in the first place. However, I must side with Marshall. Everyday, I see just plain bad use of GA, negligence being the main culprit. I don't know why, but it seems that free tend to stimulate the wrong attitude in the business world. Sure, if you get free GA, and then work with a smart consultant, 10k or 15k will bring you very far, maybe even farther than some more expensive solutions in some cases. But let us stop saying "free", and start talking "price competitiveness". Robbin, I don’t see what the client base volume has to do with anything, except to say that people rushed to some free stuff from Google. I am not even sure those were all people longing for web analytics, and hoping someone somewhere would finally bring them its blessings for free. It was Google, and we all tend to react in a very pavlovian way when the guys from Mountain View release anything. But, humour aside, you are absolutely right when you say GA ended up changing the Web Analytics world, even if it’s not a revolution all those companies were waiting for in the first place.

However, I must side with Marshall. Everyday, I see just plain bad use of GA, negligence being the main culprit. I don’t know why, but it seems that free tend to stimulate the wrong attitude in the business world.

Sure, if you get free GA, and then work with a smart consultant, 10k or 15k will bring you very far, maybe even farther than some more expensive solutions in some cases. But let us stop saying “free”, and start talking “price competitiveness”.

]]>