Comments on: streamstats: easier log analysis http://mihasya.com/blog/streamstats-easier-log-analysis/ good things now come in packages of three Sat, 23 Jul 2011 22:40:16 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: mihasya http://mihasya.com/blog/streamstats-easier-log-analysis/comment-page-1/#comment-3689 mihasya Sat, 02 Jan 2010 07:52:34 +0000 http://mihasya.com/blog/?p=288#comment-3689 <p>Ooops never replied to this.</p> <p>The problem is that the data is arbitrary. While everything you suggest is absolutely correct, this ain't the arena for it. This is meant simply as a quick and dirty tool to help parse logs on the spot to detect unusual activity, not long-term analysis. Most of the time, the red highlight is unnecessary anyway - you'll be able to tell immediately which value occurs an unusual number of times, regardless of what the distribution is. I may do away with it entirely, since it is, as you pointed out, not entirely correct.</p> Ooops never replied to this.

The problem is that the data is arbitrary. While everything you suggest is absolutely correct, this ain’t the arena for it. This is meant simply as a quick and dirty tool to help parse logs on the spot to detect unusual activity, not long-term analysis. Most of the time, the red highlight is unnecessary anyway – you’ll be able to tell immediately which value occurs an unusual number of times, regardless of what the distribution is. I may do away with it entirely, since it is, as you pointed out, not entirely correct.

]]> By: David Hall http://mihasya.com/blog/streamstats-easier-log-analysis/comment-page-1/#comment-2909 David Hall Sun, 23 Aug 2009 14:55:33 +0000 http://mihasya.com/blog/?p=288#comment-2909 <p>before you get too far, I would suggest doing some plots just to test what kind of distribution you have. You're making an assumption that mean and standard deviation are meaningful values, when they might not be. I see people do it all the time and draw pretty silly conclusions.</p> <p>Plot yourself a histogram where you are counting how many people make 1 request, 2 requests, etc (bins may need to be wider, depending on how much data you have, but I feel like you can get a lot). If you actually get a Gaussian, then you're okay using those, but you might get something else. In which case, hit the statistics books some more to try to decide what you have. (one hint, if it's bimodal, you probably have the sum of two distributions, say one is normal users, one is power users, in which case, you may be able to split out power users, treat them as a gaussian, then look for outliers 2 or 3 std dev above that mean.)</p> before you get too far, I would suggest doing some plots just to test what kind of distribution you have. You’re making an assumption that mean and standard deviation are meaningful values, when they might not be. I see people do it all the time and draw pretty silly conclusions.

Plot yourself a histogram where you are counting how many people make 1 request, 2 requests, etc (bins may need to be wider, depending on how much data you have, but I feel like you can get a lot). If you actually get a Gaussian, then you’re okay using those, but you might get something else. In which case, hit the statistics books some more to try to decide what you have. (one hint, if it’s bimodal, you probably have the sum of two distributions, say one is normal users, one is power users, in which case, you may be able to split out power users, treat them as a gaussian, then look for outliers 2 or 3 std dev above that mean.)

]]>