Comments on: Guided by voices http://www.qwghlm.co.uk/2008/04/21/guided-by-voices/ Because all the other domain names were taken Thu, 18 Feb 2010 22:04:24 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Phil http://www.qwghlm.co.uk/2008/04/21/guided-by-voices/comment-page-1/#comment-2945 Phil Mon, 21 Apr 2008 23:35:34 +0000 http://www.qwghlm.co.uk/blog/?p=1017#comment-2945 Oops - I'm using FP/FN to mean "false positive or false negative", but FP/TP to mean "ratio of FP to TP". Sorry about that. Oops – I’m using FP/FN to mean “false positive or false negative”, but FP/TP to mean “ratio of FP to TP”. Sorry about that.

]]> By: Phil http://www.qwghlm.co.uk/2008/04/21/guided-by-voices/comment-page-1/#comment-2944 Phil Mon, 21 Apr 2008 23:34:26 +0000 http://www.qwghlm.co.uk/blog/?p=1017#comment-2944 You'll be glad to know that, on those figures, the FP/FN rate can't have been anywhere near as high as 36-7%, or far more than 377 people would have been caught in the net (mostly false positives). It's a base rate fallacy thing - the rate of the actual phenomenon in the population interacts with the FP/FN rate. Assuming for simplicity that the FP and FN rates are the same, if you plug in a 'false' rate of 20% (i.e. VRA gets it wrong one in five times) you're forced to conclude that the software's identified 49 true positives and 328 false positives. Lower miss rates give better results, obviously - 5% 'false' gives 308 true positives and 69 FP - but the rate needs to be pretty low to give decent results (12.5% would give approximately 50/50 FP/TP). All this is holding the final figure of 377 constant, but since we know that number it seems reasonable to do so. You’ll be glad to know that, on those figures, the FP/FN rate can’t have been anywhere near as high as 36-7%, or far more than 377 people would have been caught in the net (mostly false positives).

It’s a base rate fallacy thing – the rate of the actual phenomenon in the population interacts with the FP/FN rate. Assuming for simplicity that the FP and FN rates are the same, if you plug in a ‘false’ rate of 20% (i.e. VRA gets it wrong one in five times) you’re forced to conclude that the software’s identified 49 true positives and 328 false positives. Lower miss rates give better results, obviously – 5% ‘false’ gives 308 true positives and 69 FP – but the rate needs to be pretty low to give decent results (12.5% would give approximately 50/50 FP/TP). All this is holding the final figure of 377 constant, but since we know that number it seems reasonable to do so.

]]>