Makes determining if a screen is successful pretty difficult!

Indeed.

I think of it this way: it's a curve of rising probability of actual "value added". You are never 100% sure...but you can be pretty darned sure at some point--functionally certain. The Buffett example.

Different metrics of success will give different curves of rising probability. The interesting work is to find the metric(s) which give the highest certainty that something "works" (topmost curve) with any given amount of out-of-sample data.

And realizing that failure is much easier to prove than success. If a strategy noticeably underperforms its universe in every 6 month interval for (say) four years post discovery, the chances of it having long run value approach zero pretty quickly. Whereas I believe four years of pretty good results would not give nearly as much certainty in the other direction.

Jim