Roger Cicala writes why it’s impossible to design the perfect lens test, but we can come close:
Lab testing, with its numbers, gives us nice, quick overviews of lens performance. It’s useful for lens reviews so that you can compare one lens to another. It’s useful for people like me who have to test lenses to make sure the optics are OK, since it eliminates some of the human variability that comes with looking at images of a test chart.
But each type of lab test has its own strengths and weaknesses that nobody ever talks about. This is important if we’re going to compare several different reviews of a lens, because we should have some idea of what the reviewers are actually analyzing. Like every scientific test, if you don’t have a grasp of the testing methods being used, you can’t possibly understand the results.
Basically there are two types of lab testing used for reviews: computerized target analysis and optical bench testing.
Computerized target analysis, using either Imatest (used by Lensrentals.com, Photozone, Lenstip, and others) or DxO Analytics (used by DPReview, DxOMark, SLRGear), are the most commonly used lab tests. Both programs work by taking carefully aligned and lighted photographs of a specific test chart which are then analyzed by a computer program. The program then analyzes the file and determines things like resolution, distortion, vignetting, etc.
A good Imatest lab costs $10,000 to $15,000. (DxO is a lot more expensive – it’s sold as a complete package, including the testing room and lighting). But an optical bench costs from $50,000 to $350,000 depending upon its capabilities.
Still, an optical bench gives us two huge advantages over target analysis:
It tests the lens at infinity focus, not close up, which can be more real-world.
It eliminates the variable of the camera body, so you can compare, for example, a Leica lens to a Nikon lens.
It has another advantage for someone like me who has to check a lot of lenses; it’s very automated. Instead of multiple images taken of an Imatest target with careful alignment and manual focus bracketing, you mount the lens, push a button, and the machine does the rest.
Here’s what an optical bench looks like:
Read the full article here, including reasons to never trust a 24mm or wider lens test and more.