I admit that crowd sourcing science experiments sounds a little risky. Since it's difficult to be fired from a job you do for free, contributors to crowd-sourced science experiments would have less incentive to do precise work or to keep their paradigms and biases from influencing the results they observe and report. Crowd sourcing science could certainly introduce unexpected and undocumented variables. On the other hand, large data sets are valuable. And there may be a side benefit to gathering large data sets from an un-characterized group of real people living real lives in real homes - the data may lead to conclusions that are more directly applicable to the populations we are trying to learn about.
We can test the viability of crowd-sourced
data collection by comparing conclusions drawn by crowd-sourced experimentation to the conclusions drawn from traditionally collected data. One example of crowd-sourced data collection is fuelly.com, where people report actual gas mileage for their vehicles. I did a quick spot check for the V6 Toyota Camry sedan. It would be an interesting exercise to repeat this check for a large group of vehicles.
On fuelly.com, the 119 participating drivers of V6 Toyota Camry sedans submitted fuel economy data for thousands of fill-ups. For this vehicle make and model, the most frequently reported gas mileage is 24 mpg, with a roughly bell curve shaped distribution of reported mileage ranging from a low of 16 mpg to a high of 32 mpg. According to fueleconomy.gov, fuel economy is measured for pre-production prototypes of new cars by the manufacturer using standardized test procedures specified by federal law. The EPA reviews the test results and spot checks 10 - 15% of them. This methodology predicted that the 2011 V6 Toyota Camry sedan would yield 20 mpg city, 29 mpg freeway, and 23 mpg combined.
In this case, the results are pretty close, but Camry drivers on fuelly.com got slightly better mileage than expected based on EPA regulated test data. I saw lots of tips on fuelly.com for getting better gas mileage...I suppose it's possible that users of fuelly.com more frequently exhibit a bias toward maximizing fuel economy than the EPA testing procedures account for.