{"id":5897,"date":"2015-03-18T15:08:23","date_gmt":"2015-03-18T09:38:23","guid":{"rendered":"http:\/\/ivyproschool.com\/blog\/?p=5897"},"modified":"2025-04-01T16:51:13","modified_gmt":"2025-04-01T11:21:13","slug":"why-data-dredging-is-trending","status":"publish","type":"post","link":"https:\/\/ivyproschool.com\/blog\/why-data-dredging-is-trending\/","title":{"rendered":"Why Data Dredging is trending"},"content":{"rendered":"

“If you torture the data long enough, it will confess to anything.” \u00a0\u00a0<\/em>— Ronald Coase, British economist<\/em><\/p>\n

\u00a0<\/em>If you thought DATA was only \u2018mined\u2019 and \u2018extracted\u2019 for analysis, take a look at this frequently used method of \u2018data dredging\u2019.<\/p>\n

As we move over from traditional eyeballing of statistical data to dig deeper into machine based techniques, the entire process of DATA extraction gets more technique based.<\/p>\n

One such DATA extraction practice is analysis of large volumes of data in the quest for ANY possible relationships. An example would be \u201cfishing\u201d in very large datasets to analyse crime clusters without understanding causation. Or say \u201csnooping\u201d into an App user\u2019s habits for finding correlations. \u00a0That is, combing data for patterns without pre-established hypotheses or objectives. Which sounds absurd, but may actually throw-up significant unseen relationships (what does the App user do at lunchtime when in the vicinity of Connaught Place, New Delhi?).<\/p>\n

With the evolution of Big Data a fundamentally different practice of experimental design has evolved. Formerly, the project \/ questions asked would decide what data to collect, for analysis of the same. Now, the low cost of data storage has caused a rethink with all kinds of data being collected first and then searched for significant patterns.<\/p>\n

This practice of \u201cdata dredging\u201d differs from traditional Data Mining practices.<\/p>\n

\"Data<\/a>Data Dredging explained<\/strong><\/p>\n

Where the sample size is not truly representative, there is \u2018confounding\u2019 or \u2018selection bias\u2019, or there exists too many hypotheses for a given dataset, there may occur some highly correlated data that are statistically significant. Whereas, there is no effect between the variables and confidence level\u00a0is .05 (5%). This is a typical case of \u201cdata dredging\u201d with false positive findings, a result of looking at too many possible associations. One way to conquer errors \u00a0of \u201cdata dredging\u201d is being stringent with \u201csignificance\u201d levels, moving to P<0.001 or beyond.<\/p>\n

Applications of Data Dredging<\/strong><\/p>\n