2012/09/11 by Jimmy Lin, Lin, Jimmy · 2 voices · 2 citations
Business, Management and Accounting · Computer Science · Decision Sciences · Engineering · #Analytics #Artificial intelligence #Big Data and Business Intelligence #Big data #Computer science #Data Quality and Management #Data mining #Data science #Engineering #GRASP #Hammer #Nail (fastener) #Perspective (graphical) #Programming language #Programming paradigm #Scale (ratio) #Simple (philosophy) #Software #Software engineering #cs.DB #cs.DC
paper · pdf · doi:10.48550/arxiv.1209.2191
published in arXiv (Cornell University) (Cornell University)
arxiv created 2012/09/11 · openalex publication_date 2012/09/11 · arxiv updated 2012/09/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Hadoop is currently the large-scale data analysis "hammer" of choice, but there exist classes of algorithms that aren't "nails", in the sense that they are not particularly amenable to the MapReduce programming model. To address this, researchers have proposed MapReduce extensions or alternative programming models in which these algorithms can be elegantly expressed. This essay espouses a very different position: that MapReduce is "good enough", and that instead of trying to invent screwdrivers, we should simply get rid of everything that's not a nail. To be more specific, much discussion in the literature surrounds the fact that iterative algorithms are a poor fit for MapReduce: the simple solution is to find alternative non-iterative algorithms that solve the same problem. This essay captures my personal experiences as an academic researcher as well as a software engineer in a "real-world" production analytics environment. From this combined perspective I reflect on the current state and future of "big data" research.