2015/08/28 by Vijay Gadepally, Jeremy Kepner · 1 voice · 24 citations
Computer Science · Engineering · Physics and Astronomy · Social Sciences · #Artificial intelligence #Big data #Complex Network Analysis Techniques #Computer science #Computer security #Data Visualization and Analytics #Data mining #Data modeling #Data science #Database #Distribution (mathematics) #Engineering #Human Mobility and Location-Based Analysis #Key (lock) #Power (physics) #Theoretical computer science #Variety (cybernetics) #Volume (thermodynamics) #Work (physics) #cs.SI
paper · pdf · doi:10.1109/hpec.2015.7322459
5 pages
arxiv created 2015/08/28 · openalex publication_date 2015/09/01 · arxiv updated 2017/01/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
The gap between data production and user ability to access, compute and produce meaningful results calls for tools that address the challenges associated with big data volume, velocity and variety. One of the key hurdles is the inability to methodically remove expected or uninteresting elements from large data sets. This difficulty often wastes valuable researcher and computational time by expending resources on uninteresting parts of data. Social sensors, or sensors which produce data based on human activity, such as Wikipedia, Twitter, and Facebook have an underlying structure which can be thought of as having a Power Law distribution. Such a distribution implies that few nodes generate large amounts of data. In this article, we propose a technique to take an arbitrary dataset and compute a power law distributed background model that bases its parameters on observed statistics. This model can be used to determine the suitability of using a power law or automatically identify high degree nodes for filtering and can be scaled to work with big data.