Incentivizing Internet-Scale Web Mining with Webcoin

This research is supported by the National Science Foundation.

Motivation and Approach

Four major search engines hold a unique position in enabling the use of the Internet, as they alone direct over 98% of Internet users to the content they seek, using proprietary indices. While the contribution of these companies is undeniable, their design is necessarily affected by their economic interests, which may or may not align with those of the users, raising concerns regarding their effect on the availability of information around the globe. One of the key barriers that hinders the proliferation of a larger number of search engines is the daunting, multi-billion-dollar-worth task of crawling and indexing the Exabyte-scale Web space. Hence, this task is carried out by a small number of players capable of committing substantial resources to conduct 24/7 Web crawling and indexing. Despite prior efforts to democratize this space, i.e., promote the use of distributed, objective and unbiased index instead, the use of such distributed systems had not become common. This is mostly due to inferior results associated with small Web indices used by such systems, induced by the lack of economic incentives for participation in distributed Web crawling, or due to the small-scale nature of the communities of users interested in collaborative Web indexing. This project argues that generating incentives to create an up-to-date, organically-generated Web index, widely available to the "masses," is the way to democratize the search engine space. Webcoin's design principles open the doors to incentivizing client participation in other Internet-scale networking tasks beyond Web indexing, and might prove useful in domains beyond networking.