Crowdvoting: How Elo Limits Disruption

Crowdvoting

Crowdvoting is an extremely useful tool for businesses. Imagine you are the CEO of a company with 5,000 employees. In your mind you have 20 goals for your company for the next year. You have a keen sense for the most important ones, but do your 5,000 employees share your priority? Ideally, you’d have all 5,000 employees rank all 20 goals. Also, ideally, you’d have an answer in hours. How can you possibly do that?

The brute force method would be to create a spreadsheet, and email it to everyone. Each employee would download the file, open it, cut and paste while they evaluate and rank 20 goals, re-save, re-attach and email it back. You would get a handy excel/python guru to consolidate the 5,000 files and give you an answer.

The pain with brute force is asking 5,000 employees to take 10 minutes to rank the goals.  That’s 21 person-weeks of effort. Among our clients, that kind of disruption is simply not allowable.

Crowdvoting and Elo

Enter Elo. Elo is a system for rating the relative strength of players in a game. It was originally developed for ranking chess players. But Elo can be applied just as easily to ideas, or in our case, corporate goals. The value of Elo is two-fold. First, only two goals are presented at a time, so the employee has a much simpler question to answer: “Which goal is more important?” Second, Elo uses every comparison available to rank goals, so even if an employee compares only two goals, he or she is contributing to the overall rank.  Because Elo can predict the final ranking with minimal data, employees don’t need to spend a lot of time ranking.

How Much Time Savings?

We’ve conducted internal studies where one group stack ranks in excel, and one group uses an Elo-enabled app. For a task where 10 items must be ranked, Elo takes about 1/5 the time. For a task with 50 items, Elo takes about 1/10 the time. In some of our customer projects, users rank hundreds of items. We don’t think this would be possible without Elo or another predictive ranking algorithm.

Conclusion

If you want a large crowd to rank a large number of ideas, brute force is not an option. You will need a way for small brush strokes from each participant to be combined to paint a clear overall picture. A pairwise system that uses Elo is a great solution.