A Statistical Analysis of Adventure Performance in the Knave 2e Adventure Game Jam

After the conclusion of the rating period for the Knave 2e Adventure Game Jam, I was curious about what factors had the biggest impact on the final rank adventures received. While there are many variables we can look at, in this article we will focus on the three criteria each adventure was rated on, and how many total ratings each adventure received.

Scoring

Each adventure was rated by fellow competitors in the jam on these three factors, on a scale of 1-5:

1. "Writing: Is the adventure original and fun to read?"

2. "Usability: Is the adventure easy to use on the fly?"

3. "Fun: Is the adventure fun to play in an OSR playstyle?"

Competitors could rate any number of other adventures, and any adventures that received below the median number of ratings (in this case 16), had their scores reduced in proportion to how many ratings they were short.

These three ratings were then averaged to give an overall score, and based on this score received their overall rank from 1st place to 118th.

Results

Every entry to the Knave 2e Adventure Game Jam, ordered by rank.

 

The data is noisy, but there is a clear relationship between all three rating factors and how an adventure did overall. Fun and Usability were correlated with rank with an R value of 0.903 (R scales from 0-1, with higher values indicating a stronger correlation), and writing was correlated with an R value of 0.884. For most adventures, usability was the rating that brought their score down the most.

 

Seeing the steep drop-off in scores for the lowest ranked submissions, I was curious how much of a factor lack of ratings had on final results.

 As you can see on the above graph, lack of ratings played an important role for final rank, especially for adventures towards the middle of the pack were competition was the tightest. Of the 118 adventures submitted, 42 of them had their scores lowered for falling below that. Of those 42x adventures,  14x received 15 ratings, 37x received 14, 2x received 13, and 1x received 11.For 4x adventures, a single additional rating could have bumped their rank up by 12 places. Simply rating an adventure at all was an important way to show interest in an adventure.

Adventures that achieved at or above the median number of adventures did not have their scores modified by number of ratings, but there is still some correlation between number of ratings and final rank, with an R value of 0.616, this suggests that on the whole adventures that were well liked were more likely to receive ratings, but number of ratings was not a strong corollary for final rank.

 No surprising conclusions, if you make an adventure that people like and are willing to put the time in to rate, your adventure will perform better. My biggest takeaways were the importance of sharing your adventure enough to hit the median number of ratings, and emphasizing usability. You can see the compiled data here if you're curious.