Robo Home | Changes | Preferences | AllPages

There is a point where you have to test your bot. It is not always evident if, after some changes, you have improved it or not. I would like this page to be a collection of strategies and tricks people use to test them. -- Albert

Test tips by Paul Evans "It's all about testing"

(originally posted at http://robocoderepository.com/jive/thread.jsp?forum=31&thread=1231&hilite=true&q=testing&start=0#9202)

Tip1 - Know the average performace of your production bot against 5 or so opponents. To do this give each opponent 300 or so rounds against you to warm up, then run a 5000 round battle and record your percentage of the total score.

Tip2 - Make a habit of recording the percentage score and number of rounds in your notes. Percentage score is (myScore/(myScore+enemyScore))*100%.

Tip3 - If you change your movement ignore the first 300 rounds against an opponent (they will be aiming where the last version of the bot was).

Tip4 - Random chance really messes up the scores. Over 100 rounds the score will be accurate only to about 5%, over 1000 rounds your score will be accurate to about 1.6% (both at 95% confidence).

Tip5 - Make sure your new bot not only beats the old bot - but also improves performance against the other benchmark bots.

Tip6 - For Team and Melee the random variation in score is even more pronounced - don't trust any scores!

The rating difference in the Eternal Rumble can be converted to an expected score percentage thus: Rating Difference Expected Score

The rating differences work for negative ratings - ie a bot with a rating 43 below an opponent would expect to get 46% of the total score.

Tip7 - Check out the Rumble and find out which bots are dragging down your rating (these are the bots that do better against you than their rating would suggest - they can be above and below you in the ratings) - concentrate on improving performace against these bots.

Tip 8 - Make your bot a big pain in the butt to test against. It should take at least half an hour to run 100 rounds against you on Dual 2.8 Ghz processors. Then no one will ever figure out your weakness, because no one's that patient. The more complex your aiming algorithm is and the more turns you skip, the better, just as long as you don't skip any turns in the first 10 rounds (which is all you need in the Rumble). I test against the entire top 10 pretty equally these days, and half of my CPU cycles go to Chameleon 0.83. -- Kawigi

Something tells me that won't be a popular technique... (with your oponents, that it) -- Tango

I have a TaskManager? class that load class implementing Task interfaces. From this I can change some parameters or a Gun or wathever. The I have a stat class that keep various informations about each round. In order to get stable result it load ancient battle data. Then you just take the data in OpenOffice? to see what's the best evolution, rolling average and so on. Usually I run 10 battle of 35 round each to estimate a Task (this may be not enough). The good thing is that you can just let it run for a big time testing various Task. --Synnalagma

Robo Home | Changes | Preferences | AllPages
Edit text of this page | View other revisions
Last edited October 19, 2003 15:42 EST by Synnalagma (diff)