[Home]TargetingChallenge2K6/PreChat

Robo Home | TargetingChallenge2K6 | Changes | Preferences | AllPages

This was the discussion that led up to TargetingChallenge2K6. (I had to remove some links to get this to post, though. I might add them back later.)


Tentative candidates: (not yet finalized)

WaveSurfers:

Non-surfers:

Bots: TC2K6_reference_bots.zip (291 kb)


 *Tango coughs*

Well, I'm not the most qualified to comment on how to benchmark a gun, but I think CassiusClay and SandboxDT 1.91 would be the first two I'd nominate. CassiusClay is a great WaveSurfer that executes pretty quickly, and DT 1.91 is some really good RandomMovement. I'm working on the gun primarily for Dookious now, so I may have some better suggestions once I make some progress there. :) -- Voidious

The problem is that there are not many good distinctive movements. We can throw in some more WaveSurfers, as they are the top-movers nowadays. Some special versions have to be made though, because SandboxDT and Tron/Shadow are the only ones I know of that you can set in challenge/reference mode without altering the code. -- GrubbmGait

Well, it's true that nearly all the best movers are WaveSurfers these days, but getting to the top of the rankings also includes really smacking down every other type of movement, too. Perhaps an even mix of WaveSurfers and other movements? I think filling half the slots with WaveSurfers would be no problem, as many are open source - there's CassiusClay, SilverSurfer, RaikoMX, PowerHouse, Chalk, Dookious, Pugilist... (I could make the special versions, that's no problem.) PulsarMax also has a .properties file, and you mentioned Shadow.

As for the non-surfers... We could keep DT 1.91 and Cigaret 1.31 from the first TC, as they generally got the lowest scores against them; MakoHT from the first TC had low scores, too. Doesn't GrubbmGrb have some pretty good movement for a non-surfer? Well, I guess we could look through the OpenSource by Movement type list to get some more ideas. -- Voidious

I can think of a few other interesting, open source movements:

I second the vote for GrubbmGrb. He's the undisputed master of multiple choice strategies (and one of my favorite bots). Regarding Chalk, I'd be more than willing to make a TC version; though there's nothing particularily unique about his movement. -- Corbos

I'll take a look at those tanks you listed when I get home, Corbos, but a couple thoughts for now... First, I really think using tanks that are fairly quick is a very good thing, and I think the overall quality of the challenge wouldn't be compromised by excluding tanks that are much slower. Also, I tend to think that two or three of the WaveSurfers should be among the most excellent surfers, and the others could be unique versions or just slightly closer to the middle of the pack. -- Voidious

Oh, and I started a tentative list at the top of the page with the two candidates I feel most strongly about ;) Feel free to add to it. -- Voidious

I modified and trimmed down Ugluk for an adaptive movement entry, pedersen.Butterfly, that you may find useful. As it is hit it cycles through various movement systems until it decides on the best one to use. If the initial choice gets beaten badly enough it will switch again. It is similar to the Musashi Trick, but more advanced. -- Martin Alan Pedersen

A TC variant of GrubbmGrb is easily made, only set the 'MC_flag' and recompile. I've tried pedersen.Butterfly and it seems quite unique and versatile. One thing has to be solved though, it is not allowed to intentionaly ram disabled opponents. -- GrubbmGait

Well, a "movement challenge" mode really shouldn't do that, but it actually shouldn't throw off the scores of a targeting challenge, right? The TC only measures bullet damage... -- Voidious

@Voidious: I agree with your comments about SlowBots. Still, you've got to admit Tile Coding is cool! :) --Corbos

Well, it certainly sounds cool, and I've wondered about it each time I've read the description ;) Now I simply have to read up on it, I guess! -- Voidious

I've tossed the source code for pedersen.Butterfly (kinda) but I can recreate it. It just won't be as 'small' of code size. I'd think ramming would be a boon since it would end matches faster once they were unwinnable and give a better spread on the scores against it, but I can disable the ramming if it is beneficial to the challenge. -- Martin Alan Pedersen

Heh, I thought the same thing about the ramming speeding things up... I guess it wouldn't make a huge difference when it's running minimized, though. The ramming shouldn't affect TC scores, but it would greatly affect any MC scores that would help guage the movement of that bot in a MovementChallenge. I'm not sure what I think of your Butterfly as a TC candidate, as Dookious got a near-perfect score against it :-\, but I would be interested to know how other tanks do against it first. -- Voidious

I didn't expect Dookious to crush the bot, since you don't annihilate me in the Rumble (though you do win by better than 60%). If it's not challenging enough then you can scrap it. -- Martin

You are right about the ramming, it does not influence the score. Ramming is not allowed in the movement-challenge though, therefor my (wrongly) remark. The naming of my flags is based upon the fact that normally I am a contender, not a reference-bot. At first GrubbmGrb could nail Butterfly easily, but after a round of 10-15 it could hardly hit it anymore. -- GrubbmGait

One thing we should keep in mind: a good TC will benefit all Robocoders. A novice might get frustrated if they don't see improvement against bots like Shadow and CassiusClay (and let's face it, improvement is a slow process against these monsters). I haven't tried Butterfly but it could provide a strong challenge that isn't as intimidating as the top-10. A wide range of styles and rankings would be nice. Thoughts? --Corbos

Yes, it is a good idea to represent various rankings in the TC. After all, your gun needs to be good against everyone, not just the top bots, to excel in the RR. --wcsv

Yep, I definitely agree. I guess I got the idea in my head that this challenge should be a lot *harder* (lower average scores) than the first one, but the truth is that it should be a consistent and accurate benchmark, first and foremost.

I still have a lot of trouble getting good scores against Cigaret 1.31, from the original TargetingChallenge, so I think that's another non-surfer that might be good to add. What do you guys think of the updated list of bots at the top? And what do you think of the the "2-3 superb surfers, 2-3 unique surfers, 5 non-surfers" distribution? As good as Toad's MC score is, he's kind of slow to execute, so I might want to skip him on the surfer selection. -- Voidious

If no one objects, why don't we ask Voidious to finish the list and set things up? That's probably the best way to avoid the fates of TargetingChallenge2K4 and TargetingChallenge2K5. ;) --Corbos

@Voidious: Do you have the time and are you willing to set up the TargetingChallenge2K6? If there is anything I can do to support you, please let me know (except bringing coffee). -- GrubbmGait

Hey guys, I'd be honored to put together the list for TC2K6. I'll try and have it together within the next week or so, and hopefully you will all approve of it. :) I'm thinking 2 top-notch surfers, 3 good or unique surfers, and 5 non-surfers. I'll be sure to test sufficiently to try and get it as well-rounded as possible. -- Voidious


If anyone's interested, I will upload some of my RoboLeague test results to:

http://www.dijitari.com/void/robocode/tc2k6

I put up the results of my first test just now. I'm going to use a dev Dookious, whose gun is pretty good against WaveSurfers, and a couple other guns to find the WaveSurfers. (This "dev" Dookious is just 0.61 plus another anti-surfing gun in the VG array.) Anyway, I won't comment with every update, but I will upload results there as they come about if anybody's curious. For reference, this gun gets about 89.5% in the original TargetingChallenge.

Luckily, all of these bot authors have relatively clean code, so it was easy to get this setup quickly. I've ruled out Dookious as a candidate for now, as I'm a little biased, of course, and it seemed odd to test his movement against his own gun. And if anyone objects to their tank being a reference, just give a holler. -- Voidious

I pit Ugluk vs. Ugluk on occasion. Ugluk's bearing offset gun is best against Ugluk's movement, which I think is best, with an 18% hit rate, which I'd prefer to reduce. -- Martin

Well, guys, I got a tentative list going sooner than expected. It's at the top of the page. (All these TC bots can be found here.) Also, I've uploaded the Excel spreadsheet I was using to tabulate my test results, and the individual RoboLeague files are in that same directory. I want to run some more tests and contemplate the bot list some more before finalizing it, but I feel pretty good about this set of bots.

For the WaveSurfers, part of my selection process was their execution speed; Shadow was about 60% as fast as Cyanide and CassiusClay, and Dookious was a bit slower than that. Tigger turned out to be pretty quick once I completely disabled his gun. Let me know what you guys think, or if you don't want your tank as part of the reference set. -- Voidious

It's just my opinion; but I'd say set it up, post the bots, and let's get busy. Thanks Voidious. --Corbos

I'll give it at least until the end of today for anyone to object, and I'll try and make sure there's nothing wrong with this set of bots. Hopefully, by tomorrow we'll have a new TargetingChallenge! -- Voidious


Ok, I ran some tests to examine the accuracy of the new TC's test scores with this set of bots. Take a look, the guns are along the top:

500 rounds Raiko 0.43 Dookious 0.611CC 1.9996Shadow 3.64
CassiusClay 59.412 67.708 57.06269.042
RaikoMX 51.146 81.466 68.24671.896
Chalk 78.140 88.190 94.06891.954
Cyanide 68.004 78.598 78.28069.340
Tigger 74.982 84.274 80.85885.344
Cigaret 84.636 78.958 89.71483.716
GrubbmGrb 91.012 85.890 92.45891.264
Butterfly 98.848 99.814 99.39098.794
RandomMovementBot 90.760 87.246 92.92487.504
NanoSatan Mu 94.436 94.818 94.81893.732
Score 79.138 84.696 84.78284.259
Original TC 92.47 89.5 94.19 ?

Clearly, both Raiko and CassiusClay do a lot better against non-surfers than Dookious, while Dookious does significantly better against WaveSurfers. The RoboRumble clearly gives a lot more rating points to a gun that's good against non-surfers. (Edit: Due to the volume, I mean, not to imply that Dookious is as good or better than CC and Raiko!) Now, personally, I think that the above set of bots is a good benchmark of a gun's actual ability; but it clearly doesn't reflect the rating a gun would give in the RoboRumble.

What do you guys think? Is the above a reasonable result, with the understanding that the non-surfer scores will affect your RR rating more? Or should there be less WaveSurfers in the reference set? -- Voidious

I added Shadow 3.64's scores to the table above. I'm really thinking that the emphasis on anti-surfer targeting is a bit too strong, and that maybe I should remove RaikoMX and/or Cyanide, and add some more distinctive non-surfing movements. Or maybe we just count the non-surfing scores double? I dunno! I'd appreciate any input. -- Voidious

Ok, I think replacing RaikoMX with FloodMini gives a more balanced view, and Kawigi's SandboxFlattener is near the top of the list of movements that I considered but initially didn't put in. Here's a look at the new scores, and I think it seems a bit closer to "reality" in the RoboRumble:

500 roundsRaiko 0.43Dookious 0.611CC 1.9996Shadow 3.64LinearTargeting
CassiusClay59.41267.70857.06269.0423.592
Chalk78.14088.19094.06891.95411.920
Cyanide68.00478.59878.28069.3403.592
Tigger74.98284.27480.85885.34430.544
Anti-Surfer Avg70.13579.69377.56778.92012.412
      
FloodMini89.47289.47893.70887.18670.044
Cigaret84.63678.95889.71483.71656.506
GrubbmGrb91.01285.89092.45891.2645.920
Butterfly98.84899.81499.39098.79434.048
RandomMovementBot90.76087.24692.92487.50486.058
DuelistMicro94.92492.08496.6594.75267.236
Non-surfer Avg91.60988.91294.14190.53653.302
      
Unweighted Score83.01985.22487.51185.89036.946
Weighted Score*84.45185.83988.61686.66439.672

The weighted score is (((2 * non-surfer avg) + surfer avg) / 3), and was just included for reference; I don't think it's necessary. What do you guys think of this list? The edited reference bots can be found in TC2K6_beta2.zip (287 kb). -- Voidious

Looks great to me. FloodMini is a nice addition to the squad; its flat movement should give some good GF guns a run for their money. By the way, Thanks for setting this up , Voidious. It would take me a *very* long time to run the test matches that you did. --wcsv

Great work Voidious! Looks very good to me. -- ABC

NanoSatan Mu has a problem, after approx 200 rounds it generates a ArrayIndexOutOfBoundsException? at line 37. Either the source has to be adapted to eliminate the use of that 'ev'-variable, or another bot has to be chosen. I do like it simple but not so simple to hit movement though. Also how fast you kill it is important, because it hits the wall quite often. The choice for FloodMini is indeed a good one, thanks for your effort! -- GrubbmGait

Thanks so much for doing this. It looks great! --Corbos

Nice spot on the NanoSatan problem, Grubbm. That variable was actually part of the targeting, so it was easily commented out. I was aware that it hit the walls a bit, and at the time I tested it I though that was OK, but now that does seem kinda stupid. I'm looking at RaikoMicro to replace NanoSatan. I should be done testing that within an hour or so, and if that works out, I'll post all the appropriate sub-pages here and move this chatter to /OldChat? or something. Thanks for the input, guys! I look forward to working on my gun again :) -- Voidious

Alright, I actually went with DuelistMicro as the replacement for NanoSatan, as RaikoMicro was very similar to FloodMini and Cigaret, and even kinda similar to RandomMovementBot. I updated the table above, and it's looking good. I'm going to make all the TC sub-pages, and move this discussion once I'm done. I'll post the current results that I have, too. Let's rock! :) -- Voidious

Damn you guys! I get thousands of ideas on what to do on CC now. =) Anyway, at least I see that I have some reading up to do on the latest developments. Truly cool this new TC thing! -- PEZ

I could quickly throw together some guns that represent dead-on, linear (already available), circular, linear mean, circular mean, partial linear, fixed forward offset (assumes linear for x distance), and my bearing offset gun (which is no great shakes) .. but .. I can't make them fire 3.0 shots without some serious refactoring. I usually fire between 2.5 and 3.0, adjusted to hit dead center of their predicted position. The dead-on gun could be easily tweaked to fire 3.0 without affecting its accuracy, but it would definitely affect the accuracy of my other guns. It's part of the reason I don't expect Ugluk to compete either. -- Martin

Hmm... you mean for the MovementChallenge2K6? I'm pretty sure we want to stick with the 3.0-only rule, but it's kind of you to offer, anyway ;) -- Voidious

Well, it makes more sense for the movement challenge, but I saw that you had a linear targeting gun as a contender in the targeting challenge, which is what I was thinking of. I'm rafactoring some of my code to better support such a challenge, but I'll keep it for internal testing for now. -- Martin

Oooh, now I see what you mean. Well, since it's part of the "rules" of the TC, we should probably stick to them for all contenders in the results, too. If you guys think it's worth while, I could run some of my versions of simple targeters to get the TC scores; I think I've got everything Martin mentioned except for "fixed forward offset", and my "average bearing offset" is highly untested and fully unsegmented, but should work other than that. -- Voidious

I have seen another (potential) problem, this time with Butterfly. It seems that it starts its rammingmode when the last bullet is still in the air and then sometimes is hit by it head-on. I do not know if this is serious enough to update Butterfly, but it would partly explain its high vulnerability for HeadOnTargeting. -- GrubbmGait

Interesting observation. I'll certainly look into it. But Butterfly shouldn't be ramming at all when it is set to be a TargetingChallenge reference bot, does it really? -- PEZ

It does not matter for the score and it is a way to shorten the battles by eliminating the timeout. -- GrubbmGait

PEZ, another "Butterfly" is a non-firing version of Ugluk, hence your confusion. There are essentially two "Butterflies" in the TC2K6 ;) Intresting point about BFly, Grubbm, I'll take a look. As long as it produces consistent scores, it should be OK to leave it in. -- Voidious

If anyone's interested, there are still a whole lot of test results that I uploaded to this directory; and there is also an Excel workbook (with 3 sheets) with most of it nicely organized. A lot of interesting info there. -- Voidious

Aha, good. I don't mind a bot named Butterfly. I haven't released a bot with that name, it just happens to be the name of my movement class. And since it is pluggable it is a quite public name. -- PEZ

Ok.. here is an improved (yet possibly still inadequate) version of pedersen.Butterfly (2.0) that has two fewer movement option (I had 'mirroring' in by mistake), ramming removed, and a new movement stats decay feature which may help balance out the 35 vs. 500 rounds effectiveness. I'm presently running a stationary (but energy managing) Ugluk against it to see how I do. -- Martin

I think it would be interesting to have a Droid that does wave surfing, which would be possible given the conditions of the challenge. You know the rounds a bullet is fired due to the fixed bulet power and gun cooling rate, and the first bullet that strikes you each round gives you enough information to determine where your opponent is. If they miss you with the first bullet and hit with the second you are still ahead, energy wise. -- Martin

Yea, that would bee an interesting thing :D But as you already said it would make no difference exept the energy bonus. IŽll tune DoBoh a bit so that he does wavesurfing against bots using static bullet power. Thanks for this hint! --Krabb


Robo Home | TargetingChallenge2K6 | Changes | Preferences | AllPages
Edit text of this page | View other revisions
Last edited January 17, 2006 19:27 EST by Krabb (diff)
Search: