|
<Snipped from David Alves/PhoenixChallenge> some preliminary testing results can be found here: /Results Voting for bots can be found here: /Voting Might as well bring this up now ... should we start planning a Targeting Challenge 2K7? I've been embarassed by my TC2K6 reference bot for nearly a year now. -- Martin Yeah, I have been pondering the TC2K7 idea for a while. I don't think it needs to be "harder" than the current one, but I think it could be a lot more accurate as a benchmark. I guess I should save some of that discussion for the TargetingChallenge2K7 page... -- Voidious |
|
/ReferenceBots - /HowTo - /FastLearning - /Results - /ResultsFastLearning - /ResultsChat - /PreChat - /Voting |
|
The only constructive thought I've had, other than if TC2K7 is necessary, is maybe separating out the surfers into an AntiSurfer sub-challenge, since it's such a small effect on RR score but still interesting. -- Voidious I have given 2k7 some thought, and I think we should have a surfer challenge. We could cut it up so we have 3 parts of the surfer challenge:
The reason for the 3rd being that some bots have alot of bias of turning on thier flatteners, and as your gun gets good vs unadaptive bots it should get worse vs the flatteners. We could also just cut out number 2 from that list. --Chase-san We might just want to have bots with flatteners permanently on or bots with no flatteners at all. Otherwise the score might change a lot depending on whether or not it was enabled, and when. How about this as a list? 5 surfers: * 2 expert surfers, e.g. Dookious, Phoenix, Shadow or CassiusClay * 2 intermediate surfers, e.g. WaveSerpent, Engineer, PhoenixOS?, Komarious * 1 beginning surfer, maybe a version of BasicSurfer updated to stay far away? 5 non-surfers: * 2 expert random movers, e.g. Cigaret or Raiko (others?) * 2 expert stop-n-go / multimode bots, e.g. GrubbmGrb or Ugluk * 1 beginning random mover, e.g. FloodMini, DuelistMicro I think we should try to pick bots by people that are active, makes it more interesting. Although we don't have active people in every category I listed above. =P --David Alves The surfer breakdown you have there looks good, and actually a lot like my general thoughts for TC2K6. However, I don't think Tigger was quite good / stable enough (his WSC scores are not great). I personally like trying to hit Shadow, but I'm of the opinion that the reference bots should be fast since we will be running so many battles against them. We should be able to find great surfing movements that are also fast. For what it's worth, I hope Dooki will be much faster when I finish the rewrite, but if it isn't I'd recommend still leaving it out... There is value in having a DC surfer in there, though, so maybe we can check out the speed of both Chalk and Shadow... What about having 5 surfers in an AntiSurfer sub-challenge, and 10 non-surfers in the non-adaptive sub-challenge? Not married to the idea, just a thought. Even with 4 surfers, and 2 expert ones, I feel like TC2K6 has too much emphasis on surfing to accurately reflect the rumble. I like Komarious as a reference bot =) I'm really proud of that quick little surfer! * Actually, Komarious may be closer to a beginner surfer than an intermediate surfer... It's not as far from BasicSurfer as you might think. -- Voidious About the flattener, I'm on the fence about it. Dookious does much better against most guns with its flattener off, but flattening also helps a lot against top guns. It'd be a much easier target for most tanks if we just left the flattener on all the time. -- Voidious I pointed out we could just drop number 2 in my list ;). Actually, my bot Velshea could fill the beginner surfer requirement (as it is and i'm still a beginner at making them). Its not alot better then BasicSurfer, and its movement in the CurveFlatteningChallenge isn't super as Komarious' is. I would vote for Shadow as a expert surfer and Komarious as a intermediate. I like the idea of Engineer in there with his neural network based surfing, and Dookious with the flattener on might even the scores a bit. Those are my ideas. -- Chase-san I guess we can cross WaveSerpent off the "intermediate surfers" list =) I know a surfer can get a lot more basic than Komarious, but I'm not sure we should have anything more basic than it as a reference bot. Very few surfers stay that simple, so I'm not sure it's much worth it to tune your gun against something like BasicSurfer. You don't need a decent AntiSurfer gun to hit a bot that doesn't roll its stats like most surfers. Compared to every other functional surfer in the rumble, Komari's movement is still very basic - she's got an extremely competent gun (see TargetingChallenge scores) and still barely breaks 2K. What do you guys think of having surfers and non-surfers as sub-challenges? I still like the sound of 10 non-surfers and 5 surfers... -- Voidious I think 10 non surfers is too many. 10 bots total in the challenge is a nice round number. Besides, it's not supposed to be a good predictor of rumble score... it's just supposed to be a challenge! Challenges should be hard, and that means (to me anyway) that we should use bots that are hard to hit, which means not too many non-surfers. --David Alves Hmm, if you say so =) I said something very similar about reflecting RR rating when we were putting together TC2K6, actually. I'm OK with 5 surfers and 5 non-surfers then. I still think it'd be good to also list separate indices for surfers and non-surfers, since the non-surfers score has so much more impact on RR rating. -- Voidious We could do it like the MC2K6? --David Alves Yeah, or just leave the format like 2K6 and add the sub-indices to the results. I kinda do this on the Dookious/DookiSaber page even for the 2K6 results. For TC2K6, after discussing a bit, it was basically put in my hands to put together the final list; I've still got a lot of interesting data from all the tests I ran for that. In particular, there this Excel workbook (3 sheets): [tc2k6.xls]. And a bunch more RoboLeague results in [that directory]. Shadow (expert surfer) and Komarious (basic surfer) would be my top candidates among the surfers; Cigaret and DuelistMicro are probably my top pics among the non-surfers. Not to toot my own horn, I think it'd be great to have Phoenix, Dookious, or Ascendant in there, since they are probably the best movements in the rumble. Engineer is really interesting, too, but I'd have to check how fast it is to execute. -- Voidious (editing conflict with Voidious) I would perfer a benchmark, and since there is such a large diversity of movement types I say we have no more then 2 or 3 surfers at best. My bot, is moot, since as a benchmark, wouldn't do well enough. So I also vote Komarious and Shadow. I also vote for Lacrimas which is based off Cigaret and uses a more advanced movement then Cigaret, should be used along with perhaps Tron 3 and its right angle movement, Vyper or GrubbmGrb would be my choice for multi-mode. DuelistMicro for its timer based movement. -- Chase-san I dunno, I like the idea of 5 surfers, personally. If you want to know how your gun will work in terms of RoboRumble rating points, just focus on the non-surfers like with TC2K6. I'm also starting to think maybe we should limit the number of bots repeated from the other two TC's - it's not like you can't run those tests, too. Just as food for thought, I'll throw out a possible list that comes to mind right now... * Shadow - Expert Surfer. We all want to hit Shadow and he's super hard to hit consistently. * Another expert surfer - Not sure, but I'd like Dookious, Phoenix, or Ascendant. I guess I think Dookious would be best because it's open source and probably the top rumble movement, but it is also a bit slow like Shadow. I think I can speed it up considerably in the coming weeks, though. Dooki's flattener is pretty well suited for enabling / disabling in this mode of operation, too. (Because I used to be addicted to MovementChallenge tests...) * Engineer - Intermediate (though almost Expert, probably) surfer. Very unique implementation and adds good variety, just gotta check its execution speed. * PhoenixOS? - Intermediate surfer, held its top 10 position for a long time after last update. Open source is good for people to learn from. * Komarious - Basic surfer. Functional, bug free, very fast to execute, but super simple and doesn't segment on much. Still relatively tough to hit - Dookious got ~86 scores against it last time I checked. * Cigaret - RandomMovement. In both other TC's, but still super hard to hit. * Raiko (or RaikoMicro?) - RandomMovement. A really good non-surfing movement. Open source. Not active, but not too many random movers coming out these days... Maybe Tityus instead? * Vyper - Top notch multi-mode movement. Open source. * Another multi-mode bot - GrubbmGrb is a great reference bot, IMO, but would be nice to get something fresh. Two of Swiffer's four modes are mirror-ish, so it wouldn't be all that useful in a TC. What about SpareParts? Ugluk could be a good choice, and I think its movement is pretty quick to execute, but it's closed source and it's kinda (but not quite) like a repeat of Butterfly. * Charo - Micro random mover. Haven't looked closely at it for a while, but Corbos is active and it's only a few places ahead of DuelistMicro in the micro rankings. Ok, that turned out to be a longer post than expected, but at least it gives us something to talk about ;) I definitely want to hear what some people other than me / Chase-san / David have to say about TC2K7... -- Voidious One thing I like about that list is that there aren't too many reactive movers. There are quite a few in the rumble, but I think a lot of the lower level bots have non reactive movement. I remember GresSuffurd gained a couple points in the rumble, but lost TC2K6 scores by adding results from non-firing waves. Having both Vyper and Raiko in is kind of reduntant because both have flat, orbital, RandomMovement with very similar movement profiles. I'm also unsure of whether Vyper and GrubbmGrb really qualify as multi-mode because they only have one real movement type (although they would be good challenge bots anyway). The other modes only turn on against simple targeters and won't really appear in the challenge. Besides that, this list looks great to me. -- Kev Actually, I just remembered Vyper's movement mode changes whenever it dies until it reaches RandomMovement. When I tested it against the wave surfing challenge bots, Vyper got stuck once and switched to its RandomMovement anyway. It would be better to use Thorn's movement instead (which is virtually identical, but doesn't have this problem). -- Kev Ah, I didn't realize that Vyper and GrubbmGrb were basically RandomMovement against a learning gun. I do think it'd be cool to have one truly MultiMode movement in there. Hmm... -- Voidious The only true MultiMode bots I can think of that are also quite recent are Ugluk/Grishnakh and ForceMajeure, alas both are not open source. Smoke can be a replacement for Cigaret, although very similar. Vyper does seem to do better against GF-guns than GrubbmGrb, so maybe is the better choice. Other than -- GrubbmGait I personally wouldn't want to many surfers for the reasons that I don't want it being like 2k6 again where you can get a decent to good score just with a anti-surfer gun as I have, I like a large selection, but I am willing to accept 5 surfers if we are lacking vairation in normal movers. In the non-surfing division I personally like Raiko, Cigaret (since the other I looked at was rather bad), Thorn, and perhaps Charo as we need a random mover(or atleast one that doesn't circle). I still like the idea of having Tron 3 in there, according to its page it doesn't surf, but its still pretty high in the RR rankings. As for a easy hit bot, so we know if our guns have a major problem, so one of the older bots like Krabby and Fractal. --Chase-san If we divide up the challenge like the MovementChallenge2K6, the number of surfers won't matter. I also agree with having Tron in there (but shouldn't it be version 2, since version 3 surfs?) so we have a non-orbital mover in there. There's a collection of quite a few old movement experiments on my computer and I could pretty quickly make a MultiMode bot with orbital random movement, go to random movement, sand box flattener movement, the musashi trick, and StopNGo movement if anyone is interested. -- Kev Tron 3 does move at right angles, I did some observing, it might use a surflike technique to not move into danger zones, but isn't really surfing. Version 3 doesn't surf, but Shadow, which is based off tron 3 does. So I personally think tron 3 is okay. --Chase-san No, it surfs... Quote from Tron page:
-- Voidious Really?? I watched Tron 3.11, turning only at right angles and moving forward with slight wiggling motions. Didn't look like any sort of surfing I had seen. -- Chase-san |
|
Welcome to the TargetingChallenge2K7. It's an attempt at measuring pure gun performance in 2007 :). The rules are simple. |
|
Yes, Tron 3 is a wavesurfer. The future position predictor is not 100% acurate, though, it doesn't take the wiggling into account. -- ABC |
|
* The Challenger must remain stationary all the time. * The Challenger must try to fire power 3 bullets when it can. ** Which means no energy management whatsoever... ** Challenger is only allowed to feed the fire() method(s) the constant 3. This often results in at least one bullet per round fired with energy lower than 3 though. * There are 10 /ReferenceBots, none of which fires, few of which does any ramming. * The Challenger meet each reference bot in battle over 500 rounds. * The Challenger's score against each reference bot is calculated like "challenger_bullet_damage / rounds" ** Average the scores against each reference bot to get the total score. |
|
Oh wow, abc, I wasn't doupting what Viodious said, just explaining what I said. --Chase-san |
|
Please post your bots' results on the /Results page. This is made very easy by using RoboLeague and the TC Calculator. |
|
I also would like to avoid having 5 orbital random movers for the non-surfers. Tron 2 seems like it might be a good reference bot. It was in the original TargetingChallenge. @Kev: I think it would be cool to throw together a MultiMode bot for the challenge. I'm also going to take a look at SpareParts to see if that might be worth considering. -- Voidious |
|
Results chatting is best carried out on the /ResultsChat page. |
|
I think the problem is that most bots that aren't surfers and aren't random movers are pretty trivial to hit for the top guns. I'd don't want any bot that we can score 99% against. Better yet, 96%. I agree that Tron 2 would be a good choice though. --David Alves Well, as I see it, there are two possible goals for the TargetingChallenge2K7. One, we could try to make it as accurate a benchmark as possible for indicating a gun's overall RoboRumble performance. This would mean it might be no harder than TC2K6, but it would have reference bots that more accurately weighted what goes into a good rumble gun. Based on how the discussion of TC2K6 went last year, I sort of had this goal in mind when we started discussing TC2K7. The other option, IMO, is to make it overall much harder, such as with no 97+ bots and 5 surfers, keeping pace with how much guns have improved over time. This means the old TC's might still be better to use when first tuning a new style of gun, but it would give newer top guns more of a chance to shine (and improve) over time. This definitely has some appeal for me, and I think it could help in pushing the envelope in targeting surfers and top notch random movements, but I'm not sure it would really appeal to everyone. Splitting up the surfer and non-surfer scores in the index would at least help to compensate for this a little... Thoughts? -- Voidious I am speculating here, but if you have a gun that can perform really well against surfers and non-surfer-but-still-hard-to-hit movements, what part of the rumble is left? More complicated guns take longer to learn, but it is up to your movement to give it the time it needs, no? If you wanted to go the TargetingBenchmark2K7 route, I suggest that rather than having top bots, you have bots that have really high specialization indices (which top bots don't have since they pwn most bots evenly). That would take some participation of the authors. I don't have any examples off hand, but I suspect that the cx.* bots are in this category. They are consistent problem bots for me. -- Martin Here is a list of suggestions for a try-out style selection for a hard-to-hit challenge:
When the ball drops, try-outs are over, and the results are determined as follows:
We could even present the rating comittee as MovementChallenge2K7 reference bots. -- Martin I think the process itself seems fair to the authors of bots, but I'm not sure we'll end up with the kind of diversity we want. Isn't it likely we'd end up with 10 surfers if 10 WaveSurfer bot authors entered theirs? Not too many active bot authors are working on RandomMovement bots... Funny though, I was about to post Komarious 1.705TC if anyone wants to take a look: [voidious.mini.Komarious_1.705TC] ... My current AntiSurfer gun (by itself, no VirtualGuns) gets about 90 against it. -- Voidious They may fall under a general heading of "wave surfer", but I imagine the implementations of wcsv, Corbos, Davis Alves, and PEZ are going to behave differently, and guns will perform differently against them. Meanwhile GrubbmGait may want to resubmit GrubbmGrb's movement, and I plan to submit my randomish grid movement. My try-out suggestions were geared as I said for a "hard-to-hit" challenge, and the proof is in the pudding, as it were. If my randomish movement is predictable, or GrubbmGrb isn't really a hard challenge, should they be in a hard-to-hit targeting challenge? It is certainly pointless to have 10 clones of the wave surfing tutorial bot, but we won't really know what we'll have until we try it. If it turns out that the challenge sucks, there's nothing stopping us from saying fuck it and making a new challenge. -- Martin Heh. I'm down to give it a whirl if others are, I guess. I do agree the resulting "committee" could really help in creating MC2K7. I'll probably submit a TC version of Dookious that will certainly (hopefully?) outrank Komarious. -- Voidious This is a cool idea; I'd love to give it a try too. I think we should take the top 5 surfing and the top 5 non surfing entrants and put them in 2 devisions. This will hopefully give a bit more balance to the challenge bots. Even then, I don't think it will be a good representation of the rumble (if that's what your aiming for) because there won't be such a variety in challengers. As more of a challenge in itself, it sounds great to me. -- Kev @GrubbmGrb re:Grishnakh - I was just reading some of the earlier posts in the thread and saw you mentioning Grishnahk as a multi-mode bot, but closed source. I have source code backups for nearly all of my releases back to v0.0.1. Grishnakh was compiled from a backup of Ugluk v0.13.0. I created it as a stand-alone bot because I had done some major overhauls to the foundation of Ugluk's targeting and movement since then. The multi-mode tuning was geared for a 35 round battle, but with a little tweaking I can adjust it to handle 500 round battles as well. Still, I fear it would get creamed nearly as badly as Butterfly did. I suppose it is worth a try though. -- Martin I just checked the results (see below) and I will submit a TC-version of the latest GrubbmGrb. It performs significantly better than the current 1.1.3TC although it still has no 'killer-movement'. -- GrubbmGait | Version | WSCBotA | WSCBotB | WSCBotC | WSC | APMC | CassiusClay | FloodHT | Shadow | CFC | Overall Score | GrubbmGrb_1.2.4TC | 89.69 | 95.88 | 93.77 | 93.11 | 27.77 | 9.83 | 16.19 | 15.11 | 13.71 | 44.86 | 5 seasons | GrubbmGrb_1.1.3TC | 85.64 | 93.88 | 91.98 | 90.50 | 17.66 | 7.33 | 9.43 | 8.41 | 8.39 | 38.85 | 2 seasons By the way, I haven't entered Dookious yet because I'd like to be more confident in the rewrite before I do so. Also, once we have a lot more bots posted, it might take a while to run all the battles for a new challenger; let me know if anyone needs me to run some reference battles for them. -- Voidious I assume everyone else has been inactive on this conversation for the same reasons as myself - holidays and/or end of Fall semester. I'll enter a version of Dookious as a movement challenger soon. It'd be nice to see some more high level surfers entered - wcsv and Corbos, your bots especially! They're unique and they're PL monsters. -- Voidious I have had a semi-unique idea, I figure that the Power Management in your gun is part of your gun almost as much as the actual aiming is. Perhaps we should have a secondary ranking that uses the bots power management. Just an idea (we all know that smaller bullets can give better scores, but such a system would also help us tweak our power management) --Chase-san I would consider the RRGunChallenge the benchmark for targeting and power management. -- Voidious I haven't really been inspired to Robocode lately. I glance at the 7 day update listing every morning but .. well I just close the tab. This challenge will not likely be viable by the turn of the year. -- Martin Maybe we should set an arbitrary deadline for entries on this? The end of January seems reasonable to me. Then we take the 5 best surfers and 5 best non-surfers and have ourselves a new TargetingChallenge? -- Voidious Eh, why not, it doesn't look like its going anywhere fast. I would enter Seraphim 0.023 as both a gun and as a surfer, I lost the source unfortunately (been using 0.025's). I can use the properties file I fitted it with to do so instead. --Chase-san Actually, I mis-stated that - the plan was to take the top entrant from the top 10 authors, not the straight top 10 entries, but I propose we also split it up to be 5 surfers and 5 non-surfers. -- Voidious Hey, if you guys are around, here are some surfers (from active authors) that I'd love to see as potential movements for this challenge: Ascendant, Phoenix, WaveSerpent, Shadow, CassiusClay, Chalk, Engineer, Cyanide. I'm willing to donate CPU cycles, too! =) |
|
If you're curious about how we arrived at this rule set, we took it from the original TargetingChallenge. If you're curious about how we arrived at this bot set, check out the /PreChat page for the discussion. |
|
Ok, this seems to have fallen by the wayside a bit. But in case y'all want to use WeeksOnEnd, let me first package up a 1.8 version, because I think it should perform better than 1.2 above. I could also profile WeeklongObsession / WeekendObsession if anyone thinks it would be valuable - but Kev has much better RM bots, I see. But where are all the entrants?? -- Simonton Well, the original plan was for bot authors to enter their own bots and only pick 1 bot per author, but given the lack of entrants, I think we should change the plan. (The above table is still helpful.) How about this: * Allow nomination of other authors' bots. Would still be nice to post them to the above table for reference; I'd volunteer some CPU cycles to help run reference battles. * Hold a vote for 5 surfers and 5 non-surfers. (Maybe a week from now and running a few days?) I do prefer the vote to the raw avg score for a bot, personally, for things like execution speed and interesting / unique movements. * Bots with the most votes are the reference bots. I would tend to just say all votes are equal, as opposed to voting with a ranked list. If there are ties, we can vote those away, too. Thoughts? -- Voidious Good idea.. Hrm, well I will probably enter Seraphim 0.052's movement anyway, however I would perfer to try and get Genesis in it, but considering its movement is a long wasys down the development path, thats unlikely. I'll run battles for Seraphim's movement. --Chase-san Mind you, I would love to see WaveSerpent as one of the test bots, its movement is awesome. --Chase-san Yeah! And it's OpenSource, so I should be able to post it myself if Kev doesn't. =) -- Voidious Seraphim is nothing special. --Chase-san Hmm. Interesting results between 1.2 and 1.10 for WeeksOnEnd. Better against some, worse against others. Personally I like 1.10 better, because the small change between the two uses an idea of my own. Well, there's the stats for anyone interested. I would like to nominate Cigaret, though I don't know that it is being actively developed. It's just so slippery! And of course Dookious. -- Simonton Gonna run WaveSerpent 1.211TC right now... Another reference gun might be useful, too, maybe we should toss Chalk, WaveSerpent, and/or CassiusClay guns in there. -- Voidious 1.211? I think he has 1.221 out by now, did you mean 1.221? --Chase-san Er, yes. (Good eye ;)) -- Voidious I ran a speed test on the surfers just now, simply timing 100 rounds vs WaveSurfingChallengeBotC:
It's worth noting that Chalk and Shadow probably slow down a lot more than the others when they are getting hit more, as their data analysis is log based and thus gets slower with more bullet hits. I'll run some 500 round tests vs a fairly quick learning gun (like CassiusClay) sometime soon and post some results. (I should run it for the non-surfers, too, but I imagine they're all pretty quick.) -- Voidious Shall we set a deadline for nominating new entries and then decide the timeline for voting on the entrants? Is everyone cool with that? As above, a vote would consist of 5 surfers and 5 non-surfers, and we tally the 5 of each with the most votes and they are the reference bots. I say we close nominations next Friday and hold votes for 2 or 3 weeks after that, since not everybody checks in all that often. -- Voidious Basically don't allow any new entries after next friday then vote from then on? If thats the case then i'm okay with that. --Chase-san |
Welcome to the TargetingChallenge2K7. It's an attempt at measuring pure gun performance in 2007 :). The rules are simple.
Please post your bots' results on the /Results page. This is made very easy by using RoboLeague and the TC Calculator.
Results chatting is best carried out on the /ResultsChat page.
If you're curious about how we arrived at this rule set, we took it from the original TargetingChallenge. If you're curious about how we arrived at this bot set, check out the /PreChat page for the discussion.
![[Home]](/images/RoboWiki.png)