Calling All Developers
Ok, I think it's time to take RoboResearch to the next level. RoboResearch 2.0. I want to network it. We'll have a roboresearch server that will keep track of all the challenges people are wanting to run. Then clients will poll the servers for battles to run, downloading bot jars from the requester as necessary. The results will be sent back to the requester, as well, to be stored on their local database. When there are no battles to run, the clients will fire up a copy of the rumble instead. People's clients will prefer their own bots' battles. Who's willing to help? -- Simonton
I'm not sure a communal RoboResearch server is a good idea. It seems like you'd end up in one of two situations. 1) Some people request way more battles than they contribute, so the people running more than their fair share get annoyed, or 2) Everyone requests about the same number of battles as they contribute, in which case what's the point of having it be networked? --David Alves
- You're right on one point - we should make sure people would actually be interested in running the client. But I can definitely see benefits. Some people may certainly run way more battles than anyone else (someone named "Simonton" comes to mind), but just like in the rumble, you can always turn off your client if you don't feel like assisting that person anymore (and, in this case, turn on the rumble client by itself ;) ). But as for people requesting the same number of battles - the point is that people can do their research at different times. Skilgannon doing research in the evening in South Africa can see results quickly, then you doing research in the evening in California can see them quickly, too, since you're both helping each other out in your times of "need". The users should be instructed briefly in RoboResearch etiquette, so that for example they only run as many battles as they need to get stable ratings (e.g. 50 seasons instead of 500, or whatever). So, I guess the question is: Who out there would actually run a client like this? -- Simonton
- I would use it. As long as I always have the option of giving my own tests highest priority, I don't see how it could do anything but help everyone. As for switching to the RoboRumble when there are no more tests to run - I think it would be equally useful to be able to reverse that: run the RoboRumble until everyone has their required battles, then return to contributing cycles towards anybody who wants to run some tests. (In other words: if there are unstable bots in the rumble, I'd give that higher priority than somebody else's gun tests.) As for the timezones thing - I do run stuff 24/7 sometimes, both while I'm at work and while I'm sleeping, but I usually try to do something that will finish before I'm back at the computer, in which case there would be some idle time at the end to contribute back. -- Voidious
- Great idea! That will require a little more logic, but I'm sure we're up to the task! -- Simonton
The Idea
I've started on a new utility something like RoboLeague. The biggest difference is that it stores battle results in a database, not an XML file. This leaves LOTS of flexibility to analyze data any way you can dream up. The other big reason I decided to start on it was to be able to run battles remotely on separate machines (which I plan to accomplish through SSH). I know RoboLeague can do this with leagues (but not divisions?), but I need it in a different format to accomplish the type of remote operation I have in mind. Would there be anyone interested in making this a community project? Here are the other differences from RoboLeague I can think of right now:
- It runs RoboCode in a separate process, interacting only with .battle files and .results files (it does not utilize any of RoboCode's APIs). This has benefits and drawbacks, but I did it so that 1) I could get a fresh JVM with each battle (to avoid memory issues as encountered with my Obsession series of bots) and 2) it's a little easier to code up the remote operation.
- It is "tweak-centric" as opposed to "bot-centric". The idea is to allow the modification of a bot's properties file, where each permutation of variable assignments defines a "tweak" of that bot. So, there may be many tweaks to each version. If I/we ever get around to it, RoboResearch will be able to automatically create tweaks of a bot and run them against a set of competitors, so you can see what combination of tuner variables brings the best payoff. Maybe it could even search for a maximum benefit & drill down to optimal values? But that's just a dream down the road.
- It should have starters you can create & save for, e.g. MC2K7, where you click on "MC2K7", then on the bot/tweak-set you wish to run, then on "go".
- There will not a be a "league" style, and there will always be a "focused competitor". The intent of this utility is to test your bot, not rank a group of bots.
- So far I plan to wipe out the robots/ and .robotcache/ directories often. It would use a directory of bot jars elsewhere, then copy them into the robots/ directory when it's time to fight. This eliminates the benefit of file saving in bots, but I'm sick of waiting for RoboCode to rebuild the robot database and robot cache so often. This should eliminate a lot of the overhead that would otherwise come from spawning a new instance of RoboCode for every battle.
Capabilites So Far
- Run a battle.
- Run any number of battles at once, each on a remote host (requires password-free SSH between the machines and that RoboResearch is installed in a common NSF directory).
- Store the results in a database.
- Read in a set of reference bots from a "RoboResearch Challenge File" and run a challenge bot against them for any number of rounds.
- Query the database and give results in wiki format! =D
- Run multiple battles at once on your local machine (for multicore/multiprocessor machines).
Next might be figuring out if it would be capable of running battles for the rumble. Also adding a GUI.
Getting Started
- Get a copy of roboresearch from its [sourceforge home] by pointing an SVN client at https://roboresearch.svn.sourceforge.net/svnroot/roboresearch. You're after the "trunk" folder.
- Continue with the (hopefully) helpful instructions found in docs/getting_started_running.txt.
Cool. Let me know if you need a hand or have trouble making it work with Robocode. Then I'll support you as much as possible! --Fnl
I need something like this for running lots of seasons of 35-round matches. If it gets integrated into the main Robocode release, that would be even better. Would you mind sharing the code? -- Skilgannon
Do you not use RoboLeague? I can't live without it in development. Actually, I remember it didn't work right with some newer versions of Robocode, but I use it with 1.07. If it still doesn't work with Robocode, that should really be something Fnl or one of us take care of. -- Voidious
- Yeah, RoboLeague does everything most people are after. I have a few special needs, like restarting the JVM between battles. Other than that, I thought saving all results to a database would be handy. Also, if I can get it to work, the distributed environment should be very powerful to anyone with access to a computer lab :). But yeah, I can shoot you the code I have so far. If you're interested in helping we should set up some kind of collaboration environment. -- Simonton
I've never tried RoboLeague. Besides not being compatible with the latest version it looked like a mission to set up as well. I'll grab a copy of 1.07 and give it a try....is the source code for it available?
- I think the source is packaged with it, but I'm not sure on the licensing if it is. And hey, it's Robocode that grew incompatible, not RoboLeague. =) It's painless to setup and I bet 95% of Robocoders use it regularly. Not to say you two can't do better (you surely can), but it's still a super useful tool available right now. -- Voidious
- Hmm, all clients that uses/runs Robocode should use the robocode.control package and access the RobocodeEngine? class. Clients that does not do this will not be able to keep up with newer versions of Robocode. It would be quite hard to make changes to Robocode. The robocode.control package is meant for Robocode clients and is always backwards compatible. If something is missing in the robocode.control package, I will be happy to extend it. ;-) --Fnl
@Simonton, how about merging it with the SourceForge? Robocode project? Things like restarting robocode each season could be set as an option, I'm assuming? What are you using for database access? SQL/ODBC? Also, is it written in Java as well? I'd like to try a bit of Python, and this sounds like something fun, but even if it's in Java I'm willing to lend a bit of my time. I just have to remember that I'm also meant to be studying =) -- Skilgannon
- I'm not sure about merging with the Robocode project. I don't plan to integrate with Robocode's source at any point in the future - only through the command line, .battle files, and .results files. So I don't know if that means it should be some separate project? But anyway, for a database I'm using [HSQLDB]. I don't know anything about sourceforge ... could we open a separate project for it pretty easily? -- Simonton
We could make some options in Robocode to support this project. I will support you with whatever you need. --Fnl
- It relies on command line arguments, .battle files and .results files, so as long as they stay the same / backward compatible I'll be in business. Do you think any of these will ever change? -- Simonton
I am wondering. How far is this project? Is there any progress? (I am just curious ;-) --Fnl
- I just finished getting it to run distributed-style. My test run is pitting the latest pattern matching against all 5 random movers as I type. It's highly un-portable, though, I'm sorry to say.
It is not currently saving to a database, though I think that functionality is in place, I'm just not activating it (if I remember correctly). That's it so far, and none of it's pretty to work with. -- Simonton
Ok - why does robocode run WAY slow when I use a copy of the jar in a directory other than the one in which I installed it? It takes a very long time for it to do something between battles. -- Simonton
- Could it be building .robotcache? --Skilgannon
- Oh, sorry, I meant between rounds. -- Simonton
- I've got no idea, but you could try a process of elimination. Do a fresh robocode install, test. Delete seemingly unnecessary files one at a time, test again. If you find it before Fnl (if he knows the reason) replies, it is time well spent =) -- Skilgannon
- I did that for a while with no fruit, so I just adapted RoboResearch to use a full install :) --Simonton
- Ok, now my normal install is doing the same thing. I upgraded to 1.4beta2. At first I just grabbed the jar out if it & put it over my current install, but then I started getting this "it takes a very long time for robots to die" problem. It's not slamming the hard drive, and it's not slamming the processor. So I ran the whole installer over top of my old install, and that didn't help. I installed in a fresh directory, and it works fine. Is anybody else experiencing anything like this? -- Simonton
- Ah HA! It has to do with the CPU constant. When I set the CPU constant really high, it sometimes waits for that long when a robot dies. -- Simonton
Alrighty, I just submitted an application to open a sourceforge project for RoboResearch. I hope you were serious about helping, Skilgannon! --Simonton
- I'll try and dig up my old SF ID, and yes I am. It might take me a few days to get the layout of the code and find time, but I'll definately help with it. -- Skilgannon
- Are you sure... --Starrynte
- I'm kinda busy with Waylander at the moment (and exams...haha), but I've messaged Simonton to add me as a developer. I'm just reading through the code now. -- Skilgannon
Oh, Fnl, I just found one more request this project could ask of RoboCode: make sure that when "-nodisplay" is used, it never tries to open *any* kind of user interface (except the console). I can't run the newest version distributed over the network; it's giving me the "can't connect to your computer" message again (since I'm behind a firewall, if it tries to open a connection back to my computer to display a window or anything like that it fails). I thought maybe it was because of the new human-interaction capabilities, so I installed version 1.3.3, and that one works great. -- Simonton
Well - sourceforge apparently ignored my request to open a project. I never heard back from them. On another note, can someone tell me whether it would work to copy robot jars and their cache directories in & out of the robots folder? I would like to get RoboResearch set up to handle data-saving bots without having to leave those bots in the robots directory (because it takes a long time for robocode to go through them all every time it starts). Would robocode re-unzip the jars & overwrite things in the data saving directory, or leave it the way it was? I suppose it wouldn't be too hard to test, I was just wondering if anyone knew. -- Simonton
- [edit] - or, then again, maybe the project is already [there] but nobody bothered to tell me! -- Simonton
Ok, it was a very long time coming - BUT THE SOURCE IS NOW ON SOURCEFORGE!! YAY! So, anyone who wishes to help out, tell me how to let you! -- Simonton
- School's almost coming, but there's still a week...so...I've never programmed anything like Robocode, so something (relatively) simple(ish) for me... --Starrynte
- Great! Um - you'll need eclipse with SVN support (I recommend using [Yoxos] and grabbing a distribution with Subclipse), and I'll have to figure out how to add you as a developer on sourceforge. Send me an email so we can communicate. My address is on the ContactInfo page. Oh ... of course if you already know how to use SVN, feel free to use any client you like instead of downloading a new eclipse! -- Simonton
- :3 I only use CVS cause its built into eclipse, SVN is alot better though. --Chase-san
- That sounds familiar ... -- Simonton
- I can't help you, i have already more than enough programming projects running :/ But i would like to try this nice tool. Is it my fault that i can't get the code from the cvs server (I'm not that familiar with cvs) or do i need a name/pw? --Krabb
""If I/we ever get around to it, RoboResearch will be able to automatically create tweaks of a bot and run them against a set of competitors,"" This is probably most easily accomplished with a GeneticAlgorithm which are in fact what my project is using, the robot is NeBot, (the one that found the bugs in RoboCode). So its not really that difficult of a concept to implement. It would be some work, having to deal with different parameters that would have different discretizations and the effects of the delta's used for those discretizations would have on the robots. But doable. -- Ntroutman 2007-08-30
- I agree. What I had in mind (and have done before) was a brute force approach. Using a GeneticAlgorithm is much more elegant, and would save a lot of computation time if you set it up well. (I have to remember that I won't always have 100+ machines to use in parallel.) Perhaps our projects could merge at some point? Or are you using [RobocodeJGAP]? Either way, if you're not currently using a database to save your results, I think it could be helpful for all of us to team up. -- Simonton
- No, I'm not using JGAP, I'm using my own Genetic Algorithm library that I've been using for a couple years now for my research at work. JGAP is genetic programming, not genetic algorithm. They do share some concepts as they are both in the evolutionary computation field but they aren't the same thing. I'm not using a database to save results, as I'm not using Robocode's scoring scheme I need my own scoring system that I can easily tweak. My only concern with RoboResearch is, are all the robots compatible with standard Robocode? -- Ntroutman 2007-09-03
- Another question, why are you starting a new JVM all the time and why aren't you using the Robocode API? It would keep you from having to parse through all the text being spewed to the console as you would have direct access to the RobotResults? objects. If its a memory issue, those have been (will be as soon as my patches hit CVS) fixed. I saw the two reasons you listed in the beginning and I disagree with them. The memory thing was an issue, but its being fixed. As for coding the remote stuff, running external programs form Java isn't exactly a pretty process. There is very little control you have over the spawned processes. I've worked with it a long time ending up having to write JNI code to get the control over a process I needed from Java. In my opinion you'd be better off keeping everything in Java, even if it takes some extra work because you'll more likely stay cross-platform compatible. If you have any question feel free to email me (an address is in my profile). I've worked a lot with java and doing research with java. -- Ntroutman 2007-09-03
- I think that the only way to do remote stuff (over SSH) is using native/scripting tools. Java isn't that friendly with native programs, but it's all a tradeoff. -- Skilgannon
- Then why use SSH? Java is very networking friendly. Just write a quick protocol over TCP/IP and that makes it even more cross-platform friendly as it means people don't need SSHD setup and configured, having SSH and SSH is not a likely story for most Windows people. -- Ntroutman 2007-09-03
- I'm a Linux junkie, so I breath SSH. =). And with SSH you don't need to set up a client, and leave it running, listening for connections. It all gets done for you. You only need to call "ssh user@host 'cd /mnt/nsf1/; java roboResearch.BattleRunner? distrib.cfg' ", and it returns all the stdout right back to you, which then gets parsed by the same mechanism, voila! -- Skilgannon
- (edit conflict) Hmm, I thought I responded to that first one about 20 minutes ago. Guess I must have failed the consciousness check or something. Anyway, as I tried to say before, yes, any robot should work with standard Robocode unless it tries to write to a directory other than the one it's supposed to (and perhaps there are a couple other restrictions?). And why use another JVM? Well, it really is just the two reasons you already responded to. As for parsing the text, that part is already done. The memory was an issue whether or not it will be in the future, so it was still a valid reason to build RoboResearch that way. But you're absolutely right about cross-platform ability-ness. Once the memory issues are fixed, it would make the program more efficient to use the API instead of re-starting the JVM (in non-distributed mode). Thanks for your offer to answer questions! As for a quick protocol, feel free to contribute one ;) -- Simonton
- By the way, it now uses .results files, not the results printed to the console. I'll fix that in the description above. -- Simonton
I'm sad. 2 hours to run the TargetingChallenge for DCResearch is a lot slower than the 2 minutes it took when I had access to the computer lab at the school ... :*( -- Simonton
I have actually optimised it to use the duo's (faster dual cores :) and it has dropped down to about 1 and a half. --Chase-san
- Sweet, you should sign up for a sorceforge account & I'll add you as a developer. -- Simonton
I'm getting this when I try to run:
C:\robocode\RoboResearch>java -cp bin;hsqldb.jar roboResearch.BattleRunner?
-s 5
-C c:\robocode\RoboResearch\challenges\mc2k7.rrc
-c "davidalves.Phoenix 0.9"
Running vs. wiki.mc2k7.HawkOnFireOS? MC2K7 ... java.io.FileNotFoundException?:
robocode_install\battles\tmp.battle.results
(The system cannot find the file specified)
at java.io.FileInputStream?.open(Native Method)
at java.io.FileInputStream?.<init>(FileInputStream?.java:106)
at java.io.FileInputStream?.<init>(FileInputStream?.java:66)
at java.io.FileReader?.<init>(FileReader?.java:41)
at roboResearch.BattleResults?.load(BattleResults?.java:22)
at roboResearch.BattleRunner?.runBattle(BattleRunner?.java:106)
at roboResearch.BattleRunner?.runBattle(BattleRunner?.java:76)
at roboResearch.BattleRunner?.runChallenge(BattleRunner?.java:143)
at roboResearch.BattleRunner?.main(BattleRunner?.java:34)
There appears to have been an error in Robocode itself.
I put a copy of Robocode 1.4.5 in the robocode_install directory and put all the MC2K7 bots in it. If run robocode manually and select the tmp.battle file, it properly loads up the two bots. Any ideas? --David Alves
Nevermind, figured it out. I had edited the batch file used to run robocode and had it pointing to a different directory. --David Alves
- :) Glad you got it figured out. I hope you like the tool! I have worked on the source a little at home to make it safe to run multiple copies of roborumble at once (in the non-distributed version), I'll commit those as soon as I get internet ... -- Simonton
Ah cool, I was going to add that when I got home tonight. 4x speed improvement would be nice, but I had to go to bed before I could add it last night. =) --David Alves
- Warning: "as soon as I get internet" means "as soon as I subscribe to an internet service at home" - so it may not be immediate. But, if you would like to make the same changes yourself, all I did was change BattleRunner?.java to use something like File.createTempFile?("tmp", ".battle", "same directory as before") instead of having every copy work off of the same "tmp.battle" file. And then make sure it and its .battle.results friend get deleted after I read in the results. Umm ... this would be somewhere in the string of runBattle(...) methods. (I'm not entirely confident of the names, but those should all be close). If you'd like to help with the tool a little, you could do something similar to the DistributedBattleRunner?.java, except just for the local host (no ssh connections), so that fast learning challenges could be kicked off in one place and run X copies in parallel :). That's on my to-do list at some point now that I have a dual core machine. (No need to announce to me that you now have a quad core just to make me feel inferior) -- Simonton
- OH WAIT!! No, that was not the only change. Oh. It requires running an HSQLDB server process that all the roboresearch processes can connect to. Hmm ... let's see. You should replace the IN_MEMORY_CONNECTION in BattleRunner?.main with SERVER_CONNECTION, and from the database directory run the following command. Ok - I think that is all you have to do. -- Simonton
java -cp ../hsqldb.jar org.hsqldb.Server -database.0 file:roboresearch -dbname.0 roboresearch
There's no harm in running multiple copies of robocode from the same install directory as long as they are just using .battle files, right? I know from experience that it can cause bad interactions if you run multiple RRAH clients from the same directory, but multiple .battle ones seems safe. I'll make the change after I get home tonight. --David Alves
- Right. As long as you don't run 2 copies of a data-saving bot at once. The trouble with RR@H is that it stores the results of its battles in a file, then reads them back in to upload. On the (distant future, maybe never implemented) to-do list of RoboResearch is to make it into a rumble client that can parallelize safely (including data saving bots). Wanna be a developper? ;) -- Simonton
It's working with multiple threads now. Do you need to add me to the project on sourceforge to allow me to commit my changes? --David Alves
It's been committed now. Enjoy! --David Alves
Very nice work! I'm just about to commit a change to print out # of seasons, if you'd like update. -- Simonton
Updated to delete .results files (they stayed forever before). Also if a thread encountered an exception when running a battle it would previously stop running. Now the thread will skip that battle and move on to the next one.
Incidentally, the battles are run in the season order, so the # of seasons complete should be
battles completed so far / # of battles per season
This doesn't account for any exceptions encountered while running battles though, if you want your season-printing thing to account for those then it gets a bit more complicated.
If you're updating to this version you might want to clear out *.results from your robocode_install/battles folder, there might be a lot of 'em. =)
--David Alves
Oops ... ok, NOW I committed the printing of the # of seasons in wiki output. Along with a few other, un-notable tweaks. Thanks again for your work, David - I'm running in two threads as we speak! -- Simonton
Just made a nice change in the way challenge battles are chosen. The number of seasons now represents the number of *completed* season you wish to have in the database altogether. The bot(s) with the least number of battles fought will be run until every pairing in the challenge has at least that many battles. It has been committed. -- Simonton
- And now I just committed a fix for a bug in that. -- Simonton
![[Home]](/images/RoboWiki.png)