Calling All Developers
Ok, I think it's time to take RoboResearch to the next level. RoboResearch 2.0. I want to network it. We'll have a roboresearch server that will keep track of all the challenges people are wanting to run. Then clients will poll the servers for battles to run, downloading bot jars from the requester as necessary. The results will be sent back to the requester, as well, to be stored on their local database. When there are no battles to run, the clients will fire up a copy of the rumble instead. People's clients will prefer their own bots' battles. Who's willing to help? -- Simonton
I'm not sure a communal RoboResearch server is a good idea. It seems like you'd end up in one of two situations. 1) Some people request way more battles than they contribute, so the people running more than their fair share get annoyed, or 2) Everyone requests about the same number of battles as they contribute, in which case what's the point of having it be networked? --David Alves
- You're right on one point - we should make sure people would actually be interested in running the client. But I can definitely see benefits. Some people may certainly run way more battles than anyone else (someone named "Simonton" comes to mind), but just like in the rumble, you can always turn off your client if you don't feel like assisting that person anymore (and, in this case, turn on the rumble client by itself ;) ). But as for people requesting the same number of battles - the point is that people can do their research at different times. Skilgannon doing research in the evening in South Africa can see results quickly, then you doing research in the evening in California can see them quickly, too, since you're both helping each other out in your times of "need". The users should be instructed briefly in RoboResearch etiquette, so that for example they only run as many battles as they need to get stable ratings (e.g. 50 seasons instead of 500, or whatever). So, I guess the question is: Who out there would actually run a client like this? -- Simonton
- I would use it. As long as I always have the option of giving my own tests highest priority, I don't see how it could do anything but help everyone. As for switching to the RoboRumble when there are no more tests to run - I think it would be equally useful to be able to reverse that: run the RoboRumble until everyone has their required battles, then return to contributing cycles towards anybody who wants to run some tests. (In other words: if there are unstable bots in the rumble, I'd give that higher priority than somebody else's gun tests.) As for the timezones thing - I do run stuff 24/7 sometimes, both while I'm at work and while I'm sleeping, but I usually try to do something that will finish before I'm back at the computer, in which case there would be some idle time at the end to contribute back. -- Voidious
- Great idea! That will require a little more logic, but I'm sure we're up to the task! -- Simonton
The Idea
I've started on a new utility something like RoboLeague. The biggest difference is that it stores battle results in a database, not an XML file. This leaves LOTS of flexibility to analyze data any way you can dream up. The other big reason I decided to start on it was to be able to run battles remotely on separate machines (which I plan to accomplish through SSH). I know RoboLeague can do this with leagues (but not divisions?), but I need it in a different format to accomplish the type of remote operation I have in mind. Would there be anyone interested in making this a community project? Here are the other differences from RoboLeague I can think of right now:
- It runs RoboCode in a separate process, interacting only with .battle files and .results files (it does not utilize any of RoboCode's APIs). This has benefits and drawbacks, but I did it so that 1) I could get a fresh JVM with each battle (to avoid memory issues as encountered with my Obsession series of bots) and 2) it's a little easier to code up the remote operation.
- It is "tweak-centric" as opposed to "bot-centric". The idea is to allow the modification of a bot's properties file, where each permutation of variable assignments defines a "tweak" of that bot. So, there may be many tweaks to each version. If I/we ever get around to it, RoboResearch will be able to automatically create tweaks of a bot and run them against a set of competitors, so you can see what combination of tuner variables brings the best payoff. Maybe it could even search for a maximum benefit & drill down to optimal values? But that's just a dream down the road.
- It should have starters you can create & save for, e.g. MC2K7, where you click on "MC2K7", then on the bot/tweak-set you wish to run, then on "go".
- There will not a be a "league" style, and there will always be a "focused competitor". The intent of this utility is to test your bot, not rank a group of bots.
- So far I plan to wipe out the robots/ and .robotcache/ directories often. It would use a directory of bot jars elsewhere, then copy them into the robots/ directory when it's time to fight. This eliminates the benefit of file saving in bots, but I'm sick of waiting for RoboCode to rebuild the robot database and robot cache so often. This should eliminate a lot of the overhead that would otherwise come from spawning a new instance of RoboCode for every battle.
Capabilites So Far
- Run a battle.
- Run any number of battles at once, each on a remote host (requires password-free SSH between the machines and that RoboResearch is installed in a common NSF directory).
- Store the results in a database.
- Read in a set of reference bots from a "RoboResearch Challenge File" and run a challenge bot against them for any number of rounds.
- Query the database and give results in wiki format! =D
- Run multiple battles at once on your local machine (for multicore/multiprocessor machines).
Next might be figuring out if it would be capable of running battles for the rumble. Also adding a GUI.
Getting Started
- Get a copy of roboresearch from its [sourceforge home] by pointing an SVN client at https://roboresearch.svn.sourceforge.net/svnroot/roboresearch. You're after the "trunk" folder.
- Continue with the (hopefully) helpful instructions found in docs/getting_started_running.txt.
Cool. Let me know if you need a hand or have trouble making it work with Robocode. Then I'll support you as much as possible! --Fnl
I need something like this for running lots of seasons of 35-round matches. If it gets integrated into the main Robocode release, that would be even better. Would you mind sharing the code? -- Skilgannon
Do you not use RoboLeague? I can't live without it in development. Actually, I remember it didn't work right with some newer versions of Robocode, but I use it with 1.07. If it still doesn't work with Robocode, that should really be something Fnl or one of us take care of. -- Voidious
- Yeah, RoboLeague does everything most people are after. I have a few special needs, like restarting the JVM between battles. Other than that, I thought saving all results to a database would be handy. Also, if I can get it to work, the distributed environment should be very powerful to anyone with access to a computer lab :). But yeah, I can shoot you the code I have so far. If you're interested in helping we should set up some kind of collaboration environment. -- Simonton
I've never tried RoboLeague. Besides not being compatible with the latest version it looked like a mission to set up as well. I'll grab a copy of 1.07 and give it a try....is the source code for it available?
- I think the source is packaged with it, but I'm not sure on the licensing if it is. And hey, it's Robocode that grew incompatible, not RoboLeague. =) It's painless to setup and I bet 95% of Robocoders use it regularly. Not to say you two can't do better (you surely can), but it's still a super useful tool available right now. -- Voidious
- Hmm, all clients that uses/runs Robocode should use the robocode.control package and access the RobocodeEngine? class. Clients that does not do this will not be able to keep up with newer versions of Robocode. It would be quite hard to make changes to Robocode. The robocode.control package is meant for Robocode clients and is always backwards compatible. If something is missing in the robocode.control package, I will be happy to extend it. ;-) --Fnl
@Simonton, how about merging it with the SourceForge? Robocode project? Things like restarting robocode each season could be set as an option, I'm assuming? What are you using for database access? SQL/ODBC? Also, is it written in Java as well? I'd like to try a bit of Python, and this sounds like something fun, but even if it's in Java I'm willing to lend a bit of my time. I just have to remember that I'm also meant to be studying =) -- Skilgannon
- I'm not sure about merging with the Robocode project. I don't plan to integrate with Robocode's source at any point in the future - only through the command line, .battle files, and .results files. So I don't know if that means it should be some separate project? But anyway, for a database I'm using [HSQLDB]. I don't know anything about sourceforge ... could we open a separate project for it pretty easily? -- Simonton
We could make some options in Robocode to support this project. I will support you with whatever you need. --Fnl
- It relies on command line arguments, .battle files and .results files, so as long as they stay the same / backward compatible I'll be in business. Do you think any of these will ever change? -- Simonton
I am wondering. How far is this project? Is there any progress? (I am just curious ;-) --Fnl
- I just finished getting it to run distributed-style. My test run is pitting the latest pattern matching against all 5 random movers as I type. It's highly un-portable, though, I'm sorry to say.
It is not currently saving to a database, though I think that functionality is in place, I'm just not activating it (if I remember correctly). That's it so far, and none of it's pretty to work with. -- Simonton
Ok - why does robocode run WAY slow when I use a copy of the jar in a directory other than the one in which I installed it? It takes a very long time for it to do something between battles. -- Simonton
- Could it be building .robotcache? --Skilgannon
- Oh, sorry, I meant between rounds. -- Simonton
- I've got no idea, but you could try a process of elimination. Do a fresh robocode install, test. Delete seemingly unnecessary files one at a time, test again. If you find it before Fnl (if he knows the reason) replies, it is time well spent =) -- Skilgannon
- I did that for a while with no fruit, so I just adapted RoboResearch to use a full install :) --Simonton
- Ok, now my normal install is doing the same thing. I upgraded to 1.4beta2. At first I just grabbed the jar out if it & put it over my current install, but then I started getting this "it takes a very long time for robots to die" problem. It's not slamming the hard drive, and it's not slamming the processor. So I ran the whole installer over top of my old install, and that didn't help. I installed in a fresh directory, and it works fine. Is anybody else experiencing anything like this? -- Simonton
- Ah HA! It has to do with the CPU constant. When I set the CPU constant really high, it sometimes waits for that long when a robot dies. -- Simonton
Alrighty, I just submitted an application to open a sourceforge project for RoboResearch. I hope you were serious about helping, Skilgannon! --Simonton
- I'll try and dig up my old SF ID, and yes I am. It might take me a few days to get the layout of the code and find time, but I'll definately help with it. -- Skilgannon
- Are you sure... --Starrynte
- I'm kinda busy with Waylander at the moment (and exams...haha), but I've messaged Simonton to add me as a developer. I'm just reading through the code now. -- Skilgannon
Oh, Fnl, I just found one more request this project could ask of RoboCode: make sure that when "-nodisplay" is used, it never tries to open *any* kind of user interface (except the console). I can't run the newest version distributed over the network; it's giving me the "can't connect to your computer" message again (since I'm behind a firewall, if it tries to open a connection back to my computer to display a window or anything like that it fails). I thought maybe it was because of the new human-interaction capabilities, so I installed version 1.3.3, and that one works great. -- Simonton
Well - sourceforge apparently ignored my request to open a project. I never heard back from them. On another note, can someone tell me whether it would work to copy robot jars and their cache directories in & out of the robots folder? I would like to get RoboResearch set up to handle data-saving bots without having to leave those bots in the robots directory (because it takes a long time for robocode to go through them all every time it starts). Would robocode re-unzip the jars & overwrite things in the data saving directory, or leave it the way it was? I suppose it wouldn't be too hard to test, I was just wondering if anyone knew. -- Simonton
- [edit] - or, then again, maybe the project is already [there] but nobody bothered to tell me! -- Simonton
Ok, it was a very long time coming - BUT THE SOURCE IS NOW ON SOURCEFORGE!! YAY! So, anyone who wishes to help out, tell me how to let you! -- Simonton
- School's almost coming, but there's still a week...so...I've never programmed anything like Robocode, so something (relatively) simple(ish) for me... --Starrynte
- Great! Um - you'll need eclipse with SVN support (I recommend using [Yoxos] and grabbing a distribution with Subclipse), and I'll have to figure out how to add you as a developer on sourceforge. Send me an email so we can communicate. My address is on the ContactInfo page. Oh ... of course if you already know how to use SVN, feel free to use any client you like instead of downloading a new eclipse! -- Simonton
- :3 I only use CVS cause its built into eclipse, SVN is alot better though. --Chase-san
- That sounds familiar ... -- Simonton
- I can't help you, i have already more than enough programming projects running :/ But i would like to try this nice tool. Is it my fault that i can't get the code from the cvs server (I'm not that familiar with cvs) or do i need a name/pw? --Krabb
""If I/we ever get around to it, RoboResearch will be able to automatically create tweaks of a bot and run them against a set of competitors,"" This is probably most easily accomplished with a GeneticAlgorithm which are in fact what my project is using, the robot is NeBot, (the one that found the bugs in RoboCode). So its not really that difficult of a concept to implement. It would be some work, having to deal with different parameters that would have different discretizations and the effects of the delta's used for those discretizations would have on the robots. But doable. -- Ntroutman 2007-08-30
- I agree. What I had in mind (and have done before) was a brute force approach. Using a GeneticAlgorithm is much more elegant, and would save a lot of computation time if you set it up well. (I have to remember that I won't always have 100+ machines to use in parallel.) Perhaps our projects could merge at some point? Or are you using [RobocodeJGAP]? Either way, if you're not currently using a database to save your results, I think it could be helpful for all of us to team up. -- Simonton
- No, I'm not using JGAP, I'm using my own Genetic Algorithm library that I've been using for a couple years now for my research at work. JGAP is genetic programming, not genetic algorithm. They do share some concepts as they are both in the evolutionary computation field but they aren't the same thing. I'm not using a database to save results, as I'm not using Robocode's scoring scheme I need my own scoring system that I can easily tweak. My only concern with RoboResearch is, are all the robots compatible with standard Robocode? -- Ntroutman 2007-09-03
- Another question, why are you starting a new JVM all the time and why aren't you using the Robocode API? It would keep you from having to parse through all the text being spewed to the console as you would have direct access to the RobotResults? objects. If its a memory issue, those have been (will be as soon as my patches hit CVS) fixed. I saw the two reasons you listed in the beginning and I disagree with them. The memory thing was an issue, but its being fixed. As for coding the remote stuff, running external programs form Java isn't exactly a pretty process. There is very little control you have over the spawned processes. I've worked with it a long time ending up having to write JNI code to get the control over a process I needed from Java. In my opinion you'd be better off keeping everything in Java, even if it takes some extra work because you'll more likely stay cross-platform compatible. If you have any question feel free to email me (an address is in my profile). I've worked a lot with java and doing research with java. -- Ntroutman 2007-09-03
- I think that the only way to do remote stuff (over SSH) is using native/scripting tools. Java isn't that friendly with native programs, but it's all a tradeoff. -- Skilgannon
- Then why use SSH? Java is very networking friendly. Just write a quick protocol over TCP/IP and that makes it even more cross-platform friendly as it means people don't need SSHD setup and configured, having SSH and SSH is not a likely story for most Windows people. -- Ntroutman 2007-09-03
- I'm a Linux junkie, so I breath SSH. =). And with SSH you don't need to set up a client, and leave it running, listening for connections. It all gets done for you. You only need to call "ssh user@host 'cd /mnt/nsf1/; java roboResearch.BattleRunner? distrib.cfg' ", and it returns all the stdout right back to you, which then gets parsed by the same mechanism, voila! -- Skilgannon
- (edit conflict) Hmm, I thought I responded to that first one about 20 minutes ago. Guess I must have failed the consciousness check or something. Anyway, as I tried to say before, yes, any robot should work with standard Robocode unless it tries to write to a directory other than the one it's supposed to (and perhaps there are a couple other restrictions?). And why use another JVM? Well, it really is just the two reasons you already responded to. As for parsing the text, that part is already done. The memory was an issue whether or not it will be in the future, so it was still a valid reason to build RoboResearch that way. But you're absolutely right about cross-platform ability-ness. Once the memory issues are fixed, it would make the program more efficient to use the API instead of re-starting the JVM (in non-distributed mode). Thanks for your offer to answer questions! As for a quick protocol, feel free to contribute one ;) -- Simonton
- By the way, it now uses .results files, not the results printed to the console. I'll fix that in the description above. -- Simonton
I'm sad. 2 hours to run the TargetingChallenge for DCResearch is a lot slower than the 2 minutes it took when I had access to the computer lab at the school ... :*( -- Simonton
I have actually optimised it to use the duo's (faster dual cores :) and it has dropped down to about 1 and a half. --Chase-san
- Sweet, you should sign up for a sorceforge account & I'll add you as a developer. -- Simonton
![[Home]](/images/RoboWiki.png)