RoboRumble/FutureVersions

I think we are moving to a "pre-defined" battles model (ie. the server defines which battles must be runed and the @home nodes run it), but this model is dangerous. Consider someone connecting to the internet on Mon and returning the results on Wed. Will the server need to wait for these results in order to update the rankings? I think it shouldn`t. Considering that the main advantage of the ELO system is that it doesn't need a fixed number of matches, nor all the competitors to play the same number of battles, it would be easier to send a current classification to the client, and then let it to decide which battles to run. It would allow for a better use of distributed power, because the clients would be optimistic (they would keep running battles even if they have no "official" assignment). -- Albert

I totally, utterly and fully agree with Albert here. -- PEZ

Yes, i see what you are saying. If the client just ran continously, and uploading every chance it got, then it could just make a best guess as to what battles need running (those with the lowest number of matches, i guess), and update its list of how many matches each bot has done when it uploads. The only problem is if a bot was updated and a client guessed it should run a battle with that bot, but only had the old version. Maybe we could say you can only update bots on Sundays or something, and then the client knows not to run any battles until it has been online after midnight saturday (i recommend using GMT/UT). -- Tango

Albert has, as usual, made a very good point. However, what we're dealing with here is the nature of a distributed system. There's really no way to get this realtime assurance out of a distributed system, and even if you just give the current classifications to a client and let it sort things out, the same problem crops up. It would just be doing the work the server would have done on the server side, but now it's doing it on the client side with data that's still only as up to date as the last time it communicated with the server. I think the advantage we have with the ELO system is as follows: if the central server is trying to get everyone the same number of matches fought, and it's giving each client an assignment with a little from each bot (so that one client failing to report won't take out all of one bot's recent matches) then as long as most of the clients report back the rankings will remain stable. Also, since ELO can handle varying numbers of matches and the rankings are long-term, I don't think it'll matter if a client gets it's match assignment on Monday and returns it on Wednesday, there's still enough data to rank on, some bots may just have one or two more matches than others until that client reports in.

The subject of rankings also brings about a new point. I think we should not use data from a bot's previous version in the calculation of it's new version's ranking. Agree or disagree? -- Kuuran

I don't think we should, but I don't really think that's in question so much, since no tournament I know about ever put old ratings into account (except maybe to choose first battles, in the case of the ER). -- Kawigi

I guess you could put the final rating of the previous version in brackets next to the current ranking (which would have started from scratch) so people can see if the new version is actually any better. -- Tango

If we allowed the client side to determine which battles to run would we not end up with each system running the whole of the league to it's conclusion? They are all going to start with the same bot list, same algorithm for chosine where to start, and run the same pairings. In essence we end up with redundancy but not paralellel processing. If you think of the Seti@Home project how did it operate? It said to the clients here is your data packet. Please search it and send me the results. It did not say here is all of the possible data, you decide what to do and send me the results. Something has to be the Droid and something has to have the intelligence.

If you are worried about having a central server, maybe a better approach would be to designate a couple of computers as master capable. The potential masters would need to share a client list. If the master was not reachable for some period of time, one of the secondary masters could come on line and take over the completion of the current running process while a human untangles the mess.

If you are worried about a machine recieving a pairing and not sending the results and holding up the whole process, put some timer on how long you will wait. If the timer is exceeded, re-issue the packet to a responsive machine. Wait and see which one responds and use the data to move on. -- jim

The ELO system does mean that even with random battles the rankings will eventually stabilize. Unfortunately, for Robocode, there is a problem, in that if you allow the clients to decide how many battles to run in each matchup, you are changing the fundamental rules of the game. You can't allow that. For instance, if my bot was a much better bot over 2 rounds than over 10 rounds, I could just have my client run matchups against my bot at 2 rounds each over and over and over again, thus modifying the rating of my bot. Since the number of rounds to run is so crucial to a bot's performance, I think that this setting at least should be fixed.

A simple fix for the original problem that Albert stated is as follows:

 1. Have the server give only a few matchups at a time (clients have to stay connected in order to get new matchups, just like SETI@Home)
 2. If the server doesn't get results back in a certain amount of time, it hands that matchup off to someone else

In addition, I think that each matchup should last longer. Most agree that 10 rounds is too short a time to judge a bot's true performance. Amarok is limited to this number of rounds because he is running all of the battles on a single machine. With RoboRumble@Home, we are not limited by computational power. If we increase the number of rounds to 50 or 100, not only will we get more accurate results, but the need to have data files will be lessened. If we upped the number of rounds to 100, we could eliminate data files completely, thus alleviating the data file problem for distributed battles.

I don't think that assuming clients will stay connected is a very big problem. I would be interested to know how many of us would actually want to run battles offline. If the number is zero or one, I don't think we should constrain ourselves in this way. Keeping clients connected to the server is a big advantage, as it solves the battle timeout problem.

If running battles offline is for some reason necessary, another option is the following:

 1. Have the server run through an entire set of pairings from beginning to end before starting again
 2. If not all pairings have been assigned, keep handing out new pairings to clients
 3. If all pairings have been assigned, but not all results have been received, hand out pairings for which the results have not been received
 4. Once battle results have been received for each pairing, start handing out pairings for the next run

This is all just my two cents. I don't know if any of this has been said before, but it sounds to me like a reasonable solution to the problem.

-- nano

It's already been suggested that the server hands out the number of rounds to play, I'm assuming it's to-do listed, but for the moment it's not a problem, the number of rounds run is reported as part of the bot's score, and no one is changing away from the 35 that we've currently decided on. We also have an advantage over SETI@Home in that our data to run is repeating. Basically, if we run all the pairings and don't get one back for awhile (say not until 4 times through the list later), when that one does come back it can be simply used for the next time that pairing comes up (the 5th iteration of the list since the pairing was assigned, but the data isn't going to waste, as it's still valid). -- Kuuran

Just a philosofical note: The underlying principle behind all the project is loose integration. The client can run unconnected, you don't need an up-to-date participants list nor all robot files to make it run, the server doesn't need to know the participants list to build the rankings!, the modules in the client comunicate using text files, so pluguin a GUI, or a new module is simple and can be done by anybody interested. It makes things easier to coordinate (just because they don't need coordination) and makes the system resistant to errors. Right now, the RobocodeRepository is down, but RR@home is working. The Wiki site could be down, but the clients would keep running and would upload the results when it is back, I could unplug my client right now, and the rankings would be valid ... Centralizing RR@H is very risky, building it through autonomous agents is safe.

Of course we can add some algorith for the clients to choose the matches smartly than now (pe. the probability to choose a bot could be proportional to the number of matches held), but it won't be a good idea to make the server plan all the work (and becoming even a little more philosofical, that's not the way Internet does things). -- Albert

Again I must, without restrictions, agree with Albert. Coordination means problems. The current autonomousness means the system is rubust and can stand major damage to the infrastructure. It's beatiful and powerful. Albert is the man. -- PEZ

It just seems to me that "completely uncoordinated client/server architecture" is a chimera that you are chasing for no good reason. After all, this system was supposedly designed after SETI@Home, and there is a great deal of coordination there. In the end, my point is merely that results would be more accurate if the server had control over the battles run. It isn't a big deal to have the server be in control. Sure, a mostly peer-to-peer RoboRumble has its poetic appeal, but I have yet to understand good practical reasons for it. -- nano

I doubt i will do many battles if you have to be online to do it. I just set the properties not the upload or download, and leave it running whenever i'm not using the computer (or even when i am, if i'm not doing anything CPU-intensive, eg. it is running now). When i get back to the computer, i upload all the results it has, often a few hundred are uploaded at a time. It works brilliantly that way. Maybe their could be a few people who volunteer to run special clients that do check how many battles each bot has, and even them up, but everyone else can run it like it is now. The robocode rules (rounds in a battle, size of battle field, etc.) should be fixed. Maybe not by the server, but hardcoded into the client. -- Tango