This article documents the whole competition and particularly the progress of Reijer's Shogi program SPEAR. Reijer has contributed much work to AI Factory's Shogi products, but his main effort is now on the Shogi program SPEAR, which has rapidly evolved to become a strong competitor at the CSA, for the first time finishing ahead of Shotest in 2007. The following pre-amble briefly documents AI Factory's entry this year.
Shotest entered this year's CSA in a kind of holding role. Work this year has been somewhat squeezed out by competing attention spent on our other game engines. In consequence the decision to compete was only made 2 weeks before the competition! Shotest has, over the years, been a robust competitor, always either being in the 2nd or final round of the competition, and twice coming 3rd overall (however that latter favourable result was in the previous millennium!). Shotest is ready for evolution to the next level, but not this year. Given that, we decided to compete with a rather modest laptop rather than try a last minute arrangement to obtain a superfast computer. This laptop was 60% of the speed of the PC we used last year. This meshed quite well with our development plans as we anyway needed to make the program faster for commercial use, so devoted time to making it faster rather than stronger.
However as the day of the competition rapidly approached, and given the escalation of hardware speed and program strength being entered into the competition, we realised that our unbroken record of always winning between 4 and 6 games (44%-66%) in the 2nd round in the 11 times we competed was no longer a given! As Reijer pointed out, there was a real danger that we could even be totally wiped out.
This realisation focussed our attention and the 2 weeks before the competition (which accounted for 90% of the work completed on Shotest that year) was fairly intensive. Testing continued back in the UK after we had already flown into Japan, with results being relayed back just hours before the first round.
After our moderated expectations Shotest actually managed to hold on to its previous record of 4 wins from 9 games in the 2nd round, so this was a relief. Indeed ours was the slowest hardware in that round: The second slowest was twice as fast. At the extreme, the FPGA based A-Class program (which we lost to after a marathon 242 move game) was searching 1000x faster than our program. Shotest also lost to Yamada for the first time ever, but we were happy with our 4 wins. This left us in a group that included TACOS, which just 3 years ago was ranked 6th in the world. A few other well established strong programs also fell sharply down the rankings.
Next year, development time permitting, Shotest will evolve to adopt the newer proprietary search developed for our Treebeard chess program. Our current work on poker is also contributing ideas that will see their way into shogi.
This year I was planning to skip the CSA tournament for a long time. I was really fed up with spending a huge amount of time tuning the evaluation function of Spear in the months running up to the tournament. Bonanza had opened my eyes (and those of many others) to a new method for automatic optimization of the evaluation function, which would basically end these fruitless hours of adding a few points here and subtracting a few points there. As I saw it, adding learning to Spear would be the only way to improve the program enough to warrant an entry.
The problem was that everybody seems to know how to implement Hoki's learning method except me. Of course I had read the papers, but I still didn't have a full grasp of the method in such a way that I could program it. The initial plan was to use the summer to try and implement Hoki's learning method, but at that time it became clear that I would start a new job from April 1st 2009 and work just seemed to snowball from there. During the summer, I generated a new opening book and set up a new evaluation function for Spear using hashed patterns, but there was no time to try and implement Bonanza's learning method.
After the summer, things only got worse and when the deadline for entering the CSA tournament arrived, I actually exchanged some E-mails with Jeff Rollason ending with the conclusion that it might be best if we both skipped the tournament. In the end, we both entered, but already seriously thinking about cancelling the entry later. There just seemed to be no point this year.
Then early February, just a few days before my 42nd birthday, I got an early present: the Bonanza source code was released! This was really a gift, because what I already suspected turned out to be true: the inner workings of Spear and Bonanza are very similar, in general based on the Crafty chess program. Therefore, for me the Bonanza source code was very easy to understand. Because of this, it took me less than two weeks (not full time, just a couple of hours in the evening) to get the learning code to work. As a result, I had more than two months to optimize the new evaluation function I had made. This got me excited, because I was now in full preparation of moving and the idea of Spear getting stronger while I was packing was almost too good to be true.
Unfortunately, things didn't go as well as hoped. The learning programming did its job, but the evaluation function I had made turned out to be unable to cover all the important aspects of shogi positions. The Bonanza method is based on the assumption that evaluation function features can be updated in such a way that the program will learn to select the moves that professionals played, but even after long learning sessions there were just too many position where the professional move was not selected by Spear. I had a version of Spear that just used the evaluation features of Bonanza and this version kept beating the other versions into the ground. This was the situation just a couple of days before moving from Yamagata to Tokyo (end of March). What to do?
In the end, I decided to further tune the original evaluation features of Bonanza, tweaking some of the learning parameters. After a few weeks of learning, this improved the program slightly over the original Bonanza evaluation features. I also added some of the hashed patterns for king defence to the evaluation and this was the version I entered in the tournament.
I had hoped for some more originality in my program and I also didn't like the fact that Spear played the Anaguma castle so often. As a shogi player, I don't like the Anaguma at all, because I think it is played by cowards who prefer not to lose than to win. My personal record (as a human) against the Anaguma is very good. The main reason is probably that when my opponent plays Anaguma, I more than anything want to win to show that this is an inferior castle. There is nothing like making one tokin after another to take away the Anaguma generals one by one (but I digress). When Spear was playing one game after another building an Anaguma, it felt like my son decided to enter a religious sect instead of going to medical school. This is not how I raised my shogi program!
Still, winning heals all wounds and I decided to enter the strongest new version, even though it kept playing Anaguma all the time. I also kept the evaluation function learning going until the morning the CSA tournament started.
This year's CSA tournament had been moved from the Kazusa Arc in Chiba to Waseda University in Tokyo for financial reasons. The Kazusa Arc is a spacious and luxury playing environment, but also very expensive. The CSA had already been in the red for a few years and despite the impressive progress in playing strength, it had been very hard to attract sponsors for the tournament. Although I don't know all the details, it was clear that lack of sponsorship was not going to improve considering the world wide economic crisis. Waseda University offered a very good location near the centre of Tokyo, not far from Shinjuku. For me this had the extra advantage that I didn't need to book a hotel, because I had moved close enough to make the trip from home to the venue by train every day (although it meant getting up at 6 on the second day).
Most participants were very happy about the new location being in Tokyo. For example, Jeff Rollason also had one of his cheapest visits to the tournament this year. Because of the weak pound, the plane ticket was cheap and because he could stay with me for the tournament, there were almost no extra costs. This was a big difference with the Kazusa Arc, which costs about 15,000 Yen a night. The disadvantage of the new location was that the playing halls were much smaller than at the Kazusa Arc, which meant that only a limited number of visitors were allowed. A former student of mine (and former participant with the program Avante) Yoshinori Higashiuchi had interest in visiting the tournament in person, but could only get permission for the first two days, not for the day of the final.
Full attention for one's program at all levels
This year there were 42 participants, a number that has been quite stable over the past few years. However, there were some notable programs absent. First of all, last year's runner up Tanase Shogi was not among the participants. Tanase didn't feel his program was ready for this tournament and he decided to skip it. Also, Nara Shogi, which finished in seventh place last year didn't enter this year. And just a couple of days before the tournament, the program Kiri (seeded for the second day) cancelled its entry. As a result, instead of eight programs, nine programs would qualify for the second day in the first preliminary stage.
There were also a number of new or returning entries that caught the attention. First of all, KCC Shogi was back after a three year absence. I thought there were political reasons why the North Korean based KCC Shogi skipped the tournament for a while. As far as I know, these political reasons still exist, so I don't really know why it entered this time. In any case, its record in the CSA tournament is very impressive so it was immediately considered to be a favourite.
The second program that seemed very strong was the new program Monju, which had been playing on the floodgate server in the weeks before the tournament and had been about as strong a Bonanza, beating all the top programs there. Finally, the program Blunder also seemed a strong new program. On the floodgate server, its results were not as impressive as those of Monju, but it seemed to have a good chance of becoming a seeded program for the second preliminary round. When I made these assessments, I didn't know that Blunder was playing on the floodgate server on a very slow machine. I would find this out the hard way...
TACOS programmer Hashimoto trying some last-minute improvements
Jeff and I took it easy on the first day, arriving at the tournament venue at about noon. KCC and Monju had already established themselves, winning their first three games. Also unbeaten was Yamada Shogi, which was a small surprise. It is a program that regularly makes it to the second day without making much impact there. Still, it seemed like it was improved considerably this year.
I ran some final tests on the learned evaluation features that I was planning to use in the tournament (tuning had finished that morning), sharing the room with Hashimoto (TACOS) and Otsuki (Otsuki Shogi), who were also doing some final tuning. Hashimoto seemed far from confident that TACOS would do well this year and Otsuki just left it all in the middle when asked how his program was doing. (I later heard that he had played his program under the names "test1" and "test2" on the floodgate server. The results were quite good, but not as good as for example the results of Monju.) I was also told that Kakinoki had already given up hope of reaching the finals this year. Kakinoki is famously pessimistic about the chances of his own program, but it was clear that the release of the Bonanza code had changed the perspective completely.
Going back to the first qualification round: KCC Shogi basically wiped the floor with everybody. Only the game against Monju was reasonably close. KCC Shogi's play was so impressive that people already starting talking about the possibility of winning every single game through three days, which had never been done before. In any case, KCC Shogi seemed to have returned with a vengeance.
Second place was for Monju, losing only against KCC Shogi. Seeing it in action, it became clear why Monju was so strong. It was running six Bonanza programs on three 8 processor machines (each Bonanza running on 4 processors). Each Bonanza program had a slight random distortion in its evaluation function. In the end, the move to play would be decided by a majority vote between these six Bonanza programs. A very simple idea, but surprisingly effective. Tests showed that Monju could win about 58% of the games against the original Bonanza. By the way, one of Monju's biggest supporters was Bonanza programmer Hoki, who generously gave a lot of advice about using his program in the weeks before the tournament. Also interesting was that one of the three machines running the Bonanza versions was actually a little more expensive (read: a little faster) than the other two. I wonder how that would have influenced the overall strength...
Third place was for Blunder, which only lost against KCC Shogi and Monju, but won all other games. Blunder was also using Bonanza's learning method, but at this stage of the tournament it was still unclear how strong it really was. The program Yumeki finished fourth, which was a big improvement from last year when it finished 13th. It also only lost against KCC Shogi and Monju, so it would be really tested on the second day.
The other programs who qualified from the first qualification round were veteran programs. WILDCAT, Yamada Shogi, Garyu, Auau Shogi have been around for a while, each qualifying for the second day multiple times. The final program to survive the first selection was ym-Shogi. Last year it finished 16th as a first year entry, so this was an important improvement. Still, it was lucky that Kiri cancelled at the last moment, or it would not have made the cut.
Results of the first qualification round
No. Program Name 1 2 3 4 5 6 7 Pt SOS SB MD 1* KCC Shogi 4+ 3+ 16+ 2+ 6+ 9+ 8+ 7.0 31.0 31.0 22.0 2* Monju 11+ 19+ 8+ 1- 5+ 4+ 3+ 6.0 32.5 25.5 18.0 3* Blunder 10+ 1- 5+ 22+ 8+ 6+ 2- 5.0 31.5 18.5 12.0 4* Yumeki 1- 10+ 23+ 13+ 11+ 2- 7+ 5.0 29.5 16.5 11.0 5* WILDCAT 12+ 15+ 3- 7+ 2- 13+ 6+ 5.0 29.0 18.0 11.0 6* Yamada Shogi 13+ 7+ 18+ 16+ 1- 3- 5- 4.0 30.0 13.0 6.0 7* Garyu 17+ 6- 14+ 5- 12+ 10+ 4- 4.0 28.0 14.0 7.0 8* Auau Shogi 24+ 21+ 2- 9+ 3- 11+ 1- 4.0 28.0 10.0 6.0 9* ym-Shogi 19+ 11- 21+ 8- 18+ 1- 15+ 4.0 25.5 10.5 5.5 10 Narikin Shogi 3- 4- 12+ 14+ 22+ 7- 19+ 4.0 25.0 11.0 5.5 11 Mattari-Yuuchan 2- 9+ 24+ 18+ 4- 8- 14+ 4.0 25.0 10.0 6.0 12 Tohske 5- 14+ 10- 17+ 7- 21+ 13+ 4.0 24.0 11.0 6.0 13 ponanza 6- 17+ 20+ 4- 16+ 5- 12- 3.0 26.5 8.5 3.0 14 Nanoha 15+ 12- 7- 10- 17+ 18+ 11- 3.0 25.0 9.0 3.0 15 Hakusa Shogi 14- 5- 17- 20+ 19+ 16+ 9- 3.0 23.0 8.0 2.5 16 Oki 23+ 20+ 1- 6- 13- 15- 21+ 3.0 23.0 6.0 2.0 17 GAsyou!!! 7- 13- 15+ 12- 14- 22+ 23+ 3.0 20.0 6.0 1.5 18 Suzuki Shogi 20+ 23+ 6- 11- 9- 14- 24+ 3.0 19.0 4.0 1.5 19 Daemon Shogi 9- 2- 22= 21+ 15- 24+ 10- 2.5 20.5 2.0 0.0 20 HIT Shogi 18- 16- 13- 15- 24+ 23= 22+ 2.5 15.0 1.5 0.0 21 Hayabusa 22+ 8- 9- 19- 23+ 12- 16- 2.0 20.5 3.0 0.0 22 Tohu 21- 24+ 19= 3- 10- 17- 20- 1.5 19.0 0.0 0.0 23 Inoue Shogi 16- 18- 4- 24+ 21- 20= 17- 1.5 18.5 0.0 0.0 24 Mangu Shogi 8- 22- 11- 23- 20- 19- 18- 0.0 19.0 0.0 0.0 * KCC Shogi, Monju, Blunder, Yumeki, WILDCAT, Yamada Shogi, Garyu, Auau Shogi and ym-Shogi qualify for the second qualification round.
Ym-shogi programmer Yamada happy with being the final program to go through to the second day
Because of the system that is being used to pair the programs on the second day, I already knew at the end of the first day that my first round opponent would be KCC Shogi and my second round opponent would be Shueso. A very tough way to start my quest for a place in the final. My own assessment was that two wins would put me in perfect position to finally get through to the final eight, one win and one loss would give me good chances and two losses would mean that I would be back to trying to defend my seeded position. Considering my results on the floodgate server, I was pretty confident that I would be able to get 5 wins, but six wins (the minimal needed to make it to the final) would be a tall order. Here is how Spear fared on the second day:
Round 1: Spear - KCC Shogi 0-1
This game looked like it would end in a quick loss for Spear, because KCC was creating a lot of pressure on the right side of the board and combined with the strong Anaguma castle this seemed decisive. However, around move 100 Spear managed to get a counter attack through the centre, weakening the Anaguma in the process. The pawn promotion on move 111 is a horrible move, ending the game immediately in KCC Shogi's favour, but Spear should have brought the promoted bishops in play earlier if it wanted to have any chance of an upset.
This game started very strangely, with Spear sacrificing a pawn on the second move. This is actually a trick opening, but should be followed by moving the rook to 4b instead of 5b as Spear played it. This was an anomaly in the new opening book, which I only discovered during this game. I knew that the opening book still needed tuning, I just didn't have the time to do it before the tournament. Spear basically never recovered from this strange opening because it invited a dubious attack that had no chance of succeeding. Shueso defended easily and then won in the counter-attack. So I lost the first two games, which meant that Spear would probably (again) not qualify for the final this year. No real surprise, but I would have really liked to pick up one of these two games, just to get the juices flowing.
It would have been a good idea to try and fix this opening book problem, but it was not trivial. I feared that I might break something in the process, so I left the book as it was. After all, I figured that there was not much chance of getting the same opening again, as Spear would pick moves randomly from a reasonably large number of alternatives in this position. I would later regret this line of thinking...
The game against Usapyon is interesting for those who want to know what happens if you get your program settings slightly wrong. Usapyon starts playing strange moves around move 17, moving the king back to where it was before. Later, the king even moves to 4g and I went over to ask Usapyon programmer Ike if he was really taking this game seriously. I even (jokingly of course) vowed that I would retire from computer shogi if I would lose against a program that played its king in this way. Fortunately, I didn't have to do that as Spear comfortable won its first game. Later Ike told me that he discovered a problem with reading certain settings from a file at the start of the game. This was fixed later, but too late to defend Usapyon's seeded position.
Jeff Rollason's Shotest used to be a program I very much looked up to. It has finished in third place in the CSA tournament twice, which is something I can only dream of. However, Shotest's development has been slow in the past couple of years and Spear has been able to first catch up and then overtake Shotest. Furthermore, Shotest was running on a slow laptop this year, so it was not really an opponent to be feared. Games that one should win always have a little bit of extra pressure, but Spear did a thorough job getting and extending an advantage. Shotest is very resilient in defending as is clear from this game, but the outcome was never in doubt.
Round 5: Spear - A-Class League 1-0
Strangely, I never really gave losing against A-Class League any thought. This confidence is based on absolutely nothing, as this program is quite strong and blindingly fast because it is running on an FPGA. Furthermore, Spear had a losing record on the floodgate server against this program (although only a few games were played). The game seemed to be going Spear's way early, but when A-Class League managed to weaken the Anaguma the endgame became a little bit of a nail biter. The fear of being mated in one move by a knight drop on 2g was very real (of course the program would not allow this, but discovering this late in the search might cause all kinds of problems). Fortunately, A-Class League moved a promoted rook to this square on move 96 and then Spear could attack anyway it liked. So, three consecutive wins and maybe back with a chance.
I was quite happy with Blunder as my next opponent. I had a very good record against Blunder on the floodgate server and didn't think the CSA tournament would be any different. I was wrong. It turned out that Blunder was running on a very slow computer when playing on the floodgate server and in the CSA tournament was running on very fast hardware. This was a huge difference as the game showed. Spear played a dubious attack that looked threatening but Blunder defended it well and won in the counter-attack. This was probably the most disappointing result of the tournament for me. Losses against KCC Shogi and Shueso were kind of within the realm of expectation, but losing against a first year entry will not do if I ever want to make it to the finals. Until we meet again, Blunder!
On the first day I talked with K-Shogi programmer Honda about the high probability of meeting and indeed we were paired again this year. Although we haven't played each other often in the CSA tournament, I feel a special relation with K-Shogi. Before the floodgate server, K-Shogi was the perfect test opponent for Spear, because the playing level was not so different. Possible improvements could be easily tested against K-Shogi. Therefore, winning against K-Shogi in the CSA tournament last year really felt like a breakthrough. These special feelings were immediately gone when Spear played the same strange opening as in the game against Shueso. So far for randomness saving the day! Spear moved into an Anaguma after that, but the generals on the other side were a horrible weakness that was easily exploited by K-Shogi. Again a loss without much of a fight. Three wins and four losses after the seventh round. Not exactly a dream scenario...
Spear again didn't make it to the final. Slight disappointment showing?
Yumeki is a relative new program and although it is not in general a weak program, in this game one of its weaknesses was exposed. Spear attacked the uncastled white king straight through the center and never gave Yumeki any chance to stop the attack or counter-attack.
Round 9: Yamada Shogi - Spear 0-1
There was the strange opening again! After the tournament, I checked if there was any problem with the random selection of opening book moves, but it all seemed fine. Still, playing the unusual 1.P7f 2.P4d three times and the also unusual 1.S6h twice clearly shows that the opening book needs some rethinking. In any case, against Yamada Shogi the strange opening finally bore some fruit. Yamada Shogi got confused and castled the king on the same side as the rook where it became an easy target for attack.
So, this tournament ended for Spear the same way as the previous two years: 5 wins and 4 losses in the second round. This time this meant 9th place, a slight improvement over last year's finish. Considering the high level of play in this group, this was a reasonable result, but I had hoped for a little more. Well, there is always next year...
Kinoa programmer Yamada Genki showing programmer's fatigue.
What happened to the other programs on this second day? Well, KCC Shogi just bulldozered its way to the finals, winning all nine games. Talk of winning this tournament by winning 23 straight games started to get stronger. It seemed that the closest KCC had come to trouble was the game against Spear, a clear indication of how dominating it was on the day.
Surprising runner-up was Monju, losing against Shueso and KCC Shogi (again), but winning every other game, including Otsuki Shogi, Kakinoki Shogi, K-Shogi, Dragon's Egg and GPS Shogi. Especially the win against GPS Shogi would become something to be proud of.
Third place was for GPS Shogi, which put in a solid performance, losing only to KCC Shogi and Monju, but winning all its other games. GPS Shogi is one of the under-achievers in the CSA tournament. It has a team of very good young programmers, but this time was only the second time it qualified for the finals. Not a surprise, really, because the new version of GPS was doing very well on the floodgate server (which the GPS team created, by the way).
KCC Shogi, Monju and GPS Shogi were the dominating programs on the second day, but there were two more spots in the final left. For a long time it looked like Shueso would easily claim one of these spots. The program started with five consecutive wins, including a win against Monju, but then lost three games in a row, including a game against Kakinoki Shogi it probably should have taken. Because of its strong opponents, it was paired in the final round against A-Class League, which must have been a relief. It won the game, finishing fourth and claiming a place in the final.
A moment of humour: Auau Shogi's king, already in dire trouble against Shotest, then suddenly chose to make it much worse!
The luckiest program of the day was Otsuki Shogi. It started strong with three wins, but then lost against KCC and Monju and also dropping the eight round game against GPS Shogi. In the end, it didn't have its fate in its own hands. It had to beat Dragon's Egg in the final round and hope that K-Shogi would beat Blunder. If Blunder would have beaten K-Shogi, it would have finished higher than Otsuki Shogi. However, both Otsuki Shogi and K-Shogi won, and Otsuki Shogi was the last program to make it to the final. Otsuki Shogi finished with the same number of points as K-Shogi, but because K-Shogi had "walked the low road" (losing early and beating relatively weak opponents after that) Otsuki qualified based on Sum of Opponent Score.
Special accolades go to first year entry Blunder, beating former finalists Bingo Shogi and Dragon's Egg, just missing a place in the finals. Dragon's Egg did more or less as expected, playing for a place in the final until the last game. Disappointing results for Kakinoki Shogi (although Kakinoki himself was happy that his program had a theoretical chance of making it to the final until it lost the last game), Bingo Shogi and especially TACOS, which was expected to do much better. Yamada Shogi had its best result ever and clinched a seeded position, something that Shotest wasn't able to do. Still, Jeff was quite happy that even this version of Shotest running on a slow machine was able to win four games at this level. With some massive improvements in both soft- and hardware (very likely ready for next year's tournament), Shotest could be a program to be reckoned with again next year. Still, it will have to deal on the first day with the likes of Tanase Shogi, Nara Shogi and Kiri.
The Otsuki team had reason to smile after just barely making the final round
Results of the second qualification round
No. Program Name 1 2 3 4 5 6 7 8 9 Pt SOS SB MD 1 KCC Shogi 9+ 12+ 17+ 5+ 3+ 4+ 2+ 7+ 10+ 9.0 49.0 49.0 38.0 2 Monju 6+ 21+ 10+ 4- 5+ 8+ 1- 14+ 3+ 7.0 51.0 36.0 26.0 3 GPS Shogi 20+ 14+ 11+ 15+ 1- 16+ 4+ 5+ 2- 7.0 48.0 32.0 23.0 4 Shueso 12+ 9+ 8+ 2+ 7+ 1- 3- 10- 13+ 6.0 51.0 30.0 19.0 5 Otsuki Shogi 23+ 16+ 7+ 1- 2- 11+ 14+ 3- 8+ 6.0 47.0 24.0 18.0 6 K-Shogi 2- 8- 24+ 13- 20+ 12+ 9+ 15+ 7+ 6.0 37.0 21.0 16.0 7 Blunder 14+ 20+ 5- 11+ 4- 9+ 8+ 1- 6- 5.0 49.0 22.0 14.0 8 Dragon's Egg 21+ 6+ 4- 10+ 15+ 2- 7- 13+ 5- 5.0 46.0 22.0 13.0 9 SPEAR 1- 4- 18+ 19+ 13+ 7- 6- 20+ 12+ 5.0 45.0 19.0 12.0 10 Kakinoki Shogi 24+ 17+ 2- 8- 14- 21+ 11+ 4+ 1- 5.0 43.0 18.0 12.0 11 Bingo Shogi 22+ 13+ 3- 7- 18+ 5- 10- 17+ 15+ 5.0 42.0 19.0 12.0 12 Yamada Shogi 4- 1- 21+ 14- 19+ 6- 22+ 16+ 9- 4.0 44.0 14.0 7.0 13 A-Class League 15- 11- 19+ 6+ 9- 17+ 16+ 8- 4- 4.0 43.0 18.0 8.0 14 TACOS 7- 3- 23+ 12+ 10+ 15+ 5- 2- 18- 4.0 43.0 14.0 8.0 15 Kinoa Shogi 13+ 22+ 16+ 3- 8- 14- 19+ 6- 11- 4.0 42.0 15.0 8.0 16 My Move 19+ 5- 15- 18+ 17+ 3- 13- 12- 23+ 4.0 38.0 13.0 8.0 17 Misaki 18+ 10- 1- 22+ 16- 13- 24+ 11- 20+ 4.0 37.0 10.0 6.0 18 Usapyon2 17- 24+ 9- 16- 11- 20- 21+ 23+ 14+ 4.0 29.0 8.0 4.0 19 Shotest 16- 23+ 13- 9- 12- 24+ 15- 21+ 22+ 4.0 28.0 7.0 4.0 20 Yumeki 3- 7- 22- 24+ 6- 18+ 23+ 9- 17- 3.0 35.0 5.0 1.0 21 WILDCAT 8- 2- 12- 23+ 22+ 10- 18- 19- 24+ 3.0 33.0 4.0 1.0 22 ym-Shogi 11- 15- 20+ 17- 21- 23+ 12- 24+ 19- 3.0 28.0 4.0 1.0 23 Auau Shogi 5- 19- 14- 21- 24+ 22- 20- 18- 16- 1.0 31.0 0.0 0.0 24 Garyu 10- 18- 6- 20- 23- 19- 17- 22- 21- 0.0 33.0 0.0 0.0 * KCC Shogi, Monju, GPS Shogi, Shueso and Otsuki Shogi qualify for the finals.
The Finals
Although Jeff and I took it a little bit easier on the final day, we arrived at the venue reasonably early, with about half of the games still being played. Still, we were in for quite a shock, because the so far invincible KCC Shogi had lost against GPS Shogi and Otsuki Shogi had beaten Bonanza. More was to come, because Monju beat Gekisashi and Shueso won against YSS. This meant that in the first round the four top seeded programs had all lost their games! This had never happened before. Sometimes one of the seeds would drop a game, but in general the status quo of the top programs would remain almost intact. I even took a picture of the score sheet after the first round because it seemed so special. I was much too early with this picture as it turned out...
The second round pretty much went the same way as the first round. GPS Shogi beat Gekisashi, Monju beat YSS and KCC Shogi lost again, this time against Otsuki Shogi. Only Bonanza showed why it was a seeded program as it beat Shueso. These results meant that after only two rounds last year's champion Gekisashi would already need a miracle to defend its title. KCC Shogi's dream of winning the CSA crown also went up in smoke after only two games.
Humans get used to the amazing very quickly, because the third round results seemed suddenly quite natural. GPS Shogi beat Bonanza, Otsuki Shogi beat YSS and Monju beat KCC Shogi. Again, only one seeded program won as Gekisashi beat Shueso. This meant the third straight loss for KCC Shogi, against an opponent it had already beaten twice in this tournament. Time to ask the operator if this was just a coincidence. It turned out that it probably wasn't. It seemed that on the evening before the final round, a new version was sent from North Korea, supposedly fixing some problem with the use of time in the opening. The operator was asked to play the new version in the final, but that probably was a big mistake. Even when the operator suggested to change back to the previous version after losing in the first round, he was told to keep using the new version. Only after three straight losses, the North Koreans seemed to be coming to their senses and allowed the operator to change back to the program that had been so successful on the previous days.
Monju running 6 Bonanza programs on 24 processors.
In the fourth round more of the same. GPS Shogi beat YSS, Otsuki Shogi beat Gekisashi, KCC Shogi beat Bonanza and Monju beat Shueso. So, after this round GPS Shogi, Otsuki Shogi and Monju all had four wins, Bonanza, KCC Shogi, Gekisashi and Shueso had one win and YSS was at the bottom of the table with four losses. The CSA tournament had become predictable again, but this time in completely reversed order! Whatever would happen next, we would have a brand new champion because either GPS Shogi, Otsuki Shogi or Monju would win this tournament.
Round 5 was huge for GPS Shogi. It beat Otsuki Shogi and because Monju lost against Bonanza, it suddenly found itself leading the tournament on its own. There seemed to be some moral justice in a single Bonanza beating six Bonanzas, but when thinking a little about it, it was not really David beating Goliath. After all, the Monju Bonanzas were only running on 4 processors each, while the "real" Bonanza was running on eight. A faster machine without randomness beating slower versions with randomness doesn't seem that unnatural. At the bottom of the table YSS lost again, this time against Gekisashi. With games against Bonanza and KCC Shogi left to play, former champion YSS seemed in serious danger of losing all its games.
Even this special cooling device didn't help YSS.
The sixth round had leader GPS Shogi playing Shueso, which wasn't making much impact in this final round. Most spectators (including me) closely followed the game between Otsuki Shogi and Monju, because if Monju would lose and GPS Shogi would win, GPS would be the new champions with one round left to play. Otsuki Shogi indeed beat Monju and when Shueso resigned a few minutes later we had a new champion: GPS Shogi. A great team effort, again by a group from prestigious Tokyo University (like former champions IS Shogi and Gekisashi). At the bottom of the table also some drama. YSS won its first game, but needed some help to do it. For some inexplicable reason, Bonanza crashed during the game with YSS and lost the game by default.
The winning GPS team with Yonenaga Kunio.
With the champion decided and also the seeded programs decided (Otsuki Shogi and Monju) this exciting tournament ended a bit on a low. The only question that remained was if GPS Shogi would be able to break through the CSA jinx that a champion crowned before the final round will lose its final game. This has happened four times in the past (in the 12th, 13th, 14th and 18th CSA championships). Only Gekishashi managed to win all seven games in the 15th CSA tournament. Monju kept the jinx alive by convincingly beating GPS Shogi in the final round.
Results of the finals
No. Program Name 1 2 3 4 5 6 7 Pt SOS SB MD 1 GPS Shogi 4+ 6+ 5+ 7+ 2+ 8+ 3- 6.0 22.0 17.0 10.0 2 Otsuki Shogi 5+ 4+ 7+ 6+ 1- 3+ 8+ 6.0 22.0 16.0 10.0 3 Monju 6+ 7+ 4+ 8+ 5- 2- 1+ 5.0 23.0 14.0 7.0 4 KCC Shogi 1- 2- 3- 5+ 8+ 6+ 7+ 4.0 24.0 7.0 3.0 5 Bonanza 2- 8+ 1- 4- 3+ 7- 6+ 3.0 25.0 8.0 2.0 6 Gekisashi 3- 1- 8+ 2- 7+ 4- 5- 2.0 26.0 2.0 0.0 7 YSS 8- 3- 2- 1- 6- 5+ 4- 1.0 27.0 3.0 0.0 8 Shueso 7+ 5- 6- 3- 4- 1- 2- 1.0 27.0 1.0 0.0
Conclusions
Three years after its shocking victory in the CSA tournament, Hoki's Bonanza program is still having a huge impact on computer shogi. The "Bonanza Method", a supervised learning method for tuning evaluation function features is now employed by all the top programs in this year's tournament. GPS Shogi, Otsuki Shogi, Monju, KCC Shogi and Bonanza (of course) have all benefited from the Bonanza Method. Gekisashi and YSS tried it, but were unable to improve their programs with the method and the results in this tournament seem to suggest that they might have to change their programs to fit the Bonanza method in order to survive.
Results before the final round: all the top programs losing!
In general the level of the programs has improved considerably since last year. GPS Shogi is estimated to have a rating that is among the 50 best players at the biggest Internet shogi site: Shogi Club24. There are quite a number of professional players anonymously active on this site, so it seems likely that GPS Shogi is already stronger than a number of professional players.
We may get a better indication of the strength of the top program next year. This year there was no exhibition match between computers and human players, but as JSR chairman Yonenaga announced at the party afterwards: it seems very likely that a professional will play a computer at a special event in 2010. Sponsorship is going to be a problem in the current recession, but it seems likely that this event will really take place.
GPS programmer Kaneko explaining to professional shogi player Sato Amahiko.
Another interesting aside was that only a few programs had actually used the code that Bonanza made public in February. Only Monju was using the full code, and most programmers even seemed to take some pride in not even reading the Bonanza code. I read the code and I can assure everyone that there are quite some gems in there that I didn't have time to add to Spear but that I will use for next year's version. As said, Spear's code is very similar to Bonanza, so this is not a difficult task.
I am sure that next year a lot of other programs will be using at least part of the Bonanza code. Bonanza will become the standard and everybody can build its own ideas on top of it. This opens the door for different programs and it will be interesting to see how this new wave of "Bonanza Children" will have an impact on next year's tournament. I truly hope that Spear will be one of the program benefiting from Hoki's generosity.
Yonenaga Kunio doesn't seem too worried about the threat to professional shogi players yet.
After the tournament...
It was only after the tournament finished that news reached me about the release of a new version of Bonanza. This version was released on May 1st, one day after the my university closed for the holidays. Because I had no Internet at home yet, I had no access to this news. Therefore, Spear played the CSA tournament with a version of the previous Bonanza feature file instead of the latest one. This was a big difference. Playing Spear on the floodgate server after the tournament with the new Bonanza evaluation features improved the program dramatically (an increase of more than 100 ELO points).
It is unclear how much impact this would have had in the actual tournament. On the floodgate server most programs play on hardware that is significantly slower than the hardware used at the CSA tournament. Furthermore, because the new Bonanza version was released so close to the CSA tournament, there would have been no time for trying any improvements, both in the learning or otherwise. Spear would have basically been a Bonanza clone, and playing it would be quite strange considering the fact that I vowed last year not to do like Bonanza at all costs.
So, maybe it was better not to know about the new version. I can now use this as a baseline and take some time until the next tournament to try to improve Spear. After the tournament Hoki said that computer shogi programming should be left to students, but after entering the tournament for 13 times, I still have hopes of making it to the final one day. Even at 42, this dream is very much alive!
Reijer Grimbergen: May 2009
