Retrospective Overview

With each of the components in place, it's easier to understand the characteristics of each technique and behavior.

Techniques

The subsumption architecture is a popular approach for implementing robot control. Despite all its limitations, it has many advantages for game developers, including explicit control and design simplicity.

Reinforcement learning is a type of AI problem. Reinforcement learning is applicable to a wide variety of designs, because evaluative feedback is powerful. However, the very concept of reward is also awkward in some situations. Specific algorithms are needed to take into account the reinforcement signal and adapt the representation accordingly. Because reinforcement problems are very broad, there are also many types of algorithms—suited to many designs.

Dynamic programming techniques are very efficient and applicable during the game development. They rely on accurate models of the world. On the other hand, Q-learning techniques are capable of dealing with adaptation during the game—without any knowledge of the environment.

The representation used for reinforcement learning is a mapping from state to action. An array allows the learning to converge to the perfect result, but consumes large amounts of memory. Various forms of abstraction are needed to speed up learning and reduce memory usage—at the cost of accuracy.

Behaviors

The deathmatch behaviors created are effective enough to challenge even the best human players. Because the tactics are a high-level component relying on existing capabilities, the result appears realistic regardless of the quality of the strategy.

Subsumption has the advantage of being designed explicitly, making it easier to adjust. Reinforcement learning finds behaviors suitable for the animat's mood, but could be adjusted to find optimal tactics.

The accumulation of reactive components produces a system that can rival a planning solution—even to the trained eye. Animats with such reflexes are suitable as deathmatch opponents, especially with adaptive skills. However, there are cases that the reactive behaviors cannot handle because of the lack of strong world models and deliberative thinking.