Purpose of Learning

As with any new technology, users can be overwhelmed by its potential and forget to take a critical perspective. The main reason for failure in projects based on learning AI is overuse. AI technology has made great progress over the past few years, but it's still at a stage where it's best kept focused on domain-specific problems.

Therefore, we'll try to approach learning from an engineer's point of view, analyzing what benefits it provides, and at what cost. By trading off both aspects of the technology, design decisions become more obvious.

Why Learn?

There are two main reasons for learning, both of which attempt to find intelligent solutions to problems. The main difference is the goal of the learning; is the desired result static or dynamic? We choose the terms optimization and adaptation to refer to these different cases.

Optimization

Optimization attempts to find the solution to a known puzzle, which does not change over time. We rely on knowledge of all the variables involved in the problem, as well as the criteria to evaluate the solution. In fact, the entire task can be considered as static.

An example of this is obstacle avoidance; the goals are clearly identifiable, and we can develop competent behaviors that do not need to change to deal with new situations.

Adaptation

In the case of adaptation, the problem actually changes over the course of time because of hidden factors affecting the behavior in unforeseeable ways. The goal may be clear, but not how to achieve it. This is the case when an animat enters a fully unknown environment.

Alternatively, adaptation is necessary when the actual goal of the learning changes—regardless of the problem being well identified. If an animat's feelings about certain players change for the worse, for example, the behavior will shift from helping them to making their life difficult.

Successful adaptation therefore involves using the facilities available in new ways, so that unforeseen factors—both changing problems and changing goals—are taken into account.

Similarities and Differences

The static and dynamic nature of the problem is the major difference between optimization and adaptation. It's important to realize that dynamic-ness arises from variables not being modeled. In many cases, information about the full problem is freely available from the game; indeed, the world is at least very predictable, if not fully deterministic (for instance, physics and logic). The designer voluntarily discards much information about the environment, for efficiency and simplicity reasons. In a few rare cases, dynamic problems arise from truly unknown parameters (for instance, emergent phenomena).

So fundamentally, both problems are really the same thing. Static problems are better identified, because most of the variables have been identified, since the problem is small or the designer has spent time on it. Dynamic problems are poorly described by the variables chosen, because the scope of the problem is too large or the designer wasn't able to identify all the parameters.

As far as the AI is concerned, the closer to a static problem the better the final result. For example, we may not want an animat that struggles to adapt to all situations, but rather one that handles a great majority of them by using a good AI optimized statically. Defining a satisfactory static problem is about knowing the variables involved and choosing all (or most) of the relevant ones. Chapter 21, "Knowledge of the Problem," discussed this in depth.

Learn What?

Different forms of learning can be applied to various tasks.

Facts and Knowledge

By observing the world, it's possible to understand it using facts or statistical predictions. "If I jump, I'll fall back down" is an example of a fact. "This platform spends most of its time in its original position" is an example of a statistical prediction. Given enough experience, this knowledge can become surprisingly accurate.

It's also possible to extract general principles from a collection of distinct facts. (For instance, "All objects that are in the air will fall to the floor.") This is a more implicit way of representing knowledge, known as a generalization. It can be quicker to learn, but less accurate.

Behaviors and Procedures

An alternative to learning facts is learning behaviors. Synthetic creatures can learn how to control their movement, to use weapons, and to move through doors. These behaviors can be simple actions as well as complex sequences guided by stimuli from the environment. It's also important for the animats to learn how to apply the behaviors in general situations.

Overlaps in Representation

In some cases, it's relatively intuitive what classes as a behavior, and what classes as a fact. However, there can still be some confusion, because it often depends what words are chosen! For example, shooting (behavior) is partly about knowing where to aim (fact). So conceptually, it's possible to learn behaviors as facts. Conversely, animats implicitly know facts about the world when they perform behaviors.

When it comes down to the representation, things are equally ambiguous between these declarative and procedural forms. For example, a decision tree stores facts but it can be used for behaviors, too. Neural networks can be used for control, but also learn information about the problem.

The implementation of the technique is the best hope of distinguishing knowledge from behaviors. Knowledge learned online (ontogenetic) will be queried by a component that's fixed (phylogenetic). On the other hand, learned behaviors will be handled by a component that changes itself dynamically. Naturally, if we look deep enough it all comes down to "information," regardless of the technology used.

In practice, game developers will almost always learn facts directly, which enables them to debug and optimize the AI component responsible for using this knowledge. It's also much easier when animats can be understood as a form of data with a developer-friendly representation.