Embedded Languages

In the preceding section, you got an idea of how to create a new programming language from scratch using simple tools. But most projects would do with a regular, C-like language, and quite likely developers do not see much added value in coding the language themselves. It would be great to have off-the-shelf languages designed specifically with scripting in mind. Embedded languages are exactly that—programming languages designed specifically to be called from a host application, much like the way plug-ins work. Thus, they provide both the internals of the programming language and an API to communicate back and forth with the host application. This way you can start with a full-featured language instead of having to code your own, which is time-consuming, and most of the time, does not make a difference in the final product.

Embedded languages exist in two fundamental flavors. Some are specifically designed to be embedded, such as Python or Lua. And others are just regular programming languages that can be embedded by using special tools. The Java Native Interface (JNI), for example, allows Java code to be executed from a C/C++ application. We will now explore one of each kind in detail. Let's begin with Lua, which is one of the most popular embedded languages.

Learning Lua

Lua is a lightweight programming language developed at the Pontifical Catholic University of Rio de Janeiro in Brazil. It follows a BASIC-like syntax with dynamic typing and can be compiled into efficient bytecodes, which can then be run from either a command-line interpreter or, more often, from a C application. Lua is very popular among game studios because it includes a great feature set while providing a low learning curve. It was used in games such as Grim Fandango, Escape from Monkey Island, Baldur's Gate, and Impossible Creatures.

We will now explore Lua programming and integration with your game code. For the rest of this section, it is recommended that you download the latest version of Lua, which can be found at www.lua.org.

Just make sure you download the precompiled binaries for your platform of choice, or the source code, should you want to build the language for a different platform. You only need the include and library files as well as the associated DLLs. Make sure you also grab a copy of the reference manual so you can learn all about Lua.

Programming

In this section, I will try to provide you with a brief overview of Lua as a programming language. Coding Lua scripts is really straightforward if you are fluent in high-level languages. The syntax is not very different from BASIC or C. You can code loops, if-then-else sequences, define and call your own functions, and so on. There are some specific syntax rules, but overall you should feel right at home using the language. Take a look at this example:

function calculate(a,b)
    local temp
    temp=a+b
    return temp
end

c=calculate(1,2)
if c==0 then
    result="zero"
elseif c<0 then
    result="negative"
else
    result="positive"
end
print("the result is: ",result)

The example adds two numbers using a subroutine and prints whether the value of the result is positive, negative, or zero. The syntax is really straightforward and can be learned in a matter of hours, not days. But Lua scripts are nothing but a curiosity if we do not embed them into actual binary code. The possibility of exchanging data back and forth between the two languages is where Lua's power really exists.

Integration

From the host application's standpoint, a Lua script is just a regular function you can call. You can pass parameters and retrieve results, and the function can have an internal state so subsequent calls to it store previous values. The main difference is that the function is an interpreted script, so it runs in a safe environment, can be modified without affecting the core code, and so on.

As an example, let's see how we can call a Lua script from C++, with no parameters or results involved. The code is really straightforward. We initialize Lua, make the call, and shut down the language interpreter:

lua_State *ls=lua_open(100);

int result=lua_dofile(ls,"d:/lua/bin/work/hello.lua");
lua_close(ls);

The first line is used to initialize both the Lua stack (which we will use in a second to send data back and forth with the script) and the lua_State variable, which is the main Lua object. Stack size should be at least 100, with larger values for scripts with lots of variables or recursive behavior. This stack is associated with the lua_State object, which is required by all calls to the Lua C API. Once Lua is up and running, the second line is used to execute a Lua script by simply passing the file name. Notice that the file can either be a pure Lua script or precompiled for higher speed. The function call should return 0 if everything goes well, and a host of error codes otherwise. It is interesting to note that Lua scripts can be run either from a file, as shown in the preceding code, or from a user-defined string using the interface:

lua_dostring(lua_State *, const char *);

Whichever the case, lua_close should be called as soon as we are done with Lua to shut down all data structures in memory.

However, running Lua scripts is useless if you can't exchange data with them. Lua is great for AI scripting; thus, we will need the results of the AI processing or any other behavior we choose to implement. Communicating with Lua is made possible by a stack where we can push parameters from the C/C++ side, so Lua can retrieve them. Once the Lua script is finished, it can dump its results to the same stack, so we can read them back from C++ code.

In the following example, we will pass an integer to a Lua script, and it will return a different string for each number. To begin with, here is the C++ code to interface with Lua:

lua_State *ls=lua_open(100);

lua_pushnumber(ls,3);
lua_setglobal(ls,"index");

int res=lua_dofile(ls,"d:/lua/bin/work/hw.lua");

int i=lua_gettop(ls);
const char *s=lua_tostring(ls,i);

lua_close(ls);

The first line is again the Lua initialization sequence. We then pass the integer 3 to the stack. To do so, we first push it to the stack, and then we use the setglobal call to assign the value at the top of the stack (the integer 3) to the variable we pass as a parameter—in this case, index. After these two calls, Lua knows it must assign the value 3 to the variable index. Then, we call the script, which we will examine in a second. For now, let's look at the different interfaces to send data to Lua:

lua_pushnumber(lua_State *, double);
lua_pushstring(lua_State *, const char *);

Once we are done, it is time to retrieve the results from the execution of the Lua script. To do so, we first call lua_gettop, which returns the position index of the element at the top of the stack. If the stack only holds one element, it will have the index 1, and so on. Then, we can directly retrieve the stack element value by passing its index to the lua_tostring call. We can directly use tostring because we know the element we are trying to retrieve is a string. But here are other options:

double lua_tonumber(lua_State *, int);
const char *lua_tostring(lua_State *, int);
int lua_strlen(lua_State *, int);

From the Lua side, the code associated with this example (the hw.lua script) is also really simple. Here it is:

if index==1 then
        do
        return "Hello world 1"
        end
end

if index==2 then
        do
        return "Hello world 2"
        end
end

if index==3 then
        do
        return "Hello world 3"
        end
end

if index==4 then
        do
        return "Hello world 4"
        end
end

if index==5 then
        do
        return "Hello world 5"
        end
end

Notice how we do not set an index to any value at the beginning of the script. The variable will already have a value set by the C++ code. We can thus use it as if it was already initialized. Then, to send a result value to the calling application, we just need to use the return function, which, in this case, returns a different string for each if-then block. These strings will then be read back from the stack by the C++ application.

A corollary to the preceding example is how do we return more than one result from a Lua script? The return call is clearly limited to single variables, and many times we will need to return more complex data structures. To do so, we only need to read Lua variables back using the getglobal call. The getglobal call receives the name of the Lua variable to retrieve and pushes it to the top of the stack so we can read it back. Here is a very simple example:

lua_State *ls=lua_open(100);
float x=5;
float y=6;
lua_pushnumber(ls, x);
lua_setglobal(ls,"xvalue");
lua_pushnumber(ls, y);
lua_setglobal(ls,"yvalue");

int res=lua_dofile(ls,"addition.lua");

lua_getglobal(ls,"xvalue");
int i=lua_gettop(ls);
x=lua_tonumber(ls,i);

lua_getglobal(ls,"yvalue");
i=lua_gettop(ls);
y=lua_tonumber(ls,i);

lua_close(ls);

Notice how getglobal pushes the variable id to the top of the stack. Then, a gettop – tonumber sequence must be used to retrieve its value. Here is the very simple addition.lua script:

xvalue=xvalue+1
yvalue=yvalue+1

The retrieved value must be incremented.

User-Defined Functions

Lua is an extensible language. If we provide a function in our game code, we can make it callable from Lua. This is a great way to customize the base language so newer, more powerful constructs are supported. Games like Baldur's Gate ran on highly customized versions of Lua, taking advantage of all core constructs such as if-then-else and for loops, but adding new routines as needed. Let's look at a complete example on how to add a new routine to Lua.

To begin with, we must implement the C function, which must conform to some Lua coding standards. The function will receive all its arguments coming from Lua by using a lua_State structure and will send back its results by pushing them in the same state used to retrieve arguments. The number of arguments is passed on the top of the stack. Then, stack positions from 1 to the number of args store the arguments. Once we have the arguments ready, we can carry out whichever task the routine has been designed for. Just remember to push the results to the stack when finished. Also, inform Lua of the number of results to expect using the standard return C call. Take a look at this function, which returns the life level and gender of a given character, assuming we have indexed them by a serial number in a large character array:

static int lifelevel (lua_State *L)
{
int nargs=lua_gettop(L);
int characterid=lua_isnumber(L,1);

// here we access the game's data structures
int result=character_array[characterid].life;
lua_pushnumber(L,result);
int result2=character_array[characterid].gender;
lua_pushnumber(L,result2);

// the next line tells Lua there were two results
return 2;
}

This is just a very simple function to expose the binding mechanism. But the philosophy is obvious: Define C functions that grant the Lua script visibility of in-game symbols and values. Now it is time to register the function so it is callable from Lua. This is achieved by using a single line:

lua_register(L, "lifelevel", lifelevel);

And that's all really. Just make sure you register the new function before you actually execute any Lua code that tries to use it, or the program will fail. Thus, the correct sequence would be as follows:

lua_open
lua_register
(...) pass arguments
lua_dofile
(...) retrieve results

Remember, Lua pushes values in direct order. Thus, a Lua call like the following expects the C side to first retrieve id1, and then id2 from the stack:

Life, gender = lifelevel(id1,id2)

As far as results go, notice how Lua supports multiple assigns. In this case, we must first push Life, and then Gender, when creating the results from the C side.

Real-World Lua Coding

I'd like to complete this section with a brief overview of what a real-world Lua-based script might look like. Lua is an interpreted language. This has the benefit of the scripts running in a safe environment, and because each script is a separate entity, Lua is able to build large AI systems consisting of thousands of scripts. Unsurprisingly, Lua was used in Baldur's Gate, one of the games with the largest AI in history. On the other hand, a binary, hard-coded approach would have been better in terms of efficiency, but would not have scaled well. Imagine the whole nonplayer character cast from Baldur's Gate in C++ code. So, the idea is to choose the best of both worlds. Enrich Lua with some C++ side functions, which are precisely those that require higher performance: complex geometrical tests, database accesses, and so on. Then, expose them through an API to Lua so programmers can use these new calls. This way your Lua code will be more elegant and performance will increase.

Another interesting idea is to use Lua as an action selector for a state-machine-based language. Code a state machine in C++, which does not really have transitions, but just a switch construct with lots of behaviors. Then, build some C++ queries visible from Lua that allow you to perform action selection, thus, setting the state for the C++ state machine. The Lua code could look like this:

if EnemyCloserThan(Myself, 10) then
      Attack()
else
      Stay()
end

Here EnemyCloserThan would be a query receiving a numerical character id and a threshold distance. The Myself string is a Lua variable that has been loaded as an argument from C. The C caller pushes the enemy identifier so it can perform its own tests. Notice how Attack and Stay are two behaviors, which we are selecting with the script. Attack involves path finding and seeking contact with the enemy, and Stay is just an idle behavior. The C++ code has the behavior implementation details, which are executed efficiently, but the task of selecting the right action for each moment is left to Lua. As a more involved example, we can have a behavior collection consisting of many different behaviors, and by changing the Lua script, recode the personality of the AI. This way we would only have a single enemy class, because behaviors are similar at all times. Then, Lua would customize each state machine and make sure the AI is rich and varied.

Java Scripting

Lua is a very popular choice for game scripting. We will now focus on an alternative option, which is to use the Java programming language as an embedded script interpreter. With Java, you get a multiplatform, object-oriented language, which has more libraries and built-in routines than you will need in your whole life. Overall, the option is not so different from Lua as you would expect. Both are embedded languages that can exchange information with the host and both support user-provided routines on the host side to be called from the script. So, in the end, differences are more related to specific language features and not so much about the design philosophy. Java will probably be more powerful than Lua, but is that extra power really needed for game AI? As usual, there are lots of opinions about this, and most times it all boils down to personal preferences.

The main difference between Lua and Java is that the former was designed to be embedded, whereas the latter is multipurpose. Thus, a specific tool set is required to make Java embeddable. We will not worry much about Java here. There are tons of books on programming in Java, and the specifics have little to do with actual game programming. We will just cover the basics of embedded Java, focusing on how to connect a Java module to an application, which is achieved with the Java Native Interface (JNI).

Java Native Interface

The JNI is a specific set of calls within the Java programming language that makes integrating Java and classic compiled languages such as C or C++ easier. The mechanism is bidirectional. From one side, a Java program can call a C/C++ routine, and thus extend its functionality. This is especially useful when the Java code needs to access platform-specific functionality. It is also interesting if you need to access legacy code that is already written in a compiler language or, even better, if you need to perform time-critical tasks that are more efficient in a low-level language.

On the other end of the spectrum (the end we will be more interested in), a C/C++ program can access methods written in Java using the Invocation API. Under this model, we have a C/C++ program, which we want to extend using Java. We can examine Java objects, call methods, or even embed the whole Java Virtual Machine in our game code for full flexibility and efficiency. Clearly, this is what we will be interested in as far as scripting goes—providing a clean method for scripts written in Java to be called from our C++ code.

Let's begin with a very simple example: a C program that calls a Java routine via JNI. The Java code is just a hello, world application, but we will use it to understand the JNI boot sequence and overall operation. Here is the code from the C side:

#define USER_CLASSPATH "." // where Prog.class is

void main()
{
JNIEnv *env;
JavaVM *jvm;
JDK1_1InitArgs vm_args;
char classpath[1024];

vm_args.version = 0x00010001;
JNI_GetDefaultJavaVMInitArgs(&vm_args);

// append where our .class files are to the classpath
sprintf(classpath, "%s;%s",vm_args.classpath, USER_CLASSPATH);
vm_args.classpath = classpath;

// create the java VM
jint res = JNI_CreateJavaVM(&jvm,&env,&vm_args);
if (res < 0)
      {
      // can't create the VM
        exit(1);
      }

jclass cls=env->FindClass("Prog");
if (cls == 0)
      {
      // can't find the class we are calling
        exit(1);
    }
jmethodID mid=env->GetStaticMethodID(cls,"main", "([Ljava/lang/String;)V");
if (mid == 0)
      {
        // can't find Prog.main
        exit(1);
      }

env->CallStaticVoidMethod(cls, mid, args);
jvm->DestroyJavaVM();
}

Now, here is the content of Prog.java, which is the module called from the previous C++ code:

public class Prog {
 public static void main(String[] args) {
  System.out.println("Hello World %d\n",args[1]);
 }
}

The operation is pretty similar to Lua. We first create the virtual machine. But we need to be very careful with the CLASSPATH, or the virtual machine will not boot. Then, the FindClass and find GetStaticMethodID are used to retrieve the class compiled Java file and the method within the file we want to call. In our case, we are just trying to call the main function. Then, all we have to do is execute the JNI call. We pass the class and method identifiers and a number of arguments that will be retrieved by the Java module.

As with Lua also, all you have to do from here is establish an interfacing method that passes arguments and retrieves results from the C/C++ side. To pass arguments, the following code can be used:

jstring jstr = env->NewStringUTF(" argument");
args = env->NewObjectArray(1, env->FindClass("java/lang/String"), jstr);
env->CallStaticVoidMethod(cls, mid, args);

Notice how we declare a UTF8 string, which is the kind of string used internally by the Java Virtual Machine. Then, we need to put it in an array, because the standard command-line argument mechanism calls for arrays. Additionally, we use the created array in the CallStaticVoidMethod array, so the standard argument passing mechanism can retrieve the arguments from the Java side. Notice that CallStaticVoidMethod was used as the method we were calling and is of type static void. Other variants exist to adapt to all the possible return values. For example, if we need to retrieve a floating-point value from C as a result of a computation performed in Java, the method would be

jfloat jf=env->CallFloatMethod(cls,mid,args);

So we get the return value in the jf variable. Also note that the method we used to locate the Prog.main routine can be changed to adapt to any routine you need to execute. This way you can have a Java file that is just a library of useful routines, which get triggered from C.