Background

This blog post describes techniques to improve the workflow of rendering non-layered materials on the Nintendo 64 console using its hardware rasterizer Reality Display Processor. Original implementation and discussion on this topic is avaiable on N64Brew discord server pinned as a source below.

This post is aimed at homebrew developers for the Nintendo 64 as a informational guide on this topic. But it doesn't cover the basics of material creation such display list setup, different RDP/RSP modes, Z-depth, or obtaining the textures.

(in this post non-layered material means a material which requires no more than one set of overlaying triangles issued to RDP for rendering)

Introduction

This post in an example of the usage of a unified way of handling diffuse and specular layer of a single material within N64's hardware restrictions with the ability to have at least 1 colored bumpmapped light. Having a good understanding of this subject can not only improve workflow of creating materials but also make the work of homebrew developers much more impressive.

Below is the theory behind this example of this advanced grate material on the N64:

Technical information

With current shading models different layers are rendered and combined together for a final result, this includes for example adding specular information on top of diffuse information gathered by the renderer for a more realistic image.

The N64'S RDP offeres a limited set of shading instructions (color combiner/color blender) on how to assemble the final result in 1 or 2 passes (called cycles). This work tries to utilize them to their full potential.

The Nintendo 64 doesn't have any hardware support for traditional normal maps or bump maps and their respected lights and reflections. But with clever use of the RDP capabilities we can at least try to mimic their behaviour.

The Material

For this material, this set of textures is used (64x64 I4 and 32x32 RGBA16):

Tex 0 (diffuse texture)

Tex 0 Alpha (specular map / bumpmap)

Tex 0 Alpha (inverted)

Tex 1 (environment texture)

Notice how the first and second textures are the same. That way only one of them has to be loaded in a I4 format in order to utilize both and save precious 4kb TMEM.

Also you can choose to use inverted alpha texture as specular map and/or bump map. In this case due to math behind the calculations the inverted texture is the one that looks like a bumpmapped result.

This doesn't mean that only this arrangment of textures can be used. In fact you can use different combinations such as two RGBA textures, IA textures and even CI textures (although without the specular component).

Building Blocks

Lets disassemble the material into visually organized instructions for RDP:

And here is how the color combiner and color blender are configured as seen in Blender with Fast64 plugin:

As you can see this is a 2cycle non-layered material where each section and each cycle of the CC/CB has its own calculation that gets used in some way

Also note that this is not the only possible configuration of the CC/CB that would give this kind of result. You can shuffle around the Prim/Env color and alpha registeres around in order to adjust the calculations for your material in particular and you can choose to use the second cycle of CB for your own needs or even swap the cycles around for intresting effects.

(Maybe you dont need a trasparent material but an emissive one for example. Then you would for example swap the framebuffer input for a fog color register).

Also you can choose to not use the specular component and alpha blending all together and you're left with a 1cycle diffuse material with bumpmapped light.

Let's examine those different calculations more closely.

Diffuse component

This is the first cycle of CC for the RGB color channel. Its main purpose is to calculate the diffuse component for our material by using multiplication between the Texture 0 and the input Vertex Colors.

Note the inclusion of Prim Color register as a B component in the calculation. You can use this technique in order to give BW texture some subtractive color information. In this particular material it is not used and the Primitive Color RGB is set to 0 or black.

Fresnel

Also note that in the visual diagram you can not only see the lights in our scene but also what appears to be fresnel effect. This effect is used to define the edges of our models and scenes. You can also see this effect in modern Physically Based Rendering solutions.

Although it is used in a diffuse component calculation instead of more realistic Specular component as to simplify calculations and to give room for the next calculations.

This fresnel effect is achieved by using some avaiable directional lights, configured on CPU to always look at the camera. Then the RSP performs lighting calculations as usual with those lights enabled.

Here is a code snippet for lights configuration:

/* Calculating the fresnel direction to our global lights */
  Vec3 lightdir; // Direction away from the camera
  { // Light 1 (Left)
    lightdir.x = cos(cam.yangle - 1) * cos(cam.xangle); // calculating direction away from the camera
    lightdir.y = cos(cam.yangle - 1) * sin(cam.xangle);
    lightdir.z = sin(cam.yangle - 1);
    v_world_lights.l[2].l.dir[0] = (u8)((lightdir.x * 128)); // applying the direction to the light's dir
    v_world_lights.l[2].l.dir[1] = (u8)((lightdir.y * 128));
    v_world_lights.l[2].l.dir[2] = (u8)((lightdir.z * 128));
  }
// this same code is performed for multiple lights to cover multiple directions for a fresnel effect
...

Specular Mask

This is the first cycle of CC for the Alpha channel. Its main purpose is to calculate the specular mask for our material by using Texture 0 Alpha and some adjustments with Prim and Env alpha registeres.

This block is where you can change a lot of stuff to make your own material more instresting since there's no defined way of how the specular mask should be made. This specific material uses the setup in this picture since its gives a broad control of the mask and the non inverted texture looks better in this case when applied.

You can shuffle the registeres around, invert the alpha texture and change the parameters for the Prim and Env alpha registeres.

Specular Component and CC Output

This is the second cycle of CC for the RGB Color channel. Its main purpose is to calculate the specular component for our material by multiplying the previous calculation and the Environment Texture together (Texture 1, note that in second cycle, tex0 and tex1 are flipped).

Also this calculation applies the specular component to the diffuse component using addition and the result is sent as a CC Output.

Note that the Combined Color Alpha which is the previous calculation of a specular mask can only be used as a C coefficient in the formula due to hardware restrictions.

The fact that alpha calculations can be used in color calculations is very intresting for the Color Combiner and is a potential use for many intresting effects.

Bumpmap

Bumpmapped light component

This is the second cycle of CC for the Alpha channel. Its main purpose is to calculate the bumpmapped light component for our material by using Texture 0 Alpha and the input Vertex Alpha with some adjustments with Prim and Env alpha registeres.

Bumpmapping's job is to make the geometric shape of the model appear more detailed than its underlying triangles.

Even though the basis for bumpmaps is just subtraction with clamping, this is a fairly complicated step of material creation. But understanding it ensures that you can add good looking bumpmaps to your materials in your creations.

First of all it is necessary to understand the problems of overlfowing or underflowing in the RDP pipeline and why the bumpmapping is done on the alpha channel instead of more usual RGB:

The RGB color information is stored as 24-bit with unsigned 8-bits for each channel and each channel of a color can only have possible values from 0 to 255. This means that when subtraction or addition is used, two colors can result in a color in which one or more channels has overflowed or underflowed which results in glitched pixels. But this is not really the case for the Alpha channel inside RDP's CC. Its values can range from approximately -0.5 to 1.5 with negative values or values over 1 clamped to 0-1 (0-255) range for the result.

Since its necessary for the bumpmap effect to use subtraction with clamping, it is impossible to have meaningful bumpmaps on RGB without at least some kind of underflowing. Thus this bumpmap effect is done on the alpha channel.

Also another reason for doing the bumpmap on the alpha channel is to give more room for calculations on the RGB portion of CC for diffuse/specular layers.

Bumpmap texture

Let's look more closely at creating the bumpmap texture:

It is a single channel grayscale texture. For simplicity's sake you can interpret the brightness level of each pixel as a theoretical inverted height of the pixel. (inverted because this texture is subtracted from the vertex alpha and not the other way around)

But more accurately different gradients of adjacent pixels define the slopeness of the surface when light for those pixels is calculated. You can see different gradients at the edges of the metal grate. In this case the slope is directed towards right-top and bottom-right in parts of the texture and is fairly flat in other parts.

Since there's only one grayscale channel, the result can only be fully accurate in a single direction and less and less accurate in others. In the opposite direction the bumpmap will appear inverted where bumps will become crevices. This unwanted effect cannot be fully removed but It is less pronounced in organic materials. Also the placement of the lights and UV alignment to those lights can help in achieving correct results.

Let's look at this picture with this texture applied:(Before the manual UV fix to the light and after)

On the left image the bumpmap effect appears correct only on the upper-right and bottom-right side of this image, while on the opposite side all bumps became crevices. While on the right image the bumpmap effect is mostly correct.

Underflow

Even though the alpha portion of the Color Combiner has a wider range than RGB channels, underflowing can still take effect, or even worse the bumpmapped light can appear in unwanted places.

There are a few ways to ensure correct result with no underflowing:

1. It is a good practice to limit the brightness of the bumpmap texture. In my practice making sure that all pixels lie within brightness levels of from 25% to 75% gives good results. Also that helps in exporting textures with alpha to .png since pixels with 0 alpha can lose their color.
2. The Prim and Env alpha registeres can help remove the underflow and adjust the overall light levels. The Prim alpha register in thismaterial limits light levels and the Env alpha register adds some light to ensure no underflow. In my practive keeping the Prim alpha at around 200 and the Env alpha at around 50-70 gives good results especially with the first advice applied. Keep in mind that if the Env alpha is too high some light can bleed into the shading regardless of input vertex alpha.

Vertex Alpha

For the bumpmap effect to work it needs a correct input of vertex alphas. Vertex alpha is interpreted as a light level for the bumpmapped light where 0 means no light and 1(255) means full light. This also means that the usage of fog is not possible since it too uses vertex alpha as an input.

There are 3 main ways of obtaining light information into the vertex alpha:

1. Static light information baked into the vertex data
2. Dynamic light information done on CPU
3. Dynamic light information done on RSP

The former approach is the easiest and requires no additional code and has no performance impact, but all lighting information is static and is painted beforehand.

The latter approach is the hardest and requires custom microcode in order to work. No current microcode at the time of writing supports sending light information to the vertex alpha.

Calculating light information on CPU is a possibility but is not recommended since there can be significant performance impacts for large scenes with many triangles and bumpmapped light sources.

Still if you wish to do so, here is an approximate code in order to generate basic lighting information used for the first demo.

void calculate_vertex_alpha (Vtx* vertices, int vamount, Vec3* lightdir){  Vec3 dir;
  for(int v = 0; v < vamount; v++){
    dir.x = ((float)-vertices[v].n.n[0]) / 128;
    dir.y = ((float)-vertices[v].n.n[1]) / 128;
    dir.z = ((float)-vertices[v].n.n[2]) / 128;
    vec3Normalize(&dir, &dir);
    float nfactor = (clamp01(vec3Dot(&dir, &lightdir)));
    vertices[v].n.a = (char)(nfactor * 255);  }
}

Shading result (Applying Bumpmap)

This is the first cycle of CB of our shading result. Its purpose is to apply bumpmapped light component to our final result. This is done by linearly interpolating between the CC Color Output and Blend Color register with the factor of CC Alpha Output.

This gives a convincing result for a bumpmapped light for fairly bright colors inside Blend Color register and it doesn't suffer from any overflow effects. Also doing it this way allows for the bumpmapped lights to be fairly bright and not limited by vertex colors or diffuse texture.

The other way of applying bumpmapped light is to linearly add the Blend Color registed multiplied by the CC Alpha Output to the CC Color Output. Doing it this way is more accurate but is very prone to overflowing.

This single color defined by the Blend Color register can be adjusted for each material. For example if you bumpmapped light color is white, you can adjust the blend color per material to make it appear that color instead of white.

Take a look at this example where different materials appear differently colored despite only being lit by a single bumpmapped light source.

Also on the second cycle of CB you can include alpha blending with the single factor of Blend Color Alpha.

Textured and colored Bumpmapped lights

There are several ways to have unlimited number of bumpmapped lights that have both their own color and diffuse texture. They all require changing the Color Blender to the appropriate settings.

Those textured and colored bumpmapped lights assume their color from the Color Combiner where typically lighting is done by multiplying Vertex Colors with the diffuse texture. We can use that information for the Color Blender calculations.

Also its important to note that any direct lights you wish to be bumpmapped should have accompanying Vertex Alpha as brightness to such light as mentioned earlier. But it should not have any ambient light information or bounce light information (i.e anywhere there's no direct light it should be set to 0).

The first way requires the RDP to be set in 2-cycle mode and it guarantees no overflow and glitched pixels. Here is the setup for it:

In the first cycle we half our CC input by half using For Alpha (set to 128). That way we can limit the range of the CC input to have head room for the second cycle.

In the second cycle we add a set Blend color to our pixel using Bumpmapped light component. The Blend color brightness should be set to the inverse of the Fog alpha (i.e 127 in this example). The reason why the CC's color should not be used instead because the resulting bumpmap effect is hard to spot in practice.

The Blend color register should be set to a shade of grey to have no bias for certain light colors although its not required.

Since we have used an additive blending the resulting color has the hue of CC's output, but we have sacrificed a bit of saturation as the disadvantage to the method.

An example material with this technique:

The second way is to multiply the CC's output with the Bumpmapped light component. This will create textured bumpmapped lighting that you can use in very dark scenes or night-time scenes.

This works in 1-cycle mode and has no overflow. Also it doesn't suffer from the reduced saturation the first method gives.

But it can be improved by adding a certain Blend color with the bumpmapped light component as in the first method. However this method also ensures that there is no ambient light at all (or very little because of the CB's limitations) in the result.

An example material with this technique:

The third way is to just add a certain Blend color to the CC's output with the bumpmapped light component as done in the first method.

This assumes that the range of CC's output is already limited (for example by a dark diffuse texture or by multiplying with a dark color).

This works with the ambient light and it can be quite bright if done correctly, but ultimately it sufferes from the possible overflows if the Blend color is set too bright. This method also reduces the saturation of lighting as mentioned in the first method.

An example material with this technique:

Bumpmapped light limitations

1. If no textured bumpmapped lights are applied, each material can have only one RGB color for its bumpmapped light when its applied in the Color Blender. The bumpmapped light in this case is calculated on Alpha channel and is applied using a single color register. This is fine when you have only one bumpmapped light in the scene, but can be a problem when you have multiple lights on a surface.
2. If no textured bumpmapped lights are applied, Applying the bumpmapped color in a Color Blender disregards any diffuse information and does not make use of a diffuse texture. That means if the object is only lit by such light with no other light sources or specular information, it will appear in the gradients of a single color. There is no such limitation when the textured bumpmapped lights are used.
3. Bumpmapped lights do not change the underlying surface and do not interact with specular information.
4. Bumpmapped lights cannot be used while fog is enabled and vice-versa.
5. Bumpmapped lights should only be used on opaque surfaces or alpha blended surfaces with a single blend alpha per material.
6. Bumpmapped lights require an alpha texture with adequate amount of brightness levels. This means that only RGBA32, I,IA, CI (in IA16 mode) textures can be used effectively.
7. Bumpmapped lights require information stored in vertex alpha colors.

Environment

Environment Texture

In the cycle 2 of RGB calculation we applied the specular component to our diffuse component. This is where the Environment texture comes into play.

This Environment texture represents the scene around our object and in an ideal situaiton would represent the reflections accurately

Unfortunately on the N64 we can only use limited methods on how to represent the environment scene for our material. There are two main ways of obtaining such texture: static painted texture (of a scene around the object) / using framebuffer as a environment texture

The latter requires shrnking the framebuffer image to the size which can fit inside TMEM which is not discussed in this topic.

The Environment texture also uses the technique known as Environment Mapping which is discussed below

Environment mapping is used to define the placement for our environment texture around our model depending on the camera view and model's normals.

There are 2 main ways that this technique can be achieved on the N64:

1. Using G_TEXTURE_GEN / G_TEXTURE_GEN_LINEAR flags

This is the officially supported way of handling Environment mapping and is done on RSP by modifying the UV coordinates of sent triangles depending on camera view and model's position and normals.
This technique is used for many materials in a lifespan of N64 by many games and developers.
Pros:

1. More accurate representation of the scene around the model with automatic inclusion of the camera view and model's normals into the calculation
2. Requires no custom code

Cons:

1. Modifies the only avaiable UV map for triangles that are sent to the RDP which means that the resulting UV map cannot be used for anything else

This one downside is enough to make this material with textured diffuse and bumpmapped light impossible to make.

2. Using tile shifting

While N64 only allowes only one UV map for the RDP, it has independent tile descriptors for all textures used.

This tile descriptor allows simple UV manipulations such as translate/power of 2 scale.

Below is the code snippet to calculate correct tile shifts for this particular material. Note that you can (and often should) customize it in order to have more accurate mapping for different objects in different positions and angles.

/* Setup the environment mapping by using tile-shifting
  ====================================================
  ====================================================*/
  gDPSetTileSize(glistp++, 1, (int)(cam.yangle*100), (int)(cam.xangle*100), 124, 124);

Pros:

1. Can be used with multiple textures
2. Doesn't modify the underlying UV map
3. Can be used in different ways for different effects

Cons:

1. Less accurate mapping results in a less convincing environment around the object
2. Requires careful control and balance of the tile calculations
3. Requires custom code to be made
4. Is prone to tearing and incorrect shift directions when different UV islands do not allign or are rotated

This material uses tile-shifting in for its calculations.

Examples or such materials on the N64

Notice how the reflections are made separate from bumpmapped lights

Demonstration of colored bumpmapped lights