Trespasser File Formats

Disclaimer: The information contained in this document is provided exclusively for Your Own Use in order to enjoy Your Legal Copy of the game Trespasser. You must not use this information in any way that would break the conditions of the Trespasser License Agreement.

This document contains the latest known information about the file formats used by Trespasser, and was written by Andres. It is based on the original formats.txt document created by PenguiN42, TSOrd, and Andres. Andres last updated it on 24 July 2003. Now it's being updated and maintained by machf.

Unless otherwise specified, all numeric values are stored as a 4 byte integer, henceforth called a long (possibly unsigned, which is mentioned explicitly if the distinction is important).

Contents

Info about Trespasser, and conventions used in the levels:
  Directory structure
  File locations and naming
  Model and Instance names
Format of the data files:
  GROFF Block structured file format used by GRFs, SCNs and Savegames
  GRF - Main level file Models, instances, scripting
  SCN/Savegame - Scene file (level startup) Savegame state, including spatial partition tree
  PID - Packed Image Directory (Texture index) Texture information
  SWP/SPZ - Swap files (Texture data) Raw and compressed textures
  TPA - Trespasser Packed Audio file format PCM/ADPCM samples
  WTD - Terrain Wavelet compressed terrain
  SMK - Video
  DDF - Menu

Directory structure

File locations and naming

Model and Instance Names

Some conventions are required by Trespasser - the level will not load if these are not followed:

• The instance names must each be unique, as they identify blocks in the scn and savegame.
• The names of all instances of a model must be identical to the model name, except for the optional postfix number (eg. "-00","-01","-999"). The postfix numbers don't have to be consecutive, just unique.

Some conventions affect the way the level works:

• Objects with names beginning with '$' are called $ objects, or physics objects. $ causes an object to be invisible, so it need not specify Visible=false in the valuetable (which is important for type1 geometry objects, because they will crash Trespasser if they are Visible objects).

The following conventions are found in the Trespasser data files, however they are not required by the game engine. (I assume they were used to assist level designers.)

• Models are named Blah-00, and instances using that model are Blah-00, Blah-01, ..., Blah-99, Blah-100, ... . However, some models that are used only once omit the '-00' and and instance that uses it also omits that suffix.
• Each model has a single valuetable entry, which has the same name. Thus all Trespasser instances of a particular model always have the same valuetable entry. (The actual value names seem to be ignored, an index is used to identify them.)
• The model names seem to be ignored by Trespasser [Is that true? I thought they had to match the instance names or trespasser crashed (see above).]

Modifying files

TODO: Changing the position of an object can cause Trespasser to crash, or objects to disappear.

If an instance has a block in the SCN/Savegame, that instance's info in the region and value tables are ignored.

GROFF - Main structure of GRFs, SCNs and Savegames

Groff is the name given to a certain block structured file format, and is named because the string "GROFF" is found in the executable. Groff files consist of a HEADER, a DATA DIRECTORY, DATA BLOCKS, and a NAME TABLE, and are the format for the GRF, SCN and Savegame files.

HEADER

DATA DIRECTORY ENTRY

The DATA DIRECTORY describes the DATA BLOCKS

NAME TABLE

The NAME TABLE stores strings which are referenced by the symbol handle. I believe the handles can be any set of unique values (as long as they match in all the places they're used). In the GRF the strings are used as the names of DATA BLOCKS, names of instances in the REGION TABLE, and names of textures in the .mapping block. In the SCN and Savegames, the strings are used as names of the DATA BLOCKS, and also are used within many of the data blocks (of which the exact format is not yet known).

Trespasser files have the name table sorted alphabetically by the string, but that is not required by the game engine.

GRF - Main level file

GRF Blocks:

- Header
- Region
- Valuetable
- Main Object Block<
- - Geometry>
- - Mapping - - Material>

General Information

GRFs hold all the object geometry, positioning, and scripting for a level. They use the Groff format, and were made with a custom 3DS MAX plugin.

GRFs can be either compressed (as in the game) or uncompressed (as in the demo). Compressed files use Lempel-Ziv (LZ) compression, and can be decompressed using WinZip. Alternatively the Microsoft MS-DOS utilities Compress.exe and Decompress.exe can be used. From within MS Visual C++, LZOpen() and releated functions can be used to decompress them.

GRFs contain three main types of data. 'Models' are actual geometry (eg. textured triangles) and are stored in seperate named data blocks. 'Instances' are instantiations of a model in the level (eg. a model reference + position,orientation,scale) and are listed in the regiontable (".region"). The valuetable contains extra data associated with instances (render parameters, scripting, ...) (".valuetable").

Notes on naming. Blocks types can be identified by either their string name or data directory 'type'.

Models consist of several blocks, a Main block with the name of the model, then optional ".geometry", ".material", and ".mapping". Main blocks contain handles to the geometry and mapping blocks, and the mapping block contains a handle to the material block. (They always seem to be written in consecutive blocks, but this should not be relied on.) There are two different types of geometry format, which I call Type1 and Type2.

Mapping strings to IDs

Textures and objects in the script are referred to by a string, which must be mapped to an integer ID which is used by the PID and TPA. This mapping is done using a hash function tresHash(), described (TODO).

Textures have three strings, for texture, opacity, and bump maps. If they are all empty strings, there is no texture. Otherwise the texture string will exist, and the opacity and bump strings seem to match the info in the PID entry ie. bumpmapped textures always have a bump string, and textures with transparency always have an opacity string (so all three may exist). To find the textureID, the texture and bump names are combined with the Diffuse value for the mesh (default 1.0).

The GRF blocks

".header"

Exactly one ".header" per GRF.

0x00	BE BA CE 0A (groff magic number)
0x04	version (always 0x8000000F?)

".region"

The regiontable lists all the objects (instances) in a level, there is exactly one ".region" per GRF. Note that when changing the position of a 'primary' instance (described TODO), many of the models listed in the script also have to be moved. They have absolute positions, yet the relative position compared to the primary instance is used to instantiate those models on other instances of a mesh. (eg. Model00s, Detail0, DetailShadow, A00s. See document on the valuetable for more information).

HEADER (4 bytes)

0x00	Number of entries (objects)
0x04	Beginning of REGION ENTRYs

REGION ENTRY (44 bytes)

0x00	Main Handle of model
0x04	symbolHandle. Name of this block, each entry must have a unique name.
0x08	Position in level (3float)
0x14	Orientation, as Euler angles, in Radians. Rotation order is x then y then z going object->world. (3float)
0x20	Scale (float)
0x24	Valuetable entry number, or 0 if 0x28 is 0
0x28	0 - No valuetable entry 1 - Has valuetable entry

POST-REGIONTABLE (4 bytes)

0x00	Main Handle of valuetable (long)

".valuetable"

The valuetable stores extra initialization information for instances in the REGIONTABLE, and there is exactly one ".valuetable" per GRF. A region instance will always reference a Group value entry which lists the values for that instance. The entry will always have the same name as the instance's model, so it's more correct that the valuetable stores info about the models, not the instances.

HEADER (20 bytes)

0x00	00 00 00 00 00 00
0x06	Number of strings in STRINGTABLE
0x0A	00 00 00 00 00 00
0x10	Number of strings in STRINGTABLE (again)
0x14	Start of STRINGTABLE
0xxx	ENTRY HEADER
0xxx	VALUETABLE ENTRYs

STRINGTABLE ENTRY

0x00	Null-terminated string
0xxx	Number of this string (numbered consecutively from 0) (long)
0xxx	Number of times this string occurs in ASSEMBLY. Seems to be ignored by Trespasser, can just be zero. (long)

ENTRY HEADER (18 bytes)

0x00	00 00 00 00 00
0x05	Number of entries
0x09	00 00 00 00 00
0x0E	Number of entries (again)
0x13	Beginning of VALUETABLE ENTRYs

VALUETABLE ENTRY

0x00	Entry type (unsigned short) 0x101 - bool 0x102 - char 0x103 - int 0x104 - float 0x105 - string 0x106 - object (Andres called this Group and used it in the .tpm format)
0x02	00
0x03	Entry number (consecutively numbered from 0)
0x07	01 00 00 00
0x0B	Entry name, index into STRINGTABLE. If this entry is directly used by an instance, this name will be the name of the instance's mesh. However, Trespasser seems to ignore it (eg. can use the string "unused" for all names).
0x0F	Number of times entry name has been used so far. Seems to be ignored by Trespasser, can just be zero.
0x13	00 00 00 00
0x17	00 00 00 00
0x1B	Entry data, depends on entry type: Boolean - Unsigned char, 0=false, 1=true 1byte Integer - (signed?) long 4 bytes Float - float 4 bytes String - zero terminated ASCII L bytes Group - Group of entries, N (long) = size, followed by 2*N longs. The first of each pair is an entry number, the second is always 0x01. (Perhaps the second is the number of times the entry is used?) 4+8*N bytes
0xxx	Entry number again (long)
0xxx	01 00 00 00

Main object block

This represents an object model or 'mesh', and is referred to by the .regiontable. A model must have a .geometry block, and has optional .mapping (with corresponding .material).

0x00	Symbol Handle (stringtable) Name of the mesh. Trespasser seems to ignore it (eg. can use string "unused" for all mesh names).
0x04	Geometry Handle (data block)
0x08	Mapping Handle (data block)
0x0C	Physics Handle (always zero)
0x10	AI Handle (always zero)
0x14	Sound Handle (always zero)
0x18	Special Handle (always zero)

".geometry"

Geometry blocks contain the actual object geometry, and contain one of two data formats. I currently use a heuristic to tell which format it contains: If there is a .mapping block, check the size of that block. If the size is 4 bytes, it is type 2, otherwise it is type 1. If there is no mapping block, check the 2nd int of the block (pivotOffset.y in type 1, mapping flag in type 2). If this is 1, the block is type 2, otherwise it is type 1. This works because the floating point interpretation of 0x01 is 1.4013e-045, which is quite unlikely, and in fact never occurs in pivotOffset.y in the trespasser data files)

Type 1:

Type 1 geometry blocks mostly contain untextured cubes (which are physics objects or triggers) and diamonds and spheres (which are triggers), although sometimes they have dummy textures (blank or with "Dummy" written on them). Sometimes they contain textured visible geometry, particularly dinosaurs. If the object is textured, the texture coordinates are stored in the .mapping block.

0x00	float[3]	Pivot Offset (how used?)
0x0C	int	(A) Vertex Count
0x10	int	(B) Face Count. Matches face count in .mapping if .mapping exists.
0x14	int	(C=3*B) Vertex Normal Count
0x18	int	(D) Wrap Vertex Count
0x1C	int[3]	Default Color (.BMP style BGR order)
0x28	float[3][A]	Vertices. Also used to find the physics bounding box. Note that the bounding box is centered about the origin, so even if all vertices are in the range [0.1,0.7], the bounding box will be [-0.7,0.7]
0x--	int[3][B]	Faces. Index into the Vertices.
0x--	float[3][B]	Face Normals. One normal per triangle.
0x--	float[3][C]	Vertex Normals. Normal for vertex j of triangle k is in position 3*k+j.
0x--	float[3][D]	Wrap Vertices. It is not known what these are used for. Trespasser uses the standard vertices (above) to calculate the bounding box. In the Trespaser data files these are all vertices from the vertex list, and so are the convex hull of the shape. However I haven't noticed any side-effects of just finding the min & max x,y,z values of the vertices and writing out the 8 vertices of this bounding cube.

Type 2:

Type 2 geometry contains texture coordinate info, so the corresponding .mapping block is only 4 bytes long.

0x00	uint	Material Handle. 0x00 if no .material block or if .material contains no textures.
0x04	uint	No Material flag, 0 if has .material, 1 if has no .material.
0x08	uint	Default Color 0x00RRGGBB. Used only if there's no .material.
0x0C	float[3]?	Pivot Offset. See description in Type 1 geometry.
0x18	int	(A) Vertex Count
0x1C	int	(B) Textured Normal Count
0x20	int	(C) Face Vertex Index Count
0x24	int	(D) Wrap Vertex Count
0x28	int	(E) Face Count
0x2C	float[3][A]	Vertices.
0x--	TexNorm[B]	Textured Normals, see below (32 bytes each)
0x--	int[C]	Face Vertex Indices (used by Faces)
0x--	FaceInfo[E]	Faces, see below (40bytes each)
0x--	float[3][D]	Wrap Vertices. See description in Type 1 geometry.

TexNorm (32 bytes)

0x00	uint	Vertex index (index into Vertices)
0x04	float[3]	Vertex normal
0x10	float[2]	Texture coordinates
0x18	uint	0x00000001
0x2C	uint	0x00000000

FaceInfo (40 bytes)

0x00	uint	Number of vertices
0x04	uint	Index of first vertex in Face Vertex List (others follow consecutively)
0x08	float[3]	Normal of polygon
0x14	float	-(p dot n), where n = face normal, p = a point on the face, dot = dot product. This is the closest distance from the origin (0,0,0) to the plane containing the face, positive if the origin is in front of the face (in the direction of the normal), negative otherwise.
0x18	uint	Face material (index into .mapping). Must be zero if there are no materials.
0x1C	uint	Type of normals: 0 - Flat shaded, use polygon normal. (Do vertex normals always equal face normal in this case?) 4 - Smooth shaded, use vertex normals. (vertex normals may not equal face normal)
0x20	uint	0x00000000
0x24	uint?	(?) Same for whole object. One of about 20 values in a level. Different levels have different sets of values. eg. 8 1-2 digit values, couple inbetween going to 9 6 digit values, then 6 8 digit values. (Do they match that unknown field in the PID entries?) Seem to be able to write zero for this.

".mapping"

If mapping.NumVerts = 0, NumFaces may be greater than 0, but all entries will be 0. So must take faces from the geometry block. NumVerts=0 seems to happen when a material has all texturenames = "", so really has no texture at all, so NumTexVerts = 0.

(TODO compare to TresView source, found out new things while writing it) Type1 geometry contains texturing info in this block, Type2 has just the material handle (as the texturing info is in the .geometry block). If this block exists, the .material does too.

struct object {
  uint materialHandle;
  uint nverts;
  uint nfaces;
  float2 tvert[nverts];
  uint face[nfaces][3] ;
  uint faceMaterial[nfaces] ;
}

0x00	uint	Material Handle
0x04	uint	Texture Vertex Count (A). Can be zero, which occurs if object is not textured, but .material is still used (will contain one material, with empty strings for textures). Note that this does not necessarily equal the vertex count in the geometry block. To find the info for vertex j of triangle i: tex coord is map.tvert[map.face[i][j]], position is geom.vert[geom.face[i][j]]
0x08	uint	Texture Face Count (B), matches the face count in the .geometry. Can be non-zero even if (A) is zero.
0x0C	float[A][2]	Texture coordinates
0x--	uint[B][3]	Face vertex indices. These index into the Texture coordinates array. eg. tex coord for vert j of face i is tvert[face[i][j]]. If (A) is zero, these are all zero.
0x--	uint[B]	Face material. These index the .material texture lists.

".material"

Contains info on textures, transparency and bump maps used by model. Also some lighting parameters (which are possibly ignored, or overwritten by data elsewhere, the valuetable perhaps). Blocks are all multiples of 24 bytes, minimum 48 bytes. This block exists if and only if there is a .mapping block too. If .mapping.nverts = 0, the map handles all point to the empty string, otherwise tmap points to a non-empty string, and omap and bmap are optional.

Type	Description
uint	Material count = N > 0
uint[N]	Texture map handle list (symbolHandles, see below)
uint[N]	Opacity map handle list (ditto)
uint[N]	Bump map handle list (ditto)
uint[3][N]	Material color list. R,G,B values, 1 long each. Usually all 0x00000080.
uint	Ambient lighting, 0-100 (usually 50 = 0x32)
uint	Diffuse lighting, 0-100 (usually 50 = 0x32)
uint	Specular lighting, 0-100 (usually 50 = 0x32)
uint	Opacity value, 0-100 (usually 100 = 0x64)
uint	Bumpiness value, (usually 0)

The symbol handles are all to strings of the form "Map\LevelName\ImageName.bmp", or an empty string if there is no texture of the type. Textures usually post-fixed with"t2", bumpmaps with "b8", and opacity maps with "o8", though that is not required. eg. Blaht2.bmp, Blahb8.bmp for a bumpmapped texture.

The strings are mapped to the TextureIDs in the PID using a hash function, see (TODO).

 

UTILITIES: GRFHack 0.10 -- extracts all the blocks from GRFs, SCNs, and SAVEGAMES.
	   GeomExt 0.12 -- extracts .geometry as 3ds files. Does most textures.
	   MasterHack 0.1 -- can sorta change savegame positions...

SCN/Savegame - Scene file (level startup)

The SCN and Savegame files use the Groff format, and consist largely of data update blocks which replace information from the GRF. The SCN is basically an 'initial savegame', though the Savegame files contain a few extra blocks specific to savegames. A savegame can be renamed as a SCN, and will then show up on the level load cheat screen (if put in the data directory along with all the other SCNs).

I don't believe the SCN contains any level information that is not derivable from the GRF - the SCN would have been generated from the GRF using a utility, after the GRFs were exported from MAX.

SCN and Savegame blocks

SCNs contain a Header, Partitions, Hierarchy, TerrainHierarchy, and TriggerHierarchy, along with many data blocks (named with a hexadecimal string). That hex number is a hash of a string. Most are of instance names from the GRF, some are strings from the .exe ("DefaultCon", "Render DBase", "AI System", "Joe", "CBloodSplats", "Audio Daemon", "Terrain").

Important note: It appears to be possible to trick Trespasser into creating most of the SCN for us, which is a good thing because we don't need to figure out the format of the other blocks. All blocks except the Header can be deleted and the level will still load. Immediately save the level, then rename the savegame as the required SCN. Trespasser will have recreated all the blocks that are in a normal SCN. (If the blocks are not recreated in this way the level will not restart correctly as there would be no InstanceUpdate blocks to reset the level state.) However, Trespasser will not create a proper Partitions block, so levels may run very slow.

About the GROFF data blocks: The data type must always be 0x80000000. I believe the block handles are ignored. The blocks with hexadecimal names are the ones that we believe update information originally in the GRF. The hexadecimal values are hashes of name from the GRF. For example, the 36byte blocks update CInstances in the GRF, and hence the hex name will be a hash of the instance name in the GRF.

SCN block: "Header" (1348 bytes)

The Hierarchy blocks

These are used along with that Partitions block to define a spatial tree of axis-aligned bounding boxes. If nodeID is the hash of a (case sensitive) instance name, the node's bounding box is the bounding box of that instance. Otherwise nodeID is the hash of a PartitionEntry (see description of hash above). nodeID is always non-zero. There seems to be no limit to the number of children a node can have. The partitions are fixed, so as objects move around a level they are placed into different hierarchy nodes.

In the original Trespasser levels: Each hierarchy block has its own set of partitions. In each hierarchy block there are often more than one node with no parent. These often enclose disjoint parts of the level (eg. one root node for main level objects, one for some outlying objects a long way away).

// Stored in SCN blocks named "Hierarchy", "TriggerHierarchy", and "TerrainHierarchy"

struct HierarchyBlock {
  HierarchyNode nodes[]; // Calculate array size from size of data block??? eg. size/8
}

struct HierarchyNode {
  uint nodeID;   // Hash of node name or of a PartitionEntry.

  uint parentID; // Hash of parent node's name, or zero if no parent.
} // sizeof(HierarchyNode) = 8 bytes

SCN Block: "Hierarchy"
Contains all classes not in other hierarchy blocks? Certainly all of the following: CAnimal, CEntityWater, CGun, CInstance,CLightDirectional, Player, Teleport, ""(no class), CWaterDisturbance. Physics subobjects (all those with instance name beginning with '$') are not included in the hierarchy, except for the body parts of CAnimal and Player. However, levels seem to work fine even if the body parts aren't included in the hierarchy block.

Most nodeIDs use the hash of their instance name, however there are some exceptions. class "Player" uses the string "Player" instead of the instance name (normally "Anne"). Dino body parts (CAnimal values "Body", "Head", and "Tail") and player body parts ("Body", "Foot", and "Hand") are named eg. "$RaptorBody+RaptorB" where "$RaptorBody" is the body part of instance "RaptorB". Note that although there is only one "$RaptorBody" there are entries for the body parts of every instance of dinos eg. there might be a "$RaptorBody+RaptorB300-00". Also, there are entries named eg. "LFoot+31519BCE" and "RFoot+31519BCE" where the number is the hash of "RaptorB" (not sure if should include leading zeros). I don't know which instances these correspond to or how their position is determined (probably two of the joints), but as mentioned above it seems to be okay to not have hierarchy entries for any of these body parts.

SCN Block: "TriggerHierarchy"
Contains all triggers (classes named "*Trigger"). Only CLocationTrigger are stored as a partition tree structure, the other triggers have no 'location', and are stored with parentID = 0.

SCN Block: "TerrainHierarchy"
Contains all CTerrainObj classes. There seem to be some (as yet unknown) restrictions on the terrain hierarchy - with the tree structures I build, some levels to crash on pentium Trespasser builds (but not the K6 build). As a work around I create a single partition containing all the terrain objects as children.

SCN block: "Partitions"

This a list of bounding boxes used by the Hierarchy blocks. The partitions are static (ie. they are not modified, created or deleted by Trespasser, although the ordering (and hence names) may change). In the hierarchy blocks, a partition is identified by a uint, which is the sum of the hashes of a partition's position and size. ie. nodeID = tresHash(&partition.position, 3*sizeof(float)) + tresHash(&partition.size, 3*sizeof(float)). Size of partition node is size of bounding box of child nodes + <0.01,0.01,0.01>

struct PartitionsBlock { // Stored in SCN block named "Partitions"

  char unknown;          // Always 0x10? (Or is it 0x01?)
  PartitionEntry partitions[]; // Calculate array size from size of data block??? eg. (size-1)/28
}

struct PartitionEntry {

  uint flags;      // Flags. When saving a level I set this to 0x10, then the level seems
                   // to work fine. Trespasser seems to recreate the proper flags as they
                   // appear in the savegame if the new level is saved.
  float3 position; // Center of bounding box
  float3 size;     // Half length of bounding box sides

} // sizeof(PartitionEntry) = 28 bytes, except in TestScnNght where size = 24! (no flags)

The Object Update blocks

These hold dynamic state information for objects in the level. The string name of the block is the hash (uppercase hex, without leading zeros) of the instance name (case sensitive) eg."79815A3" for "PDomino-07". The format for most of the object update blocks is unknown. 

struct InstanceUpdateBlock {  // sizeof(block) = 36 or 44 bytes
  float3 position;
  float4 rotation; // Quaternion in order [w,x,y,z]
  float  scale;
  uint   flags;    // (unknown, but similar to PartitionEntry flags)
  // 44 byte blocks have two more fields:

  uint   unknown1; // (usually a small number)
  uint   unknown2; // (always zero?)
}

// Format of other object update blocks (triggers, dinos, teleports etc) is unknown.

Savegame blocks

In addition to the SCN blocks, Savegames contain three extra blocks:

Savegame block: "SaveGameName"

Zero terminated string, name of savegame, as seen in the Trespasser load level dialog box. (Possibly 14 extra unknown bytes after the string.)

Savegame block: "SaveBMP"

Image in standard BMP format. This image is displayed in the Trespasser load level dialog box.

Savegame block: "SaveSCNName"

Zero terminated string, name of the SCN this savegame is based on. Seems to be ignored by Trespasser. (Possibly 6 extra unknown bytes after the string.)

*** From Andres's hacking diary ***)

Found some parsing related stings near CAnimationScript in exe. Nearby strings
were ".asa" and ".asb". Keywords are possibly: scale, rot, pos, frame, end_object,
object, forced_rate, version, Camera. Couldn't figure out how to use them.

*** Other stuff ***

"We used dummy meshes to represent gameplay objects like triggers and
defined game behavior of all objects by typing code into their
object properties buffers"

Utilities

TresEd, unless you're interested in hex data. Then there's GRFHack from the original THS to extract the raw blocks, and there's the unreleased MasterHack by TSOrd.

PID - Packed Image Directory (Texture index)

Packed Image Directory files, or PIDs, describe the data in the corresponding SWP files. They consist of a HEADER followed by PALETTE BLOCK(s) then PID ENTRY(s). See the section on the SWP below for information on how the textures must be packed into the SWP.

HEADER

PALETTE BLOCK

PID ENTRY

Utilities

The classic SWPAdd and SWPExt can be used to add/extract textures, but they are hard to use.

SWP/SPZ - Swap files (Texture data)

The SWP/SPZs are swap files holding raw bitmap data for all the texture maps, bump maps and pre-computed mip maps used in the game. An SPZ file is a compressed SWP file, and the data in the SWP is described in the PID file.

If the SWP doesn't exist for a level, or if the 'modified' time/date stamp on the SWP doesn't match the modified time/date stamp on the SPZ, then Trespasser uncompresses the SPZ on to your hard drive to run the level on load-time (this is reflected in the "Level File Copy" progress bar). Thus Trespasser will overwrite a tampered-with SWP if its 'modified' time/date is different

Also, if Trespasser determines it doesn't need the data in the SPZ (ie. SWP already exists and has the correct 'modified' date/time stamp), then it doesn't even look at the data within the SPZ - any non-zero length "dummy SPZ" will satisfy the engine. (Andres - Actually the SPZ must contain the correct swp length in the first 4 bytes, but the rest of the file is ignored if length + date/time match the SWP.)

SPZ

An SPZ is a compressed SWP file. The compression is a variant of run-length encoding. The Demo does not use SPZs.

The SPZ data is divided into blocks consisting of a single byte Block Code, then some Block Data. The decompressed data is appended to the SWP file as the decompression progresses.

Block code: 8 bits specifying how the block data is to be interpreted. A '1' bit specifies a raw data byte, and a '0' bit specifies a two byte command code. Lowest bit corresponds to the first data byte.

Command code: 2 bytes, copy 'count' bytes from swp file 'offset'. Note that the source and destination may overlap.
Offset = command byte 1 + 256 * command byte 2 hi nibble
Count = 3 + command byte 2 low nibble

I'm not certain of the interpretation of the offset, but I believe:
- Must add 0x12 to get offset from startt of the output stream (offset is not relative to the current position)
- Offset is signed (so must sign extend))
- If the offset is before the start of tthe output stream, output zeros. For encoding a string of zeros, Trespasser initially uses an offset -0x12 - NumZeros
- Not sure what happens when required offfset is larger than 3 nibbles. Taken from start of file? From start of current 0x100000 block?

Example:

Data in spz: 9D 77 EE F3 1A 40 74 F6 F0 FA F0 02
Data in swp: 77 77 77 77 77 77 77 1A 40 74 40 40 40 40 02

SWP

The SWP file holds the raw data for all the texture maps, bump maps and pre-computed mip maps used in the game. The bitmaps are described in the PID file, one bitmap per PID ENTRY. The bitmaps are arranged into blocks that are usually 512 or 1024 bytes wide. I assume bitmaps were organised into blocks in order to simplify the memory management needed for the texture caching used by Trespasser. (I'm not sure how high the blocks are, but almost certainly a multiple of the largest texture height, which is 256.) The bitmaps are all palette based, either standard 8-bit bitmap data, or a custom 16-bit format which includes bumpmap information. For more details of the bitmap format see the "bit depth" field of the PID ENTRY description.

The SWP is divided into two sections. The first is 'non-swapable', it is loaded during level startup and remains in memory. This occurs about 3/4 of the way through level loading. Changing the size of this section (in the PID header) noticeably creates a pause during level loading, as more data is being loaded. This section MUST contain at least the smallest mipmap of every texture, or the level won't load (or will crash randomly). If a texture has no mipmaps, the whole texture must be in this section. (except for the large dino textures at the end of some pids?) The second section contains the rest of the textures, which are loaded as they are needed. The area between the first and second sections is (always?) zero, and can be used when adding new textures to the swp.

It appears that textures can be written sequentially one after the other, instead of packed into pages. (This is how all the small dino textures with non-power of two sizes are stored). However, doing this breaks texture wrapping and makes the mipmapping screwy. If the textures are packed into 512 byte wide pages, texture wrapping works fine and happens automatically. (Haven't verified that mipmaps work correctly).

Most textures require all mipmaps to exist until one of the dimensions is less than 16. eg. 16x32 would get divided again, but 15x32 or 8x32 are okay as the smallest mipmap. There are some odd textures with no mipmaps (don't know how they work, haven't been able to write one myself).

Utilities

Again, the classic SWPAdd and SWPExt can be used to add/extract textures, but they are hard to use.

TPA - Trespasser Packed Audio file format

tpa -- Trespasser Packed Audio Files

Found in the \Trespasser Directory, \Demo\data directory, and the D:\data directory. 
Purposely named (Effects, Ambient, Menu, Stream), the sounds in each one of them are accessed
and played back differently.

They basically hold all the sound data (PCM or DVI/IMA ADPCM [not Microsoft ADPCM])
for the game, and caption data. TSORD - I've tested substituting the Microsoft
ADPCM (which is about 75% smaller) but it doesn't work because Trespasser doesn't
know how to play the data.

Because of the repeating string "WBOR" throughout the file, TSOrd and Andres
named most of the elements after this "WBOR". Since it turned out "WBOR" is
just the "magic number" associated with the format which is just "ROBW" backwards and
stands for Rob Wyatt, I've changed the names to be more descriptive. Here are my names:
First, there's a HEADER,
Then, an ID Table (old: WBOR Table, WBTable), consisting of a number of ID Headers + Sample
Headers together (old: WBOR Table Entries, WTE's)
Then the rest of the file is composed of a Sample Table (old: WBOR Blocks, WBBlocks), each entry
consists of a Sample Header and optional Caption Block (old: WBOR Block Header, WBH) and the
Data Block (old: WBOR Block Data, WBD).
At the end of Effects.tpa there's an additional "Foley Table" or Collision Table.

Most data is aligned to unsigned longs / intel byte order.

Other notes:
-ADPCM uses the 'new' DVI form ie. decode low nibble then high nibble.
-PCM data can be 8bit or 16bit, mono or  stereo.
-For compressed blocks, the UnCompSize field is as if each ADPCM block
 writes the predicted value to the stream, plus one extra sample. But
 CoolEdit seems not to write the predicted value. Who is right?


Here are the details so far:

HEADER
The header is 0x18 (decimal 24) bytes long.

    

        
0x00
        
Version number (Tres will only accept 0x150)
    
    

        
0x04
        
Number of Sound Samples (UL)*
    
    

        
0x08
        
Offset to ID Table
    
    

        
0x0C
        
Number of Collisions (zero except for Effects.tpa)
    
    

        
0x10
        
Offset to Foley Table (= file size if Number of Collisons = 0) (UL)
    
    

        
0x14
        
Number of Sound IDs (equal to or greater than # of Sound Samples) (UL)
    



*Notes from TSOrd: 
This (# of Sound Samples) should equal the number of Sound IDs, 
sometimes it doesn't. The reason for this discrepancy is because the same 
sound sample sometimes has two (or more?) Sound IDs! There are always more 
Sound IDs than Sound Samples when this happens. (Sound IDs are stored 
in the ID Table and not in the Sample Data.) My program compensates 
for this by writing multiple WBOR blocks - later I may work on fixing this 
problem properly. [To fix this problem requires naming the sound files,
which will be necessary to use them properly elsewhere. I'm thinking more
on this.]

ID TABLE
The number of Sound IDs in the table is specified in the previous header. 
Each entry in the ID Table has a length of 0x38 or 56 bytes, and consists of
an ID Header (length 0x14 or 20 bytes)
and a Sample Header (duplicated from the Sample Table) (length 0x24 or 36 bytes)

ID HEADER

    

        
0x00
        
Sound ID (UL) (first one is always D9 7B A0 F0 - "MISSING")*
    
    

        
0x04
        
Offset of Sample Table entry from beginning of file (UL)
    
    

        
0x08
        
Length of Sample Table entry in bytes (UL)
    
    

        
0x0C
        
(float) Attenuation factor, in dB/m. Dinosaur vocals have values of 0.5, 1.5, 1.75,
             other sounds have 0, and when new dinosaur vocals added with TPAadd
             have had this value left at 0, they sound as if the dinos were right
             next to the player, even though they are far away.
    
    

        
0x10
        
(float) Master volume, in dB. Usually 0, but sometimes not, will contain
             negative values for lower volume.

    

SAMPLE HEADER

    

        
0x00
        
Magic number 'WBOR' ('ROBW' backwards, for Rob Wyatt who designed the format)
    
    

        
0x04
        
Version number 6E 00 00 00 (110 decimal)
    
    

        
0x08
        
Offset to Sample Data (UL) (0x24 unless captions)
    
    

        
0x0C
        
ADPCM Block Length
    
    

        
0x10
        
Length of Sample Data in bytes (UL)
    
    

        
0x14
        
Uncompressed Sample Data length (UL)
    
    

        
0x18
        
Sample Rate (UL)
    
    

        
0x1C
        
Bits per sample (UC)
    
    

        
0x1D
        
# of channels 01 -- MONO, 02 -- STEREO (UC)
    
    

        
0x1E
        
compression type 00 -- PCM, 01 -- DVI/IMA ADPCM (UC)
    
    

        
0x1F
        
flags, always 00 ?
    
    

        
0x20
        
Offset to Caption Data (0x00000000 if no captions, 0x00000024 if captions)
    



*Notes from TSOrd:
The first ID Table Entry points to a silent sample that is 16536 or 22120 bytes 
in length. This first entry is the default sound/caption block used whenever the TPA 
is missing the Sound called for. This is a special case but it is *NOT* the only 
case where the sound ID is duplicated between TPA files. It appears that there is 
one discrepancy in that normally a duplicate ID results in a duplicate sound. See 
the DITTO table for more information.

SAMPLE TABLE
BEGINNING OF Sample Table (Offset = 0x38 * Number of ID Table Entries + 0x18)
Described by the WTE's
Each entry of the Sample Table consists of a Sample Header, an optional Caption Block and a Data Block.

SAMPLE HEADER
WBH (info better agree with WTE info)

    

        
0x00
        
Magic number 'WBOR' ('ROBW' backwards, for Rob Wyatt who designed the format)
    
    

        
0x04
        
Version number 6E 00 00 00 (110 decimal)
    
    

        
0x08
        
Offset to Sample Data (UL) (0x24 unless captions)
    
    

        
0x0C
        
ADPCM Block Length
    
    

        
0x10
        
Length of Sample Data in bytes (UL)
    
    

        
0x14
        
Uncompressed Sample Data length (UL)
    
    

        
0x18
        
Sample Rate (UL)
    
    

        
0x1C
        
Bits per sample (UC)
    
    

        
0x1D
        
# of channels 01 -- MONO, 02 -- STEREO (UC)
    
    

        
0x1E
        
compression type 00 -- PCM, 01 -- DVI/IMA ADPCM (UC)
    
    

        
0x1F
        
flags, always 00 ?
    
    

        
0x20
        
Offset to Caption Data (0x00 if no captions, 0x24 if captions)
    

CAPTION BLOCK
Caption Block -- variable length (Optional -- only present if offset in Sample Header == 0x24)
Each Caption Block consists of a Caption Header, 8 bytes long, and Caption Data of variable length.


CAPTION HEADER
The Caption Header consists of two 4-byte integer entries:

    

        
0x00
        
Length of Caption Block
    
    

        
0x04
        
Number of caption lines (can't be 0)
    

CAPTION DATA
Caption Data consist of as many entries as specified by the number of caption lines indicated
in the header, of the following form:

    

        
0x00
        
Duration to display this caption in 200ms increments (I think)
    
    

        
0x04
        
Caption text - \0 terminated string (Length of caption can be zero)
    

(both fields repeated for each line)



DATA BLOCK
The Data Block contains the actual Sample Data, the number of bytes it takes up is indicated
in the Sample Header, together with the format and other info.



FOLEY TABLE
Extra data at the end of Effects.tpa has been identified as a 'Foley table', listing
pairs of SoundMaterial hashes, the sound(s) to play when objects with those
materials collide, and a bunch of additional parameters (all floats).
There are as many entries in the Foley Table as indicated by the Number of Collisions
entry in the Header. Each Foley Table entry has a length of 0x98 or 154 bytes, like this:


FOLEY TABLE ENTRY (FTE) (0x98 bytes)

    

        
0x00
        
SoundMaterial1 hash
    
    

        
0x04
        
SoundMaterial2 hash
    
    

        
0x08
        
NumEffectSounds 

     bits 0-1: = number of Effect hashes (always 1 or 2). See 0x14.

     bit 7: 0, or 1 if has FrictionSoundEffect hash (ie. 1=0x80 because this is the high bit of the byte).
    
    

        
0x0C
        
(float) Minimum time delay. Most entries have this value set to 0.35 or 0.3,
     but "Bullet" entries and some other ones have it set to 0.
    
    

        
0x10
        
(float) Time last used. All entries have this value set to 0.
    
    

        
0x14
        
SoundEffect1 hash
    
    

        
0x18
        
SoundEffect2 hash
    
    

        
0x1C
        
FrictionSoundEffect hash
    
    

        
0x20
        
Transfer function parameter table for SoundEffect1
    
    

        
0x48
        
Transfer function parameter table for SoundEffect2
    
    

        
0x70
        
Transfer function parameter table for FrictionSoundEffect
    


Each Transfer Function Parameter Table consists of 10 float entries, for a length of 0x28
or 40 bytes, as follows:

TRANSFER FUNCTION PARAMETER TABLE (0x28 bytes)

     

        
0x00
        
(float) Maximum volume
    
    

        
0x04
        
(float) Minimum volume
    
    

        
0x08
        
(float) Volume slope
    
    

        
0x0C
        
(float) Volume intersection
    
    

        
0x10
        
(float) Maximum pitch
    
    

        
0x14
        
(float) Minimum pitch
    
    

        
0x18
        
(float) Pitch slope
    
    

        
0x1C
        
(float) Pitch intersection
    
    

        
0x20
        
(float) Attenuation in dB/m
    
    

        
0x24
        
(float) Minimum velocity
    


For non-used effects, the corresponding values appear to be just garbage (though they
may be something else, but interpreted as float values they are absurd).

This is how the transfer functions are built based on the corresponding parameters
for the different sound effects:

Transfer functions:
Volume = Volume Intersection + (Volume Slope * Velocity)
Pitch = Pitch Intersection + (Pitch Slope * Velocity)
Volume = Min(Max(Volume, Minimum Volume), Maximum Volume)
Pitch = Min(Max(Pitch, Minimum Pitch), Maximum Pitch)
Final Volume = Volume / 10 ^ (Attenuation / 10)

Or, to express it in single lines:
Volume = (Min(Max(Volume Intersection + (Volume Slope * Velocity), Minimum Volume), Maximum Volume))/ 10 ^ (Attenuation / 10)
Pitch = Min(Max(Pitch Intersection + (Pitch Slope * Velocity), Minimum Pitch), Maximum Pitch)

To understand better the terms "slope" and "intersection", imagine the functions
being expressed as the equation of a line in a Cartesian coordinates system:

y=a*x+b

Here, y is either the volume or pitch (which we want to calculate), x is the velocity at
which the collision takes place (our input variable), and a and b are the equation's
parameters: b is the value at which the line intersects the Y-axis (the value of y
when x = 0) and a is the slope of the line (the value of y when x = 1, minus b).
The minimum and maximum values actually override the values y should take for certain
values of x. Then the attenuation is used to divide the result to further lower it.




UTILITIES: TPAHack 0.2 -- extract all the data in the tpa's to wavs.
MasterHack 0.1 -- extract and replace wav data. 
TPAAdd 0.0 -- Add and replace wave data.

WTD - Terrain

The WTD (World Terrain Data) holds wavelet compressed terrain data. It consists of three sections: a header, a parameter table, and the zerotree.

WTD HEADER

WTD TERRAIN PARAMETERS

The following parameters are used to convert back and forth between world coordinates and grid (or quad, or zerotree) coordinates, both horizontal (XY) and vertical (Z, height).
NOTE: Things from now on are read using a reversed endian format. Four bytes are read at a time, and their order must be reversed.

WTD ZEROTREE

The wavelet coefficients are stored in a zerotree (which is like a quadtree storing children only if they are significant), which is how the terrain supports varying level of detail. Ask me if you want to know the details of how the wavelet encoding/decoding actually works, it's a bit involved and I don't want to go into it here.

WTD TREENODE

Created: 03/02/2003

SMK - Video

(copied directly from formats.txt)

Updated: 12/30/1998

smk -- Smacker Video Files

Found in the \Demo\data\menu directory and D:\data\menu directory. Arbitrarily named.

Contains the movies used at the beginning and end of trespasser. This is a standardized 
format rather than a DreamWorks Proprietary format -- Mechwarrior 2 also uses SMK files.

SMACKW32.DLL holds the routines to play them back, but we don't care about that :) ... 

I got a free SMK compressor from www.radgametools.com ... cool stuff.

UTILITIES: Smacker Utilities, available for free download.

DDF - Menu

(copied directly from formats.txt)

Updated: 1/21/2000

Figure these out at some point, they look easy.

Text files which script the appearance and actions of buttons and
windows and such. Basically the GUI layer of the game.

Found in the \Demo\data\menu, D:\data\menu and Trespasser\menu directories.

Formats related to the game but not used within it

TPM - Models

(copied directly from Remdul's draft)

TPM File Format

Introduction

Conventions

Elements

Fileinfo

fileinfo
{
 formatversion = 1.0.1
 name = "NewRaptor.max"
 version = 0.1.3
 source = "NewRaptor.max"
 date = 7/2/2007 6:03:23 PM
 comments = "unfinished yet"
}

Instance

instance "RaptorB"
{
 mesh = "RaptorB"
 position = (0.00585938,-0.51088,0.0339626)
 rotation = (0,0,0)
 scale = 3.05637
}

Material

material "RaptorGreen2"
{
 colormap = "RaptorGreen2color.bmp"
 bumpmap = "RaptorGreen2bump.bmp"
}

Mesh

mesh "Plane01"
{
 m = "wood04"
 m = "stone03"
 v = (-1.0,-0.5,0.0)
 v = (1.0,-0.5,0.0)
 v = (-1.0,0.5,0.0)
 v = (1.0,0.5,0.0)
 t = (0.0,0.0)
 t = (1.0,0.0)
 t = (0.0,1.0)
 t = (1.0,1.0)
 n = (0.0,0.0,1.0)
 f = (3,1,4),(3,1,4),(1,1,1),2
 f = (2,4,1),(2,4,1),(1,1,1),1
}

Skin

skin "TestSkin"
{
 m = "default"
 v = (-1.0,-0.5,0.0),1
 v = (1.0,-0.5,0.0),1
 v = (-1.0,0.5,0.0),3
 v = (1.0,0.5,0.0),2
 t = (0.0,0.0)
 t = (1.0,0.0)
 t = (0.0,1.0)
 t = (1.0,1.0)
 n = (0.0,0.0,1.0)
 f = (3,1,4),(3,1,4),(1,1,1),1
 f = (2,4,1),(2,4,1),(1,1,1),1
}

Bone

bone "$JRaptorB23"
{
 position = (0.002930,1.740082,0.810120)
 rotation = (31.740742,-0.000009,0.000010)
 Anim00 = 7
 Anim01 = 8
 Ratio = 0.500000
 RotationRatio = 0.500000
}

Acknowledgments

CAU - Compressed Audio

SAMPLE HEADER

0x00	Magic number 'WBOR' ('ROBW' backwards, for Rob Wyatt who designed the format)
0x04	Version number 6E 00 00 00 (110 decimal)
0x08	Offset to Sample Data (UL) (0x24 unless captions)
0x0C	ADPCM Block Length
0x10	Length of Sample Data in bytes (UL)
0x14	Uncompressed Sample Data length (UL)
0x18	Sample Rate (UL)
0x1C	Bits per sample (UC)
0x1D	# of channels 01 -- MONO, 02 -- STEREO (UC)
0x1E	compression type 00 -- PCM, 01 -- DVI/IMA ADPCM (UC)
0x1F	flags, always 00 ?
0x20	Offset to Caption Data (0x00 if no captions, 0x24 if captions)

CAPTION BLOCK

Caption Block -- variable length (Optional -- only present if offset in Sample Header == 0x24) Each Caption Block consists of a Caption Header, 8 bytes long, and Caption Data of variable length.

CAPTION HEADER

The Caption Header consists of two 4-byte integer entries:

0x00	Length of Caption Block
0x04	Number of caption lines (can't be 0)

CAPTION DATA

Caption Data consist of as many entries as specified by the number of caption lines indicated in the header, of the following form:

0x00	Duration to display this caption in 200ms increments (I think)
0x04	Caption text - \0 terminated string (Length of caption can be zero)

(both fields repeated for each line)

DATA BLOCK

The Data Block contains the actual Sample Data, the number of bytes it takes up is indicated in the Sample Header, together with the format and other info.

TRI - Terrain Triangulation

numTris -- # of triangles in the terrain, integer number
Next comes a list of triangles consisting of #numTris triVerts
numVerts -- # of vertices in the terrain, integer number
Next comes a vertList (list of vertices) consisting of #numVerts vertCoords

V1 V2 V3

X Y Z

TRR - Terrain

X Y Z