DLT Format

From ModdingWiki
Jump to navigation Jump to search
DLT Format
Format typeArchive
Max files65,535
File Allocation Table (FAT)Embedded
Filenames?Yes, 8.3, encrypted
Metadata?None
Supports compression?Yes
Supports encryption?Yes
Supports subdirectories?No
Hidden data?No
Games

The DLT format is used to store most of the data for Stargunner.

File format

Signature

The first four characters of the file are "DAVE". Each compressed subfile begins with "PGBP", while uncompressed subfiles do not. It is probably not possible to store a file that has "PGBP" as its first four bytes unless it is first compressed.

Header

The file starts with a single field indicating the number of files present.

Data type Description
char signature[4] "DAVE"
UINT16LE version File version
UINT16LE numFiles File count

The file version must be 0x100 ("00 01") to work with .exe version 1.0b (the version in the full freeware release.)

File entry

After the header, the following structure is repeated numFiles times.

Data type Description
BYTE filename[32] Encrypted filename (8.3, with a path, backslash as separator)
UINT32LE lastModified Last modified date, as number of seconds since 1980-01-01 (no timezone, so local time wherever the archive was created)
UINT32LE size File size
BYTE data[size] File content, size bytes long

! Does the filename need to be null terminated before encryption (31 chars max) or can it be the full 32 chars?

Filename encryption

The filenames are encrypted using an XOR cipher. The first character is in cleartext, then the second and subsequent characters are XOR'd with the previous character's cleartext value incremented by its offset from the start of the string: 

for (int i = 1; i < 32; i++) name[i] ^= name[i - 1] + i;

Compression

Many of the files inside the DLT are compressed using a variant of byte-pair encoding. If a file begins with the four bytes "PGBP" then it is compressed.

The_coder has decompiled the decompression algorithm and produced C++ code which can extract and decompress the files. A cleaned up version of this, with only the decompression algorithm, follows: 

//
// unstargun.cpp
//
// Stargunner decompression utility.
//
// Written by Adam Nielsen <malvineous@shikadi.net>.  Based on The_coder's work
// for tombexcavator.
//
// Use: unstargun < infile > outfile
//

#include <iostream>
#include <cstring>
#include <cassert>
#include <stdint.h>

/// Chunk size used during compression.  Each chunk expands to this amount of data.
#define CHUNK_SIZE 4096

/// Largest possible chunk of compressed data.  (No compression + worst case dictionary size.)
#define CMP_CHUNK_SIZE (CHUNK_SIZE + 256)

/// Read 32-bit little-endian int, in an endian-neutral way
uint32_t get_u32()
{
	uint8_t sig[4];
	std::cin.read((char *)sig, 4);
	return sig[0] | (sig[1] << 8) | (sig[2] << 16) | (sig[3] << 24);
}

/// Read 16-bit little-endian int, in an endian-neutral way
uint16_t get_u16()
{
	uint8_t sig[2];
	std::cin.read((char *)sig, 2);
	return sig[0] | (sig[1] << 8);
}

/// Decompress a data chunk.
/**
 * @param in
 *   Input data.  First byte is the one immediately following the chunk length.
 *
 * @param expanded_size
 *   The size of the input chunk after decompression.  The output buffer must
 *   be able to hold this many bytes.
 *
 * @param out
 *   Output buffer.
 */
unsigned int explode_chunk(const uint8_t* in, size_t expanded_size, uint8_t* out)
{
	uint8_t tableA[256], tableB[256];
	unsigned int inpos = 0;
	unsigned int outpos = 0;

	while (outpos < expanded_size) {
		// Initialise the dictionary so that no bytes are codewords (or if you
		// prefer, each byte expands to itself only.)
		for (int i = 0; i < 256; i++) tableA[i] = i;

		//
		// Read in the dictionary
		//

		uint8_t code;
		unsigned int tablepos = 0;
		do {
			code = in[inpos++];

			// If the code has the high bit set, the lower 7 bits plus one is the
			// number of codewords that will be skipped from the dictionary.  (Those
			// codewords were initialised to expand to themselves in the loop above.)
			if (code > 127) {
				tablepos += code - 127;
				code = 0;
			}
			if (tablepos == 256) break;

			// Read in the indicated number of codewords.
			for (int i = 0; i <= code; i++) {
				assert(tablepos < 256);
				uint8_t data = in[inpos++];
				tableA[tablepos] = data;
				if (tablepos != data) {
					// If this codeword didn't expand to itself, store the second byte
					// of the expansion pair.
					tableB[tablepos] = in[inpos++];
				}
				tablepos++;
			}
		} while (tablepos < 256);

		// Read the length of the data encoded with this dictionary
		int len = in[inpos++];
		len |= in[inpos++] << 8;

		//
		// Decompress the data
		//

		int expbufpos = 0;
		// This is the maximum number of bytes a single codeword can expand to.
		uint8_t expbuf[32];
		while (1) {
			if (expbufpos) {
				// There is data in the expansion buffer, use that
				code = expbuf[--expbufpos];
			} else {
				// There is no data in the expansion buffer, use the input data
				if (--len == -1) break; // no more input data
				code = in[inpos++];
			}

			if (code == tableA[code]) {
				// This byte is itself, write this to the output
				out[outpos++] = code;
			} else {
				// This byte is actually a codeword, expand it into the expansion buffer
				assert(expbufpos < (signed)sizeof(expbuf) - 2);
				expbuf[expbufpos++] = tableB[code];
				expbuf[expbufpos++] = tableA[code];
			}
		}
	}
	return outpos - expanded_size;
}

int main(void)
{
	std::cerr << "Stargunner decompressor.\n"
		"Written by Adam Nielsen <malvineous@shikadi.net>\n"
		"Decompression algorithm decompiled by The_coder" << std::endl;

	// Make sure the input data has a valid "PGBP" signature.
	char sig[4];
	std::cin.read(sig, 4);
	if (strncmp(sig, "PGBP", 4) != 0) {
		std::cerr << "ERROR: Input file is not a Stargunner compressed file."
			<< std::endl;
		return 1;
	}

	// Read the header
	uint32_t outSize = get_u32();
	std::cerr << "Writing " << outSize << " bytes" << std::flush;
	uint8_t *chunk = new uint8_t[CMP_CHUNK_SIZE];
	uint8_t *chunkOut = new uint8_t[CHUNK_SIZE];

	while (!std::cin.eof()) {
		long chunkSize = get_u16();
		if (chunkSize > CMP_CHUNK_SIZE) {
			std::cerr << "\nERROR: Compressed chunk is too large!"
				<< std::endl;
			return 2;
		}
		std::cerr << "." << std::flush;
		std::cin.read((char *)chunk, chunkSize);
		int lenOut;
		if (outSize < CHUNK_SIZE) lenOut = outSize;
		else lenOut = CHUNK_SIZE;
		if (explode_chunk(chunk, lenOut, chunkOut)) {
			std::cerr << "\nERROR: Failed to explode chunk!" << std::endl;
			return 2;
		}
		std::cout.write((char *)chunkOut, lenOut);
		outSize -= lenOut;
	}
	std::cerr << "done." << std::endl;

	return 0;
}

Tools

The following tools are able to work with files in this format.

Name PlatformExtract files? Decompress on extract? Create new? Modify? Compress on insert? Access hidden data? Edit metadata? Notes
Camoto Linux/WindowsYesNoYesYesNoN/AN/A
Camoto/gamearchive.js AnyYesYesYesYesYesN/AN/A
tombexcavator Any/consoleYesYesNoNoNoN/AN/A

See also

Credits

The filename encryption algorithm was reverse engineered by a user on the Xentax forum. The compression algorithm was reverse engineered by The_coder. If you find this information helpful in a project you're working on, please give credit where credit is due. (A link back to this wiki would be nice too!)

Retrieved from ‘https://moddingwiki.shikadi.net/w/index.php?title=DLT_Format&oldid=11760’