Visage Format

From ModdingWiki
(Redirected from Visage Palette Format)
Jump to navigation Jump to search
Visage Format
Visage Format.png
Format typeTileset
HardwareVGA
Max tile count
PaletteInternal/External
Tile names?No
Minimum tile size (pixels)1×1
Maximum tile size (pixels)65536×65536
Plane count1
Plane arrangementLinear
Transparent pixels?Palette-based
Hitmap pixels?No
Metadata?None
Supports sub-tilesets?No
Compressed tiles?Yes
Hidden data?No
Games

The Visage Format is a graphic format that can store multiple images and palettes. It is used in Bodyworks Voyager: Missions in Anatomy, The Lost Files of Sherlock Holmes: The Case of the Serrated Scalpel and The Lost Files of Sherlock Holmes: The Case of the Rose Tattoo to store various backgrounds and sprites. Each block of data can be uncompressed or use RLE. Visage files generally use the .vgs extension, though some others like .lbv and .all also appear, and palette-only ones typically use the .pal extension.

File format

Although the format can contain multiple blocks, there is no header to tell how many data blocks exist. You just need to check for End of File to know when you're done reading.

Offset Data type Name Description
0x00 UINT16LE LastXCoord The last X coordinate on the frame. Add one to get the actual frame width.
0x02 UINT16LE LastYCoord The last Y coordinate on the frame. Add one to get the actual frame height.
0x04 UINT8 Unknown04 0 usually, disassembly shows this can control scaling somehow.
0x05 UINT8 Compressed 0 for uncompressed, 1 for RLE-compressed. See below.
0x06 UINT8 XOffset Added to drawing position to offset an image. Works as an anchor to allow sprite frames with varying dimensions to 'line up'.
0x07 UINT8 YOffset Same as XOffset, but for Y.
0x08 BYTE[] Data The image data.


With the header loaded, you can read the Data block. In uncompressed format, its size will be (LastXCoord+1) * (LastYCoord+1). In compressed format, it will start with a UINT16LE CompressedSize value which indicates the size of the entire block including the headers, meaning the size of the Data byte array itself is CompressedSize - 8, and the actual data inside it will only start after those two bytes, and will have a length of CompressedSize - 10.

Palettes

In some files, the first block is not an actual image, but a palette to use for the rest of the file's frames. It fully matches the normal format, but is never compressed, and its X and Y offsets are always zero. The image data will always have a size of 780 bytes, indicated in the header as image dimensions 390×2, and it will start with the ASCII string "VGA palette" followed by byte 0x1A. The palette data follows immediately behind that, and is 0x300 bytes long. It is in classic 6-bit RGB 256-colour VGA palette format.

This palette format also appears as separate .pal files in Bodyworks Voyager, but they don't appear to be used; they have the same filenames as .VDA files that already contain a palette, and while there are often small differences between the external and internal palettes, screenshots taken in DOSBox revealed that the game seems to use the internal one.

Graphics

Graphic data is implied if the data block isn't a palette. The graphic data is 8-bit VGA data, but, depending on the compression flag, can be either linear VGA data or use a type of RLE as described below.

Unlike most indexed graphics formats, it uses index 0xFF as transparency, rather than 0x00.

RLE Compression

The compressed data in all three games uses the following header:

DataType Name Description
UINT16LE CompressedSize Size of the data. As mentioned before, this is the size of the entire block, meaning the size of the actual compressed data part (including this header) is CompressedSize - 8.
BYTE FlagMarker Flag value to signal a repeated byte in the flag-based RLE compression. Not used in the collapsed transparency compression, but still present there, so it should be skipped.

This means that the size of the actual compressed data behind this header is CompressedSize - 11.

Flag-based RLE

Serrated Scalpel and Bodyworks Voyager both use a rather classic flag-based RLE, which can be decompressed as follows:

  1. Read a byte as testByte.
    • If testByte == FlagMarker, read a BYTE as val, read a UINT8 as num, and copy num bytes of val to the output.
    • Else, add testByte to the output.
  2. Loop until out of data.

Note that this can be applied over the entire data, but the existing files actually never let repeating ranges cross over to a next row in the image. Because of this, if you write an algorithm to compress the data, it is advised to take these row boundaries into account, to ensure the game has no issues reading the files.

Since any run-length command is three bytes long, the original compression does not compress ranges of only two bytes. The only exception to this is the flag marker value itself; there is no escaping system for it, so any bytes in the image data that have the flag marker's value are stored as three-byte run-length commands.

Collapsed transparency RLE

The compression used in Rose Tattoo is row-based, and aims to collapse transparent areas around the real image data. Since a sprite placed on a transparent background, when seen per row, will generally have a chunk of transparent pixels, then a chunk of the actual image data, and then another chunk of transparent pixels, the compression divides each row into alternating chunks of transparent and opaque pixels. The transparent ones get collapsed, the opaque ones get copied without compression. This compression technique is similar to the way code-based RLE works, but the command to execute is inferred automatically. Because of the row-based structure, the decompression process requires the image width.

The decompression repeats the following steps until it runs out of data:

  1. Read a UINT8 as fillSize.
  2. Add fillSize bytes of 0xFF to output. 0xFF is the transparent colour index.
  3. If the end of the data is reached, stop. If the end of the row is reached, loop from the first step. Else, continue.
  4. Read a UINT8 as copySize.
  5. Copy copySize bytes from input to output.
  6. If the end of the data is reached, stop. Else, loop from the start.

Corruption in this data can easily be detected if the operations do not align perfectly to the width of a single row, which makes it easier to distinguish the two compression types.

Tools

The following tools are able to work with files in this format.

Name PlatformView images in this format? Convert/export to another file/format? Import from another file/format? Access hidden data? Edit metadata? Notes
Engie File Converter WindowsYesYesYesN/AN/A Full compression loading and saving support as of v1.2.9.
Westwood Font Editor WindowsYesYesYesN/AN/A No compression support, and no support for embedded palettes. Can't display more than 136 frames due to handling the frames as font characters.

Source Code

File loading (FreeBASIC)

The following FreeBASIC code will load through each block of a Visage file and display the palette or graphic data. It doesn't yet properly decode the RLE compressed graphics. 

' Visage File Viewer.

' Visage files contain 1 or more blocks of data that can include either palette or graphic data.
' Each block begins with a header of four 16-bit integers: width, height, RLE compression, and unknown.
' Blocks containing palette data begin with "VGA palette", all other blocks are graphic data.
' Palette blocks are typical 8-bit VGA palettes, 256 indexes, each with three 6-bit color values.
' Graphic blocks are raw 8-bit indexed VGA pixel data.

' The file path to the palette file you want to open.
Dim As String VGSFile = "H:\Programs\LIB Extractor\vgs\bigmap.vgs"

' Open the Visage file.
If Open(VGSFile For Binary Access Read As #1) <> 0 Then
    Print "File: " + VGSFile + " not found!"
    Sleep
    End
End If

Dim Y As Integer, X As Integer
Dim Index As UByte, Length As UByte
Dim XSize As UShort, YSize As UShort
Dim RLEFlag As UShort, Unknown As UShort
Dim BlockSize As UInteger
Dim As String PaletteCheck = ""
Dim ColorIndex As UShort, ColorOffset As UShort
Dim Red As UByte, Green As UByte, Blue As UByte
Dim As UByte Block = 0
Dim Offset As UInteger, Delta As UShort, I As UShort, FullLength As UInteger

Screen 13
Do
    CLS
    
    ' Load the block header.
    Get #1, , XSize
    Get #1, , YSize
    Get #1, , RLEFlag
    Get #1, , Unknown

    'If XSize = 389 And YSize = 1 Then
    '    ' Trap for palette which doesn't store the correct size to load.
    '    XSize = 780
    'End If
    
    BlockSize = (XSize + 1) * (YSize + 1)

    Print "    Block #: " + Str(Block)
    Print "X, Y (Size): " + Str(XSize) + ", " + Str(YSize) + " (" + Str(BlockSize) + " bytes)"
    Print " Compressed: " + Str(RLEFlag)
    Print "    Unknown: " + Str(Unknown)

    ' Create an array for the block data.
    ReDim VGSData(0 To Blocksize) As UByte

    ' Load this block's data into the array.
    For Y = 0 To YSize
        For X = 0 To XSize
            Get #1, , VGSData(Y * XSize + X)
        Next X
    Next Y

    ' Check for palette.
    PaletteCheck = ""
    For X = 0 To 11
        PaletteCheck = PaletteCheck + Chr(VGSData(X))
    Next X
    If PaletteCheck = "VGA palette" + Chr(26) Then
        Print "       Type: Palette Data"
        Sleep
        Cls

        ' Loop through all of the colors.
        ColorOffset = 12
        ColorIndex = 0
        Do
            ' Read the color attributes from the file.
            Red   = VGSData(ColorOffset)
            Green = VGSData(ColorOffset + 1)
            Blue  = VGSData(ColorOffset + 2)
            
            Red   = Red   * 4
            Green = Green * 4
            Blue  = Blue  * 4
            
            ' Change the palette.
            Palette ColorIndex, Red, Green, Blue
            
            Line(ColorIndex, 0)-(ColorIndex, 199), ColorIndex
            ColorIndex = ColorIndex + 1
            ColorOffset = ColorOffset + 3
        Loop Until ColorOffset >= BlockSize
    Else
        Print "       Type: Graphic Data"
        Sleep
        Cls

        ' If it's not a palette, it's graphic data.
        If RLEFlag = 0 Then
            ' Uncompressed image.
            For Y = 0 To YSize
                For X = 0 To XSize
                    PSet(X, Y), VGSData(Y * XSize + X)
                Next X
            Next Y
        Else
            ' Compressed image.
            Y = 0
            X = 0
            FullLength = VGSData(1) * 256 + VGSData(0)      ' Data Stream Length
            ' Unknown: Always FE FE.
            Offset = 4
            ' To Do: Decode RLE.
        End If
    End If
    
    Sleep
    Block = Block + 1
Loop While Not EOF(1)

Close #1

Flag-based RLE

Decompression (C#)

This code is written in C# by Nyerguds, for the Engie File Converter tool. 

/// <summary>
/// Decodes the Mythos Software flag-based RLE compression.
/// </summary>
/// <param name="buffer">Input buffer.</param>
/// <param name="startOffset">Start offset. Leave null to start at the start.</param>
/// <param name="endOffset">End offset. Leave null to take the length of the buffer.</param>
/// <param name="decompressedSize">Decompressed size. If given, the initial output buffer will be initialised to this.</param>
/// <param name="abortOnError">Abort and return null whenever an error occurs. If a decompressedSize was given, it will also abort when exceeding it.</param>
/// <returns>The decoded data, or null if decoding failed.</returns>
public Byte[] FlagRleDecode(Byte[] buffer, UInt32? startOffset, UInt32? endOffset, Int32 decompressedSize, Boolean abortOnError)
{
    UInt32 offset = startOffset ?? 0;
    UInt32 end = (UInt32) buffer.LongLength;
    if (endOffset.HasValue)
        end = Math.Min(endOffset.Value, end);
    UInt32 origOutLength = decompressedSize != 0 ? (UInt32) decompressedSize : ((end - offset) * 4);
    UInt32 outLength = origOutLength;
    Byte[] output = new Byte[outLength];
    UInt32 writeOffset = 0;
    if (end - offset < 3)
        return abortOnError ? null : new Byte[0];
    // Skip size bytes
    offset += 2;
    // Get flag byte
    Byte flag = buffer[offset++];
    while (offset < end)
    {
        Byte val = buffer[offset++];
        if (val == flag)
        {
            if (offset + 1 >= end)
            {
                if (abortOnError)
                    return null;
                break;
            }
            Byte repeatVal = buffer[offset++];
            Byte repeatNum = buffer[offset++];
            if (outLength < writeOffset + repeatNum)
            {
                if (abortOnError && decompressedSize != 0)
                    return null;
                output = this.ExpandBuffer(output, Math.Max(origOutLength, repeatNum));
                outLength = (UInt32) output.LongLength;
            }
            for (; repeatNum > 0; repeatNum--)
                output[writeOffset++] = repeatVal;
        }
        else
        {
            if (outLength <= writeOffset)
            {
                if (abortOnError && decompressedSize != 0)
                    return null;
                output = this.ExpandBuffer(output, origOutLength);
                outLength = (UInt32) output.LongLength;
            }
            output[writeOffset++] = val;
        }
    }
    if (abortOnError && decompressedSize != 0 && decompressedSize != writeOffset)
        return null;
    if (writeOffset < output.Length)
    {
        Byte[] finalOut = new Byte[writeOffset];
        Array.Copy(output, 0, finalOut, 0, writeOffset);
        output = finalOut;
    }
    return output;
}

/// <summary>
/// Expands the buffer by copying its contents into a new, larger byte array.
/// </summary>
/// <param name="buffer">Buffer to expand</param>
/// <param name="expandSize">amount of bytes to add to the buffer.</param>
/// <returns></returns>
private Byte[] ExpandBuffer(Byte[] buffer, UInt32 expandSize)
{
    Byte[] newBuf = new Byte[buffer.Length + expandSize];
    Array.Copy(buffer, 0, newBuf, 0, buffer.Length);
    return newBuf;
}

Compression (C#)

This code is written in C# by Nyerguds, for the Engie File Converter tool. The output it gives is identical to the original files. 

/// <summary>
/// Encodes data to the Mythos Software flag-based RLE compression.
/// </summary>
/// <param name="buffer">Input buffer.</param>
/// <param name="flag">Byte to use as flag value.</param>
/// <param name="lineWidth">Line width. If not zero, the compression will be aligned to fit into separate rows.</param>
/// <param name="headerSize">Header size, to correctly put the full block length at the start.</param>
/// <returns>The encoded data.</returns>
public Byte[] FlagRleEncode(Byte[] buffer, Byte flag, Int32 lineWidth, Int32 headerSize)
{
    if (headerSize + 3 >= 0x10000)
        throw new OverflowException("Header too big!");
    UInt32 outLen = (UInt32)(0x10000 - headerSize - 3);
    Byte[] bufferOut = new Byte[outLen];
    UInt32 len = (UInt32) buffer.Length;
    UInt32 inPtr = 0;
    UInt32 outPtr = 0;
    UInt32 rowWidth = (lineWidth == 0) ? len : (UInt32) lineWidth;
    UInt32 curLineEnd = rowWidth;
    while (inPtr < len)
    {
        if (outLen == outPtr)
            throw new OverflowException("Compressed data is too big to be stored as Mythos compressed format!");
        Byte cur = buffer[inPtr];
        // only one pixel required to write a repeat code if the value is the flag.
        UInt32 requiredRepeat = (UInt32) (cur == flag ? 1 : 3);
        UInt32 detectedRepeat;
        if ((curLineEnd - inPtr >= requiredRepeat) && (detectedRepeat = RepeatingAhead(buffer, len, inPtr, requiredRepeat)) == requiredRepeat)
        {
            // Found more than 2 bytes (or a flag byte). Worth compressing. Apply run-length encoding.
            UInt32 start = inPtr;
            UInt32 end = Math.Min(inPtr + 0xFF, curLineEnd);
            // Already checked these in the RepeatingAhead function.
            inPtr += detectedRepeat;
            // Increase inptr to the last repeated.
            for (; inPtr < end && buffer[inPtr] == cur; inPtr++) { }
            UInt32 repeat = inPtr - start;
            // check buffer overflow
            if (outLen <= outPtr + 3)
                throw new OverflowException("Compressed data is too big to be stored as Mythos compressed format!");
            // write code
            bufferOut[outPtr++] = flag;
            // Add value to repeat
            bufferOut[outPtr++] = cur;
            // add amount of repeats.
            bufferOut[outPtr++] = (Byte) repeat;
        }
        else
        {
            bufferOut[outPtr++] = cur;
            inPtr++;
        }
        if (inPtr == curLineEnd)
            curLineEnd = inPtr + rowWidth;
    }
    Byte[] finalOut = new Byte[outPtr + 3];
    Array.Copy(bufferOut, 0, finalOut, 3, outPtr);
    outPtr += 3 + (UInt32) headerSize;
    if (outPtr > UInt16.MaxValue)
        throw new OverflowException("Compressed data is too big to be stored as Mythos compressed format!");
    // Store size in first two bytes.            
    finalOut[0] = (Byte) (outPtr & 0xFF);
    finalOut[1] = (Byte) ((outPtr >> 8) & 0xFF);
    // Store flag value in third byte.
    finalOut[2] = flag;
    return finalOut;
}

/// <summary>
/// Checks if there are enough repeating bytes ahead.
/// </summary>
/// <param name="buffer">Input buffer.</param>
/// <param name="max">The maximum offset to read inside the buffer.</param>
/// <param name="ptr">The current read offset inside the buffer.</param>
/// <param name="minAmount">Minimum amount of repeating bytes to search for.</param>
/// <returns>The amount of detected repeating bytes.</returns>
protected static UInt32 RepeatingAhead(Byte[] buffer, UInt32 max, UInt32 ptr, UInt32 minAmount)
{
    Byte cur = buffer[ptr];
    for (UInt32 i = 1; i < minAmount; i++)
        if (ptr + i >= max || buffer[ptr + i] != cur)
            return i;
    return minAmount;
}

Collapsed transparency

Decompression (C#)

This code is written in C# by Nyerguds, for the Engie File Converter tool.

It uses the ExpandBuffer function from the above code. 

/// <summary>
/// Decodes the Mythos Software transparency-collapsing RLE compression.
/// </summary>
/// <param name="buffer">Input buffer.</param>
/// <param name="startOffset">Start offset. Leave null to start at the start.</param>
/// <param name="endOffset">End offset. Leave null to take the length of the buffer.</param>
/// <param name="decompressedSize">Decompressed size. If given, the initial output buffer will be initialised to this.</param>
/// <param name="lineWidth">Byte length of one line of image data.</param>
/// <param name="transparentIndex">Transparency value to collapse.</param>
/// <param name="abortOnError">Abort and return null whenever an error occurs. If a decompressedSize was given, it will also abort when exceeding it.</param>
/// <returns>The decoded data, or null if decoding failed.</returns>
public Byte[] CollapsedTransparencyDecode(Byte[] buffer, UInt32? startOffset, UInt32? endOffset, Int32 decompressedSize, Int32 lineWidth, Byte transparentIndex, Boolean abortOnError)
{
    UInt32 offset = startOffset ?? 0;
    UInt32 end = (UInt32)buffer.LongLength;
    if (endOffset.HasValue)
        end = Math.Min(endOffset.Value, end);
    UInt32 origOutLength = decompressedSize != 0 ? (UInt32) decompressedSize : ((end - offset) * 4);
    UInt32 outLength = origOutLength;
    Byte[] output = new Byte[outLength];
    UInt32 writeOffset = 0;
    // Skip size bytes and unused flag byte
    offset += 3;
    UInt32 curLineEnd = (UInt32) lineWidth;
    while (offset < end)
    {
        // Handle fill part
        Byte fillSize = buffer[offset++];
        if (outLength < writeOffset + fillSize)
        {
            if (abortOnError && decompressedSize != 0)
                return null;
            output = this.ExpandBuffer(output, origOutLength);
            outLength = (UInt32) output.LongLength;
        }
        for (; fillSize > 0; fillSize--)
            output[writeOffset++] = transparentIndex;
        // Handle copy part
        if (writeOffset >= curLineEnd)
        {
            if (writeOffset != curLineEnd && abortOnError)
                return null;
            writeOffset = curLineEnd;
            curLineEnd += (UInt32) lineWidth;
            continue;
        }
        if (offset >= end) // also view as error? Dunno if the format does that.
            break;
        Byte copySize = buffer[offset++];
        if (end < offset + copySize)
        {
            if (abortOnError)
                return null;
            copySize = (Byte) (end - offset);
        }
        if (outLength < writeOffset + copySize)
        {
            if (abortOnError && decompressedSize != 0)
                return null;
            output = this.ExpandBuffer(output, origOutLength);
            outLength = (UInt32) output.LongLength;
        }
        Array.Copy(buffer, offset, output, writeOffset, copySize);
        offset += copySize;
        writeOffset += copySize;
        if (writeOffset >= curLineEnd)
        {
            if (writeOffset != curLineEnd && abortOnError)
                return null;
            writeOffset = curLineEnd;
            curLineEnd += (UInt32) lineWidth;
        }
    }
    if (abortOnError && decompressedSize != 0 && decompressedSize != writeOffset)
        return null;
    if (writeOffset < output.Length)
    {
        Byte[] finalOut = new Byte[writeOffset];
        Array.Copy(output, 0, finalOut, 0, writeOffset);
        output = finalOut;
    }
    return output;
}

Compression (C#)

This code is written in C# by Nyerguds, for the Engie File Converter tool. The output it gives is identical to the original files. 

/// <summary>
/// Encodes data to the Mythos Software transparency-collapsing RLE compression.
/// </summary>
/// <param name="buffer">Input buffer.</param>
/// <param name="transparentIndex">Transparency value to collapse.</param>
/// <param name="lineWidth">Line width.</param>
/// <param name="headerSize">Header size, to correctly put the full block length at the start. Should normally be '8'.</param>
/// <returns>The encoded data.</returns>
public Byte[] CollapsedTransparencyEncode(Byte[] buffer, Byte transparentIndex, Int32 lineWidth, Int32 headerSize)
{
    if (headerSize + 3 >= 0x10000)
        throw new OverflowException("Header too big!");
    UInt32 outLen = (UInt32)(0x10000 - headerSize - 3);
    Byte[] bufferOut = new Byte[outLen];
    UInt32 len = (UInt32) buffer.Length;
    UInt32 inPtr = 0;
    UInt32 outPtr = 0;
    UInt32 rowWidth = (UInt32) lineWidth;
    UInt32 curLineEnd = rowWidth;
    Boolean writingTransparency = true;
    while (inPtr < len)
    {
        if (outLen == outPtr)
            throw new OverflowException("Compressed data is too big to be stored as Mythos compressed format!");
        Byte cur = buffer[inPtr];
        Boolean isTrans = cur == transparentIndex;
        if (writingTransparency && isTrans)
        {
            // Get repeat length. Limit to current line end.
            UInt32 start = inPtr;
            UInt32 end = Math.Min(inPtr + 0xFF, curLineEnd);
            // Increase inptr to the last repeated.
            for (; inPtr < end && buffer[inPtr] == transparentIndex; inPtr++) { }
            // write repeat value
            bufferOut[outPtr++] = (Byte) (inPtr - start);
        }
        else if (!writingTransparency && !isTrans)
        {
            // Get copy length. Limit to current line end.
            UInt32 start = inPtr;
            UInt32 end = Math.Min(inPtr + 0xFF, curLineEnd);
            // Increase inptr to the last repeated.
            for (; inPtr < end && buffer[inPtr] != transparentIndex; inPtr++) { }
            // write repeat value
            Byte copySize = (Byte) (inPtr - start);
            bufferOut[outPtr++] = copySize;
            // Boundary checking
            if (outLen < outPtr + copySize)
                throw new OverflowException("Compressed data is too big to be stored as Mythos compressed format!");
            // Write uncollapsed data
            Array.Copy(buffer, start, bufferOut, outPtr, copySize);
            outPtr += copySize;
        }
        else
        {
            // Somehow writing transparent while in non-transparent mode or vice versa. Could happen
            // if a line starts with non-transparent, or the amount of consecutive transparent pixels
            // exceeds 255. Just set a 0 and continue without incrementing the read ptr.
            bufferOut[outPtr++] = 0;
        }
        if (inPtr >= len)
            break;
        if (inPtr == curLineEnd)
        {
            // Reset to next row
            curLineEnd = inPtr + rowWidth;
            writingTransparency = true;
        }
        else
        {
            // Switch between transparency and opaque data.
            writingTransparency = !writingTransparency;
        }
    }
    Byte[] finalOut = new Byte[outPtr + 3];
    Array.Copy(bufferOut, 0, finalOut, 3, outPtr);
    outPtr += 3 + (UInt32) headerSize;
    if (outPtr > UInt16.MaxValue)
        throw new OverflowException("Compressed data is too big to be stored as Mythos compressed format!");
    // Store size in first two bytes.
    finalOut[0] = (Byte) (outPtr & 0xFF);
    finalOut[1] = (Byte) ((outPtr >> 8) & 0xFF);
    // Store (unused) flag value in third byte.
    finalOut[2] = 0xFE;
    return finalOut;
}

Credits

This file format was reverse engineered by TheAlmightyGuru. The RLE compression was reverse engineered by Ceidwad. If you find this information helpful in a project you're working on, please give credit where credit is due. (A link back to this wiki would be nice too!)

Retrieved from ‘https://moddingwiki.shikadi.net/w/index.php?title=Visage_Format&oldid=10921#Palettes’