Task 066: .BPS File Format

1. List of Properties of the .BPS File Format

The .BPS file format, used for binary patching in ROM hacking, has the following intrinsic properties based on its structure. These properties define the file's layout and are essential for parsing and validation:

• Magic String: A fixed 4-byte identifier "BPS1" at the start of the file, confirming the format.
• Source Size: A variable-length integer (VLI) indicating the size of the original (source) file in bytes.
• Target Size: A VLI indicating the size of the modified (target) file in bytes.
• Metadata Size: A VLI indicating the length of optional metadata. If zero, no metadata is present (encoded as 0x80).
• Metadata: A string of bytes with length equal to the metadata size, containing domain-specific information (e.g., XML in UTF-8); optional if size is zero.
• Actions: A sequence of commands that describe modifications, continuing until the file offset reaches the file size minus 12 bytes (to account for checksums). Each action consists of:
• A VLI encoding the command type (lowest 2 bits) and length (shifted right by 2, plus 1).
• Command types:
• 0 (SourceRead): Copies 'length' bytes from the source file at the current output offset; no additional data.
• 1 (TargetRead): Includes 'length' bytes of new data directly in the patch.
• 2 (SourceCopy): Followed by a VLI relative offset (signed); copies 'length' bytes from the source at the adjusted relative offset.
• 3 (TargetCopy): Followed by a VLI relative offset (signed); copies 'length' bytes from the target (previously written data) at the adjusted relative offset.
• Source Checksum: A 4-byte CRC32 value verifying the original source file.
• Target Checksum: A 4-byte CRC32 value verifying the resulting target file after patching.
• Patch Checksum: A 4-byte CRC32 value of all preceding bytes in the patch file, ensuring integrity.

Variable-length integers are encoded with 7 bits per byte for data and the 8th bit as a continuation flag (0 for more bytes, 1 for end), with an adjustment subtracting 1 during encoding to avoid ambiguities.

2. Two Direct Download Links for .BPS Files

• https://dl.smwcentral.net/21606/Sunshine Secret Book 64.bps
• https://dl.smwcentral.net/14812/VLDC9 v1.11.bps

3. Ghost Blog Embedded HTML JavaScript for Drag-and-Drop .BPS File Dumper

The following is a complete, self-contained HTML page with embedded JavaScript. It can be embedded in a Ghost blog post or used standalone. Users can drag and drop a .BPS file onto the designated area, after which the script parses the file and displays all properties on the screen.

4. Python Class for .BPS File Handling

The following Python class can open a .BPS file, decode and read its properties, print them to the console, and write the file back (or a modified version) to disk.

import struct
import zlib  # For CRC32, though not used for validation here

class BPSFile:
    def __init__(self, filename):
        with open(filename, 'rb') as f:
            self.data = bytearray(f.read())
        self.properties = self.read_properties()

    def decode_vli(self, pos):
        data = 0
        shift = 1
        while True:
            x = self.data[pos]
            pos += 1
            data += (x & 0x7f) * shift
            if x & 0x80:
                break
            shift <<= 7
            data += shift
        return data, pos

    def decode_signed_vli(self, pos):
        value, pos = self.decode_vli(pos)
        sign = value & 1
        value = (value >> 1) * (-1 if sign else 1)
        return value, pos

    def read_properties(self):
        pos = 0
        properties = {}

        # Magic
        properties['magic'] = self.data[pos:pos+4].decode('ascii')
        pos += 4
        if properties['magic'] != 'BPS1':
            raise ValueError('Invalid BPS file')

        # Source size
        properties['source_size'], pos = self.decode_vli(pos)

        # Target size
        properties['target_size'], pos = self.decode_vli(pos)

        # Metadata size
        properties['metadata_size'], pos = self.decode_vli(pos)

        # Metadata
        properties['metadata'] = ''
        if properties['metadata_size'] > 0:
            properties['metadata'] = self.data[pos:pos + properties['metadata_size']].decode('utf-8', errors='ignore')
            pos += properties['metadata_size']

        # Actions
        properties['actions'] = []
        end_actions = len(self.data) - 12
        while pos < end_actions:
            data, pos = self.decode_vli(pos)
            command = data & 3
            length = (data >> 2) + 1
            action = {'type': command, 'length': length}

            if command == 1:  # TargetRead
                action['data'] = self.data[pos:pos + length].hex()
                pos += length
            elif command in (2, 3):  # SourceCopy or TargetCopy
                action['relative_offset'], pos = self.decode_signed_vli(pos)

            properties['actions'].append(action)

        # Checksums
        properties['source_checksum'] = self.data[pos:pos+4].hex()
        pos += 4
        properties['target_checksum'] = self.data[pos:pos+4].hex()
        pos += 4
        properties['patch_checksum'] = self.data[pos:pos+4].hex()

        return properties

    def print_properties(self):
        props = self.properties
        print(f"Magic: {props['magic']}")
        print(f"Source Size: {props['source_size']}")
        print(f"Target Size: {props['target_size']}")
        print(f"Metadata Size: {props['metadata_size']}")
        print(f"Metadata: {props['metadata']}")
        print("Actions:")
        for i, action in enumerate(props['actions']):
            print(f"  Action {i+1}:")
            print(f"    Type: {action['type']} ({['SourceRead', 'TargetRead', 'SourceCopy', 'TargetCopy'][action['type']]})")
            print(f"    Length: {action['length']}")
            if 'data' in action:
                print(f"    Data: {action['data']}")
            if 'relative_offset' in action:
                print(f"    Relative Offset: {action['relative_offset']}")
        print(f"Source Checksum: {props['source_checksum']}")
        print(f"Target Checksum: {props['target_checksum']}")
        print(f"Patch Checksum: {props['patch_checksum']}")

    def write(self, filename):
        # For simplicity, writes the original data; can be modified to reconstruct from properties
        with open(filename, 'wb') as f:
            f.write(self.data)

# Example usage:
# bps = BPSFile('example.bps')
# bps.print_properties()
# bps.write('output.bps')

5. Java Class for .BPS File Handling

The following Java class can open a .BPS file, decode and read its properties, print them to the console, and write the file back to disk.

import java.io.*;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
import java.util.List;

public class BPSFile {
    private byte[] data;
    private Map<String, Object> properties;

    public BPSFile(String filename) throws IOException {
        data = Files.readAllBytes(new File(filename).toPath());
        properties = readProperties();
    }

    private int[] decodeVLI(int pos) {
        long dataVal = 0;
        long shift = 1;
        while (true) {
            int x = data[pos] & 0xFF;
            pos++;
            dataVal += (x & 0x7f) * shift;
            if ((x & 0x80) != 0) break;
            shift <<= 7;
            dataVal += shift;
        }
        return new int[]{(int) dataVal, pos};
    }

    private int[] decodeSignedVLI(int pos) {
        int[] res = decodeVLI(pos);
        int value = res[0];
        int sign = value & 1;
        value = (value >> 1) * (sign == 1 ? -1 : 1);
        return new int[]{value, res[1]};
    }

    private Map<String, Object> readProperties() {
        Map<String, Object> props = new HashMap<>();
        int pos = 0;

        // Magic
        props.put("magic", new String(data, pos, 4));
        pos += 4;
        if (!"BPS1".equals(props.get("magic"))) {
            throw new IllegalArgumentException("Invalid BPS file");
        }

        // Source size
        int[] res = decodeVLI(pos);
        props.put("source_size", res[0]);
        pos = res[1];

        // Target size
        res = decodeVLI(pos);
        props.put("target_size", res[0]);
        pos = res[1];

        // Metadata size
        res = decodeVLI(pos);
        props.put("metadata_size", res[0]);
        pos = res[1];

        // Metadata
        String metadata = "";
        if ((int) props.get("metadata_size") > 0) {
            metadata = new String(data, pos, (int) props.get("metadata_size"));
            pos += (int) props.get("metadata_size");
        }
        props.put("metadata", metadata);

        // Actions
        List<Map<String, Object>> actions = new ArrayList<>();
        int endActions = data.length - 12;
        while (pos < endActions) {
            res = decodeVLI(pos);
            int dataVal = res[0];
            pos = res[1];
            int command = dataVal & 3;
            int length = (dataVal >> 2) + 1;
            Map<String, Object> action = new HashMap<>();
            action.put("type", command);
            action.put("length", length);

            if (command == 1) { // TargetRead
                StringBuilder sb = new StringBuilder();
                for (int i = 0; i < length; i++) {
                    sb.append(String.format("%02x", data[pos + i] & 0xFF));
                }
                action.put("data", sb.toString());
                pos += length;
            } else if (command == 2 || command == 3) { // SourceCopy or TargetCopy
                res = decodeSignedVLI(pos);
                action.put("relative_offset", res[0]);
                pos = res[1];
            }

            actions.add(action);
        }
        props.put("actions", actions);

        // Checksums
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < 4; i++) sb.append(String.format("%02x", data[pos + i] & 0xFF));
        props.put("source_checksum", sb.toString());
        pos += 4;

        sb = new StringBuilder();
        for (int i = 0; i < 4; i++) sb.append(String.format("%02x", data[pos + i] & 0xFF));
        props.put("target_checksum", sb.toString());
        pos += 4;

        sb = new StringBuilder();
        for (int i = 0; i < 4; i++) sb.append(String.format("%02x", data[pos + i] & 0xFF));
        props.put("patch_checksum", sb.toString());

        return props;
    }

    public void printProperties() {
        System.out.println("Magic: " + properties.get("magic"));
        System.out.println("Source Size: " + properties.get("source_size"));
        System.out.println("Target Size: " + properties.get("target_size"));
        System.out.println("Metadata Size: " + properties.get("metadata_size"));
        System.out.println("Metadata: " + properties.get("metadata"));
        System.out.println("Actions:");
        List<Map<String, Object>> actions = (List<Map<String, Object>>) properties.get("actions");
        for (int i = 0; i < actions.size(); i++) {
            Map<String, Object> action = actions.get(i);
            System.out.println("  Action " + (i + 1) + ":");
            int type = (int) action.get("type");
            System.out.println("    Type: " + type + " (" + new String[]{"SourceRead", "TargetRead", "SourceCopy", "TargetCopy"}[type] + ")");
            System.out.println("    Length: " + action.get("length"));
            if (action.containsKey("data")) {
                System.out.println("    Data: " + action.get("data"));
            }
            if (action.containsKey("relative_offset")) {
                System.out.println("    Relative Offset: " + action.get("relative_offset"));
            }
        }
        System.out.println("Source Checksum: " + properties.get("source_checksum"));
        System.out.println("Target Checksum: " + properties.get("target_checksum"));
        System.out.println("Patch Checksum: " + properties.get("patch_checksum"));
    }

    public void write(String filename) throws IOException {
        // Writes original data; can be extended to reconstruct
        Files.write(new File(filename).toPath(), data);
    }

    // Example usage:
    // public static void main(String[] args) throws IOException {
    //     BPSFile bps = new BPSFile("example.bps");
    //     bps.printProperties();
    //     bps.write("output.bps");
    // }
}

6. JavaScript Class for .BPS File Handling

The following JavaScript class can open a .BPS file (using Node.js for file I/O), decode and read its properties, print them to the console, and write the file back to disk.

const fs = require('fs');

class BPSFile {
    constructor(filename) {
        this.data = fs.readFileSync(filename);
        this.properties = this.readProperties();
    }

    decodeVLI(pos) {
        let data = 0n;
        let shift = 1n;
        while (true) {
            const x = this.data[pos++];
            data += BigInt(x & 0x7f) * shift;
            if (x & 0x80) break;
            shift <<= 7n;
            data += shift;
        }
        return { value: Number(data), pos };
    }

    decodeSignedVLI(pos) {
        const res = this.decodeVLI(pos);
        const sign = res.value & 1;
        res.value = (res.value >> 1) * (sign ? -1 : 1);
        return res;
    }

    readProperties() {
        let pos = 0;
        const properties = {};

        // Magic
        properties.magic = this.data.slice(pos, pos + 4).toString('ascii');
        pos += 4;
        if (properties.magic !== 'BPS1') throw new Error('Invalid BPS file');

        // Source size
        let res = this.decodeVLI(pos);
        properties.source_size = res.value;
        pos = res.pos;

        // Target size
        res = this.decodeVLI(pos);
        properties.target_size = res.value;
        pos = res.pos;

        // Metadata size
        res = this.decodeVLI(pos);
        properties.metadata_size = res.value;
        pos = res.pos;

        // Metadata
        properties.metadata = '';
        if (properties.metadata_size > 0) {
            properties.metadata = this.data.slice(pos, pos + properties.metadata_size).toString('utf-8');
            pos += properties.metadata_size;
        }

        // Actions
        properties.actions = [];
        const endActions = this.data.length - 12;
        while (pos < endActions) {
            res = this.decodeVLI(pos);
            const dataVal = res.value;
            pos = res.pos;
            const command = dataVal & 3;
            const length = (dataVal >> 2) + 1;
            const action = { type: command, length };

            if (command === 1) { // TargetRead
                action.data = Array.from(this.data.slice(pos, pos + length))
                    .map(b => b.toString(16).padStart(2, '0')).join('');
                pos += length;
            } else if (command === 2 || command === 3) { // SourceCopy or TargetCopy
                res = this.decodeSignedVLI(pos);
                action.relative_offset = res.value;
                pos = res.pos;
            }

            properties.actions.push(action);
        }

        // Checksums
        properties.source_checksum = Array.from(this.data.slice(pos, pos + 4))
            .map(b => b.toString(16).padStart(2, '0')).join('');
        pos += 4;
        properties.target_checksum = Array.from(this.data.slice(pos, pos + 4))
            .map(b => b.toString(16).padStart(2, '0')).join('');
        pos += 4;
        properties.patch_checksum = Array.from(this.data.slice(pos, pos + 4))
            .map(b => b.toString(16).padStart(2, '0')).join('');

        return properties;
    }

    printProperties() {
        const props = this.properties;
        console.log(`Magic: ${props.magic}`);
        console.log(`Source Size: ${props.source_size}`);
        console.log(`Target Size: ${props.target_size}`);
        console.log(`Metadata Size: ${props.metadata_size}`);
        console.log(`Metadata: ${props.metadata}`);
        console.log('Actions:');
        props.actions.forEach((action, i) => {
            console.log(`  Action ${i + 1}:`);
            console.log(`    Type: ${action.type} (${['SourceRead', 'TargetRead', 'SourceCopy', 'TargetCopy'][action.type]})`);
            console.log(`    Length: ${action.length}`);
            if (action.data) console.log(`    Data: ${action.data}`);
            if (action.relative_offset !== undefined) console.log(`    Relative Offset: ${action.relative_offset}`);
        });
        console.log(`Source Checksum: ${props.source_checksum}`);
        console.log(`Target Checksum: ${props.target_checksum}`);
        console.log(`Patch Checksum: ${props.patch_checksum}`);
    }

    write(filename) {
        // Writes original data; can be extended
        fs.writeFileSync(filename, this.data);
    }
}

// Example usage:
// const bps = new BPSFile('example.bps');
// bps.printProperties();
// bps.write('output.bps');

7. C++ Class for .BPS File Handling

The following C++ class can open a .BPS file, decode and read its properties, print them to the console, and write the file back to disk.

#include <iostream>
#include <fstream>
#include <vector>
#include <string>
#include <iomanip>
#include <map>

class BPSFile {
private:
    std::vector<unsigned char> data;
    std::map<std::string, std::string> properties; // Simplified to strings for printing
    std::vector<std::map<std::string, std::string>> actions;

    std::pair<long long, size_t> decodeVLI(size_t pos) {
        long long value = 0;
        long long shift = 1;
        while (true) {
            unsigned char x = data[pos++];
            value += (x & 0x7f) * shift;
            if (x & 0x80) break;
            shift <<= 7;
            value += shift;
        }
        return {value, pos};
    }

    std::pair<long long, size_t> decodeSignedVLI(size_t pos) {
        auto res = decodeVLI(pos);
        long long value = res.first;
        int sign = value & 1;
        value = (value >> 1) * (sign ? -1 : 1);
        return {value, res.second};
    }

    void readProperties() {
        size_t pos = 0;

        // Magic
        std::string magic(data.begin() + pos, data.begin() + pos + 4);
        properties["magic"] = magic;
        pos += 4;
        if (magic != "BPS1") {
            throw std::runtime_error("Invalid BPS file");
        }

        // Source size
        auto res = decodeVLI(pos);
        properties["source_size"] = std::to_string(res.first);
        pos = res.second;

        // Target size
        res = decodeVLI(pos);
        properties["target_size"] = std::to_string(res.first);
        pos = res.second;

        // Metadata size
        res = decodeVLI(pos);
        long long metadataSize = res.first;
        properties["metadata_size"] = std::to_string(metadataSize);
        pos = res.second;

        // Metadata
        std::string metadata;
        if (metadataSize > 0) {
            metadata = std::string(data.begin() + pos, data.begin() + pos + metadataSize);
            pos += metadataSize;
        }
        properties["metadata"] = metadata;

        // Actions
        size_t endActions = data.size() - 12;
        while (pos < endActions) {
            res = decodeVLI(pos);
            long long dataVal = res.first;
            pos = res.second;
            int command = dataVal & 3;
            long long length = (dataVal >> 2) + 1;
            std::map<std::string, std::string> action;
            action["type"] = std::to_string(command);
            action["length"] = std::to_string(length);

            if (command == 1) { // TargetRead
                std::stringstream ss;
                for (long long i = 0; i < length; ++i) {
                    ss << std::hex << std::setw(2) << std::setfill('0') << (int)data[pos + i];
                }
                action["data"] = ss.str();
                pos += length;
            } else if (command == 2 || command == 3) { // SourceCopy or TargetCopy
                res = decodeSignedVLI(pos);
                action["relative_offset"] = std::to_string(res.first);
                pos = res.second;
            }

            actions.push_back(action);
        }

        // Checksums
        std::stringstream ss;
        for (int i = 0; i < 4; ++i) ss << std::hex << std::setw(2) << std::setfill('0') << (int)data[pos + i];
        properties["source_checksum"] = ss.str();
        pos += 4;

        ss.str("");
        for (int i = 0; i < 4; ++i) ss << std::hex << std::setw(2) << std::setfill('0') << (int)data[pos + i];
        properties["target_checksum"] = ss.str();
        pos += 4;

        ss.str("");
        for (int i = 0; i < 4; ++i) ss << std::hex << std::setw(2) << std::setfill('0') << (int)data[pos + i];
        properties["patch_checksum"] = ss.str();
    }

public:
    BPSFile(const std::string& filename) {
        std::ifstream file(filename, std::ios::binary);
        if (!file) throw std::runtime_error("Cannot open file");
        data = std::vector<unsigned char>((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
        readProperties();
    }

    void printProperties() const {
        std::cout << "Magic: " << properties.at("magic") << std::endl;
        std::cout << "Source Size: " << properties.at("source_size") << std::endl;
        std::cout << "Target Size: " << properties.at("target_size") << std::endl;
        std::cout << "Metadata Size: " << properties.at("metadata_size") << std::endl;
        std::cout << "Metadata: " << properties.at("metadata") << std::endl;
        std::cout << "Actions:" << std::endl;
        const std::string types[] = {"SourceRead", "TargetRead", "SourceCopy", "TargetCopy"};
        for (size_t i = 0; i < actions.size(); ++i) {
            const auto& action = actions[i];
            int type = std::stoi(action.at("type"));
            std::cout << "  Action " << (i + 1) << ":" << std::endl;
            std::cout << "    Type: " << type << " (" << types[type] << ")" << std::endl;
            std::cout << "    Length: " << action.at("length") << std::endl;
            if (action.count("data")) std::cout << "    Data: " << action.at("data") << std::endl;
            if (action.count("relative_offset")) std::cout << "    Relative Offset: " << action.at("relative_offset") << std::endl;
        }
        std::cout << "Source Checksum: " << properties.at("source_checksum") << std::endl;
        std::cout << "Target Checksum: " << properties.at("target_checksum") << std::endl;
        std::cout << "Patch Checksum: " << properties.at("patch_checksum") << std::endl;
    }

    void write(const std::string& filename) const {
        std::ofstream file(filename, std::ios::binary);
        file.write(reinterpret_cast<const char*>(data.data()), data.size());
    }
};

// Example usage:
// int main() {
//     try {
//         BPSFile bps("example.bps");
//         bps.printProperties();
//         bps.write("output.bps");
//     } catch (const std::exception& e) {
//         std::cerr << e.what() << std::endl;
//     }
//     return 0;
// }