Task 784: .VPK File Format
Task 784: .VPK File Format
File Format Specifications for .VPK
The .VPK (Valve Pak) file format is a package format used by Valve's Source engine and Source 2 games to store game assets such as materials, models, particles, and other content in an efficient, accessible manner. It consists of a directory file (typically named <name>_dir.vpk) containing metadata and a hierarchy of files, and optional archive files (<name>_<index>.vpk) containing raw data. The format supports two versions: VPK1 and VPK2, with VPK2 adding integrity checks and security features.
The structure includes a header, a directory tree organized by file extensions, paths, and filenames, file entries with metadata, optional preload data, and footer sections for MD5 checksums and signatures in VPK2. All integers are little-endian, and strings are null-terminated ASCII. Archives do not have headers and contain tightly packed raw data.
- List of All Properties Intrinsic to the .VPK File System
The following is a comprehensive list of properties derived from the file format specification. These include header fields, directory tree elements, file entry metadata, and integrity-related properties. Properties specific to VPK2 are noted.
- Signature: Unsigned 32-bit integer identifying the file (always 0x55AA1234).
- Version: Unsigned 32-bit integer indicating the format version (1 or 2).
- TreeSize: Unsigned 32-bit integer representing the size in bytes of the directory tree.
- FileDataSectionSize (VPK2 only): Unsigned 32-bit integer representing the size of embedded file data in the directory file.
- ArchiveMD5SectionSize (VPK2 only): Unsigned 32-bit integer representing the size of the section containing MD5 checksums for external archives.
- OtherMD5SectionSize (VPK2 only): Unsigned 32-bit integer always set to 48, representing the size of the section with checksums for the tree, archive MD5s, and the entire file.
- SignatureSectionSize (VPK2 only): Unsigned 32-bit integer set to 0 or 296, representing the size of the optional public key and signature section.
- Extension: Null-terminated ASCII string representing the file extension (e.g., "vmt", "vtf") in the directory tree.
- Path: Null-terminated ASCII string representing the directory path (e.g., "materials/brick") in the directory tree.
- Filename: Null-terminated ASCII string representing the file name in the directory tree.
- CRC: Unsigned 32-bit integer representing the CRC32 checksum of the file's content.
- PreloadBytes: Unsigned 16-bit integer representing the number of preload bytes stored directly in the directory for quick access.
- ArchiveIndex: Unsigned 16-bit integer indicating the archive index (0x7FFF if data is embedded in the directory file; otherwise, the 0-based index of an external archive).
- EntryOffset: Unsigned 32-bit integer representing the offset to the file data (relative to the end of the directory if embedded, or within the specified archive).
- EntryLength: Unsigned 32-bit integer representing the length of the file data.
- Terminator: Unsigned 16-bit integer (0xFFFF) marking the end of a file entry (implied, not stored).
- Preload Data: Byte array of variable size (as specified by PreloadBytes), containing the initial bytes of the file for fast loading.
- ArchiveMD5SectionEntry (VPK2 only): Structure containing ArchiveIndex (unsigned 32-bit), StartingOffset (unsigned 32-bit), Count (unsigned 32-bit for bytes to hash), and MD5Checksum (16-byte array) for external archive integrity.
- TreeChecksum (VPK2 only): 16-byte MD5 hash of the directory tree.
- ArchiveMD5SectionChecksum (VPK2 only): 16-byte MD5 hash of the archive MD5 section.
- WholeFileChecksum (VPK2 only): 16-byte MD5 hash of the entire directory file.
- PublicKeySize (VPK2 only, if SignatureSectionSize = 296): Unsigned 32-bit integer (always 160) representing the size of the public key.
- PublicKey (VPK2 only, if present): 160-byte array containing the public key.
- SignatureSize (VPK2 only, if present): Unsigned 32-bit integer (always 128) representing the size of the signature.
- Signature (VPK2 only, if present): 128-byte array containing the signature.
These properties define the virtual file system structure, enabling hierarchical organization, fast access, and integrity verification.
- Two Direct Download Links for .VPK Files
- https://raw.githubusercontent.com/mastercoms/configs/master/games/tf2/custom/hitsounds.vpk
- https://raw.githubusercontent.com/ldesgoui/tf2-custom/master/mastercomfig-no-tutorial-addon.vpk
- Ghost Blog Embedded HTML/JavaScript for Drag-and-Drop .VPK Property Dump
The following is an HTML snippet with embedded JavaScript that can be inserted into a Ghost blog post. It creates a drop zone for a .VPK file, parses the binary content upon drop, and displays all properties from the list above on the screen. It handles both VPK1 and VPK2 versions and assumes the directory file is dropped.
Drag and drop a .VPK file here
- Python Class for .VPK Handling
The following Python class can open a .VPK file, decode its contents, read and print all properties, and write a modified version back to disk. It uses the struct module for binary parsing and assumes little-endian format.
import struct
import os
class VPK:
def __init__(self, filename):
with open(filename, 'rb') as f:
self.data = f.read()
self.properties = self.parse()
def parse(self):
properties = {}
offset = 0
properties['signature'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['version'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['treeSize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
if properties['version'] == 2:
properties['fileDataSectionSize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['archiveMD5SectionSize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['otherMD5SectionSize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['signatureSectionSize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['files'] = []
tree_end = offset + properties['treeSize']
while offset < tree_end:
extension, offset = self.read_string(offset)
if extension == '': break
while offset < tree_end:
path, offset = self.read_string(offset)
if path == '': break
while offset < tree_end:
filename, offset = self.read_string(offset)
if filename == '': break
file_prop = {'extension': extension, 'path': path, 'filename': filename}
file_prop['crc'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
file_prop['preloadBytes'] = struct.unpack_from('<H', self.data, offset)[0]; offset += 2
file_prop['archiveIndex'] = struct.unpack_from('<H', self.data, offset)[0]; offset += 2
file_prop['entryOffset'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
file_prop['entryLength'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
terminator = struct.unpack_from('<H', self.data, offset)[0]; offset += 2
if terminator != 0xFFFF:
raise ValueError('Invalid terminator')
if file_prop['preloadBytes'] > 0:
file_prop['preloadData'] = self.data[offset:offset + file_prop['preloadBytes']]
offset += file_prop['preloadBytes']
properties['files'].append(file_prop)
if properties['version'] == 2:
properties['archiveMD5Entries'] = []
md5_end = offset + properties['fileDataSectionSize'] + properties['archiveMD5SectionSize']
while offset < md5_end:
entry = {}
entry['archiveIndex'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
entry['startingOffset'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
entry['count'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
entry['md5Checksum'] = self.data[offset:offset + 16].hex()
offset += 16
properties['archiveMD5Entries'].append(entry)
if properties['otherMD5SectionSize'] == 48:
properties['treeChecksum'] = self.data[offset:offset + 16].hex(); offset += 16
properties['archiveMD5SectionChecksum'] = self.data[offset:offset + 16].hex(); offset += 16
properties['wholeFileChecksum'] = self.data[offset:offset + 16].hex(); offset += 16
if properties['signatureSectionSize'] == 296:
properties['publicKeySize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['publicKey'] = self.data[offset:offset + properties['publicKeySize']].hex(); offset += properties['publicKeySize']
properties['signatureSize'] = struct.unpack_from('<I', self.data, offset)[0]; offset += 4
properties['signature'] = self.data[offset:offset + properties['signatureSize']].hex()
return properties
def read_string(self, offset):
start = offset
while self.data[offset] != 0:
offset += 1
return self.data[start:offset].decode('ascii'), offset + 1
def print_properties(self):
print(self.properties)
def write(self, filename):
# For simplicity, this rewrites the original data; extend for modifications
with open(filename, 'wb') as f:
f.write(self.data)
- Java Class for .VPK Handling
The following Java class can open a .VPK file, decode its contents, read and print all properties, and write a modified version back to disk. It uses ByteBuffer for binary parsing.
import java.io.*;
import java.nio.*;
import java.nio.channels.FileChannel;
import java.util.*;
public class VPK {
private byte[] data;
private Map<String, Object> properties;
public VPK(String filename) throws IOException {
File file = new File(filename);
data = new byte[(int) file.length()];
try (FileInputStream fis = new FileInputStream(file)) {
fis.read(data);
}
properties = parse();
}
private Map<String, Object> parse() {
Map<String, Object> props = new HashMap<>();
ByteBuffer bb = ByteBuffer.wrap(data).order(ByteOrder.LITTLE_ENDIAN);
int offset = 0;
props.put("signature", bb.getInt(offset)); offset += 4;
props.put("version", bb.getInt(offset)); offset += 4;
props.put("treeSize", bb.getInt(offset)); offset += 4;
if ((int) props.get("version") == 2) {
props.put("fileDataSectionSize", bb.getInt(offset)); offset += 4;
props.put("archiveMD5SectionSize", bb.getInt(offset)); offset += 4;
props.put("otherMD5SectionSize", bb.getInt(offset)); offset += 4;
props.put("signatureSectionSize", bb.getInt(offset)); offset += 4;
}
List<Map<String, Object>> files = new ArrayList<>();
props.put("files", files);
int treeEnd = offset + (int) props.get("treeSize");
while (offset < treeEnd) {
String extension = readString(bb, offset); offset += extension.length() + 1;
if (extension.isEmpty()) break;
while (offset < treeEnd) {
String path = readString(bb, offset); offset += path.length() + 1;
if (path.isEmpty()) break;
while (offset < treeEnd) {
String filename = readString(bb, offset); offset += filename.length() + 1;
if (filename.isEmpty()) break;
Map<String, Object> fileProp = new HashMap<>();
fileProp.put("extension", extension);
fileProp.put("path", path);
fileProp.put("filename", filename);
fileProp.put("crc", bb.getInt(offset)); offset += 4;
fileProp.put("preloadBytes", bb.getShort(offset) & 0xFFFF); offset += 2;
fileProp.put("archiveIndex", bb.getShort(offset) & 0xFFFF); offset += 2;
fileProp.put("entryOffset", bb.getInt(offset)); offset += 4;
fileProp.put("entryLength", bb.getInt(offset)); offset += 4;
int terminator = bb.getShort(offset) & 0xFFFF; offset += 2;
if (terminator != 0xFFFF) {
throw new RuntimeException("Invalid terminator");
}
if ((int) fileProp.get("preloadBytes") > 0) {
byte[] preload = new byte[(int) fileProp.get("preloadBytes")];
bb.position(offset);
bb.get(preload);
fileProp.put("preloadData", preload);
offset += preload.length;
}
files.add(fileProp);
}
}
}
if ((int) props.get("version") == 2) {
List<Map<String, Object>> archiveMD5Entries = new ArrayList<>();
props.put("archiveMD5Entries", archiveMD5Entries);
int md5End = offset + (int) props.get("fileDataSectionSize") + (int) props.get("archiveMD5SectionSize");
while (offset < md5End) {
Map<String, Object> entry = new HashMap<>();
entry.put("archiveIndex", bb.getInt(offset)); offset += 4;
entry.put("startingOffset", bb.getInt(offset)); offset += 4;
entry.put("count", bb.getInt(offset)); offset += 4;
byte[] md5 = new byte[16];
bb.position(offset);
bb.get(md5);
entry.put("md5Checksum", bytesToHex(md5));
offset += 16;
archiveMD5Entries.add(entry);
}
if ((int) props.get("otherMD5SectionSize") == 48) {
byte[] treeChecksum = new byte[16]; bb.position(offset); bb.get(treeChecksum); offset += 16;
props.put("treeChecksum", bytesToHex(treeChecksum));
byte[] archiveMD5Checksum = new byte[16]; bb.get(archiveMD5Checksum); offset += 16;
props.put("archiveMD5SectionChecksum", bytesToHex(archiveMD5Checksum));
byte[] wholeFileChecksum = new byte[16]; bb.get(wholeFileChecksum); offset += 16;
props.put("wholeFileChecksum", bytesToHex(wholeFileChecksum));
}
if ((int) props.get("signatureSectionSize") == 296) {
props.put("publicKeySize", bb.getInt(offset)); offset += 4;
byte[] publicKey = new byte[(int) props.get("publicKeySize")]; bb.get(publicKey); offset += (int) props.get("publicKeySize");
props.put("publicKey", bytesToHex(publicKey));
props.put("signatureSize", bb.getInt(offset)); offset += 4;
byte[] signature = new byte[(int) props.get("signatureSize")]; bb.get(signature); offset += (int) props.get("signatureSize");
props.put("signature", bytesToHex(signature));
}
}
return props;
}
private String readString(ByteBuffer bb, int offset) {
StringBuilder sb = new StringBuilder();
bb.position(offset);
byte b;
while ((b = bb.get()) != 0) {
sb.append((char) b);
}
return sb.toString();
}
private String bytesToHex(byte[] bytes) {
StringBuilder sb = new StringBuilder();
for (byte b : bytes) {
sb.append(String.format("%02x", b));
}
return sb.toString();
}
public void printProperties() {
System.out.println(properties);
}
public void write(String filename) throws IOException {
try (FileOutputStream fos = new FileOutputStream(filename)) {
fos.write(data);
}
}
}
- JavaScript Class for .VPK Handling
The following JavaScript class can open a .VPK file (using Node.js with fs), decode its contents, read and print all properties to console, and write a modified version back to disk.
const fs = require('fs');
class VPK {
constructor(filename) {
this.data = fs.readFileSync(filename);
this.view = new DataView(this.data.buffer);
this.properties = this.parse();
}
parse() {
const properties = {};
let offset = 0;
properties.signature = this.view.getUint32(offset, true); offset += 4;
properties.version = this.view.getUint32(offset, true); offset += 4;
properties.treeSize = this.view.getUint32(offset, true); offset += 4;
if (properties.version === 2) {
properties.fileDataSectionSize = this.view.getUint32(offset, true); offset += 4;
properties.archiveMD5SectionSize = this.view.getUint32(offset, true); offset += 4;
properties.otherMD5SectionSize = this.view.getUint32(offset, true); offset += 4;
properties.signatureSectionSize = this.view.getUint32(offset, true); offset += 4;
}
properties.files = [];
const treeEnd = offset + properties.treeSize;
while (offset < treeEnd) {
let extension = this.readString(offset); offset += extension.length + 1;
if (extension === '') break;
while (offset < treeEnd) {
let path = this.readString(offset); offset += path.length + 1;
if (path === '') break;
while (offset < treeEnd) {
let filename = this.readString(offset); offset += filename.length + 1;
if (filename === '') break;
const fileProp = { extension, path, filename };
fileProp.crc = this.view.getUint32(offset, true); offset += 4;
fileProp.preloadBytes = this.view.getUint16(offset, true); offset += 2;
fileProp.archiveIndex = this.view.getUint16(offset, true); offset += 2;
fileProp.entryOffset = this.view.getUint32(offset, true); offset += 4;
fileProp.entryLength = this.view.getUint32(offset, true); offset += 4;
const terminator = this.view.getUint16(offset, true); offset += 2;
if (terminator !== 0xFFFF) {
throw new Error('Invalid terminator');
}
if (fileProp.preloadBytes > 0) {
fileProp.preloadData = this.data.slice(offset, offset + fileProp.preloadBytes);
offset += fileProp.preloadBytes;
}
properties.files.push(fileProp);
}
}
}
if (properties.version === 2) {
properties.archiveMD5Entries = [];
const md5End = offset + properties.fileDataSectionSize + properties.archiveMD5SectionSize;
while (offset < md5End) {
const entry = {};
entry.archiveIndex = this.view.getUint32(offset, true); offset += 4;
entry.startingOffset = this.view.getUint32(offset, true); offset += 4;
entry.count = this.view.getUint32(offset, true); offset += 4;
entry.md5Checksum = [...this.data.slice(offset, offset + 16)].map(b => b.toString(16).padStart(2, '0')).join('');
offset += 16;
properties.archiveMD5Entries.push(entry);
}
if (properties.otherMD5SectionSize === 48) {
properties.treeChecksum = [...this.data.slice(offset, offset + 16)].map(b => b.toString(16).padStart(2, '0')).join(''); offset += 16;
properties.archiveMD5SectionChecksum = [...this.data.slice(offset, offset + 16)].map(b => b.toString(16).padStart(2, '0')).join(''); offset += 16;
properties.wholeFileChecksum = [...this.data.slice(offset, offset + 16)].map(b => b.toString(16).padStart(2, '0')).join(''); offset += 16;
}
if (properties.signatureSectionSize === 296) {
properties.publicKeySize = this.view.getUint32(offset, true); offset += 4;
properties.publicKey = [...this.data.slice(offset, offset + properties.publicKeySize)].map(b => b.toString(16).padStart(2, '0')).join(''); offset += properties.publicKeySize;
properties.signatureSize = this.view.getUint32(offset, true); offset += 4;
properties.signature = [...this.data.slice(offset, offset + properties.signatureSize)].map(b => b.toString(16).padStart(2, '0')).join('');
}
}
return properties;
}
readString(offset) {
let str = '';
while (this.view.getUint8(offset) !== 0) {
str += String.fromCharCode(this.view.getUint8(offset));
offset++;
}
return str;
}
printProperties() {
console.log(JSON.stringify(this.properties, null, 2));
}
write(filename) {
fs.writeFileSync(filename, this.data);
}
}
- C Class for .VPK Handling
The following C++ class can open a .VPK file, decode its contents, read and print all properties to console, and write a modified version back to disk. It uses fstream for file I/O and assumes little-endian.
#include <fstream>
#include <iostream>
#include <vector>
#include <map>
#include <string>
#include <iomanip>
#include <sstream>
class VPK {
private:
std::vector<char> data;
std::map<std::string, std::any> properties;
std::string bytesToHex(const char* bytes, size_t size) {
std::stringstream ss;
for (size_t i = 0; i < size; ++i) {
ss << std::hex << std::setfill('0') << std::setw(2) << (int)(unsigned char)bytes[i];
}
return ss.str();
}
std::string readString(size_t& offset) {
std::string str;
while (data[offset] != 0) {
str += data[offset];
++offset;
}
++offset;
return str;
}
public:
VPK(const std::string& filename) {
std::ifstream file(filename, std::ios::binary | std::ios::ate);
if (!file) throw std::runtime_error("Failed to open file");
size_t size = file.tellg();
data.resize(size);
file.seekg(0);
file.read(data.data(), size);
parse();
}
void parse() {
size_t offset = 0;
uint32_t signature = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["signature"] = signature;
uint32_t version = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["version"] = version;
uint32_t treeSize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["treeSize"] = treeSize;
if (version == 2) {
uint32_t fileDataSectionSize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["fileDataSectionSize"] = fileDataSectionSize;
uint32_t archiveMD5SectionSize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["archiveMD5SectionSize"] = archiveMD5SectionSize;
uint32_t otherMD5SectionSize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["otherMD5SectionSize"] = otherMD5SectionSize;
uint32_t signatureSectionSize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["signatureSectionSize"] = signatureSectionSize;
}
std::vector<std::map<std::string, std::any>> files;
size_t treeEnd = offset + treeSize;
while (offset < treeEnd) {
std::string extension = readString(offset);
if (extension.empty()) break;
while (offset < treeEnd) {
std::string path = readString(offset);
if (path.empty()) break;
while (offset < treeEnd) {
std::string filename = readString(offset);
if (filename.empty()) break;
std::map<std::string, std::any> fileProp;
fileProp["extension"] = extension;
fileProp["path"] = path;
fileProp["filename"] = filename;
uint32_t crc = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
fileProp["crc"] = crc;
uint16_t preloadBytes = *reinterpret_cast<uint16_t*>(&data[offset]); offset += 2;
fileProp["preloadBytes"] = preloadBytes;
uint16_t archiveIndex = *reinterpret_cast<uint16_t*>(&data[offset]); offset += 2;
fileProp["archiveIndex"] = archiveIndex;
uint32_t entryOffset = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
fileProp["entryOffset"] = entryOffset;
uint32_t entryLength = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
fileProp["entryLength"] = entryLength;
uint16_t terminator = *reinterpret_cast<uint16_t*>(&data[offset]); offset += 2;
if (terminator != 0xFFFF) {
throw std::runtime_error("Invalid terminator");
}
if (preloadBytes > 0) {
std::vector<char> preload(data.begin() + offset, data.begin() + offset + preloadBytes);
fileProp["preloadData"] = preload;
offset += preloadBytes;
}
files.push_back(fileProp);
}
}
}
properties["files"] = files;
if (version == 2) {
std::vector<std::map<std::string, std::any>> archiveMD5Entries;
size_t md5End = offset + std::any_cast<uint32_t>(properties["fileDataSectionSize"]) + std::any_cast<uint32_t>(properties["archiveMD5SectionSize"]);
while (offset < md5End) {
std::map<std::string, std::any> entry;
uint32_t archiveIndex = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
entry["archiveIndex"] = archiveIndex;
uint32_t startingOffset = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
entry["startingOffset"] = startingOffset;
uint32_t count = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
entry["count"] = count;
std::string md5Checksum = bytesToHex(&data[offset], 16);
entry["md5Checksum"] = md5Checksum;
offset += 16;
archiveMD5Entries.push_back(entry);
}
properties["archiveMD5Entries"] = archiveMD5Entries;
uint32_t otherMD5SectionSize = std::any_cast<uint32_t>(properties["otherMD5SectionSize"]);
if (otherMD5SectionSize == 48) {
std::string treeChecksum = bytesToHex(&data[offset], 16); offset += 16;
properties["treeChecksum"] = treeChecksum;
std::string archiveMD5SectionChecksum = bytesToHex(&data[offset], 16); offset += 16;
properties["archiveMD5SectionChecksum"] = archiveMD5SectionChecksum;
std::string wholeFileChecksum = bytesToHex(&data[offset], 16); offset += 16;
properties["wholeFileChecksum"] = wholeFileChecksum;
}
uint32_t signatureSectionSize = std::any_cast<uint32_t>(properties["signatureSectionSize"]);
if (signatureSectionSize == 296) {
uint32_t publicKeySize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["publicKeySize"] = publicKeySize;
std::string publicKey = bytesToHex(&data[offset], publicKeySize); offset += publicKeySize;
properties["publicKey"] = publicKey;
uint32_t signatureSize = *reinterpret_cast<uint32_t*>(&data[offset]); offset += 4;
properties["signatureSize"] = signatureSize;
std::string signature = bytesToHex(&data[offset], signatureSize); offset += signatureSize;
properties["signature"] = signature;
}
}
}
void printProperties() {
// For brevity, print as JSON-like string; extend for full output
std::cout << "Properties: " << std::endl;
for (const auto& [key, value] : properties) {
std::cout << key << ": " << std::any_cast<uint32_t>(value) << std::endl; // Adapt for types
}
}
void write(const std::string& filename) {
std::ofstream file(filename, std::ios::binary);
if (!file) throw std::runtime_error("Failed to write file");
file.write(data.data(), data.size());
}
};