Task 745: .TZST File Format
Task 745: .TZST File Format
File Format Specifications for .TZST
The .TZST file format is a Unix TAR archive compressed using the Zstandard (zstd) algorithm. It combines the TAR container format (typically in UStar variant) for archiving multiple files with zstd for compression to reduce size. This format is commonly used in backups (e.g., by software like Duplicati or Plesk) and package distributions (e.g., Arch Linux packages, often with extensions like .pkg.tar.zst, which follow the same structure). The specifications are derived from the POSIX UStar TAR standard and the Zstandard compression format (as defined in RFC 8878 and the zstd documentation). The file begins with a zstd frame, and the decompressed content is a TAR stream consisting of 512-byte headers followed by file data, padded to 512-byte multiples.
List of All Properties Intrinsic to the .TZST File Format
The properties are primarily those embedded in the UStar TAR headers after zstd decompression, representing filesystem-like metadata for each archived item. These are intrinsic as they mirror file system attributes such as permissions, ownership, and timestamps. The zstd layer adds compression-related properties, but the core filesystem-intrinsic ones are from the TAR structure. For each file/entry in the archive, the properties (fields) are:
- File name: The path and name of the file (up to 100 bytes, ASCII-encoded).
- File mode: Permissions and file type (8 bytes, octal, e.g., 0000644 for regular file with read/write for owner).
- Owner's numeric user ID (UID): Numeric ID of the file owner (8 bytes, octal).
- Group's numeric user ID (GID): Numeric ID of the file group (8 bytes, octal).
- File size: Size of the file content in bytes (12 bytes, octal).
- Last modification time (mtime): Unix timestamp of last modification (12 bytes, octal).
- Checksum: Header checksum for validation (8 bytes, octal).
- Type flag: Indicates file type (1 byte; possible values: '0' or NUL = normal file, '1' = hard link, '2' = symbolic link, '3' = character special, '4' = block special, '5' = directory, '6' = FIFO, '7' = contiguous file, 'g' = global extended header, 'x' = extended header, 'A'–'Z' = vendor-specific, others reserved).
- Name of linked file: Target path for links (100 bytes, ASCII).
- UStar indicator (magic): "ustar\0" for format identification (6 bytes).
- UStar version: "00" (2 bytes).
- Owner user name (uname): String name of the owner (32 bytes, ASCII).
- Owner group name (gname): String name of the group (32 bytes, ASCII).
- Device major number: Major device number for special files (8 bytes, octal).
- Device minor number: Minor device number for special files (8 bytes, octal).
- Filename prefix: Prefix for long file names (155 bytes, ASCII).
These fields occupy fixed offsets in each 512-byte TAR header (e.g., name at offset 0, mode at 100). The zstd compression does not alter these but wraps the entire TAR stream.
Two Direct Download Links for .TZST Files
Note that .TZST files are equivalently .tar.zst files in practice. The following are direct links to sample Arch Linux package files in this format:
- https://mirror.pkgbuild.com/core/os/x86_64/acl-2.3.2-1-x86_64.pkg.tar.zst
- https://mirror.pkgbuild.com/core/os/x86_64/attr-2.5.2-1-x86_64.pkg.tar.zst
Ghost Blog Embedded HTML JavaScript for Drag-and-Drop .TZST File Dump
The following is an embeddable HTML snippet with JavaScript for a Ghost blog (or similar). It creates a drag-and-drop area. Upon dropping a .TZST file, it reads the file as an ArrayBuffer, decompresses it using the fflate library (loaded from CDN for zstd support), parses the TAR structure manually, and displays the properties for each entry on the screen. Include this in a blog post's code block or custom HTML.
Drag and drop .TZST file here
Python Class for .TZST Handling
The following Python class uses the zstandard and tarfile libraries to open, decode (decompress and read), print properties, and write .TZST files. Install dependencies via pip install zstandard.
import zstandard as zstd
import tarfile
import io
import os
class TZSTHandler:
def __init__(self, filepath):
self.filepath = filepath
self.properties_list = []
def read_and_print(self):
with open(self.filepath, 'rb') as f:
dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(f) as reader:
with io.BytesIO(reader.read()) as decompressed_io:
with tarfile.open(fileobj=decompressed_io, mode='r:') as tar:
for member in tar.getmembers():
props = {
'name': member.name,
'mode': oct(member.mode),
'uid': member.uid,
'gid': member.gid,
'size': member.size,
'mtime': member.mtime,
'typeflag': member.type,
'linkname': member.linkname,
'uname': member.uname,
'gname': member.gname,
'devmajor': member.devmajor,
'devminor': member.devminor
}
self.properties_list.append(props)
print(props)
return self.properties_list
def write(self, output_path, files_to_add):
# files_to_add: list of file paths to archive
with io.BytesIO() as tar_io:
with tarfile.open(fileobj=tar_io, mode='w:') as tar:
for file_path in files_to_add:
tar.add(file_path, arcname=os.path.basename(file_path))
tar_io.seek(0)
cctx = zstd.ZstdCompressor()
with open(output_path, 'wb') as f:
with cctx.stream_writer(f) as writer:
writer.write(tar_io.read())
Java Class for .TZST Handling
The following Java class uses Apache Commons Compress for TAR and zstd-jni for zstd. Add dependencies: com.github.luben:zstd-jni:1.5.6-10 and org.apache.commons:commons-compress:1.26.0.
import com.github.luben.zstd.ZstdInputStream;
import com.github.luben.zstd.ZstdOutputStream;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import java.io.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class TZSTHandler {
private String filepath;
private List<Map<String, Object>> propertiesList = new ArrayList<>();
public TZSTHandler(String filepath) {
this.filepath = filepath;
}
public List<Map<String, Object>> readAndPrint() throws IOException {
try (FileInputStream fis = new FileInputStream(filepath);
ZstdInputStream zis = new ZstdInputStream(fis);
TarArchiveInputStream tis = new TarArchiveInputStream(zis)) {
TarArchiveEntry entry;
while ((entry = tis.getNextTarEntry()) != null) {
Map<String, Object> props = new HashMap<>();
props.put("name", entry.getName());
props.put("mode", Integer.toOctalString(entry.getMode()));
props.put("uid", entry.getUserId());
props.put("gid", entry.getGroupId());
props.put("size", entry.getSize());
props.put("mtime", entry.getLastModifiedDate().getTime() / 1000);
props.put("typeflag", entry.getLinkFlag());
props.put("linkname", entry.getLinkName());
props.put("uname", entry.getUserName());
props.put("gname", entry.getGroupName());
props.put("devmajor", entry.getDevMajor());
props.put("devminor", entry.getDevMinor());
propertiesList.add(props);
System.out.println(props);
}
}
return propertiesList;
}
public void write(String outputPath, List<String> filesToAdd) throws IOException {
try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
TarArchiveOutputStream taos = new TarArchiveOutputStream(baos)) {
for (String filePath : filesToAdd) {
File file = new File(filePath);
TarArchiveEntry entry = new TarArchiveEntry(file, file.getName());
taos.putArchiveEntry(entry);
try (FileInputStream fis = new FileInputStream(file)) {
byte[] buffer = new byte[1024];
int len;
while ((len = fis.read(buffer)) > 0) {
taos.write(buffer, 0, len);
}
}
taos.closeArchiveEntry();
}
taos.finish();
try (FileOutputStream fos = new FileOutputStream(outputPath);
ZstdOutputStream zos = new ZstdOutputStream(fos)) {
zos.write(baos.toByteArray());
}
}
}
}
JavaScript Class for .TZST Handling
The following JavaScript class is for Node.js, using fs for file I/O, @node-rs/zstd for zstd, and manual TAR parsing (no built-in lib). Install via npm install @node-rs/zstd.
const fs = require('fs');
const zstd = require('@node-rs/zstd');
class TZSTHandler {
constructor(filepath) {
this.filepath = filepath;
this.propertiesList = [];
}
readAndPrint() {
const compressed = fs.readFileSync(this.filepath);
const decompressed = zstd.decompress(compressed);
let offset = 0;
while (offset < decompressed.length) {
if (decompressed[offset] === 0) break;
const props = {
name: this.getString(decompressed, offset, 100),
mode: this.getOctal(decompressed, offset + 100, 8),
uid: this.getOctal(decompressed, offset + 108, 8),
gid: this.getOctal(decompressed, offset + 116, 8),
size: this.getOctal(decompressed, offset + 124, 12),
mtime: this.getOctal(decompressed, offset + 136, 12),
checksum: this.getOctal(decompressed, offset + 148, 8),
typeflag: String.fromCharCode(decompressed[offset + 156]),
linkname: this.getString(decompressed, offset + 157, 100),
magic: this.getString(decompressed, offset + 257, 6),
version: this.getString(decompressed, offset + 263, 2),
uname: this.getString(decompressed, offset + 265, 32),
gname: this.getString(decompressed, offset + 297, 32),
devmajor: this.getOctal(decompressed, offset + 329, 8),
devminor: this.getOctal(decompressed, offset + 337, 8),
prefix: this.getString(decompressed, offset + 345, 155)
};
this.propertiesList.push(props);
console.log(props);
const fileSize = parseInt(props.size, 10);
offset += 512 + Math.ceil(fileSize / 512) * 512;
}
return this.propertiesList;
}
write(outputPath, filesToAdd) {
// Simplified: create TAR buffer, then compress
let tarBuffer = Buffer.alloc(0);
filesToAdd.forEach(filePath => {
const stat = fs.statSync(filePath);
const header = Buffer.alloc(512);
// Populate header fields (simplified, add full impl as needed)
this.setString(header, 0, filePath, 100);
this.setOctal(header, 100, stat.mode & 0o777, 8);
this.setOctal(header, 108, stat.uid, 8);
this.setOctal(header, 116, stat.gid, 8);
this.setOctal(header, 124, stat.size, 12);
this.setOctal(header, 136, Math.floor(stat.mtime.getTime() / 1000), 12);
this.setString(header, 157, '', 100); // linkname
this.setString(header, 257, 'ustar', 6);
this.setString(header, 263, '00', 2);
this.setString(header, 265, 'user', 32); // uname placeholder
this.setString(header, 297, 'group', 32); // gname
// Checksum calculation
let chksum = 8 * 32; // spaces
for (let i = 0; i < 512; i++) if (i < 148 || i >= 156) chksum += header[i];
this.setOctal(header, 148, chksum, 8);
const fileData = fs.readFileSync(filePath);
const paddedSize = Math.ceil(stat.size / 512) * 512;
const fileBuffer = Buffer.alloc(paddedSize);
fileData.copy(fileBuffer);
tarBuffer = Buffer.concat([tarBuffer, header, fileBuffer]);
});
// End with two zero blocks
tarBuffer = Buffer.concat([tarBuffer, Buffer.alloc(1024)]);
const compressed = zstd.compress(tarBuffer);
fs.writeFileSync(outputPath, compressed);
}
getString(buf, off, len) {
return buf.slice(off, off + len).toString('utf8').replace(/\0/g, '').trim();
}
getOctal(buf, off, len) {
return parseInt(buf.slice(off, off + len).toString('utf8').trim(), 8);
}
setString(buf, off, str, len) {
buf.write(str.slice(0, len - 1), off, len - 1, 'utf8');
}
setOctal(buf, off, val, len) {
const oct = val.toString(8).padStart(len - 1, '0');
buf.write(oct, off, len - 1, 'utf8');
buf[off + len - 1] = 0;
}
}
C++ Class for .TZST Handling (Note: C does not have classes; using C++ for "class" implementation)
The following C++ class uses libzstd for decompression/compression and manual TAR parsing. Compile with -lzstd.
#include <iostream>
#include <fstream>
#include <vector>
#include <map>
#include <string>
#include <zstd.h>
#include <sys/stat.h>
class TZSTHandler {
private:
std::string filepath;
std::vector<std::map<std::string, std::string>> properties_list;
public:
TZSTHandler(const std::string& fp) : filepath(fp) {}
std::vector<std::map<std::string, std::string>> read_and_print() {
std::ifstream file(filepath, std::ios::binary);
if (!file) return {};
file.seekg(0, std::ios::end);
size_t compressed_size = file.tellg();
file.seekg(0);
std::vector<char> compressed(compressed_size);
file.read(compressed.data(), compressed_size);
size_t const decompressed_size = ZSTD_findDecompressedSize(compressed.data(), compressed_size);
std::vector<char> decompressed(decompressed_size);
size_t const dSize = ZSTD_decompress(decompressed.data(), decompressed_size, compressed.data(), compressed_size);
if (ZSTD_isError(dSize)) {
std::cerr << "Decompression error: " << ZSTD_getErrorName(dSize) << std::endl;
return {};
}
size_t offset = 0;
while (offset < dSize) {
if (decompressed[offset] == 0) break;
std::map<std::string, std::string> props;
props["name"] = get_string(decompressed, offset, 100);
props["mode"] = std::to_string(get_octal(decompressed, offset + 100, 8));
props["uid"] = std::to_string(get_octal(decompressed, offset + 108, 8));
props["gid"] = std::to_string(get_octal(decompressed, offset + 116, 8));
props["size"] = std::to_string(get_octal(decompressed, offset + 124, 12));
props["mtime"] = std::to_string(get_octal(decompressed, offset + 136, 12));
props["checksum"] = std::to_string(get_octal(decompressed, offset + 148, 8));
props["typeflag"] = std::string(1, decompressed[offset + 156]);
props["linkname"] = get_string(decompressed, offset + 157, 100);
props["magic"] = get_string(decompressed, offset + 257, 6);
props["version"] = get_string(decompressed, offset + 263, 2);
props["uname"] = get_string(decompressed, offset + 265, 32);
props["gname"] = get_string(decompressed, offset + 297, 32);
props["devmajor"] = std::to_string(get_octal(decompressed, offset + 329, 8));
props["devminor"] = std::to_string(get_octal(decompressed, offset + 337, 8));
props["prefix"] = get_string(decompressed, offset + 345, 155);
properties_list.push_back(props);
// Print
for (const auto& p : props) {
std::cout << p.first << ": " << p.second << std::endl;
}
std::cout << std::endl;
size_t file_size = std::stoul(props["size"]);
offset += 512 + ((file_size + 511) / 512) * 512;
}
return properties_list;
}
void write(const std::string& output_path, const std::vector<std::string>& files_to_add) {
std::vector<char> tar_data;
for (const auto& file_path : files_to_add) {
struct stat st;
if (stat(file_path.c_str(), &st) != 0) continue;
std::vector<char> header(512, 0);
set_string(header, 0, file_path.substr(file_path.find_last_of('/') + 1), 100);
set_octal(header, 100, st.st_mode & 0777, 8);
set_octal(header, 108, st.st_uid, 8);
set_octal(header, 116, st.st_gid, 8);
set_octal(header, 124, st.st_size, 12);
set_octal(header, 136, st.st_mtime, 12);
set_string(header, 157, "", 100); // linkname
header[156] = '0'; // typeflag normal file
set_string(header, 257, "ustar", 6);
set_string(header, 263, "00", 2);
set_string(header, 265, "user", 32);
set_string(header, 297, "group", 32);
// Checksum
unsigned chksum = 8 * ' ';
for (size_t i = 0; i < 512; ++i) if (i < 148 || i >= 156) chksum += static_cast<unsigned char>(header[i]);
set_octal(header, 148, chksum, 8);
std::ifstream file_data(file_path, std::ios::binary);
file_data.seekg(0, std::ios::end);
size_t fsize = file_data.tellg();
file_data.seekg(0);
std::vector<char> fdata(fsize);
file_data.read(fdata.data(), fsize);
size_t padded = ((fsize + 511) / 512) * 512;
std::vector<char> padded_data(padded, 0);
std::copy(fdata.begin(), fdata.end(), padded_data.begin());
tar_data.insert(tar_data.end(), header.begin(), header.end());
tar_data.insert(tar_data.end(), padded_data.begin(), padded_data.end());
}
// Add two zero blocks
std::vector<char> zero(1024, 0);
tar_data.insert(tar_data.end(), zero.begin(), zero.end());
size_t cbuf_size = ZSTD_compressBound(tar_data.size());
std::vector<char> compressed(cbuf_size);
size_t cSize = ZSTD_compress(compressed.data(), cbuf_size, tar_data.data(), tar_data.size(), 3);
if (ZSTD_isError(cSize)) {
std::cerr << "Compression error: " << ZSTD_getErrorName(cSize) << std::endl;
return;
}
std::ofstream out(output_path, std::ios::binary);
out.write(compressed.data(), cSize);
}
private:
std::string get_string(const std::vector<char>& buf, size_t off, size_t len) {
std::string s(buf.begin() + off, buf.begin() + off + len);
s.erase(std::find(s.begin(), s.end(), '\0'), s.end());
return s;
}
unsigned long get_octal(const std::vector<char>& buf, size_t off, size_t len) {
std::string s(buf.begin() + off, buf.begin() + off + len);
s.erase(std::find(s.begin(), s.end(), '\0'), s.end());
return std::stoul(s, nullptr, 8);
}
void set_string(std::vector<char>& buf, size_t off, const std::string& str, size_t len) {
size_t copy_len = std::min(str.size(), len - 1);
std::copy(str.begin(), str.begin() + copy_len, buf.begin() + off);
buf[off + copy_len] = '\0';
}
void set_octal(std::vector<char>& buf, size_t off, unsigned long val, size_t len) {
char oct[32];
snprintf(oct, sizeof(oct), "%0*lo", static_cast<int>(len - 1), val);
std::copy(oct, oct + len - 1, buf.begin() + off);
buf[off + len - 1] = '\0';
}
};