.TQL

Task 735: .TQL File Format

Task 735: .TQL File Format

thinkmelt@protonmail.com

• 6 min read

The .TQL file format is a text-based format primarily associated with the Striim platform's Transformation Query Language (TQL), which is utilized for defining real-time data integration and streaming applications. It consists of SQL-like statements for creating sources, streams, continuous queries, and targets, with syntax including data types (e.g., byte, string, DateTime), operators (e.g., +, =, AND), functions (e.g., AVG, TO_DATE, maskPhoneNumber), and reserved keywords (e.g., CREATE, SELECT, FROM). The format has no binary header or encoded structure; it is plain text adhering to the TQL grammar as documented by Striim. Files of this type can be opened and edited with any text editor or Striim's development tools, such as Microsoft SQL Server Management Studio in cases where .tql is used for SQL templates.

  1. The following is a comprehensive list of properties intrinsic to the .TQL file format within its file system context. These represent standard file system metadata attributes, as the format itself is text-based and lacks format-specific binary properties such as magic numbers or headers:
  • File name
  • File path
  • File size (in bytes)
  • Creation time (or birth time, depending on the operating system)
  • Last modification time
  • Last access time
  • Owner user ID (UID)
  • Owner group ID (GID)
  • File permissions (mode, e.g., read/write/execute bits)

Note that availability of certain properties (e.g., creation time) may depend on the underlying file system (e.g., NTFS, ext4).

  1. Two direct download links for example .TQL files (from Striim's public GitHub repository, containing sample TQL applications):
  1. The following is an embedded HTML and JavaScript snippet suitable for a Ghost blog post. It creates a drag-and-drop area where a user can drop a .TQL file, after which the script retrieves and displays the available file system properties (limited to those accessible via the browser's File API, such as name, size, type, and last modified time; server-side properties like owner or permissions are not available in client-side JavaScript).
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Drag and Drop .TQL File Analyzer</title>
    <style>
        #dropZone {
            border: 2px dashed #ccc;
            padding: 20px;
            text-align: center;
            margin: 20px;
        }
        #output {
            margin: 20px;
            white-space: pre-wrap;
        }
    </style>
</head>
<body>
    <div id="dropZone">Drag and drop a .TQL file here</div>
    <div id="output"></div>
    <script>
        const dropZone = document.getElementById('dropZone');
        const output = document.getElementById('output');

        dropZone.addEventListener('dragover', (e) => {
            e.preventDefault();
            dropZone.style.borderColor = '#000';
        });

        dropZone.addEventListener('dragleave', () => {
            dropZone.style.borderColor = '#ccc';
        });

        dropZone.addEventListener('drop', (e) => {
            e.preventDefault();
            dropZone.style.borderColor = '#ccc';
            const file = e.dataTransfer.files[0];
            if (file && file.name.endsWith('.tql')) {
                let props = `File name: ${file.name}\n`;
                props += `File size (in bytes): ${file.size}\n`;
                props += `Last modification time: ${new Date(file.lastModified).toISOString()}\n`;
                props += `File type: ${file.type || 'text/plain'}\n`;
                output.textContent = props;
            } else {
                output.textContent = 'Please drop a valid .TQL file.';
            }
        });
    </script>
</body>
</html>
  1. The following is a Python class for handling .TQL files. It can open a file, retrieve and print the file system properties, read the content (as text, since no decoding is required beyond standard UTF-8), and write a new .TQL file with example content.
import os
import stat
import time

class TQLFileHandler:
    def __init__(self, filepath):
        self.filepath = filepath
        self.content = None

    def open_and_read(self):
        """Open and read the .TQL file content."""
        with open(self.filepath, 'r', encoding='utf-8') as f:
            self.content = f.read()
        return self.content

    def get_properties(self):
        """Retrieve file system properties."""
        st = os.stat(self.filepath)
        properties = {
            'File name': os.path.basename(self.filepath),
            'File path': os.path.abspath(self.filepath),
            'File size (in bytes)': st.st_size,
            'Creation time': time.ctime(st.st_birthtime) if hasattr(st, 'st_birthtime') else 'Not available',
            'Last modification time': time.ctime(st.st_mtime),
            'Last access time': time.ctime(st.st_atime),
            'Owner user ID (UID)': st.st_uid,
            'Owner group ID (GID)': st.st_gid,
            'File permissions (mode)': oct(st.st_mode)
        }
        return properties

    def print_properties(self):
        """Print all file system properties to console."""
        props = self.get_properties()
        for key, value in props.items():
            print(f"{key}: {value}")

    def write_example(self, new_filepath):
        """Write an example .TQL file."""
        example_content = """
CREATE SOURCE ExampleSource USING FileReader (
    directory: 'data',
    wildcard: '*.csv'
) OUTPUT TO ExampleStream;
"""
        with open(new_filepath, 'w', encoding='utf-8') as f:
            f.write(example_content)
        print(f"Example .TQL file written to {new_filepath}")

# Example usage:
# handler = TQLFileHandler('example.tql')
# handler.open_and_read()
# handler.print_properties()
# handler.write_example('new_example.tql')
  1. The following is a Java class for handling .TQL files. It can open a file, retrieve and print the file system properties, read the content, and write a new .TQL file with example content.
import java.io.*;
import java.nio.file.*;
import java.nio.file.attribute.*;
import java.time.Instant;

public class TQLFileHandler {
    private Path filepath;
    private String content;

    public TQLFileHandler(String filepathStr) {
        this.filepath = Paths.get(filepathStr);
    }

    public String openAndRead() throws IOException {
        content = new String(Files.readAllBytes(filepath));
        return content;
    }

    public void printProperties() throws IOException {
        BasicFileAttributes attrs = Files.readAttributes(filepath, BasicFileAttributes.class);
        PosixFileAttributes posixAttrs = null;
        try {
            posixAttrs = Files.readAttributes(filepath, PosixFileAttributes.class);
        } catch (UnsupportedOperationException e) {
            // Posix not supported on this OS
        }

        System.out.println("File name: " + filepath.getFileName());
        System.out.println("File path: " + filepath.toAbsolutePath());
        System.out.println("File size (in bytes): " + attrs.size());
        System.out.println("Creation time: " + Instant.ofEpochMilli(attrs.creationTime().toMillis()));
        System.out.println("Last modification time: " + Instant.ofEpochMilli(attrs.lastModifiedTime().toMillis()));
        System.out.println("Last access time: " + Instant.ofEpochMilli(attrs.lastAccessTime().toMillis()));
        if (posixAttrs != null) {
            System.out.println("Owner user ID (UID): " + posixAttrs.owner().getName());  // Name instead of UID
            System.out.println("Owner group ID (GID): " + posixAttrs.group().getName());  // Name instead of GID
            System.out.println("File permissions (mode): " + posixAttrs.permissions());
        } else {
            System.out.println("Owner user ID (UID): Not available");
            System.out.println("Owner group ID (GID): Not available");
            System.out.println("File permissions (mode): Not available");
        }
    }

    public void writeExample(String newFilepathStr) throws IOException {
        Path newPath = Paths.get(newFilepathStr);
        String exampleContent = """
                CREATE SOURCE ExampleSource USING FileReader (
                    directory: 'data',
                    wildcard: '*.csv'
                ) OUTPUT TO ExampleStream;
                """;
        Files.writeString(newPath, exampleContent);
        System.out.println("Example .TQL file written to " + newPath.toAbsolutePath());
    }

    // Example usage:
    // public static void main(String[] args) throws IOException {
    //     TQLFileHandler handler = new TQLFileHandler("example.tql");
    //     handler.openAndRead();
    //     handler.printProperties();
    //     handler.writeExample("new_example.tql");
    // }
}
  1. The following is a JavaScript class (designed for Node.js, as browser JavaScript lacks access to full file system properties). It can open a file, retrieve and print the file system properties, read the content, and write a new .TQL file with example content.
const fs = require('fs');
const path = require('path');

class TQLFileHandler {
    constructor(filepath) {
        this.filepath = filepath;
        this.content = null;
    }

    openAndRead() {
        this.content = fs.readFileSync(this.filepath, 'utf-8');
        return this.content;
    }

    getProperties() {
        const stats = fs.statSync(this.filepath);
        return {
            'File name': path.basename(this.filepath),
            'File path': path.resolve(this.filepath),
            'File size (in bytes)': stats.size,
            'Creation time': stats.birthtime.toISOString(),
            'Last modification time': stats.mtime.toISOString(),
            'Last access time': stats.atime.toISOString(),
            'Owner user ID (UID)': stats.uid,
            'Owner group ID (GID)': stats.gid,
            'File permissions (mode)': stats.mode.toString(8)
        };
    }

    printProperties() {
        const props = this.getProperties();
        for (const [key, value] of Object.entries(props)) {
            console.log(`${key}: ${value}`);
        }
    }

    writeExample(newFilepath) {
        const exampleContent = `
CREATE SOURCE ExampleSource USING FileReader (
    directory: 'data',
    wildcard: '*.csv'
) OUTPUT TO ExampleStream;
`;
        fs.writeFileSync(newFilepath, exampleContent, 'utf-8');
        console.log(`Example .TQL file written to ${path.resolve(newFilepath)}`);
    }
}

// Example usage:
// const handler = new TQLFileHandler('example.tql');
// handler.openAndRead();
// handler.printProperties();
// handler.writeExample('new_example.tql');
  1. The following is a C++ class (as C does not natively support classes; using C++ for object-oriented structure) for handling .TQL files. It can open a file, retrieve and print the file system properties, read the content, and write a new .TQL file with example content. This assumes a Unix-like system for stat functions.
#include <iostream>
#include <fstream>
#include <string>
#include <sys/stat.h>
#include <ctime>
#include <unistd.h>

class TQLFileHandler {
private:
    std::string filepath;
    std::string content;

public:
    TQLFileHandler(const std::string& fp) : filepath(fp) {}

    std::string openAndRead() {
        std::ifstream file(filepath);
        if (file) {
            content.assign((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
            file.close();
        }
        return content;
    }

    void printProperties() {
        struct stat st;
        if (stat(filepath.c_str(), &st) == 0) {
            std::cout << "File name: " << filepath.substr(filepath.find_last_of("/\\") + 1) << std::endl;
            std::cout << "File path: " << filepath << std::endl;
            std::cout << "File size (in bytes): " << st.st_size << std::endl;
            std::cout << "Creation time: " << std::ctime(&st.st_ctime);
            std::cout << "Last modification time: " << std::ctime(&st.st_mtime);
            std::cout << "Last access time: " << std::ctime(&st.st_atime);
            std::cout << "Owner user ID (UID): " << st.st_uid << std::endl;
            std::cout << "Owner group ID (GID): " << st.st_gid << std::endl;
            std::cout << "File permissions (mode): " << std::oct << st.st_mode << std::endl;
        } else {
            std::cerr << "Error retrieving properties." << std::endl;
        }
    }

    void writeExample(const std::string& newFilepath) {
        std::ofstream file(newFilepath);
        if (file) {
            file << R"(
CREATE SOURCE ExampleSource USING FileReader (
    directory: 'data',
    wildcard: '*.csv'
) OUTPUT TO ExampleStream;
)";
            file.close();
            std::cout << "Example .TQL file written to " << newFilepath << std::endl;
        }
    }
};

// Example usage:
// int main() {
//     TQLFileHandler handler("example.tql");
//     handler.openAndRead();
//     handler.printProperties();
//     handler.writeExample("new_example.tql");
//     return 0;
// }