Task 192: .ETL File Format

File Format Specifications for the .ETL File Format

The .ETL file format is the Microsoft Event Trace Log format used by Event Tracing for Windows (ETW). It is a binary format for storing trace messages generated during trace sessions. The format is not officially documented in detail by Microsoft, but through reverse engineering and available sources, it is known to consist of a sequence of fixed-size trace buffers flushed from memory. Each buffer starts with a 0x48-byte WMI_BUFFER_HEADER structure, followed by a sequence of variable-size events, each beginning with a trace header (such as EVENT_TRACE_HEADER or EVENT_HEADER) and event data. The file is compressed in some cases, and events require provider-specific metadata (like PDB or TMF files) for full decoding. The format is version-dependent, with changes across Windows versions (e.g., Windows XP to Windows 10+).

1. List of all the properties of this file format intrinsic to its file system:

Based on the WMI_BUFFER_HEADER structure and associated fields (intrinsic to the format's structure for buffer management and event logging), the key properties are:

BufferSize: The total size of the buffer in bytes (ULONG, typically at offset 0x00).

SavedOffset: The offset to the end of saved (valid) data in the buffer (ULONG, typically at offset 0x04).

CurrentOffset: The current write offset in the buffer (ULONG, typically at offset 0x08).

ReferenceCount: Reference count for the buffer (LONG volatile, typically at offset 0x0C).

TimeStamp: The timestamp when the buffer was filled (LARGE_INTEGER, typically at offset 0x10).

SequenceNumber: The sequence number of the buffer (ULONG, typically at offset 0x18).

ClockType: The type of clock used for timestamps (ULONG, typically at offset 0x1C).

Frequency: The clock frequency for timestamp calculations (LARGE_INTEGER, typically at offset 0x20).

ClientContext: The context of the client (ETW_BUFFER_CONTEXT structure, typically at offset 0x28), containing:

• LoggerId: The ID of the logger session (USHORT).
• ProcessorIndex: The processor index (UCHAR).
• Alignment: Alignment byte (UCHAR).

State: The state of the buffer (ETW_BUFFER_STATE enum, typically at offset 0x2C).

Offset: The used offset in the buffer (ULONG, typically at offset 0x30).

EventsLost: The number of events lost (ULONG, in earlier versions at offset 0x34; replaced in later versions).

BufferFlag: Flags for the buffer (USHORT, in later versions at offset 0x34).

BufferType: Type of the buffer (USHORT, in later versions at offset 0x36).

ReferenceTime: The reference time for the buffer (ETW_REF_CLOCK structure, typically at offset 0x38), containing:

• Time: The reference time (LARGE_INTEGER).
• Freq: The reference frequency (LARGE_INTEGER).

These properties are intrinsic to the format's buffer-based structure and are repeated for each buffer in the file. The file has no global magic number or header; identification is based on the first buffer's structure.

1. Two direct download links for files of format .ETL:

I was unable to find public direct download links for sample .ETL files during my search, as they are typically generated locally using tools like xperf, logman, or PerfView, and not commonly hosted due to their binary nature and potential size. However, you can generate your own using Windows tools (e.g., logman start trace -o sample.etl). For example purposes, here are links to repos where you can download and extract samples if available in test directories (not direct .ETL, but close):

• https://github.com/microsoft/etl2pcapng/raw/main/test/simple.etl (assumed test file; check repo for availability).
• https://github.com/airbus-cert/etl-parser/raw/master/test/test.etl (assumed test file; check repo for availability).

1. Ghost blog embedded HTML JavaScript for drag and drop .ETL file to dump properties:

1. Python class to open, decode, read, write, and print properties to console:

import struct
import os

class ETLFile:
    HEADER_SIZE = 0x48
    HEADER_FMT = '<IIIiQQIQQHHBIHHQQ'  # Approximate unpack format for WMI_BUFFER_HEADER (adjust for exact fields)

    def __init__(self, filename):
        self.filename = filename
        self.buffers = []

    def read(self):
        with open(self.filename, 'rb') as f:
            data = f.read()
        position = 0
        while position < len(data):
            if len(data) - position < self.HEADER_SIZE:
                break
            header_data = data[position:position + self.HEADER_SIZE]
            unpacked = struct.unpack(self.HEADER_FMT, header_data)
            buffer_size = unpacked[0]
            if buffer_size <= self.HEADER_SIZE or position + buffer_size > len(data):
                break
            self.buffers.append(unpacked)
            position += buffer_size

    def print_properties(self):
        for idx, buf in enumerate(self.buffers, 1):
            print(f"Buffer {idx}:")
            print(f"  BufferSize: {buf[0]}")
            print(f"  SavedOffset: {buf[1]}")
            print(f"  CurrentOffset: {buf[2]}")
            print(f"  ReferenceCount: {buf[3]}")
            print(f"  TimeStamp: {buf[4]}")
            print(f"  SequenceNumber: {buf[5]}")
            print(f"  ClockType: {buf[6]}")
            print(f"  Frequency: {buf[7]}")
            print(f"  LoggerId: {buf[8]}")
            print(f"  ProcessorIndex: {buf[9]}")
            print(f"  State: {buf[10]}")
            print(f"  Offset: {buf[11]}")
            print(f"  BufferFlag: {buf[12]}")
            print(f"  BufferType: {buf[13]}")
            print(f"  ReferenceTime.Time: {buf[14]}")
            print(f"  ReferenceTime.Freq: {buf[15]}")
            print()

    def write(self, new_filename):
        with open(new_filename, 'wb') as f:
            for buf in self.buffers:
                header_data = struct.pack(self.HEADER_FMT, *buf)
                f.write(header_data)
                # Write dummy event data to pad to BufferSize (in real use, copy from original)
                pad = b'\0' * (buf[0] - self.HEADER_SIZE)
                f.write(pad)

# Example usage:
# etl = ETLFile('sample.etl')
# etl.read()
# etl.print_properties()
# etl.write('new.etl')

1. Java class to open, decode, read, write, and print properties to console:

import java.io.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

public class ETLFile {
    private static final int HEADER_SIZE = 0x48;
    private ByteBuffer[] buffers;

    public ETLFile(String filename) throws IOException {
        try (FileInputStream fis = new FileInputStream(filename);
             BufferedInputStream bis = new BufferedInputStream(fis)) {
            byte[] data = bis.readAllBytes();
            ByteBuffer bb = ByteBuffer.wrap(data).order(ByteOrder.LITTLE_ENDIAN);
            int position = 0;
            java.util.List<ByteBuffer> bufferList = new java.util.ArrayList<>();
            while (position < data.length) {
                if (data.length - position < HEADER_SIZE) break;
                int bufferSize = bb.getInt(position);
                if (bufferSize <= HEADER_SIZE || position + bufferSize > data.length) break;
                ByteBuffer header = ByteBuffer.allocate(HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
                header.put(data, position, HEADER_SIZE);
                header.flip();
                bufferList.add(header);
                position += bufferSize;
            }
            buffers = bufferList.toArray(new ByteBuffer[0]);
        }
    }

    public void printProperties() {
        for (int idx = 0; idx < buffers.length; idx++) {
            ByteBuffer buf = buffers[idx];
            buf.position(0);
            System.out.println("Buffer " + (idx + 1) + ":");
            System.out.println("  BufferSize: " + buf.getInt());
            System.out.println("  SavedOffset: " + buf.getInt());
            System.out.println("  CurrentOffset: " + buf.getInt());
            System.out.println("  ReferenceCount: " + buf.getInt());
            System.out.println("  TimeStamp: " + buf.getLong());
            System.out.println("  SequenceNumber: " + buf.getInt());
            System.out.println("  ClockType: " + buf.getInt());
            System.out.println("  Frequency: " + buf.getLong());
            System.out.println("  LoggerId: " + buf.getShort());
            System.out.println("  ProcessorIndex: " + (buf.get() & 0xFF));
            System.out.println("  State: " + buf.getInt());
            System.out.println("  Offset: " + buf.getInt());
            System.out.println("  BufferFlag: " + buf.getShort());
            System.out.println("  BufferType: " + buf.getShort());
            System.out.println("  ReferenceTime.Time: " + buf.getLong());
            System.out.println("  ReferenceTime.Freq: " + buf.getLong());
            System.out.println();
        }
    }

    public void write(String newFilename) throws IOException {
        try (FileOutputStream fos = new FileOutputStream(newFilename)) {
            for (ByteBuffer buf : buffers) {
                buf.position(0);
                byte[] headerData = new byte[HEADER_SIZE];
                buf.get(headerData);
            fos.write(headerData);
                // Pad with zeros for event data
                int padSize = buf.getInt(0) - HEADER_SIZE;
                byte[] pad = new byte[padSize];
                fos.write(pad);
            }
        }
    }

    // Example usage:
    // public static void main(String[] args) throws IOException {
    //   ETLFile etl = new ETLFile("sample.etl");
    //   etl.printProperties();
    //   etl.write("new.etl");
    // }
}

1. JavaScript class to open, decode, read, write, and print properties to console:

const fs = require('fs');

class ETLFile {
  constructor(filename) {
    this.filename = filename;
    this.buffers = [];
  }

  read() {
    const data = fs.readFileSync(this.filename);
    let position = 0;
    while (position < data.length) {
      if (data.length - position < 0x48) break;
      const view = new DataView(data.buffer, data.byteOffset + position, 0x48);
      const bufferSize = view.getUint32(0, true);
      if (bufferSize <= 0x48 || position + bufferSize > data.length) break;
      const buf = {
        bufferSize: view.getUint32(0, true),
        savedOffset: view.getUint32(4, true),
        currentOffset: view.getUint32(8, true),
        referenceCount: view.getInt32(12, true),
        timeStamp: Number(view.getBigInt64(16, true)),
        sequenceNumber: view.getUint32(24, true),
        clockType: view.getUint32(28, true),
        frequency: Number(view.getBigInt64(32, true)),
        loggerId: view.getUint16(40, true),
        processorIndex: view.getUint8(42),
        state: view.getUint32(44, true),
        offset: view.getUint32(48, true), // Adjusted offsets for JS
        bufferFlag: view.getUint16(52, true),
        bufferType: view.getUint16(54, true),
        referenceTimeTime: Number(view.getBigInt64(56, true)),
        referenceTimeFreq: Number(view.getBigInt64(64, true)),
      };
      this.buffers.push(buf);
      position += bufferSize;
    }
  }

  printProperties() {
    this.buffers.forEach((buf, idx) => {
      console.log(`Buffer ${idx + 1}:`);
      Object.entries(buf).forEach(([key, value]) => {
        console.log(`  ${key}: ${value}`);
      });
      console.log('');
    });
  }

  write(newFilename) {
    const totalLength = this.buffers.reduce((acc, buf) => acc + buf.bufferSize, 0);
    const outputBuffer = Buffer.alloc(totalLength);
    let position = 0;
    this.buffers.forEach(buf => {
      const view = new DataView(outputBuffer.buffer, outputBuffer.byteOffset + position, 0x48);
      view.setUint32(0, buf.bufferSize, true);
      view.setUint32(4, buf.savedOffset, true);
      view.setUint32(8, buf.currentOffset, true);
      view.setInt32(12, buf.referenceCount, true);
      view.setBigInt64(16, BigInt(buf.timeStamp), true);
      view.setUint32(24, buf.sequenceNumber, true);
      view.setUint32(28, buf.clockType, true);
      view.setBigInt64(32, BigInt(buf.frequency), true);
      view.setUint16(40, buf.loggerId, true);
      view.setUint8(42, buf.processorIndex);
      view.setUint32(44, buf.state, true);
      view.setUint32(48, buf.offset, true);
      view.setUint16(52, buf.bufferFlag, true);
      view.setUint16(54, buf.bufferType, true);
      view.setBigInt64(56, BigInt(buf.referenceTimeTime), true);
      view.setBigInt64(64, BigInt(buf.referenceTimeFreq), true);
      // Pad the rest with zeros
      position += buf.bufferSize;
    });
    fs.writeFileSync(newFilename, outputBuffer);
  }
}

// Example usage:
 // const etl = new ETLFile('sample.etl');
 // etl.read();
 // etl.printProperties();
 // etl.write('new.etl');

1. C class to open, decode, read, write, and print properties to console:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>

typedef struct {
  uint32_t bufferSize;
  uint32_t savedOffset;
  uint32_t currentOffset;
  int32_t referenceCount;
  int64_t timeStamp;
  uint32_t sequenceNumber;
  uint32_t clockType;
  int64_t frequency;
  uint16_t loggerId;
  uint8_t processorIndex;
  uint8_t alignment; // Padding
  uint32_t state;
  uint32_t offset;
  uint16_t bufferFlag;
  uint16_t bufferType;
  int64_t referenceTimeTime;
  int64_t referenceTimeFreq;
} BufferHeader;

typedef struct {
  BufferHeader* buffers;
  size_t numBuffers;
} ETLFile;

ETLFile* etl_open(const char* filename) {
  FILE* f = fopen(filename, "rb");
  if (!f) return NULL;

  fseek(f, 0, SEEK_END);
  size_t size = ftell(f);
  fseek(f, 0, SEEK_SET);

  uint8_t* data = malloc(size);
  fread(data, 1, size, f);
  fclose(f);

  ETLFile* etl = malloc(sizeof(ETLFile));
  etl->buffers = NULL;
  etl->numBuffers = 0;

  size_t position = 0;
  while (position < size) {
    if (size - position < 0x48) break;
    BufferHeader header;
    memcpy(&header, data + position, 0x48);
    if (header.bufferSize <= 0x48 || position + header.bufferSize > size) break;
    etl->buffers = realloc(etl->buffers, sizeof(BufferHeader) * (etl->numBuffers + 1));
    etl->buffers[etl->numBuffers] = header;
    etl->numBuffers++;
    position += header.bufferSize;
  }
  free(data);
  return etl;
}

void etl_print_properties(ETLFile* etl) {
  for (size_t idx = 0; idx < etl->numBuffers; idx++) {
    BufferHeader* buf = &etl->buffers[idx];
    printf("Buffer %zu:\n", idx + 1);
    printf("  BufferSize: %u\n", buf->bufferSize);
    printf("  SavedOffset: %u\n", buf->savedOffset);
    printf("  CurrentOffset: %u\n", buf->currentOffset);
    printf("  ReferenceCount: %d\n", buf->referenceCount);
    printf("  TimeStamp: %lld\n", (long long)buf->timeStamp);
    printf("  SequenceNumber: %u\n", buf->sequenceNumber);
    printf("  ClockType: %u\n", buf->clockType);
    printf("  Frequency: %lld\n", (long long)buf->frequency);
    printf("  LoggerId: %hu\n", buf->loggerId);
    printf("  ProcessorIndex: %u\n", buf->processorIndex);
    printf("  State: %u\n", buf->state);
    printf("  Offset: %u\n", buf->offset);
    printf("  BufferFlag: %hu\n", buf->bufferFlag);
    printf("  BufferType: %hu\n", buf->bufferType);
    printf("  ReferenceTime.Time: %lld\n", (long long)buf->referenceTimeTime);
    printf("  ReferenceTime.Freq: %lld\n", (long long)buf->referenceTimeFreq);
    printf("\n");
  }
}

void etl_write(ETLFile* etl, const char* newFilename) {
  FILE* f = fopen(newFilename, "wb");
  if (!f) return;

  for (size_t idx = 0; idx < etl->numBuffers; idx++) {
    BufferHeader* buf = &etl->buffers[idx];
    fwrite(buf, 1, 0x48, f);
    // Pad with zeros for event data
    size_t padSize = buf->bufferSize - 0x48;
    uint8_t* pad = calloc(padSize, 1);
    fwrite(pad, 1, padSize, f);
    free(pad);
  }
  fclose(f);
}

void etl_close(ETLFile* etl) {
  free(etl->buffers);
  free(etl);
}

// Example usage:
// int main() {
//   ETLFile* etl = etl_open("sample.etl");
//   etl_print_properties(etl);
//   etl_write(etl, "new.etl");
//   etl_close(etl);
//   return 0;
// }