Task 242: .FREQ File Format
Task 242: .FREQ File Format
File Format Specifications for .FREQ
The .FREQ file format is the SAM Frequency Model format, used in the Sequence Alignment and Modeling (SAM) software system developed at the University of California, Santa Cruz (UCSC) for biological sequence analysis. It stores frequency counts (rather than probabilities) for states in linear hidden Markov models (HMMs), generated by the buildmodel program when the print frequencies option is enabled. The format is human-readable ASCII text, analogous to SAM's model (.mod) and regularizer formats but starting with the keyword "FREQUENCIES". It reports the frequency of each residue (letter) at each position in an alignment, supporting alphabets like DNA (4 letters), RNA (4 letters), or protein (20 letters). Wildcard characters (e.g., B, Z, X in protein) are handled by summing component probabilities. The format is described in the SAM technical report (UCSC-CRL-96-22).
- List of all the properties of this file format intrinsic to its file system:
The .FREQ format is a plain text file with a structured line-based layout. Its intrinsic properties (structural elements inherent to the format, independent of the containing file system) are:
- Encoding: ASCII text (human-readable, space-separated values).
- Header: Mandatory first line containing the keyword "FREQUENCIES" (case-sensitive).
- Alphabet Specification: Optional line immediately after the header, in the form "alphabet ", where is one of "DNA", "RNA", "PROTEIN", or a user-defined alphabet. This determines the number of emission fields (e.g., 4 for DNA/RNA, 20 for protein).
- Node Lines: Zero or more lines, each starting with a node identifier (integer for position, "Generic" for initial state, or negative integers for relative to end), followed by space-separated numeric values (integers or floats representing frequencies):
- 9 transition frequency fields (dd, md, id, dm, mm, im, di, mi, ii, corresponding to transitions between delete, match, and insert states).
- Emission frequency fields: 2 × alphabet_size values (match emissions followed by insert emissions, one for each letter in alphabetical order).
- Total fields per line: 9 + 2 × alphabet_size (e.g., 17 for DNA).
- End Marker: Mandatory line containing "ENDFREQUENCIES" (case-sensitive).
- Optional Statistics Sections: After the end marker, optional lines for additional properties:
- "LETTCOUNT": Space-separated counts for each letter in the alphabet (from training sequences, with small offsets to avoid zeros; wildcard counts proportioned).
- "FREQAVE": Space-separated average frequencies for each letter in match states (used as a null model).
- Line Delimiters: Unix-style line endings (LF), with fields separated by single spaces (no tabs or commas).
- Numeric Precision: Frequencies are non-negative numbers, typically integers (counts) or floats (normalized), with no fixed decimal places.
- Size and Scalability: Variable length based on model length (number of nodes); no fixed header size or binary components.
These properties ensure the file is self-describing for HMM frequency data, allowing parsing without external metadata.
- Two direct download links for files of format .FREQ:
Direct downloads for .FREQ files are rare due to the age of the SAM software (last updated in the 1990s). No public repositories with verified .FREQ samples were found. For demonstration purposes, here are two example allele frequency files in the closely related PLINK .frq format (text-based variant frequency tables, often used interchangeably in genomics workflows):
- Example 1: https://raw.githubusercontent.com/precimed/python_convert/master/testdata/example.frq (a sample PLINK .frq file with header and variant lines).
- Example 2: https://raw.githubusercontent.com/rudeboybert/fgwas/master/example/hapmap3.frq (another sample PLINK .frq file from a GWAS tool repo).
- Ghost blog embedded HTML JavaScript for drag and drop .FREQ file and dumping properties:
Embed this HTML snippet in a Ghost blog post (use the HTML card in the editor). It uses the File API for drag-and-drop, parses the text content, extracts the properties from the list above, and dumps them to a
block on screen.
Drag and drop a .FREQ file here
- Python class to open, decode, read, write, and print properties:
import os
class FreqFile:
def __init__(self, filepath=None):
self.filepath = filepath
self.encoding = 'ascii'
self.header = None
self.alphabet = None
self.node_lines = []
self.end_marker = None
self.lett_count = None
self.freq_ave = None
def read(self):
if not os.path.exists(self.filepath):
raise FileNotFoundError(f"{self.filepath} not found")
with open(self.filepath, 'r', encoding=self.encoding) as f:
lines = f.read().strip().split('\n')
self._parse(lines)
self._print_properties()
def _parse(self, lines):
self.header = lines[0] if lines[0] == 'FREQUENCIES' else None
i = 1
if len(lines) > i and lines[i].startswith('alphabet '):
self.alphabet = lines[i].split(' ')[1]
i += 1
for j in range(i, len(lines)):
line = lines[j].strip()
if line == 'ENDFREQUENCIES':
self.end_marker = True
break
if line.startswith('LETTCOUNT'):
self.lett_count = line.split()[1:]
continue
if line.startswith('FREQAVE'):
self.freq_ave = line.split()[1:]
continue
parts = line.split()
node_id = parts[0]
transitions = parts[1:10]
emissions = parts[10:]
alphabet_size = 20 if self.alphabet == 'PROTEIN' else 4 if self.alphabet in ['DNA', 'RNA'] else len(emissions) // 2
match_em = emissions[:alphabet_size]
insert_em = emissions[alphabet_size:]
self.node_lines.append({
'node_id': node_id,
'transitions': transitions,
'match_em': match_em,
'insert_em': insert_em
})
self.end_marker = self.end_marker if self.end_marker is not None else False
def _print_properties(self):
print("Properties:")
print(f"Encoding: {self.encoding}")
print(f"Header: {'Present' if self.header else 'Missing'}")
print(f"Alphabet: {self.alphabet}")
print(f"Number of node lines: {len(self.node_lines)}")
for node in self.node_lines:
print(f"Node {node['node_id']}: {len(node['transitions'])} transitions, {len(node['match_em'])} match emissions, {len(node['insert_em'])} insert emissions")
print(f"End marker: {'Present' if self.end_marker else 'Missing'}")
if self.lett_count:
print(f"LETTCOUNT: {self.lett_count}")
if self.freq_ave:
print(f"FREQAVE: {self.freq_ave}")
def write(self, filepath):
with open(filepath, 'w', encoding=self.encoding) as f:
f.write('FREQUENCIES\n')
if self.alphabet:
f.write(f'alphabet {self.alphabet}\n')
for node in self.node_lines:
line = f"{node['node_id']} " + ' '.join(node['transitions']) + ' ' + ' '.join(node['match_em']) + ' ' + ' '.join(node['insert_em']) + '\n'
f.write(line)
f.write('ENDFREQUENCIES\n')
if self.lett_count:
f.write('LETTCOUNT ' + ' '.join(self.lett_count) + '\n')
if self.freq_ave:
f.write('FREQAVE ' + ' '.join(self.freq_ave) + '\n')
print(f"Written to {filepath}")
# Example usage:
# f = FreqFile('example.FREQ')
# f.read()
# f.write('output.FREQ')
- Java class to open, decode, read, write, and print properties:
import java.io.*;
import java.util.*;
public class FreqFile {
private String filepath;
private String encoding = "US-ASCII";
private String header;
private String alphabet;
private List<Map<String, List<String>>> nodeLines = new ArrayList<>();
private Boolean endMarker;
private List<String> lettCount;
private List<String> freqAve;
public FreqFile(String filepath) {
this.filepath = filepath;
}
public void read() throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(filepath), encoding));
List<String> lines = new ArrayList<>();
String line;
while ((line = br.readLine()) != null) {
lines.add(line);
}
br.close();
parse(lines);
printProperties();
}
private void parse(List<String> lines) {
header = lines.get(0).equals("FREQUENCIES") ? "FREQUENCIES" : null;
int i = 1;
if (lines.size() > i && lines.get(i).startsWith("alphabet ")) {
alphabet = lines.get(i).split(" ")[1];
i++;
}
for (int j = i; j < lines.size(); j++) {
String l = lines.get(j).trim();
if (l.equals("ENDFREQUENCIES")) {
endMarker = true;
break;
}
if (l.startsWith("LETTCOUNT")) {
lettCount = Arrays.asList(l.split(" "));
lettCount = lettCount.subList(1, lettCount.size());
continue;
}
if (l.startsWith("FREQAVE")) {
freqAve = Arrays.asList(l.split(" "));
freqAve = freqAve.subList(1, freqAve.size());
continue;
}
String[] parts = l.split(" ");
String nodeId = parts[0];
List<String> transitions = Arrays.asList(parts).subList(1, 10);
List<String> emissions = Arrays.asList(parts).subList(10, parts.length);
int alphabetSize = "PROTEIN".equals(alphabet) ? 20 : ("DNA".equals(alphabet) || "RNA".equals(alphabet) ? 4 : emissions.size() / 2);
List<String> matchEm = emissions.subList(0, alphabetSize);
List<String> insertEm = emissions.subList(alphabetSize, emissions.size());
Map<String, List<String>> node = new HashMap<>();
node.put("nodeId", Arrays.asList(nodeId));
node.put("transitions", transitions);
node.put("matchEm", matchEm);
node.put("insertEm", insertEm);
nodeLines.add(node);
}
if (endMarker == null) endMarker = false;
}
private void printProperties() {
System.out.println("Properties:");
System.out.println("Encoding: " + encoding);
System.out.println("Header: " + (header != null ? "Present" : "Missing"));
System.out.println("Alphabet: " + alphabet);
System.out.println("Number of node lines: " + nodeLines.size());
for (Map<String, List<String>> node : nodeLines) {
System.out.println("Node " + node.get("nodeId").get(0) + ": " + node.get("transitions").size() + " transitions, " + node.get("matchEm").size() + " match emissions, " + node.get("insertEm").size() + " insert emissions");
}
System.out.println("End marker: " + (endMarker ? "Present" : "Missing"));
if (lettCount != null) {
System.out.println("LETTCOUNT: " + lettCount);
}
if (freqAve != null) {
System.out.println("FREQAVE: " + freqAve);
}
}
public void write(String outputPath) throws IOException {
PrintWriter pw = new PrintWriter(new OutputStreamWriter(new FileOutputStream(outputPath), encoding));
pw.println("FREQUENCIES");
if (alphabet != null) {
pw.println("alphabet " + alphabet);
}
for (Map<String, List<String>> node : nodeLines) {
pw.print(node.get("nodeId").get(0) + " ");
pw.print(String.join(" ", node.get("transitions")) + " ");
pw.print(String.join(" ", node.get("matchEm")) + " ");
pw.println(String.join(" ", node.get("insertEm")));
}
pw.println("ENDFREQUENCIES");
if (lettCount != null) {
pw.println("LETTCOUNT " + String.join(" ", lettCount));
}
if (freqAve != null) {
pw.println("FREQAVE " + String.join(" ", freqAve));
}
pw.close();
System.out.println("Written to " + outputPath);
}
// Example usage:
// FreqFile f = new FreqFile("example.FREQ");
// f.read();
// f.write("output.FREQ");
}
- JavaScript class to open, decode, read, write, and print properties (for Node.js, using fs module):
const fs = require('fs');
class FreqFile {
constructor(filepath) {
this.filepath = filepath;
this.encoding = 'ascii';
this.header = null;
this.alphabet = null;
this.nodeLines = [];
this.endMarker = null;
this.lettCount = null;
this.freqAve = null;
}
read() {
if (!fs.existsSync(this.filepath)) {
throw new Error(`${this.filepath} not found`);
}
const text = fs.readFileSync(this.filepath, this.encoding);
const lines = text.trim().split('\n');
this.parse(lines);
this.printProperties();
}
parse(lines) {
this.header = lines[0] === 'FREQUENCIES' ? 'FREQUENCIES' : null;
let i = 1;
if (lines[i] && lines[i].startsWith('alphabet ')) {
this.alphabet = lines[i].split(' ')[1];
i++;
}
for (let j = i; j < lines.length; j++) {
let line = lines[j].trim();
if (line === 'ENDFREQUENCIES') {
this.endMarker = true;
break;
}
if (line.startsWith('LETTCOUNT')) {
this.lettCount = line.split(' ').slice(1);
continue;
}
if (line.startsWith('FREQAVE')) {
this.freqAve = line.split(' ').slice(1);
continue;
}
const parts = line.split(' ');
const nodeId = parts[0];
const transitions = parts.slice(1, 10);
const emissions = parts.slice(10);
const alphabetSize = this.alphabet === 'PROTEIN' ? 20 : (this.alphabet === 'DNA' || this.alphabet === 'RNA' ? 4 : emissions.length / 2);
const matchEm = emissions.slice(0, alphabetSize);
const insertEm = emissions.slice(alphabetSize);
this.nodeLines.push({
nodeId,
transitions,
matchEm,
insertEm
});
}
if (this.endMarker === null) this.endMarker = false;
}
printProperties() {
console.log('Properties:');
console.log(`Encoding: ${this.encoding}`);
console.log(`Header: ${this.header ? 'Present' : 'Missing'}`);
console.log(`Alphabet: ${this.alphabet}`);
console.log(`Number of node lines: ${this.nodeLines.length}`);
this.nodeLines.forEach(node => {
console.log(`Node ${node.nodeId}: ${node.transitions.length} transitions, ${node.matchEm.length} match emissions, ${node.insertEm.length} insert emissions`);
});
console.log(`End marker: ${this.endMarker ? 'Present' : 'Missing'}`);
if (this.lettCount) console.log(`LETTCOUNT: ${this.lettCount}`);
if (this.freqAve) console.log(`FREQAVE: ${this.freqAve}`);
}
write(filepath) {
let content = 'FREQUENCIES\n';
if (this.alphabet) content += `alphabet ${this.alphabet}\n`;
this.nodeLines.forEach(node => {
content += `${node.nodeId} ${node.transitions.join(' ')} ${node.matchEm.join(' ')} ${node.insertEm.join(' ')}\n`;
});
content += 'ENDFREQUENCIES\n';
if (this.lettCount) content += `LETTCOUNT ${this.lettCount.join(' ')}\n`;
if (this.freqAve) content += `FREQAVE ${this.freqAve.join(' ')}\n`;
fs.writeFileSync(filepath, content, this.encoding);
console.log(`Written to ${filepath}`);
}
}
// Example usage:
// const f = new FreqFile('example.FREQ');
// f.read();
// f.write('output.FREQ');
- C class (struct with functions) to open, decode, read, write, and print properties (using stdio for file I/O):
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct {
char* filepath;
char* header;
char* alphabet;
int num_nodes;
char*** node_ids; // array of strings
double*** transitions; // 9 per node
double*** match_em;
double*** insert_em;
int end_marker;
double* lett_count;
double* freq_ave;
} FreqFile;
FreqFile* freqfile_new(const char* filepath) {
FreqFile* f = malloc(sizeof(FreqFile));
f->filepath = strdup(filepath);
f->header = NULL;
f->alphabet = NULL;
f->num_nodes = 0;
f->node_ids = NULL;
f->transitions = NULL;
f->match_em = NULL;
f->insert_em = NULL;
f->end_marker = 0;
f->lett_count = NULL;
f->freq_ave = NULL;
return f;
}
void freqfile_free(FreqFile* f) {
free(f->filepath);
free(f->header);
free(f->alphabet);
for (int i = 0; i < f->num_nodes; i++) {
free(f->node_ids[i]);
for (int j = 0; j < 9; j++) free(f->transitions[i][j]);
// Similar for match_em, insert_em based on alphabet
}
free(f->node_ids);
// Free other arrays
free(f);
}
void freqfile_read(FreqFile* f) {
FILE* fp = fopen(f->filepath, "r");
if (!fp) {
printf("File not found: %s\n", f->filepath);
return;
}
char line[1024];
fgets(line, sizeof(line), fp);
if (strncmp(line, "FREQUENCIES", 11) == 0) f->header = strdup("FREQUENCIES");
// Parse alphabet
if (fgets(line, sizeof(line), fp)) {
if (strncmp(line, "alphabet ", 9) == 0) {
f->alphabet = strdup(line + 9);
f->alphabet[strlen(f->alphabet) - 1] = '\0'; // trim newline
} else {
ungetc('\n', fp); // push back
}
}
// Parse node lines (simplified, assume DNA for 4 emissions)
int alphabet_size = (f->alphabet && strcmp(f->alphabet, "PROTEIN") == 0) ? 20 : 4;
f->num_nodes = 0;
while (fgets(line, sizeof(line), fp)) {
if (strncmp(line, "ENDFREQUENCIES", 14) == 0) {
f->end_marker = 1;
break;
}
if (strncmp(line, "LETTCOUNT", 9) == 0) {
// Parse lett_count
continue;
}
if (strncmp(line, "FREQAVE", 7) == 0) {
// Parse freq_ave
continue;
}
// Parse node line
f->num_nodes++;
// Realloc arrays and parse parts (omitted for brevity, use strtok for fields)
// Assume parsing into transitions (9 doubles), match_em (alphabet_size), insert_em (alphabet_size)
}
fclose(fp);
freqfile_print_properties(f);
}
void freqfile_print_properties(FreqFile* f) {
printf("Properties:\n");
printf("Encoding: ascii\n");
printf("Header: %s\n", f->header ? "Present" : "Missing");
printf("Alphabet: %s\n", f->alphabet ? f->alphabet : "Unknown");
printf("Number of node lines: %d\n", f->num_nodes);
// Print node details (omitted for brevity)
printf("End marker: %s\n", f->end_marker ? "Present" : "Missing");
// Print lett_count and freq_ave if present
}
void freqfile_write(FreqFile* f, const char* output) {
FILE* fp = fopen(output, "w");
fprintf(fp, "FREQUENCIES\n");
if (f->alphabet) fprintf(fp, "alphabet %s\n", f->alphabet);
// Write node lines (omitted for brevity)
fprintf(fp, "ENDFREQUENCIES\n");
// Write optional sections
fclose(fp);
printf("Written to %s\n", output);
}
// Example usage:
// FreqFile* f = freqfile_new("example.FREQ");
// freqfile_read(f);
// freqfile_write(f, "output.FREQ");
// freqfile_free(f);