Interpreter

The Pfp interpreter is quite simple: it uses py010parser to parse the template into an abstract-syntax-tree, and then handles each of the nodes in the tree appropriately.

The main method for handling nodes is the _handle_node function. The _handle_node function performs basic housekeeping, logging, decides if the user should be dropped into the interactive debugger, and of course, handles the node itself.

If a methods are not implemented to handle a certain AST node, an pfp.errors.UnsupportedASTNode error will be raised. Implemented methods to handle AST node types are found in the _node_switch dict:

self._node_switch = {
        AST.FileAST:            self._handle_file_ast,
        AST.Decl:                       self._handle_decl,
        AST.TypeDecl:           self._handle_type_decl,
        AST.ByRefDecl:          self._handle_byref_decl,
        AST.Struct:                     self._handle_struct,
        AST.Union:                      self._handle_union,
        AST.StructRef:          self._handle_struct_ref,
        AST.IdentifierType:     self._handle_identifier_type,
        AST.Typedef:            self._handle_typedef,
        AST.Constant:           self._handle_constant,
        AST.BinaryOp:           self._handle_binary_op,
        AST.Assignment:         self._handle_assignment,
        AST.ID:                         self._handle_id,
        AST.UnaryOp:            self._handle_unary_op,
        AST.FuncDef:            self._handle_func_def,
        AST.FuncCall:           self._handle_func_call,
        AST.FuncDecl:           self._handle_func_decl,
        AST.ParamList:          self._handle_param_list,
        AST.ExprList:           self._handle_expr_list,
        AST.Compound:           self._handle_compound,
        AST.Return:                     self._handle_return,
        AST.ArrayDecl:          self._handle_array_decl,
        AST.InitList:           self._handle_init_list,
        AST.If:                         self._handle_if,
        AST.For:                        self._handle_for,
        AST.While:                      self._handle_while,
        AST.DeclList:           self._handle_decl_list,
        AST.Break:                      self._handle_break,
        AST.Continue:           self._handle_continue,
        AST.ArrayRef:           self._handle_array_ref,
        AST.Enum:                       self._handle_enum,
        AST.Switch:                     self._handle_switch,
        AST.Cast:                       self._handle_cast,
        AST.Typename:           self._handle_typename,
        AST.EmptyStatement: self._handle_empty_statement,

        StructDecls:            self._handle_struct_decls,
        UnionDecls:                     self._handle_union_decls,
}

Interpreter Reference Documentation

Python format parser

pfp.interp.LazyField(lookup_name, scope)[source]

Super non-standard stuff here. Dynamically changing the base class using the scope and the lazy name when the class is instantiated. This works as long as the original base class is not directly inheriting from object (which we’re not, since our original base class is fields.Field).

class pfp.interp.PfpInterp(debug=False, parser=None, int3=True)[source]
classmethod add_native(name, func, ret, interp=None, send_interp=False)[source]

Add the native python function func into the pfp interpreter with the name name and return value ret so that it can be called from within a template script.

Note

The @native decorator exists to simplify this.

All native functions must have the signature def func(params, ctxt, scope, stream, coord [,interp]), optionally allowing an interpreter param if send_interp is True.

Example:

The example below defines a function Sum using the add_native method.

import pfp.fields
from pfp.fields import PYVAL

def native_sum(params, ctxt, scope, stream, coord):
    return PYVAL(params[0]) + PYVAL(params[1])

pfp.interp.PfpInterp.add_native("Sum", native_sum, pfp.fields.Int64)
Parameters:
  • name (basestring) – The name the function will be exposed as in the interpreter.
  • func (function) – The native python function that will be referenced.
  • ret (type(pfp.fields.Field)) – The field class that the return value should be cast to.
  • interp (pfp.interp.PfpInterp) – The specific pfp interpreter the function should be defined in.
  • send_interp (bool) – If true, the current pfp interpreter will be added as an argument to the function.
classmethod add_predefine(template)[source]

Add a template that should be run prior to running any other templates. This is useful for predefining types, etc.

Parameters:template (basestring) – The template text (unicode is also fine here)
cont()[source]

Continue the interpreter

classmethod define_natives()[source]

Define the native functions for PFP

eval(statement, ctxt=None)[source]

Eval a single statement (something returnable)

get_bitfield_direction()[source]

Return if the bitfield direction

Note

This should be applied AFTER taking into account endianness.

get_bitfield_padded()[source]

Return if the bitfield input/output stream should be padded

Returns:True/False
get_curr_lines()[source]

Return the current line number in the template, as well as the surrounding source lines

get_filename()[source]

Return the filename of the data that is currently being parsed

Returns:The name of the data file being parsed.
get_types()[source]

Return a types object that will contain all of the typedefd structs’ classes.

Returns:Types object

Example:

Create a new PNG_CHUNK object from a PNG_CHUNK type that was defined in a template:

types = interp.get_types() chunk = types.PNG_CHUNK()

load_template(template)[source]

Load a template and all required predefines into this interpreter. Future calls to parse will not require the template to be parsed.

parse(stream, template=None, predefines=True, orig_filename=None, keep_successful=False, printf=True)[source]

Parse the data stream using the template (e.g. parse the 010 template and interpret the template using the stream as the data source).

Stream:The input data stream
Template:The template to parse the stream with
Keep_successful:
 Return whatever was successfully parsed before an error. _pfp__error will contain the exception (if one was raised)
Parameters:printf (bool) – If False, printfs will be noops (default=``True``)
Returns:Pfp Dom
set_bitfield_direction(val)[source]

Set the bitfields to parse from left to right (1), the default (None), or right to left (-1)

set_bitfield_padded(val)[source]

Set if the bitfield input/output stream should be padded

Val:True/False
Returns:None
set_break(break_type)[source]

Set if the interpreter should break.

Returns:TODO
step_into()[source]

Step over/into the next statement

step_over()[source]

Perform one step of the interpreter

class pfp.interp.PfpTypes(interp, scope)[source]

A class to hold all typedefd types in a template. Note that types are instantiated by having them parse a null-stream. This means that type creation will not work correctly for complicated structs that have internal control-flow

class pfp.interp.Scope(logger, parent=None)[source]

A class to keep track of the current scope of the interpreter

add_local(field_name, field)[source]

Add a local variable in the current scope

Field_name:The field’s name
Field:The field
Returns:None
add_refd_struct_or_union(name, refd_name, interp, node)[source]

Add a lazily-looked up typedef struct or union

Name:name of the typedefd struct/union
Node:the typedef node
Interp:the 010 interpreter
add_type(new_name, orig_names)[source]

Record the typedefd name for orig_names. Resolve orig_names to their core names and save those.

New_name:TODO
Orig_names:TODO
Returns:TODO
add_type_class(name, cls)[source]

Store the class with the name

add_type_struct_or_union(name, interp, node)[source]

Store the node with the name. When it is instantiated, the node itself will be handled.

Name:name of the typedefd struct/union
Node:the union/struct node
Interp:the 010 interpreter
add_var(field_name, field, root=False)[source]

Add a var to the current scope (vars are fields that parse the input stream)

Field_name:TODO
Field:TODO
Returns:TODO
clear_meta()[source]

Clear metadata about the current statement

clone()[source]

Return a new Scope object that has the curr_scope pinned at the current one :returns: A new scope object

get_id(name, recurse=True)[source]

Get the first id matching name. Will either be a local or a var.

Name:TODO
Returns:TODO
get_local(name, recurse=True)[source]

Get the local field (search for it) from the scope stack. An alias for get_var

Name:The name of the local field
get_meta(meta_name)[source]

Get the current meta value named meta_name

get_type(name, recurse=True)[source]

Get the names for the typename (created by typedef)

Name:The typedef’d name to resolve
Returns:An array of resolved names associated with the typedef’d name
get_var(name, recurse=True)[source]

Return the first var of name name in the current scope stack (remember, vars are the ones that parse the input stream)

Name:The name of the id
Recurse:Whether parent scopes should also be searched (defaults to True)
Returns:TODO
level()[source]

Return the current scope level

pop()[source]

Leave the current scope :returns: TODO

pop_meta(name)[source]

Pop metadata about the current statement from the metadata stack for the current statement.

Name:The name of the metadata
push(new_scope=None)[source]

Create a new scope :returns: TODO

push_meta(meta_name, meta_value)[source]

Push metadata about the current statement onto the metadata stack for the current statement. Mostly used for tracking integer promotion and casting types

pfp.interp.StructUnionTypeRef(curr_scope, typedef_name, refd_name, interp, node)[source]

Create a typedef that resolves itself dynamically. This is needed in situations like:

struct MY_STRUCT {
    char magic[4];
    unsigned int filesize;
};
typedef struct MY_STRUCT ME;
LittleEndian();
ME s;

The typedef ME is handled before the MY_STRUCT declaration actually occurs. The typedef value for ME should not the empty struct that is resolved, but should be a dynamically-looked up struct definition when a ME instance is actually declared.

Python format parser

pfp.interp.LazyField(lookup_name, scope)[source]

Super non-standard stuff here. Dynamically changing the base class using the scope and the lazy name when the class is instantiated. This works as long as the original base class is not directly inheriting from object (which we’re not, since our original base class is fields.Field).

class pfp.interp.PfpInterp(debug=False, parser=None, int3=True)[source]
classmethod add_native(name, func, ret, interp=None, send_interp=False)[source]

Add the native python function func into the pfp interpreter with the name name and return value ret so that it can be called from within a template script.

Note

The @native decorator exists to simplify this.

All native functions must have the signature def func(params, ctxt, scope, stream, coord [,interp]), optionally allowing an interpreter param if send_interp is True.

Example:

The example below defines a function Sum using the add_native method.

import pfp.fields
from pfp.fields import PYVAL

def native_sum(params, ctxt, scope, stream, coord):
    return PYVAL(params[0]) + PYVAL(params[1])

pfp.interp.PfpInterp.add_native("Sum", native_sum, pfp.fields.Int64)
Parameters:
  • name (basestring) – The name the function will be exposed as in the interpreter.
  • func (function) – The native python function that will be referenced.
  • ret (type(pfp.fields.Field)) – The field class that the return value should be cast to.
  • interp (pfp.interp.PfpInterp) – The specific pfp interpreter the function should be defined in.
  • send_interp (bool) – If true, the current pfp interpreter will be added as an argument to the function.
classmethod add_predefine(template)[source]

Add a template that should be run prior to running any other templates. This is useful for predefining types, etc.

Parameters:template (basestring) – The template text (unicode is also fine here)
cont()[source]

Continue the interpreter

classmethod define_natives()[source]

Define the native functions for PFP

eval(statement, ctxt=None)[source]

Eval a single statement (something returnable)

get_bitfield_direction()[source]

Return if the bitfield direction

Note

This should be applied AFTER taking into account endianness.

get_bitfield_padded()[source]

Return if the bitfield input/output stream should be padded

Returns:True/False
get_curr_lines()[source]

Return the current line number in the template, as well as the surrounding source lines

get_filename()[source]

Return the filename of the data that is currently being parsed

Returns:The name of the data file being parsed.
get_types()[source]

Return a types object that will contain all of the typedefd structs’ classes.

Returns:Types object

Example:

Create a new PNG_CHUNK object from a PNG_CHUNK type that was defined in a template:

types = interp.get_types() chunk = types.PNG_CHUNK()

load_template(template)[source]

Load a template and all required predefines into this interpreter. Future calls to parse will not require the template to be parsed.

parse(stream, template=None, predefines=True, orig_filename=None, keep_successful=False, printf=True)[source]

Parse the data stream using the template (e.g. parse the 010 template and interpret the template using the stream as the data source).

Stream:The input data stream
Template:The template to parse the stream with
Keep_successful:
 Return whatever was successfully parsed before an error. _pfp__error will contain the exception (if one was raised)
Parameters:printf (bool) – If False, printfs will be noops (default=``True``)
Returns:Pfp Dom
set_bitfield_direction(val)[source]

Set the bitfields to parse from left to right (1), the default (None), or right to left (-1)

set_bitfield_padded(val)[source]

Set if the bitfield input/output stream should be padded

Val:True/False
Returns:None
set_break(break_type)[source]

Set if the interpreter should break.

Returns:TODO
step_into()[source]

Step over/into the next statement

step_over()[source]

Perform one step of the interpreter

class pfp.interp.PfpTypes(interp, scope)[source]

A class to hold all typedefd types in a template. Note that types are instantiated by having them parse a null-stream. This means that type creation will not work correctly for complicated structs that have internal control-flow

class pfp.interp.Scope(logger, parent=None)[source]

A class to keep track of the current scope of the interpreter

add_local(field_name, field)[source]

Add a local variable in the current scope

Field_name:The field’s name
Field:The field
Returns:None
add_refd_struct_or_union(name, refd_name, interp, node)[source]

Add a lazily-looked up typedef struct or union

Name:name of the typedefd struct/union
Node:the typedef node
Interp:the 010 interpreter
add_type(new_name, orig_names)[source]

Record the typedefd name for orig_names. Resolve orig_names to their core names and save those.

New_name:TODO
Orig_names:TODO
Returns:TODO
add_type_class(name, cls)[source]

Store the class with the name

add_type_struct_or_union(name, interp, node)[source]

Store the node with the name. When it is instantiated, the node itself will be handled.

Name:name of the typedefd struct/union
Node:the union/struct node
Interp:the 010 interpreter
add_var(field_name, field, root=False)[source]

Add a var to the current scope (vars are fields that parse the input stream)

Field_name:TODO
Field:TODO
Returns:TODO
clear_meta()[source]

Clear metadata about the current statement

clone()[source]

Return a new Scope object that has the curr_scope pinned at the current one :returns: A new scope object

get_id(name, recurse=True)[source]

Get the first id matching name. Will either be a local or a var.

Name:TODO
Returns:TODO
get_local(name, recurse=True)[source]

Get the local field (search for it) from the scope stack. An alias for get_var

Name:The name of the local field
get_meta(meta_name)[source]

Get the current meta value named meta_name

get_type(name, recurse=True)[source]

Get the names for the typename (created by typedef)

Name:The typedef’d name to resolve
Returns:An array of resolved names associated with the typedef’d name
get_var(name, recurse=True)[source]

Return the first var of name name in the current scope stack (remember, vars are the ones that parse the input stream)

Name:The name of the id
Recurse:Whether parent scopes should also be searched (defaults to True)
Returns:TODO
level()[source]

Return the current scope level

pop()[source]

Leave the current scope :returns: TODO

pop_meta(name)[source]

Pop metadata about the current statement from the metadata stack for the current statement.

Name:The name of the metadata
push(new_scope=None)[source]

Create a new scope :returns: TODO

push_meta(meta_name, meta_value)[source]

Push metadata about the current statement onto the metadata stack for the current statement. Mostly used for tracking integer promotion and casting types

pfp.interp.StructUnionTypeRef(curr_scope, typedef_name, refd_name, interp, node)[source]

Create a typedef that resolves itself dynamically. This is needed in situations like:

struct MY_STRUCT {
    char magic[4];
    unsigned int filesize;
};
typedef struct MY_STRUCT ME;
LittleEndian();
ME s;

The typedef ME is handled before the MY_STRUCT declaration actually occurs. The typedef value for ME should not the empty struct that is resolved, but should be a dynamically-looked up struct definition when a ME instance is actually declared.