BAP Public API

==============

Abstract
--------

This patch adds JSON Public API to BAP. It also introduces some bug
fixes and extensions. As an example of API usage a small python binding
is provided.

Infrastructure Changes
----------------------

Since BAP now has more that 50 dependencies (most are optional), opam
needs further guidance, otherwise it can't find a proper solution. This
patch adds such guidance, properly specifying packages versions.

Also, this patch fixes warnings issued by a build system on a newer
compiler version.

Core Lwt library
----------------

This patch introduces a Core Lwt library, that is a thin wrapper around
Lwt library to provide a concise Core-like interface. I'm going to
remove this library from BAP sooner or later, and push it to opam as a
separate entity.

Consistent Constructor Names
----------------------------

In order to satisfy the requirements of ADT data format, and for
consistency purposes, I've title-cased all constructor names. This is a
breaking change.

Debugging Support
-----------------

Many were extended to support `sexp_of` or `sexp` protocols. Also, all
regular types got a plethora of new primitives, that can be used with
different format specifiers.

Changes to Bitvector
--------------------

Functions `of_int32` and `of_int64` now accepts optional `width`
parameter. Also, a new `string_of_value` function was added, that
converts to string only vector value, dropping information about the
size.

Changes to Image
----------------

Switched to bap's size type instead of core's.

Python Bindings
---------------

This PR introduces a python binding to BAP. This binding allows one to
disassemble arbitrary strings, and to load and analyze binary files. The
library is packed with distutils, that means, that in order to use, you
can just `pip install` it. The binding is rather deep, that means that
instead of textly typed dictionaries you will get first-class python
value of the appropriate type. All known to us ARM operands, registers
and instructions are lifted to python classes. See documentation for
more info.

Public API
----------

BAP now can be called from any language, using JSON API. This API is
implemented in a new program called `bap-server`. The server will also
take care on storage and persistence problems, i.e., it will store date
for you. Currently, the following set of protocols are implemented:

- http
- mmap
- file
- zmq

This set allows to use the following physical media for interaction:

- tcp and udp sockets
- unix sockets
- regular files
- shared memory

Author: ivg

__init__.py

@@ -0,0 +1,104 @@
+r"""Python inteface to BAP.
+
+In a few keystrokes:
+
+    >>> import bap
+    >>> print '\n'.join(insn.asm for insn in bap.disasm("\x48\x83\xec\x08"))
+        decl    %eax
+        subl    $0x8, %esp
+
+A more complex example:
+
+    >>> img = bap.image('coreutils_O0_ls')
+    >>> sym = img.get_symbol('main')
+    >>> print '\n'.join(insn.asm for insn in bap.disasm(sym))
+        push    {r11, lr}
+        add     r11, sp, #0x4
+        sub     sp, sp, #0xc8
+        ... <snip> ...
+
+Bap package exposes two functions:
+
+#. ``disasm`` returns a disassembly of the given object
+#. ``image``  loads given file
+
+Disassembling things
+====================
+
+``disasm`` is a swiss knife for disassembling things. It takes either a
+string object, or something returned by an ``image`` function, e.g.,
+images, sections and symbols.
+
+``disasm`` function returns a generator yielding instances of class
+``Insn`` defined in module :mod:`asm`. It has the following attributes:
+
+* name - instruction name, as undelying backend names it
+* addr - address of the first byte of instruction
+* size - overall size of the instruction
+* operands - list of instances of class ``Op``
+* asm - assembler string, in native assembler
+* kinds - instruction meta properties, see :mod:`asm`
+* target - instruction lifter to a target platform, e.g., see :mod:`arm`
+* bil - a list of BIL statements, describing instruction semantics.
+
+``disasm`` function also accepts a bunch of keyword arguments, to name a few:
+
+* server - either an url to a bap server or a dictionay containing port
+  and/or executable name
+* arch
+* endian  (instance of ``bil.Endian``)
+* addr    (should be an instance of type ``bil.Int``)
+* backend
+* stop_conditions
+
+All attributes are self-describing I hope. ``stop_conditions`` is a list of
+``Kind`` instances defined in :mod:`asm`. If disassembler meets instruction
+that is instance of one of this kind, it will stop.
+
+Reading files
+=============
+
+To read and analyze file one should load it with ``image``
+function. This function  returns an instance of class ``Image`` that
+allows one to discover information about the file, and perform different
+queries. It has function ``get_symbol`` function to lookup symbol in
+file by name, and the following set of attributes (self describing):
+
+* arch
+* entry_point
+* addr_size
+* endian
+* file (file name)
+* sections
+
+Sections is a list of instances of ``Section`` class, that also has a
+``get_symbol`` function and the following attributes:
+
+* name
+* perm (a list of ['r', 'w', 'x'])
+* addr
+* size
+* memory
+* symbols
+
+Symbols is a list of, you get it, ``Symbol`` class, each having the
+following attributes:
+
+* name
+* is_function
+* is_debug
+* addr
+* chunks
+
+Where chunks is a list of instances of ``Memory`` class, each having the
+following attributes:
+
+* addr
+* size
+* data
+
+Where data is actual string of bytes.
+"""
+__all__ = ['disasm', 'image']
+
+from .bap import disasm, image

adt.py

@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+"""
+Algebraic Data Types (ADT) is used to represent two kinds of things:
+
+1. A discrimintated union of types, called sum
+2. A combination of some types, called product.
+
+# Sum types
+
+Sum types represents a concept of generalizing. For example,
+on ARM R0 and R1 are all general purpose registers (GPR). Also on ARM
+we have Condition Code registers (CCR) :
+
+   class Reg(ADT) : pass
+   class GPR(Reg) : pass
+   class CCR(Reg) : pass
+   class R0(GPR)  : pass
+   class R1(GPR)  : pass
+
+
+That states that a register can be either R0 or R1, but not both.
+
+# Product types
+
+Product types represent a combination of other types. For example,
+mov instruction has two arguments, and the arguments are also ADT's by
+itself:
+
+   def Insn(ADT) : pass
+   def Mov(Insn) : pass
+
+   Mov(R0(), R1())
+
+
+# Comparison
+
+ADT objects are compared structurally: if they have the same class and
+and their values are structurally the same, then they are equal, i.e.,
+
+   assert(R0() == R0())
+   assert(R1() != R0())
+
+"""
+
+from collections import Iterable
+
+class ADT(object):
+    """ Algebraic Data Type.
+
+    This is a base class for all ADTs. ADT represented by a tuple of arguments,
+    stored in a val field. Arguments should be instances of ADT class, or numbers,
+    or strings. Empty set of arguments is permitted.
+    A one-tuple is automatically untupled, i.e., `Int(12)` has value `12`, not `(12,)`.
+    For convenience, a name of the constructor is provided in `name` field.
+
+    A structural comparison is provided.
+
+    """
+    def __init__(self, *args):
+        self.name = self.__class__.__name__
+        self.val = args if len(args) != 1 else args[0]
+
+    def __cmp__(self,other):
+        return self.__dict__.__cmp__(other.__dict__)
+
+    def __repr__(self):
+        def qstr(x):
+            if isinstance(x, int) or isinstance(x, ADT):
+                return str(x)
+            else:
+                return '"{0}"'.format(x)
+        def args():
+            if isinstance(self.val, tuple):
+                return ", ".join(qstr(x) for x in self.val)
+            else:
+                return qstr(self.val)
+
+        return "{0}({1})".format(self.name, args())
+
+
+class Visitor(object):
+    """ ADT Visitor.
+    This class helps to perform iterations over arbitrary ADTs.
+
+    This visitor supports, subtyping, i.e. you can match not only on
+    leaf constructors, but also on their bases. For example, with
+    the `Exp` hierarchy, provided below, you can visit all binary operators,
+    by overriding `visit_BinOp` method. See `run` method description for
+    more infromation.
+    """
+
+    def visit_ADT(self, adt):
+        """Default visitor.
+
+        This method will be called for those data types that has
+        no specific visitors. It will recursively descent into all
+        ADT values.
+        """
+        if isinstance(adt.val, tuple):
+            for e in adt.val:
+                self.run(e)
+
+    def run(self, adt):
+        """ADT.run(adt-or-iterable) -> None
+
+        if adt is iterable, the run is called recursively for each member
+        of adt.
+
+        Otherwise, for an ADT of type C the method `visit_C` is looked up in the
+        visitors methods dictionary. If it doesn't exist, then `visit_B` is
+        looked up, where `D` is the base class of `C`. The process continues,
+        until the method is found. This is guaranteed to terminate,
+        since visit_ADT method is defined.
+
+        Note: Non ADTs will be silently ignored.
+
+        Once the method is found it is called. It is the method's responsiblity
+        to recurse into sub-elements, e.g., call run method.
+
+        For example, suppose that we want to count negative values in a given
+        BIL expression:
+
+        class CountNegatives(Visitor):
+            def __init__(self):
+                self.neg = False
+                self.count = 0
+
+            def visit_Int(self, int):
+                if int.val < 0 and not self.neg \
+                  or int.val > 0 and self.neg:
+                    self.count += 1
+
+            def visit_NEG(self, op):
+                was = self.neg
+                self.neg = not was
+                self.run(op.val)
+                self.neg = was
+
+        We need to keep track on the unary negation operator, and, of
+        course, we need to look for immediates, so we override two methods:
+        visit_Int for Int constructor and visit_NEG for counting unary minuses.
+        (Actually we should count for bitwise NOT operation also, since it will
+        change the sign bit also, but lets forget about it for the matter of the
+        excercise (and it can be easily fixed just by matching visit_UnOp)).
+
+        When we hit visit_NEG we toggle current sign, storing its previous value
+        and recurse into the operand. After we return from the recursion, we restore
+        the sign.
+        """
+        if isinstance(adt, Iterable):
+            for s in adt:
+                self.run(s)
+        if isinstance(adt, ADT):
+            for c in adt.__class__.mro():
+                name = ("visit_%s" % c.__name__)
+                fn = getattr(self, name, None)
+                if fn is not None:
+                    return fn(adt)
+
+
+if __name__ == "__main__":
+    class Fruit(ADT) : pass
+    class Bannana(Fruit) : pass
+    class Apple(Fruit) : pass
+
+    assert(Bannana() == Bannana())
+    assert(Bannana() != Apple())
+    assert(  Apple() <  Bannana())