From Zynamics BinCrowd presentation

Bincrowd, a project I had the pleasure of contributing to, has been released by Zynamics. Bincrowd simplifies the collaborative option in reverse engineering and brings it en masse. Any function a user has ever submitted documentation for can be found quickly in your target. It also introduces greater flexibility and reliability by adding additional signature methods for functions. Further, server access and clients for various disassemblers are free (ida client here).

To fully understand how this could create an evolutionary step for software reverse engineering (RE) lets walk look at a typical RE session. You open your target in your favorite disassembler and find it
has several hundred functions. If you are lucky you will have a signature database that matches the OS version, library versions and compiler version your target was built with.  This will let you find the system calls quickly, fiddling down the undocumented code you need to RE to a hundred or so functions. Once you have reverse engineered a function that is missing a signature you could create one for it but until now it has been difficult to share this with others. With Bincrowd you can share this information quickly for team based RE efforts or just for posterity sake.  If 10 years from now someone runs into similar code somewhere else this information becomes useful. Further, Bincrowd goes further by adding more flexible signature routines to the mix. Signatures often used today are based on the raw bytes of a compiled function. The added signatures in Bincrowd are based on the flow of the function instead. The flexibility this introduces means you might find details and signatures for functions that were compiled with a different version compiler, different version of libraries, different compiler and libraries entirely, different hardware architectures or even different OS’s. Research from Zynamics has been the ground breaking in this field.

My efforts on Bincrowd were supported by Recurity-Labs. Both Recurity-Labs and Zynamics are great companies to work with.

JTAGenum is an open source Arduino based hardware platform I built last year with three primary goals: [1. Given a large set of pins on a device determine which are JTAG lines 2. Enumerate the Instruction Register to find undocumented functionality 3. be easy to build and apply] The development of a device has various distinct stages handled by different people/companies that each assume the other has properly secured their part. The security of devices often rely on obfuscation which makes it dificult for any part of the chain to evaluate the security of the whole. This is a problem that JTAGenum helps address. This was built for personal research and while working on various projects at Recurity Labs. Please feel free to contact me with any questions, problems, targets or updates. I would be more than happy to share credit.

Related work: There are two other tools for finding JTAG pins: JTAGScan presented by Benedikt Heinz (hunz) at ph-neutral which inspired Arduinull by Sébastien Bourdeauducq (lekernel). JTAGenum is most similar to the latter with the added feature of finding undocumented functionality. Felix Domke (tmbinc) recently gave a lecture on enumarating undocumented JTAG instructions and anyone considering using JTAGenum would do well to check his paper(cache)/lecture from the 26c3.

About JTAG

JTAG is a common hardware debugging interface. It is used throughout the development chain of a device. Layout designers and board manufactures that employ pick-and-place machines will use JTAG to test interconnectivity of components. ASIC designers use it to test the internal state of the chips they build. Software developers often use it to load firmware onto the device and to debug software.

For a varity of reasons JTAG is often left in the final product. As such each stage of the development chain will attempt to obfuscate its existence or functionality. ASIC manufactures often build in added functionality (such as logic analysis tools) and avoid mentioning both extended and often basic functionality from their final documentation. Layout designers might remove JTAG pins from the board, spread their contacts throughout on the board, remove contacts and hide JTAG lines on inner layers of the board. As mentioned before, this can make it difficult for any one part of the development chain to evaluate the security of the device as a whole. If you are unfamiliar with the inner workings of JTAG skip to the A bit more about JTAG section for the basics.

Hardware

To use JTAGenum you need an arduino compatible microcontroller. Arduino is a simple development enviornment (IDE) for various microcontrollers. At the moment AVR and PIC variants are available and can be purchased anywhere from $10 to $50. JTAGenum has been tested on the official Arduino Duemilanove, Arduino Mega (see hackerspace.be modifications), RBBB clone and Teensy++. When picking your microcontroller platform consider two issues: 1. How many pins do you want to check on your target. 2. what voltage level does your target device require.  Concerning voltage most Arduinos work at 5 volts. Some are switchable but even those that are not can be modified. For example revision 1.0 of the Teensy++ with over 30 pins of i/o can be modified by hand to operate at 3.3 volts. I show where to cut lines and install a voltage regulator over here (Update: Teensy++ version 2.0 has a spot on the pcb to install a 3.3v regulator). For voltages other than 3.3v and 5v there are a variety of solutions that depend on if you need uni-directional or bi-directional support on your i/o lines.

When connecting the microcontroller to the pins of your target one thing to be aware of is possible cross-talk between wires. I’ve been using a patch cable from Amontec that has a lot of cross talk.  JTAGenum has a mode that helps check for this which I will get into more detail later.

Usage

Download the JTAGenum code and open it in the Arduino IDE. The following needs to be changed in the code depending on your microcontroller:

• pins[] define which pins on the microcontroller are being used to connect to the target
• pinname[] is a convenient way to map the pins to names which correspond to the names of pins on your target
• IR_LEN defines the length of the JTAG instruction register. If you change this you should also add ’0′s to each of the coresponding IR_** instruction definitions. You can find the IR_LEN in the documentation for your target. If you cannot find it just guess. (10 is the current value, 8 is also common)

Upload the sketch to your microcontroller and open the serial console with a baud of 115200.  Sending a ‘h’ to the console will print usage information that describes each function. Each function is enacted by sending the defined one character code:

v > verbose

Toggles verbose output. At times verbose might present too much information or without it too little.

l > loopback check

Find loopback pairs that will generate false-positives for other tests. After running you should remove any loopback pairs from your pins[]/pinnames[]. Looback pairs are found by sending a predetermined pattern[] to all possible pins while checking all pins for matching output.  Because the JTAG clock (TCK) and state (TMS) pins are NOT being stimulated the input/output pairs where the pattern is found represent loopbacks. NOTE: you should probably run this once with and without internal pull-up resistors set (‘r’) to avoid problems of cross-talk which is discussed in detail later.

s > scan

This routine is used to check all possible pins and find JTAG  clock, state, input and output pins lines (TCK,TMS,TDI,TDO). This is done by setting the JTAG state (TMS) into Shift_IR mode and then sending pattern[] to TDI and checking for it on TDO while clocking TCK. This check is run for every possible pin combination and it is important that you remove loopback pins before running. While this scan is meant to determine all of the JTAG pins required it is possible that the  TMS pin found is incorrect.  This depends on if the target uses the bypass register by default (described later). If an IDCODE register is present then bypass mode is not the default and you can assume that the pin this scan defines as TMS is correct.  Otherwise, only the TCK, TDI and TDO pins can be determined.  NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

y > brute force IR search

This will set the instruction register (IR) to all possible values and check the output. This can be used to find undocumented instructions and examine their results via the data register (DR). To run this scan you should have already determined the 4 JTAG pins and define pins[] as such: [0]=TCK [1]=TMS [2]=TDO [3]=TDI.  NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

x > boundary scan

This will return the state of all the pins on the target.  Actually it is not just the pins but the contents of the scan/sample register. This should be a rather large register and is defined in the code by SCAN_LEN+100. You can check your targets documentation and specify this or just leave it as a large number (currently 1800). To run this scan you should have already determined the 4 JTAG pins and define pins[] as such: [0]=TCK [1]=TMS [2]=TDO [3]=TDI.  NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

i > idcode scan

The JTAG standards specify that if an idcode register is present it should be set as the default data register (DR) and attached to output (TDO) by default. Meaning, regardless of the state of the JTAG chip (set with TMS line) and regardless of input being sent to the chip (TDI) by clocking the chip (TCK) it should return the contents of the idcode to the output (TDO). Hence, this routine iterates through all possible TCK,TDO pairs of pins, CLK’ing each bit along the way, and prints the output when there is any change (we assume an idcode will not be all 0′s or 1′s). You should examine the documentation of your target(s) to see if the idcode matches. NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

b > shift_bypass

Broken atm (need to add TCK enumeration). The JTAG standards specify that if and idcode register is NOT present on the chip then the bypass register (length of 1) should be the default DR. Essentially this means what is sent to the input (TDI) should come out on the output (TDI) with a one clock delay (TCK). It is important that you remove loopbacks before running this test otherwise the loopback pins will look like valid JTAG lines. NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

r > set pull-up resistors & cross-talk

If like me the cables you use to connect between JTAGenum to your targets are flimsy or uninsulated you might run into issues of cross-talk whereby when one pin is transmitting a nearby pin picks up the transmission even though they are not connected. To avoid this you can turn on the internal pull-up resistors which will force the pin to a default state. If for some reason you continue to have sporadic issues run the following in sequence to check if the problem is the cable, target or other:

1. Disconnect the cables between your target and JTAGenum. Disconnected them entirely from JTAGenum as well.
2. Run a loopback check (‘l’) with pull-ups off. In this state the pins are in open mode and might fluctuate.  You’ll notice that as you move the microcontroller around, turn lights on and off or move other devices close to or away from it that the results change.
3. Turn on pull-ups (‘r’) and run the test again. The results should now be consistent. If they aren’t, then let me know.
4. Now attach your cables to JTAGenum but not the target.  Run steps 2 and 3 again. Step 2 will give you a feel for how much inconsistency the cable may add. If the loopback check results in actual pattern matches then your cable has cross-talk. Step 3 should still result in a consistent state of either all high (1′s) or all low (0′s) and if it doesn’t then your cross-talk issues are such that all JTAGenum tests are going to be buggy at best. Feel free to give me an emailand I will happily try to help solve the problem.

Code references

Examining code of various jtag scanning derivitives is probably the best way to modify JTAGenum however you want or when wanting to cross reference JTAG logic to debug a problem.

• JTAG Finder – the microcontroller provides a pin protocol/interface to a C client running on the PC which does the actual scanning logic. This was one of the first (if not the) JTAG scanner.  It trades simplicty of code and protocol for efficiency and speed.
• JRev – Not meant for scanning pins to find JTAG. It does however implement some logic not in any of the JTAG scanners mentioned here.  In paticular, support for chain mapping, IR length detection and perhaps others I’m missing.
• Arduinull – The first simplified scanner running all in the microcontroller.
• JTAG pinout detector – a port of Arduinull and JTAGenum for the Arduino Mega board.

Additionally Jal2 and Zoobab have written their branches of JTAGenum.  I will try to merge their features to the JTAGenum branch as quickly as possible but if you run into problems or for general curiosity you can check their code.

A bit more about JTAG

Basic understanding of how JTAG works will be helpful when using JTAGenum. There are 4 lines/pins: TDO=output, TDI=input, TCK=clock, TMS=state machine control.  Say you want to read the ID of the chip. First you would send the IDCODE instruction to the instruction register (IR). The JTAG controller then places the actual id code value of the chip in a data register which you could then read out. You would think that it would be enough to have one input line going to the IR and one output coming from the DR but JTAG also supports writing to the DR. As apposed to adding another input line specific to the DR instead JTAG works by moving the input and output lines between IR and DR. The TMS line is used to switch TDI/TDO to IR when you want to place an instruction and back to DR when you want to read or write data. With all operations, be it state change (TMS) reading (TDI) or writing (TDO), the clock line must be cycled once (TCK) for every bit or change. This was a brutal and drastic simplification but with that understood reading the Usage section should be comprehensible.

Update: added routine to print out hex data for blocks where entropy passes a given threshold.
Update: GUI display of graph (using TK) was not working on my system. Code now saves an image of the results in addition to attempting to display with GUI.

Ero Carrera responded yesterday to a request on OpenRCE concerning using entropy analysis to find RSA keys and other random blocks of data in binaries. Here in is a full wrapper for the code he gives. We use matplotlib instead of Mathematica to generate the graph. Also if you plan to scan files larger than 100k I’d highly recommend downloading the modified progressBar class included here.

example output:

Target data:

data = ''.join (
  [chr (random.randint (0, 64)) for x in xrange (1024)] +
  [chr (random.randint (0, 255)) for x in xrange (1024)] +
  [chr (random.randint (0, 64)) for x in xrange (1024)] )

[==================================100%=====================================]
     949 7.00: 1a060113050c2d0d 17302e091d2d0117 →♠☺‼♣♀- ↨0. ↔-☺↨

entropy_graph.py

""" Entropy scan
    H() and entropy_scan() originally by Ero Carrera (blog.dkbza.org)

    Modified May 2007 by cyphunk (deadhacker.com)
    Modified Dec 2009 by cyphunk

    USAGE:
    cmd [target_path]
    """

# FLAGS:
SHOWPROGRESS = 1       # Show console progress bar?
PRINTONTHRESHOLD = 6.8 # When block is > than threshold
                       # print first 16 bytes in both
                       # hex and ascii.  Set to 0 to turn
                       # off.
ONLYFIRSTBLOCK = 0     # Set to 1 it will only print the first
                       # block that goes over threshold and not
                       # blocks > threshold that are only offset
                       # by 1.  By setting to zero block windows
                       # that match will be printed.
BLOCKSIZE = 256        # size of blocks scanned.

import math
import random
from pylab import *
from matplotlib.ticker import MultipleLocator, FormatStrFormatter
import tkFileDialog
from Tkinter import *
from progressBar import *
from binascii import hexlify
import string
import os
import cPickle # cache results

def H(data):
  if not data:
    return 0
  entropy = 0
  for x in range(256):
    p_x = float(data.count(chr(x)))/len(data)
    if p_x > 0:
      entropy += - p_x*math.log(p_x, 2)
  return entropy

def entropy_scan (data, block_size) :
  if SHOWPROGRESS:
      progress = progressBar(0, len(data) - block_size, 77)
  # creates blocks of block_size for all possible offsets ('x'):
  blocks = (data[x : block_size + x] for x in range (len (data) - block_size))
  i = 0
  for block in (blocks) :
    i += 1
    if SHOWPROGRESS:
        progress(i)
    yield H (block)

# performance improvement if you have psyco
try:
  import psyco
  psyco.full()
  print "got psyco"
except ImportError:
  pass

# get target file as argument var or from dialog:
filename = ""
if sys.argv[1:]:
    filename = sys.argv[1]
else:
    root = Tk()
    root.withdraw()
    filename = tkFileDialog.askopenfilename(title="Target binary",
                                        filetypes=[("All files", "*")])

# run, print graph:

if filename:
    # Open file and scan for entropy:
    if os.path.splitext(filename)[1] == ".entropy":
        print "File is a cached '.entropy' from previous scan"
        results = cPickle.load(open(filename, 'rb'))
        filename = os.path.splitext(filename)[0]
        print filenamea
        raw = open(filename, 'rb').read()
    else:
        raw = open(filename, 'rb').read()
        # debug with test data:
        """
        import random
        raw = ''.join (
        [chr (random.randint (0, 64)) for x in xrange (1024)] +
        [chr (random.randint (0, 255)) for x in xrange (1024)] +
        [chr (random.randint (0, 64)) for x in xrange (1024)] )
        """
        results = list( entropy_scan(raw,BLOCKSIZE) )
        print "saving cache of entropy scan data to %s" % filename+".entropy"
        cPickle.dump(results, open(filename+".entropy", 'wb')) 

    # Print blocks that are above a defined threshold of entropy:
    if PRINTONTHRESHOLD > 0:
        print
        found = 0
        for i in range(len(results)):
            if results[i] > PRINTONTHRESHOLD:
                if found == 0:
                    table = string.maketrans("rnt", '   ') # don't like newlines
                    #blockstr = string.translate(str(raw[i : i+16]), table) # translate to string value
                    print "0x%8x %.2f: %s %s" % (i, results[i], hexlify(raw[i : i+8]),
                                                     hexlify(raw[i+8 : i+16]))
                    #%.3f - %016X / %s" % (i, results[i], raw[i : i + 16], raw[i : i + 16])
                    found = ONLYFIRSTBLOCK
            else:
                found = 0

    # Plot
    filesize = os.path.getsize(filename)
    imgdpi = 100
    imgwidth = filesize / imgdpi

    if imgwidth > 327:
      imgwidth = 327

    majorLocator   = MultipleLocator(0x400)   # mark every 1024 bytes
    majorFormatter = FormatStrFormatter('%X') # change to %d to see decimal offsets

    ax = subplot(111)
    plot(results, linewidth=2.0, antialiased=False)
    subplots_adjust(left=0.02, right=0.99, bottom=0.2)

    ax.axis([0,filesize,0,8])
    ax.xaxis.set_major_locator(majorLocator)
    ax.xaxis.set_major_formatter(majorFormatter)
    xticks(rotation=315)

    xlabel('block (byte offset)')
    ylabel('entropy')
    title('Entropy levels')

    grid(True)

    img = gcf()
    img.set_size_inches(imgwidth, 6)
    img.savefig(filename+".png", dpi=imgdpi)

    draw()
    show()

progressBar.py (originally from active state but modified for our use)

import sys

class progressBar:
    """ Creates a text-based progress bar. Call the object with the `print'
        command to see the progress bar, which looks something like this:

        [=======>        22%                  ]

        You may specify the progress bar's width, min and max values on init.
    """
    def __init__(self, minValue = 0, maxValue = 100, totalWidth=80):
        self.progBar = "[]"   # This holds the progress bar string
        self.min = minValue
        self.max = maxValue
        self.span = maxValue - minValue
        self.width = totalWidth
        self.amount = 0       # When amount == max, we are 100% done
        self.updateAmount(0)  # Build progress bar string
        self._old_pbar = ""   # used to track change
        self.pbar_str = ""

    def updateAmount(self, newAmount = 0):
        """ Update the progress bar with the new amount (with min and max
            values set at initialization; if it is over or under, it takes the
            min or max value as a default. """
        if newAmount > self.max: newAmount = self.max
        self.amount = newAmount

        # Figure out the new percent done, round to an integer
        diffFromMin = float(self.amount - self.min)
        percentDone = (diffFromMin / float(self.span)) * 100.0
        percentDone = int(round(percentDone))

        # Figure out how many hash bars the percentage should be
        allFull = self.width - 2
        numHashes = (percentDone / 100.0) * allFull
        numHashes = int(round(numHashes))

        # Build a progress bar with an arrow of equal signs; special cases for
        # empty and full
        if numHashes == 0:
            self.progBar = "[>%s]" % (' '*(allFull-1))
        elif numHashes == allFull:
            self.progBar = "[%s]\n" % ('='*allFull)
        else:
            self.progBar = "[%s>%s]" % ('='*(numHashes-1),
                                        ' '*(allFull-numHashes))

        # figure out where to put the percentage, roughly centered
        percentPlace = (len(self.progBar) / 2) - len(str(percentDone))
        percentString = str(percentDone) + "%"

        # slice the percentage into the bar
        self.progBar = ''.join([self.progBar[0:percentPlace], percentString,
                                self.progBar[percentPlace+len(percentString):]
                                ])

    def __str__(self):
        return str(self.progBar)

    def __call__(self, value):
        """ Updates the amount, and writes to stdout. Prints a carriage return
            first, so it will overwrite the current line in stdout."""

        self.updateAmount(value)
        self.pbar_str = str(self)
        if self.pbar_str != self._old_pbar:
            self._old_pbar = self.pbar_str
            sys.stdout.write(self.pbar_str + "\r")
            sys.stdout.flush()