From Zynamics BinCrowd presentation

Bincrowd, a project I had the pleasure of contributing to, has been released by Zynamics. Bincrowd simplifies the collaborative option in reverse engineering and brings it en masse. Any function a user has ever submitted documentation for can be found quickly in your target. It also introduces greater flexibility and reliability by adding additional signature methods for functions. Further, server access and clients for various disassemblers are free (ida client here).

To fully understand how this could create an evolutionary step for software reverse engineering (RE) lets walk look at a typical RE session. You open your target in your favorite disassembler and find it
has several hundred functions. If you are lucky you will have a signature database that matches the OS version, library versions and compiler version your target was built with.  This will let you find the system calls quickly, fiddling down the undocumented code you need to RE to a hundred or so functions. Once you have reverse engineered a function that is missing a signature you could create one for it but until now it has been difficult to share this with others. With Bincrowd you can share this information quickly for team based RE efforts or just for posterity sake.  If 10 years from now someone runs into similar code somewhere else this information becomes useful. Further, Bincrowd goes further by adding more flexible signature routines to the mix. Signatures often used today are based on the raw bytes of a compiled function. The added signatures in Bincrowd are based on the flow of the function instead. The flexibility this introduces means you might find details and signatures for functions that were compiled with a different version compiler, different version of libraries, different compiler and libraries entirely, different hardware architectures or even different OS’s. Research from Zynamics has been the ground breaking in this field.

My efforts on Bincrowd were supported by Recurity-Labs. Both Recurity-Labs and Zynamics are great companies to work with.

JTAGenum is an open source Arduino based hardware platform I built last year with three primary goals: [1. Given a large set of pins on a device determine which are JTAG lines 2. Enumerate the Instruction Register to find undocumented functionality 3. be easy to build and apply] The development of a device has various distinct stages handled by different people/companies that each assume the other has properly secured their part. The security of devices often rely on obfuscation which makes it dificult for any part of the chain to evaluate the security of the whole. This is a problem that JTAGenum helps address. This was built for personal research and while working on various projects at Recurity Labs. Please feel free to contact me with any questions, problems, targets or updates. I would be more than happy to share credit.

Related work: There are two other tools for finding JTAG pins: JTAGScan presented by Benedikt Heinz (hunz) at ph-neutral which inspired Arduinull by Sébastien Bourdeauducq (lekernel). JTAGenum is most similar to the latter with the added feature of finding undocumented functionality. Felix Domke (tmbinc) recently gave a lecture on enumarating undocumented JTAG instructions and anyone considering using JTAGenum would do well to check his paper(cache)/lecture from the 26c3.

About JTAG

JTAG is a common hardware debugging interface. It is used throughout the development chain of a device. Layout designers and board manufactures that employ pick-and-place machines will use JTAG to test interconnectivity of components. ASIC designers use it to test the internal state of the chips they build. Software developers often use it to load firmware onto the device and to debug software.

For a varity of reasons JTAG is often left in the final product. As such each stage of the development chain will attempt to obfuscate its existence or functionality. ASIC manufactures often build in added functionality (such as logic analysis tools) and avoid mentioning both extended and often basic functionality from their final documentation. Layout designers might remove JTAG pins from the board, spread their contacts throughout on the board, remove contacts and hide JTAG lines on inner layers of the board. As mentioned before, this can make it difficult for any one part of the development chain to evaluate the security of the device as a whole. If you are unfamiliar with the inner workings of JTAG skip to the A bit more about JTAG section for the basics.

Hardware

To use JTAGenum you need an arduino compatible microcontroller. Arduino is a simple development enviornment (IDE) for various microcontrollers. At the moment AVR and PIC variants are available and can be purchased anywhere from $10 to $50. JTAGenum has been tested on the official Arduino Duemilanove, Arduino Mega (see hackerspace.be modifications), RBBB clone and Teensy++. When picking your microcontroller platform consider two issues: 1. How many pins do you want to check on your target. 2. what voltage level does your target device require.  Concerning voltage most Arduinos work at 5 volts. Some are switchable but even those that are not can be modified. For example revision 1.0 of the Teensy++ with over 30 pins of i/o can be modified by hand to operate at 3.3 volts. I show where to cut lines and install a voltage regulator over here (Update: Teensy++ version 2.0 has a spot on the pcb to install a 3.3v regulator). For voltages other than 3.3v and 5v there are a variety of solutions that depend on if you need uni-directional or bi-directional support on your i/o lines.

When connecting the microcontroller to the pins of your target one thing to be aware of is possible cross-talk between wires. I’ve been using a patch cable from Amontec that has a lot of cross talk.  JTAGenum has a mode that helps check for this which I will get into more detail later.

Usage

Download the JTAGenum code and open it in the Arduino IDE. The following needs to be changed in the code depending on your microcontroller:

• pins[] define which pins on the microcontroller are being used to connect to the target
• pinname[] is a convenient way to map the pins to names which correspond to the names of pins on your target
• IR_LEN defines the length of the JTAG instruction register. If you change this you should also add ’0′s to each of the coresponding IR_** instruction definitions. You can find the IR_LEN in the documentation for your target. If you cannot find it just guess. (10 is the current value, 8 is also common)

Upload the sketch to your microcontroller and open the serial console with a baud of 115200.  Sending a ‘h’ to the console will print usage information that describes each function. Each function is enacted by sending the defined one character code:

v > verbose

Toggles verbose output. At times verbose might present too much information or without it too little.

l > loopback check

Find loopback pairs that will generate false-positives for other tests. After running you should remove any loopback pairs from your pins[]/pinnames[]. Looback pairs are found by sending a predetermined pattern[] to all possible pins while checking all pins for matching output.  Because the JTAG clock (TCK) and state (TMS) pins are NOT being stimulated the input/output pairs where the pattern is found represent loopbacks. NOTE: you should probably run this once with and without internal pull-up resistors set (‘r’) to avoid problems of cross-talk which is discussed in detail later.

s > scan

This routine is used to check all possible pins and find JTAG  clock, state, input and output pins lines (TCK,TMS,TDI,TDO). This is done by setting the JTAG state (TMS) into Shift_IR mode and then sending pattern[] to TDI and checking for it on TDO while clocking TCK. This check is run for every possible pin combination and it is important that you remove loopback pins before running. While this scan is meant to determine all of the JTAG pins required it is possible that the  TMS pin found is incorrect.  This depends on if the target uses the bypass register by default (described later). If an IDCODE register is present then bypass mode is not the default and you can assume that the pin this scan defines as TMS is correct.  Otherwise, only the TCK, TDI and TDO pins can be determined.  NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

y > brute force IR search

This will set the instruction register (IR) to all possible values and check the output. This can be used to find undocumented instructions and examine their results via the data register (DR). To run this scan you should have already determined the 4 JTAG pins and define pins[] as such: [0]=TCK [1]=TMS [2]=TDO [3]=TDI.  NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

x > boundary scan

This will return the state of all the pins on the target.  Actually it is not just the pins but the contents of the scan/sample register. This should be a rather large register and is defined in the code by SCAN_LEN+100. You can check your targets documentation and specify this or just leave it as a large number (currently 1800). To run this scan you should have already determined the 4 JTAG pins and define pins[] as such: [0]=TCK [1]=TMS [2]=TDO [3]=TDI.  NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

i > idcode scan

The JTAG standards specify that if an idcode register is present it should be set as the default data register (DR) and attached to output (TDO) by default. Meaning, regardless of the state of the JTAG chip (set with TMS line) and regardless of input being sent to the chip (TDI) by clocking the chip (TCK) it should return the contents of the idcode to the output (TDO). Hence, this routine iterates through all possible TCK,TDO pairs of pins, CLK’ing each bit along the way, and prints the output when there is any change (we assume an idcode will not be all 0′s or 1′s). You should examine the documentation of your target(s) to see if the idcode matches. NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

b > shift_bypass

Broken atm (need to add TCK enumeration). The JTAG standards specify that if and idcode register is NOT present on the chip then the bypass register (length of 1) should be the default DR. Essentially this means what is sent to the input (TDI) should come out on the output (TDI) with a one clock delay (TCK). It is important that you remove loopbacks before running this test otherwise the loopback pins will look like valid JTAG lines. NOTE: run with pull-ups on (‘r’) as any cross-talk might result in false-positives.

r > set pull-up resistors & cross-talk

If like me the cables you use to connect between JTAGenum to your targets are flimsy or uninsulated you might run into issues of cross-talk whereby when one pin is transmitting a nearby pin picks up the transmission even though they are not connected. To avoid this you can turn on the internal pull-up resistors which will force the pin to a default state. If for some reason you continue to have sporadic issues run the following in sequence to check if the problem is the cable, target or other:

1. Disconnect the cables between your target and JTAGenum. Disconnected them entirely from JTAGenum as well.
2. Run a loopback check (‘l’) with pull-ups off. In this state the pins are in open mode and might fluctuate.  You’ll notice that as you move the microcontroller around, turn lights on and off or move other devices close to or away from it that the results change.
3. Turn on pull-ups (‘r’) and run the test again. The results should now be consistent. If they aren’t, then let me know.
4. Now attach your cables to JTAGenum but not the target.  Run steps 2 and 3 again. Step 2 will give you a feel for how much inconsistency the cable may add. If the loopback check results in actual pattern matches then your cable has cross-talk. Step 3 should still result in a consistent state of either all high (1′s) or all low (0′s) and if it doesn’t then your cross-talk issues are such that all JTAGenum tests are going to be buggy at best. Feel free to give me an emailand I will happily try to help solve the problem.

Code references

Examining code of various jtag scanning derivitives is probably the best way to modify JTAGenum however you want or when wanting to cross reference JTAG logic to debug a problem.

• JTAG Finder – the microcontroller provides a pin protocol/interface to a C client running on the PC which does the actual scanning logic. This was one of the first (if not the) JTAG scanner.  It trades simplicty of code and protocol for efficiency and speed.
• JRev – Not meant for scanning pins to find JTAG. It does however implement some logic not in any of the JTAG scanners mentioned here.  In paticular, support for chain mapping, IR length detection and perhaps others I’m missing.
• Arduinull – The first simplified scanner running all in the microcontroller.
• JTAG pinout detector – a port of Arduinull and JTAGenum for the Arduino Mega board.

Additionally Jal2 and Zoobab have written their branches of JTAGenum.  I will try to merge their features to the JTAGenum branch as quickly as possible but if you run into problems or for general curiosity you can check their code.

A bit more about JTAG

Basic understanding of how JTAG works will be helpful when using JTAGenum. There are 4 lines/pins: TDO=output, TDI=input, TCK=clock, TMS=state machine control.  Say you want to read the ID of the chip. First you would send the IDCODE instruction to the instruction register (IR). The JTAG controller then places the actual id code value of the chip in a data register which you could then read out. You would think that it would be enough to have one input line going to the IR and one output coming from the DR but JTAG also supports writing to the DR. As apposed to adding another input line specific to the DR instead JTAG works by moving the input and output lines between IR and DR. The TMS line is used to switch TDI/TDO to IR when you want to place an instruction and back to DR when you want to read or write data. With all operations, be it state change (TMS) reading (TDI) or writing (TDO), the clock line must be cycled once (TCK) for every bit or change. This was a brutal and drastic simplification but with that understood reading the Usage section should be comprehensible.

First, the directory structure:

• Attacks (segmented by attack class with logs on attempts and all information needed to replicate)
• Logs (all non-attack specific notes and logs)
• Photos
• References (datasheets, application notes, documentation)
• Reports (information used to document or report findings)
• Tools (software, schematics, non-attack specific custom built tools)

Workflow:

1. Objectives
   This isn’t represented as a directory but instead typically the first note sheet I start to write and store in the Logs directory. An objective comes from either a client or research goals. With clients this might come in the form of their general concern or attack vectors. For research this is often in the form of low hanging fruit (objective milestones).
2. References
   Build an overview of the device by determining the function and relation between components. Meaning, find all the datasheets or application notes you can, store them here and read them. I typically have a separate note sheet with very short summarization of this information which I store in the Logs directory.
3. Targets
   Again, not a directory. When I find a pin cluster I will test all the electrical characteristics (resistance to GND, voltage levels at different states, etc). I make note of these either on note sheets or visual notations and I store both in the Logs directory. electrical_10pinheader.txt, electrical_4pinpad.txt, electrical_3statebusctrl.txt, network_TCPIPstates.txt, firmware_interestingSymbols.txt, etc. This becomes extremely useful information not just for the current project but for cross referencing in future projects.
4. Attacks
   Make a sub directory for every class of attack (jtag, serial, i2c, spi, firmware, network, etc). If and when we have to build specific tools or record results, all of it will be here. In this case, for notes I store them with each attack, not in the Logs directory.
5. Reports
   When an attack is note worthy, be it a success or not, I will copy the relevent photos, notations, to this directory so that writing reports or documenting the work at the end is easier.

Additional repositories that are useful:

• Photos
  Take photos of everything. This is essential for documentation.
• Tools
  If I have to download software libraries, tools or schematics for building software or hardware tools I will store the originals here. If I write wind up writing custom code for e a specific attack ill store it in its attack directory (with dependencies here). Otherwise if it is a custom made tool used across many attacks I will store it here as well. Some of the tools I find useful eventually find their way into the sectk github.
• Logs
  All Notes or logs that do not relate to a specific attack go here. Such as general network captures, logic analysis logs, electrical testing notations, etc. At times I will go off on a general hunch that has no clear attack, target or objective and I will store the logs in a subdirectory. Also, any time I am working with software or a terminal (console) I keep a screen log (man screen) of the work and store and label these here as “screen_weekN_dayN.log”. I have one or two logs for every day while I am working on the project which gives me a very low level point to return to if I need to find something later that I might have forgotten to document. To retain absolute continuity of these logs I always append to the log for each day and when the log needs to be included in a directory for a specific attack I will copy it to that directory, retaining the original here.

Idealy all information should be digestable by one person. Verbose notes and documentation are essential for tracking down methods and pecularities later but ultimately you want to be able to quickly document the essentials either to pass on others, share with the community or include in a report to your client. I haven’t found a good medium between detail and summary in an active way so what I tend to do with my notes is repeat prior knowledge required to replicate findings, even if the information was already noted earlier in the same note sheet. The result is that when going back over notes one would start at the very end of the note sheet and work their way back up. Often times I will make note of this at the top of the sheet when I feel I have reached a closure point for the thread.

These are simple basic suggestions that are absolutely unexciting but often helpful. My experiance has been mostly with embedded analysis but this field can quickly forge into software level reverse engineering so this structure can be applied to some degree there as well. I would be elated if anyone has constructive suggestions or would like to contribute link or comment on their own workflow.

photo by Clint McManaman

Update Mar.2010: Deti has found a way modify the image enough to start a “service” menu. See his thread, he explains this and much more (german thread)

This is a semi-professional HD video camera that sells for $4000-5000. Regardless if you have this camera or not the methods discussed herein should encourage you to pull apart any device you have.  Even with such a large pricetag the hardware and software of the device are rather comprehendable, in some cases laughably so. In such cases I’m not laughing at Panasonic but rather at myself for thinking that this would be more complicated.  That is the objective of this tutorial, to help the reader realize that, if nothing else, the initial stage of diving in is easy and fun.

The camera is popular with indipendent producers but has its areas that could be improved. The camera runs a Montavista Linux operating system ontop a SuperH SH4 subtype SH7751 CPU.  Storing of HD video is done using “Panasonic P2″ cards. These appear to be PCMCIA cards with flash chips and a RAID controller which greatly increase data writing speeds.  There are many improvements that a homebrew system could bring to this camera. As an example of some low hanging fruit: Adding support for offloading video from the P2 card in the camera to an external disk using either a usb or firewire controller. While there are many other modifications that come to mind this would let someone use a single P2 card to record endless ammounts of video instead of paying for a mountain of P2 cards at $500 and $600 a pop. Another would be dissection of the P2 card and replacement of the flash chips of a cheap 4GB card to increase the size to 64GB?

INDEX

1. Obtaining Firmware
2. Firmware Dissection
3. Hardware Overview
4. Target List Assessment
5. Obtaining Debug Console
6. Developing Software For Camera
7. Modifying Firmware
8. …

• Modification wishlist

Future updates will be placed in this document. The index above provides a map for where I think research is heading.

1. Obtaining Firmware

There are a few routes to obtain the firmware, in order of difficulty: 1. obtain a manufacturers firmware update and hope that it contains a system image, 2. use the debug mechanism and ports (Serial, Network and TFTP, JTAG?) to download the running system image from internal FLASH, 3. desolder the FLASH or ROM memory chips found inside the target and dump the contents with a compatible memory programmer.  In the case of the Panasonic HVX200 the firmware updates contain an entire image of the system and can be obtained from the Panasonic website (ref2). Important: the version I am working with is  v4.06-00-0.00 (aka version “vsi4598g”) which can be downloaded directly from panasonic or found in the project working directory.

1.1. Decompressing Firmware Updates

Update: A shorter method for decompessing is… just untar and unzip the update file with “tar -zxvf <updatefile>”. I will retain the old direction because it’s useful knowledge for other projects/targets you might run into.

The firmware update is compressed. I use a program called STUNS which searches the file attempting to decompress any block of data using various compression routines. The developers website has since gone offline but I’ve placed a copy with the source code and compiled version for windows in the project working directory. Execute the program on the compressed firmware. Upon completion you will find directories created for each type of compression and inside are thousands of files for each block it attempted to decompress.  The largest block is the winner.  In the case of the HVX it the firmware is compressed with the deflate compression routine. I’ve placed a copy of the uncompressed version in the project working directory.  Even in this state you can already open the file in a Hex editor or run it through `strings` to find a wealth of information.

2. Firmware Dissection

2.1. Basic Examination of Strings

Looking at the strings of the decompressed firmware it should stand out that this is a Linux system and that the image contains several filesystems. Before pulling the file systems out lets just look at the strings to see what we can determine.

SH IPL+g version 0.9, Copyright (C) 2000 Free Software Foundation, Inc.

This is the bootloader. “It is an Initial Program Loader(IPL) of many SH-based systems booting Linux kernel. It functions as gdb-stub to provide the way to communicate with GNU gdb debugger. Also, it acts as a BIOS of the system.” (ref1, ref2)

# CONFIG_CPU_SUBTYPE_SH7300 is not set
# CONFIG_CPU_SUBTYPE_SH7707 is not set
# CONFIG_CPU_SUBTYPE_SH7708 is not set
# CONFIG_CPU_SUBTYPE_SH7709 is not set
# CONFIG_CPU_SUBTYPE_SH7750 is not set
CONFIG_CPU_SUBTYPE_SH7751=y
# CONFIG_CPU_SUBTYPE_ST40STB1 is not set
# CONFIG_CPU_SH3 is not set
CONFIG_CPU_SH4=y
CONFIG_CPU_LITTLE_ENDIAN=y
# CONFIG_P2PF_K200 is not set
CONFIG_P2PF_K230=y
# CONFIG_P2PF_K250 is not set
CONFIG_P2PF_HW_VERSION=2
CONFIG_MEMORY_START=08000000
CONFIG_MEMORY_SIZE=04000000
CONFIG_MEMORY_SET=y

This is part of  the configuration file for compiling a Linux kernel. It shows us that the CPU is a SuperH SH4 subtype SH7751. This family of CPUs are also used with Sega gamesystems and some PayTV Set Top Boxes.

Linux version 2.4.20_mvl31-ms7751rse01-sh_sh4_le (root@rx7)
(gcc version 3.3.1 (MontaVista 3.3.1-7.0.42.0600552 2006-04-30))
#13 Thu Mar 22 19:28:14 JST 2007

We are running a 2.4.20 kernel compiled with gcc version 3.3.1.  We also see the version of MontaVista linux being used.  These details are important because if we ever want to compile our own software for this system we will need to setup, at a minimum, an enviornment that includes the Linux 2.4.20 headers, the version of glibc being used in this MontaVista version and a gcc 3.3.1 crosscompiler for SH4 platforms (or crosstoll.)

description=NetChip 2280 USB Peripheral Controller

This is part of the net2280.o kernel module and indicates the USB driver.  This already gives some promise to obtaining our objective of supporting an external USB disk. Though we would still need to find USB host drivers somewhere.  It is so important to determine which drivers are already available because if we have to add and compile our own drivers we will likely need more than just the Linux headers, glibc and compatible gcc version.  We would perhaps need the entire MontaVista development kit so that we could both compile the driver and add support to the kernel for the driver, meaning perhaps we even have to recompile the entire kernel.  Yuk.

7791c147ef8f33e7c9844d20992e4fd8  netboot-1st.binary
9004619ae34aad3bbbc7cca53e3b7444  netboot1.binary
21910d91dcbae05a0688acce08ef86df  netboot2.binary
e0be87ce4e9c70deddb828a17f931488  vmlinux.bin
cedfadc76e06cca69602424d3a277a39  rootfs.image
b21555e3f845831d455d9f679277ac57  home.image
e9a74405904af1544bdfe5cb1363858b  vmlinux-vup.bin
f5fcfa007950817cca499da3ae13bf53  ramdisk.gz
c003003dc8b8abbb3e3c11367530d454  vup.sh
51ffed17a3b940c6ad5b7ca727a8d259  vup2.sh
df7e7b23160729e90add6b437db2b278  tx.bin
f1d3ff8443297732862df21dc4e57262  txvups.bin
a54f0041a9e15b050f25c463f1db7449  txvupe.bin
d9a885fc28e4f61d3268fea6ccb7be09  txvup.sh

This appears at the very end of the uncompressed firmware image and indicates the various parts of the image.  We also find tons of scripts in in the file but it would be nice to map these to their filenames.

2.2. Mounting Firmware Filesystem

There are a few images/filesystems contained internally in the uncompressed firmware. I made a lucky guess that these were CramFS images. If this was not the case, to determine what type of filesystem for the internal image we would use a program that attempts to match “magic” strings (mime/type, magic/type databases) to search over the file looking for matches.  At the moment I do not remember the name for such a program but contact me if you need this for other projects.

Confirmation of a CramFS image came in the form of these strings found in the firmware image:

Now that we have the filesystem I’d like to both start compiling a list of targets (software to be reverse engineered) and to better understand the internal hardware and funcional components. This requires some familiarity with linux systems so that you can deduce what is common to basic linux systems and what is specifically added by Panasonic. Lets start walking through it.

2.3.1. first_cramfs_img_toEOF.mount (rootfs.image)

2.3.1.1 /dev/

The device handles that are new to me here are /dev/rt, /dev/vcs and /dev/vcsa. /dev/rt is still a mystery for me but the vcs and vcsa devices are Virtual Console Capture devices (according to a common linux device list). From the Linux Cookbook (ref2):

If the target console is the first virtual console (which you would see by typing [ALT]-[F1]), the device to cat is `/dev/vcs1′. To take a screen shot of the fourth virtual console, and save it to a file called `screenshot’, type:

$ cat /dev/vcs4 > screenshot [RET]

In order to re-display the screenshot just paste it back into a free virtual console, type:

$ cat screenshot > /dev/vcs4 [RET]

Hence, we might be able to use this for debugging, provided we find some way of grabbing the screencapture file. Then again, maybe just setting up a serial terminal is going to be easier.

2.3.1.2 /etc/init.d/

This directory commonly contains the startup scripts for a linux system that are used to start services that will run in the background while the system is up, load drivers and enable devices. When examining these files I will only include excerpts from the scripts.

/etc/init.d/pcmcia

# Panasonic P2 original pcmcia script 2004/07/30
. /lib/modules/`uname -r`/pcmcia-config/pcmcia
    action=$1
    case "$action" in
    start)
	/sbin/modprobe pcmcia_core
	/sbin/modprobe i82365 irq_mode=0
	/sbin/modprobe ds
	/sbin/modprobe cb_enabler
	/sbin/modprobe spd_mod > /dev/null 2>&1
	/sbin/cardmgr -q -c /lib/modules/`uname -r`/pcmcia-config
	touch /var/lock/subsys/pcmcia
	EXITCODE=0
	;;

All of the modules loaded with this script are common except for spd_mod. i82365 indicates that we are using a Intel i82365sl PCMCIA host controller (there are many clones, ref1). Let’s examine briefly the strings in ./first_cramfs_img_toEOF.mount/lib/modules/2.4.20_mvl31-ms7751rse01-sh_sh4_le/pcmcia/spd_mod.o.

./first_cramfs_img_toEOF.mount/lib/modules/2.4.20_mvl31-ms7751rse01-sh_sh4_le/pcmcia/spd_mod.o

[spd]2.4.1.61(K230)
<3>[spd]4:%s:%d:R5C812 on Unkown
<3>[spd]4:%s:%d:R5C812 on WRITE
<3>[spd]4:%s:%d:R5C812 on READ
spdu
<3>[spd]4:%s:%d:register_chrdev() failed(%d)
spd_udev.c
<3>[spd]4:%s:%d:invalid minor number(%d)
<3>[spd]4:%s:%d:unknown ioctl command(%d)
<3>[spd]4:%s:%d:copy_to_user() failed(%d)
<3>[spd]4:%s:%d:spd_identify_device() failed(%d)
<3>[spd]4:%s:%d:copy_form_user() failed(%d)
<3>[spd]4:%s:%d:spd_read_sector() failed(%d)
<3>[spd]4:%s:%d:udev_make_sg() failed(%d)
<3>[spd]4:%s:%d:spd_write_sector() failed(%d)
<3>[spd]4:%s:%d:spd_log_sense() failed(%d)
<3>[spd]4:%s:%d:spd_log_write() failed(%d)
<3>[spd]4:%s:%d:spd_block_erase() failed(%d)
<3>[spd]4:%s:%d:spd_sector_erase() failed(%d)
<1>[spd]2:%s:%d:ALERT:P2card firmware size 61h only but size=%02xh
<3>[spd]4:%s:%d:spd_firm_update() failed(%d)
<3>[spd]4:%s:%d:scatterlist overflow(%d)
<3>[spd]4:%s:%d:spd_read_capacity() failed(%d)

It is obvious from the various symbols found that this driver is used for writing to some form of memory card. We also find two important strings, “ALERT:P2card” which indicates that this driver is specific to the P2 card and “R5C812“. R5C812 indicates that we have a PCMCIA controller from Rocah, in particular the model which supports two PCMCIA/PCcard slots, such is the case on the HVX200. Examination of the product sheet confirms that this device is also used to control the SD card slot of the camera. We can put this driver into a list of potentially high value targets. With a file size of 98KB and the wonderful gift of a symbol table left in this might not be so dificult to analyze.

/etc/init.d/StartUp_Cam.sh

### execute application for MontaVista Linux Pro. 3.1
##   by Panasonic
#########################
# preload drivers       #
# set environment value #
#########################
. /etc/p2pfenv
ulimit -c 0
#####################
# application start #
#####################
cd /var
/home/apli/sg &

A look at the /etc/p2pfenv shows that we preload a driver for the SD card but also gives another possible target in the commenting out of a module that enables the PC mode of the camera (scullp.o). This is the mode the camera is put into when plugged up to a PC for offloading video. Herein we might find code for both reading the P2 card filesystem and setting up as a USB client.

/etc/p2pfenv

###################
# preload drivers #
###################
insmod /lib/modules/2.4.20_mvl31-ms7751rse01-sh_sh4_le/kernel/drivers/sdcard/sdcard.o \
> /dev/null 2>&1
#use only PC mode
#insmod /home/usb/scullp.o > /dev/null 2>&1

In scullp.o we have strings which indicate the developers login name and the project name in addition to plenty of symbols that confirm that this can access and control the P2 card (not shown here) so lets add scullp.o to our list of Panasonic specific targets:

../../../../src/contrib/K230_Kernel/include/linux
../../../../src/contrib/K230_Kernel/include/asm
/opt/montavista/pro/devkit/sh/sh4_le/lib/gcc-lib/sh4-hardhat-linux/3.3.1/include
../../../../src/contrib/K230_Kernel/include/linux/sunrpc
../../../../src/contrib/K230_Kernel/include/p2
../../../../src/contrib/K230_Kernel/include/asm-generic
/home/tanaka/work/K230/myapplication/p2pf/src/driver/usb/buffer

StarUp_Cam.sh also executes .//second_cramfs_img_toEOF.mount/apli/sg. When examining this file we get a load of other targets:

ZION VGA Initialize(NTSC) Success!!
ROM VERSION AREA INITIALIZE ERROR
/home/apli/pm
Create UM
/home/apli/sm
PID[SG_CHILD_PROC_SM] = %d
Create SM
/home/apli/mm
/home/apli/nano-X
nano-X
createSV::arg error eSgStatuse=%d
/home/apli/ext_prgrm.sh
ext_prgrm.sh
execl[EXT] error(%s)
/usr/local/bin/vup-start.sh
vup-start.sh
execl[VUP] error(%s)
/dev/fb0
libev.so
liblg.so
LGInit
libsc.so
libsg.so
liblb.so
libsp.so
libro.so
libm.so.6
GLIBCPP_3.2
GLIBCPP_3.2.3
CXXABI_1.2
GCC_3.0
GLIBC_2.2

…

… to be continued

There are several frameworks for such debugging available.  DTrace with RE:Trace (osx, sun), SystemTap on linux and vtrace for win32+linux, all scriptable.  My favorate as yet is Subterfugue though old its keep-it-simple-stupid methods have kept me coming back. Here is an example that changes the argument passed to a write() into rot13 ascii:

trans = string.maketrans('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ',
                         'nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM')

class Rot13(Trick):
    def callbefore(self, pid, call, args):
        m = getMemory(pid)
        address = args[1]
        size = args[2]
        data = m.peek(address, size)
        m.poke(address, string.translate(data, trans), self)

    def callmask(self):
        return { 'write' : 1 }

And the output:

bash-2.03$ sf --tri=Rot13 date
Jrq Sro  2 02:55:34 PFG 2000
bash-2.03$ sf --tri=Rot13 --tri=Rot13 date
Wed Feb  2 02:55:37 CST 200

So because Im too lazy to make a CVS commit, ill explain how you can revive it yourself. Hey! Really this is better. Its future proof: You wont have to worry about the software dieing if I go off to work at some draconian anti-opensource company just like all the other wonderful security engineers out there (Im looking at you Boomerang Decompiler). You wont have to worry because… within the next 5 minutes youll know how to maintain it yourself, kinda.

1. Downgrade python:
   download and install python 1.5.2. You could try your luck with later versions but the object c methods are different and subterfugue needs these for heavy use of ptrace() hooking. Lets race to see who recodes them first. Anyway, whatever version to try be sure you have the Makefile.pre.in from the python install sources.
2. Update system call map:
   grab the strace sources. The system call map that subterfugue is using is dated from 2001 or so and needs to be updated for newer kernels. compare the syscallmap.py in subterfugue to the syscallent.h of strace. From about array index 250+ is where the new entries start. To add them I just cut and paste to a new file, ran a replace routine for line in f.readlines(): print line.translate(string.maketrans(‘{}/*’,'()##’)). Also needed to be sure there there was no more than one flag in each array.
3. make install and then test with a trick from /usr/lib/subterfugue/tricks/: sf –tri=Count date

If time permits I would like to rewrite the ptrace c shell using python 2+ methods. Until then, this works.

Update: added routine to print out hex data for blocks where entropy passes a given threshold.
Update: GUI display of graph (using TK) was not working on my system. Code now saves an image of the results in addition to attempting to display with GUI.

Ero Carrera responded yesterday to a request on OpenRCE concerning using entropy analysis to find RSA keys and other random blocks of data in binaries. Here in is a full wrapper for the code he gives. We use matplotlib instead of Mathematica to generate the graph. Also if you plan to scan files larger than 100k I’d highly recommend downloading the modified progressBar class included here.

example output:

Target data:

data = ''.join (
  [chr (random.randint (0, 64)) for x in xrange (1024)] +
  [chr (random.randint (0, 255)) for x in xrange (1024)] +
  [chr (random.randint (0, 64)) for x in xrange (1024)] )

[==================================100%=====================================]
     949 7.00: 1a060113050c2d0d 17302e091d2d0117 →♠☺‼♣♀- ↨0. ↔-☺↨

entropy_graph.py

""" Entropy scan
    H() and entropy_scan() originally by Ero Carrera (blog.dkbza.org)

    Modified May 2007 by cyphunk (deadhacker.com)
    Modified Dec 2009 by cyphunk

    USAGE:
    cmd [target_path]
    """

# FLAGS:
SHOWPROGRESS = 1       # Show console progress bar?
PRINTONTHRESHOLD = 6.8 # When block is > than threshold
                       # print first 16 bytes in both
                       # hex and ascii.  Set to 0 to turn
                       # off.
ONLYFIRSTBLOCK = 0     # Set to 1 it will only print the first
                       # block that goes over threshold and not
                       # blocks > threshold that are only offset
                       # by 1.  By setting to zero block windows
                       # that match will be printed.
BLOCKSIZE = 256        # size of blocks scanned.

import math
import random
from pylab import *
from matplotlib.ticker import MultipleLocator, FormatStrFormatter
import tkFileDialog
from Tkinter import *
from progressBar import *
from binascii import hexlify
import string
import os
import cPickle # cache results

def H(data):
  if not data:
    return 0
  entropy = 0
  for x in range(256):
    p_x = float(data.count(chr(x)))/len(data)
    if p_x > 0:
      entropy += - p_x*math.log(p_x, 2)
  return entropy

def entropy_scan (data, block_size) :
  if SHOWPROGRESS:
      progress = progressBar(0, len(data) - block_size, 77)
  # creates blocks of block_size for all possible offsets ('x'):
  blocks = (data[x : block_size + x] for x in range (len (data) - block_size))
  i = 0
  for block in (blocks) :
    i += 1
    if SHOWPROGRESS:
        progress(i)
    yield H (block)

# performance improvement if you have psyco
try:
  import psyco
  psyco.full()
  print "got psyco"
except ImportError:
  pass

# get target file as argument var or from dialog:
filename = ""
if sys.argv[1:]:
    filename = sys.argv[1]
else:
    root = Tk()
    root.withdraw()
    filename = tkFileDialog.askopenfilename(title="Target binary",
                                        filetypes=[("All files", "*")])

# run, print graph:

if filename:
    # Open file and scan for entropy:
    if os.path.splitext(filename)[1] == ".entropy":
        print "File is a cached '.entropy' from previous scan"
        results = cPickle.load(open(filename, 'rb'))
        filename = os.path.splitext(filename)[0]
        print filenamea
        raw = open(filename, 'rb').read()
    else:
        raw = open(filename, 'rb').read()
        # debug with test data:
        """
        import random
        raw = ''.join (
        [chr (random.randint (0, 64)) for x in xrange (1024)] +
        [chr (random.randint (0, 255)) for x in xrange (1024)] +
        [chr (random.randint (0, 64)) for x in xrange (1024)] )
        """
        results = list( entropy_scan(raw,BLOCKSIZE) )
        print "saving cache of entropy scan data to %s" % filename+".entropy"
        cPickle.dump(results, open(filename+".entropy", 'wb')) 

    # Print blocks that are above a defined threshold of entropy:
    if PRINTONTHRESHOLD > 0:
        print
        found = 0
        for i in range(len(results)):
            if results[i] > PRINTONTHRESHOLD:
                if found == 0:
                    table = string.maketrans("rnt", '   ') # don't like newlines
                    #blockstr = string.translate(str(raw[i : i+16]), table) # translate to string value
                    print "0x%8x %.2f: %s %s" % (i, results[i], hexlify(raw[i : i+8]),
                                                     hexlify(raw[i+8 : i+16]))
                    #%.3f - %016X / %s" % (i, results[i], raw[i : i + 16], raw[i : i + 16])
                    found = ONLYFIRSTBLOCK
            else:
                found = 0

    # Plot
    filesize = os.path.getsize(filename)
    imgdpi = 100
    imgwidth = filesize / imgdpi

    if imgwidth > 327:
      imgwidth = 327

    majorLocator   = MultipleLocator(0x400)   # mark every 1024 bytes
    majorFormatter = FormatStrFormatter('%X') # change to %d to see decimal offsets

    ax = subplot(111)
    plot(results, linewidth=2.0, antialiased=False)
    subplots_adjust(left=0.02, right=0.99, bottom=0.2)

    ax.axis([0,filesize,0,8])
    ax.xaxis.set_major_locator(majorLocator)
    ax.xaxis.set_major_formatter(majorFormatter)
    xticks(rotation=315)

    xlabel('block (byte offset)')
    ylabel('entropy')
    title('Entropy levels')

    grid(True)

    img = gcf()
    img.set_size_inches(imgwidth, 6)
    img.savefig(filename+".png", dpi=imgdpi)

    draw()
    show()

progressBar.py (originally from active state but modified for our use)

import sys

class progressBar:
    """ Creates a text-based progress bar. Call the object with the `print'
        command to see the progress bar, which looks something like this:

        [=======>        22%                  ]

        You may specify the progress bar's width, min and max values on init.
    """
    def __init__(self, minValue = 0, maxValue = 100, totalWidth=80):
        self.progBar = "[]"   # This holds the progress bar string
        self.min = minValue
        self.max = maxValue
        self.span = maxValue - minValue
        self.width = totalWidth
        self.amount = 0       # When amount == max, we are 100% done
        self.updateAmount(0)  # Build progress bar string
        self._old_pbar = ""   # used to track change
        self.pbar_str = ""

    def updateAmount(self, newAmount = 0):
        """ Update the progress bar with the new amount (with min and max
            values set at initialization; if it is over or under, it takes the
            min or max value as a default. """
        if newAmount > self.max: newAmount = self.max
        self.amount = newAmount

        # Figure out the new percent done, round to an integer
        diffFromMin = float(self.amount - self.min)
        percentDone = (diffFromMin / float(self.span)) * 100.0
        percentDone = int(round(percentDone))

        # Figure out how many hash bars the percentage should be
        allFull = self.width - 2
        numHashes = (percentDone / 100.0) * allFull
        numHashes = int(round(numHashes))

        # Build a progress bar with an arrow of equal signs; special cases for
        # empty and full
        if numHashes == 0:
            self.progBar = "[>%s]" % (' '*(allFull-1))
        elif numHashes == allFull:
            self.progBar = "[%s]\n" % ('='*allFull)
        else:
            self.progBar = "[%s>%s]" % ('='*(numHashes-1),
                                        ' '*(allFull-numHashes))

        # figure out where to put the percentage, roughly centered
        percentPlace = (len(self.progBar) / 2) - len(str(percentDone))
        percentString = str(percentDone) + "%"

        # slice the percentage into the bar
        self.progBar = ''.join([self.progBar[0:percentPlace], percentString,
                                self.progBar[percentPlace+len(percentString):]
                                ])

    def __str__(self):
        return str(self.progBar)

    def __call__(self, value):
        """ Updates the amount, and writes to stdout. Prints a carriage return
            first, so it will overwrite the current line in stdout."""

        self.updateAmount(value)
        self.pbar_str = str(self)
        if self.pbar_str != self._old_pbar:
            self._old_pbar = self.pbar_str
            sys.stdout.write(self.pbar_str + "\r")
            sys.stdout.flush()

Update: Just use their RSS feeds. They have updated them to include full details rendering my own versions mute.

One of the resources I use to monitor for current cryptography papers is the Cryptology ePrint Archive, a routinely updated repository of all cryptography papers. Recently the Archive setup their own RSS feeds. Their feed provides a link to the article summaries. For me this isn’t enough and for a while I’ve had my own bot building an RSS feed listing the latest additions to the archive including their full summary inside the feed itself, not just a link to it. It was too buggy to link publicly so last night I fixed what should be the last of the problems to providing a stable feed. I have a feed for just newly published articles and another for all articles new or updated.

Incomplete: Need to discuss how the author discovered attacks. Need to check my description using the detailed equations provided. I must illustrate the attack methods.

The interest in this thesis is due to its reference in Heard Hash Functions and many other papers relating to hash algorithms. In Chapter 5 a fixed point attack against hash algorithms is discussed. Methods are given for overcoming the appended message length specified in Merkle-Damgård (cache) constructed hash functions.

What is a Fixed Point Attack?
A Fixed Point Attack involves finding a random block whose properties allow the attacker to insert the block into the original message without changing the final hash. As a result two different messages are created with the same hash (the original message and the original+the special block). To produce this special block first make note of all the internal hash states produced after each block is compressed (see: SHA-1 Illustrated). Next generate random blocks (Xi) until you find one that meets two properties:

1. The hash state before compression of block Xi is the same as the hash state returned after compression.
2. The hash state of Xi equals one of the internal hash states of the original message.

After finding such a block it can be inserted into the message directly after the message block whose internal hash state it matched.

Overcoming Message Length
MD5, MD4 and SHA use Merkle-Damgård construction (cache) which specifies that the length of the entire message be appended to it. Therefor, a simple Fixed Point Attack will not do because the message length will change when the special block is inserted. This intern changes the hash of the last block thereby changing the final hash returned. The paper gives 3 methods to overcome this.

1. The length is a 64 bit integer so add the special block 2^64 times, in affect causing the number to loop. This does not work on SHA because SHA does not cover messages greater than 2^64 bits.

2. Look for any two internal hash states in the message that equal each other. If you are lucky enough to have such a message you can delete all the blocks between the two and then expand the message back to the original size using the Fixed Point Attack.

3. Run a Fixed Point Attack and make note of the place in the original message where you can insert the special block. Now, remember that the block compression function adds the resulting hash state to the previous hash state. That means that compression is a function of the current block and the previous blocks hash state. With that understood, we want to find another random block that means the following two requirements:

• The hash state before compression of block Xj is set to the first hash state of the original message.
• The resulting hash state of Xj equals one of the internal hash states of the original message which is less than the hash state matched by the Fixed Point attack.

You’ll notice that this block uses the same method as the Fixed Point only it initializes the incoming hash state to the compression function to be that of the first hash state of the message.

Now we can insert the Xj into the message directly after the message block whose internal hash state it matched. Since the compression of Xj takes the first hash state of the message we delete all the blocks up until that point, effectively making Xj the first block and reducing the size of the message. Now the Fixed Point block Xi can be inserted into the message directly after the message block whose internal hash state it matched and can be repeated to bring the message back to it’s original size.

This chapter also discusses how the attack was found as well as possible solutions. which I still need to cover.

References
References I must find:

• [PvO95] Bart Preneel and Paul C. van Oorschot. MDx-MAC and building fast MACs from hash functions. In Don Coppersmith, editor, Proc. CRYPTO 95, pages 1–14. Springer, 1995. Lecture Notes in Computer Science No. 963.

To find the attacks the author used Binary Decision Diagrams to look at the logical structure of MD5,MD5,SHA-1. References I must find:

• [Bry92] Randal E. Bryant. Symbolic boolean manipulation with ordered binary decision diagrams. ACM Computing Surveys, 24(3):293–318, September 1992.
• [Hu97] Alan J. Hu. Formal hardware verification with BDDs: An introduction. In IEEE Pacific Rim Conference on Communications, Computers, and Signal Processing, pages 677–682, 1997.

Incomplete: must create detailed diagram for compression functions.

The following simplifies the specification of SHA-1 in an easy to digest form. First we will cover the general structure of the algorithm. Detail of the expansion and compression routines are covered separately.

First we start with a message. The message is padded and the length of the message is added to the end. It is then split into blocks of 512 bits (Figure 2).

(Figure 2)

The blocks are then processed one at a time. Each block must be expanded and compressed. The value after each compression is added to a 160bit buffer called the current hash state. After the last block is processed the current hash state is returned as the final hash. A overview of this procedure can be seen in Figure 3.

(Figure 3)

Let’s look more closely at the expansion and compression functions. For expansion each 512 bit message block is separated into chunks of 32 bits. As you can see in Figure 3 these 16 chunks are then used to create 64 more chunks for a total of 80. Details of how this is done are described later.

(Figure 4)

Now all 80 of these chunks are compressed into a 160 bit value which is added to the current hash state (Figure 5):

(Figure 5)

Figure 5 shows one block being processed. The expansion and compression functions are repeated for each block with the return constantly being added to the current hash state buffer. Once all blocks have been processed it is this value that is returned as the hash of the message.

3 tasks were generalized above: How the message is prepared before processing, how exactly the block is expanded to 80 chunks (Figure 4) and how those chunks are compressed (Figure 5). It is not essential to understand them in detail but should you desire, here are the details.

Message Preparation

The message is prepared in 4 steps:

1. Append a single binary 1 bit to the message
2. Split into blocks of 512 bits each (Figure 2 above)
3. The last block must be equal to 448 so that we can append the message length (next step). If it is under pad with binary 0 bits until equal to 448. If over, pad until it is 512 bits and create an additional block of 448 binary 0 bits.
4. Append the length of the original message to the last block. Represent this length as a 64 bit integer (making the last block equal to 512 bits).

I should also mention that before we process any blocks we must initiate the hash state buffer. The buffer is actually 5 separate 32 bit integers:

• h0 = 67452301
• h1 = EFCDAB89
• h2 = 98BADCFE
• h3 = 10325476
• h4 = C3D2E1F0

Block expansion
Each 512 bit block is split further into 32 bit chunks (“words“) as seen in Figure 4. These 16 chunks are then expanded to a total of 80. The processes of expansion is a simple XOR of 4 values. For instance, the next chunk, chunk 17, is created by XOR’ing together chunk 17-3, 17-8, 17-14 and 17-16. For chunk 18 run the same processes but subtracting from 18 instead of 17. This continues until all 80 have been created. This can clearly be seen in the animation to the right. (If the animation is not playing reload the page.)

Block compression

This work is licensed under a Creative Commons Public Domain License and may be used however you wish.  For sources to Dia based diagrams, contact me.

Update 2006.02.11: clearer explanation of CTFP preimage resistance.

This is a very good introduction to what a hash algorithm is, what it is for and what collisions are all about. It does not cover specific details, only the general understanding. It’s a quick read so I’ll forgo summarizing the contents.

The article explains the common terms used in most papers that discuss collisions. These terms are used to classify the type of collision attacks possible and are necessary to understand when reading other papers:

• Collision resistance measures how difficult it is to create two inputs which produce any hash value which is the same for both inputs. In this scenario the attacker can control both inputs.
• Preimage resistance measures how difficult it is to create one input which matches the hash value of an unknown input. Here the attacker does not know the other input and is restricted by needing to create a specific hash value.
• Second preimage resistance measures how difficult it is to create one input which matches the hash value of a known input. Here the attacker can see both inputs but only controls one. Attacker is still restricted by having to create an input which matches the specific hash value of the other. However, knowing the input that produced the hash might be of assistance.

Both preimage and second preimage are similar in that the objective is to get one input to match a predefined hash which is not controlled by the attacker. Also, in the Herding Hash Functions by John Kelsey and Tadayoshi Kohno they that there is a 4rth resistance value:

• Chosen Target Forced Prefix preimage resistance measures how difficult it is to create a collision when the first input is known while the second input is not know yet. This is similar to preimage resistance except that here the attacker controls the first input and not the second. Well, almost. The attacker is permitted to append data to the second input. The attacker must determine the hash first using the first input and then “herd” the second input to the same hash. Herding is done by adding data to the second input to make it collide. Is a process that involves carefully predetermining the first input and using internal states from its hash generation in the appended data to the second.