Malware Lineage in the Wild

Irfan Ul Haq, Sergio Chica, Juan Caballero, Somesh Jha
In COSE Journal: Elsevier Computer and Security Journal
August 2018.

Abstract: Malware lineage studies the evolutionary relationships among malware and has important applications for malware analysis. A persistent limitation of prior malware lineage approaches is to consider every input sample a separate malware version. This is problematic since a majority of malware are packed and the packing process produces many polymorphic variants (i.e., executables with different file hash) of the same malware version. Thus, many samples correspond to the same malware version and it is challenging to identify distinct malware versions from polymorphic variants. This problem does not manifest in prior malware lineage approaches because they work on synthetic malware, malware that are not packed, or packed malware for which unpackers are available.

In this work, we propose a novel malware lineage approach that works on malware samples collected in the wild. Given a set of malware executables from the same family, for which no source code is available and which may be packed, our approach produces a lineage graph where nodes are versions of the family and edges describe the relationships between versions. To enable our malware lineage approach, we propose the first technique to identify the versions of a malware family and a scalable code indexing technique for determining shared functions between any pair of input samples. We have evaluated the accuracy of our approach on 13 open-source programs and have applied it to produce lineage graphs for 10 popular malware families. Our malware lineage graphs achieve on average a 26 times reduction from number of input samples to number of versions.


author = {Irfan Ul Haq and Sergio Chica and Juan Caballero and Somesh Jha},
title = {{Malware Lineage in the Wild}},
journal = {Computers \& Security},
publisher = {Elsevier},
volume = {78},
number = {C},
month = {August},
year = {2018},
pages = {347-363},
issn = {0167-4048},
doi = {10.1016/j.cose.2018.07.012},
jcr = {2.862},

Replay PANDA malware recordings

PANDA provides a record and replay system. It executes a binary, records its execution and later provides the facility to replay the recording. A huge number of malware recordings exist on http://panda.gtisc.gatech.edu/malrec/.

But before you may run the replay, you need to patch the given recording to recover a snapshot file. To patch, we need (1) a tool called bpatch.py and (2) reference snapshots that can be downloaded from here. Please follow following instructions to download, patch and replay PANDA recordings:

  1. Create a file bpatch.py with following code:

#!/usr/bin/env python

import sys
import shutil
import os

if len(sys.argv) < 2:     print >>sys.stderr, "usage: %s <patchfile>" % sys.argv[0]

f = open(sys.argv[1])
ref = f.readline().strip()

if not os.path.exists(ref):
    print >>sys.stderr, "error: couldn't find reference snapshot %s" % ref
    print "Using reference %s as a base" % ref

basename = os.path.splitext(sys.argv[1])[0]
patched = basename + '-rr-snp'
print "Creating patched snapshot %s" % patched
shutil.copy(ref, patched)

of = open(patched, 'rb+')

for line in f:
    off, val = line.strip().split()
    off = int(off, 16)
    val = val.decode('hex')


print "All done, no errors." 

Create a directory ‘references’ anywhere on your system. We refer to this as REF_DIR.

$ cd REF_DIR
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/0000c18e-a947-42ea-abb2-234ea18facdc-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/0002f074-cd1b-4523-aacd-eeccd61c0f96-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/00568419-706b-4c2e-ad3a-4de0add3780d-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/023c870e-4be8-4f1c-a712-340e21c67565-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/0335fe75-a7bd-4963-8304-da7e59005692-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/097b607a-735e-4ac0-b853-c15dc58b58fc-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/1b5091e3-98a5-4058-a944-c5d6f87fe103-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/5ad7f823-1b5f-4f99-8f2b-53bf69e0fc08-rr-snp

Alternatively, you can manually download reference snapshots from http://panda.gtisc.gatech.edu/malrec/rr/references/.

$ cd /tmp
$ mkdir malware_recordings
$ cd malware_recordings

Download and unzip any `rrlog’ malware recordings from http://panda.gtisc.gatech.edu/malrec/ Or
http://giantpanda.gtisc.gatech.edu/malrec/ into `malware_recordings`.

For example, you can download recording with UUID 3618dea7-fe33-4726-b3c6-befc5f9b32d0 using following command:
$ wget http://panda.gtisc.gatech.edu/malrec/rr/3618dea7-fe33-4726-b3c6-befc5f9b32d0.txz
$ tar xvf 3618dea7-fe33-4726-b3c6-befc5f9b32d0.txz

Apply patch using bpatch.py tool.
$ ln -s REF_DIR logs/rr
$ python bpatch.py logs/rr/3618dea7-fe33-4726-b3c6-befc5f9b32d0.patch

If successful it will print following message:
“All done, no errors“

Run PANDA with following -replay command:
“ -replay /tmp/malware_recordings/logs/rr/3618dea7-fe33-4726-b3c6-befc5f9b32d“

Decaf trace reader installation program

First of all, there is no configuration file, as mentioned in INSTALL file, inside trace_reader directory. You should not worry about it.

When you run ‘make’ command, sometime compiler raises error for various undefined functions. You should make sure that binutils-multiarch is installed on the system. So, following instruction may help you — at least it worked for me 🙂 :

$ sudo apt-get install binutils-multiarch
$ sudo mv /usr/lib/libbfd.so /usr/lib/libbfd.so.bk
$ sudo mv /usr/lib/libopcodes.so /usr/lib/libopcodes.so.bk
$ sudo ln -s /usr/lib/libbfd-<VERSION-INFO>-multiarch.so /usr/lib/libbfd.so
$ sudo ln -s /usr/lib/libopcodes-<VERSION-INFO>-multiarch.so /usr/lib/libopcodes.so

$ make