Malware Lineage in the Wild

Irfan Ul Haq, Sergio Chica, Juan Caballero, Somesh Jha
In COSE Journal: Elsevier Computer and Security Journal
August 2018.

Abstract: Malware lineage studies the evolutionary relationships among malware and has important applications for malware analysis. A persistent limitation of prior malware lineage approaches is to consider every input sample a separate malware version. This is problematic since a majority of malware are packed and the packing process produces many polymorphic variants (i.e., executables with different file hash) of the same malware version. Thus, many samples correspond to the same malware version and it is challenging to identify distinct malware versions from polymorphic variants. This problem does not manifest in prior malware lineage approaches because they work on synthetic malware, malware that are not packed, or packed malware for which unpackers are available.

In this work, we propose a novel malware lineage approach that works on malware samples collected in the wild. Given a set of malware executables from the same family, for which no source code is available and which may be packed, our approach produces a lineage graph where nodes are versions of the family and edges describe the relationships between versions. To enable our malware lineage approach, we propose the first technique to identify the versions of a malware family and a scalable code indexing technique for determining shared functions between any pair of input samples. We have evaluated the accuracy of our approach on 13 open-source programs and have applied it to produce lineage graphs for 10 popular malware families. Our malware lineage graphs achieve on average a 26 times reduction from number of input samples to number of versions.


author = {Irfan Ul Haq and Sergio Chica and Juan Caballero and Somesh Jha},
title = {{Malware Lineage in the Wild}},
journal = {Computers \& Security},
publisher = {Elsevier},
volume = {78},
number = {C},
month = {August},
year = {2018},
pages = {347-363},
issn = {0167-4048},
doi = {10.1016/j.cose.2018.07.012},
jcr = {2.862},

Replay PANDA malware recordings

PANDA provides a record and replay system. It executes a binary, records its execution and later provides the facility to replay the recording. A huge number of malware recordings exist on http://panda.gtisc.gatech.edu/malrec/.

But before you may run the replay, you need to patch the given recording to recover a snapshot file. To patch, we need (1) a tool called bpatch.py and (2) reference snapshots that can be downloaded from here. Please follow following instructions to download, patch and replay PANDA recordings:

  1. Create a file bpatch.py with following code:

#!/usr/bin/env python

import sys
import shutil
import os

if len(sys.argv) < 2:     print >>sys.stderr, "usage: %s <patchfile>" % sys.argv[0]

f = open(sys.argv[1])
ref = f.readline().strip()

if not os.path.exists(ref):
    print >>sys.stderr, "error: couldn't find reference snapshot %s" % ref
    print "Using reference %s as a base" % ref

basename = os.path.splitext(sys.argv[1])[0]
patched = basename + '-rr-snp'
print "Creating patched snapshot %s" % patched
shutil.copy(ref, patched)

of = open(patched, 'rb+')

for line in f:
    off, val = line.strip().split()
    off = int(off, 16)
    val = val.decode('hex')


print "All done, no errors." 

Create a directory ‘references’ anywhere on your system. We refer to this as REF_DIR.

$ cd REF_DIR
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/0000c18e-a947-42ea-abb2-234ea18facdc-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/0002f074-cd1b-4523-aacd-eeccd61c0f96-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/00568419-706b-4c2e-ad3a-4de0add3780d-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/023c870e-4be8-4f1c-a712-340e21c67565-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/0335fe75-a7bd-4963-8304-da7e59005692-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/097b607a-735e-4ac0-b853-c15dc58b58fc-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/1b5091e3-98a5-4058-a944-c5d6f87fe103-rr-snp
$ wget http://panda.gtisc.gatech.edu/malrec/rr/references/5ad7f823-1b5f-4f99-8f2b-53bf69e0fc08-rr-snp

Alternatively, you can manually download reference snapshots from http://panda.gtisc.gatech.edu/malrec/rr/references/.

$ cd /tmp
$ mkdir malware_recordings
$ cd malware_recordings

Download and unzip any `rrlog’ malware recordings from http://panda.gtisc.gatech.edu/malrec/ Or
http://giantpanda.gtisc.gatech.edu/malrec/ into `malware_recordings`.

For example, you can download recording with UUID 3618dea7-fe33-4726-b3c6-befc5f9b32d0 using following command:
$ wget http://panda.gtisc.gatech.edu/malrec/rr/3618dea7-fe33-4726-b3c6-befc5f9b32d0.txz
$ tar xvf 3618dea7-fe33-4726-b3c6-befc5f9b32d0.txz

Apply patch using bpatch.py tool.
$ ln -s REF_DIR logs/rr
$ python bpatch.py logs/rr/3618dea7-fe33-4726-b3c6-befc5f9b32d0.patch

If successful it will print following message:
“All done, no errors“

Run PANDA with following -replay command:
“ -replay /tmp/malware_recordings/logs/rr/3618dea7-fe33-4726-b3c6-befc5f9b32d“

On Mitigating Sampling-Induced Accuracy Loss in Traffic Anomaly Detection Systems

Sardar Ali, Irfan Ul Haq, Sajjad Rizvi, Naurin Rasheed, Unum Sarfraz, Syed Ali Khayam, and Fauzan Mirza
ACM SIGCOMM Computer Communication Review (CCR)
Volume 40, Issue 3, July 2010, ACM New York, NY, USA.

Abstract: Real-time Anomaly Detection Systems (ADSs) use packet sampling to realize traffic analysis at wire speeds. While recent studies have shown that a considerable loss of anomaly detection accuracy is incurred due to sampling, solutions to mitigate this loss are largely unexplored. In this paper, we propose a Progressive Security-Aware Packet Sampling (PSAS) algorithm which enables a real-time inline anomaly detector to achieve higher accuracy by sampling larger volumes of malicious traffic than random sampling, while adhering to a given sampling budget. High malicious sampling rates are achieved by deploying inline ADSs progressively on a packet’s path. Each ADS encodes a binary score (malicious or benign) of a sampled packet into the packet before forwarding it to the next hop node. The next hop node then samples packets marked as malicious with a higher probability. We analytically prove that under certain realistic conditions, irrespective of the intrusion detection algorithm used to formulate the packet score, PSAS always provides higher malicious packet sampling rates.

To empirically evaluate the proposed PSAS algorithm, we simultaneously collect an Internet traffic dataset containing DoS and portscan attacks at three different deployment points in our university’s network. Experimental results using four existing anomaly detectors show that PSAS, while having no extra communication overhead and extremely low complexity, allows these detectors to achieve significantly higher accuracies than those operating on random packet samples.


  title={On mitigating sampling-induced accuracy loss in traffic anomaly detection systems},
  author={Ali, Sardar and Haq, Irfan Ul and Rizvi, Sajjad and Rasheed, Naurin and Sarfraz, Unum and Khayam, Syed Ali and Mirza, Fauzan},
  journal={ACM SIGCOMM Computer Communication Review},

What is the Impact of P2P Traffic on Anomaly Detection?

Irfan Ul Haq, Sardar Ali, Hassan Khan, and Syed Ali Khayam
13th International Symposium on Recent Advances in Intrusion Detection (RAID)
September 15-17, 2010, Ottawa, Canada.

Acceptance Rate = 23.1%

Abstract: Recent studies estimate that peer-to-peer (p2p) traffic comprises 40-70% of today’s Internet traffic. Surprisingly, the impact of p2p traffic on anomaly detection has not been investigated. In this paper, we collect and use a labeled dataset containing diverse network anomalies (portscans, TCP floods, UDP floods, at varying rates) and p2p traffic (encrypted and unencrypted with BitTorrent, Vuze, Flashget, µTorrent, Deluge, BitComet, Halite, eDonkey and Kademlia clients) to empirically quantify the impact of p2p traffic on anomaly detection. Four prominent anomaly detectors (TRW-CB, Rate Limiting, Maximum Entropy  and NETAD) are evaluated on this dataset.

Our results reveal that: 1) p2p traffic results in up to 30% decrease in detection rate and up to 45% increase in false positive rate; 2) due to a partial overlap of traffic behaviors, p2p traffic inadvertently provides an effective evasion cover for high- and low-rate attacks; and 3) training an anomaly detector on p2p traffic, instead of improving accuracy, introduces a significant accuracy degradation for the anomaly detector. Based on these results, we argue that only p2p traffic filtering can provide a pragmatic, yet short-term, solution to this problem. We incorporate two prominent p2p traffic classifiers (OpenDPI and Karagiannis’ Payload Classifier(KPC)) as pre-processors into the anomaly detectors and show that the existing non-proprietary p2p traffic classifiers do not have sufficient accuracies to mitigate the negative impacts of p2p traffic on anomaly detection.

Given the premise that p2p traffic is here to stay, our work demonstrates the need to rethink the classical anomaly detection design philosophy with a focus on performing anomaly detection in the presence of p2p traffic. We make our dataset publicly available for evaluation of future anomaly detectors that are designed to operate with p2p traffic.



  title = {What is the impact of p2p traffic on anomaly detection?},
  author={Haq, Irfan Ul and Ali, Sardar and Khan, Hassan and Khayam, Syed Ali},
  booktitle={Recent Advances in Intrusion Detection},