A few days ago we at Astyran needed a tool that could extract data from a network capture file. A quick Baidu search and we found the tool Xplico.
The goal of Xplico is to extract application data from internet network capture. Brilliant, exactly what we needed. In layman’s terms: you feed Xplico a capture file, and it will extract all data (pictures, emails, VOIP call, …). And even better, it is free!
According to the website the features of the current version (0.7.1) are as follows:
- Protocols supported: HTTP, SIP, IMAP, POP, SMTP, TCP, UDP, IPv6, …;
- Port Independent Protocol Identification (PIPI) for each application protocol;
- Multithreading;
- Output data and information in SQLite database or MySQL database and/or files;
- At each data reassembled by Xplico is associated a XML file that uniquely identifies the flows and the pcap containing the data reassembled;
- Real time elaboration (depends on the number of flows, the types of protocols and by the performance of computer -RAM, CPU, HD access time, …;
- TCP reassembly with ACK verification for any packet or soft ACK verification;
- Reverse DNS lookup from DNS packages contained in the inputs files (pcap), not from external DNS server;
- No size limit on data entry or the number of files entrance (the only limit is HD size);
- IPv4 and IPv6 support;
- Modularity. Each Xplico component is modular. The input interface, the protocol decoder (Dissector) and the output interface (dispatcher) are all modules;
- The ability to easily create any kind of dispatcher with which to organize the data extracted in the most appropriate and useful to you;
But don’t be scared of these technical details, Xplico is really simple to use, that is, if you can run it… Source code is available and some packages for Fedora and Ubuntu Linux server and workstation.
Alas, no package on Amazon AMI version and thus we had no choice but to compile Xplico on AMI. Ok, we had a choice, but we thought it was fun (that was, before the third sleepless night) to have it running on our AMI. Anyway, here is the result!
Request a 64-bit Amazon Linux AMI instance. use SSH to connect and type the following:
1: sudo su -
2: yum install sqlite-devel
3: yum install zlib-devel
4: yum install openssl-devel
5: yum install flac-devel
6: yum install libjpeg-devel
7: yum install libpng-devel
8: yum install libvorbis-devel
9: yum install libao-devel
10: yum install gnutls-devel
11: yum install libtiff-devel
12: yum install speex-devel
13: yum install gcc
14: yum install gcc-c++
15: yum install make
16: yum install libtool
17: yum install patch
18: yum install git
19: yum install php-devel
20: yum install mysql
21: yum install mysql-libs
22: yum install mysql-devel
23: yum install libpcap-devel
24: yum install wireshark-devel
25: yum install libXt-devel
26: yum install libXaw-devel
27: cd
28: git clone git://github.com/pinard/Recode.git
29: cd Recode
30: ./configure
31: make
32: make install
33: make clean
34: cd
35: git clone git://github.com/yasm/yasm.git
36: cd yasm
37: ./autogen.sh
38: make
39: make install
40: make clean
41: cd
42: wget http://downloads.sourceforge.net/project/opencore-amr/vo-amrwbenc/vo-amrwbenc-0.1.1.tar.gz
43: tar -xzvf vo-amrwbenc-0.1.1.tar.gz
44: cd vo-amrwbenc-0.1.1
45: ./configure
46: make
47: make install
48: make clean
49: cd
50: wget http://ftp.penguin.cz/pub/users/utx/amr/amrwb-10.0.0.0.tar.bz2
51: tar -xvjf amrwb-10.0.0.0.tar.bz2
52: cd amrwb-10.0.0.0
53: ./configure
54: make
55: make install
56: make clean
57: cd
58: wget http://ftp.penguin.cz/pub/users/utx/amr/amrnb-10.0.0.0.tar.bz2
59: tar -xvjf amrnb-10.0.0.0.tar.bz2
60: cd amrnb-10.0.0.0
61: ./configure
62: make
63: make install
64: make clean
65: cd
66: wget http://www.ladspa.org/download/ladspa_sdk.tgz
67: tar -xzvf ladspa_sdk.tgz
68: cd ladspa_sdk/src
69: cp ladspa.h /usr/local/include
70: cd
71: wget ftp://ftp.mars.org/pub/mpeg/libmad-0.15.1b.tar.gz
72: tar -xzvf libmad-0.15.1b.tar.gz
73: cd libmad-0.15.1b
74: sed ‘/fforce-mem/d’ configure > sedtmp
75: mv sedtmp configure
76: chmod +x configure
77: make
78: make install
79: make clean
80: cd
81: wget ftp://ftp.mars.org/pub/mpeg/libid3tag-0.15.1b.tar.gz
82: tar -xzvf libid3tag-0.15.1b.tar.gz
83: cd libid3tag-0.15.1b
84: ./configure
85: make
86: make install
87: make clean
88: cd
89: wget http://www.mpg123.de/download/mpg123-1.13.4.tar.bz2
90: tar -xvjf mpg123-1.13.4.tar.bz2
91: cd mpg123-1.13.4
92: ./configure
93: make
94: make install
95: make clean
96: cd
97: wget http://sourceforge.net/projects/lame/files/lame/3.99/lame-3.99.4.tar.gz
98: tar -xzvf lame-3.99.4.tar.gz
99: cd lame-3.99.4
100: ./configure
101: make
102: make install
103: make clean
104: cd
105: wget http://www.wavpack.com/wavpack-4.60.1.tar.bz2
106: tar -xvjf wavpack-4.60.1.tar.bz2
107: cd wavpack-4.6.0.1
108: ./configure
109: make
110: make install
111: make clean
112: cd
113: wget http://mikmod.raphnet.net/files/libmikmod-3.2.0-beta2.tar.gz
114: tar -xzvf libmikmod-3.2.0-beta2
115: cd libmikmod-3.2.0-beta2
116: ./configure
117: make
118: make install
119: make clean
120: cd
121: git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg
122: cd ffmpeg
123: ./configure --enable-nonfree --prefix=/usr --enable-gnutls --enable-libmp3lame --enable-version3
124: make
125: make install
126: make clean
127: cd
128: git clone git://sox.git.sourceforge.net/gitroot/sox/sox
129: cd sox
130: autoreconf -i
131: ./configure
132: make -s
133: make install
134: make clean
135: cd
136: wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.8.tar.gz
137: tar -zxvf GeoIP-1.4.8.tar.gz
138: cd GeoIP-1.4.8
139: autoreconf -i
140: ./configure
141: make
142: cd
143: wget http://sourceforge.net/projects/xplico/files/Xplico%20versions/version%200.7.1/xplico-0.7.1.tgz
144: tar -zxvf xplico-0.7.1.tgz
145: cd xplico-0.7.1
146: wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
147: gzip -d GeoLiteCity.dat.gz
148: rm -f *dat.gz
149: make
150: cd
151: wget http://downloads.ghostscript.com/public/ghostpdl-9.04.tar.gz
152: tar -xvzf ghostpdl-9.04.tar.gz
153: rm -f ghostpdl-9.04.tar.gz
154: dd if=/dev/zero of=/tmp/tmpswap bs=1024 count=2000000
155: mkswap /tmp/tmpswap 2000000
156: swapon /tmp/tmpswap
157: cd ghostpdl-9.04
158: ./configure
159: make
160: cd
161: cp ghostpdl-9.04/main/obj/pcl6 xplico-0.7.1
162: rm -rf ghostpdl-9.04
163: git clone git://github.com/sam-github/libnet.git
164: cd libnet/libnet
165: ./autogen.sh
166: ./configure
167: make
168: make install
169: wget http://sourceforge.net/projects/ucsniff/files/videosnarf/videosnarf-0.63.tar.gz
170: tar -xzvf videosnarf-0.63.tar.gz
171: cd videosnarf-0.63
172: ./configure
173: make
174: cd
175: cp videosnarf-0.63/src/videosnarf xplico-0.7.1
176: cd xplico-0.7.1
177: sed -i ‘s/DLT_IPV4/228/’ dissectors/ip/ip.c
178: sed -i ‘s/DLT_IPV6/229/’ dissectors/ipv6/ipv6.c
179: sed -i ‘s/apache2/httpd/’ Makefile
180: mkdir /etc/httpd/sites-enabled
181: make install
182: cd
183: rm -f *.tar.gz
184: rm -f *.bz2
185: rm -f *.tgz
186: swapoff /tmp/tmpswap
187: rm -f /tmp/tmpswap
At this moment, you can test the installation and rund xplico from commandline.

Next time, we try to get the web-enabled version running! Meanwhile, have fun!
History
2012/02/08 First version. Still some rough edges.