THIS IS A TEST INSTANCE. Feel free to ask and answer questions, but take care to avoid triggering too many notifications.
Ask Your Question
0

tshark export to ek json too slow?

I'm using this version of tshark

TShark (Wireshark) 2.4.5 (Git v2.4.5 packaged as 2.4.5-1)

running on Ubuntu Server 18.04 LTS, and the export to ek json (-T ek > file.json) is being too slow. Here an example:

alvaro@alvaro-server:~/PF_RING/userland/examples_zc$ time tshark -r file.pcap -T ek > file2.json 
^C

real    18m33.665s
user    14m2.707s
sys 4m3.751s

It didn't even finish and took 18 min...

updated 2018-07-28 08:34:28 +0000
This post is a wiki. Anyone with karma >750 is welcome to improve it.
edit flag offensive 0 remove flag close merge delete

Comments

What's the size of the input file, use capinfos to show the details? Have you tried turning off name resolution -n? Can you share the capture file somewhere publicly?

grahamb's avatar grahamb (2018-07-28 09:50:21 +0000) edit
more
  • delete

These are the details of the pcap file:

alvaro@alvaro-server:~/PF_RING/userland/examples_zc$ capinfos file.pcap 
File name:           file.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
File timestamp precision:  microseconds (6)
Packet size limit:   file hdr: 262144 bytes
Number of packets:   33 M
File size:           2577 MB
Data size:           2035 MB
Capture duration:    29.995985 seconds
First packet time:   2018-07-28 09:17:49.004955
Last packet time:    2018-07-28 09:18:19.000940
Data byte rate:      67 MBps
Data bit rate:       542 Mbps
Average packet size: 60.00 bytes
Average packet rate: 1130 kpackets/s
SHA1:                5b0fd55620b8a3759362b68a3e88e3ec476ea634
RIPEMD160:           ce86d8074cb92fee3a66d6200cb5bdeebd7ab3d6
MD5:                 5806f2cfad7fefcdb1065ed2991be326
Strict time order:   True
Number of interfaces in file: 1
Interface #0 info:
                     Encapsulation = Ethernet (1 - ether)
                     Capture length = 262144
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Number of stat entries = 0
                     Number of packets = 33916811

I tried now turning off name resolution, but it doesn ... (more)

alvarospunk's avatar alvarospunk (2018-07-28 10:03:40 +0000) edit
more
  • delete

file uploaded

alvarospunk's avatar alvarospunk (2018-07-28 11:35:46 +0000) edit
more
  • delete
add a comment see more comments

1 Answer

0

I think you're just hitting the limits of working with large capture files.

The file has 33M packets. Splitting the file into 1M packets chunks using editcap -c 1000000 file.pcap out.pcapng and then running the same tshark output on a 1M file took 144 seconds (i7-8550U with SSD), the tshark process peaked at 2GB RAM and the output file was 3.88 GB.

grahamb's avatar
23.8k
grahamb
answered 2018-07-28 18:28:00 +0000
edit flag offensive 0 remove flag delete link
more

Comments

Thank you for your answer. Unfortunately, I think you're right and makes sense (I have to deal with larger pcap files and will try to find another solution).

alvarospunk's avatar alvarospunk (2018-07-28 21:45:03 +0000) edit
more
  • delete
add a comment see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss.

Add Answer