First time here? Check out the FAQ!
THIS IS A TEST INSTANCE. Feel free to ask and answer questions, but take care to avoid triggering too many notifications.
Ask Your Question
0

Converting Pcap file to CSV file while defautly keeping all features/fields defined in pcap

Hi all, I have been searching for solutions that can transform PCAP files to CSV format while keeping all the details defined in the pcap file, without explicitly defining which features/fields that i want to include in CSV format.

With Wireshark&Tshark, i have tested the following two ways:

  1. the first one is to use tshark commands such as:

    tshark -r traffic.pcap > traffic.csv

    tshark -r traffic.pcap -T fields -e ip.src -E separator=, -E occurrence=f > traffic.csv

    In any case, the traffic.csv contains only general information (e.g., No., Time, Source, Destination, Length, Protocol, Info) of the traces without any packet detail(Packet Bytes).

  2. I also tried with Wireshark by selecting Export Packet Dissections, however i got differet results by exporting the same pcap file to csv, json, and plain text format. In general, the exported csv file still contains only general information of packets, without packet detail (Packet Bytes), even i selected the field (Packet Bytes during export).

I wonder if there is a way i can also have the the details of each packet in the csv file? Thanks.

Artemis's avatar
5
Artemis
asked 2020-05-12 15:03:58 +0000
edit flag offensive 0 remove flag close merge delete

Comments

What would you expect a CSV of all fields to look like? A CSV file is a representation of data in a row\column format, with the rows as packets and the columns as fields. Depending on the data the number of columns could be enormous.

Maybe you could explain what you want to do with the data so alternatives could be suggested?

grahamb's avatar grahamb (2020-05-12 15:54:13 +0000) edit
more
  • delete

the dimension of the data is not an issue, since i would like to explore the different column combinations to support the machine learning work later. One thing i found wierd is that, if I choose the json or plain text as an export format, the packet byte is in the file, it does not work with csv file.

Artemis's avatar Artemis (2020-05-12 15:59:37 +0000) edit
more
  • delete

CSV needs to know what the columns are in order to put the value into the correct columns. Maybe look at the ElasticSearch output, -T ek.

grahamb's avatar grahamb (2020-05-12 16:19:27 +0000) edit
more
  • delete

I don't know if i'm being naive thinking that the automatic conversion can leave fields as empty if certain features are not appicable to specific packets, I will check the ElasticSearch as suggested thanks.

Artemis's avatar Artemis (2020-05-12 19:22:00 +0000) edit
more
  • delete
add a comment see more comments

1 Answer

0

It's not clear that the concept of a CSV of all fields is well-defined.

A row of a CSV file is just a Comma-Separated sequence of Values; there are no tags to indicate what the values are values of. A CSV row, containing the values of all the fields in a packet, would just be a sequence of values, with no indication what those values signify - and not all rows would have the same number of values.

If you used the first row as a table of field names, to solve that problem, that would require that the first row have the name of each field that appears in a packet in the file, and that elements in subsequent rows may be empty (if the fed in question isn't in the packet corresponding to that row). It might also require either that a field name may appear more than once in the first row, to handle packets with more than one instance of a field.

There is currently no code in Wireshark to do that.

One thing i found wierd is that, if I choose the json or plain text as an export format, the packet byte is in the file, it does not work with csv file.

What do you mean by "the packet byte(s)"? What's exported as JSON are the packet fields; what's shown is the value, in a somewhat human-readable form, which isn't necessarily the raw bytes.

What's exported as CSV are the columns; the columns have the advantage that there is a fixed set of columns specified, so that you don't have the problems I mentioned above with CSV dumping of packet fields.

I.e., there is a reason why CSV export is different from JSON or PDML exports.

For plain text exports, which are intended for humans to read, you can choose to export the columns as displayed in the packet list pane, the packet details as displayed in the packet details pane, or the raw hex/characters dump as displayed in the hex/characters dump pane; more than one of those can be chosen.

Guy Harris's avatar
19.9k
Guy Harris
answered 2020-05-12 19:19:37 +0000
edit flag offensive 0 remove flag delete link
more

Comments

Thanks for the kind isnght, I understand better now the first point. Regarding the second one "it does not work with csv file", i mean is it normal that CSV does not contain any information from packet details pane? i understand the json and plain text formats only structure the packet details in a different way, is csv file supposed to contain all fileds from packet detail panes (the packet byte such as frame attrbutes, eth attrbutes), the situation is that i only got attributes as (No., Time, Source, Destination, Length, Protocol, Info) in csv format, even if i selected the "packet byte" option during export.

Artemis's avatar Artemis (2020-05-12 19:42:28 +0000) edit
more
  • delete

is it normal that CSV does not contain any information from packet details pane?

Yes.

i understand the json and plain text formats only structure the packet details in a different way,

The JSON format is packet details; the plain text format can be packet summaries, packet details, raw packet data, or any combination thereof.

is csv file supposed to contain all fileds from packet detail panes (the packet byte such as frame attrbutes, eth attrbutes),

No.

the situation is that i only got attributes as (No., Time, Source, Destination, Length, Protocol, Info) in csv format, even if i selected the "packet byte" option during export.

The only way you can select any of those during export to CSV is to choose another format that allows you to select them, and then change the format to CSV. If the format chosen is CSV, those options are greyed-out to indicate that ... (more)

Guy Harris's avatar Guy Harris (2020-05-12 23:48:23 +0000) edit
more
  • delete

Thanks a lot for the clarification.

Artemis's avatar Artemis (2020-05-13 07:55:03 +0000) edit
more
  • delete
add a comment see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss.

Add Answer