I'm trying to ingest packet captures into Elasticsearch, using filebeat. Seems like the ringbuffer doesn'twork when using the option "-T ek". I'm using wireshark 3.0.2 on CentOS 7
The ring buffer is a mechanism in the raw binary capture code path (in dumpcap).
There is no ring buffer mechanism for any -T text formats. Note that those mechanisms write to the standard output stream, not to files created by TShark, so a ring buffer mechanism for text output would require that TShark create the text files itself.
If rotatelogs reads its standard input and writes it to a sequence of log files, switching to a new file based on time or file size (as the documentation implies but does not explicitly state, perhaps assuming it's obvious), then piping the output of TShark to rotatelogs should work.
Comments
What is the full command line you're using for
tshark?Do you get them same warning as in this question:
https://discuss.elastic.co/t/tshark-a...
It's an extra step but can you write the files in raw mode with ring buffer then process back through
tsharkto feed JSON to filebeat?The ring buffer mechanism is for raw capture files (pcap or pcapng files), not for dissected output such as text, JSON, PSML/PDML, or ElasticSearch; the dissected output just gets written to the standard output in TShark, and that inherently has no mechanism for rotating files, so the ElasticSearch output has no ring buffer. Are you saying that you're capturing to a raw capture file and using "-T ek" to get dissected output for the same capture, and the raw capture files aren't being treated as a ring buffer?
I was only using the -T ek not using the raw capture output. Would piping it to rotatelogs from Apache be an option to get the ringbuffer behaviour? Or would I need to look into the 2-step process? I wanted to get a near real-time feed of tshark into Elastic, so not sure if that's possible with tshark -T ek. Not sure if vichargrave/espcap would be better suited for this
Thanks for the help