From 36abf06c6f59b66bde48e7b3028b4823ecc6ed85 Mon Sep 17 00:00:00 2001 From: Luca Deri Date: Tue, 29 Aug 2023 17:34:04 +0200 Subject: Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic approach for handling Internet domain names. For switching back to Aho-Corasick it is necessary to edit ndpi-typedefs.h and uncomment the line // #define USE_LEGACY_AHO_CORASICK [1] With Aho-Corasick $ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory nDPI Memory statistics: nDPI Memory (once): 37.34 KB Flow Memory (per flow): 960 B Actual Memory: 33.09 MB Peak Memory: 33.09 MB [2] With the new algorithm $ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory nDPI Memory statistics: nDPI Memory (once): 37.31 KB Flow Memory (per flow): 960 B Actual Memory: 7.42 MB Peak Memory: 7.42 MB In essence from ~33 MB to ~7 MB This new algorithm will enable larger lists to be loaded (e.g. top 1M domans https://s3-us-west-1.amazonaws.com/umbrella-static/index.html) In ./lists there are file names that are named as _.list With -G ndpiReader can load all of them at startup --- tests/cfgs/default/pcap/bets.pcapng | Bin 0 -> 10740 bytes tests/cfgs/default/result/bets.pcapng.out | 30 ++++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 tests/cfgs/default/pcap/bets.pcapng create mode 100644 tests/cfgs/default/result/bets.pcapng.out (limited to 'tests') diff --git a/tests/cfgs/default/pcap/bets.pcapng b/tests/cfgs/default/pcap/bets.pcapng new file mode 100644 index 000000000..090cf6f22 Binary files /dev/null and b/tests/cfgs/default/pcap/bets.pcapng differ diff --git a/tests/cfgs/default/result/bets.pcapng.out b/tests/cfgs/default/result/bets.pcapng.out new file mode 100644 index 000000000..c17db6426 --- /dev/null +++ b/tests/cfgs/default/result/bets.pcapng.out @@ -0,0 +1,30 @@ +Guessed flow protos: 0 + +DPI Packets (TCP): 6 (6.00 pkts/flow) +Confidence DPI : 1 (flows) +Num dissector calls: 1 (1.00 diss/flow) +LRU cache ookla: 0/0/0 (insert/search/found) +LRU cache bittorrent: 0/0/0 (insert/search/found) +LRU cache zoom: 0/0/0 (insert/search/found) +LRU cache stun: 0/0/0 (insert/search/found) +LRU cache tls_cert: 0/2/0 (insert/search/found) +LRU cache mining: 0/0/0 (insert/search/found) +LRU cache msteams: 0/0/0 (insert/search/found) +LRU cache stun_zoom: 0/0/0 (insert/search/found) +Automa host: 1/0 (search/found) +Automa domain: 1/0 (search/found) +Automa tls cert: 0/0 (search/found) +Automa risk mask: 0/0 (search/found) +Automa common alpns: 2/2 (search/found) +Patricia risk mask: 0/0 (search/found) +Patricia risk: 0/0 (search/found) +Patricia protocols: 1/1 (search/found) + +TLS 33 9228 1 + +JA3 Host Stats: + IP Address # JA3C + 1 192.168.10.2 1 + + + 1 TCP 192.168.10.2:60099 <-> 13.224.103.22:443 [proto: 91/TLS][IP: 265/AmazonAWS][Encrypted][Confidence: DPI][DPI packets: 6][cat: Gambling/107][17 pkts/1469 bytes <-> 16 pkts/7759 bytes][Goodput ratio: 39/89][0.19 sec][Hostname/SNI: www.1084bets10.com][(Advertised) ALPNs: h2;http/1.1][TLS Supported Versions: TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.682 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 7/9 46/45 16/17][Pkt Len c2s/s2c min/avg/max/stddev: 52/52 86/485 380/1420 78/609][TLSv1.3][JA3C: 375c6162a492dfbf2795909110ce8424][JA3S: f4febc55ea12b31ae17cfb7e614afda8][Firefox][Cipher: TLS_AES_128_GCM_SHA256][Plen Bins: 14,14,21,0,7,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,28,0,0,0,0,0] -- cgit v1.2.3