Rework how hostname/SNI info is saved (#1330)

Looking at `struct ndpi_flow_struct` the two bigger fields are
`host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and
`protos.tls_quic.client_requested_server_name[256]`
(for TLS/QUIC SNIs).

This commit aims to reduce `struct ndpi_flow_struct` size, according to
two simple observations:
 1) maximum one of these two fields is used for each flow. So it seems safe
to merge them;
 2) even if hostnames/SNIs might be very long, in practice they are rarely
longer than a fews tens of bytes. So, using a (single) large buffer is a
waste of memory for all kinds of flows. If we need to truncate the name,
we keep the *last* characters, easing domain matching.

Analyzing some real traffic, it seems safe to assume that the vast
majority of hostnames/SNIs is shorter than 80 bytes.

Hostnames/SNIs are always converted to lowercase.

Attention was given so as to be sure that unit-tests outputs are not
affected by this change.

Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the
*first* 64 ones): as a consequence, there were some "Suspicious DGA
domain name" and "TLS Certificate Mismatch" false positives.

3 files changed, 5 insertions, 5 deletions

diff --git a/tests/result/dns_exfiltration.pcap.out b/tests/result/dns_exfiltration.pcap.out
index 726e452b9..245d6cc57 100644
--- a/tests/result/dns_exfiltration.pcap.out
+++ b/tests/result/dns_exfiltration.pcap.out
@@ -4,4 +4,4 @@ DPI Packets (UDP):	2	(2.00 pkts/flow)
 
 DNS	300	73545	1
 
-	1	UDP 192.168.220.56:56373 <-> 192.168.203.167:53 [proto: 5/DNS][ClearText][cat: Network/14][150 pkts/32419 bytes <-> 150 pkts/41126 bytes][Goodput ratio: 81/85][59.99 sec][Host: dnscat.546b03f50000000000a6023ed4df184d6ac5c2628b47714fdee584fed739.5a03b5b1e1aa8f8fdb1bbe8d5e04952141f7d4f82c7e3b06dcc8b87fad7a.19e4d098dc8c618f8d81cfeb02][::][bytes ratio: -0.118 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 1/1 398/397 1035/1015 491/489][Pkt Len c2s/s2c min/avg/max/stddev: 101/148 216/274 300/386 97/97][Risk: ** Suspicious DGA domain name **][Risk Score: 100][PLAIN TEXT (dnscat)][Plen Bins: 0,24,0,23,0,0,0,0,26,26,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	1	UDP 192.168.220.56:56373 <-> 192.168.203.167:53 [proto: 5/DNS][ClearText][cat: Network/14][150 pkts/32419 bytes <-> 150 pkts/41126 bytes][Goodput ratio: 81/85][59.99 sec][Host: e1aa8f8fdb1bbe8d5e04952141f7d4f82c7e3b06dcc8b87fad7a.19e4d098dc8c618f8d81cfeb02][::][bytes ratio: -0.118 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 1/1 398/397 1035/1015 491/489][Pkt Len c2s/s2c min/avg/max/stddev: 101/148 216/274 300/386 97/97][Risk: ** Suspicious DGA domain name **][Risk Score: 100][PLAIN TEXT (dnscat)][Plen Bins: 0,24,0,23,0,0,0,0,26,26,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
diff --git a/tests/result/fuzz-2006-06-26-2594.pcap.out b/tests/result/fuzz-2006-06-26-2594.pcap.out
index 60bd4a672..65cd73735 100644
--- a/tests/result/fuzz-2006-06-26-2594.pcap.out
+++ b/tests/result/fuzz-2006-06-26-2594.pcap.out
@@ -17,7 +17,7 @@ SIP	85	39540	15
 	1	UDP 212.242.33.35:5060 <-> 192.168.1.2:5060 [proto: 100/SIP][ClearText][cat: VoIP/10][23 pkts/11772 bytes <-> 37 pkts/14743 bytes][Goodput ratio: 91/89][1521.43 sec][bytes ratio: -0.112 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 19/227 32597/38366 167478/304738 41340/57147][Pkt Len c2s/s2c min/avg/max/stddev: 344/47 512/398 711/1118 86/358][PLAIN TEXT (SIP/2.0 401 Unauthorized)][Plen Bins: 29,0,0,0,0,0,0,0,0,3,6,0,3,6,8,13,1,0,3,0,1,15,0,0,0,5,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 	2	UDP 192.168.1.2:5060 <-> 200.68.120.81:5060 [proto: 100/SIP][ClearText][cat: VoIP/10][9 pkts/4647 bytes <-> 3 pkts/1944 bytes][Goodput ratio: 92/93][66.58 sec][bytes ratio: 0.410 (Upload)][IAT c2s/s2c min/avg/max/stddev: 507/34556 8170/34556 32608/34556 10578/0][Pkt Len c2s/s2c min/avg/max/stddev: 417/637 516/648 864/656 186/8][PLAIN TEXT (INVITEKsip)][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,59,0,0,0,0,0,0,8,16,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 	3	UDP 192.168.1.2:137 -> 192.168.1.255:137 [proto: 10/NetBIOS][ClearText][cat: System/18][71 pkts/6532 bytes -> 0 pkts/0 bytes][Goodput ratio: 54/0][1527.12 sec][Host: eci_domain][bytes ratio: 1.000 (Upload)][IAT c2s/s2c min/avg/max/stddev: 741/0 20522/0 93225/0 24163/0][Pkt Len c2s/s2c min/avg/max/stddev: 92/0 92/0 92/0 0/0][PLAIN TEXT ( EFEDEJ)][Plen Bins: 0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
-	4	TCP 192.168.1.2:2720 <-> 147.234.1.253:21 [proto: 1/FTP_CONTROL][ClearText][cat: Download/7][11 pkts/624 bytes <-> 14 pkts/1080 bytes][Goodput ratio: 4/27][0.32 sec][Host: ProFTPD][bytes ratio: -0.268 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 24/7 115/18 38/8][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 57/77 70/113 5/19][Risk: ** Unsafe Protocol **][Risk Score: 10][PLAIN TEXT (220 ProFTPD Server In ECI Telec)][Plen Bins: 66,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	4	TCP 192.168.1.2:2720 <-> 147.234.1.253:21 [proto: 1/FTP_CONTROL][ClearText][cat: Download/7][11 pkts/624 bytes <-> 14 pkts/1080 bytes][Goodput ratio: 4/27][0.32 sec][Host: proftpd][bytes ratio: -0.268 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 24/7 115/18 38/8][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 57/77 70/113 5/19][Risk: ** Unsafe Protocol **][Risk Score: 10][PLAIN TEXT (220 ProFTPD Server In ECI Telec)][Plen Bins: 66,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 	5	UDP 192.168.1.2:5060 -> 212.242.33.35:17860 [proto: 100/SIP][ClearText][cat: VoIP/10][1 pkts/1118 bytes -> 0 pkts/0 bytes][Goodput ratio: 96/0][< 1 sec][PLAIN TEXT (INVITE six)][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 	6	UDP 192.168.1.2:30000 -> 212.242.33.36:40392 [proto: 87/RTP][ClearText][cat: Media/1][5 pkts/1070 bytes -> 0 pkts/0 bytes][Goodput ratio: 80/0][0.05 sec][PLAIN TEXT (goxcffj)][Plen Bins: 0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 	7	UDP 192.168.1.2:68 <-> 192.168.1.1:67 [proto: 18/DHCP][ClearText][cat: Network/14][1 pkts/342 bytes <-> 1 pkts/590 bytes][Goodput ratio: 87/93][0.00 sec][Host: d002465][DHCP Fingerprint: 1,15,3,6,44,46,47,31,33,43][DHCP Class Ident: MSFT 5.0][PLAIN TEXT (002465Q)][Plen Bins: 0,0,0,0,0,0,0,0,0,50,0,0,0,0,0,0,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
diff --git a/tests/result/reddit.pcap.out b/tests/result/reddit.pcap.out
index 2becdb7e1..83220c286 100644
--- a/tests/result/reddit.pcap.out
+++ b/tests/result/reddit.pcap.out
@@ -2,10 +2,10 @@ Guessed flow protos:	1
 
 DPI Packets (TCP):	445	(7.42 pkts/flow)
 
-TLS	612	384211	15
+TLS	577	373106	14
 Twitter	863	686585	3
 YouTube	881	966947	3
-Google	618	334683	13
+Google	653	345788	14
 Amazon	100	59185	2
 Reddit	8337	9059073	20
 GoogleServices	271	87487	4
@@ -39,7 +39,7 @@ JA3 Host Stats:
 	22	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:56782 <-> [64:ff9b::68f4:2ac8]:443 [proto: 91.120/TLS.Twitter][Encrypted][cat: SocialNetwork/6][23 pkts/5030 bytes <-> 22 pkts/7292 bytes][Goodput ratio: 61/74][4.33 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.184 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/0 216/228 2512/2545 565/587][Pkt Len c2s/s2c min/avg/max/stddev: 86/86 219/331 854/1474 227/405][TLSv1.2][Client: syndication.twitter.com][JA3C: b32309a26951912be7dba376398abc3b][ServerNames: syndication.twitter.com,syndication.twimg.com,syndication-o.twitter.com,syndication-o.twimg.com,cdn.syndication.twitter.com,cdn.syndication.twimg.com][JA3S: 8d2a028aa94425f76ced7826b1f39039][Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 High Assurance Server CA][Subject: C=US, ST=California, L=San Francisco, O=Twitter, Inc., OU=lon3, CN=syndication.twitter.com][Certificate SHA-1: 09:D3:FE:9A:3E:39:A7:E2:90:5B:C9:1F:3B:7D:CE:7C:7E:08:1C:6F][Chrome][Validity: 2020-01-02 00:00:00 - 2020-12-24 12:00:00][Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,15,0,20,0,0,4,4,4,15,0,0,4,0,4,0,4,0,0,0,4,0,4,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9,0,0,0,0]
 	23	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:56640 <-> [64:ff9b::9765:798c]:443 [proto: 91.205/TLS.Reddit][Encrypted][cat: SocialNetwork/6][23 pkts/3696 bytes <-> 22 pkts/8527 bytes][Goodput ratio: 46/78][0.57 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.395 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 27/30 307/307 76/75][Pkt Len c2s/s2c min/avg/max/stddev: 86/86 161/388 603/1134 157/388][TLSv1.2][Client: gateway.reddit.com][JA3C: b32309a26951912be7dba376398abc3b][ServerNames: reddit.com,*.reddit.com][JA3S: 16c0b3e6a7b8173c16d944cfeaeee9cf][Issuer: C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA][Subject: C=US, ST=California, L=San Francisco, O=Reddit Inc., CN=*.reddit.com][Certificate SHA-1: DB:E9:D5:FE:EB:EF:68:34:55:FD:62:BA:C9:BB:04:D4:E3:22:18:81][Chrome][Validity: 2020-08-26 00:00:00 - 2021-02-22 12:00:00][Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,14,20,4,4,0,0,0,4,0,0,4,9,0,0,9,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 	24	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:46646 <-> [64:ff9b::345f:7ca5]:443 [proto: 91.178/TLS.Amazon][Encrypted][cat: Web/5][14 pkts/3201 bytes <-> 13 pkts/8450 bytes][Goodput ratio: 62/87][0.22 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.451 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 15/17 60/42 22/16][Pkt Len c2s/s2c min/avg/max/stddev: 86/86 229/650 762/1446 254/571][TLSv1.2][Client: aax-eu.amazon-adsystem.com][JA3C: b32309a26951912be7dba376398abc3b][ServerNames: aax-eu.amazon-adsystem.com,aax.amazon-adsystem.com,aax-cpm.amazon-adsystem.com,aax-dtb-web.amazon-adsystem.com][JA3S: 49b45fc1ab090aa3a159778313fc9b9e][Issuer: C=US, O=Amazon, OU=Server CA 1B, CN=Amazon][Subject: CN=aax-eu.amazon-adsystem.com][Certificate SHA-1: 5D:18:8E:CB:B7:91:5C:79:26:B5:08:49:FF:2C:24:D8:06:54:91:8B][Chrome][Validity: 2020-06-15 00:00:00 - 2021-06-15 12:00:00][Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,0,0,8,0,0,0,0,8,0,0,0,8,0,0,8,8,0,0,0,8,16,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,34,0,0,0,0,0]
-	25	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:59624 <-> [2a00:1450:4007:80b::2001]:443 [proto: 91/TLS][Encrypted][cat: Web/5][18 pkts/2649 bytes <-> 17 pkts/8456 bytes][Goodput ratio: 41/83][0.15 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.523 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 7/5 34/33 12/10][Pkt Len c2s/s2c min/avg/max/stddev: 86/86 147/497 603/1294 137/490][Risk: ** Suspicious DGA domain name **][Risk Score: 100][TLSv1.3][Client: 8a755a3fef0b189d8ab5b0d10758f68a.safeframe.googlesyndication.co][JA3C: b32309a26951912be7dba376398abc3b][JA3S: eb1d94daa7e0344597e756a1fb6e7054][Chrome][Cipher: TLS_AES_128_GCM_SHA256][Plen Bins: 12,12,12,0,0,0,0,0,0,0,6,0,6,0,6,0,6,0,6,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,25,0,0,0,0,0,0,0,0,0,0]
+	25	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:59624 <-> [2a00:1450:4007:80b::2001]:443 [proto: 91.126/TLS.Google][Encrypted][cat: Web/5][18 pkts/2649 bytes <-> 17 pkts/8456 bytes][Goodput ratio: 41/83][0.15 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.523 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 7/5 34/33 12/10][Pkt Len c2s/s2c min/avg/max/stddev: 86/86 147/497 603/1294 137/490][TLSv1.3][Client: 8a755a3fef0b189d8ab5b0d10758f68a.safeframe.googlesyndication.com][JA3C: b32309a26951912be7dba376398abc3b][JA3S: eb1d94daa7e0344597e756a1fb6e7054][Chrome][Cipher: TLS_AES_128_GCM_SHA256][Plen Bins: 12,12,12,0,0,0,0,0,0,0,6,0,6,0,6,0,6,0,6,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,25,0,0,0,0,0,0,0,0,0,0]
 	26	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:46808 <-> [2a00:1450:4007:808::2001]:443 [proto: 91/TLS][Encrypted][cat: Web/5][15 pkts/1843 bytes <-> 13 pkts/9101 bytes][Goodput ratio: 32/88][0.12 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.663 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 5/5 32/32 11/10][Pkt Len c2s/s2c min/avg/max/stddev: 74/86 123/700 603/1294 129/569][TLSv1.3][Client: cdn.ampproject.org][JA3C: b32309a26951912be7dba376398abc3b][JA3S: eb1d94daa7e0344597e756a1fb6e7054][Chrome][Cipher: TLS_AES_128_GCM_SHA256][Plen Bins: 0,0,10,0,10,0,0,0,0,0,0,0,0,0,0,0,10,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,60,0,0,0,0,0,0,0,0,0,0]
 	27	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:46810 <-> [2a00:1450:4007:808::2001]:443 [proto: 91/TLS][Encrypted][cat: Web/5][15 pkts/1843 bytes <-> 13 pkts/9100 bytes][Goodput ratio: 32/88][0.12 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.663 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 5/6 31/34 11/11][Pkt Len c2s/s2c min/avg/max/stddev: 74/86 123/700 603/1294 129/569][TLSv1.3][Client: cdn.ampproject.org][JA3C: b32309a26951912be7dba376398abc3b][JA3S: eb1d94daa7e0344597e756a1fb6e7054][Chrome][Cipher: TLS_AES_128_GCM_SHA256][Plen Bins: 0,0,10,0,10,0,0,0,0,0,0,0,0,0,0,0,10,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,60,0,0,0,0,0,0,0,0,0,0]
 	28	TCP [2a01:cb01:2049:8b07:991d:ec85:28df:f629]:46814 <-> [2a00:1450:4007:808::2001]:443 [proto: 91/TLS][Encrypted][cat: Web/5][14 pkts/1769 bytes <-> 13 pkts/9102 bytes][Goodput ratio: 33/88][0.12 sec][ALPN: h2;http/1.1][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.675 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 10/5 43/36 15/11][Pkt Len c2s/s2c min/avg/max/stddev: 74/86 126/700 603/1294 133/569][TLSv1.3][Client: cdn.ampproject.org][JA3C: b32309a26951912be7dba376398abc3b][JA3S: eb1d94daa7e0344597e756a1fb6e7054][Chrome][Cipher: TLS_AES_128_GCM_SHA256][Plen Bins: 0,0,10,0,10,0,0,0,0,0,0,0,0,0,0,0,10,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,60,0,0,0,0,0,0,0,0,0,0]