New API for library configuration

This is the first step into providing (more) configuration options in nDPI.

The idea is to have a simple way to configure (most of) nDPI: only one
function (`ndpi_set_config()`) to set any configuration parameters
(in the present or on in the future) and we try to keep this function
prototype as agnostic as possible.

You can configure the library:
* via API, using `ndpi_set_config()`
* via a configuration file, in a text format

This way, anytime we need to add a new configuration parameter:
* we don't need to add two public functions (a getter and a setter)
* we don't break API/ABI compatibility of the library; even changing
the parameter type (from integer to a list of integer, for example)
doesn't break the compatibility.

The complete list of configuration options is provided in
`doc/configuration_parameters.md`.

As a first example, two configuration knobs are provided:
* the ability to enable/disable the extraction of the sha1 fingerprint of
the TLS certificates.
* the upper limit on the number of packets per flow that will be subject
to inspection

8 files changed, 72 insertions, 1 deletions

diff --git a/tests/cfgs/disable_metadata/config.txt b/tests/cfgs/disable_metadata/config.txt
new file mode 100644
index 000000000..1e5e30bf8
--- /dev/null
+++ b/tests/cfgs/disable_metadata/config.txt
@@ -0,0 +1 @@
+--cfg=tls,metadata.sha1_fingerprint.enable,0
diff --git a/tests/cfgs/disable_metadata/pcap/tls_verylong_certificate.pcap b/tests/cfgs/disable_metadata/pcap/tls_verylong_certificate.pcap
new file mode 120000
index 000000000..2f722f28e
--- /dev/null
+++ b/tests/cfgs/disable_metadata/pcap/tls_verylong_certificate.pcap
@@ -0,0 +1 @@
+../../default/pcap/tls_verylong_certificate.pcap
\ No newline at end of file
diff --git a/tests/cfgs/disable_metadata/result/tls_verylong_certificate.pcap.out b/tests/cfgs/disable_metadata/result/tls_verylong_certificate.pcap.out
new file mode 100644
index 000000000..049a4cec0
--- /dev/null
+++ b/tests/cfgs/disable_metadata/result/tls_verylong_certificate.pcap.out
@@ -0,0 +1,33 @@
+DPI Packets (TCP):	11	(11.00 pkts/flow)
+Confidence DPI              : 1 (flows)
+Num dissector calls: 1 (1.00 diss/flow)
+LRU cache ookla:      0/0/0 (insert/search/found)
+LRU cache bittorrent: 0/0/0 (insert/search/found)
+LRU cache zoom:       0/0/0 (insert/search/found)
+LRU cache stun:       0/0/0 (insert/search/found)
+LRU cache tls_cert:   0/0/0 (insert/search/found)
+LRU cache mining:     0/0/0 (insert/search/found)
+LRU cache msteams:    0/0/0 (insert/search/found)
+LRU cache stun_zoom:  0/0/0 (insert/search/found)
+Automa host:          1/1 (search/found)
+Automa domain:        1/0 (search/found)
+Automa tls cert:      0/0 (search/found)
+Automa risk mask:     0/0 (search/found)
+Automa common alpns:  1/1 (search/found)
+Patricia risk mask:   0/0 (search/found)
+Patricia risk mask IPv6: 0/0 (search/found)
+Patricia risk:        0/0 (search/found)
+Patricia risk IPv6:   0/0 (search/found)
+Patricia protocols:   2/0 (search/found)
+Patricia protocols IPv6: 0/0 (search/found)
+
+Cybersec	48	22229	1
+
+Safe                            48 22229         1            
+
+JA3 Host Stats: 
+		 IP Address                  	 # JA3C     
+	1	 192.168.1.160            	 1      
+
+
+	1	TCP 192.168.1.160:54804 <-> 151.101.66.49:443 [proto: 91.283/TLS.Cybersec][IP: 0/Unknown][Encrypted][Confidence: DPI][DPI packets: 11][cat: Cybersecurity/33][24 pkts/2404 bytes <-> 24 pkts/19825 bytes][Goodput ratio: 35/92][0.09 sec][Hostname/SNI: feodotracker.abuse.ch][(Advertised) ALPNs: http/1.1][(Negotiated) ALPN: http/1.1][bytes ratio: -0.784 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 4/4 15/21 5/7][Pkt Len c2s/s2c min/avg/max/stddev: 54/66 100/826 583/1434 109/662][TLSv1.2][JA3C: 2a26b1a62e40d25d4de3babc9d532f30][JA4: t12d6707ht_2955a3196ffa_b1760ac0ffd7][ServerNames: p2.shared.global.fastly.net,*.12wbt.com,*.2bleacherreport.com,*.3bleacherreport.com,*.4bleacherreport.com,*.8bleacherreport.com,*.abuse.ch,*.acdn-it.ps-pantheon.com,*.cdn.livingmap.com,*.content.plastiq.com,*.dimensions.ai,*.dollarshaveclub.co.uk,*.dollarshaveclub.com,*.dontpayfull.com,*.ebisubook.com,*.foreignaffairs.com,*.fs.jibjab.com,*.fs.unitprints.com,*.ggleap.com,*.goodeggs.com,*.huevosbuenos.com,*.indy.myomnigon.com,*.jwatch.org,*.kingsfordcharcoal.com.au,*.lancenters.com,*.madebywe.com,*.minirodini.com,*.modcloth.net,*.orionlabs.io,*.ps-pantheon.com,*.scodle.com,*.steelseries.com,*.theforeman.org,*.uploads.eversign.com,*.uploads.schoox.com,*.vts.com,*.x.stg1.ebisubook.com,*.yang2020.com,12wbt.com,2bleacherreport.com,3bleacherreport.com,4bleacherreport.com,8bleacherreport.com,abuse.ch,brita.com,cdn.fwupd.org,cdn.livingmap.com,cdn.seated.com,cdn.skillacademy.com,clinicaloptions.com,clorox.com,content-preprod.beaverbrooksweb2.co.uk,content.beaverbrooks.co.uk,content.plastiq.com,coolmathgames.com,copterroyale.coolmathgames.com,d8-dev.coolmathgames.com,deflyio.coolmathgames.com,delivery-api.evadacms.com,dimensions.ai,dollarshaveclub.co.uk,dollarshaveclub.com,dontpayfull.com,eluniverso.com,email.amg-group.co,email.tekoforlife.co.uk,feedmarket.fr,freshstep.com,ggleap.com,goodeggs.com,heap.io,huevosbuenos.com,identity.linuxfoundation.org,joebiden.com,jwatch.org,kingsford.co.nz,kingsfordcharcoal.com.au,lancenters.com,lists.linuxfoundation.org,m-stage.coolmathgames.com,m.coolmathgames.com,madebywe.com,minirodini.com,modcloth.net,orionlabs.io,puritanmedproducts.com,reviews.org,rg-video-staging.ruangguru.com,rg-video.ruangguru.com,ruangguru.com,scodle.com,stage.coolmathgames.com,staging.appblade.com,steelseries.com,stg.platform.eluniverso.com,test.brita.com,test.heap.io,test.joebiden.com,test.ruangguru.com,theforeman.org,video-cdn.quipper.com,videos.calcworkshop.com,vts.com,www.101network.com,www.autos101.com,www.brita.com,www.clorox.com,www.collider.com,www.coolmathgames.com,www.eluniverso.com,www.flinto.com,www.freshstep.com,www.heap.io,www.holagente.com,www.icsydney.com.au,www.joebiden.com,www.kingsford.co.nz,www.mrnatty.com,www.myjewellerystory.com.au,www.myjs.com,www.netacea.com,www.parenting101.com,www.puritanmedproducts.com,www.reviews.org,www.sba.sa,www.shashatcom.sa,www.uat.ontariocolleges.ca,www.vacation101.com,www.walterspeople.co.uk,www.westwayelectricsupply.com][JA3S: ae53107a2e47ea20c72ac44821a728bf][Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign CloudSSL CA - SHA256 - G3][Subject: C=US, ST=California, L=San Francisco, O=Fastly, Inc., CN=p2.shared.global.fastly.net][Firefox][Validity: 2019-11-19 01:31:22 - 2020-08-29 17:19:32][Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256][Plen Bins: 12,16,0,4,0,4,4,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,55,0,0,0,0,0]
diff --git a/tests/cfgs/packets_limit_per_flow/config.txt b/tests/cfgs/packets_limit_per_flow/config.txt
new file mode 100644
index 000000000..df3eed5d2
--- /dev/null
+++ b/tests/cfgs/packets_limit_per_flow/config.txt
@@ -0,0 +1 @@
+--cfg=packets_limit_per_flow,4
diff --git a/tests/cfgs/packets_limit_per_flow/pcap/tls_verylong_certificate.pcap b/tests/cfgs/packets_limit_per_flow/pcap/tls_verylong_certificate.pcap
new file mode 120000
index 000000000..2f722f28e
--- /dev/null
+++ b/tests/cfgs/packets_limit_per_flow/pcap/tls_verylong_certificate.pcap
@@ -0,0 +1 @@
+../../default/pcap/tls_verylong_certificate.pcap
\ No newline at end of file
diff --git a/tests/cfgs/packets_limit_per_flow/result/tls_verylong_certificate.pcap.out b/tests/cfgs/packets_limit_per_flow/result/tls_verylong_certificate.pcap.out
new file mode 100644
index 000000000..19f147637
--- /dev/null
+++ b/tests/cfgs/packets_limit_per_flow/result/tls_verylong_certificate.pcap.out
@@ -0,0 +1,33 @@
+DPI Packets (TCP):	5	(5.00 pkts/flow)
+Confidence DPI              : 1 (flows)
+Num dissector calls: 1 (1.00 diss/flow)
+LRU cache ookla:      0/0/0 (insert/search/found)
+LRU cache bittorrent: 0/0/0 (insert/search/found)
+LRU cache zoom:       0/0/0 (insert/search/found)
+LRU cache stun:       0/0/0 (insert/search/found)
+LRU cache tls_cert:   0/0/0 (insert/search/found)
+LRU cache mining:     0/0/0 (insert/search/found)
+LRU cache msteams:    0/0/0 (insert/search/found)
+LRU cache stun_zoom:  0/0/0 (insert/search/found)
+Automa host:          1/1 (search/found)
+Automa domain:        1/0 (search/found)
+Automa tls cert:      0/0 (search/found)
+Automa risk mask:     0/0 (search/found)
+Automa common alpns:  1/1 (search/found)
+Patricia risk mask:   0/0 (search/found)
+Patricia risk mask IPv6: 0/0 (search/found)
+Patricia risk:        0/0 (search/found)
+Patricia risk IPv6:   0/0 (search/found)
+Patricia protocols:   2/0 (search/found)
+Patricia protocols IPv6: 0/0 (search/found)
+
+Cybersec	48	22229	1
+
+Safe                            48 22229         1            
+
+JA3 Host Stats: 
+		 IP Address                  	 # JA3C     
+	1	 192.168.1.160            	 1      
+
+
+	1	TCP 192.168.1.160:54804 <-> 151.101.66.49:443 [proto: 91.283/TLS.Cybersec][IP: 0/Unknown][Encrypted][Confidence: DPI][DPI packets: 5][cat: Cybersecurity/33][24 pkts/2404 bytes <-> 24 pkts/19825 bytes][Goodput ratio: 35/92][0.09 sec][Hostname/SNI: feodotracker.abuse.ch][(Advertised) ALPNs: http/1.1][bytes ratio: -0.784 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 4/4 15/21 5/7][Pkt Len c2s/s2c min/avg/max/stddev: 54/66 100/826 583/1434 109/662][TLSv1.2][JA3C: 2a26b1a62e40d25d4de3babc9d532f30][JA4: t12d6707ht_2955a3196ffa_b1760ac0ffd7][Firefox][Plen Bins: 12,16,0,4,0,4,4,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,55,0,0,0,0,0]
diff --git a/tests/do.sh.in b/tests/do.sh.in
index b71618f9b..e8a08f035 100755
--- a/tests/do.sh.in
+++ b/tests/do.sh.in
@@ -30,7 +30,7 @@ PCRE2_ENABLED=@PCRE2_ENABLED@
 PCRE_PCAPS="WebattackRCE.pcap"
 NBPF_ENABLED=@NBPF_ENABLED@
 NBPF_PCAPS="h323-overflow.pcap"
-READER="${CMD_PREFIX} ../../../example/ndpiReader${EXE_SUFFIX} -A -p ../../../example/protos.txt -c ../../../example/categories.txt -r ../../../example/risky_domains.txt -j ../../../example/ja3_fingerprints.csv -S ../../../example/sha1_fingerprints.csv -G ../../../lists -q -K JSON -k /dev/null -t -v 2"
+READER="${CMD_PREFIX} ../../../example/ndpiReader${EXE_SUFFIX} --cfg=filename.config,../../../example/config.txt -A -p ../../../example/protos.txt -c ../../../example/categories.txt -r ../../../example/risky_domains.txt -j ../../../example/ja3_fingerprints.csv -S ../../../example/sha1_fingerprints.csv -G ../../../lists -q -K JSON -k /dev/null -t -v 2"
 
 RC=0
 
diff --git a/tests/ossfuzz.sh b/tests/ossfuzz.sh
index e2adb3bf8..d41454b2d 100644
--- a/tests/ossfuzz.sh
+++ b/tests/ossfuzz.sh
@@ -59,6 +59,7 @@ cp example/categories.txt $OUT/
 cp example/risky_domains.txt $OUT/
 cp example/ja3_fingerprints.csv $OUT/
 cp example/sha1_fingerprints.csv $OUT/
+cp example/config.txt $OUT/
 cp fuzz/ipv4_addresses.txt $OUT/
 cp fuzz/bd_param.txt $OUT/
 cp fuzz/splt_param.txt $OUT/