ahoсorasick. Code review. Part 2. (#1236)

Simplified the process of adding lines to AC_AUTOMATA_t.
Use the ndpi_string_to_automa() function to add patterns with domain names.
For other cases can use ndpi_add_string_value_to_automa().

ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding
and compare data in a case insensitive manner. For mandatory pattern comparison
from the end of the line, the "ac_pattern.rep.at_end=1" flag is used.
This eliminated unnecessary conversions to lowercase and adding "$" for
end-of-line matching in domain name patterns.

ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified.
ac_domain_match_handler() looks for the template with the highest domain level.
For special cases it is possible to manually specify the domain level.
Added test for checking ambiguous domain names like:
 - short.weixin.qq.com is QQ, not Wechat
 - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook

If you specify a NULL handler when creating the AC_AUTOMATA_t structure,
then a pattern with the maximum length that satisfies the search conditions will be found
(exact match, from the beginning of the string, from the end of the string, or a substring).

Added debugging for ac_automata_search.
To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and
enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG).
The search will display "name" and a list of matching patterns.
Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates
and which templates were found.

The ac_automata_dump() prototype has been changed. Now it outputs data to a file.
If it is specified as NULL, then the output will be directed to stdout.
If you need to get data as a string, then use open_memstream().

Added the ability to run individual tests via the do.sh script

3 files changed, 25 insertions, 1 deletions

diff --git a/tests/do.sh.in b/tests/do.sh.in
index 85ef1ff63..546166b57 100755
--- a/tests/do.sh.in
+++ b/tests/do.sh.in
@@ -36,6 +36,11 @@ build_results() {
 
 check_results() {
 	for f in $PCAPS; do
+	    if [ -n "$*" ]; then
+	    	SKIP_PCAP=1
+		for i in $* ; do [ "$f" = "$i" ] && SKIP_PCAP=0 && break ; done
+		[ $SKIP_PCAP = 1 ] && continue
+	    fi
 	    SKIP_PCAP=0
 	    if [ $GCRYPT_ENABLED -eq 0 ]; then
 	      for g in $GCRYPT_PCAPS; do
@@ -73,6 +78,6 @@ if [ $FUZZY_TESTING_ENABLED -eq 1 ]; then
     fuzzy_testing
 fi
 build_results
-check_results
+check_results $*
 
 exit $RC
diff --git a/tests/pcap/dns_ambiguous_names.pcap b/tests/pcap/dns_ambiguous_names.pcap
new file mode 100644
index 000000000..87ef756e2
--- /dev/null
+++ b/tests/pcap/dns_ambiguous_names.pcap
diff --git a/tests/result/dns_ambiguous_names.pcap.out b/tests/result/dns_ambiguous_names.pcap.out
new file mode 100644
index 000000000..0378634bb
--- /dev/null
+++ b/tests/result/dns_ambiguous_names.pcap.out
@@ -0,0 +1,19 @@
+QQ	2	212	1
+Google	2	208	1
+Instagram	2	220	1
+PlayStore	2	474	1
+ApplePush	2	414	1
+GoogleServices	2	235	1
+Teams	6	790	3
+AppleSiri	2	234	1
+
+	1	UDP 10.200.2.11:57632 <-> 8.8.8.8:53 [proto: 5.228/DNS.PlayStore][cat: SoftwareUpdate/19][1 pkts/97 bytes <-> 1 pkts/377 bytes][Goodput ratio: 56/89][0.03 sec][Host: android.clients.google.com][108.177.14.101][PLAIN TEXT (android)][Plen Bins: 0,50,0,0,0,0,0,0,0,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	2	UDP 10.200.2.11:48375 <-> 8.8.8.8:53 [proto: 5.238/DNS.ApplePush][cat: Cloud/13][1 pkts/96 bytes <-> 1 pkts/318 bytes][Goodput ratio: 56/87][0.04 sec][Host: 41-courier.push.apple.com][17.57.146.139][PLAIN TEXT (courier)][Plen Bins: 0,50,0,0,0,0,0,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	3	UDP 10.200.2.11:57051 <-> 8.8.8.8:53 [proto: 5.250/DNS.Teams][cat: Collaborative/15][1 pkts/90 bytes <-> 1 pkts/221 bytes][Goodput ratio: 53/81][0.03 sec][Host: api.teams.skype.com][52.113.194.131][PLAIN TEXT (trafficmanager)][Plen Bins: 0,50,0,0,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	4	UDP 10.200.2.11:42790 <-> 8.8.8.8:53 [proto: 5.250/DNS.Teams][cat: Collaborative/15][1 pkts/92 bytes <-> 1 pkts/166 bytes][Goodput ratio: 54/74][0.08 sec][Host: _.teams.microsoft.com][::][PLAIN TEXT (microsoft)][Plen Bins: 0,50,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	5	UDP 10.200.2.11:46134 <-> 8.8.8.8:53 [proto: 5.239/DNS.GoogleServices][cat: Web/5][1 pkts/92 bytes <-> 1 pkts/143 bytes][Goodput ratio: 54/70][0.03 sec][Host: alt2-mtalk.google.com][173.194.202.188][PLAIN TEXT (google)][Plen Bins: 0,50,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	6	UDP 10.200.2.11:52541 <-> 8.8.8.8:53 [proto: 5.254/DNS.AppleSiri][cat: VirtualAssistant/32][1 pkts/88 bytes <-> 1 pkts/146 bytes][Goodput ratio: 52/71][0.06 sec][Host: guzzoni.apple.com][17.130.21.5][PLAIN TEXT (guzzoni)][Plen Bins: 0,50,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	7	UDP 10.200.2.11:57290 <-> 8.8.8.8:53 [proto: 5.250/DNS.Teams][cat: Collaborative/15][1 pkts/86 bytes <-> 1 pkts/135 bytes][Goodput ratio: 51/68][0.03 sec][Host: teams.skype.com][13.107.3.128][PLAIN TEXT (msedge)][Plen Bins: 0,50,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	8	UDP 10.200.2.11:44883 <-> 8.8.8.8:53 [proto: 5.211/DNS.Instagram][cat: SocialNetwork/6][1 pkts/102 bytes <-> 1 pkts/118 bytes][Goodput ratio: 58/64][0.04 sec][Host: instagram.faae1-1.fna.fbcdn.net][41.220.158.96][PLAIN TEXT (instagram)][Plen Bins: 0,50,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	9	UDP 10.200.2.11:53951 <-> 8.8.8.8:53 [proto: 5.48/DNS.QQ][cat: Chat/9][1 pkts/90 bytes <-> 1 pkts/122 bytes][Goodput ratio: 53/65][0.34 sec][Host: short.weixin.qq.com][203.205.254.77][PLAIN TEXT (weixin)][Plen Bins: 0,50,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
+	10	UDP 10.200.2.11:44198 <-> 8.8.8.8:53 [proto: 5.126/DNS.Google][cat: Web/5][1 pkts/96 bytes <-> 1 pkts/112 bytes][Goodput ratio: 56/62][0.03 sec][Host: wide-youtube.l.google.com][64.233.164.198][PLAIN TEXT (youtube)][Plen Bins: 0,50,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]