aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAge
...
* Line app support (#1759)sharonenoch2022-10-01
| | | | | | | | | | | | | | | | | | | * Standard support for LINE app * Added test pcap for LINE app * make check result for LINE app * Make check success as 1kxun has LINE packets * Added the ASN inc file for LINE * Removed extra lines as its effecting make check * Editing the SNI required a new pcap output file for TLS.Line format * Run Configure with --with-pcre --with-maxminddb to enable the generation of h323-overflow.pcap.out Co-authored-by: Sharon Enoch <sharone@amzetta.com>
* TLS/DTLS: we process certificate for UDP flows, tooNardi Ivan2022-09-30
| | | | | | Note that current code access `certificate_processed` state even before setting the protocol classification, so this piece of information can't be saved in `flow->protos` union.
* Fixed invalid unidirectional traffic alert for unidirectional protocols ↵Luca2022-09-29
| | | | (e.g. sFlow)
* Remove unused codeNardi Ivan2022-09-28
| | | | | | | | | LRU callbacks have been added in 460ff3c7a, but they have never been used and they have never been extended to the other LRU caches. `ndpi_search_tcp_or_udp()` basically returns the classification by port/ip of the flow; calling it from the dissector is useless. The same for TOR detection: ips are checked in the generic code
* Fix value of `ndpi_protocol->protocol_by_ip`Nardi Ivan2022-09-25
| | | | Fix: a7c2734b
* DNS: change category of DNS flowsNardi Ivan2022-09-25
| | | | | | DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category, regardless of the subprotocol (if any). Follow-up of 83de3e47
* Changed some protocol categories to align with others.Toni Uhlig2022-09-23
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Sizes of LRU caches are now configurableNardi Ivan2022-09-23
| | | | | | | | 0 as size value disable the cache. The diffs in unit tests are due to the fact that some lookups are performed before the first insert: before this change these lookups weren't counted because the cache was not yet initialized, now they are.
* DNS: change category of DNS flowsNardi Ivan2022-09-22
| | | | | DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category, regardless of the subprotocol (if any).
* Extend content match listsNardi Ivan2022-09-22
|
* NAT-PMP: fix metadata extractionNardi Ivan2022-09-21
|
* Improved NATPMP dissection. (#1745)Toni2022-09-21
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Build ndpiReader and run regression tests.Toni Uhlig2022-09-21
| | | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Moved `NDPI_API_VERSION` to `src/include/ndpi_define.h.in` where it belongs.Toni Uhlig2022-09-21
| | | | | | | That way, we can make `src/include/ndpi_api.h` independent of autoconf. MSBuild will like that. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add proj and sln for compile of dynamic x64 lib under Visual Studio 2019.mark andrews2022-09-21
| | | | | | * add CI support via MSBuild Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix CI after nBPF integration (#1746)Ivan Nardi2022-09-21
| | | Add one CI job testing nBPF
* Copilation fixLuca Deri2022-09-21
|
* Finalized nBPF support. You can now define custom protocols such asLuca Deri2022-09-21
| | | | | | | | | | (see exaple/protos.txt) nbpf:"host 192.168.1.1 and port 80"@HomeRouter In order to have nBPF support, you need to compile nDPI with it. Just download https://github.com/ntop/PF_RING in the same directory where you have downloaded nDPI and compile PF_RING/userland/nbpf
* Remove classification "by-ip" from protocol stack (#1743)Ivan Nardi2022-09-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Basically: * "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is NEVER returned in the protocol stack (i.e. `flow->detected_protocol_stack[]`); * if the application is interested into such information, it can access `ndpi_protocol->protocol_by_ip` itself. There are mainly 4 points in the code that set the "classification by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/ `ndpi_detection_giveup()` functions and the HTTP/STUN dissectors. In the unit tests output, a print about `ndpi_protocol->protocol_by_ip` has been added for each flow: the huge diff of this commit is mainly due to that. Strictly speaking, this change is NOT an API/ABI breakage, but there are important differences in the classification results. For examples: * TLS flows without the initial handshake (or without a matching SNI/certificate) are simply classified as `TLS`; * similar for HTTP or QUIC flows; * DNS flows without a matching request domain are simply classified as `DNS`; we don't have `DNS/Google` anymore just because the server is 8.8.8.8 (that was an outrageous behaviour...); * flows previusoly classified only "by-ip" are now classified as `NDPI_PROTOCOL_UNKNOWN`. See #1425 for other examples of why adding the "classification by-ip" in the protocol stack is a bad idea. Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e. `ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for IPv6 flows. Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed. Close #1687
* Fix warnings on OpenWrt CCs, caused by the memcpy builtin. (#1741)Toni2022-09-18
| | | | | | | | | | | | | | | | | | | In file included from ../include/ndpi_includes.h:31, from ../include/ndpi_main.h:27, from ../include/ndpi_api.h:28, from protocols/quic.c:27: In function 'memcpy', inlined from 'tls13_hkdf_expand_label_context' at protocols/quic.c:473:5, inlined from 'tls13_hkdf_expand_label' at protocols/quic.c:498:10, inlined from 'quic_hkdf_expand_label.constprop' at protocols/quic.c:512:6: /home/build/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.3.0_musl/include/fortify/string.h:53:16: error: argument 2 null where non-null expected [-Werror=nonnull] 53 | return __builtin_memcpy(__od, __os, __n); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ protocols/quic.c: In function 'quic_hkdf_expand_label.constprop': /home/build/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.3.0_musl/include/fortify/string.h:53:16: note: in a call to built-in function '__builtin_memcpy' Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added MAX_NBPF_CUSTOM_PROTO defineLuca Deri2022-09-17
|
* Initial work towards custom BPF protocol definitionLuca2022-09-16
|
* Add CryNetwork dissector.Toni Uhlig2022-09-16
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add Syncthing dissector.Toni Uhlig2022-09-16
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add NATPMP dissector. (#1738)Toni2022-09-16
| | | | | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Remove a case of guessed sub-classificationNardi Ivan2022-09-14
| | | | | | | | | | | | | | | | | | This code is triggered only for "unknown" flows with a valid sni/hostname. Why in that case the guessed classification should be something like `DNS/Subprotocol_depending_on_hostname`? Why DNS as master and not HTTP or TLS or QUIC? Furthermore, I have not been able to trigger a positive match from that lookup. I strongly think that if we had a valid subprotocol, we would have a valid master in the first place. In doubt, remove it completely. As a follow up, we should investigate why some dissectors (the HTTP one, at least) set the sni/hostname field without setting a valid protocol, in the first place. This behaviour seems quite suspicious, if not plainly buggy.
* Dissectors shouldn't update `flow->guessed_host_protocol_id`Nardi Ivan2022-09-14
| | | | | | | | | | | | | | | | The field `flow->guessed_host_protocol_id` is set at the beginning of the flow analysis and it represents the "classification by ip" of the flow itself. This field should never be changed. Dissectors which want to provide an "hint" about the classification, should update `flow->guessed_protocol_id` instead. Such "hint" is useless if the dissector set the "extra-dissection" data-path. Rename such field to `guessed_protocol_id_by_ip` to better describe its role. Preliminary work necessary for #1687
* Anydesk: improve detection (#1735)Ivan Nardi2022-09-13
|
* HTTP: remove some wrong codeNardi Ivan2022-09-13
| | | | | | | | | | This code seems wrong or in the wrong place, at least: * "classification by port" and "classification by ip" protocols (i.e "guessed" protocols) should be used to set the protocol stack only after trying all the dissectors, and only by the generic code * there are no reason (for a dissector) to update the "guessed" information using the protocol stack values: it is usually the other way around (see previous point)
* Fix `ndpi_do_guess()` (#1731)Ivan Nardi2022-09-12
| | | | | Avoid a double call of `ndpi_guess_host_protocol_id()`. Some code paths work for ipv4/6 both Remove some never used code.
* STUN: several improvementsNardi Ivan2022-09-11
| | | | | | | | | Add detection over TCP and fix detection over IPv6. Rename some variables since Stun dissector is no more "udp-centric". Stun dissector should always classified the flow as `STUN` or `STUN/Something`. Don't touch `flow->guessed_host_protocol_id` field, which should be always be related to "ip-classification" only.
* HTTP: correctly set the classification protocolsNardi Ivan2022-09-06
| | | | | | | | | | | | Classification should always be set via `ndpi_set_detected_protocol()` to be sure to set a correct `confidence` value, too. Having a "known" protocol stack with `NDPI_CONFIDENCE_UNKNOWN` as confidence, is not valid. This code in HTTP dissector likely needs some more thoughts (the classification itself of the attached example doesn't make a lot of sense), but the goal of this commit is only to always have a valid `confidence` value.
* Serialize nDPI detection/dissection data as object. (#1725)Toni2022-09-06
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix some warningsNardi Ivan2022-09-05
| | | | | | | | | | | | | | | | | | | | | Ubuntu-20.04, clang-16 (nightly build) ``` Making all in src/lib protocols/smpp.c:70:17: warning: variable 'pdu_c' set but not used [-Wunused-but-set-variable] u_int16_t pdu_c = 1; ^ 1 warning generated. third_party/src/ahocorasick.c:173:20: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] thiz->root->root = 1; ^ ~ third_party/src/ahocorasick.c:336:15: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] n->ff = 1; ^ ~ third_party/src/ahocorasick.c:716:21: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] node->final = 1; [...] ```
* Avoid useless host automa lookup (#1724)Ivan Nardi2022-09-05
| | | | | | | | | | | | | | | | | | | The host automa is used for two tasks: * protocol sub-classification (obviously); * DGA evaluation: the idea is that if a domain is present in this automa, it can't be a DGA, regardless of its format/name. In most dissectors both checks are executed, i.e. the code is something like: ``` ndpi_match_host_subprotocol(..., flow->host_server_name, ...); ndpi_check_dga_name(..., flow->host_server_name,...); ``` In that common case, we can perform only one automa lookup: if we check the sub-classification before the DGA, we can avoid the second lookup in the DGA function itself.
* Fix string copy truncation warning. (#1723)Toni2022-09-05
| | | | | | | | | | protocols/ubntac2.c: In function ‘ndpi_search_ubntac2’: protocols/ubntac2.c:69:4: warning: ‘strncpy’ output may be truncated copying between 0 and 31 bytes from a string of length 255 [-Wstringop-truncation] 69 | strncpy(flow->protos.ubntac2.version, (const char *)version, len); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed FastCGI memory issue (was not using nDPI's malloc wrapper). (#1722)Toni2022-09-05
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix endianess in IP-Port lookupNardi Ivan2022-09-05
| | | | | Port parameter in `ndpi_network_port_ptree_match()` must be in network order.
* Fix broken ndpi_timeval_to_(milli|micro)seconds (>UINT_MAX). (#1720)Toni2022-09-04
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add support to opportunistic TLSNardi Ivan2022-09-04
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A lot of protocols provide the feature to upgrade their plain text connections to an encrypted one, via some kind of "STARTTLS" command. Add generic code to support this extension, and allow dissection of the entire TLS handshake. As examples, SMTP, POP, IMAP and FTP dissectors have been updated. Since this feature requires to process more packets per flow, add the possibility to disable it. Fix some log messages. Slight improvement on TCP sequence number tracking. As a side effect, this commit fix also a memory leak found by oss-fuzzer ``` ==108966==ERROR: LeakSanitizer: detected memory leaks Direct leak of 22 byte(s) in 1 object(s) allocated from: #0 0x55f8b367a0be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5480be) (BuildId: 94debacb4a6784c30420ab748c8bf3cc59621063) #1 0x55f8b36e1345 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:321:10 #2 0x55f8b379c7d2 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:25 #3 0x55f8b379cb18 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13 #4 0x55f8b386ce46 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34 #5 0x55f8b385ebf7 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5 #6 0x55f8b39e708c in ndpi_extra_search_mail_smtp_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/mail_smtp.c:422:9 #7 0x55f8b37e636c in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5884:9 #8 0x55f8b37edc05 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6276:5 #9 0x55f8b3701ffc in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1619:31 #10 0x55f8b36faf14 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2189:10 #11 0x55f8b36b6a50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 ``` See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50765
* Fixes on hw calculationLuca Deri2022-09-01
|
* Serializer supports double for JSON and CSV only for the time being (TLV ↵Alfredo Cardigliano2022-09-01
| | | | needs to be extended)
* Temporarily restore ndpi_serialization_unknown = 0Alfredo Cardigliano2022-09-01
|
* Restore ndpi_serialization_type item values. Use 0 for the new ↵Alfredo Cardigliano2022-09-01
| | | | ndpi_serialization_double as there is no more key left (4 bit). Use 16 for ndpi_serialization_unknown which is not used in TLV encoded data.
* Improve AES-NI checkAlfredo Cardigliano2022-08-30
|
* FastCGI: dissect PARAMS (#1715)Toni2022-08-30
| | | | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com>
* Improved MGCP dissector. (#1717)Toni2022-08-30
| | | | | | | | * typ0s fixed * dissect endpoint hostnames Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Internal gcrypt: fix memory leakNardi Ivan2022-08-30
| | | | Fix: ef99eb67
* Improved AES-NI presence check on LinuxLuca Deri2022-08-29
|
* QUIC: fix heap-buffer-overflowNardi Ivan2022-08-25
| | | | | | | | | | | | | | | | | | | | | | | | | | ``` ==12318==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000032 at pc 0x55a59ec97959 bp 0x7fffee67fdd0 sp 0x7fffee67fdc8 READ of size 1 at 0x604000000032 thread T0 #0 0x55a59ec97958 in may_be_0rtt /home/ivan/svnrepos/nDPI/src/lib/protocols/quic.c:1483:24 #1 0x55a59ec9515f in ndpi_search_quic /home/ivan/svnrepos/nDPI/src/lib/protocols/quic.c:1708:13 #2 0x55a59ec32e95 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5428:6 #3 0x55a59ec33c5b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5464:10 #4 0x55a59ec335fc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5497:12 #5 0x55a59ec44615 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6322:15 #6 0x55a59eb8884e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5 #7 0x55a59eb889c7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17 #8 0x7fb5b3ba2082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #9 0x55a59eac742d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x47d42d) (BuildId: 712c87b21cf5c05f64174745909c693d3ba0b62e) 0x604000000032 is located 0 bytes to the right of 34-byte region [0x604000000010,0x604000000032) allocated by thread T0 here: #0 0x55a59eb4bfee in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x501fee) (BuildId: 712c87b21cf5c05f64174745909c693d3ba0b62e) #1 0x55a59eb8899c in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:87:17 ``` Found by CI tests. See: https://github.com/ntop/nDPI/runs/7996151458?check_suite_focus=true
Powered by cgit, Themed by Ted Yin.