aboutsummaryrefslogtreecommitdiff
path: root/example
Commit message (Collapse)AuthorAge
* Finalized nBPF support. You can now define custom protocols such asLuca Deri2022-09-21
| | | | | | | | | | (see exaple/protos.txt) nbpf:"host 192.168.1.1 and port 80"@HomeRouter In order to have nBPF support, you need to compile nDPI with it. Just download https://github.com/ntop/PF_RING in the same directory where you have downloaded nDPI and compile PF_RING/userland/nbpf
* Remove classification "by-ip" from protocol stack (#1743)Ivan Nardi2022-09-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Basically: * "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is NEVER returned in the protocol stack (i.e. `flow->detected_protocol_stack[]`); * if the application is interested into such information, it can access `ndpi_protocol->protocol_by_ip` itself. There are mainly 4 points in the code that set the "classification by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/ `ndpi_detection_giveup()` functions and the HTTP/STUN dissectors. In the unit tests output, a print about `ndpi_protocol->protocol_by_ip` has been added for each flow: the huge diff of this commit is mainly due to that. Strictly speaking, this change is NOT an API/ABI breakage, but there are important differences in the classification results. For examples: * TLS flows without the initial handshake (or without a matching SNI/certificate) are simply classified as `TLS`; * similar for HTTP or QUIC flows; * DNS flows without a matching request domain are simply classified as `DNS`; we don't have `DNS/Google` anymore just because the server is 8.8.8.8 (that was an outrageous behaviour...); * flows previusoly classified only "by-ip" are now classified as `NDPI_PROTOCOL_UNKNOWN`. See #1425 for other examples of why adding the "classification by-ip" in the protocol stack is a bad idea. Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e. `ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for IPv6 flows. Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed. Close #1687
* Replace obsolete linux macroAlfredo Cardigliano2022-09-13
|
* Avoid useless host automa lookup (#1724)Ivan Nardi2022-09-05
| | | | | | | | | | | | | | | | | | | The host automa is used for two tasks: * protocol sub-classification (obviously); * DGA evaluation: the idea is that if a domain is present in this automa, it can't be a DGA, regardless of its format/name. In most dissectors both checks are executed, i.e. the code is something like: ``` ndpi_match_host_subprotocol(..., flow->host_server_name, ...); ndpi_check_dga_name(..., flow->host_server_name,...); ``` In that common case, we can perform only one automa lookup: if we check the sub-classification before the DGA, we can avoid the second lookup in the DGA function itself.
* Add support to opportunistic TLSNardi Ivan2022-09-04
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A lot of protocols provide the feature to upgrade their plain text connections to an encrypted one, via some kind of "STARTTLS" command. Add generic code to support this extension, and allow dissection of the entire TLS handshake. As examples, SMTP, POP, IMAP and FTP dissectors have been updated. Since this feature requires to process more packets per flow, add the possibility to disable it. Fix some log messages. Slight improvement on TCP sequence number tracking. As a side effect, this commit fix also a memory leak found by oss-fuzzer ``` ==108966==ERROR: LeakSanitizer: detected memory leaks Direct leak of 22 byte(s) in 1 object(s) allocated from: #0 0x55f8b367a0be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5480be) (BuildId: 94debacb4a6784c30420ab748c8bf3cc59621063) #1 0x55f8b36e1345 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:321:10 #2 0x55f8b379c7d2 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:25 #3 0x55f8b379cb18 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13 #4 0x55f8b386ce46 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34 #5 0x55f8b385ebf7 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5 #6 0x55f8b39e708c in ndpi_extra_search_mail_smtp_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/mail_smtp.c:422:9 #7 0x55f8b37e636c in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5884:9 #8 0x55f8b37edc05 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6276:5 #9 0x55f8b3701ffc in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1619:31 #10 0x55f8b36faf14 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2189:10 #11 0x55f8b36b6a50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 ``` See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50765
* Fixed serializing bug caused by an unitialized serializer.Toni Uhlig2022-08-24
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Support serialization of double-precision floating-point numbers. Fixes #1702.lns2022-08-24
| | | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add TiVoConnect dissector. Fixes #1697. (#1699)Toni2022-08-08
| | | | | * added static assert if supported, to complain if the flow struct changes Signed-off-by: lns <matzeton@googlemail.com>
* Add Discord dissector. (#1694)Toni2022-08-03
| | | | | * fixed RiotGames false positive Signed-off-by: lns <matzeton@googlemail.com>
* Improved nDPI JSON serialization. (#1689)Toni2022-08-02
| | | | | | | | * fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors * introduced portable version of gmtime_r aka ndpi_gmtime_r * do as most as possible of the serialization work in ndpi_utils.c * use flow2json in ndpiReader Signed-off-by: lns <matzeton@googlemail.com>
* SMTPS, POPS, IMAPS: fix classification and extra dissection (#1685)Ivan Nardi2022-07-30
| | | | | | The big change in TLS code is to allow "master" protocols other than TLS/DTLS, like SMTPS, POPS and IMAPS. This change will allow, in a future, a proper and complete TLS dissection for all these protocols with "STARTTLS"-like messages.
* Add Softether dissector. (#1679)Toni2022-07-29
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)Ivan Nardi2022-07-29
| | | | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; enabled by default in the unit tests. This new option enables/disables dumping of "num dissectors calls" values, too (see b4cb14ec).
* Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics ↵Ivan Nardi2022-07-29
| | | | | (#1677)" (#1682) This reverts commit bb83899985c25097341b947c2c535f56254a075c.
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)Ivan Nardi2022-07-29
| | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; disabled by default to avoid too much fuss with the unit tests.
* Improve handling of HTTP-Proxy and HTTP-Connect (#1673)Ivan Nardi2022-07-25
| | | | | | | | | | | | Treat HTTP-Proxy and HTTP-Connect flows like the HTTP ones: print/serialize all the attributes and allow parsing of replies. The line about "1kxun" has been removed to avoid regressions in 1KXUN classification in `tests/pcap/1kxun.pcap`. I haven't fully understod what was happening but the comment at the beginning of `static ndpi_category_match category_match[]` says that we can't have overlaps between `host_match` and `category_match` lists and that is no longer true since 938e89ca. Bottom line: removing this line seems the right thing to do, anyway.
* Add support for flow client/server information (#1671)Ivan Nardi2022-07-24
| | | | | | | | | | | | | | | | | | In a lot of places in ndPI we use *packet* source/dest info (address/port/direction) when we are interested in *flow* client/server info, instead. Add basic logic to autodetect this kind of information. nDPI doesn't perform any "flow management" itself but this task is delegated to the external application. It is then likely that the application might provide more reliable hints about flow client/server direction and about the TCP handshake presence: in that case, these information might be (optionally) passed to the library, disabling the internal "autodetect" logic. These new fields have been used in some LRU caches and in the "guessing" algorithm. It is quite likely that some other code needs to be updated.
* reader_util: stop processing a flow (#1666)Ivan Nardi2022-07-20
| | | | We should stop processing a flow if all protocols have been excluded or if we have already processed too many packets.
* Do not interfere with CFLAGS/LDFLAGS env anymore. (#1659)Toni2022-07-13
| | | | | | | * CI fixes * some build systems do not like that (e.g. OpenWrt) * fixed some rrdtool related build warnings/errors Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Keep track of how many dissectors calls we made for each flow (#1657)Ivan Nardi2022-07-11
|
* Updated JA3/SSL fingerprints.Toni Uhlig2022-07-04
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bins: add support for 64bit bins (#1626)Ivan Nardi2022-07-03
|
* Improved TFTP. Dissect Read/Write Request filenames. (#1617)Toni2022-07-03
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix byte-order issue during ndpiReader tcp/udp src/dst port serialization. ↵Toni2022-07-03
| | | | | | | | Fixes #1608. (#1614) * fixed possible memory leak caused by an invalid call to `node_proto_guess_walker()` during serialization * execute serialization code while running regression tests Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix compilation and sync unit tests results (#1606)Ivan Nardi2022-06-20
|
* Added unidirectional traffic flow riskLuca Deri2022-06-20
|
* Added collectd dissector (again). (#1601)Toni2022-06-17
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Replaced nDPI's internal hashmap with uthash. (#1602)Toni2022-06-17
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Add some statistics to ndpiReader (#1587)claudio_burrafato2022-06-15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add some statistics to ndpiReader The purpose of this version of ndpiReader is too adding some other statistics printed by ndpiReader. In this simple version the domain names(in the flows) that are collected are: flow-> ssh_tls.server_info flow-> host_server_name and are placed in a UT_hash_table, ordering them by number of occurrences. * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update example/ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com> * Update ndpiReader.c * Update ndpiReader.c * Update ndpiReader.c * Update ndpiReader.c Co-authored-by: Toni <matzeton@googlemail.com>
* Fix invalid memory access (#1596)Ivan Nardi2022-06-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | We can access `flow->protos` union only after checking the protocol. Checking `flow->detected_protocol.master_protocol` is redundant because we already check it in `is_ndpi_proto` ``` AddressSanitizer:DEADLYSIGNAL ================================================================= ==29739==ERROR: AddressSanitizer: SEGV on unknown address 0x000000353820 (pc 0x7f9b64dd2717 bp 0x7fff161a52f0 sp 0x7fff161a4aa8 T0) ==29739==The signal is caused by a READ memory access. #0 0x7f9b64dd2717 /build/glibc-SzIz7B/glibc-2.31/string/../sysdeps/x86_64/multiarch/strlen-avx2.S:96 #1 0x555c65e597d8 in __interceptor_strlen (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x6407d8) (BuildId: 11ac8ec30f1d49fb0276c9b03368e491505d2bba) #2 0x555c65fd85fa in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:269:13 #3 0x555c65f3e8c6 in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1188:36 #4 0x555c65f52cab in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1567:2 #5 0x555c65f4b632 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2110:10 #6 0x555c65f04d29 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:109:7 #7 0x555c65f054bb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:181:17 #8 0x7f9b64c6e082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #9 0x555c65e4253d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x62953d) (BuildId: 11ac8ec30f1d49fb0276c9b03368e491505d2bba) ``` Found by oss-fuzzer. See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=48020
* Improved DNS traffic analysisLuca Deri2022-06-13
| | | | Added ability to identify application and network protocols
* Added gprof CPU/HEAP profiling support. (#1592)Toni2022-06-12
| | | | | * Some small auto{conf,make} improvements Signed-off-by: lns <matzeton@googlemail.com>
* Removed Makefile references to legacy code. (#1589)Toni2022-06-08
| | | | | - feature removed with f5545a80f95a90a5219bbb319b346f964b1e1548 Signed-off-by: lns <matzeton@googlemail.com>
* Updated README.md (#1562)Toni2022-05-30
| | | | | | | * make check great again (not so much) * make doc/doc-view * CI updates Signed-off-by: lns <matzeton@googlemail.com>
* Added ability to return risk info in JSON format in ndpi_get_flow_risk_info()Luca2022-05-30
|
* Fixed dispay bug for risk_infoLuca Deri2022-05-30
|
* Updated tests resultsLuca Deri2022-05-30
| | | | Code cleanup
* Added RiskInfo stringLuca Deri2022-05-30
|
* Compilation fixes for old gcc compilerLuca Deri2022-05-28
|
* Added script to compare and verify the output of `make dist'. (#1551)Toni2022-05-26
| | | | | | | This fixes some build/test issues resulting when using tarballs. * nDPI uses autotools (especially autoconf) in a wrong way, see #1163 Signed-off-by: lns <matzeton@googlemail.com>
* Removed MacOS XCode integration.Toni Uhlig2022-05-16
| | | | | | | | | * It is outdated (from 2018) and will most likely not work anymore due to changed protocol files and outdated API usage in the example integration * Removal is the only option besides fixing those issues and provide a CI integration for future automatic checks Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Replaced ndpiReader's libjson-c support with libnDPI's internal ↵Toni2022-05-07
| | | | | | | | | | | serialization interface. (#1535) * Fixes #1528 * Serialization Interface should also fuzzed * libjson-c may only be used in the unit test to verify the internal serialization interface * Serialization Interface supports tlv(broken), csv and json * Unit test does work again and requires libjson-c Signed-off-by: lns <matzeton@googlemail.com>
* Added BPF filtering for discarding non-IP packetsLuca Deri2022-04-27
|
* Added ability to store custom category file in patricia treeLuca Deri2022-04-26
|
* Added generic user agent setter. (#1530)Toni2022-04-25
| | | | | * ndpiReader: Print user agent if one was set and not just for certain protocols. Signed-off-by: lns <matzeton@googlemail.com>
* XIAOMI: add detection of Xiaomi traffic (#1529)Ivan Nardi2022-04-25
| | | Most of the credits should go to @utoni (see #1521)
* Reduce ndpiReader's `-h' spam. `-H' does this job now. (#1523)Toni2022-04-21
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Removed superfluous ifdef'd includes. (#1519)Toni2022-04-19
| | | Signed-off-by: lns <matzeton@googlemail.com>
* DGA improvementsLuca Deri2022-04-16
|
* Fixed msys2 build warnings and re-activated CI Mingw64 build.fix/windows-msys2Toni Uhlig2022-04-14
| | | | | | | * Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.