aboutsummaryrefslogtreecommitdiff
path: root/src/lib/ndpi_main.c
Commit message (Collapse)AuthorAge
* fuzz: some improvements and add two new fuzzers (#1881)Ivan Nardi2023-02-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from `fuzz/Makefile.am`; it is already included by the main configure script (when fuzzing). Add a knob to force disabling of AESNI optimizations: this way we can fuzz also no-aesni crypto code. Move CRC32 algorithm into the library. Add some fake traces to extend fuzzing coverage. Note that these traces are hand-made (via scapy/curl) and must not be used as "proof" that the dissectors are really able to identify this kind of traffic. Some small updates to some dissectors: CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting with "VS01" will be classified as STEAM (see steam.c around line 111). Googling it, it seems right so. XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore that condition is false. RTP, STUN: removed useless "break"s Zattoo: `flow->zattoo_stage` is never set to any values greater or equal to 5, so these checks are never true. PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it. TeamSpeak: we check for `flow->packet_counter == 3` just above, so the following check `flow->packet_counter >= 3` is always false.
* Add VK detection (#1880)0xA50C1A12023-02-02
|
* Improved connection refused detectionLuca Deri2023-01-25
|
* Fix compilation and CI (#1875)Ivan Nardi2023-01-25
| | | | | | | | | | | | | | | | | | | | | ubuntu-latest s390x CI doesn't like snapshot bigger than 262144 bytes. Fix an error found by fuzz CI ``` ================================================================= ==55399==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x561e41cb684d bp 0x7ffd54ce3650 sp 0x7ffd54ce3520 T0) ==55399==The signal is caused by a READ memory access. ==55399==Hint: address points to the zero page. #0 0x561e41cb684d in ndpi_network_ptree_match /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2321:41 #1 0x561e41d30879 in ndpi_guess_undetected_protocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7767:8 #2 0x561e41ca804d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_config.cpp:104:5 #3 0x561e41bb96a0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4726a0) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #4 0x561e41ba3c2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x45cc2f) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #5 0x561e41ba96f6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4626f6) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #6 0x561e41bd22e2 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x48b2e2) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #7 0x7f94f0e5c082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #8 0x561e41b9eb0d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x457b0d) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) ```
* CenOS compilation fixLuca Deri2023-01-25
|
* Compilation fixesLuca Deri2023-01-25
|
* Add some fuzzers to test other data structures. (#1870)Ivan Nardi2023-01-25
| | | | | | | Start using a dictionary for fuzzing (see: https://llvm.org/docs/LibFuzzer.html#dictionaries). Remove some dead code. Fuzzing with debug enabled is not usually a great idea (from performance POV). Keep the code since it might be useful while debugging.
* Some small changes (#1869)Ivan Nardi2023-01-25
| | | | | | | | All dissector callbacks should not be exported by the library; make static some other local functions. The callback logic in `ndpiReader` has never been used. With internal libgcrypt, `gcry_control()` should always return no errors. We can check `categories` length at compilation time.
* Added new risk NDPI_TCP_ISSUESLuca Deri2023-01-24
|
* fuzz: fix memory allocation failure logic (#1867)Ivan Nardi2023-01-20
| | | | | We *do* want to have some allocation errors. Fix some related bugs Fix: 29be01ef
* Minor changes in `ndpi_detection_giveup()` (#1861)Ivan Nardi2023-01-19
| | | | | | Check the mining cache at the end of the function, like all the others LRU caches. Rewrite the STUN checks: same identical semantic but in a cleaner way, hopefully
* Fix undefined-behaviour in ahocorasick callback (#1864)Ivan Nardi2023-01-19
| | | | | | | ``` ndpi_main.c:2119:28: runtime error: left shift of 1 by 31 places cannot be represented in type 'int' ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=55226
* LRU caches: add a generic (optional and configurable) expiration logic (#1855)Ivan Nardi2023-01-18
| | | Two caches already implemented a similar mechanism: make it generic.
* Add detection of Tailscale (#1857)Ivan Nardi2023-01-17
| | | Extend the example of wireguard traffic
* Add Meraki Cloud protocol and improve other protocols (#1854)Ivan Nardi2023-01-17
| | | | | Avoid some LineCall and Jabber false positives. Detect Discord mid flows. Fix Bittorrent detection.
* Add some fuzzers to test algorithms and data structures (#1852)Ivan Nardi2023-01-17
| | | Fix some issues found with these new fuzzers
* Fix classification "by-port" (#1851)Ivan Nardi2023-01-17
| | | | | | | | | | | Classification "by-port" should be the last possible effort, *after* having test all the LRU caches. Remove some dead code from `ndpi_detection_giveup()`: `flow->guessed_protocol_id` is never set to any od those voip protocols and at that point in this function we never have both a master *and* a application protocols. Coverage reports (both from unit tests and from fuzzing) confirms that was dead code.
* Improved DGA detectionLuca Deri2023-01-12
|
* Fix some warnings and two errors found while fuzzing (#1844)Ivan Nardi2023-01-03
| | | | Fix CI See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54614
* Added NDPI_MINOR_ISSUES risk used for storing generic/relevant information ↵Luca Deri2022-12-31
| | | | about issues found on traffic.
* Added NDPI_PERIODIC_FLOW flow risk to be used by apps based on nDPILuca Deri2022-12-30
|
* Implemented EDNS(0) support in DNS dissectorLuca Deri2022-12-29
| | | | Improved DNS dissection
* fuzz: add fuzzer testing nDPI (initial) configurations (#1830)Ivan Nardi2022-12-23
| | | | | | | | | | | | | | | | | The goal of this fuzzer is to test init and deinit of the library, with different configurations. In details: * random memory allocation failures, even during init phase * random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()` * random LRU caches sizes * random bitmask of enabled protocols * random parameters of `ndpi_set_detection_preferences()` * random initialization of opportunistic TLS * random load/don't load of configuration files This new fuzzer is a C++ file, because it uses `FuzzedDataProvider` class (see https://github.com/google/fuzzing/blob/master/docs/split-inputs.md). Note that the (existing) fuzzers need to be linked with C++ compiler anyway, so this new fuzzer doesn't add any new requirements.
* Remove some old protocols (#1832)Ivan Nardi2022-12-23
| | | | | | | | | | | | | | | | | | | | | These protocols: * have been addeded in the OpenDPI era * have never been updated since then * we don't have any pcap examples [*] If (and it is a big if...) some of these protocols are still somehow used and if someone is still interested in them, we can probably re-add them starting from scratch (because the current detection rules are probably outdated) Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT, OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA, FLORENSIA, AIMINI, SOPCAST PPSTREAM dissector works (...) only on UDP. [*]: with do have an AIMINI test pcap but it was some trivial http traffic detected only by hostname matching, on domains no more available...
* Fix infinite loop when a custom rule has port 65535 (#1833)Ivan Nardi2022-12-21
| | | Close #1829
* Added TP-LINK Smart Home Protocol dissector. (#1841)Darryl Sokoloski2022-12-20
| | | | | Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca> Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
* Added TUYA LAN Protocol dissector. (#1838)Darryl Sokoloski2022-12-19
| | | | | | | | Tuya IoTOS Embedded Wi-Fi and BLE SDK for bk7231n. Used by many "smart" devices such as LED light strips, bulbs, etc. Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca> Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
* Add protocol disabling feature (#1808)Ivan Nardi2022-12-18
| | | | | | | | | | | | | | | | | | | | | | The application may enable only some protocols. Disabling a protocol means: *) don't register/use the protocol dissector code (if any) *) disable classification by-port for such a protocol *) disable string matchings for domains/certificates involving this protocol *) disable subprotocol registration (if any) This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`. Custom protocols are always enabled. Technically speaking, this commit doesn't introduce any API/ABI incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()` is now mandatory, just after having called `ndpi_init_detection_module()`. Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to the removing of some function parameters. Fix the low level macro `NDPI_LOG`. This issue hasn't been detected sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
* STUN: add detection of ZOOM peer-to-peer flows (#1825)Ivan Nardi2022-12-11
| | | | See: "Enabling Passive Measurement of Zoom Performance in Production Networks" https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
* DTLS: handle (certificate) fragments (#1811)Ivan Nardi2022-12-10
| | | | | Keep using the existing function to handle reassembling buffer: rename it from `ndpi_search_tls_tcp_memory` to `ndpi_search_tls_memory` and make it "transport" agnostic
* Fix undefined-behavior when setting empty user-agent (#1821)Ivan Nardi2022-12-10
| | | | | | | ``` ndpi_main.c:9111:35: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:44:28: note: nonnull attribute specified here SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ndpi_main.c:9111:35 in ```
* fuzz: add a new fuzzer testing memory allocation failures (#1818)Ivan Nardi2022-12-06
| | | | | | | | | Try to fuzz error paths triggered by allocation errors. Fix some errors already found by this new fuzzer. Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to (not)compile specific code in fuzzing builds. See: https://llvm.org/docs/LibFuzzer.html
* Exported HTTP server in metadataLuca Deri2022-12-05
|
* CI: GitHub is updating Ubuntu runners (#1817)Ivan Nardi2022-12-03
| | | | | | | | | | | GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our dependencies. See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/ This is the reason of the recent random failures in CI. Update "newest" tested gcc to gcc-12. Fix a memory error introduced in 557bbcfc5a5165c9eb43bbdd78435796239cd3c9
* Make LRU caches ipv6 aware (#1810)Ivan Nardi2022-12-03
| | | | | | | | | | | | | Simplest solution, keeping the existing cache data structure TLS certificate cache is used for DTLS traffic, too. Note that Ookla cache already works with ipv6 flows. TODO: * make the key/hashing more robust (extending the key size?) * update bittorrent cache too. That task is quite difficult because ntopng uses a public function (`ndpi_guess_undetected_protocol()`) intrinsically ipv4 only...
* Improved HTTP header parsing skipping lines that won't matchLuca Deri2022-12-03
|
* Fix classification "by-port" of QUIC flows (#1785)Ivan Nardi2022-11-22
| | | | | | | | | nDPI is able to properly classify QUIC flows only if it elaborates the very first packets of the flow. The protocol list in `is_udp_guessable_protocol()` is basically a list of protocols which can be detected from *any* packets in the flow. Rename such function to `is_udp_not_guessable_protocol()`: the name is still quite cryptic, but at least not plainly wrong
* TLS: improve handling of ALPN(s) (#1784)Ivan Nardi2022-10-25
| | | | | | | | Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is extracted from the CH, the latter from the SH. Add some entries to the known ALPN list. Fix printing of "TLS Supported Versions" field.
* Fix memory corruption on TCP flows (#1786)Ivan Nardi2022-10-25
|
* Add Elasticsearch protocol dissector. (#1782)Toni2022-10-21
| | | | | | | * all credits goes to @verzulli Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add Munin protocol dissector.Toni Uhlig2022-10-20
| | | | | | * all credits goes to @verzulli Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Updated FortiClient default portsLuca Deri2022-10-19
|
* LINE_CALL: add detection of LINE voip calls (#1761)Ivan Nardi2022-10-06
| | | | | | | | These flows are classifed as `LINE_CALL`; another option was `RTP/LINE_CALL`. No sure about the best solution... Extend LINE domains list. Remove RTP dead code.
* Added new flow risk NDPI_HTTP_OBSOLETE_SERVER. Currently Apache and nginx ↵Luca2022-10-04
| | | | are supported
* Fix compilation and sync utests resultsNardi Ivan2022-10-04
|
* Added the ability to track the payload via -E and via the new option ↵Luca2022-10-04
| | | | 'ndpi_track_flow_payload'
* Line app support (#1759)sharonenoch2022-10-01
| | | | | | | | | | | | | | | | | | | * Standard support for LINE app * Added test pcap for LINE app * make check result for LINE app * Make check success as 1kxun has LINE packets * Added the ASN inc file for LINE * Removed extra lines as its effecting make check * Editing the SNI required a new pcap output file for TLS.Line format * Run Configure with --with-pcre --with-maxminddb to enable the generation of h323-overflow.pcap.out Co-authored-by: Sharon Enoch <sharone@amzetta.com>
* Fixed invalid unidirectional traffic alert for unidirectional protocols ↵Luca2022-09-29
| | | | (e.g. sFlow)
* Remove unused codeNardi Ivan2022-09-28
| | | | | | | | | LRU callbacks have been added in 460ff3c7a, but they have never been used and they have never been extended to the other LRU caches. `ndpi_search_tcp_or_udp()` basically returns the classification by port/ip of the flow; calling it from the dissector is useless. The same for TOR detection: ips are checked in the generic code
* Fix value of `ndpi_protocol->protocol_by_ip`Nardi Ivan2022-09-25
| | | | Fix: a7c2734b
Powered by cgit, Themed by Ted Yin.