aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* Add Meraki Cloud protocol and improve other protocols (#1854)Ivan Nardi2023-01-17
| | | | | Avoid some LineCall and Jabber false positives. Detect Discord mid flows. Fix Bittorrent detection.
* Add some fuzzers to test algorithms and data structures (#1852)Ivan Nardi2023-01-17
| | | Fix some issues found with these new fuzzers
* Fix classification "by-port" (#1851)Ivan Nardi2023-01-17
| | | | | | | | | | | Classification "by-port" should be the last possible effort, *after* having test all the LRU caches. Remove some dead code from `ndpi_detection_giveup()`: `flow->guessed_protocol_id` is never set to any od those voip protocols and at that point in this function we never have both a master *and* a application protocols. Coverage reports (both from unit tests and from fuzzing) confirms that was dead code.
* Improved DGA detectionLuca Deri2023-01-12
|
* Reduce the size of some traces used as unit test (#1845)Ivan Nardi2023-01-05
| | | | | | | | | | | | | | | | | | | No traces and no flows has been removed; only long sessions has been reduced, keeping only their first packets. This is quite important in fuzzing systems, since these pcaps are used as initial seed. There is no value in fuzzing long sessions, because only the very first packets are really used/processed by nDPI. Before: ``` du -h tests/pcap/ 200M tests/pcap/ ``` After: ``` du -h tests/pcap/ 98M tests/pcap/ ```
* PPSTREAM: fix some heap-buffer overflows (#1846)Ivan Nardi2023-01-05
| | | | Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54802
* Fix missing bracket at ppstream (#1843)Ege Çetin2023-01-03
| | | | | | | * add missing bracket * Sync unit test results Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
* Fix some warnings and two errors found while fuzzing (#1844)Ivan Nardi2023-01-03
| | | | Fix CI See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54614
* Added missing prototypeLuca Deri2023-01-03
|
* Added NDPI_MINOR_ISSUES risk used for storing generic/relevant information ↵Luca Deri2022-12-31
| | | | about issues found on traffic.
* Added NDPI_PERIODIC_FLOW flow risk to be used by apps based on nDPILuca Deri2022-12-30
|
* Implemented EDNS(0) support in DNS dissectorLuca Deri2022-12-29
| | | | Improved DNS dissection
* fuzz: add fuzzer testing nDPI (initial) configurations (#1830)Ivan Nardi2022-12-23
| | | | | | | | | | | | | | | | | The goal of this fuzzer is to test init and deinit of the library, with different configurations. In details: * random memory allocation failures, even during init phase * random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()` * random LRU caches sizes * random bitmask of enabled protocols * random parameters of `ndpi_set_detection_preferences()` * random initialization of opportunistic TLS * random load/don't load of configuration files This new fuzzer is a C++ file, because it uses `FuzzedDataProvider` class (see https://github.com/google/fuzzing/blob/master/docs/split-inputs.md). Note that the (existing) fuzzers need to be linked with C++ compiler anyway, so this new fuzzer doesn't add any new requirements.
* Remove some old protocols (#1832)Ivan Nardi2022-12-23
| | | | | | | | | | | | | | | | | | | | | These protocols: * have been addeded in the OpenDPI era * have never been updated since then * we don't have any pcap examples [*] If (and it is a big if...) some of these protocols are still somehow used and if someone is still interested in them, we can probably re-add them starting from scratch (because the current detection rules are probably outdated) Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT, OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA, FLORENSIA, AIMINI, SOPCAST PPSTREAM dissector works (...) only on UDP. [*]: with do have an AIMINI test pcap but it was some trivial http traffic detected only by hostname matching, on domains no more available...
* postgres: improve detection (#1831)Ivan Nardi2022-12-22
| | | Remove some dead code (found via coverage report)
* Fix infinite loop when a custom rule has port 65535 (#1833)Ivan Nardi2022-12-21
| | | Close #1829
* Fix some errors found by oss-fuzz (#1834)Ivan Nardi2022-12-20
|
* Remove any references to LGTM (#1837)Ivan Nardi2022-12-20
|
* Added TP-LINK Smart Home Protocol dissector. (#1841)Darryl Sokoloski2022-12-20
| | | | | Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca> Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
* Sync unit tests results (#1840)Ivan Nardi2022-12-19
| | | | Update the documentation, hinting how to test/update *all* the unit tests.
* Added TUYA LAN Protocol dissector. (#1838)Darryl Sokoloski2022-12-19
| | | | | | | | Tuya IoTOS Embedded Wi-Fi and BLE SDK for bk7231n. Used by many "smart" devices such as LED light strips, bulbs, etc. Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca> Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
* Add protocol disabling feature (#1808)Ivan Nardi2022-12-18
| | | | | | | | | | | | | | | | | | | | | | The application may enable only some protocols. Disabling a protocol means: *) don't register/use the protocol dissector code (if any) *) disable classification by-port for such a protocol *) disable string matchings for domains/certificates involving this protocol *) disable subprotocol registration (if any) This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`. Custom protocols are always enabled. Technically speaking, this commit doesn't introduce any API/ABI incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()` is now mandatory, just after having called `ndpi_init_detection_module()`. Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to the removing of some function parameters. Fix the low level macro `NDPI_LOG`. This issue hasn't been detected sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
* Updated decriptionLuca Deri2022-12-17
|
* README: another try to fix build badge (#1836)Ivan Nardi2022-12-16
|
* README: fix build badge link (#1835)Ivan Nardi2022-12-16
| | | See: https://github.com/badges/shields/issues/8671
* Added ability to decode DNS PTR records used for reverse address resolutionLuca Deri2022-12-15
|
* Minor RTP fixLuca Deri2022-12-15
|
* STUN: add detection of ZOOM peer-to-peer flows (#1825)Ivan Nardi2022-12-11
| | | | See: "Enabling Passive Measurement of Zoom Performance in Production Networks" https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
* Centos7 fixLuca Deri2022-12-11
|
* Rocky9 fixLuca Deri2022-12-11
|
* fuzz: some enhancements (#1827)Ivan Nardi2022-12-10
| | | | | | | | | | Load some custom configuration (like in the unit tests) and factorize some (fuzzing) common code. There is no way to pass file paths to the fuzzers as parameters. The safe solution seems to be to load them from the process working dir. Anyway, missing file is not a blocking error. Remove some dead code (found looking at the coverage report)
* DTLS: handle (certificate) fragments (#1811)Ivan Nardi2022-12-10
| | | | | Keep using the existing function to handle reassembling buffer: rename it from `ndpi_search_tls_tcp_memory` to `ndpi_search_tls_memory` and make it "transport" agnostic
* Fix undefined-behavior when setting empty user-agent (#1821)Ivan Nardi2022-12-10
| | | | | | | ``` ndpi_main.c:9111:35: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:44:28: note: nonnull attribute specified here SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ndpi_main.c:9111:35 in ```
* Packaing changesLuca Deri2022-12-10
|
* fuzz: fix signed-integer-overflow (#1822)Ivan Nardi2022-12-10
| | | | | | | | | | ``` fuzz_ndpi_reader.c:33:29: runtime error: signed integer overflow: 214013 * 24360337 cannot be represented in type 'int' #0 0x4c1cf7 in fastrand ndpi/fuzz/fuzz_ndpi_reader.c:33:29 #1 0x4c1cf7 in malloc_wrapper ndpi/fuzz/fuzz_ndpi_reader.c:38:11 #2 0x523057 in ndpi_malloc ndpi/src/lib/ndpi_main.c:220:25 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54112
* Build: fix `make install` when `--prefix=` is used in configure script (#1824)Ivan Nardi2022-12-10
| | | | | Use the same logic already used in `example/Makefile.in` Close #1823
* Added Zoom screen share detectionLuca Deri2022-12-09
|
* Added RTP stream type in flow metadataLuca Deri2022-12-09
|
* Improved Zoom protocol detectionLuca Deri2022-12-08
|
* fuzz: add a new fuzzer testing memory allocation failures (#1818)Ivan Nardi2022-12-06
| | | | | | | | | Try to fuzz error paths triggered by allocation errors. Fix some errors already found by this new fuzzer. Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to (not)compile specific code in fuzzing builds. See: https://llvm.org/docs/LibFuzzer.html
* Fix compilation (#1819)Ivan Nardi2022-12-05
|
* Updated test resultsLuca Deri2022-12-05
|
* Exported HTTP server in metadataLuca Deri2022-12-05
|
* thread_index may by negative. (#1814)Gowa20172022-12-05
| | | | | | | | * thread_index may by negative. Like: 192.168.8.155:55848 --> 183.3.224.139 * reader thread index also need to uint32
* Add support for LTO and Gold linker (#1812)Ivan Nardi2022-12-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit add (optional) support for Link-Time-Optimization and Gold linker. This is the first, mandatory step needed to make nDPI compliant with "introspector" sanitizer requirements in OSS-Fuzz: see https://github.com/google/oss-fuzz/issues/8939 Gold linker is not supported by Windows and by macOS, so this feature is disabled by default. It has been enable in CI in two linux targets ("latest" gcc and clang). Fix some warnings triggered by LTO. The changes in `src/lib/ndpi_serializer.c` seams reasonable. However, the change in `tests/unit/unit.c` is due to the following warning, which seems to be a false positive. ``` unit.c: In function ‘serializerUnitTest’: ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here 67 | ndpi_serializer serializer, deserializer; | ^ ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here 67 | ndpi_serializer serializer, deserializer; ``` Since this warning is triggered only with an old version of gcc and `tests/unit/unit.c` is used only during the tests, the easiest fix has been applied. Some (unknown to me) combinations of OS and compiler trigger the following warnings at linker time (with sanitizer and gold linker) ``` /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm' [..] ``` I have not found any references to this kind of message, with the only exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975 which seems to suggest that these messages can be safely ignored. In any case, the compilation results are sound. Fix `clean` target in the Makefile in the `example` directory. In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error (as always, when the gold linker is involved...). It's come out that the culprit was the `tempnam` function: the code has been changed to use `tmpfile` instead. No sure why... :( Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a` anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds as workaround for the "missing dependencies errors" described in https://github.com/google/oss-fuzz/issues/8939
* CI: GitHub is updating Ubuntu runners (#1817)Ivan Nardi2022-12-03
| | | | | | | | | | | GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our dependencies. See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/ This is the reason of the recent random failures in CI. Update "newest" tested gcc to gcc-12. Fix a memory error introduced in 557bbcfc5a5165c9eb43bbdd78435796239cd3c9
* Make LRU caches ipv6 aware (#1810)Ivan Nardi2022-12-03
| | | | | | | | | | | | | Simplest solution, keeping the existing cache data structure TLS certificate cache is used for DTLS traffic, too. Note that Ookla cache already works with ipv6 flows. TODO: * make the key/hashing more robust (extending the key size?) * update bittorrent cache too. That task is quite difficult because ntopng uses a public function (`ndpi_guess_undetected_protocol()`) intrinsically ipv4 only...
* Updated tests resultsLuca Deri2022-12-03
|
* Added check for empty/missing user-agent in HTTPLuca Deri2022-12-03
|
* Improved HTTP header parsing skipping lines that won't matchLuca Deri2022-12-03
|
Powered by cgit, Themed by Ted Yin.