aboutsummaryrefslogtreecommitdiff
path: root/src/lib
Commit message (Collapse)AuthorAge
* Added the ability to define custom protocols with arbitrary Ids in proto.txtLuca Deri2023-03-22
| | | | | | | | | | Example - ip:213.75.170.11/32:443@CustomProtocol nDPI assigns an is that can change based on protos.txt content - ip:213.75.170.11/32:443@CustomProtocol=9999 nDPI assigns 9999 as protocolId to CustomProtocol and won't change when protos.txt content will chaneg
* Improved checks for duplicated entries in protocols fileLuca Deri2023-03-21
|
* DisneyPlus/Hulu ip lists should be auto-generated (#1905)Ivan Nardi2023-03-20
| | | | | | Remove two stale ip lists: 1) these 3 ips are in the Amazon ranges (now)... 2) the Instagram list originated from AS32934, which is now a Facebook AS; see https://github.com/ntop/nDPI/pull/1264/commits/8dabd06301a802dd38616ba8684a1d995783e023
* Add a new protocol id for generic Adult Content traffic (#1906)Ivan Nardi2023-03-20
| | | | The list has been taken from https://www.similarweb.com/top-websites/adult/ Fix a GoTo false positive.
* Add a new protocol id for generic advertisement/analytics/tracking stuff (#1904)Ivan Nardi2023-03-20
|
* fuzz: add a new fuzzer to test TLS certificates (#1901)Ivan Nardi2023-03-20
|
* FreeBSD compilation fixLuca Deri2023-03-14
| | | | (C) update
* fuzz: simplify fuzzers dependencies in CIFuzz (#1896)Ivan Nardi2023-03-14
| | | | | | | | | | | | | | | | | | | | | | CIFuzz (based on oss-fuzz) is the GitHub action/CI job that runs fuzz targets on pull requests. It only runs the fuzzers affected by a pull request/commit. Otherwise it will divide up the allotted fuzzing time among all fuzzers in the project. Since: * we have more than 20 fuzzers and most of them use the custom memory allocation functions (to force allocation failures) even if they are not strictly about DPI stuff; * we need to keep fuzzing time relatively small (to avoid waiting the CI results for a long time) it is important that fuzzers dependencies (which are based on *files* changed by the single commit/PR) are as small as possible. Bottom line: move all the low-level allocation callbacks to a dedicated file; this way most of the fuzzers don't depend anymore on `ndpi_main.c` file (which is touched by ever commit/PR). The goal is to have only the "most important" fuzzers running during (most of) the CI.
* Added ability to define an unlimited number of custom rules IP:port for the ↵Luca Deri2023-03-13
| | | | same IP (it used tobe limited to 2)
* Minor cleanupLuca Deri2023-03-13
|
* Added fix for better handling exceptions rollback in case of later matchLuca Deri2023-03-06
|
* Add a new flow risk about literal IP addresses used as SNI (#1892)Ivan Nardi2023-03-02
| | | | | | | | | | RFC 6066 3: "Literal IPv4 and IPv6 addresses are not permitted in "HostName"." Don't set this risk if we have a valid sub-classification (example: via certificate) Since a similar risk already exists for HTTP hostnames, reuse it, with a more generic name.
* Fix insert of ip addresses into patricia tree(s) (#1895)Ivan Nardi2023-03-02
| | | Fix: d0b46dcf
* Keep master protocol in `ndpi_reconcile_protocols`Nardi Ivan2023-03-01
| | | | | | | | | | | | | | | | | | | As a side effect of this change, the following memory leak has been fixed. ``` ==55523==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2 byte(s) in 1 object(s) allocated from: #0 0x55657d01b7be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_alloc_fail+0x5b27be) (BuildId: 9dfa7bd9081421417b8b1c2e8b44633a9bfcfe52) #1 0x55657d169182 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:243:25 #2 0x55657d1694c8 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:310:13 #3 0x55657d25ddc7 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2308:46 #4 0x55657d2701b9 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:895:5 #5 0x55657d26a408 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1223:11 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56466
* SoftEther: fix invalid memory accessNardi Ivan2023-03-01
| | | | | | | | | | | | | | | | | | | | We can't write `flow->protos` union until we are really sure about protocol classification ``` ==28334==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x558db5554512 bp 0x000000000000 sp 0x7ffcb22c2880 T0) ==28334==The signal is caused by a READ memory access. ==28334==Hint: this fault was caused by a dereference of a high value address (see register values below). Disassemble the provided pc to learn which register was used. #0 0x558db5554512 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x48e512) (BuildId: 2f71e395637a7b748f36d5a04c7281f18b1128d7) #1 0x558db55ea54b in __interceptor_free (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x52454b) (BuildId: 2f71e395637a7b748f36d5a04c7281f18b1128d7) #2 0x558db56977ca in ndpi_free /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:274:7 #3 0x558db56c20e3 in ndpi_free_flow_data /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5175:2 #4 0x558db569783f in ndpi_free_flow /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8394:5 #5 0x558db5627936 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:38:3 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56272
* Fixed uninitialized variableLuca Deri2023-02-27
|
* Added missing breakLuca Deri2023-02-27
|
* Improved protocol detection exploiting IP-based guessLuca Deri2023-02-27
| | | | Reworked ndpi_reconcile_protocols() that is now called only in front of a match (less overhead)
* Enhanced custom port definition and improved error reporting in case of ↵Luca Deri2023-02-24
| | | | duplications
* fuzz: extend fuzz coverage (#1888)Ivan Nardi2023-02-16
|
* Fix export/serialization of `flow->risk` (#1885)Ivan Nardi2023-02-14
| | | | Any risks bigger than 31 was never exported/serialized; found looking a the fuzz coverage report.
* Fix packet counters (#1884)Ivan Nardi2023-02-13
| | | | | | We need to keep separete counters to keep tracks of packet numbers with and without any payload. Regression introduced in 5849863ef
* Fix detection of packet direction and NDPI_UNIDIRECTIONAL_TRAFFIC risk (#1883)Ivan Nardi2023-02-13
|
* Centos7 fixesLuca Deri2023-02-13
|
* Update roaring bitmap codeLuca Deri2023-02-12
|
* Add Yandex services detection (#1882)0xA50C1A12023-02-09
| | | | | Add Yandex services detection Add VK and Yandex to the TLS certificate match list
* fuzz: some improvements and add two new fuzzers (#1881)Ivan Nardi2023-02-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from `fuzz/Makefile.am`; it is already included by the main configure script (when fuzzing). Add a knob to force disabling of AESNI optimizations: this way we can fuzz also no-aesni crypto code. Move CRC32 algorithm into the library. Add some fake traces to extend fuzzing coverage. Note that these traces are hand-made (via scapy/curl) and must not be used as "proof" that the dissectors are really able to identify this kind of traffic. Some small updates to some dissectors: CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting with "VS01" will be classified as STEAM (see steam.c around line 111). Googling it, it seems right so. XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore that condition is false. RTP, STUN: removed useless "break"s Zattoo: `flow->zattoo_stage` is never set to any values greater or equal to 5, so these checks are never true. PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it. TeamSpeak: we check for `flow->packet_counter == 3` just above, so the following check `flow->packet_counter >= 3` is always false.
* Add VK detection (#1880)0xA50C1A12023-02-02
|
* File rename to avoid name clashes (e.g. on Windows)Luca Deri2023-01-31
|
* Updated (C)Luca Deri2023-01-31
|
* fuzz: add a new fuzzer to test serialization/deserialization code (#1876)Ivan Nardi2023-01-27
| | | | | | | | | | Autodetecting the needed buffer size is quite complex (especially with float/double values) so it is mandatory to properly check for `ndpi_snprintf` truncation. These issues have been undetected so far probably because the default buffer is big enough for all common cases. Add an example of usage of `ndpi_deserialize_clone_all()` (taken from `ntopng`)
* Improved connection refused detectionLuca Deri2023-01-25
|
* Fix compilation and CI (#1875)Ivan Nardi2023-01-25
| | | | | | | | | | | | | | | | | | | | | ubuntu-latest s390x CI doesn't like snapshot bigger than 262144 bytes. Fix an error found by fuzz CI ``` ================================================================= ==55399==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x561e41cb684d bp 0x7ffd54ce3650 sp 0x7ffd54ce3520 T0) ==55399==The signal is caused by a READ memory access. ==55399==Hint: address points to the zero page. #0 0x561e41cb684d in ndpi_network_ptree_match /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2321:41 #1 0x561e41d30879 in ndpi_guess_undetected_protocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7767:8 #2 0x561e41ca804d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_config.cpp:104:5 #3 0x561e41bb96a0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4726a0) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #4 0x561e41ba3c2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x45cc2f) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #5 0x561e41ba96f6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4626f6) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #6 0x561e41bd22e2 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x48b2e2) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) #7 0x7f94f0e5c082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #8 0x561e41b9eb0d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x457b0d) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9) ```
* Update every ip lists (#1872)Ivan Nardi2023-01-25
|
* CenOS compilation fixLuca Deri2023-01-25
|
* Compilation fixesLuca Deri2023-01-25
|
* Add some fuzzers to test other data structures. (#1870)Ivan Nardi2023-01-25
| | | | | | | Start using a dictionary for fuzzing (see: https://llvm.org/docs/LibFuzzer.html#dictionaries). Remove some dead code. Fuzzing with debug enabled is not usually a great idea (from performance POV). Keep the code since it might be useful while debugging.
* Some small changes (#1869)Ivan Nardi2023-01-25
| | | | | | | | All dissector callbacks should not be exported by the library; make static some other local functions. The callback logic in `ndpiReader` has never been used. With internal libgcrypt, `gcry_control()` should always return no errors. We can check `categories` length at compilation time.
* Added new risk NDPI_TCP_ISSUESLuca Deri2023-01-24
|
* fuzz: fix memory allocation failure logic (#1867)Ivan Nardi2023-01-20
| | | | | We *do* want to have some allocation errors. Fix some related bugs Fix: 29be01ef
* Bittorrent: fix detection over TCP (#1868)Ivan Nardi2023-01-19
| | | Close #1866
* Minor changes in `ndpi_detection_giveup()` (#1861)Ivan Nardi2023-01-19
| | | | | | Check the mining cache at the end of the function, like all the others LRU caches. Rewrite the STUN checks: same identical semantic but in a cleaner way, hopefully
* Fix undefined-behaviour in ahocorasick callback (#1864)Ivan Nardi2023-01-19
| | | | | | | ``` ndpi_main.c:2119:28: runtime error: left shift of 1 by 31 places cannot be represented in type 'int' ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=55226
* Bittorrent: fix heap-buffer-overflow (#1863)Ivan Nardi2023-01-19
| | | | | | | | | | | | | | | | | ``` ==258287==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600068ff9d at pc 0x5653a6e35def bp 0x7ffeef5aa620 sp 0x7ffeef5a9dc8 READ of size 22 at 0x60600068ff9d thread T0 #0 0x5653a6e35dee in strncmp (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x4d2dee) (BuildId: 133b8c3c8ff99408109fcb9be2538bb8341f07f7) #1 0x5653a70d6624 in ndpi_search_bittorrent /home/ivan/svnrepos/nDPI/src/lib/protocols/bittorrent.c:500:71 #2 0x5653a6ff255a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5686:6 #3 0x5653a6ff331b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5722:10 #4 0x5653a6ff2cbc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5755:12 #5 0x5653a70016bf in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6578:15 #6 0x5653a6f1836d in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1678:31 #7 0x5653a6f140a1 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2256:10 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=55218 Fix: 470eaa6f
* Shrinked symbolic flow risks labelsLuca Deri2023-01-19
|
* LRU caches: add a generic (optional and configurable) expiration logic (#1855)Ivan Nardi2023-01-18
| | | Two caches already implemented a similar mechanism: make it generic.
* POP3: improve detection (#1856)Ivan Nardi2023-01-18
|
* Improve support for Snapchat voip calls (#1858)Ivan Nardi2023-01-17
| | | | Latest Snapchat versions use QUICv1 for their audio/video real time sessions. See c50a8d480
* Add detection of Tailscale (#1857)Ivan Nardi2023-01-17
| | | Extend the example of wireguard traffic
* Add Meraki Cloud protocol and improve other protocols (#1854)Ivan Nardi2023-01-17
| | | | | Avoid some LineCall and Jabber false positives. Detect Discord mid flows. Fix Bittorrent detection.
Powered by cgit, Themed by Ted Yin.