aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
...
* python: reformatted, fixed bugs (#2504)Petr2024-07-17
|
* .gitignore: reformatted, added patterns for IDEs, for deb packages and for ↵Petr2024-07-16
| | | | test results (#2503)
* Add HLS support (#2502)Vladimir Gavrilov2024-07-16
|
* Refactor ndpi_strnstr to use ndpi_memmem (#2500)Vladimir Gavrilov2024-07-15
|
* ndpi_memmem: optimized, fixed bug, added tests (#2499)Petr2024-07-15
|
* Optimize performance of ndpi_strnstr() and possible bugfix (#2494)Petr2024-07-15
|
* Fixed mistake in shebang (SC1113) (#2498)Petr2024-07-15
|
* fuzzing: improve coverage (#2495)Ivan Nardi2024-07-12
| | | | | | | | | | | | | | Fix detection of WebDAV and Gnutella (over HTTP) Fix detection of z3950 Add two fuzzers to test `ndpi_memmem()` and `ndpi_strnstr()` Remove some dead code: * RTP: the same exact check is performed at the very beginning of the function * MQTT: use a better helper to exclude the protocol * Colletd: `ndpi_hostname_sni_set()` never fails Update pl7m code (fix a Use-of-uninitialized-value error)
* Improve detection of Cloudflare WARP traffic (#2491)Ivan Nardi2024-07-04
| | | See: #2484
* Add infrastructure for explicit support of Fist Packet Classification (#2488)Ivan Nardi2024-07-03
| | | | | Let's start with some basic helpers and with FPC based on flow addresses. See: #2322
* Minor fix in CI action (#2489)Ivan Nardi2024-07-03
|
* Reduce snaplen of some traces (#2490)Ivan Nardi2024-07-03
| | | | | | To avoid the following error with some old libpcap versions: ``` ERROR: could not open pcap file: invalid file capture length 524288, bigger than maximum of 262144 ```
* Add detection of Twitter bot (#2487)Ivan Nardi2024-07-03
| | | Update the global list of crawlers ips
* Make the CI faster (#2475)Ivan Nardi2024-07-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | Without the `-fsanitize-memory-track-origins` flag, MSAN job is ~30% faster. Since this flag is useful only while debugging (and not to simply discover memory issues), avoid it on the CI. Note that, by default it is still enabled by default. Right now, MingW runs on *every* ubuntu builds: limit it only to the standard matrix (i.e. ubuntu 20.04, 22.04, 24.04 with default configuration), without any sanitizers (note that MingW doesn't support *san anyway). armhf job is by far the longest job in the CI: remove asan configuration to make it faster. Note that we already have a lot of different jobs (on x86_64) with some sanitizers, and that the other 2 jobs on arm/s390x don't have asan support anyway. If we really, really want a job with arm + asan we can add it as a async/scheduled job. Remove an old workaround for ubuntu jobs Avoid installing packages needed only for the documentation About `check_symbols.sh` script: even if uses the compiled library/objects, it basicaly only checks if we are using, in the source code, same functions that we shoudn't. We don't need to perform the same kind of check so many times..
* Modified separator from , (comma) to | (pipe) as some fields such as the ↵Luca Deri2024-07-01
| | | | HTTP user agent as sometimes they contain commas and create parsing problems
* tunnelbear: improve detection over wireguard (#2485)Ivan Nardi2024-07-01
| | | See #2484
* Improve detection of Twitter/X (#2482)Ivan Nardi2024-07-01
|
* Add detection of OpenAI ChatGPT bots (#2481)Ivan Nardi2024-07-01
|
* fuzz: improve fuzzers using pl7m (#2486)Ivan Nardi2024-06-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some unclear reasons, fuzzers using pl7m create huge corpus, triggering OOM in oss-fuzz runs (where the memory RSS limit is set to 2560Mb). Example: ``` ==25340== ERROR: libFuzzer: out-of-memory (used: 2564Mb; limit: 2560Mb) To change the out-of-memory limit use -rss_limit_mb=<N> Live Heap Allocations: 2364004039 bytes in 133791 chunks; quarantined: 60662293 bytes in 3664 chunks; 176432 other chunks; total chunks: 313887; showing top 95% (at most 8 unique contexts) 1285841683 byte(s) (54%) in 2956 allocation(s) #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0 #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3 #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9 #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10 #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16 #6 0x56f814de7df2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:829:7 #7 0x56f814de8127 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3 #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6 #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 1031350683 byte(s) (43%) in 2468 allocation(s) #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0 #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3 #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9 #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10 #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16 #6 0x56f814de7635 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19 #7 0x56f814de8425 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5 #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6 #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 ``` See: https://oss-fuzz.com/testcase-detail/4717811415449600 See: https://oss-fuzz.com/testcase-detail/6164130982068224 Let's *try* the following workaround: set the parameter `-max-len` to 512K, to force the engine to not genereate inputs (i.e. pcap files...) larger than 512K. Right now the value used is 1MB, i.e the default, because we have file larger than 1MB in the initial seeds (i.e. `/tests/pcaps/*`). Let's hope than having smaller files lead to smaller corpus... Update pl7m code (fix a Use-of-uninitialized-value error)
* fuzz: pl7m: add a custom mutator for better fuzzing of pcap files (#2483)Ivan Nardi2024-06-27
| | | | | | | | | | | Pl7m is a custom mutator (used for structure aware fuzzing) for network traffic packet captures (i.e. pcap files). The output of the mutator is always a valid pcap file, containing the same flows/sessions of the input file. That's it: the mutator only changes the packet payload after the TCP/UDP header, keeping all the original L2/L3 information (IP addresses and L4 ports). See: https://github.com/IvanNardi/pl7m
* Remove duplicate API for `ndpi_serialize_uint32_double` in ndpi_api.h (#2480)Anshul Thakur2024-06-26
| | | Co-authored-by: Anshul Thakur <athakur@cdot.in>
* wireshark: lua: export some metadataNardi Ivan2024-06-25
| | | | | | | | | | | | | | | Export some metadata (for the moment, SNI and TLS fingerprints) to Wireshark/tshark via extcap. Note that: * metadata are exported only once per flow * metadata are exported (all together) when nDPI stopped processing the flow Still room for a lot of improvements! In particular: * we need to add some boundary checks (if we are going to export other attributes) * we should try to have a variable length trailer
* wireshark: lua: filtering and trailer dissection work with tshark, tooNardi Ivan2024-06-25
| | | | | | | | | | | | | ``` ivan@ivan-Latitude-E6540:~/$ tshark -C "nDPI extcap" -i ndpi -o extcap.ndpi.i:/home/ivan/svnrepos/nDPI/tests/pcap/anydesk.pcapng -Y "ndpi.protocol.name contains DNS" Capturing on 'nDPI interface: ndpi' 62 22635386.425683 192.168.1.187 DNS.AnyDesk 192.168.1.1 128 Standard query 0xec22 A relay-3185a847.net.anydesk.com 63 22635386.439540 192.168.1.1 DNS.AnyDesk 192.168.1.187 144 Standard query response 0xec22 A relay-3185a847.net.anydesk.com A 37.61.223.15 64 22635386.721277 192.168.1.187 DNS.AnyDesk 192.168.1.1 128 Standard query 0xea89 A relay-9b6827f2.net.anydesk.com 65 22635386.732444 192.168.1.1 DNS.AnyDesk 192.168.1.187 144 Standard query response 0xea89 A relay-9b6827f2.net.anydesk.com A 138.199.36.115 4 packets captured ```
* wireshark: lua: fix DNS dissectionNardi Ivan2024-06-25
| | | | | Not sure when we (or Wireshark, or Lua...) broke it, but we can't call tonumber() on Bool variables.
* wireshark: extcap: restore filtering mechanismNardi Ivan2024-06-25
|
* wireshark: lua: latest Wireshark versions correctly handle 64 bit maskNardi Ivan2024-06-25
|
* wireshark: lua: minor improvementsNardi Ivan2024-06-25
| | | | | * Use a proper TVB to parse the nDPI trailer * Fix some flow risks definitions
* Improved logic for checking invalid DNS queriesLuca Deri2024-06-23
|
* Added default port mappings to ndpiReader help -H (#2477)Mark Jeffery2024-06-19
| | | Close #2125
* Zoom: fix detection of screen sharing (#2476)Ivan Nardi2024-06-17
|
* fuzz: improve fuzzing coverage (#2474)Ivan Nardi2024-06-17
| | | | | | Remove some code never triggered AFP: the removed check is included in the following one MQTT: fix flags extraction
* Sync unit tests resultsNardi Ivan2024-06-17
|
* Add Ripe Atlas probe protocol. (#2473)Toni2024-06-17
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Zoom: harden RTP/RTCP detectionNardi Ivan2024-06-17
|
* Zoom: remove "stun_zoom" LRU cacheNardi Ivan2024-06-17
| | | | | Since 070a0908b we are able to detect P2P calls directly from the packet content, without any correlation among flows
* Update main Readme and remove commented out code (#2471)Ivan Nardi2024-06-17
| | | | Avoid that useless code is copy-and-pasted into new dissector, as in https://github.com/ntop/nDPI/pull/2470#discussion_r1639384434
* Added protocol - JRMI - Java Remote Method Invocation (#2470)Mark Jeffery2024-06-15
|
* Improved detection of Android connectiity checksLuca2024-06-12
|
* Added NDPI_PROTOCOL_NTOP assert and removed percentage comparison (#2460)Mark Jeffery2024-06-10
| | | Close #2413
* Zoom: fix integer overflow (#2469)Ivan Nardi2024-06-10
| | | | | | | | | | | | | | ``` AddressSanitizer:DEADLYSIGNAL ================================================================= ==29508==ERROR: AddressSanitizer: SEGV on unknown address 0x50710145d51d (pc 0x55cb788f25fe bp 0x7ffcfefa15f0 sp 0x7ffcfefa1240 T0) ==29508==The signal is caused by a READ memory access. #0 0x55cb788f25fe in ndpi_search_zoom /home/ivan/svnrepos/nDPI/src/lib/protocols/zoom.c:210:24 #1 0x55cb787e9418 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7174:6 #2 0x55cb7883f753 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7209:10 #3 0x55cb7883bc9d in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7240:12 ``` Found by oss-fuzzer See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=69520
* RTP/STUN: look for STUN packets after RTP/RTCP classification (#2465)Ivan Nardi2024-06-07
| | | | | | | | | | After a flow has been classified as RTP or RTCP, nDPI might analyse more packets to look for STUN/DTLS packets, i.e. to try to tell if this flow is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with STUN/DTLS. Useful for proper (sub)classification when the beginning of the flows are not captured or if there are lost packets in the the captured traffic. Disabled by default
* Zoom: faster detection of P2P flows (#2467)Ivan Nardi2024-06-07
|
* STUN: add support for Microsoft Multiplexed TURN channels (#2464)Ivan Nardi2024-06-05
|
* TLS: add support for DTLS (over STUN) over TCP (#2463)Ivan Nardi2024-06-05
| | | | | TODO: TCP reassembler on top of UDP reassembler See: #2414
* CI: more parallel work (#2459)Ivan Nardi2024-06-05
|
* Update unit tests results (#2466)Ivan Nardi2024-06-05
|
* Fixes accounting of number of processed packets in ↵Luca Deri2024-06-04
| | | | ndpi_internal_detection_process_packet()
* RTP: fix detection over TCP (#2462)Ivan Nardi2024-05-29
| | | | | | RFC4571 is not the only way to wrap RTP messages in TCP streams. For example, when RTP is encapsulated over TURN flows (i.e. via DATA attribute) there is no additional framing. See also 6127e0490
* support rtp/rtcp over tcp (#2422) (#2457)Maatuq2024-05-28
| | | | | Support rtp/rtcp over tcp as per rfc4571. Signed-off-by: mmaatuq <mahmoudmatook.mm@gmail.com>
* Add ZUG consensus protocol dissector. (#2458)Toni2024-05-28
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.