aboutsummaryrefslogtreecommitdiff
path: root/src/lib
Commit message (Collapse)AuthorAge
* Fixed msys2 build warnings and re-activated CI Mingw64 build.fix/windows-msys2Toni Uhlig2022-04-14
| | | | | | | * Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com>
* Kerberos: fix some memory access errors (#1514)Ivan Nardi2022-04-13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ``` ==19724==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00000045e at pc 0x5620b8b3d3cc bp 0x7ffe0fda6b50 sp 0x7ffe0fda6310 READ of size 2 at 0x60e00000045e thread T0 #0 0x5620b8b3d3cb in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x63f3cb) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6) #1 0x5620b8d9b69c in strncpy_lower /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:208:4 #2 0x5620b8d995a0 in krb_parse /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:316:5 #3 0x5620b8d97a90 in ndpi_search_kerberos /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:687:12 #4 0x5620b8bcef35 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4996:4 #5 0x5620b8bd1be8 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5072:10 #6 0x5620b8bd159c in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5105:12 #7 0x5620b8be323a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15 #8 0x5620b8b8f7e0 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:3 #9 0x5620b8b8fd1b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:84:17 #10 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16 #11 0x5620b8acf47d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5d147d) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6) 0x60e00000045e is located 0 bytes to the right of 158-byte region [0x60e0000003c0,0x60e00000045e) allocated by thread T0 here: #0 0x5620b8b5283e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x65483e) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6) #1 0x5620b8b8fc86 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:70:17 #2 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16 ``` ``` protocols/kerberos.c:79:52: runtime error: left shift of 255 by 24 places cannot be represented in type 'int' ``` Found by oss-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46670 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46636
* Extended list of cybersecurity domainsLuca Deri2022-04-13
|
* fix(ndpi_main):Fix memory leak about ndpi_str; (#1513)Wayne2022-04-12
| | | Co-authored-by: 林文烽 <wenfeng.lin@baishan.com>
* TINC: fix invalid memory read (#1512)Ivan Nardi2022-04-10
| | | | | | | | | | | | | | | | | | | | | | | | | ``` ================================================================= ==19324==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600061be96 at pc 0x55b4a4cb4460 bp 0x7ffc7b461a70 sp 0x7ffc7b461a68 READ of size 1 at 0x60600061be96 thread T0 #0 0x55b4a4cb445f in ndpi_check_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:105:9 #1 0x55b4a4cb1888 in ndpi_search_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:135:5 #2 0x55b4a4b4a6e1 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5013:6 #3 0x55b4a4b4c2d4 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5084:12 #4 0x55b4a4b4bf77 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5103:12 #5 0x55b4a4b5dcca in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15 #6 0x55b4a4a87734 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1519:31 #7 0x55b4a4a80761 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2093:10 #8 0x55b4a4a39c8d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 #9 0x55b4a4a3a46b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17 #10 0x7f69c63760b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16 #11 0x55b4a497954d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x61654d) (BuildId: 705ebc5c412d267294a65cb01f03a1f012aeaf20) 0x60600061be96 is located 0 bytes to the right of 54-byte region [0x60600061be60,0x60600061be96) allocated by thread T0 here: [...] ``` Found by oss-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46499
* Improved ASN.1 parsing for Keberos. Fixes #1492. (#1497)Toni2022-04-10
| | | | | | * This is a quick fix, the Kerberos protocol dissector requires some refactoring effort. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com>
* QUIC: handle retransmissions and overlapping fragments in reassembler ↵Vinicius Silva Nogueira2022-04-07
| | | | | | | | | | | | | | | | | | | | | (#1195) (#1498) * QUIC: handle retransmissions and overlapping fragments in reassembler * Trigger CI * minor fix: parentheses * Changing ndpi_malloc to ndpi_calloc * fix memory leak * quic_reasm_buf calloc to malloc * change order of is_ch_complete && is_reasm_buf_complete call * is_reasm_buf_complete: added handling for case where frame size is not multiple of 8 * add extra check
* Added ndpi_find_outliers() API call using Z-ScoreLuca Deri2022-04-04
|
* Removed un-necessary guess in miningLuca Deri2022-04-01
|
* DGA improvementsLuca Deri2022-04-01
|
* Improvements for CUSTOM_NDPI_PROTOCOLSLuca Deri2022-04-01
|
* Moved geneated file to a separate folderLuca Deri2022-04-01
|
* Improved twitter detectionLuca Deri2022-04-01
|
* Removed SRV record from suspicious DNS trafficLuca Deri2022-03-31
|
* Improved DGA detectionLuca Deri2022-03-31
|
* Add support for Pluralsight site (#1503)Ivan Nardi2022-03-27
|
* Reducing the size of the ndpi_detection_module_struct structure. (#1490)Vitaly Lavrov2022-03-27
| | | | | | | | | | | | | | The ndpi_detection_module_struct structure contains 5 arrays "struct ndpi_call_function_struct" size 286*144=41 kB size, which are occupied by a small number of elements. At the moment we have callback_buffer_size = 172, tcp_with_payload=114, tcp_no_payload=8, udp=93, other 8. NDPI_MAX_SUPPORTED_PROTOCOLS = 285. Size of struct ndpi_detection_module_struct is 253136 bytes. Size of all structs ndpi_call_function_struct 5*286*144=205920 bytes. Real use memory size for struct ndpi_call_function_struct is (173+224)*144=57168 bytes.
* [SSDP] Extract HTTP user-agent when available. (#1500)Darryl Sokoloski2022-03-27
| | | | | | [SSDP] Added capture file with UA header. [SSDP] Added pcap test output log file. Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
* Improved DGA detection skipping names containign at least 3 consecutive ↵Luca Deri2022-03-26
| | | | digits in the first word
* QUIC: add support for version 2 draft 01 (#1493)Ivan Nardi2022-03-25
| | | | | | Support for v2-00 has been removed (it has never been used in real networks and it is incompatible with v2-01). Chrome already supports v2-01 in latest versions in Chrome Beta channel.
* Mining: cleanup registration (#1496)Ivan Nardi2022-03-25
| | | | | Use the same pattern of all the other dissectors: one registration and one callback. Spotted by @dsokoloski
* Trying to improve QUIC reassembler (#1195) (#1489)Vinicius Silva Nogueira2022-03-24
| | | | | | | | | | | * handling QUIC out-of-order fragments * minor fix * updated quic_frags_ch_out_of_order_same_packet_craziness.pcapng.out * quic test: buf_len + last_pos * QUIC: comment update in __reassemble function and minor change is_ch_complete function
* Extended the list of cybersecurity protocolLuca Deri2022-03-21
|
* Bug fixing. (#1487)Vitaly Lavrov2022-03-15
| | | Using the protocol_id instead of its index.
* QUIC: convert logs to standard mechanism (#1485)Ivan Nardi2022-03-15
|
* QUIC: fix dissection of draft-34 (#1484)dev-1Ivan Nardi2022-03-09
| | | | QUIC-34 is probably not used in production, but fixing it is trivial and it doesn't add any noise to the already complex QUIC code.
* Extend tests coverage (#1476)Ivan Nardi2022-03-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Now there is at least one flow under `tests/pcap` for 249 protocols out of the 284 ones supported by nDPI. The 35 protocols without any tests are: * P2P/sharing protocols: DIRECT_DOWNLOAD_LINK, OPENFT, FASTTRACK, EDONKEY, SOPCAST, THUNDER, APPLEJUICE, DIRECTCONNECT, STEALTHNET * games: CSGO, HALFLIFE2, ARMAGETRON, CROSSFIRE, DOFUS, FIESTA, FLORENSIA, GUILDWARS, MAPLESTORY, WORLD_OF_KUNG_FU * voip/streaming: VHUA, ICECAST, SHOUTCAST, TVUPLAYER, TRUPHONE * other: AYIYA, SOAP, TARGUS_GETDATA, RPC, ZMQ, REDIS, VMWARE, NOE, LOTUS_NOTES, EGP, SAP Most of these protocols (expecially the P2P and games ones) have been inherited by OpenDPI and have not been updated since then: even if they are still used, the detection rules might be outdated. However code coverage (of `lib/protocols`) only increases from 65.6% to 68.9%. Improve Citrix, Corba, Fix, Aimini, Megaco, PPStream, SNMP and Some/IP dissection. Treat IPP as a HTTP sub protocol. Fix Cassandra false positives. Remove `NDPI_PROTOCOL_QQLIVE` and `NDPI_PROTOCOL_REMOTE_SCAN`: these protocol ids are defined but they are never used. Remove Collectd support: its code has never been called. If someone is really interested in this protocol, we can re-add it later, updating the dissector. Add decoding of PPI (Per-Packet Information) data link type.
* Improved ASN/IP update scripts and CI integration. (#1474)Toni2022-03-09
| | | | | * CI will print a warning if ASN/IP addresses changed. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Some small fixes (#1481)Ivan Nardi2022-03-08
| | | | | | FTP: if the authentication fails, stop analyzing the flow WSD: call the initialization routine; the dissector code has never been triggered MINING: fix dissection
* Errors fixed (#1482)Vitaly Lavrov2022-03-08
| | | | | | | | | | | | | | | Fixed errors for bigendian platforms in ndpiReader. All address and port comparisons and hash calculations are done with endian in mind. The get_ndpi_flow_info() function searched for an existing flow for the forward and reverse direction of the packet. The ndpi_workflow_node_cmp() function looked for a flow regardless of the packet's direction. This is what led to an error in determining the direction of transmission of the packet. Fixed error in "synscan" test: the number of packets in the forward and reverse direction is incorrectly defined (verified via tcpdump). Fixed bug with icmp protocol checksum check for big endian platforms.
* EthernetIP: fix integer conversion on big-endian archs (#1477)Ivan Nardi2022-03-05
|
* Fixed a bug for BE architectures (#1478)Vitaly Lavrov2022-03-05
| | | Fixed a bug in the internal implementation of libgcrypt for bigendian architectures
* configure: fix usage of libgpg-error with `--with-local-libgcrypt` (#1472)Ivan Nardi2022-03-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Right now, using external libgcrypt, nDPI is not linked to libgpg-error because configure script never checks for it. ``` ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ CC=gcc-11 CXX=g++-11 CFLAGS="-O3 -g -Werror" ./autogen.sh --enable-debug-messages --with-pcre --with-local-libgcrypt && make -s -j [...] checking for numa_available in -lnuma... yes checking for pcap_open_live in -lpcap... yes checking for pthread_setaffinity_np in -lpthread... yes checking for gcry_cipher_checktag in -lgcrypt... yes <------- missing check for libgpg-error checking for pcre_compile in -lpcre... yes checking that generated files are newer than configure... done [...] ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ grep HAVE_LIBGPG_ERROR src/include/ndpi_config.h /* #undef HAVE_LIBGPG_ERROR */ ``` Make both libgcrypt and libgpg-error mandatory if `--with-local-libgcrypt` is used. Technically speaking, libgpg-error might be optional, because it is used only for debug messages. However having both libraries mandatory slightly simplified the logic. In most environments, libgpg-error is a dependency of libgcrypt anyway, so having both libraries should be the standard case.
* Drop support for non-gcrypt builds. (#1469)Toni2022-03-02
| | | | | | | | * As there is now a builtin, lightweight libgcrypt there is no need to disable tls-clho decryption. * It is still possible to use a host libgcrypt with `--with-local-libgcrypt'. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Internal crypto: increase size of authentication buffer (#1468)Ivan Nardi2022-03-02
| | | | | | | Some QUIC flows are not properly decoded while using internal crypto code: the authentication buffer is too small. The new value (like the old one) is arbitrary. Close #1463
* Add ICMP checksum check and set risk if mismatch detected. (#1464)Toni2022-03-02
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added configureable ndpi packet processing limit. (#1466)Toni2022-03-02
| | | | | | | | | * The current behaviour ignores any user preferences and was also incorrectly implemented, because the flow->num_processed_pkts wraps every 65535 and nDPI will process packets again until NDPI_MAX_NUM_PKTS_PER_FLOW_TO_DISSECT reached. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix libgcrypt(-light/-internal) compile error. (#1465)Toni2022-03-02
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add a new flow risk `NDPI_ANONYMOUS_SUBSCRIBER` (#1462)Ivan Nardi2022-02-28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main goal of a DPI engine is usually to determine "what", i.e. which types of traffic flow on the network. However the applications using DPI are often interested also in "who", i.e. which "user/subscriber" generated that traffic. The association between a flow and a subscriber is usually done via some kind of DHCP/GTP/RADIUS/NAT mappings. In all these cases the key element of the flow used to identify the user is the source ip address. That usually happens for the vast majority of the traffic. However, depending on the protocols involved and on the position on the net where the traffic is captured, the source ip address might have been changed/anonymized. In that case, that address is useless for any flow-username association. Example: iCloud Private Relay traffic captured between the exit relay and the server. See the picture at page 5 on: https://www.apple.com/privacy/docs/iCloud_Private_Relay_Overview_Dec2021.PDF This commit adds new generic flow risk `NDPI_ANONYMOUS_SUBSCRIBER` hinting that the ip addresses shouldn't be used to identify the user associated with the flow. As a first example of this new feature, the entire list of the relay ip addresses used by Private Relay is added. A key point to note is that list is NOT used for flow classification (unlike all the other ip lists present in nDPI) but only for setting this new flow risk. TODO: IPv6
* Removed some unused fields (#1461)Ivan Nardi2022-02-28
|
* Bug fixing. (#1459)Vitaly Lavrov2022-02-28
| | | | | The '--enable-debug-messages' option works again. Fixed warning in ahocorasick.c Fixed integer overflow in ndpiReader.c for 32bit systems.
* Add some scripts to easily update some IPs lists (#1449)Ivan Nardi2022-02-25
| | | | | | | | | | | While the lists in a6ff0dd0 and 2f5f445f are somehow provided by the companies themselves (or by some interested parties), these new lists are directly extracted from BGP information, via AS prefixes. *Usually*, these new lists are far more stable than the previous ones. TODO: * add some other ASNs (see `src/lib/ndpi_content_match.c.inc`) * IPv6, as usual :-(
* Provide some API functions for convenience. (#1456)Toni2022-02-25
| | | | | * Extended JSON serializsation: risk, risk score, confidence Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Win fixesLuca Deri2022-02-23
|
* Replaced strdup with ndpi_strupLuca Deri2022-02-23
|
* Directly drop malformed packets (#1455)Ivan Nardi2022-02-22
| | | | TCP/UDP/ICMP/ICMPv6 packets with invalid L4 header length should be ignored.
* DTLS: fix access to certificate cache (#1450)Ivan Nardi2022-02-21
| | | | | | | | | | | | | | | | ``` protocols/tls.c:650:54: runtime error: member access within null pointer of type 'const struct ndpi_tcphdr' SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in protocols/tls.c:650:54: runtime error: load of null pointer of type 'const u_int16_t' (aka 'const unsigned short') SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in AddressSanitizer:DEADLYSIGNAL ================================================================= ==47401==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x55f7a61b661c bp 0x7f38190f91b0 sp 0x7f38190f70e0 T1) ==47401==The signal is caused by a READ memory access. ==47401==Hint: address points to the zero page. #0 0x55f7a61b661c in processCertificateElements /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:650:41 #1 0x55f7a61ac3cc in processCertificate /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:792:7 #2 0x55f7a61d34e1 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:846:13 ```
* EthernetIP: add missing initialization (#1448)Ivan Nardi2022-02-20
| | | Fix:1e1cfb89
* Add support for Google Cloud (#1447)Ivan Nardi2022-02-20
| | | | Differentiate between Google its own apps/services and Google Cloud. We already do something similar for Amazon vs AWS and Microsoft vs Azure.
* Added lightweight implementation of libgcrypt. (#1444)Vitaly Lavrov2022-02-20
| | | | | | | | | | | | | | | | | | | | | Implementation borrowed from the https://github.com/ARMmbed/mbedtls.git project (v3.1.0) Speed testing (Xeon(R) CPU E3-1230 V2 @ 3.30GHz): gcrypt-gnu Test md 2897 ms enc 2777 ms dec 942 ms gcrypt-int Test md 3668 ms enc 1312 ms dec 2836 ms gcrypt-int-noaesni Test md 3652 ms enc 1916 ms dec 4458 ms gcrypt-gnu-nonopt Test md 3763 ms enc 4978 ms dec 3999 ms gcrypt-gnu-nonopt - libgcrypt compiled without hardware acceleration --disable-padlock-support --disable-aesni-support \ --disable-shaext-support --disable-pclmul-support \ --disable-sse41-support --disable-drng-support \ --disable-avx-support --disable-avx2-support \ --disable-neon-support --disable-arm-crypto-support \ --disable-ppc-crypto-support --disable-amd64-as-feature-detection
Powered by cgit, Themed by Ted Yin.