aboutsummaryrefslogtreecommitdiff
path: root/example
Commit message (Collapse)AuthorAge
* Extend tests coverage (#1476)Ivan Nardi2022-03-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Now there is at least one flow under `tests/pcap` for 249 protocols out of the 284 ones supported by nDPI. The 35 protocols without any tests are: * P2P/sharing protocols: DIRECT_DOWNLOAD_LINK, OPENFT, FASTTRACK, EDONKEY, SOPCAST, THUNDER, APPLEJUICE, DIRECTCONNECT, STEALTHNET * games: CSGO, HALFLIFE2, ARMAGETRON, CROSSFIRE, DOFUS, FIESTA, FLORENSIA, GUILDWARS, MAPLESTORY, WORLD_OF_KUNG_FU * voip/streaming: VHUA, ICECAST, SHOUTCAST, TVUPLAYER, TRUPHONE * other: AYIYA, SOAP, TARGUS_GETDATA, RPC, ZMQ, REDIS, VMWARE, NOE, LOTUS_NOTES, EGP, SAP Most of these protocols (expecially the P2P and games ones) have been inherited by OpenDPI and have not been updated since then: even if they are still used, the detection rules might be outdated. However code coverage (of `lib/protocols`) only increases from 65.6% to 68.9%. Improve Citrix, Corba, Fix, Aimini, Megaco, PPStream, SNMP and Some/IP dissection. Treat IPP as a HTTP sub protocol. Fix Cassandra false positives. Remove `NDPI_PROTOCOL_QQLIVE` and `NDPI_PROTOCOL_REMOTE_SCAN`: these protocol ids are defined but they are never used. Remove Collectd support: its code has never been called. If someone is really interested in this protocol, we can re-add it later, updating the dissector. Add decoding of PPI (Per-Packet Information) data link type.
* Errors fixed (#1482)Vitaly Lavrov2022-03-08
| | | | | | | | | | | | | | | Fixed errors for bigendian platforms in ndpiReader. All address and port comparisons and hash calculations are done with endian in mind. The get_ndpi_flow_info() function searched for an existing flow for the forward and reverse direction of the packet. The ndpi_workflow_node_cmp() function looked for a flow regardless of the packet's direction. This is what led to an error in determining the direction of transmission of the packet. Fixed error in "synscan" test: the number of packets in the forward and reverse direction is incorrectly defined (verified via tcpdump). Fixed bug with icmp protocol checksum check for big endian platforms.
* reader_util: fix parsing of MPLS packets (#1467)Ivan Nardi2022-03-02
| | | | | | | | | | | | | | ``` ==20492==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000578c at pc 0x55c47455e3ea bp 0x7ffc62ca1eb0 sp 0x7ffc62ca1ea8 READ of size 4 at 0x60300000578c thread T0 #0 0x55c47455e3e9 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:1840:16 #1 0x55c47451b9cd in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 #2 0x55c47451c1ab in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17 #3 0x7f661b50e0b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16 #4 0x55c47445b54d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x61054d) (BuildId: eba4bb4cd43b7101e4f0028ec0fb79087bae0e37) 0x60300000578d is located 0 bytes to the right of 29-byte region [0x603000005770,0x60300000578d) ```
* Bug fixing. (#1459)Vitaly Lavrov2022-02-28
| | | | | The '--enable-debug-messages' option works again. Fixed warning in ahocorasick.c Fixed integer overflow in ndpiReader.c for 32bit systems.
* reader_util: fix TZSP decapsulation (#1460)Ivan Nardi2022-02-27
| | | | | | | | | | | | | | | | | ``` ==38674==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60400008957f at pc 0x5653fcee6434 bp 0x7ffe9b554b50 sp 0x7ffe9b554b48 READ of size 1 at 0x60400008957f thread T0 #0 0x5653fcee6433 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2050:18 #1 0x5653fce9fc0d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 #2 0x5653fcea03eb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17 #3 0x7fe71dc3d0b2 in __libc_start_main /build/glibc-YbNSs7/glibc-2.31/csu/../csu/libc-start.c:308:16 #4 0x5653fcddf67d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x58a67d) (BuildId: 525418a27e8c37d6c492cc3220e0e97809c40f98) 0x60400008957f is located 0 bytes to the right of 47-byte region [0x604000089550,0x60400008957f) allocated by thread T0 here: ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=45036
* reader_util: fix parsing of IPv6 extension headers (#1453)Ivan Nardi2022-02-22
| | | | Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44942
* reader_util: fix infinite loop in packet dissection (#1454)Ivan Nardi2022-02-22
| | | | Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44944
* fuzz: purge old sessions (#1451)Ivan Nardi2022-02-21
| | | | | | | | At every fuzz iteration (i.e for every trace file): * keep the same ndpi context (`ndpi_init_detection_module` is very slow); * reset the flow table, otherwise it grows indefinitely. This change should fix the "out-of-memory" errors reported by oss-fuzz.
* Added newflow risk NDPI_HTTP_CRAWLER_BOTLuca Deri2022-02-17
|
* Increment current/total number of active flows on successful flow insertion ↵ol-andreyizrailev2022-02-09
| | | | | | | | (#1434) Memory allocation or ndpi_tsearch might fail, so the two values should be incremented only when insertion actually happened. Co-authored-by: Andrey Izrailev <Andrey.Izrailev@oktetlabs.ru>
* Remove `struct ndpi_id_struct` (#1427)Ivan Nardi2022-01-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the last uses of `struct ndpi_id_struct`. That code is not really used and it has not been updated for a very long time: see #1279 for details. Correlation among flows is achieved via LRU caches. This change allows to further reduce memory consumption (see also 91bb77a8). At nDPI 4.0 (more precisly, at a6b10cf, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 235.27 KB Flow Memory (per flow): 688 B <-------- ``` i.e. memory usage per flow has been reduced by 77%. Close #1279
* Fix some race conditions by using atomic operations. (#1420)Toni2022-01-29
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Make some protocols more "big-endian" friendly (#1402)Ivan Nardi2022-01-29
| | | See #1312
* Kerberos, TLS, example: fix some memory errors (#1419)Ivan Nardi2022-01-27
| | | | | | Detected by oss-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43823 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43921 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43925
* Fixed wrong ip tuple comparison. #1386 (#1418)Toni2022-01-26
| | | | | | | * Added u32 pads to `union ip_tuple` so btree search should now work as expected. The bug caused new flow's when the remote answers, resulting in two Flows per direction. Fail. * Fixed a race condition during shutdown phase. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* TLS, H323, examples: fix some memory errors (#1414)Ivan Nardi2022-01-18
| | | | | | | Detected by oss-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26880 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26906 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43782 https://oss-fuzz.com/testcase-detail/6334089358082048
* build: respect environment options more (#1392)Sam James2022-01-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * build: update m4/ax_pthread.m4 from serial 23 -> serial 31 Update ax_pthread.m4 to the latest version from the autoconf-archive project. Signed-off-by: Sam James <sam@gentoo.org> * build: properly detect AR, CC, RANLIB It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain variables/tools for cross-compilation, testing with other toolchains, and to ensure the compiler chosen by the user is actually used for the build. Previously, GNU_PREFIX was kind-of used for this but this isn't a standard variable (at all) and it wasn't applied consistently anyway. We now use the standard autoconf mechanisms for finding these tools. (RANLIB is already covered by LT_INIT.) Signed-off-by: Sam James <sam@gentoo.org> * build: use $(MAKE) This ensures that parallel make works correctly, as otherwise, a fresh make job will be started without the jobserver fd, and hence not know about its parent, forcing -j1. * build: respect CPPFLAGS, LDFLAGS - CPPFLAGS is for the C preprocessor (usually for setting defines) - LDFLAGS should be placed before objects for certain flags to work (e.g. -Wl,--as-needed) Signed-off-by: Sam James <sam@gentoo.org> Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Adds some risk exceptions for popular services and domain namesLuca2022-01-17
| | | | via a new (internal) function named ndpi_add_domain_risk_exceptions()
* Minor cosmetic changesLuca Deri2022-01-16
|
* Added the ability to specify trusted issueDN often used in companies to ↵Luca Deri2022-01-13
| | | | | | | | | | | self-signed certificates This allows to avoid triggering alerts for trusted albeit private certificate issuers. Extended the example/protos.txt with the new syntax for specifying trusted issueDN. Example: trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
* Removed outdated commentLuca Deri2022-01-11
|
* Removed legacy codeLuca Deri2022-01-11
|
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Improved user agent analysisLuca Deri2022-01-09
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Fix configure script (after fb85dac9) (#1381)Ivan Nardi2021-12-04
| | | Fix/disable some LGTM warnings
* Added example for finding similarities in RRDs using nDPI statistical APIsLuca Deri2021-12-04
|
* ndpiReader: slight simplificaton of the output (#1378)Ivan Nardi2021-11-27
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Reduce memory used by `ndpiReader` (#1371)Ivan Nardi2021-11-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `ndpiReader` is only an example, aiming to show nDPI capabilities and integration, without any claim about performances. Nonetheless its memory usage per flow is *huge*, limiting the kinds of traces that we can test on a "normal" hardware (example: scan attacks). The key reason of that behaviour is that we preallocate all the memory needed for *all* the available features. Try to reduce memory usage simply allocating some structures only when they are really needed. Most significant example: JOY algorithms. This way we should use a lot less memory in the two most common user-cases: * `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i $FILENAME_OR_IFACE`) * internal unit tests Before (on x86_64): ``` struct ndpi_flow_info { [...] /* size: 7320, cachelines: 115, members: 72 */ ``` After: ``` struct ndpi_flow_info { [...] /* size: 2128, cachelines: 34, members: 75 */ ```
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Implemented ndpi_ses_fitting() and ndpi_des_fitting()Luca Deri2021-10-12
| | | | for comuting the best alpha/beta values for exponential smoothing
* Fix compilation with clang-13 or if some debug macros are enabled (#1326)Ivan Nardi2021-10-06
|
* Improved DGA detection for skipping potential DGAs of known/popular domain namesLuca Deri2021-10-05
|
* Added -a <num> to ndpiReader for generating OPNsense configurationLuca Deri2021-10-04
| | | | See https://github.com/ntop/opnsense
* Removed traceLuca Deri2021-10-03
|
* Fix warningAlfredo Cardigliano2021-09-28
|
* Added unit test for bitmap iterationLuca Deri2021-09-27
|
* Warnign fixLuca Deri2021-09-27
|
* Added API for handling compressed bitmapsLuca Deri2021-09-26
| | | | | | | | | | | | | | ndpi_bitmap* ndpi_bitmap_alloc(); void ndpi_bitmap_free(ndpi_bitmap* b); u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b); void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value); bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_clear(ndpi_bitmap* b); size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf); ndpi_bitmap* ndpi_bitmap_deserialize(char *buf); based on https://github.com/RoaringBitmap/CRoaring
* Fix parsing of ipv6 packets with extension headersNardi Ivan2021-09-19
| | | | | | | | Decoding of ipv6 traffic with extension headers was completely broken, since the beginning of the L4 header was always set to a wrong value. Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first one and drop the others.
* Fix unused varAlfredo Cardigliano2021-09-03
|
* Fixed memory statsLuca Deri2021-08-26
|
* Compilation fixLuca Deri2021-08-20
|
* Fix access to some TLS fields in flow structure (#1277)Ivan Nardi2021-08-20
| | | | Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by QUIC (i.e UDP...), too.
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Fixed Mingw64 build, SonerCloud-CI and more. (#1273)Toni2021-08-18
| | | | | | | | | | | | | * Added ARM build and unit test run for SonarCloud-CI. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Fixed Mingw64 build. * adapted to SonarCloud-CI workflow * removed broken and incomplete Windows example (tested on VS2017/VS2019) * removed unnecessary include (e.g. pthread.h for the library which does not make use of it) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.