aboutsummaryrefslogtreecommitdiff
path: root/src/lib/ndpi_main.c
Commit message (Collapse)AuthorAge
...
* Fix access to packet/flow information (#2013)Ivan Nardi2023-06-17
| | | We can't access to `ndpi_str->packet` from `ndpi_detection_giveup()`
* Reworked teams handlingLuca Deri2023-06-15
|
* Fix compilation (#2011)Ivan Nardi2023-06-15
|
* Implemented Zoom/Teams stream type detectionLuca Deri2023-06-14
|
* Fix missing u_char, u_short and u_int typedefs for some platforms e.g.: (#2009)Toni2023-06-12
| | | | | | | | | | In file included from ../include/ndpi_main.h:34, from ../include/ndpi_api.h:28, from protocols/activision.c:26: ../include/ndpi_typedefs.h:294:3: error: unknown type name 'u_char' 294 | u_char h_dest[6]; /* destination eth addr */ | ^~~~~~ Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* ProtonVPN: add basic detection (#2006)Ivan Nardi2023-06-08
|
* Add bitcoing protocol dissector. (#1992)Maatuq2023-05-31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add bitcoing protocol dissector. * remove bitcoin protcol detection from mining.c * add a new bitcoin deissector. * add a new category: Cryptocurrency. Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com> * Remove useless checks and add missing windows and docs file. Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com> * update affected tests. Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com> * add a brief version. Add notes on the difference between normal bitcoin protocol and the mining protocol. Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com> * update enable_payload_stat test after dev rebasing. Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com> --------- Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
* Changed logging callback function sig. (#2000)Toni2023-05-30
| | | | | * make user data available for any build config Signed-off-by: lns <matzeton@googlemail.com>
* Fix some memory errors triggered by allocation failures (#1995)Ivan Nardi2023-05-29
| | | | | | | | | | | | Some low hanging fruits found using nallocfuzz. See: https://github.com/catenacyber/nallocfuzz See: https://github.com/google/oss-fuzz/pull/9902 Most of these errors are quite trivial to fix; the only exception is the stuff in the uthash. If the insertion fails (because of an allocation failure), we need to avoid some memory leaks. But the only way to check if the `HASH_ADD_*` failed, is to perform a new lookup: a bit costly, but we don't use that code in any critical data-path.
* Avoid calling `ndpi_reconcile_protocols()` twice in ↵Ivan Nardi2023-05-29
| | | | | | `ndpi_detection_giveup()` (#1996) `ndpi_reconcile_protocols()` is already called by `ndpi_set_detected_protocol()`
* Add support for Epic Games and GeForceNow/Nvidia (#1990)Ivan Nardi2023-05-27
|
* Cosmetic changesLuca Deri2023-05-23
|
* Improved missing usage of nDPIs malloc wrapper. Fixes #1978. (#1979)Toni2023-05-20
| | | | | * added CI check Signed-off-by: lns <matzeton@googlemail.com>
* Added scripts to auto generate hostname/SNI *.inc files. (#1984)Toni2023-05-20
| | | | | * add illegal gambling sites (Belgium) Signed-off-by: lns <matzeton@googlemail.com>
* Fix classification-by-ip in `ndpi_detection_giveup` (#1981)Ivan Nardi2023-05-17
| | | | | | | | | | | | | | Return the "classification-by-ip" as protocol results only if no other results are available. In particular, never return something like "protocol_by_port/protocol_by_ip" (i.e. `NTP/Apple`, BitTorrent/GoogleCloud`, `Zoom/AWS`) because this kind of classification is quite confusing, if not plainly wrong. Notes: * the information about "classification-by-ip" is always available, so no information is lost with this change; * in the unit tests, the previous classifications with confidence `NDPI_CONFIDENCE_DPI_PARTIAL` were wrong, as noted in #1957
* Add support for SRTP (#1977)Ivan Nardi2023-05-16
| | | | | | | The goal is to have Zoom flows classified as "Encrypted" and not as "Cleartext". Start documenting the list of protocols supported by nDPI; format, verbosity and content are still a work-in-progress.
* HTTP: rework state machine (#1966)Ivan Nardi2023-05-16
| | | | | | | | | | | | | | | | | The goal if to correlate the right request-response pair, exporting metadata from only one transaction (for example, the right url & return state pair) As a nice side effect, the code should be much cleaner, but that is a matter of taste. Two differences respect to the previous code: * as it happens in the CI, if in the flow there are only one response (before) and one request (after), only the metadata of the response are saved/exported * for performance reasons, we don't call `ndpi_parse_packet_line_info()` anymore for ALL packets triggering the HTTP dissector, but only for the packets that we already know belong to an HTTP flow. This is the reason for the changes in RTSP/SOAP/... code
* Fix MS Teams detection with heuristic (#1972)Ivan Nardi2023-05-10
| | | | | | When we reconcile a TLS session to Teams, we need to keep TLs as master. This way: * we keep exporting all the TLS metadata * we avoid some memory leaks (of these metadata themeselves)
* Improved MS Teams detection with heuristicLuca2023-05-10
|
* Enhanced MS teams STUN/Azure detectionLuca2023-05-10
|
* Remove special handling of some TCP flows without SYN (#1965)Ivan Nardi2023-05-09
| | | | | | | | | | | | | | | | | | | | | This piece of code has multiple problems: * nDPI is able to detect some TCP protocols even with mid-flows (i.e. without the initial packets of the session); TLS is the most significative example * since e6b332aa4a1399e33df68998cf8351bccaee3fc4 it is perfectly valid to not pass the TCP Handshake packets to nDPI * in any case, we shouldn't call `ndpi_detection_giveup()`. That function is usually called by the application and we end up calling it twice in some cases. The simple solution is to completely remove that code: process these kinds of flows like everyone else. Note that the application can always avoid to pass to nDPI any TCP flows without the initial handshake; the flow managemnt is always up to the application. Looking at the CI results, some rare flows are now processed significantly longer. As a follow-up we could look into that.
* Improve detection of crawler/bot traffic (#1956)Ivan Nardi2023-05-04
|
* Make Bittorrent LRU cache IPv6 aware. (#1909)Ivan Nardi2023-04-29
| | | | It was the only remaining LRU cache without IPv6 support. See 81e1ea545ca465cda064e7cc80333fe7f0ef2aff
* Added missing checkLuca Deri2023-04-28
|
* Added ndpi_guess_undetected_protocol_v4() to restore logic thatLuca Deri2023-04-28
| | | | was removed in https://github.com/ntop/nDPI/commit/032e778a6dc1ee2173c4cdb2c671180c5b5df245
* Updated (C)Luca Deri2023-04-28
|
* Add an heuristic to detect/ignore some anomalous TCP ACK packets (#1948)Ivan Nardi2023-04-25
| | | | | | | | | | | | | | | | | | | In some networks, there are some anomalous TCP flows where the smallest ACK packets have some kind of zero padding. It looks like the IP and TCP headers in those frames wrongly consider the 0x00 Ethernet padding bytes as part of the TCP payload. While this kind of packets is perfectly valid per-se, in some conditions they might be treated by the TCP reassembler logic as (partial) overlaps, deceiving the classification engine. Add an heuristic to detect these packets and to ignore them, allowing correct detection/classification. This heuristic is configurable. Default value: * in the library, it is disabled * in `ndpiReader` and in the fuzzers, it is enabled (to ease testing) Credit to @vel21ripn for the initial patch. Close #1946
* fuzz: extend fuzzers coverage (#1952)Ivan Nardi2023-04-25
|
* Add "Heroes of the Storm" video game signature detection. (#1949)nikitamishagin2023-04-22
|
* Improved debug output. (#1951)Toni2023-04-21
| | | | | | * try to get rid of some `printf(..)`s as they do not belong to a shared library * replaced all `exit(..)`s with `abort()`s to indicate an abnormal process termination Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added OICQ dissector. (#1950)Toni2023-04-21
| | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Simplify `ndpi_internal_guess_undetected_protocol()` (#1941)Ivan Nardi2023-04-12
| | | | | | | | | | | | | | | | | | | `ndpi_guess_undetected_protocol()/ndpi_internal_guess_undetected_protocol()` is a strange function: * it is exported by the library and it is actively used by `ntopng` * it is intrinsecally ipv4-only * it returns basically something like "classification_by_ip"/"classification_by_port" (these information have already been calculated in `ndpi_do_guess()`...) * it access the bittorrent LRU caches (similarly to `ndpi_detection_giveup()` but without all the other caches...) So: * make the interface IPv4/6 agnostic * use the classifications already available This work will allow to make the Bittorrent caches IPV6-aware (see 81e1ea5). Handle Dropbox classification "by-port" in the "standard" way.
* Added BACnet dissector. (#1940)Toni2023-04-11
| | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved debug logging.lns2023-04-11
| | | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved invalid logging via printf().lns2023-04-11
| | | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added Source Engine dissector. (#1937)Toni2023-04-11
| | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Ookla: rework detection (#1922)Ivan Nardi2023-03-30
| | | | | | | | | | | | | | | | | | | | | | | | | | The logic of the LRU cache has been changed: once we know an ip has connected to an Ookla server, all the following (unknown) flows (for a short time interval) from the same ip to the port 8080 are treated as Ookla ones. Most of the changes in this commit are about introducing the concept of "aggressive detection". In some cases, to properly detect a protocol we might use some statistical/behavior logic that, from one side, let us to identify the protocol more often but, from the other side, might lead to some false positives. To allow the user/application to easily detect when such logic has been triggered, the new confidence value `NDPI_CONFIDENCE_DPI_AGGRESSIVE` has been added. It is always possible to disable/configure this kind of logic via the API. Detection of Ookla flows using plain TLS over port 8080 is the first example of aggressive detection in nDPI. Tested with: * Android 9.0 with app 4.8.3 * Ubuntu 20.04 with Firefox 110 * Win 10 with app 1.15 and 1.16 * Win 10 with Chrome 108, Edge 108 and Firefox 106
* Added the ability to define custom protocols with arbitrary Ids in proto.txtLuca Deri2023-03-22
| | | | | | | | | | Example - ip:213.75.170.11/32:443@CustomProtocol nDPI assigns an is that can change based on protos.txt content - ip:213.75.170.11/32:443@CustomProtocol=9999 nDPI assigns 9999 as protocolId to CustomProtocol and won't change when protos.txt content will chaneg
* Improved checks for duplicated entries in protocols fileLuca Deri2023-03-21
|
* DisneyPlus/Hulu ip lists should be auto-generated (#1905)Ivan Nardi2023-03-20
| | | | | | Remove two stale ip lists: 1) these 3 ips are in the Amazon ranges (now)... 2) the Instagram list originated from AS32934, which is now a Facebook AS; see https://github.com/ntop/nDPI/pull/1264/commits/8dabd06301a802dd38616ba8684a1d995783e023
* Add a new protocol id for generic Adult Content traffic (#1906)Ivan Nardi2023-03-20
| | | | The list has been taken from https://www.similarweb.com/top-websites/adult/ Fix a GoTo false positive.
* Add a new protocol id for generic advertisement/analytics/tracking stuff (#1904)Ivan Nardi2023-03-20
|
* FreeBSD compilation fixLuca Deri2023-03-14
| | | | (C) update
* fuzz: simplify fuzzers dependencies in CIFuzz (#1896)Ivan Nardi2023-03-14
| | | | | | | | | | | | | | | | | | | | | | CIFuzz (based on oss-fuzz) is the GitHub action/CI job that runs fuzz targets on pull requests. It only runs the fuzzers affected by a pull request/commit. Otherwise it will divide up the allotted fuzzing time among all fuzzers in the project. Since: * we have more than 20 fuzzers and most of them use the custom memory allocation functions (to force allocation failures) even if they are not strictly about DPI stuff; * we need to keep fuzzing time relatively small (to avoid waiting the CI results for a long time) it is important that fuzzers dependencies (which are based on *files* changed by the single commit/PR) are as small as possible. Bottom line: move all the low-level allocation callbacks to a dedicated file; this way most of the fuzzers don't depend anymore on `ndpi_main.c` file (which is touched by ever commit/PR). The goal is to have only the "most important" fuzzers running during (most of) the CI.
* Added ability to define an unlimited number of custom rules IP:port for the ↵Luca Deri2023-03-13
| | | | same IP (it used tobe limited to 2)
* Minor cleanupLuca Deri2023-03-13
|
* Added fix for better handling exceptions rollback in case of later matchLuca Deri2023-03-06
|
* Add a new flow risk about literal IP addresses used as SNI (#1892)Ivan Nardi2023-03-02
| | | | | | | | | | RFC 6066 3: "Literal IPv4 and IPv6 addresses are not permitted in "HostName"." Don't set this risk if we have a valid sub-classification (example: via certificate) Since a similar risk already exists for HTTP hostnames, reuse it, with a more generic name.
* Fix insert of ip addresses into patricia tree(s) (#1895)Ivan Nardi2023-03-02
| | | Fix: d0b46dcf
* Keep master protocol in `ndpi_reconcile_protocols`Nardi Ivan2023-03-01
| | | | | | | | | | | | | | | | | | | As a side effect of this change, the following memory leak has been fixed. ``` ==55523==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2 byte(s) in 1 object(s) allocated from: #0 0x55657d01b7be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_alloc_fail+0x5b27be) (BuildId: 9dfa7bd9081421417b8b1c2e8b44633a9bfcfe52) #1 0x55657d169182 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:243:25 #2 0x55657d1694c8 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:310:13 #3 0x55657d25ddc7 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2308:46 #4 0x55657d2701b9 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:895:5 #5 0x55657d26a408 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1223:11 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56466
Powered by cgit, Themed by Ted Yin.