aboutsummaryrefslogtreecommitdiff
path: root/src/include
Commit message (Collapse)AuthorAge
* Improved Zoom protocol detectionLuca Deri2022-01-23
|
* Updated confidence typeLuca Deri2022-01-18
|
* H323: fix a use-after-poison error (#1412)Ivan Nardi2022-01-17
| | | | | | | Detected by oss-fuzz See: https://oss-fuzz.com/testcase-detail/6730505580576768 Fix a function prototype Update a unit test results
* Added Badoo detectionLuca Deri2022-01-17
|
* Added performance tests toolsLuca Deri2022-01-16
|
* Reduced Patricia tree bucket memory footprintLuca Deri2022-01-16
|
* FreeBSD fixesLuca Deri2022-01-13
|
* Added the ability to specify trusted issueDN often used in companies to ↵Luca Deri2022-01-13
| | | | | | | | | | | self-signed certificates This allows to avoid triggering alerts for trusted albeit private certificate issuers. Extended the example/protos.txt with the new syntax for specifying trusted issueDN. Example: trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
* Added EthernetIP dissectorLuca Deri2022-01-12
|
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Remove some unused fields (#1393)Ivan Nardi2022-01-08
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Added support for Log4J/Log4Shell detection in nDPI via a new flow risk ↵Luca Deri2021-12-23
| | | | named NDPI_POSSIBLE_EXPLOIT
* Add support for ICloud Private Relay (#1390)Ivan Nardi2021-12-22
| | | | | | | See: https://www.apple.com/privacy/docs/iCloud_Private_Relay_Overview_Dec2021.PDF TODO: an up-to-date list of egress IP ranges is publicly available. Can we use it somehow?
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Added Microsoft Azure supportLuca Deri2021-12-19
|
* Improve/add several protocols (#1383)Ivan Nardi2021-12-18
| | | | | | | | | | | | | | | | | | | Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok detection. Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection. Try to differentiate between Messenger/Signal standard flows (i.e chat) and their VOIP (video)calls (like we already do for Whatsapp and Snapchat). Add a partial list of some ADS/Tracking stuff. Fix Cassandra, Radius and GTP false positives. Fix DNS, Syslog and SIP false negatives. Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and GTP_PRIME. Fix 3 LGTM warnings.
* Improved BitTorrent classificationLuca Deri2021-12-07
|
* Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1380)Ivan Nardi2021-12-04
| | | | | | | | | | | There are no valid reasons for a (generic) protocol to ignore IPv6 traffic. Note that: * I have not found the specifications of "CheckPoint High Availability Protocol", so I don't know how/if it supports IPv6 * all LRU caches are still IPv4 only Even if src_id/dst_id stuff is probably useless (see #1279), the right way to update the protocol classification is via `ndpi_set_detected_protocol()`
* Make serialize risk and proto not dependant on any flow. (#1377)Toni2021-12-04
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added example for finding similarities in RRDs using nDPI statistical APIsLuca Deri2021-12-04
|
* Added ndpi_serializer_skip_header() serialization APILuca Deri2021-11-26
|
* Added Salesforce detectionLuca Deri2021-11-26
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* bittorrent old code cleanup. Enlarged BT cacheLuca Deri2021-11-16
|
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* IMAP, POP3, SMTP: improve dissection (#1368)Ivan Nardi2021-11-11
| | | Avoid NATS false positives
* Differentiate between standard Amazon stuff (i.e market) and AWS (#1369)Ivan Nardi2021-11-04
|
* Improved BitTorrent detectionLuca Deri2021-11-04
|
* TLS: fix two warnings (#1365)Ivan Nardi2021-11-02
| | | | | | | | Disable unit tests on CI for big-endian target. We know we have multiple issues on big-endian architectures (see #1312) and so the unit tests always fail there. Ignore this error for the time being and let the CI pass if we don't have other issues. Remove an unused automa definition
* Serialize additional information stored in the flow struct. (#1362)Toni2021-10-27
| | | | | | | * Changed function signature of ndpi_flow2json (removed unused vlan_id; API break) * Serialize NTP information. * Improved QUIC serialization. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Detect invalid characters in text and set a risk. Fixes #1347. (#1363)Toni2021-10-26
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix QUIC log and remove SoulSeek leftovers after b97dc6ba (#1351)Ivan Nardi2021-10-19
| | | | | Update .gitignore file Fix a function prototype Close #1349
* Fix some invalid memory reads (#1350)Ivan Nardi2021-10-19
| | | | | | | | `ndpi_detection_giveup()` (and any functions called by it) can't access `ndpi_detection_module_struct->packet` anymore since 730c236. Sync unit tests results Close #1348
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Implemented RDP over UDP dissectionLuca Deri2021-10-18
|
* Reworked flow risks asignmentLuca Deri2021-10-16
| | | | Added esceptions for windows update and binary application transfer risk
* Removed outdated (and broken) soulseek dissectorLuca Deri2021-10-15
|
* Implemented ndpi_ses_fitting() and ndpi_des_fitting()Luca Deri2021-10-12
| | | | for comuting the best alpha/beta values for exponential smoothing
* Win fixLuca Deri2021-10-11
|
* Cleaned up code moving specific includes in files their are using it. Thi ↵Luca Deri2021-10-11
| | | | prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* Added -a <num> to ndpiReader for generating OPNsense configurationLuca Deri2021-10-04
| | | | See https://github.com/ntop/opnsense
* Adds sections labels with risk id to the docsSimone Mainardi2021-10-01
|
* Remove `detected_protocol_stack` field from `ndpi_packet_struct` (#1317)Ivan Nardi2021-09-29
| | | | | | | | | | | | | This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`: * at the very beginning of packet dissection, the value saved in `flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack` (via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`) * every time we update `flow->detected_protocol_stack` we update `packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()` -> `ndpi_int_change_packet_protocol()`) These two fields are always in sync: keeping the same value in two different places is useless.
* Compilation fixed on CentOS 7Luca Deri2021-09-27
| | | | Bitmap APi changes
* Fix armhf (#1315)Gianfranco Costamagna2021-09-26
| | | | | | | | | | | | | | | | * Fix unaligned memory accesses with get_u_int64_t at armhf see: https://bugs.debian.org/993627 * Use get_u_int64_t to avoid unaligned memory access at armhf see: https://bugs.debian.org/993627 * Update src/include/ndpi_define.h.in Drop const type from get_u_int64_t, from lnslbrty Co-authored-by: Bernhard Übelacker <bernhardu@mailbox.org> Co-authored-by: Toni <matzeton@googlemail.com>
* Added include for defining bools (not present on all platforms)Luca Deri2021-09-26
|
* Added API for handling compressed bitmapsLuca Deri2021-09-26
| | | | | | | | | | | | | | ndpi_bitmap* ndpi_bitmap_alloc(); void ndpi_bitmap_free(ndpi_bitmap* b); u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b); void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value); bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_clear(ndpi_bitmap* b); size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf); ndpi_bitmap* ndpi_bitmap_deserialize(char *buf); based on https://github.com/RoaringBitmap/CRoaring
Powered by cgit, Themed by Ted Yin.