aboutsummaryrefslogtreecommitdiff
path: root/src/lib/ndpi_main.c
Commit message (Collapse)AuthorAge
...
* H323: fix a use-after-poison error (#1412)Ivan Nardi2022-01-17
| | | | | | | Detected by oss-fuzz See: https://oss-fuzz.com/testcase-detail/6730505580576768 Fix a function prototype Update a unit test results
* Adds some risk exceptions for popular services and domain namesLuca2022-01-17
| | | | via a new (internal) function named ndpi_add_domain_risk_exceptions()
* Added performance tests toolsLuca Deri2022-01-16
|
* Added the ability to specify trusted issueDN often used in companies to ↵Luca Deri2022-01-13
| | | | | | | | | | | self-signed certificates This allows to avoid triggering alerts for trusted albeit private certificate issuers. Extended the example/protos.txt with the new syntax for specifying trusted issueDN. Example: trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
* Added EthernetIP dissectorLuca Deri2022-01-12
|
* Fix two use-of-uninitialized-value errors (#1398)Ivan Nardi2022-01-12
| | | | | | | Found by oss-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40269 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41432 Fix fuzz compilation (follow-up of f5545a80)
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Remove some unused fields (#1393)Ivan Nardi2022-01-08
|
* Improved bittorrent detectionLuca Deri2022-01-06
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Improved BitTorrent guessLuca Deri2021-12-26
|
* Added support for Log4J/Log4Shell detection in nDPI via a new flow risk ↵Luca Deri2021-12-23
| | | | named NDPI_POSSIBLE_EXPLOIT
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Improved bittorrent heuristicLuca Deri2021-12-21
|
* Added Microsoft Azure supportLuca Deri2021-12-19
|
* Improve/add several protocols (#1383)Ivan Nardi2021-12-18
| | | | | | | | | | | | | | | | | | | Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok detection. Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection. Try to differentiate between Messenger/Signal standard flows (i.e chat) and their VOIP (video)calls (like we already do for Whatsapp and Snapchat). Add a partial list of some ADS/Tracking stuff. Fix Cassandra, Radius and GTP false positives. Fix DNS, Syslog and SIP false negatives. Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and GTP_PRIME. Fix 3 LGTM warnings.
* Improved BitTorrent classificationLuca Deri2021-12-07
|
* Fix configure script (after fb85dac9) (#1381)Ivan Nardi2021-12-04
| | | Fix/disable some LGTM warnings
* Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1380)Ivan Nardi2021-12-04
| | | | | | | | | | | There are no valid reasons for a (generic) protocol to ignore IPv6 traffic. Note that: * I have not found the specifications of "CheckPoint High Availability Protocol", so I don't know how/if it supports IPv6 * all LRU caches are still IPv4 only Even if src_id/dst_id stuff is probably useless (see #1279), the right way to update the protocol classification is via `ndpi_set_detected_protocol()`
* Removed spaces from some protocol names as they create problems with some ↵Luca Deri2021-11-30
| | | | applications
* Aded default port to HTTP_CONNECTLuca Deri2021-11-25
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* BitTorrent detection improvementsLuca Deri2021-11-17
|
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Update ndpi_main.c for HTTP line parser optimization and cleaning (#1372)ragostino2021-11-11
| | | | | | | * Update ndpi_main.c HTTP line parser: optimization and cleaning * Update ndpi_main.c
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* IMAP, POP3, SMTP: improve dissection (#1368)Ivan Nardi2021-11-11
| | | Avoid NATS false positives
* Improved BitTorrent detectionLuca Deri2021-11-04
|
* Fix use-of-uninitialized-value (#1366)Ivan Nardi2021-11-03
| | | Close #1346
* Avoid overwriting valid protocol in `ndpi_detection_giveup` (#1360)Ivan Nardi2021-10-27
| | | | | | | | | | | | | | | | | We should avoid updating any valid protocol in `ndpi_detection_giveup`; we should try to find a proper classification only if the flow is still completely unclassified. For example in the attached pcap there is a valid TLS session, recognized as such by TLS dissector. However, the `ndpi_detection_giveup`function updates it to "HTTP/TLS" (!?) simply because the server port is 80. Note that the real issue is not the wrong classification, but the wrong access to `flow->protos` union. If we already set some fields of `flow->protos` and we change the protocol in `ndpi_detection_giveup`, we might end up freeing some invalid pointers in `ndpi_free_flow_data` (no wonder this issue has been found while fuzzing #1354) Fix GIT and TLS dissectors (issues found by CI fuzzer)
* Improved STUN and RTP detectionLuca Deri2021-10-27
|
* Detect invalid characters in text and set a risk. Fixes #1347. (#1363)Toni2021-10-26
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix some invalid memory reads (#1350)Ivan Nardi2021-10-19
| | | | | | | | `ndpi_detection_giveup()` (and any functions called by it) can't access `ndpi_detection_module_struct->packet` anymore since 730c236. Sync unit tests results Close #1348
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked flow risks asignmentLuca Deri2021-10-16
| | | | Added esceptions for windows update and binary application transfer risk
* Removed outdated (and broken) soulseek dissectorLuca Deri2021-10-15
|
* Added missing call to ndpi_reconcile_protocols() with protocol guessLuca Deri2021-10-14
|
* Fixed issue on DGA numeric IP detectionLuca Deri2021-10-13
|
* IndemtLuca Deri2021-10-13
|
* Compilation fixLuca Deri2021-10-05
|
* Numeric IPs are not considered for DGA checksLuca Deri2021-10-05
|
* Improved DGA detection for skipping potential DGAs of known/popular domain namesLuca Deri2021-10-05
|
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* Added -a <num> to ndpiReader for generating OPNsense configurationLuca Deri2021-10-04
| | | | See https://github.com/ntop/opnsense
* Remove `detected_protocol_stack` field from `ndpi_packet_struct` (#1317)Ivan Nardi2021-09-29
| | | | | | | | | | | | | This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`: * at the very beginning of packet dissection, the value saved in `flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack` (via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`) * every time we update `flow->detected_protocol_stack` we update `packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()` -> `ndpi_int_change_packet_protocol()`) These two fields are always in sync: keeping the same value in two different places is useless.
* FTP: fix support for START-TLS sessionsNardi Ivan2021-09-21
| | | | | | When TLS-over-FTP is used, the credentials are encrypted. So we must not wait for the username and the password commands, otherwise we elaborate a lot of packets for nothing.
* Fix parsing of ipv6 packets with extension headersNardi Ivan2021-09-19
| | | | | | | | Decoding of ipv6 traffic with extension headers was completely broken, since the beginning of the L4 header was always set to a wrong value. Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first one and drop the others.
* Small fixes after latest commits (#1308)Ivan Nardi2021-09-18
|
* A (tiny) effort to reduce memory usage per flow (#1304)Ivan Nardi2021-09-18
| | | | | | | Follow-up of 22241a1d Only trivial changes: * remove completely unused fields * remove fields only written (but never read) * CSGO protocol only handles UDP traffic
* DNS dissection fixesLuca Deri2021-09-17
|
Powered by cgit, Themed by Ted Yin.