aboutsummaryrefslogtreecommitdiff
path: root/python
Commit message (Collapse)AuthorAge
* Added NDPI_ERROR_CODE_DETECTED riskLuca Deri2022-02-03
|
* Added new IDN/Punycode risk for spotting internationalized domain namesLuca2022-02-03
|
* Added NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE flow riskLuca Deri2022-01-26
| | | | Added ndpi_set_tls_cert_expire_days() API call to modify the number of days for triggering the above alert that by default is set to 30 days
* build: respect environment options more (#1392)Sam James2022-01-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * build: update m4/ax_pthread.m4 from serial 23 -> serial 31 Update ax_pthread.m4 to the latest version from the autoconf-archive project. Signed-off-by: Sam James <sam@gentoo.org> * build: properly detect AR, CC, RANLIB It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain variables/tools for cross-compilation, testing with other toolchains, and to ensure the compiler chosen by the user is actually used for the build. Previously, GNU_PREFIX was kind-of used for this but this isn't a standard variable (at all) and it wasn't applied consistently anyway. We now use the standard autoconf mechanisms for finding these tools. (RANLIB is already covered by LT_INIT.) Signed-off-by: Sam James <sam@gentoo.org> * build: use $(MAKE) This ensures that parallel make works correctly, as otherwise, a fresh make job will be started without the jobserver fd, and hence not know about its parent, forcing -j1. * build: respect CPPFLAGS, LDFLAGS - CPPFLAGS is for the C preprocessor (usually for setting defines) - LDFLAGS should be placed before objects for certain flags to work (e.g. -Wl,--as-needed) Signed-off-by: Sam James <sam@gentoo.org> Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Remove some unused fields (#1393)Ivan Nardi2022-01-08
|
* Added support for Log4J/Log4Shell detection in nDPI via a new flow risk ↵Luca Deri2021-12-23
| | | | named NDPI_POSSIBLE_EXPLOIT
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* TLS: fix two warnings (#1365)Ivan Nardi2021-11-02
| | | | | | | | Disable unit tests on CI for big-endian target. We know we have multiple issues on big-endian architectures (see #1312) and so the unit tests always fail there. Ignore this error for the time being and let the CI pass if we don't have other issues. Remove an unused automa definition
* Fix QUIC log and remove SoulSeek leftovers after b97dc6ba (#1351)Ivan Nardi2021-10-19
| | | | | Update .gitignore file Fix a function prototype Close #1349
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* FTP: fix support for START-TLS sessionsNardi Ivan2021-09-21
| | | | | | When TLS-over-FTP is used, the credentials are encrypted. So we must not wait for the username and the password commands, otherwise we elaborate a lot of packets for nothing.
* A (tiny) effort to reduce memory usage per flow (#1304)Ivan Nardi2021-09-18
| | | | | | | Follow-up of 22241a1d Only trivial changes: * remove completely unused fields * remove fields only written (but never read) * CSGO protocol only handles UDP traffic
* Progetto esame Gestione di Reti - Debora Cerretini (#1290)deboracerretini2021-09-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Removing unused fields from the id_struct structure. (#1293)Vitaly Lavrov2021-09-13
| | | Reduced by 40 bytes.
* Added new risk for clear text credentialsLuca Deri2021-09-10
|
* Fix access to some TLS fields in flow structure (#1277)Ivan Nardi2021-08-20
| | | | Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by QUIC (i.e UDP...), too.
* Clode cleanup (after last merge)Luca Deri2021-08-08
|
* Added TLS fatal alert flow riskLuca Deri2021-08-07
|
* Improved TFTP detection. Fixes #1242, #1256 (#1262)Toni2021-07-25
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added NDPI_DESKTOP_OR_FILE_SHARING_SESSION flow riskLuca Deri2021-04-11
|
* Refactored nDPI subprotocol handling and aimini protocol detection. (#1156)Toni2021-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Refactored and merged callback buffer routines for non-udp-tcp / udp / tcp / tcp-wo-payload. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Try to detect one subprotocol if a detected protocol can have one. * This adds a performance overhead due to much more protocol detection routine calls. See #1148 for more information. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Refactor subprotocol handling (1/2). Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Refactor subprotocol handling (2/2). Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Prevent some code duplication by using macros for ndpi_int_one_line_struct string comparision. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Refactored aimini HTTP detection parts (somehow related to #1148). Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Added aimini client/server test pcap. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Removed master protocol as it was only used for STUN and via also removed API function ndpi_get_protocol_id_master_proto * Adjusted Python code to conform to the changes made during the refactoring process. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* HTTP: fix user-agent parsing (#1124)Ivan Nardi2021-02-03
| | | | | | | | | | | User-agent information is used to try to detect the user OS; since the UA is extracted for QUIC traffic too, the "detected_os" field must be generic and not associated to HTTP flows only. Otherwise, you might overwrite some "tls_quic_stun" fields (SNI...) with random data. Strangely enough, the "detected_os" field is never used: it is never logged, or printed, or exported...
* Added missing libraryLuca Deri2021-01-22
|
* Added HTTP suspicious content securirty risk (useful for tracking trickbot)Luca Deri2021-01-02
|
* Added missing files to `make dist' target which are not required to build ↵Toni2020-09-29
| | | | | nDPI but still somehow essential. (#1024) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added risks for checkingLuca Deri2020-09-21
| | | | | - invalid DNS traffic (probably carrying exfiltrated data) - TLS traffic with no SNI extension
* Update example.aouinizied2020-09-14
|
* Added new risk for NDPI_UNSAFE_PROTOCOL that identifies protocols that are ↵Luca Deri2020-08-30
| | | | not condidered safe/secure
* Add Connectivity check category and blacklisted host risk.aouinizied2020-08-10
|
* Add risk flag about suspicious ESNI usageNardi Ivan2020-08-05
| | | | | | | | | | | | | | | In a Client Hello, the presence of both SNI and ESNI may obfuscate the real domain of an HTTPS connection, fooling DPI engines and firewalls, similarly to Domain Fronting. Such technique is reported in a presentation at DEF CON 28: "Domain Fronting is Dead, Long Live Domain Fronting: Using TLS 1.3 to evade censors, bypass network defenses, and blend in with the noise" Full credit for the idea must go the original author At the moment, the only way to get the pdf presention and related video is via https://forum.defcon.org/node/234492 Hopefully a direct link (and an example pcap) will be available soon
* Sync TLS definitions.aouinizied2020-08-01
|
* Update CFFI definitions.aouinizied2020-07-29
|
* added modified risksMrRadix2020-07-22
|
* added new risks to ndpi_risk_enumMrRadix2020-07-22
|
* Add ssl_version_str to flow structure.aouinizied2020-07-13
|
* Add NDPI_MALFORMED_PACKET risk.aouinizied2020-07-09
|
* Synchronize type definitions.aouinizied2020-06-12
|
* Update APIs and structures.aouinizied2020-05-22
|
* Fix flow_printer example.Zied Aouini2020-04-15
|
* Update Python cffi bindings.Zied Aouini2020-02-21
|
* nDPI now detect RCE in HTTP GET requestsMrTiz92020-01-24
|
* Update ndpi_flow_struct definition.Zied Aouini2019-12-23
|
* Implement cffi bindings.aouinizied2019-11-28
|
* Polish.aouinizied2019-11-15
|
* Improve python bindings.aouinizied2019-11-15
|
* Reworked categories handlingLuca Deri2019-09-29
| | | | | Removed GenericProtocol and replaced with categories Removed ndpi_pref_enable_category_substring_match option: substring matching is now default
* Add files via uploadbohmax2019-09-29
|
Powered by cgit, Themed by Ted Yin.