aboutsummaryrefslogtreecommitdiff
path: root/src/lib/protocols/http.c
Commit message (Collapse)AuthorAge
* Fix compilation and sync unit tests results (#1445)Ivan Nardi2022-02-19
| | | | 'strcasestr' is not defined in all enviroments: quicker fix is to use 'ndpi_strncasestr' instead.
* Added newflow risk NDPI_HTTP_CRAWLER_BOTLuca Deri2022-02-17
|
* Added NDPI_ERROR_CODE_DETECTED riskLuca Deri2022-02-03
|
* Renamed DCERPC to more generic RPC protocol so we can use also for other ↵Luca Deri2022-02-03
| | | | | | | types of RPCs (not limited to DCE) Extended HTTP plugin to support RPC Improved HTTP crear text detection to limit it to Basic and Digest
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Improved user agent analysisLuca Deri2022-01-09
|
* Invalid checkLuca Deri2022-01-06
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Added support for Log4J/Log4Shell detection in nDPI via a new flow risk ↵Luca Deri2021-12-23
| | | | named NDPI_POSSIBLE_EXPLOIT
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* HTTP proxy improvementLuca Deri2021-11-25
|
* Reworked HTTP protocol dissection including HTTP proxy and HTTP connectLuca Deri2021-11-25
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked flow risks asignmentLuca Deri2021-10-16
| | | | Added esceptions for windows update and binary application transfer risk
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* Remove `detected_protocol_stack` field from `ndpi_packet_struct` (#1317)Ivan Nardi2021-09-29
| | | | | | | | | | | | | This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`: * at the very beginning of packet dissection, the value saved in `flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack` (via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`) * every time we update `flow->detected_protocol_stack` we update `packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()` -> `ndpi_int_change_packet_protocol()`) These two fields are always in sync: keeping the same value in two different places is useless.
* A (tiny) effort to reduce memory usage per flow (#1304)Ivan Nardi2021-09-18
| | | | | | | Follow-up of 22241a1d Only trivial changes: * remove completely unused fields * remove fields only written (but never read) * CSGO protocol only handles UDP traffic
* Added new risk for clear text credentialsLuca Deri2021-09-10
|
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Skip whitespaces between HTTP method and URL. (#1271)Toni2021-08-08
| | | | | * be less case-restrictive, RFC2616 wants it that way Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved RTSP detection and fixed HTTP false-positive. Fixes #1229. (#1266)Toni2021-07-31
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked flow risk implementationLuca Deri2021-07-23
|
* Code cleanup. (#1246)Vitaly Lavrov2021-07-16
| | | | ndpi_utils.c: use ndpi_malloc,ndpi_calloc,ndpi_free genshin_impact.c, git.c, hpvirtgrp.c, http.c, z3950.c: removed "#include stdlib.h"
* Fix for #1230 (#1235)Vitaly Lavrov2021-07-07
| | | | | | | | | | | | | | | * Revert "Fix return value of ndpi_match_string_subprotocol() (#1230)" This reverts commit 58665e93a98d014b53d131b2481ccab074efc9ff. * Checking the return code after calling ndpi_match_string_subprotocol() ndpi_api.h: Description of the returned error codes for the ndpi_match_string_subprotocol() function. If the ndpi_match_string_subprotocol() function returned an error, then return NDPI_PROTOCOL_UNKNOWN. http: The "Content-type" header is only checked if it is not empty.
* Improved RTSP detection the second. (#1232) (#1233)Toni2021-07-07
| | | | | | | * RTSP is no subprotocol of HTTP (most of the time) * detection patterns should stay in rtsp.c * set detected HTTP protocol only if at least a valid HTTP method detected Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Refactored nDPI subprotocol handling and aimini protocol detection. (#1156)Toni2021-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Refactored and merged callback buffer routines for non-udp-tcp / udp / tcp / tcp-wo-payload. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Try to detect one subprotocol if a detected protocol can have one. * This adds a performance overhead due to much more protocol detection routine calls. See #1148 for more information. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Refactor subprotocol handling (1/2). Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Refactor subprotocol handling (2/2). Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Prevent some code duplication by using macros for ndpi_int_one_line_struct string comparision. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Refactored aimini HTTP detection parts (somehow related to #1148). Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Added aimini client/server test pcap. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Removed master protocol as it was only used for STUN and via also removed API function ndpi_get_protocol_id_master_proto * Adjusted Python code to conform to the changes made during the refactoring process. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* HTTP: fix memory access in ndpi_http_parse_subprotocol() (#1151)Ivan Nardi2021-03-09
|
* Added Ookla detection over IPv6Luca Deri2021-03-09
|
* Ookla fixesLuca Deri2021-03-09
|
* Improved detection of Ookla speedtest and openspeedtest.comLuca Deri2021-03-09
|
* Improved DGA detection with trigrams. Disadvantage: slower startup timeLuca Deri2021-03-03
| | | | | Reworked Tor dissector embedded in TLS (fixes #1141) Removed false positive on HTTP User-Agent
* HTTP: fix user-agent parsing (#1124)Ivan Nardi2021-02-03
| | | | | | | | | | | User-agent information is used to try to detect the user OS; since the UA is extracted for QUIC traffic too, the "detected_os" field must be generic and not associated to HTTP flows only. Otherwise, you might overwrite some "tls_quic_stun" fields (SNI...) with random data. Strangely enough, the "detected_os" field is never used: it is never logged, or printed, or exported...
* HTTP: fix logs when NDPI_ENABLE_DEBUG_MESSAGES is defined (#1123)Ivan Nardi2021-02-03
|
* Increased number of extra packets that is necessary since the frgament ↵Luca Deri2021-02-03
| | | | mananger introduction
* (C) UpdateLuca Deri2021-01-07
|
* HTTP: fix compilation and a memory error when NDPI_ENABLE_DEBUG_MESSAGES is ↵Ivan Nardi2021-01-07
| | | | defined (#1109)
* Split HTTP request from response Content-Type. Request Content-Type should ↵Luca Deri2021-01-06
| | | | be present with POSTs and not with other methods such as GET
* Added check for invalid HTTP contentLuca Deri2021-01-06
|
* Fixed missing symbolLuca Deri2021-01-02
|
* Added HTTP suspicious content securirty risk (useful for tracking trickbot)Luca Deri2021-01-02
|
* Added known protocol on unknown port for ntopLuca Deri2020-12-28
|
* Type change to avoid Windows compilation issuesLuca Deri2020-12-17
|
* Improved HTTP dissectionLuca Deri2020-12-16
|
* Fix for detecting numeric IPsLuca Deri2020-11-01
|
* Fixes #1033Luca Deri2020-10-21
|
* Minor UA handling improvement to avoid heap-overflowLuca Deri2020-09-22
|
* Merge pull request #1012 from IvanNardi/uaLuca Deri2020-09-17
|\ | | | | QUIC: extract User Agent information
Powered by cgit, Themed by Ted Yin.