aboutsummaryrefslogtreecommitdiff
path: root/nDPId.c
Commit message (Collapse)AuthorAge
...
* Process additional layer 3 protocols.Toni Uhlig2022-01-31
| | | | | | | | | * bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af * run_tests.sh: improved execution from non-git directories e.g. via `make dist` * updated JSON schema to be more restrictive * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId: Added some error messages when workflow init fails.Toni Uhlig2022-01-26
| | | | | | | * Fixed invalid array subscript typ0 (caused some trouble..) * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved Plotly/Dash example. It is now somehow informative.Toni Uhlig2022-01-25
| | | | | | | * TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea8570609cc0c8cf6edadc81f8f5bb76 * py-flow-info: reset 'guessed' flag after detection/detection-update received Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Major nDPId extension. Sorry for the huge commit.Toni Uhlig2022-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - nDPId: fixed invalid IP4/IP6 tuple compare - nDPIsrvd: fixed caching issue (finally) - added tiny c example (can be used to check flow manager sanity) - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow` - README.md update: added example JSON sequence - nDPId: added new flow event `update` necessary for correct timeout handling (and other future use-cases) - nDPIsrvd.h and nDPIsrvd.py: switched to an instance (consists of an alias/source tuple) based flow manager - every flow related event **must** now serialize `alias`, `source`, `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout handling and verification process work correctly - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation - nDPIsrvd.py: removed PcapPacket class (unused) - py-flow-dashboard and py-flow-multiprocess: fixed race condition - py-flow-info: print statusbar with probably useful information - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`) to a generic flow event timestamp `ts_msec` - nDPId-test: added additional checks - nDPId: increased ICMP flow timeout - nDPId: using event based i/o if capturing packets from a device - nDPIsrvd: fixed memory leak on shutdown if remote descriptors were still connected Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Use blocking I/O to prevent data loss if nDPIsrvd too slow.Toni Uhlig2021-11-16
| | | | | | * Fixed MemoryProfiler stack overflow. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* MemoryProfiling: Advanced flow usage logging.Toni Uhlig2021-11-13
| | | | | | | | * nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined * nDPId-test: mock syslog flags and functions * gitlab-ci: force -Werror Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Additional semantic validation tests.Toni Uhlig2021-11-02
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed invalid "flow_last_seen" timestamp for the first packet.Toni Uhlig2021-10-08
| | | | | | | | | | * After the first packet was processed, "flow_last_seen" was still 0. This behaviour is invalid as the first packet may contain l4 payload data e.g. for UDP and it also breaks nDPId json consistency "flow_first_seen" > 0, but "flow_last_seen" == 0. * JSON schema: set minimum timestamp value for Epoch timestamps to 24710 for flow_*_seen and 1 for pcap packet ts. Those values are dependant on some manipulated pcap's in libnDPI/tests/pcap. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed some typ0's and reduced ICMP timeout to 10s.Toni Uhlig2021-09-16
| | | | | | | * nDPId: Renamed some of the misleading terms, still TODO for nDPIsrvd * CMake improvments Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fixed Warnings / build error / cosmeticsToni Uhlig2021-09-14
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved and Fixed another buffering issue caused by removing an outgoing fd ↵Toni Uhlig2021-08-05
| | | | | | too early from epoll queue (EPOLLOUT). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Enable memory profiling for nDPId-test.Toni Uhlig2021-06-18
| | | | | | * print a summary Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added zLib compression parameters to control compression conditions.Toni Uhlig2021-06-16
| | | | | | | | * more structs are now "compressable" * fixed missing DAEMON_RECONNECT event * improved memory profiler Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Support for zLib flow memory compression. Experimental.Toni Uhlig2021-06-14
| | | | | | | Please use this feature only for testing purposes. It will change or be removed in the future. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Git tag/commit version printing for nDPId/nDPIsrvd. Reduces confusion.Toni Uhlig2021-06-08
| | | | | | * disabled subshell spawn for run_tests.sh, common pitfall while using counters Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Unified IO buffer mgmt.Toni Uhlig2021-06-07
| | | | | | | * c-collectd gives the user control over collectd-exec instance name * added missing collectd type `flow_l4_icmp_count` Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked layer 4 flow length naming/calculation.Toni Uhlig2021-05-20
| | | | | | | * nDPIsrvd services usually do not care about layer4 data length, payload length is quite more essential for further processing Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Cosmetic fixes.Toni Uhlig2021-05-13
| | | | | | * daemon.sh script to simplify daemon testing Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Use layer4 specific flow timeouts.Toni Uhlig2021-05-13
| | | | | | * default values "stolen" from nf_conntrack Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added missing datalink types.Toni Uhlig2021-05-12
| | | | | | * basicially C&P from nDPI reader_utils but with some more sanity checks Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added nDPId semantic validation test.Toni Uhlig2021-04-14
| | | | | | | | | | | * fixed inconsistent processing of remaining flows during nDPId shutdown phase * fixed multiple `detected' flow events (instead only `detection-update' flow events can occur after a `detected' flow event) * fixed nDPIsrvd.py invalid message buffer handling * improved run_tests.sh so only valid pcap capture files are getting processed (and some more cosmetics + logging) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added IPv6 support for -I / -E.Toni Uhlig2021-04-11
| | | | | | | * added another Python search path and try-catch ModuleNotFoundError again * run_tests.sh checks for OpenBSD netcat (required for -q) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId-test: mimic full nDPId lifecycleToni Uhlig2021-04-09
| | | | | | | * generate DAEMON_EVENT_INIT as well as DAEMON_EVENT_SHUTDOWN * process remaining flows before shutdown (and generate events) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId-test: added JSON distribution + JSON parsing (Multithreaded design ↵Toni Uhlig2021-04-09
| | | | | | | | | | | | | | re-using most of nDPId/nDPIsrvd core) * improved Makefile.old install targets * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future) * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced * test run script checks and fails if certain files are are missing (PCAP file <=> result output file) * removed not very useful "internal format error" JSON serialization if a BUG for same exists * fixed invalid l4 type statistics counters for nDPIsrvd-collectd Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reverted internal ndpi structs to dynamic memory allocation hoping that it ↵Toni Uhlig2021-03-26
| | | | | | | | | | will help to reduce the average memory consumption. How? After the detection finished, internal ndpi structs can be free'd as they are not needed anymore. * Set the amount of max. packets to process via subopt. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added nDPId-test as all-in-one JSON dumper.Toni Uhlig2021-03-24
| | | | | | | | | * fixed invalid flow event schema type * added run_tests.sh to generate/diff JSON dumps * renamed lot's of vars/fns in nDPId.c/nDPIsrvd.c, so nDPId-test.c can include "*.c" * improved CMake dependency checks Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added JA3 / SSL SHA1 fingerprint blacklists.Toni Uhlig2021-02-27
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Finalized examples/c-captured to dump packet bytes to PCAP for further analysis.Toni Uhlig2021-02-24
| | | | | | * Fixed memory holes in nDPId structs. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Extended nDPIsrvd.h with address parsing.Toni Uhlig2021-02-23
| | | | | | | | | * nDPId supports looading of custom nDPI protocol/category files * extended JSON schemas according to nDPI / nDPId JSON serializing * removed memory holes in nDPId * extended examples/c-captured Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd C API overhaul and massive simplification.Toni Uhlig2021-02-06
| | | | | | | | * nDPIsrvd.h does flow mgmt out of the box * dissect received JSON strings via callback * added new JSON key/values for packet-flows (usecTimestamp/L3/L4 info) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed braindead failure - Increase of NETWORK_BUFFER_MAX_SIZE > 9999 means ↵Toni Uhlig2021-01-27
| | | | | | | | | to also increase NETWORK_BUFFER_LENGTH_DIGITS to 5. * Fixed ARM32 xcompile warnings; Other GCC versions, other uint64_t's.. * Replaced ridiculous nDPIsrvd_JSON_BYTES with NETWORK_BUFFER_LENGTH_DIGITS. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Increased JSON buffer size to 12288 (libnDPI serializes more and more ↵Toni Uhlig2021-01-27
| | | | | | | | | | | information). * Making Compare&Fetch mandatory. * Added some more Compare&Fetch to prevent TSAN complaining about data races. Fixed possible but more ore less harmless data races during shutdown process. * Shrink SIGNAL handler to a minimum. SYSV Signal handling and MT-safety is awkward. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* All colors are beautiful for py-flow-info.Toni Uhlig2021-01-26
| | | | | | * instance alias is now mandatory (will be required for future use-cases) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Switched to a slightly modified base64 encoder basicially from wikibooks.Toni Uhlig2021-01-20
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added internal/external packet processing modes. #2Toni Uhlig2021-01-10
| | | | | | * fixed incorrect handling of skipped flows Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added internal/external packet processing modes.Toni Uhlig2021-01-06
| | | | | | | | * only IPv4 supported for now * refactored nDPId's internal IP address storage * use fresh ndpi_free_flow_data() to free nDPI's dynamic allocated data Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Declared nDPI id and flow structs in nDPId flow info struct.Toni Uhlig2020-12-30
| | | | | | | | Two reasons: * reduce heap memory allocations * nDPId flow info struct may be inflated in the future (more bytes to compress) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Set an optional bpf filter string for pcap based packet capture.Toni Uhlig2020-12-29
| | | | | | * serialize and send datalink layer type Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Distributor clients which are too slow can cause buffer bloat.Toni Uhlig2020-12-19
| | | | | | | | | Switching back to blocking mode works as a quick fix but is not sufficient. See comments. * nDPId prints more accurate error messages if command line argument validation failed Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId option validation is now less restrictiveToni Uhlig2020-12-14
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Let travis use pcap files from nDPI to produce some JSON output. (disabled, ↵Toni Uhlig2020-12-01
| | | | | | | | | | | needs further testing..) * Added pcap diff script * Added \n to JSON string end (useful for debugging and readability) * Use first host/server name character for hash calculation as well * Removed error'ing EPOLLHUP handling in nDPIsrvd (connection closing will be detected via read()) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Moved datalink layer processing into a separate function.Toni Uhlig2020-11-18
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* More detailed `nanosleep() before close()' description.Toni Uhlig2020-11-11
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved default capture device handling if none set via `-i'.Toni Uhlig2020-11-08
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Make sure that DAEMON_EVENT_SHUTDOWN gets transmitted before close() (hacky, ↵Toni Uhlig2020-11-08
| | | | | | but works). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId/nDPIsrvd: change_user_group does now chown/chmod collector/distributor ↵Toni Uhlig2020-10-07
| | | | | | socket paths Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId: Fixed BUG if flow was guessed but "Unknown" detected protocol serialized.Toni Uhlig2020-10-01
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added additional datalink header check for DLT_NULL/IPv6.Toni Uhlig2020-09-28
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added alias support for an nDPId instance.Toni Uhlig2020-09-27
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added daemon events.Toni Uhlig2020-09-27
| | | | | | * Added missing IPv4 / IPv6 datalink layer (if no layer2 protocol used). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.