summaryrefslogtreecommitdiff
path: root/nDPIsrvd.c
Commit message (Collapse)AuthorAge
* Fixed possible buffer underflow.Toni Uhlig2023-11-20
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added SonarCloud exclusions for third-party files and files lacking relevance.Toni Uhlig2023-11-20
| | | | | | * fixed two other "bugs" Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed more SonarCloud complaints.Toni Uhlig2023-11-07
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Apple/BSD port (#30)Toni2023-11-06
| | | | | | | | * Add MacOS to Github CI builds. * Fixed libnDPI-4.8 CI build. * Fixed missing include for `struct sockaddr*`. * Reworked IPv4 address and netmask retrieval. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Event I/O abstraction layer. (#28)Toni2023-11-06
| | | | | | | | | | | * Finalize Event I/O abstraction layer. * Fix possible fd leakage, Gitlab-CI build and error logging. * Fixed possible uninitialized signalfd variable. * Fixed possible memory leak. * Fixed some SonarCloud complaints. * Fixed nDPId-test nDPIsrvd-arpa-mockup stuck indefinitely. * Add nDPId / nDPIsrvd command line option to use poll() on Linux instead of the default epoll(). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed bug which may happen if additional write buffers are empty but main ↵Toni Uhlig2023-10-19
| | | | | | | | write buffer not. * may cause nDPIsrvd to hang indefinitly if no more data received from a collector Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Log error if collector unix socket can not be removed.Toni Uhlig2023-09-11
| | | | | | * systemd: add post stop hook to forcefully remove the collector unix socket Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* keras-autoencoder.py: fixed invalid preprocessing of received base64 packet dataToni Uhlig2023-08-15
| | | | | | | * split logic into seperate jobs; nDPIsrvd and Keras * nDPIsrvd: break event processing and re-run `epoll_wait()` after client disconnected Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Correctly handle EINTR while doing I/O..Toni Uhlig2023-07-18
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Limit the size of base64 serialized raw packet data (8192 bytes per packet).Toni Uhlig2022-12-06
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId-test: Reworked I/O handling to prevent some endless loop scenarios. ↵Toni Uhlig2022-12-02
| | | | | | | | Fixed a race condition in the memory wrapper as well. * nDPId: Instead of sending too long JSON strings, log an error and some parts. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Unified nDPId/nDPIsrvd command line argument storage.Toni Uhlig2022-11-21
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Provide workaround for change user/group.Toni Uhlig2022-11-13
| | | | | | | | * nDPId/nDPIsrvd/c-examples: Parameter parsing needs to be improved if `strdup()` in combination with static strings is used. * Other non-critical fixes. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Experimental systemd support.Toni Uhlig2022-11-06
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPid-test: add buffer testToni Uhlig2022-10-30
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed heap overflow on shutdown caused by missing remotes size/used reset.Toni Uhlig2022-10-05
| | | | | | * introduced with 22a8d04c748ff3e04ca11c5c29b4433c7136f5f5 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Refactored client distributor C API.Toni Uhlig2022-09-25
| | | | | | | | | | | * Still not perfect, but the code before was not even able to deal with JSON arrays. Use common "speaking" function names for all functions in nDPIsrvd.h * Provide a more or less generic and easy extendable JSON walk function. * Modified C examples to align with the changed C API. * c-collectd: Reduced lot's of code duplication by providing mapping tables. * nDPId: IAT array requires one slot less (first packet has always an IAT of 0). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved flown analyse event:Toni Uhlig2022-09-22
| | | | | | | | * store packet directions * merged direction based IATs * merged direction based PKTLENs Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.Toni Uhlig2022-09-19
| | | | | | | | | | | | * nDPId: Fixed broken validation tests. * nDPId: Removed TICK_RESOLUTION, not required anymore. * c-collectd: Improved total layer4 payload calculation/update handling. * c-collectd: Updated RRD Graph script according to total layer4 payload changes. * py-flow-info.py: Fixed several bugs and syntax errors. * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py. * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* New flow event: 'analysis'.Toni Uhlig2022-09-13
| | | | | | | | | * The goal was to provide a separate event for extracted feature that are not required and only useful for a few (e.g. someone who wants do ML). * Increased network buffer size to 32kB (8192 * 4). * Switched timestamp precision from ms to us for *ALL* timestamps. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Fixed NUL pointer deref during logging attempt.Toni Uhlig2022-09-10
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId: Merged nDPId_flow_(info|finished) into nDPId_flowlns2022-04-24
| | | | | | | * nDPIsrvd: Fixed buffer allocation error due to missing memset() on disconnect * nDPIsrvd: Removed unused struct members Signed-off-by: lns <matzeton@googlemail.com>
* Added proper DLT_RAW dissection for IPv4 and IPv6.lns2022-04-24
| | | | | | | | | | * nDPId: Improved TCP timeout handling if FIN/RST seen which caused Midstream TCP flows when there shouldn't be any. * nDPIsrvd: Unified remote descriptor resource cleanup on disconnects/shutdown. * nDPIsrvd: Added additional error messages for remote descriptors. * py-flow-info: Better daemon status message printing. Signed-off-by: lns <matzeton@googlemail.com>
* Refactored buffer subsystem.lns2022-04-16
| | | | Signed-off-by: lns <matzeton@googlemail.com>
* Disconnect nDPIsrvd clients immediately instead waiting for a failed write().Toni Uhlig2022-03-13
| | | | | | | | | * nDPIsrvd: Collector/Distributor logging improved * nDPIsrvd: Command line option for max remote descriptors * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again * nDPId: Refactored some variable names and events Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added nDPIsrvd TCP/IP support for distributors.Toni Uhlig2022-03-10
| | | | | | | | * nDPIsrvd: Improved distributor client disconnect detection * nDPIsrvd: Fixed invalid usage of epoll_add instead of epoll_mod * nPDIsrvd: Improved logging for distributor clients Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added logging interface used by nDPId, nDPIsrvd and nDPId-test.Toni Uhlig2022-02-03
| | | | | | | | | * fixed GitLab pipeline * nDPId: added static assert (just for a test) * nDPId: memory profiling for total bytes compressed * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Fixed memory leak caused be not clearing buffer cache after a ↵Toni Uhlig2022-01-26
| | | | | | | | client disconnected. * README.md: Fixed a typ0 and added a meh image from examples/py-flow-dashboard/flow-dash.py Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Major nDPId extension. Sorry for the huge commit.Toni Uhlig2022-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - nDPId: fixed invalid IP4/IP6 tuple compare - nDPIsrvd: fixed caching issue (finally) - added tiny c example (can be used to check flow manager sanity) - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow` - README.md update: added example JSON sequence - nDPId: added new flow event `update` necessary for correct timeout handling (and other future use-cases) - nDPIsrvd.h and nDPIsrvd.py: switched to an instance (consists of an alias/source tuple) based flow manager - every flow related event **must** now serialize `alias`, `source`, `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout handling and verification process work correctly - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation - nDPIsrvd.py: removed PcapPacket class (unused) - py-flow-dashboard and py-flow-multiprocess: fixed race condition - py-flow-info: print statusbar with probably useful information - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`) to a generic flow event timestamp `ts_msec` - nDPId-test: added additional checks - nDPId: increased ICMP flow timeout - nDPId: using event based i/o if capturing packets from a device - nDPIsrvd: fixed memory leak on shutdown if remote descriptors were still connected Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* MemoryProfiling: Advanced flow usage logging.Toni Uhlig2021-11-13
| | | | | | | | * nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined * nDPId-test: mock syslog flags and functions * gitlab-ci: force -Werror Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId-test: Set max buffer size for remote descriptors useful to test ↵Toni Uhlig2021-11-04
| | | | | | caching/buffering. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Fixed anther bug, introduced during refactoring -_-Toni Uhlig2021-09-15
| | | | | | nDPId-test: Collect information about JSON string length's. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fixed Warnings / build error / cosmeticsToni Uhlig2021-09-14
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved and Fixed another buffering issue caused by removing an outgoing fd ↵Toni Uhlig2021-08-05
| | | | | | too early from epoll queue (EPOLLOUT). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved nDPIsrvd buffer bloat handling using caching.Toni Uhlig2021-08-04
| | | | | | | * still allow blocking mode (with send timeout) * improved daemon start/stop test script Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Git tag/commit version printing for nDPId/nDPIsrvd. Reduces confusion.Toni Uhlig2021-06-08
| | | | | | * disabled subshell spawn for run_tests.sh, common pitfall while using counters Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Build system cleanup / cosmetics.1.4Toni Uhlig2021-06-07
| | | | | | * libnDPI submodule update Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Unified IO buffer mgmt.Toni Uhlig2021-06-07
| | | | | | | * c-collectd gives the user control over collectd-exec instance name * added missing collectd type `flow_l4_icmp_count` Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* travis-ci build CMake projectToni Uhlig2021-04-15
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId-test: added JSON distribution + JSON parsing (Multithreaded design ↵Toni Uhlig2021-04-09
| | | | | | | | | | | | | | re-using most of nDPId/nDPIsrvd core) * improved Makefile.old install targets * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future) * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced * test run script checks and fails if certain files are are missing (PCAP file <=> result output file) * removed not very useful "internal format error" JSON serialization if a BUG for same exists * fixed invalid l4 type statistics counters for nDPIsrvd-collectd Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added nDPId-test as all-in-one JSON dumper.Toni Uhlig2021-03-24
| | | | | | | | | * fixed invalid flow event schema type * added run_tests.sh to generate/diff JSON dumps * renamed lot's of vars/fns in nDPId.c/nDPIsrvd.c, so nDPId-test.c can include "*.c" * improved CMake dependency checks Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reduced code duplication. Preps for nDPId-test.Toni Uhlig2021-03-23
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd collectd-exec overhaul.Toni Uhlig2021-03-15
| | | | | | | * Install targets updated. * Removed nDPIsrvd.h token validation function (done automatically by token_get). Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd refactoringToni Uhlig2021-03-05
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Finalized examples/c-captured to dump packet bytes to PCAP for further analysis.Toni Uhlig2021-02-24
| | | | | | * Fixed memory holes in nDPId structs. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Extended nDPIsrvd.h with address parsing.Toni Uhlig2021-02-23
| | | | | | | | | * nDPId supports looading of custom nDPI protocol/category files * extended JSON schemas according to nDPI / nDPId JSON serializing * removed memory holes in nDPId * extended examples/c-captured Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed braindead failure - Increase of NETWORK_BUFFER_MAX_SIZE > 9999 means ↵Toni Uhlig2021-01-27
| | | | | | | | | to also increase NETWORK_BUFFER_LENGTH_DIGITS to 5. * Fixed ARM32 xcompile warnings; Other GCC versions, other uint64_t's.. * Replaced ridiculous nDPIsrvd_JSON_BYTES with NETWORK_BUFFER_LENGTH_DIGITS. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd: Distributor clients which are too slow can cause buffer bloat.Toni Uhlig2020-12-19
| | | | | | | | | Switching back to blocking mode works as a quick fix but is not sufficient. See comments. * nDPId prints more accurate error messages if command line argument validation failed Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Let travis use pcap files from nDPI to produce some JSON output. (disabled, ↵Toni Uhlig2020-12-01
| | | | | | | | | | | needs further testing..) * Added pcap diff script * Added \n to JSON string end (useful for debugging and readability) * Use first host/server name character for hash calculation as well * Removed error'ing EPOLLHUP handling in nDPIsrvd (connection closing will be detected via read()) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId/nDPIsrvd: change_user_group does now chown/chmod collector/distributor ↵Toni Uhlig2020-10-07
| | | | | | socket paths Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.