aboutsummaryrefslogtreecommitdiff
path: root/schema
Commit message (Collapse)AuthorAge
* bump libnDPI to c49d126d3642d5b1f5168d049e3ebf0ee3451edcHEADtmpmainToni Uhlig2025-03-06
| | | | | | * fix API issue with a changed function signature Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Initial tunnel decoding (GRE - Layer4 only atm) (#55)Toni2025-02-25
| | | | | | | | | | Initial tunnel decoding (GRE - Layer4 only atm). Fixes #53 * make finally use of the thread distribution seed * Handle GRE/PPP subprotocol the right way * Add `-t` command line / config option * Removed duplicated and obsolete IP{4,6}_SIZE_SMALLER_THAN_HEADER which is the same as IP{4,6}_PACKET_TOO_SHORT * Updated error event schema Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bump libnDPI to e946f49aca13e4447a7d7b2acae6323a4531fb55Toni Uhlig2025-01-25
| | | | | | * incorporated upstream changes Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId daemon status event: serialize nDPI API version + Size/FlowToni Uhlig2024-11-17
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bump libnDPI to 59ee1fe1156be234fed796972a29a31a0589e25aToni Uhlig2024-11-13
| | | | | | | | | | * set minimum nDPI version to 4.12.0 (incompatible API changes) * fixed `ndpi_debug_printf()` function signature * JSON schema (flow): added risk `56`: "Obfuscated Traffic" * JSON schema (flow): added "domainame" * fixed OpenWrt build Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added `vlan_id` dissection of the most outer (first) 802.1Q header. Fixes #50Toni Uhlig2024-11-02
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Updated JSON schema files and a test to make use of the UUID feature.Toni Uhlig2024-10-29
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bump libnDPI to 92507c014626bc542f2ab11c729742802c0bc345Toni Uhlig2024-09-09
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Updated flow event schema with risk names/severites.Toni Uhlig2024-09-03
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Extend flow JSON schema with more properties from nDPI JSON serializer.Toni Uhlig2024-08-22
| | | | | | * unfortunately, JSON schema definitions could not be used to make this easier to read and maintain Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add PF_RING support. (#38)Toni2024-08-19
|
* Added global (heap) memory stats for daemon status events.Toni Uhlig2024-06-19
| | | | | | | | * added new CMake option `ENABLE_MEMORY_STATUS` to restore the old behavior (and increase performance) * splitted `ENABLE_MEMORY_PROFILING` into `ENABLE_MEMORY_STATUS` and `ENABLE_MEMORY_PROFILING` Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Replaced ambiguous naming of "JSON string" to more accurate "JSON message".Toni Uhlig2023-12-04
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Serialize nDPId / libnDPI versions within daemon events.Toni Uhlig2023-09-10
| | | | | | * changed nDPI version hints / requirements Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bump libnDPI to ...Toni Uhlig2023-05-24
| | | | | | | | * upstream changed regression test interface, needed to adapt * improved libnDPI helper build script * updated JSON schema Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Moved `datalink` json key/value from error to packet events and renamed it ↵Toni Uhlig2023-04-21
| | | | | | to `pkt_datalink`. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved the point of time when to append the raw packet base64 data to the ↵Toni Uhlig2022-11-15
| | | | | | | | | serializer. * nDPId-test: Increased the max-packets-per-flow-to-send from 3 to 5. This is quite useful for TCP as the first 3 packets are usually part of the three-way-handshake. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Sync unit tests.Toni Uhlig2022-11-13
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bump libnDPI to db9f6ec1b4018164e5bff05f115dc60711bb711bToni Uhlig2022-10-31
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId: Added error event threshold to prevent event spamming which may be ↵Toni Uhlig2022-10-17
| | | | | | abused. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added nDPIsrvd-analysed to generate CSV files from analyse events.Toni Uhlig2022-09-30
| | | | | | | * nDPIsrvd.h: iterate over JSON arrays * nDPId: calculate l3 payload packet entropies for analysis Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved flown analyse event:Toni Uhlig2022-09-22
| | | | | | | | * store packet directions * merged direction based IATs * merged direction based PKTLENs Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Merge branch 'main' of github.com:utoni/nDPIdToni Uhlig2022-09-19
|\
| * Improve READMEDamiano Verzulli2022-09-19
| | | | | | | | | | | | | | - link to already-existing JSON-schemas have been added - a graphical schema detailing flow-events timeline have been added in both PNG and source-Drawio formats. Link to PNG have been included in the README
* | nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.Toni Uhlig2022-09-19
|/ | | | | | | | | | | | * nDPId: Fixed broken validation tests. * nDPId: Removed TICK_RESOLUTION, not required anymore. * c-collectd: Improved total layer4 payload calculation/update handling. * c-collectd: Updated RRD Graph script according to total layer4 payload changes. * py-flow-info.py: Fixed several bugs and syntax errors. * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py. * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added layer4 payload length bins.Toni Uhlig2022-09-19
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* New flow event: 'analysis'.Toni Uhlig2022-09-13
| | | | | | | | | * The goal was to provide a separate event for extracted feature that are not required and only useful for a few (e.g. someone who wants do ML). * Increased network buffer size to 32kB (8192 * 4). * Switched timestamp precision from ms to us for *ALL* timestamps. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Bump libnDPI to 37f918322c0a489b5143a987c8f1a44a6f78a6f3 and updated flow ↵Toni Uhlig2022-09-06
| | | | | | | | | json schema file. * export env vars AR / CMAKE_C_COMPILER_AR and RANLIB / CMAKE_C_COMPILER_RANLIB while building libnDPI * nDPId check API version during startup (macro vs. function call) and print a warning if they are different Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Split `*_l4_payload_len' into `*_src_l4_payload_len' and `*_dst_l4_payload_len'.Toni Uhlig2022-08-15
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Split `flow_packets_processed' into `flow_src_packets_processed' and ↵lns2022-08-15
| | | | | | | | | `flow_dst_packets_processed'. * no use for `flow_avg_l4_payload_len' -> removed * test/run_tests.sh does not fail if git-worktree's are used Signed-off-by: lns <matzeton@googlemail.com>
* bump libnDPI to 8f6a006e36eef0ae386f7e663d3ebecfad6a2dc9Toni Uhlig2022-07-01
| | | | | | | * try to use same wording wherever possible e.g. renamed workflow->total_l4_data_len to workflow->total_l4_payload_len Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* bump libnDPI to bb12837ca75efc2691ecb18fd5f56e2d097ef26bToni Uhlig2022-03-24
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked nDPIsrvd.h C-API.Toni Uhlig2022-03-21
| | | | | | | | | | | | * nDPIsrvd.h: Provide nDPId thread storage. * nDPIsrvd.py: Fixed instance cleanup bug. * nDPIsrvd.h: Support for instance/thread user data and cleanup callback. * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht. * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling. * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!) * nDPId: Added information about ZLib compressions to daemon status/shutdown events. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Renamed basic event to error event for the sake of the logic.Toni Uhlig2022-03-13
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Disconnect nDPIsrvd clients immediately instead waiting for a failed write().Toni Uhlig2022-03-13
| | | | | | | | | * nDPIsrvd: Collector/Distributor logging improved * nDPIsrvd: Command line option for max remote descriptors * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again * nDPId: Refactored some variable names and events Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)Toni Uhlig2022-03-06
| | | | | | | | | | * Improved distributor timeout handling (per-thread). * flow-info.py / flow-dash.py: Distinguish between flow risk severities. * nDPId: Skip tag switch datalink packet dissection / processing. * nDPId: Fixed incorrect value for current active flows. * Improved JSON schema's. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added additional (minimalistic) detection information to flow updates.Toni Uhlig2022-02-04
| | | | | | | | | | | | | This will only affect flows with the state `FT_FINISHED' (detection done). * nDPIsrvd.py: force use of JSON schema Draft 7 validator * flow-dash.py: gather/use total processed layer4 payload size * flow-info.py: added additional event filter * flow-info.py: prettified flow events printing whose detection is in progress * py-semantic-validation.py: added validation checks for FT_FINISHED * updated flow event JSON schema Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Process additional layer 3 protocols.Toni Uhlig2022-01-31
| | | | | | | | | * bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af * run_tests.sh: improved execution from non-git directories e.g. via `make dist` * updated JSON schema to be more restrictive * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId: Added some error messages when workflow init fails.Toni Uhlig2022-01-26
| | | | | | | * Fixed invalid array subscript typ0 (caused some trouble..) * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Major nDPId extension. Sorry for the huge commit.Toni Uhlig2022-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - nDPId: fixed invalid IP4/IP6 tuple compare - nDPIsrvd: fixed caching issue (finally) - added tiny c example (can be used to check flow manager sanity) - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow` - README.md update: added example JSON sequence - nDPId: added new flow event `update` necessary for correct timeout handling (and other future use-cases) - nDPIsrvd.h and nDPIsrvd.py: switched to an instance (consists of an alias/source tuple) based flow manager - every flow related event **must** now serialize `alias`, `source`, `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout handling and verification process work correctly - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation - nDPIsrvd.py: removed PcapPacket class (unused) - py-flow-dashboard and py-flow-multiprocess: fixed race condition - py-flow-info: print statusbar with probably useful information - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`) to a generic flow event timestamp `ts_msec` - nDPId-test: added additional checks - nDPId: increased ICMP flow timeout - nDPId: using event based i/o if capturing packets from a device - nDPIsrvd: fixed memory leak on shutdown if remote descriptors were still connected Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed invalid "flow_last_seen" timestamp for the first packet.Toni Uhlig2021-10-08
| | | | | | | | | | * After the first packet was processed, "flow_last_seen" was still 0. This behaviour is invalid as the first packet may contain l4 payload data e.g. for UDP and it also breaks nDPId json consistency "flow_first_seen" > 0, but "flow_last_seen" == 0. * JSON schema: set minimum timestamp value for Epoch timestamps to 24710 for flow_*_seen and 1 for pcap packet ts. Those values are dependant on some manipulated pcap's in libnDPI/tests/pcap. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked layer 4 flow length naming/calculation.Toni Uhlig2021-05-20
| | | | | | | * nDPIsrvd services usually do not care about layer4 data length, payload length is quite more essential for further processing Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Use layer4 specific flow timeouts.Toni Uhlig2021-05-13
| | | | | | * default values "stolen" from nf_conntrack Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added JSON schema validation to run_tests.shToni Uhlig2021-04-09
| | | | | | | * Python3 scripts are now compatible with versions <3.6 * improved and prettified run_tests.sh Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Updated README.mdToni Uhlig2021-03-24
| | | | | | * added examples/README.md and schema/README.md Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added nDPId-test as all-in-one JSON dumper.Toni Uhlig2021-03-24
| | | | | | | | | * fixed invalid flow event schema type * added run_tests.sh to generate/diff JSON dumps * renamed lot's of vars/fns in nDPId.c/nDPIsrvd.c, so nDPId-test.c can include "*.c" * improved CMake dependency checks Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Extended nDPIsrvd.h with address parsing.Toni Uhlig2021-02-23
| | | | | | | | | * nDPId supports looading of custom nDPI protocol/category files * extended JSON schemas according to nDPI / nDPId JSON serializing * removed memory holes in nDPId * extended examples/c-captured Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added JSON schema files and a Python schema validator.Toni Uhlig2021-02-17
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.