Ingest New Documentation

docs/Developer and Contributor Corner/Database Engine.mdx

@@ -101,7 +101,7 @@ Each **datafile** has two **journal files** with metadata related to the stored
 
 - **journal file v1**, with filename suffix `.njf`, holds information about the transactions in its **datafile** and provides the ability to recover as much data as possible, in case either the datafile or the journal files get corrupted. This journal file has a maximum transaction size of 4KB, so in case data are corrupted on disk transactions of 4KB are lost. Each transaction holds the metadata of one **extent** (this is why DBENGINE supports up to 64 pages per extent).
 
-- **journal file v2**, with filename suffix `.njfv2`, which is a disk-based index for all the **pages** and **extents**. This file is memory mapped at runtime and is consulted to find where the data of a metric are in the datafile. This journal file is automatically re-created from **journal file v1** if it is missing. It is safe to delete these files (when Netdata does not run). Netdata will re-create them on the next run. Journal files v2 are supported in Netdata Agents with version `netdata-1.37.0-115-nightly`. Older versions maintain the journal index in memory.
+- **journal file v2**, with filename suffix `.njfv2`, which is a disk-based index for all the **pages** and **extents**. This file is memory mapped at runtime and is consulted to find where the data of a metric are in the datafile. If this journal file is missing or fails its checks at startup, Netdata rebuilds it from **journal file v1**. This requires the journal file v1 to be readable; otherwise the data of that datafile are not available. It is safe to delete these files (when Netdata does not run), as long as their journal file v1 is intact. Netdata will re-create them on the next run. Journal files v2 are supported in Netdata Agents with version `netdata-1.37.0-115-nightly`. Older versions maintain the journal index in memory.
 
   Newer agents also write an optional **samples section** in each **journal file v2**: the number of slots (points, empty ones included) of every metric in that file, derived from the time ranges of its pages, and their total in the section descriptor (so that startup reads only the header page). Only pages the index can reach are counted: if a file holds two pages of a metric with the same start time, the index keeps one of them (the one that ends later; if both end together, the one written earlier in the datafile) and only that one is counted. The per-tier `samples` reported in `db_size` by `/api/v2/info` is the sum of what each datafile holds: exact for files that have this section, estimated from the metric time ranges for files without it (files written by older agents). A page whose descriptor on disk is malformed is counted as what queries read from it (after repair), which can differ from the slots physically stored. Pages are counted when they are written to disk, so pages still waiting to be flushed are not included yet. Agents that do not know this section ignore it.
 

docs/Developer and Contributor Corner/External Plugins/External Plugins.mdx

@@ -540,16 +540,18 @@ When the function to be called is to receive a payload of parameters, the call l
 
 ```
 FUNCTION_PAYLOAD transaction_id timeout "name and parameters of the function as one quoted parameter" "user permissions value" "source of request" "content/type"
-body of the payload, formatted according to content/type
-FUNCTION PAYLOAD END
+body of the payload, a JSON document
+FUNCTION_PAYLOAD_END
 ```
 
 In this case, Netdata will send:
 
 - A line starting with `FUNCTION_PAYLOAD` together with the required metadata for the function, like the transaction id, the function name and its parameters, the timeout and the content type. This line ends with a newline.
-- Then, the payload itself (which may or may not have newlines in it). The payload should be parsed according to the content type parameter.
+- Then, the payload itself: a JSON document, which may or may not have newlines in it.
 - Finally, a line starting with `FUNCTION_PAYLOAD_END`, so it is expected like `\nFUNCTION_PAYLOAD_END\n`.
 
+The payload is not escaped or length-prefixed, so Netdata only sends payloads that are a single strict JSON document (RFC 8259: double-quoted strings without raw control characters, no comments, nothing but whitespace after the document) that do not contain the text `FUNCTION_PAYLOAD_END` anywhere and have no line starting (after spaces, tabs or carriage returns) with a quoted plugins.d keyword (`"FUNCTION"`, `"FUNCTION_PAYLOAD"`, `"FUNCTION_CANCEL"`, `"FUNCTION_PROGRESS"` or `"QUIT"`), and always labels them `application/json`. An empty payload means no payload block and is accepted. Any other payload is refused with HTTP 400. Plugins must treat the payload as opaque data: only a line that is exactly `FUNCTION_PAYLOAD_END` ends it, and no other line inside it is a command.
+
 Note 1: The plugins.d protocol allows parameters without single or double quotes if they don't contain spaces. However, the plugin should be able to parse parameters even if they are enclosed in single or double quotes. If the first character of a parameter is a single quote, its last character should also be a single quote too, and similarly for double quotes.
 
 Note 2: Netdata always sends the function and its parameters enclosed in double quotes. If the function command and its parameters contain quotes, they are converted to single quotes.
@@ -755,7 +757,7 @@ or (for commands: `update`, `add` and `test`):
 
 ```
 FUNCTION_PAYLOAD transaction_id timeout "config id command" "user permissions value" "source string" "content/type"
-body of the payload formatted according to content/type
+body of the payload, a JSON document
 FUNCTION_PAYLOAD_END
 ```
 

ingest/generated_sidebar_order.json

@@ -2,7 +2,7 @@
   "schema_version": 1,
   "source": "netdata/docs/.map/map.yaml",
   "source_sha256": "5c613e8fa52bfee42b4c57d2ad18bf26ad3c1865ca5a5654eca0c66042d0c280",
-  "source_corpus_sha256": "de7a5675d246ea0d3da864bc525006c09463672a1d9943328c850602ed3d9dc6",
+  "source_corpus_sha256": "775250a2c1866cf310c5325ea254fa2f4bab7b110e815cd3082923706ed86fe9",
   "order": [
     {
       "parent_path": "Alerts & Notifications",
@@ -1810,5 +1810,5 @@
       "position": 190
     }
   ],
-  "full_ingest_identity_sha256": "4bbffdbaa1142572c6ad53d687e955c07c5fa9af12c57b86fdfa4f3df142dfc8"
+  "full_ingest_identity_sha256": "e6d29899bdff5baef8461c1e2aca8addfe869af9c05c451e9b5edaba67bdf866"
 }

ingest/generated_sidebar_order.json.sha256

@@ -1 +1 @@
-1b413ac0b98666f66ea65ae63a16b4b7ca0ca59bf7172e2cb53160757fb2e307  generated_sidebar_order.json
+1af7ee2472871ccbde34091774fa15d106a9585f19a3a2e816592be8ff29f931  generated_sidebar_order.json