structured JSON logging

[darkexplosiveqwx]

Thank you for your contribution to the Pi-hole Community!

Please read the comments below to help us consider your Pull Request.

We are all volunteers and completing the process outlined will help us review your commits quicker.

Please make sure you

1. Base your code and PRs against the repositories developmental branch.
2. Sign Off all commits as we enforce the DCO for all contributions
3. Sign all your commits as they must have verified signatures
4. File a pull request for any change that requires changes to our documentation at our documentation repo

---

What does this PR aim to accomplish?:

Builds upon #2958 and #2960
Discussed in #2897

Todo: possibly upstream the don't redirect stdout patch to dnsmasq

How does this PR accomplish the above?:

Link documentation PRs if any are needed to support this PR:

---

By submitting this pull request, I confirm the following:

1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
2. I have commented my proposed changes within the code and I have tested my changes.
3. I am willing to help maintain this change if there are issues with it later.
4. It is compatible with the EUPL 1.2 license
5. I have squashed any insignificant commits. (git rebase)
6. I have checked that another pull request for this purpose does not exist.
7. I have considered, and confirmed that this submission will be valuable to others.
8. I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
9. I give this submission freely, and claim no ownership to its content.

---

• I have read the above and my PR is ready for review. Check this box to confirm

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[DL6ER]

Following up on the allocation point from #2897, with more detail on why it matters here.

my_syslog() runs inside dnsmasq's request handling, once or twice per query, and #2960 drops log-async, so the write is already synchronous on the DNS thread. This is the absolute hot path - anything we add lands on query latency. Today it allocates nothing: snprintf() into a stack buffer and a write(), no cJSON in sight. log_to_json() adds an object, two cJSON_strdup()s per field, and cJSON_PrintUnformatted() with its growing buffer, so roughly fifteen malloc/free pairs per line from a thread already contending for the allocator.

The schema is six fixed keys, so please emit it directly into a stack buffer. Only message needs escaping, the other five are ours. escape_string() and escape_data() are not reusable - they emit C-style \xNN and allocate - but this is enough:

// Escape into a caller-supplied buffer so the log path needs no allocation.
// Never splits an escape, so a short buffer truncates to valid JSON.
static size_t json_escape(char *out, const size_t outlen, const char *in)
{
	static const char hex[] = "0123456789abcdef";
	size_t o = 0;

	for(const unsigned char *p = (const unsigned char *)in; *p != '\0'; p++)
	{
		// Widest form is \u00XX, and every escape starts with a backslash
		char esc[6] = { '\\' };
		size_t len = 2;

		switch(*p)
		{
			case '"':  esc[1] = '"';  break;
			case '\\': esc[1] = '\\'; break;
			case '\b': esc[1] = 'b';  break;
			case '\f': esc[1] = 'f';  break;
			case '\n': esc[1] = 'n';  break;
			case '\r': esc[1] = 'r';  break;
			case '\t': esc[1] = 't';  break;

			default:
				// Printable and UTF-8 bytes pass through
				if(*p >= 0x20)
				{
					esc[0] = (char)*p;
					len = 1;
					break;
				}

				// Other control characters have no short form
				esc[1] = 'u';
				esc[2] = '0';
				esc[3] = '0';
				esc[4] = hex[*p >> 4];
				esc[5] = hex[*p & 0x0f];
				len = 6;
				break;
		}

		// Stop on the last character that fits, keeping room for the NUL
		if(o + len >= outlen)
			break;

		memcpy(out + o, esc, len);
		o += len;
	}

	out[o] = '\0';
	return o;
}

Valid UTF-8 survives as is. Also, printf("%s\n", out) bypasses the cached descriptor writer from #2960, so this path gets stdio buffering rather than the O_APPEND write - worth folding in.

[github-actions]

Conflicts have been resolved.

[DL6ER]

set_log_path(), ctime_r(), the syslog fallback for dnsmasq lines, the reopen warning after a flush and the off < 0 clamps all look right now.

Three things left.

log_to_json() is unchanged: still an object, six cJSON_AddStringToObject() and cJSON_PrintUnformatted() per line, then printf(). That is the allocation churn on the DNS hot path from my earlier comment, and the printf() also bypasses the cached descriptor writer this branch introduces.

open_log_fds(false) still sits after readFTLconf() in main.c, and write_dnsmasq_config() runs inside it, so config-test output still meets a descriptor of -1. The new syslog fallback may well cover that window now - worth confirming rather than assuming.

Small one: the ctime_r() failure fallback is "Jan 1 00:00:00 ", but it is read as ctime_str + 4, which skips a weekday the fallback does not have and leaves " 1 00:00:00 ". It wants a full ctime-shaped string, e.g. "Thu Jan 1 00:00:00 1970".

validate_filepath_dash is still gone, so files.log.dnsmasq = "-" stays on the breaking-change list.

[darkexplosiveqwx]

log_to_json() is unchanged: still an object, six cJSON_AddStringToObject() and cJSON_PrintUnformatted() per line, then printf(). That is the allocation churn on the DNS hot path from my earlier comment, and the printf() also bypasses the cached descriptor writer this branch introduces.

This has already been addressed in f4b1477

open_log_fds(false) still sits after readFTLconf() in main.c, and write_dnsmasq_config() runs inside it, so config-test output still meets a descriptor of -1. The new syslog fallback may well cover that window now - worth confirming rather than assuming.

Part of #2960 and already addressed in 91c3ed2

Small one: the ctime_r() failure fallback is "Jan 1 00:00:00 ", but it is read as ctime_str + 4, which skips a weekday the fallback does not have and leaves " 1 00:00:00 ". It wants a full ctime-shaped string, e.g. "Thu Jan 1 00:00:00 1970".

Part of #2960 and now addressed in b8c1697 (I have not yet rebased this branch onto that commit)

validate_filepath_dash is still gone, so files.log.dnsmasq = "-" stays on the breaking-change list.

Do you think injecting log-facility=- if it is - and disabling out fallback paths would be better than making this a breaking change? (We cannot remove echo_stderr anyway, since it is needed for test_dnsmasq_config())

[DL6ER]

You are right on all three, and my point about open_log_fds() was simply wrong - 91c3ed2 adds the early call in readFTLconf() before write_dnsmasq_config(), which is exactly what I asked for. I checked the ordering in main.c, saw it unchanged and stopped there, without looking at config.c in the same commit. Sorry for the noise.

One thing on the new log_to_json(). escaped_msg and line are both 8192, but line also carries the timestamp, level, component, pid and about 90 bytes of framing, and msg arrives from a json_buffer[8192] where escaping can grow it up to sixfold. A long or escape-heavy message therefore overflows line, and snprintf() cuts it mid-string - no closing quote, no brace, no newline. That is the one outcome the escaper's careful truncation is meant to avoid, so line should be big enough for escaped_msg plus the framing, or truncation should be detected and a shorter valid record emitted instead.

On files.log.dnsmasq = "-": the retirement itself is fine, I only want it written down. v7 is a breaking release and the logging rework is already on the breaking-change list, so retiring one config value fits what is being announced anyway. Keeping it would mean carrying a second writer for a single setting, leaving the dnsmasq path reachable and in need of testing rather than letting it become dead code, and that is the more expensive option, so I would leave it retired.

Dropping validate_filepath_dash() was the right call as the branch stands - once FTL owns the file and stops emitting log-facility, the dash would reach open() and create a file called -. Just list it with the rest of the logging changes so nobody still using - meets it as a surprise. echo_stderr staying for test_dnsmasq_config() is unrelated and fine.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.