rest_client: optional response cache

Adds an opt-in cache for the HTTP responses this module fetches.  When
cachedb_url is set, repeated queries are served from a cachedb backend
instead of going back to the origin, in both the blocking and the
asynchronous flavour.  Nothing changes unless it is configured.

The module had no caching of any kind, so every rest_get() was a fresh
round trip even when the same worker had fetched the same URL moments
earlier.  For the usual OpenSIPS pattern - a per-call lookup against a
rating, routing or authorization API - that round trip is the most
expensive thing in the request path.

The module replicates nothing itself, so the backend URL alone decides
whether the cache is private to one instance or shared by a cluster.
Both are useful: an in-process backend gives the cheapest possible hit
and each instance warms its own cache, while a shared redis lets one
node's fetch serve the whole cluster and cuts the load on the origin, at
a network round trip per hit.  The rule is not local versus remote, only
that the backend be cheaper than the call it replaces.

Freshness does not rely on the backend acting on the TTL it is handed.
cachedb_mongodb, for one, takes the expires argument of its set() and
ignores it, so an entry written there would never expire; each entry
therefore carries an absolute expiry of its own and a hit past it is
treated as a miss.  That backend remains unsuitable regardless, since
nothing ever reclaims the space.

By default the origin decides.  A response is stored only if its own
Cache-Control and Expires headers permit it, and only for as long as
they allow; no-store, no-cache, private, Set-Cookie and Vary are all
refused, as are non-200 replies and bodies over cache_max_body.  Because
many internal APIs send no cache headers at all, cache_policy offers
heuristic (supply a lifetime only where the origin was silent) and force
(ignore origin directives), both rejected at startup without a positive
cache_ttl.  rest_cache_ctl() overrides a single call instead, which is
usually what is wanted - loosening the policy globally reinterprets
every endpoint, including the ones that were already right.

The cache key is MD5(method, URL, appended header list).  The headers
matter: rest_append_hf() queues a process-global list consumed by the
next request, so the same URL is routinely fetched with different
Authorization or tenant headers and legitimately returns different
bodies.  A URL-only key would hand one subscriber's response to another
and would pass any single-credential test.  Header names are lower-cased
and sorted so append order is irrelevant, and values are hashed rather
than stored, so a bearer token never reaches the backend's key space.

The key does not cover the request body, which is why only GET responses
are stored; rest_post() and rest_put() consult the cache and report
miss:method but never write to it.

Each function takes an optional trailing output variable reporting where
the answer came from - hit, miss, miss:<reason> or bypass - alongside
the rest_cache_hits/misses/stores/skipped statistics and a computed hit
rate.  Both exist because a cache that stores nothing, because every
origin forbids it, is otherwise indistinguishable from a working one.

Where the core provides the zero-copy get_buf cachedb endpoint and the
configured backend advertises it, it is used automatically; otherwise
get() is.  The dependency is soft, so this builds and works as is.

Author: Yury Kirsanov

modules/rest_client/README

@@ -0,0 +1,137 @@
+/*
+ * Copyright (C) 2026 OpenSIPS Project
+ *
+ * This file is part of opensips, a free SIP server.
+ *
+ * opensips is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * opensips is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+#ifndef _REST_CACHE_H_
+#define _REST_CACHE_H_
+
+#include "../../str.h"
+#include "../../cachedb/cachedb.h"
+#include "../../statistics.h"
+#include <curl/curl.h>
+
+/* what a lookup/store decided, reported to the script and the log */
+#define RCC_SRC_BYPASS      "bypass"
+#define RCC_SRC_HIT         "hit"
+#define RCC_SRC_MISS        "miss"
+
+/* module-wide policy: how far to go beyond what the origin explicitly allows */
+enum rcc_policy {
+	RCC_POLICY_OFF = 0,   /* never cache                                     */
+	RCC_POLICY_STRICT,    /* only what the origin explicitly permits         */
+	RCC_POLICY_HEURISTIC, /* also cache when the origin says nothing         */
+	RCC_POLICY_FORCE,     /* also cache when the origin says not to          */
+};
+
+/* per-call override, set by rest_cache_ctl() for the NEXT request only, in the
+ * same "pending state" style the module already uses for header_list */
+#define RCC_CTL_UNSET   (-1)   /* no override: module policy applies */
+#define RCC_CTL_BYPASS   (0)   /* skip the cache entirely for this call */
+
+/* upper bound on one cached response: header + ctype + body */
+#define RCC_MAX_VALUE   (64 * 1024 + 512)
+
+/*
+ * The caching directives worth keeping out of a response.  header_func() fills
+ * this in as the headers stream past; everything else is still discarded.
+ */
+struct rcc_resp_hdrs {
+	int   have_cc;          /* a Cache-Control header was seen              */
+	int   cc_no_store;
+	int   cc_no_cache;
+	int   cc_private;
+	int   have_maxage;      /* s-maxage or max-age was parsed               */
+	long  maxage;
+	int   have_expires;
+	time_t expires;         /* absolute, from the Expires header            */
+	long  age;              /* Age header, 0 if absent                      */
+	int   have_vary;
+	int   have_setcookie;
+};
+
+/* everything the cache needs to know about one in-flight request */
+struct rcc_ctx {
+	int          enabled;       /* cache consulted for this call at all     */
+	int          ctl_ttl;       /* per-call rest_cache_ctl(), captured at    */
+	                            /* dispatch: -1 none, 0 bypass, >0 force     */
+	int          store_ttl;     /* >0 = store for this many seconds         */
+	str          key;           /* MD5 hex of method+url+headers            */
+	char         keybuf[33];
+	const char  *src;           /* provenance string, see RCC_SRC_*         */
+	char         srcbuf[32];    /* holds "miss:<reason>"                    */
+	struct rcc_resp_hdrs hdrs;
+};
+
+extern char *rcc_cdb_url;
+extern char *rcc_policy_str;
+extern int   rcc_cache_ttl;
+extern int   rcc_max_body;
+extern enum rcc_policy rcc_policy;
+
+/* module life cycle */
+int  rcc_init(void);          /* parse policy, connect the backend          */
+void rcc_destroy(void);
+int  rcc_enabled(void);       /* a backend is configured and policy != off  */
+
+/* per-call override (rest_cache_ctl), cleared with the header list */
+extern int rcc_ctl_ttl;
+void rcc_ctl_reset(void);
+
+/*
+ * Build the cache key for this request.  @extra_hdrs is the pending
+ * rest_append_hf() list - it MUST take part, since the same URL can be fetched
+ * with different Authorization/Accept headers and get different answers.
+ */
+int  rcc_build_key(struct rcc_ctx *ctx, const char *method, const str *url,
+                   struct curl_slist *extra_hdrs);
+
+/*
+ * Prepare @ctx for a request: decide whether the cache applies, capture the
+ * per-call rest_cache_ctl() override, and build the key.  Returns 1 if the
+ * cache is engaged (ctx->enabled set), 0 if this call bypasses it entirely.
+ * The global override is cleared here, so it never leaks to a later request -
+ * which matters for async, whose store runs long after the next call started.
+ */
+int  rcc_prepare(struct rcc_ctx *ctx, const char *method, const str *url,
+                 struct curl_slist *extra_hdrs);
+
+/* returns 1 and fills body/ctype/code on a hit, 0 on a miss */
+int  rcc_lookup(struct rcc_ctx *ctx, str *body, str *ctype, int *code);
+
+/* decide from policy + origin headers whether to store, and for how long */
+void rcc_decide(struct rcc_ctx *ctx, const char *method, int code);
+
+/* store the response if rcc_decide() allowed it */
+void rcc_store(struct rcc_ctx *ctx, const str *body, const str *ctype, int code);
+
+/* statistics - the aggregate view of what source_pv reports per call */
+extern stat_var *rcc_st_hits;
+extern stat_var *rcc_st_misses;
+extern stat_var *rcc_st_stores;
+extern stat_var *rcc_st_skipped;
+unsigned long rcc_stat_hit_rate(void *unused);
+
+/* publish ctx->src into a script variable (no-op when source_pv is NULL) */
+struct sip_msg;
+void rcc_set_source(struct sip_msg *msg, void *source_pv, struct rcc_ctx *ctx);
+
+/* one header line, as seen by header_func() */
+void rcc_parse_header(struct rcc_resp_hdrs *h, const char *line, int len);
+
+#endif /* _REST_CACHE_H_ */

modules/rest_client/doc/rest_client_admin.xml

@@ -79,10 +79,19 @@ size_t write_func(char *ptr, size_t size, size_t nmemb, void *body)
 size_t header_func(char *ptr, size_t size, size_t nmemb, void *userdata)
 {
 	int len, left;
-	str *st = (str *)userdata;
+	struct rest_hdr_sink *sink = (struct rest_hdr_sink *)userdata;
+	str *st = sink->ctype;
 
 	len = left = size * nmemb;
 
+	if (sink->cache)
+		rcc_parse_header(sink->cache, ptr, len);
+
+	if (!st) {
+		LM_DBG("Received: %.*s\n", len, ptr);
+		return len;
+	}
+
 	if (len > CONTENT_TYPE_HDR_LEN && *ptr == 'C' &&
 	    strncasecmp(ptr, HTTP_HDR_CONTENT_TYPE, CONTENT_TYPE_HDR_LEN) == 0) {
 

modules/rest_client/rest_cache.c

@@ -28,6 +28,7 @@
 #include "rest_client.h"
 
 #include "../../str.h"
+#include "rest_cache.h"
 #include "../../mem/mem.h"
 #include "../../error.h"
 #include "../../dprint.h"
@@ -40,6 +41,19 @@
 #define MAX_HEADER_FIELD_LEN	 1024 /* arbitrary */
 
 size_t write_func(char *ptr, size_t size, size_t nmemb, void *userdata);
+/*
+ * What header_func() writes into.  Both members are optional: a script that asks
+ * for no content-type still needs the caching headers parsed, which is why the
+ * callback is installed unconditionally rather than only when a ctype output
+ * variable was supplied - otherwise rest_get(url,$b) and rest_get(url,$b,$ct)
+ * would cache differently, and the async path (which only installed it when a
+ * ctype was present) would never see a Cache-Control header at all.
+ */
+struct rest_hdr_sink {
+	str *ctype;                    /* Content-Type, if the caller wants it */
+	struct rcc_resp_hdrs *cache;   /* caching directives, if caching is on */
+};
+
 size_t header_func(char *ptr, size_t size, size_t nmemb, void *userdata);
 
 #endif /* _REST_CB_H_ */