feat(go-server): add function allowlist (#1150)

Author: derekperkins

README.md

@@ -53,7 +53,7 @@ This repository contains a set of related packages, spanning the core Mosaic arc
 
 * [`duckdb-server`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server): A Python-based server that runs a local DuckDB instance and support queries over Web Sockets or HTTP, returning data in either [Apache Arrow](https://arrow.apache.org/) or JSON format.
 * [`duckdb-server-rust`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-rust): A Rust-based server similar to `duckdb-server` (Python) and `mosaic-duckdb` (Node.js) with additional support for HTTP/2. We are still evaluating what server component works best. DuckDB support for Rust is often delayed compared to Python.
-* [`duckdb-server-go`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-go): A Go-based server similar to `duckdb-server` (Python) and `mosaic-duckdb` (Node.js) with additional support for HTTP/2. It has experimental support for multi-tenant access control, function blocklisting, and other features to harden the server for production use.
+* [`duckdb-server-go`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-go): A Go-based server similar to `duckdb-server` (Python) and `mosaic-duckdb` (Node.js) with additional support for HTTP/2. It has experimental support for multi-tenant access control, function allowlisting and blocklisting, and other features to harden the server for production use.
 * [`mosaic-duckdb`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb): A Promise-based Node.js API to DuckDB, along with a data server that supports transfer of [Apache Arrow](https://arrow.apache.org/) and JSON data over either Web Sockets or HTTP. Due to quality and maintenance issues involving the Node.js DuckDB client and Arrow extension, we recommend using the Python-based `duckdb-server` package instead. However, we retain this package for both backwards compatibility and internal testing use.
 
 ### Examples (`packages/examples`)

packages/server/duckdb-server-go/README.md

@@ -33,7 +33,8 @@ You can customize the server behavior with the following command-line flags:
 -   `--key <path>`: Path to a TLS private key file to enable HTTPS.
 -   `--schema-match-headers`: Comma-separated list of headers to match against schema names for multi-tenant access control (e.g., `X-Tenant-Id,verified-user-id`).
 -   `--load-extensions`: Comma-separated list of extensions to install and load at startup. Use a pipe after the extension name to specify a DuckDB repository alias. Unspecified repositories use DuckDB's default (e.g. `mysql_scanner,netquack|community,aws|core_nightly`).
--   `--function-blocklist`: Comma-separated list of functions to block, useful for blocking functions that may pose security or performance risks. (e.g., 'bigquery_query,read_parquet')`
+-   `--function-blocklist`: Comma-separated list of exact function names to block, useful for blocking functions that may pose security or performance risks (e.g. `bigquery_query,read_parquet`).
+-   `--function-allowlist`: Comma-separated list of exact function names to add to the reviewed defaults. Names are matched case-insensitively, repeated flags accumulate names, and an explicitly empty value enables only the defaults.
 
 By default, the server will look for `localhost.pem` and `localhost-key.pem` in the current directory to enable HTTPS if the `--cert` and `--key` flags are not provided.
 
@@ -79,6 +80,80 @@ errors are logged and returned as sanitized 500 responses. Authorization can all
 and exact SQL, but cannot rewrite SQL or sandbox the shared process, filesystem, network, extensions, catalogs, or
 credentials.
 
+### Function Policies
+
+Use an allowlist when the server should accept only reviewed functions and operators. An explicitly empty value enables
+the defaults without adding application-specific names:
+
+```sh
+duckdb-server-go --function-allowlist=
+```
+
+Without `--function-allowlist`, the server remains unrestricted. The binary intentionally exposes only policy
+activation and exact additions; use a custom binary embedding `pkg/query` for exclusions, exact-only policies, or
+extension groups.
+
+Programs embedding `pkg/query` can apply the same policy and add application functions with:
+
+```go
+query.WithFunctionAllowlist(query.FunctionAllowlistOptions{
+	Include: append(functionset.Spatial.Elevated(), "my_function"),
+})
+```
+
+By default, configured policies use `functionset.DefaultFunctions()`, which contains reviewed built-ins and every
+[core extension](https://duckdb.org/docs/current/core_extensions/overview)'s `Compute()` group. `Elevated()` requires
+explicit admission, and `All()` returns both groups. These Go helpers return fresh slices; the CLI accepts exact names only.
+
+The table records unique names reviewed against DuckDB 1.5.5. A name is elevated if any overload has elevated behavior.
+An empty row means the extension has no reviewed function-call names, not that it has no other capabilities.
+
+| Extension | Compute | Elevated | Classification and status |
+| --- | ---: | ---: | --- |
+| `Autocomplete` | 1 | 3 | Parser check; completion and parser controls are elevated. |
+| `Avro` | 0 | 1 | Reader only. |
+| `AWS` | 0 | 1 | Credential and provider operation. |
+| `Azure` | 0 | 0 | Filesystem integration with no reviewed function-call names. |
+| `Delta` | 2 | 9 | Local parser/test helpers; scans, metadata I/O, and writes are elevated. |
+| `DuckLake` | 1 | 21 | Local hash helper; catalog, scan, metadata, and mutation operations are elevated. |
+| `Encodings` | 0 | 0 | CSV codec integration with no reviewed function-call names. |
+| `Excel` | 2 | 1 | Value conversion; the sheet reader is elevated. |
+| `FTS` | 1 | 2 | Text stemming; index creation and mutation are elevated. |
+| `HTTPFS` | 0 | 0 | Filesystem integration with no reviewed function-call names. |
+| `Iceberg` | 2 | 14 | Value helpers; scans, catalogs, metadata I/O, and writes are elevated. |
+| `ICU` | 179 | 7 | Deterministic collation and calendar computation; current-time names are elevated. |
+| `Inet` | 11 | 0 | IP value operations only. |
+| `JSON` | 33 | 9 | Value parsing and serialization; readers, SQL execution, and plan inspection are elevated. |
+| `Lance` | 0 | 12 | Source-pinned scans and metadata operations. |
+| `MotherDuck` | 0 | 198 | Best-effort observed proprietary runtime snapshot; all names are elevated. |
+| `MySQL` | 0 | 5 | Connector and scanner operations. |
+| `ODBC` | 0 | 11 | Connector and scanner operations. |
+| `Parquet` | 2 | 9 | `VARIANT` conversion; file, metadata, bloom, and key operations are elevated. |
+| `Postgres` | 2 | 8 | Value helpers; connector and scanner operations are elevated. |
+| `Quack` | 3 | 9 | Protocol value helpers; remote and session operations are elevated. |
+| `Spatial` | 151 | 13 | Geometry computation; readers, index/catalog access, random generation, and resource-capable transforms are elevated. |
+| `SQLite` | 0 | 3 | Connector and scanner operations. |
+| `TPCDS` | 2 | 2 | Query and answer text; data generators are elevated. |
+| `TPCH` | 2 | 2 | Query and answer text; data generators are elevated. |
+| `UI` | 0 | 5 | HTTP server lifecycle, URL, and status operations. |
+| `UnityCatalog` | 0 | 4 | Attached-catalog and checkpoint operations; the generated registry is incomplete. |
+| `Vortex` | 0 | 2 | Readers verified against the pinned nested source revision. |
+| `VSS` | 0 | 5 | Index access and management operations. |
+
+These groups authorize names only; extension loading and file or network access are separate concerns. Validation is
+syntactic and name-only: it does not bind function identity, inspect arguments, expand macros or views, recursively inspect
+SQL strings, or cover replacement scans and attached-table binding. Keep catalogs and the search path trusted, and enforce
+resource access outside this policy. Pre-provisioned views and attached tables can deliberately expose curated datasets
+while reader functions remain excluded; catalog integrity and process resource controls then carry the boundary.
+
+In Go, `Exclude` wins over `Include`, and `DisableDefaults` creates an exact-only policy. Omitting
+`WithFunctionAllowlist` is unrestricted; configuring an exact-empty policy denies all function calls. A function
+allowlist cannot be combined with a non-empty blocklist, and any configured function policy rejects `exec` requests.
+
+Spatial compute defaults cover Mosaic rendering over existing geometry data, but the `ST_Read` loader remains elevated.
+Current-time functions are omitted from defaults because persistent cache entries do not expire by default; keyword forms
+such as `CURRENT_DATE` are not function nodes and remain outside this policy.
+
 ### Multi-Tenant Access Control
 
 `schema-match-headers` isn't part of the mosaic server API, but is provided here as an example of how to have
@@ -108,19 +183,19 @@ multiple users / customers share the same DuckDB server instance while restricti
 
 _Note:_ Schema matching authorizes schema references in submitted SQL; it does not isolate the shared DuckDB process,
 filesystem, network, extensions, or credentials. It assumes a single catalog; attached catalogs are outside this policy
-boundary, and explicitly catalog-qualified table, `SHOW`, and function references are rejected. The function blocklist
-applies only to explicit function calls. Schema matching does not restrict catalog metadata returned by functions such as
-`duckdb_tables()` and `pragma_table_info()`. If metadata is sensitive, add the exact metadata-function names exposed by the
-deployment to `--function-blocklist`; wildcard patterns such as `duckdb_*` are not supported, and the list must be reviewed
+boundary, and explicitly catalog-qualified table, `SHOW`, and function references are rejected. Function allowlists and
+blocklists apply only to explicit function calls. Schema matching does not restrict catalog metadata returned by functions
+such as `duckdb_tables()` and `pragma_table_info()`. If metadata is sensitive, allow or block the exact metadata-function
+names exposed by the deployment; wildcard patterns such as `duckdb_*` are not supported, and the policy must be reviewed
 when DuckDB or its extensions change. To restrict file-reading functions, also enable schema matching so DuckDB replacement
 scans such as `FROM 'data.parquet'` are rejected as unqualified table references. These controls are not a sandbox: run the
 server with access only to external resources that are safe for every tenant.
 
-If either `--schema-match-headers` or `--function-blocklist` is configured, `json` and `arrow` requests are limited to
-statements DuckDB can serialize for validation; unsupported forms such as `PRAGMA` and `SET` are rejected, with HTTP
-requests receiving a 400 response. All `exec` requests are also rejected until full-statement authorization is supported.
-This includes every `Coordinator.exec(...)` call, such as data loading, preloading, and DDL/DML. Mosaic pre-aggregation
-also uses `exec` to create schemas and tables, so set `preagg: { enabled: false }` in this mode.
+If `--schema-match-headers`, `--function-blocklist`, or `--function-allowlist` is configured, `json` and `arrow` requests
+are limited to statements DuckDB can serialize for validation; unsupported forms such as `PRAGMA` and `SET` are rejected,
+with HTTP requests receiving a 400 response. All `exec` requests are also rejected until full-statement authorization is
+supported. This includes every `Coordinator.exec(...)` call, such as data loading, preloading, and DDL/DML. Mosaic
+pre-aggregation also uses `exec` to create schemas and tables, so set `preagg: { enabled: false }` in this mode.
 
 ## API
 

packages/server/duckdb-server-go/flags.go

@@ -0,0 +1,20 @@
+package main
+
+import "strings"
+
+type optionalCommaListFlag struct {
+	values []string
+	set    bool
+}
+
+func (f *optionalCommaListFlag) Set(value string) error {
+	f.set = true
+	if value != "" {
+		f.values = append(f.values, strings.Split(value, ",")...)
+	}
+	return nil
+}
+
+func (f *optionalCommaListFlag) String() string {
+	return strings.Join(f.values, ",")
+}

packages/server/duckdb-server-go/flags_test.go

@@ -0,0 +1,30 @@
+package main
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestOptionalCommaListFlag(t *testing.T) {
+	var flag optionalCommaListFlag
+
+	assert.False(t, flag.set)
+	assert.Empty(t, flag.values)
+
+	require.NoError(t, flag.Set("md5,range"))
+	require.NoError(t, flag.Set("+"))
+	assert.True(t, flag.set)
+	assert.Equal(t, []string{"md5", "range", "+"}, flag.values)
+	assert.Equal(t, "md5,range,+", flag.String())
+}
+
+func TestOptionalCommaListFlagPreservesExplicitEmpty(t *testing.T) {
+	var flag optionalCommaListFlag
+
+	require.NoError(t, flag.Set(""))
+	assert.True(t, flag.set)
+	assert.Empty(t, flag.values)
+	assert.Empty(t, flag.String())
+}

packages/server/duckdb-server-go/main.go

@@ -19,6 +19,10 @@ import (
 )
 
 func main() {
+	os.Exit(run())
+}
+
+func run() int {
 	dbPath := flag.String("database", ":memory:", "Path of database file (e.g., \"database.db\". \":memory:\" for in-memory database)")
 	address := flag.String("address", "localhost", "HTTP Address")
 	port := flag.String("port", "3000", "HTTP Port")
@@ -31,6 +35,8 @@ func main() {
 	schemaMatchHeadersStr := flag.String("schema-match-headers", "", "Comma-separated list of headers to match against schema names for multi-tenant access control (e.g., \"X-Tenant-Id,verified-user-id\")")
 	extensionsStr := flag.String("load-extensions", "", "Comma-separated list of extensions to install and load at startup. Use a pipe after the extension name to specify a DuckDB repository alias. Unspecified repositories use DuckDB's default (e.g. mysql_scanner,netquack|community,aws|core_nightly).")
 	functionBlocklistStr := flag.String("function-blocklist", "", "Comma-separated list of functions to block, useful for blocking functions that may pose security or performance risks. (e.g., 'bigquery_query,read_parquet')")
+	var functionAllowlist optionalCommaListFlag
+	flag.Var(&functionAllowlist, "function-allowlist", "Comma-separated exact names to add to the reviewed default allowlist. An empty value enables only the defaults; names are matched case-insensitively.")
 	flag.Parse()
 
 	var schemaMatchHeaders []string
@@ -52,7 +58,7 @@ func main() {
 
 	if err := extensions.Validate(*extensionsStr); err != nil {
 		logger.Error("main: invalid load-extensions", "error", err, "load-extensions", *extensionsStr)
-		return
+		return 1
 	}
 
 	// If no certificate files are specified, check for default localhost certificates
@@ -72,7 +78,7 @@ func main() {
 	})
 	if err != nil {
 		logger.Error("main: error creating duckdb connector", "error", err)
-		return
+		return 1
 	}
 	defer func() {
 		err = connector.Close()
@@ -84,20 +90,27 @@ func main() {
 	ttl, err := time.ParseDuration(*ttlStr)
 	if err != nil {
 		logger.Error("main: invalid cache-ttl", "error", err)
-		return
+		return 1
 	}
 
-	db, err := query.New(ctx, connector,
+	queryOptions := []query.OptionFunc{
 		query.WithMaxConnections(*poolSize),
 		query.WithMaxCacheEntries(*maxCacheEntries),
 		query.WithMaxCacheBytes(*maxCacheBytes),
 		query.WithTTL(ttl),
 		query.WithLogger(logger),
 		query.WithFunctionBlocklist(functionBlocklist),
-	)
+	}
+	if functionAllowlist.set {
+		queryOptions = append(queryOptions, query.WithFunctionAllowlist(query.FunctionAllowlistOptions{
+			Include: functionAllowlist.values,
+		}))
+	}
+
+	db, err := query.New(ctx, connector, queryOptions...)
 	if err != nil {
 		logger.Error("main: error creating query DB", "error", err)
-		return
+		return 1
 	}
 	defer db.Close()
 
@@ -113,7 +126,7 @@ func main() {
 	)
 	if err != nil {
 		logger.Error("main: error creating server", "error", err)
-		return
+		return 1
 	}
 	logger.Warn("DuckDB Server permits all HTTP and WebSocket origins for compatibility; enforce an outer origin or CSRF policy before exposing it to untrusted browsers")
 
@@ -129,13 +142,16 @@ func main() {
 		"ttl":                  ttl,
 		"max_cache_bytes":      *maxCacheBytes,
 		"load_extensions":      *extensionsStr,
+		"function_blocklist":   *functionBlocklistStr,
+		"function_allowlist":   functionAllowlist.String(),
+		"allowlist_configured": functionAllowlist.set,
 	}
 	logger.Info("DuckDB Server configuration", "config", config)
 
 	extensions, err := db.GetExtensions(ctx)
 	if err != nil {
 		logger.Error("main: error getting extensions", "error", err)
-		return
+		return 1
 	}
 
 	logger.Info("DuckDB Server Extensions", "extensions", extensions)
@@ -163,6 +179,7 @@ func main() {
 	}
 	if err != nil {
 		logger.Error("main: error running HTTP server", "error", err)
-		return
+		return 1
 	}
+	return 0
 }

packages/server/duckdb-server-go/pkg/functionset/AGENTS.md

@@ -0,0 +1,13 @@
+# Function allowlist inventories
+
+When changing the reviewed inventories in this package:
+
+- Treat the DuckDB release bundled by the `duckdb-go` version in [go.mod](../../go.mod) as the source of truth. Inspect that tag's function registrations, parser rewrites, and macro bodies; use `duckdb_functions()` as a classification cross-check, not as a generated allowlist.
+- Keep names lowercase, sorted, deduplicated, and grouped by the serialized `function_name`. Audit side effects, volatility, resource I/O, dynamic SQL or dispatch, and type or macro collisions. Leave uncertain names out.
+- Update the catalog exemptions, reviewed macros, and collision allowlists in `functionset_test.go` only when the matching DuckDB source justifies the exception. Verify parser-generated operators and syntax helpers with `json_serialize_sql` and executable SQL because some have no catalog row.
+- Keep `CoreExtensions` aligned with the bundled DuckDB release's core-extension roster, with an explicit inventory entry even when both groups are empty. Review external extensions against the exact revision pinned by DuckDB's descriptor; generated `extension_entries.hpp` data is not exhaustive.
+- For initial inventories and DuckDB upgrades, verify installed extension revisions and diff `duckdb_functions()` before and after loading each available extension in a fresh database, including new overloads of existing names. Treat statically linked baseline entries, dependency-loaded names classified under another extension, and lazy catalog-scoped or proprietary registrations as source-audit exceptions, not catalog-equality failures.
+- Keep each extension's source pin and `Compute`/`Elevated` arrays together in `<extension>.go`, using the DuckDB extension ID for the filename (for example, `unity_catalog.go`).
+- Put reviewed local computation or embedded static data with no resource or state effects in `Compute`; it is enabled by `DefaultFunctions`. Put resource, mutation, dynamic dispatch, catalog/session inspection, current-time, and source-limited runtime-verified names in `Elevated`. Treat catalog volatility as elevated unless exact pinned source proves the function is pure by argument. Classify a shared name by its most capable overload because validation does not bind signatures.
+- Update the pinned counts and classification table in [README.md](../../README.md) with the inventories. Document source-only or runtime-only limitations, especially MotherDuck's non-exhaustive proprietary runtime snapshot, and keep loading, autoloading, replacement scans, `ATTACH`, settings, and other non-function mechanisms outside the function-group claim.
+- As a secondary compatibility check, compare the functions emitted by [Mosaic SQL](../../../../mosaic/sql/src/index.ts). Do not copy `aggregateNames` or exports wholesale: they can be stale or include unsafe macros and non-function syntax.

packages/server/duckdb-server-go/pkg/functionset/autocomplete.go

@@ -0,0 +1,12 @@
+package functionset
+
+// These inventories match DuckDB 1.5.5 at d8cdaa33fda8df955cc76ef58a280f68f4cd43fa.
+var autocompleteComputeFunctions = [...]string{
+	"check_peg_parser",
+}
+
+var autocompleteElevatedFunctions = [...]string{
+	"disable_peg_parser",
+	"enable_peg_parser",
+	"sql_auto_complete",
+}

packages/server/duckdb-server-go/pkg/functionset/avro.go

@@ -0,0 +1,8 @@
+package functionset
+
+// Source: https://github.com/duckdb/duckdb-avro/tree/f9d590297485f0318f480372c70bdd852826e258
+var avroComputeFunctions = [...]string{}
+
+var avroElevatedFunctions = [...]string{
+	"read_avro",
+}