Jack Eadie

Spice v2.4.0 is a feature release that expands data ingestion, caching, and AI decisions, and makes Spice Cayenne the default accelerator. It also upgrades the query engine to Apache DataFusion v55 and Apache Arrow v59.3. Datasets and views that do not set acceleration.engine now use Cayenne instead of Arrow, except on native Windows. Other accelerators are deprecated and will be removed in v2.5. In tests, Cayenne served uncached primary-key lookups about twice as fast, and query latency stayed steady during full refreshes.

SQL queries and the new OpenAI-compatible /v1/decisions API can now ask TypeSafe or chat models for decisions. With refresh_mode: changes, accelerated S3 datasets read new objects from S3 when SQS delivers S3 event notifications. This release also adds SQL results-cache warmup, Enterprise acceleration snapshots, and new data connectors and formats.

Read Breaking Changes before you upgrade. SQL results-cache entries now expire only by TTL, not when a dataset refreshes. Cayenne ignores on_conflict and keeps one row for each primary key. Some datasets that loaded on Arrow fail to load on Cayenne unless they set engine: arrow.

Highlights in v2.4.0 include:

• Spice Cayenne by Default: Datasets without acceleration.engine now use Cayenne instead of Arrow, except on native Windows. Cayenne adds automatic primary-key handling, persistent secondary indexes, and adaptive tuning (preview).
• DataFusion v55: the query engine upgrades to DataFusion v55 and Arrow v59.3. In DataFusion benchmarks, the TopK query suite ran about 43% faster. Ballista v55, Vortex v0.86, and Iceberg v0.11 also ship in this release.
• Faster Cayenne Lookups and Refreshes: in tests, uncached Cayenne primary-key lookups ran about twice as fast. P99 query latency during full refreshes fell from 130–690 ms to 44–50 ms.
• AI Decisions in SQL and HTTP: five SQL decision functions and an OpenAI-compatible POST /v1/decisions API, with TypeSafe models or any chat model.
• S3 Event-Driven Refresh: with refresh_mode: changes, accelerated S3 datasets read new objects when SQS delivers S3 event notifications. A periodic listing loads objects whose notification did not arrive.
• Acceleration Snapshots (Enterprise): serve a dataset from published snapshots without its source connector. Snapshots use the shared state location and expire automatically.
• SQL Results-Cache Warmup: record query shapes, then replay them after a restart so that the first queries use a warm cache.
• New Data Connectors and Formats: query Hugging Face datasets, ORC files, and Iceberg tables without a catalog.

What's New in v2.4.0​

Spice Cayenne by Default​

Spice Cayenne now accelerates datasets and views that omit acceleration.engine. Before this release, they used Arrow. Native Windows keeps Arrow. On native Windows, run Spice in Docker or WSL to use Cayenne. Storage still defaults to mode: memory. Set mode: file for persistent acceleration. To move an existing acceleration, read the Cayenne migration guide.

Automatic ON CONFLICT handling for Primary Keys​

A Cayenne dataset with a primary_key now keeps one row for each key in every refresh mode, without an on_conflict setting. This table shows which version of a repeated key Cayenne keeps:

With a time_column, a late row with an older time does not replace a newer stored row. Sources such as Iceberg, where writers can commit out of time order, keep the newest version of each key.

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    time_column: updated_at
    params:
      file_format: parquet
    acceleration:
      enabled: true
      mode: file
      primary_key: id

Cayenne no longer uses on_conflict to select its conflict policy. For read-write datasets whose writes must stay in the acceleration, set access: read_write and acceleration.write_mode: acceleration. The source need not support writes. This setting cannot accompany a change-stream refresh (refresh_mode: changes, or a CDC source whose default refresh mode is changes).

An UPDATE that creates a primary-key collision fails before it changes rows. See duplicate primary keys and the migration guidance.

Runtime-Wide Adaptive Tuning (Preview)​

Cayenne can now change its own settings to meet runtime-wide targets for freshness, replication lag, query latency, and throughput. Set runtime.adaptive_tuning to enable it for Cayenne datasets and catalogs. Adaptive tuning is in preview.

runtime:
  adaptive_tuning: enabled
  params:
    target_freshness: 30s

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      enabled: true
      mode: file
      refresh_mode: changes
      primary_key: id

Adaptive tuning accepts four target objectives and one pacing setting in runtime.params. Cayenne changes its settings in small steps to move each measured value toward its target:

• target_replication_lag: Desired maximum age of the newest applied source commit. Applies only to change-stream (refresh_mode: changes) datasets.
• target_freshness: Desired maximum delay from a source commit until its data becomes queryable. Applies only to change-stream datasets.
• target_query_latency: Desired P99 latency of the queries that read each Cayenne dataset.
• target_qph: Desired throughput, in queries per hour (QPH), of queries that read Cayenne datasets.
• target_convergence_window: Time budget for tuning to approach its targets. Default: 60s.

Adaptive tuning defaults to disabled. Targets apply only when adaptive tuning is enabled. A refresh_mode: full dataset does not use adaptive tuning. Each refresh replaces the whole table, so Cayenne turns off background compaction (cayenne_compaction_background_interval_ms: 0). The tuner runs on the compaction task, so it does not run for that dataset. The runtime logs a warning for each such dataset. See Cayenne self-tuning and Cayenne Tuning Settings.

Indexes, Initial Loads, and Compaction​

• Persistent secondary indexes: File-mode Cayenne stores index runs with its acceleration. Writes maintain the indexes, and acceleration snapshots retain them. Dynamic join filters can probe secondary indexes.
• Keyed initial loads: Cayenne can build the primary-key filter during the first load within its memory budget. Otherwise, it retains the existing warmup path. Bulk loads avoid per-row routing and a subsequent whole-table fold.
• Maintained aggregates: Views share a compact index of per-key contributions. Rebuilds capture concurrent writes and apply them after the scan. A bounded query pool assigns 10% to aggregates without the former 512 MiB cap. An unbounded pool retains that cap.
• Append-only compaction: For unkeyed datasets without position deletes, Cayenne encodes merged files outside the write lock. Publication still takes the lock. Datasets with position deletes retain the full-rewrite path.
• Configuration updates: A restart applies changed write settings to an existing acceleration. Identity changes can rebuild eligible full-refresh accelerations from their source. See the Cayenne documentation.

Apache DataFusion v55 and Query Engine Upgrades​

Spice now runs on Apache DataFusion v55 and Apache Arrow v59.3. This release also upgrades Apache Ballista to v55, Vortex to v0.86.1, and Apache Iceberg to v0.11.0. The numbers in this section come from each project's own benchmarks, not from Spice tests.

Apache DataFusion v55​

The DataFusion v55 release adds these changes:

• TopK pruning: As the threshold of an ORDER BY ... LIMIT query tightens, Parquet scans check the unread row groups again. They skip groups that cannot change the result. Compound ORDER BY keys are also supported. The topk_tpch suite ran about 43% faster in total.
• Join planning: The optimizer converts eligible inner joins to semi joins. It removes redundant sides of LEFT and RIGHT joins and orders filter predicates by estimated cost. ClickBench Q21 ran 10–13% faster.
• Planning speed: The optimizer collapses chained projections and rewrites subqueries in place. One planning case fell from 623 ms to 155 ms, and TPC-DS optimization fell from 220 ms to 170 ms.
• Functions: IN lists with few primitive values evaluate up to 14.7× faster in microbenchmarks. approx_distinct over 50,000 groups fell from 1,723 ms to 17 ms. More string functions keep dictionary encoding.
• Aggregation: Multi-column GROUP BY keeps column-oriented storage for all key types, such as fixed-size binary UUIDs. Grouping 200 million UUIDs ran about 5% faster and used less memory.
• Parquet scans: Scans skip nested fields that the table schema does not declare. They also skip the page index when a file has none. ClickBench Q1 on a single file ran 38% faster.
• Spills: A sort under memory pressure spills its largest stream again in smaller batches instead of failing. Merges also limit their number of input streams.
• SQL: EXPLAIN accepts PostgreSQL-style options and FORMAT pgjson. New array functions add, subtract, scale, sum, and average array elements.

Spice carries these changes through its query plans and keeps statistics across its plan wrappers. Cayenne does not split Vortex scans below 10 MiB, to avoid repeated footer reads.

Apache Ballista v55​

Spice.ai Enterprise feature. See Distributed Query.

Spice distributed query runs on a Spice fork of Apache Ballista v55. The Ballista v55 release adds these upstream changes:

• Adaptive query execution by default: The scheduler replans between stages with row counts from completed stages. All 99 TPC-DS queries complete and match single-process DataFusion in the Ballista test suite.
• Fewer tasks: One task can cover several input partitions, up to the available virtual cores of its executor. This change reduces planning, serialization, and dispatch work.
• Shuffle: Remote shuffle reads receive blocks without a copy. Sort shuffle writes one file for each task.
• Protocol version: Clients receive a clear error when their major version differs from the version of the scheduler.
• Benchmark: On TPC-H at scale factor 1,000, Ballista ran the 22 queries in 502.4 seconds. Apache Spark 4.1.3 took 898.2 seconds on the same cluster.

Schedulers and executors must run the same version, so upgrade all cluster components together.

Vortex v0.79.0 to v0.86.1​

Spice v2.3.3 used the Vortex v0.79.0 fork. This upgrade covers every upstream release from v0.79.0 through v0.86.1:

• Types and arithmetic: Vortex adds native Map arrays, union arrays, and decimal addition, subtraction, multiplication, and division.
• Row selection: Piecewise-sequence indices represent contiguous selections without an index entry for every row.
• Scans and expressions: Layout scans get a physical plan and expression optimization. Filters can pass through scalar functions with more than one argument. Filters on wide lists read only the selected range of child elements.
• Compression: Binary arrays support FSST compression with variable-length offsets. OnPair becomes a stable encoding with a dictionary in storage.
• File metadata: Files can store custom metadata. Readers cache decoded type descriptions.
• Correctness and validation: Vortex corrects NULL handling for dictionary predicates, BETWEEN bounds, and empty arrays. Readers validate footer offsets, compression metadata, and array indices.

See the Vortex release notes for v0.80.0 through v0.86.0 and v0.86.1.

Apache Iceberg v0.11.0​

Spice updates its Iceberg reader and catalog integrations from v0.10.1 to the v0.11.0 fork for DataFusion v55 and Arrow v59. The upstream v0.11 changes extend reads and catalog compatibility:

• Iceberg v3 reads: The reader applies deletion vectors from Puffin files. It carries row identifiers and sequence numbers through scans.
• Metadata-only scans: Projections of metadata only do not read data columns. Positional-delete processing buffers runs instead of one key for each row.
• REST catalogs: Clients negotiate the endpoints that the server advertises. They use OAuth2 authentication for each session.

The upgrade also aligns Iceberg storage with OpenDAL v0.58.

Performance​

Spice Cayenne serves lookups, loads, and refreshes faster in this release. Each number below comes from the test in the linked pull request. The tests ran on different machines, so do not compare numbers between them.

Spice Cayenne​

• Steady query latency during refreshes: Full and append refreshes now write on dedicated refresh threads by default. Before, a large refresh write ran on the threads that serve queries. In a test, a client sent 500 point-lookup joins per second to a Cayenne table of 2.2 million rows and about 8 GB. P99 latency during each full-refresh commit fell from 130–690 ms to the steady 44–50 ms. With runtime.params.dedicated_thread_pool: disabled, refresh writes run as before. See #15262.
• Primary-key point lookups: Cayenne serves uncached primary-key point lookups about twice as fast. On a file-mode table of 10 million rows, P50 latency improved 2.11–2.41× at 1 to 128 clients. See #14433.
• Lookups after the first load: Uncached primary-key lookups right after the first full refresh are faster. On a table of 10 million rows, P50 latency at one client fell from 6.0 ms to 1.1 ms. At 16 clients, P99 latency fell from 16.3 ms to 3.9 ms. See #14314.
• First refresh after a load: Cayenne builds the primary-key filter during the first load, so the first append refresh does not rebuild it. On a keyed table of 40.2 million rows, the first append refresh fell from a median of 4.2 s to 0.11 s. See #15099.
• Keyed bulk loads: A first load into an empty keyed table no longer routes each row separately. On an Iceberg table of 40.2 million rows, the slowest first load fell from 21.0 s to 13.5 s. Iceberg scans also decode at the session batch size instead of 1,024 rows. See #15076 and #15072.
• Append visibility on clustered tables: Cayenne orders each scan partition of a whole-table rewrite separately, then merges them. On an append table of 20 million rows with cayenne_cluster_by, P99 time from append to query visibility fell from 17.2 s to 3.0 s. See #14488.
• Append-only ingestion: Small-file compaction no longer stalls writers on tables without a primary key or position deletes. In an OTLP log-ingestion test, export P99 latency fell from about 1.2 s to 0.12 s. Throughput rose from about 210,000 to 775,000 records per second. See #15227.
• Metastore writes: Cayenne batches manifest rewrites in its SQLite metastore and runs every write on one writer connection. At 128 clients, P99 latency of a 1,000-file manifest rewrite fell from 5.39 s to 374 ms. See #14369.
• Filtered key deletes: A filtered delete on a keyed table now reads only the key and filter columns. In a test, P99 latency of a key-only delete fell from 27.7 ms to 8.9 ms. See #14374.

Caching​

• Caching mode under sustained misses: Cayenne datasets with refresh_mode: caching now batch cache replacements. In a test with 500 lookups per second over 2.5 million random keys, request P99 latency fell from 363 ms to 9 ms. Dropped cache fills fell from 122,508 to 0. See #15253.
• Caching mode without a primary key: Lookups stay fast when the cache replaces entries continuously on a dataset without a primary key. In a two-core test at 300 lookups per second, P99 latency fell from 1,377 ms to 9.3 ms. See #15362.
• SQL results-cache expiry: Expiry work now grows with the number of expired entries, not with the size of the cache. With 3 million entries, insert P99.9 latency fell from 10.8–15.7 ms to 70–84 µs. See #14919.

Object-Store Listings​

• Metadata-only queries: GROUP BY, DISTINCT, MIN, and MAX over partition or metadata columns no longer scan whole files. Spice answers them from the file listing, or from the first record of each file. count, sum, and avg still scan the files. See #14116.
• Listing pruning: Filters on _last_modified and other metadata columns now prune the file listing, so Spice does not open the excluded objects. Timestamp bounds joined by AND prune the listing. Filters with OR or NOT read the full listing. See #14265 and #14303.

AI Decisions in SQL and HTTP​

SQL queries and HTTP clients can now ask a model for yes-or-no answers, probabilities, labels, and scores. This release adds five SQL decision functions and an OpenAI-compatible POST /v1/decisions API. TypeSafe decision models return calibrated probabilities. Any chat model can also answer decisions, but its probabilities are uncalibrated estimates.

Add a TypeSafe model to the Spicepod:

models:
  - from: typesafe:jev
    name: support_model
    params:
      typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Then call a decision function in SQL. Each function selects its model with the model argument:

SELECT
    id,
    ai_classify(body, ['billing', 'technical', 'account'],
                model => 'support_model') AS team,
    ai_probability(body, 'The customer requests a refund',
                   model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI decisions.

S3 Event-Driven Refresh​

With refresh_mode: changes, accelerated S3 datasets read new objects from S3 when SQS delivers S3 event notifications. Set refresh_mode: changes and s3_changes_queue_url together. The queue must serve only this dataset, and from must be a prefix, not a single object or a glob:

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: parquet
      s3_region: us-east-1
      s3_changes_queue_url: ${secrets:events_queue_url}
    acceleration:
      enabled: true
      mode: file
      refresh_mode: changes

A periodic listing of the prefix (s3_changes_backfill_interval, default 1h) loads objects whose notification did not arrive. Spice loads each object key one time. By default, rows from deleted objects stay in the acceleration (s3_on_object_removed: ignore).

Acceleration Snapshots​

Spice.ai Enterprise feature. See Acceleration Snapshots.

A dataset can now serve published acceleration snapshots without its original source connector. Set file_format: snapshot, and set from to the snapshot location:

datasets:
  - from: s3://my-bucket/spice/snapshots/
    name: orders
    params:
      file_format: snapshot
      s3_region: us-east-1

Snapshots now expire after 28 days, except for the current and newest snapshot of each dataset. A reader that cannot load a newer snapshot keeps serving its loaded snapshot and tries again on the next poll. Each failure increments dataset_acceleration_refresh_errors{mode="snapshot", reason="snapshot_download_failed"}. OSS builds refuse snapshot datasets and do not publish snapshots. See Serve a dataset from published snapshots and Shared State and Snapshot Migration.

One Location for Shared Runtime State​

runtime.state sets one object-store location for the state that the runtime keeps across restarts and instances. Each of these features stores its state in its own subdirectory:

• Cluster state
• SQL results-cache warmup
• Shared HTTP rate control (Spice.ai Enterprise)
• Acceleration snapshots (Spice.ai Enterprise)

runtime:
  state:
    location: s3://my-bucket/spice/
    params:
      s3_region: us-east-1

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      mode: file
      snapshots: enabled

Earlier per-feature locations, such as runtime.scheduler.state_location, still work but are deprecated. See Shared State and Snapshot Migration.

SQL Results Cache and HTTP Caching​

SQL Results-Cache Warmup​

After a restart, the first queries can now read from a warm SQL results cache. The cache records query shapes and stores them. When Spice starts again, it replays the stored shapes after the first full or append refresh of each accelerated dataset:

runtime:
  caching:
    sql_results:
      enabled: true
      warmup: on_first_refresh

Spice records the first 10 distinct shapes of cacheable, unauthenticated queries. Queries that differ only in equality-filter values share one shape. During replay, warmup reads up to 1,024 distinct combinations of filter-column values for each shape. It reruns the shape with each combination until the cache is full. Datasets do not report ready until warmup completes. The runtime does not start when a Spicepod sets both warmup and cache_key_type: sql.

Cache Backend​

The SQL results, search, and embedding caches now use the Spice sharded cache. The runtime accepts engine: moka and engine: pingora for compatibility, but these values no longer select a backend.

Caching Refresh Mode​

These changes apply to datasets with refresh_mode: caching, such as HTTP datasets:

• Shared cache-miss fetches: Concurrent cache misses for the same key now share one origin fetch.
• stale-if-error: Cached HTTP responses now follow the RFC 5861 stale-if-error rules.

New Data Connectors and Formats​

Hugging Face Datasets​

The Hugging Face data connector queries and accelerates Parquet, CSV, TSV, JSON, and ORC files from the Hub:

datasets:
  - from: hf://datasets/stanfordnlp/imdb/plain_text/
    name: imdb
    acceleration:
      enabled: true

ORC and Object Metadata​

Listing connectors now support file_format: orc. Enable metadata columns to query the location and modification time of each row's source object:

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: orc
      s3_region: us-east-1
    metadata:
      _location: enabled
      _last_modified: enabled

For example, query objects modified since October 1:

SELECT DISTINCT _location, _last_modified
FROM events
WHERE _last_modified >= TIMESTAMP '2026-10-01 00:00:00';

Apache Iceberg Without a Catalog​

Apache Iceberg datasets can read a table directly from its root directory on local or object storage. Spice finds the current metadata file on every query and every refresh, so it reads new commits without a restart:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1

To read one fixed table version, set iceberg_metadata_path to a metadata file. A pinned dataset is read-only, so it fails to load with access: read_write:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis_v3
    params:
      iceberg_s3_region: us-east-1
      iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Iceberg Append Refresh Follows Table Snapshots​

An Iceberg dataset with refresh_mode: append and a time_column now loads the rows that each new commit adds, whatever their time values. This applies to Cayenne datasets and to datasets without a primary key. Before this release, an append refresh read only rows later than the stored maximum time minus refresh_append_overlap. A writer that committed out of time order could put rows outside that window. The refresh now saves the last Iceberg snapshot that it loaded, and the next refresh reads the data files that later snapshots added. See append refresh.

Operations and Observability​

The runtime can now send structured logs through the existing OTLP exporter:

runtime:
  telemetry:
    otel_exporter:
      endpoint: otel-collector:4317
      signals: [metrics, logs]
  query:
    log_errors: execution

Log records carry severity, typed attributes, and the query trace ID. The runtime batches delivery on a dedicated export runtime. signals defaults to [metrics].

Failure records contain error_code and datasets. They do not contain SQL text. See runtime.query.log_errors and observability.

Other changes:

• Adaptive HTTP rate control: HTTP rate control now reduces admission after upstream failures and stays within the configured request limits. In Spice.ai Enterprise, instances that share runtime.state also share per-second and per-minute limits. See HTTP rate control.
• Open-file limits on Unix: The runtime raises its soft open-file limit to the hard limit at startup. On macOS, kern.maxfilesperproc also bounds the result. If the hard limit remains too low, the Cayenne error names the effective limit and the remedy.
• Shared-state metrics: state_store_request_duration_ms{feature, operation}, state_store_conflicts_total{feature}, and state_store_cleanup_deleted_total{feature} report requests, conditional-write conflicts, and cleanup.
• Prometheus scrapes: Derived summary families emit one HELP and TYPE block per family. Strict parsers no longer reject the scrape for repeated metadata.
• Datadog dashboard: New panels cover snapshot freshness, Cayenne segment caches, and pod health.

Security Improvements​

• Brotli decoder (CVE-2026-111497): The DuckDB that Spice bundles no longer writes out of bounds when it decodes a malformed Brotli-compressed Parquet page. Most Parquet reads in Spice do not go through DuckDB. The fix matters where DuckDB decodes a Parquet file that an untrusted party can supply.
• Flight sessions: Authentication precedes session creation. Sessions remain bound to their creating principal. TLS handshakes have a 10-second deadline, and Flight SQL statement updates count against the Flight write limit.
• HTTP paths: The HTTP connector rejects request_path values that URL parsing would rewrite outside their allowed paths. This covers encoded dot segments and control-character rewrites.

Bug Fixes​

The fixes with the widest effect are below. The changelog lists every fix.

• Wrong rows from Cayenne queries: Cayenne queries no longer return wrong rows from maintained views, Vortex IN lists, or shared dynamic filters. See #14427.
• NOT IN and NULL values: Queries with NOT IN no longer lose their NULL semantics after a join reorder or the Cayenne sort-merge rewrite. See #14429.
• Wrong results from DataFusion: The DataFusion upgrade includes upstream fixes for wrong results from filter pushdown, expression simplification, and planning. See #14430.
• Federated pushdown: Federated queries no longer send some casts and filters to a source that evaluates them differently from Spice. Affected sources include DuckDB, PostgreSQL, MySQL, BigQuery, DynamoDB, Cosmos DB, and MongoDB. See #14484, #14601, and #14419.
• Accelerated HTTP datasets: An HTTP error response no longer replaces the rows of an accelerated HTTP dataset. See #13538.
• Unavailable sources at startup: A dataset whose source is unreachable at startup now retries. An existing acceleration keeps serving queries while its source is unavailable. See #14623 and #14624.
• Stale plans and results after a reload: Queries no longer use cached plans or results after a hot reload replaces a catalog or unloads a dataset. See #13914 and #14365.
• CDC positions at shutdown: Change-data-capture sources now record their positions before shutdown closes the accelerations. See #14702.
• Tool calls: Tool-use streams no longer end on a stale tool_calls finish. /v1/responses runs Spice tools and honors tool_recursion_limit when it streams. See #14548 and #14930.
• Vector search on chunked columns: vector_search returns its top k rows by score on chunked and multi-vector columns. See #15056.

SDK Updates​

Updated SDKs release alongside v2.4.0:

• spice.js v3.3.0: adds query cancellation with an AbortSignal, runtimeStatus(), and refreshDataset(). refreshAcceleration() is deprecated. Fixes intermittent HTTP query failures on Node, API keys on plaintext gRPC, and typed Flight parameters. If you set only one of flightUrl or httpUrl, the client now uses the matching address for the other. An empty URL or a Flight authentication failure now throws.
• spicepy v4.0.1: sends API keys as a Bearer token on Flight and ADBC calls to self-managed runtimes, and binds numpy and pandas scalars as query parameters. refresh_dataset() and runtime_status() now raise SpiceAIError with the runtime's error message instead of requests.HTTPError. Requires certifi 2026.10.19 or later.
• spice-rs v5.0.0: upgrades to Arrow 59 and binds chrono and std::time values as query parameters. Parameterized queries now work with API keys, and queries retry after a connection reset. Applications that also depend on arrow must move to Arrow 59.
• spice-java v0.8.1: accepts any non-empty API key, and sends it on refreshDataset. Fixes the NoClassDefFoundError that v0.8.0 raised on every TLS connection, including Spice.ai Cloud.
• spice-dotnet v0.5.0: adds GetRuntimeStatusAsync() for /v1/status, which all six SDKs now support. Accepts any non-empty API key, and supports grpc:// Flight addresses.
• gospice v9.1.0: binds time.Time, time.Duration, pointers, sql.Null* types, and named types as query parameters. Accepts any non-empty API key, and uses one Flight session for each client. Requires Go 1.26 or later. Decimal128Param and Decimal256Param now read Arrow little-endian bytes, so check those calls before you upgrade.

Agent Skills Updates​

Updated Agent Skills release alongside v2.4.0:

• spiceai/skills v2.4.0:
  • New launch skill: deploys a Spice.ai Cloud project, validates that it serves data and model responses, and sets up monitors and alerts.
  • New sdk skill: covers the six client SDKs.
  • cloud skill (previously spice-cloud-management): now works through the management MCP server and can fork projects.
  • All skills are updated for v2.4.0, including Cayenne as the default accelerator and the deprecation of other accelerators.

Dependency Updates​

Breaking Changes​

Accelerator Selection and Primary-Key Policies​

Spice Cayenne is now the default accelerator on supported platforms. Datasets and views without acceleration.engine switch from Arrow to Cayenne. Native Windows keeps Arrow. If datasets rely on another accelerator's behavior, review the Cayenne migration guide before the upgrade.

Other accelerators are deprecated for removal in v2.5. This applies to Arrow, DuckDB, SQLite, Turso, and PostgreSQL acceleration, and to the DuckDB vector engine. They remain available in v2.4. A startup warning groups affected components by accelerator.

On native Windows, run Spice in Docker or WSL to use Cayenne before v2.5.

Some datasets that loaded on Arrow fail to load on Cayenne. These configurations loaded on Arrow when they did not set engine. They now fail to load unless the dataset sets engine: arrow:

• partition_by in memory mode, or with no mode
• A column of type Interval, Duration, FixedSizeBinary, Union, or RunEndEncoded, such as a PostgreSQL interval column
• A Debezium or cdc: source with no key
• A numeric time_column with retention_period
• maintained_aggregates with MIN or MAX and no primary key

To retain Arrow temporarily in v2.4, name it explicitly:

acceleration:
  enabled: true
  engine: arrow

Plan acceleration migration before v2.5. Do not delete acceleration-only data to change accelerators. Keep those datasets on v2.4 until a supported migration can preserve their data. The migration guide describes other limitations and vector-score changes.

on_conflict is deprecated for removal in v3.0. Cayenne accepts the field but no longer applies its policy. A primary key selects automatic version resolution. Before an upgrade, review drop policies and policies that reject conflicts. Other accelerators retain their existing policies in v2.4.

on_conflict no longer routes writes to the acceleration, for all accelerators. In v2.3, a read-write dataset that set on_conflict sent its writes to the acceleration, so its source did not need to accept writes. In v2.4, only write_mode selects where writes go. Such a dataset does not load if its source cannot accept writes. To keep writes in the acceleration, set write_mode: acceleration:

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    access: read_write
    params:
      file_format: parquet
    acceleration:
      enabled: true
      mode: file
      primary_key: id
      write_mode: acceleration

SQL Results-Cache TTL and Freshness​

SQL results-cache entries now expire only by TTL. Full and append refreshes, CDC batches, and retention passes no longer invalidate them. Queries can return data from before the latest refresh until item_ttl expires. The stale_while_revalidate_ttl setting can extend this period. Review both settings before the upgrade. Reduce them if results must reflect refreshed data sooner. See Cached Results and Dataset Refresh.

HTTP Rate-Control Wait Timeouts​

If rate_control_acquire_timeout is unset, an HTTP request now waits up to the connector's client_timeout for rate-control capacity. Requests previously waited without a limit. Set rate_control_acquire_timeout: 0 under the dataset's params to retain unbounded waits. Otherwise, choose a duration that matches the request budget, such as 30s.

Datasets that share an HTTP origin share one rate controller. They must use the same max_concurrent_requests, requests_per_second_limit, requests_per_minute_limit, rate_control_jitter_min, rate_control_jitter_max, rate_control_failure_threshold, and rate_control_window values. If the values differ, the dataset fails to load with an error that names these settings. See rate control parameters.

Iceberg REST Compatibility​

GET /v1/namespaces/{namespace}/tables/{table} now returns HTTP 400 with BadRequestException for accelerated datasets, views, and other unsupported datasets. The endpoint no longer synthesizes metadata for those datasets. For schema discovery, use SQL DESCRIBE or information_schema.columns. For queries, use /v1/sql or Arrow Flight SQL. Eligible unaccelerated Iceberg datasets expose their source metadata, and clients need their own storage credentials.

Instance Identity and Cluster Upgrades​

Instance identity resolves in this order: nonempty SPICE_INSTANCE_ID, HOSTNAME, COMPUTERNAME, then the host name. A scheduler's default changes from host:port to the host name. Executors also register under the instance identity.

If several schedulers or executors share a host, give each a distinct SPICE_INSTANCE_ID. Upgrade schedulers before executors. See distributed query. Older schedulers cannot fan out task-history requests to newer identity-based registrations. All cluster components must use compatible Ballista protocol versions.

PostgreSQL replication slots keep their names when a nonempty identity environment variable already selected the instance name. Without those variables, the host-name fallback creates a new slot. An empty SPICE_INSTANCE_ID also changes the previous slot name. After the new slot works, drop the unused old slot to stop its WAL retention. See PostgreSQL replication:

SELECT pg_drop_replication_slot('<old slot name>');

Shared State and Snapshot Migration​

The top-level snapshots section and runtime.scheduler.state_location and params are deprecated. They continue to work, and when set they take priority over runtime.state for their feature. Prefer runtime.state, with per-dataset overrides (acceleration.snapshots_location, acceleration.snapshots_params) when necessary.

For upgrades from v2.4.0-rc.1, move state from the shared root into these paths:

An explicit legacy scheduler location retains its earlier layout until migrated. In rc.1, Azure state ignored the URL path and used the container root. Move that state into the configured path and feature subdirectory.

The configured store must support writes and conditional writes. The runtime refuses newer state formats and stores that ignore write conditions. Check storage permissions before the upgrade.

Snapshots now expire automatically after 28 days, except for each dataset's current and newest snapshot. Rollback cannot select a pruned snapshot. Review external retention requirements before the upgrade.

Upgrade snapshot readers before, or alongside, their writers. Cayenne writes format 2 of the metastore slice when a snapshot contains partition children or pending write-back markers. Older readers reject that format. Other slices retain format 1.

Cayenne cannot publish acceleration snapshots for datasets with cayenne_datalake_location. Review snapshot settings for those datasets.

For a serving snapshot reader, bootstrap_on_failure_behavior: retry no longer retries a newer snapshot indefinitely inside one refresh. The reader reports the failure and retries at the next poll. Startup behavior does not change.

Cayenne Tuning Settings​

Replace retired per-dataset and per-catalog tuning settings with runtime settings:

The runtime no longer applies the old names. It warns with their replacements. Targets are runtime-wide. runtime.adaptive_tuning accepts only enabled or disabled. A Boolean such as true fails Spicepod loading.

Upgrading​

Review Breaking Changes before upgrading. Upgrade all distributed cluster components together, with schedulers before executors.

Upgrade snapshot readers before, or alongside, their writers so they can read the newer Cayenne metastore slice format.

To upgrade to v2.4.0, use one of these methods:

CLI:

spice upgrade v2.4.0

Homebrew:

brew upgrade spiceai/spiceai/spice

Homebrew follows the version published in the tap.

Docker:

docker pull spiceai/spiceai:2.4.0

See Docker Hub for available tags.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0

AWS Marketplace:

Spice is available in the AWS Marketplace. Marketplace availability follows its published versions.

Cookbook Updates​

• New Decisions API recipe: call /v1/decisions with TypeSafe Jev and GPT-6 Luna.

The Spice Cookbook provides recipes to help you get started with Spice.

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

What's Changed​

Changelog​

• fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
• fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
• fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
• fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
• fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
• fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
• fix(cayenne): release the keyset bytes an abandoned PK checkout accounted (fixes #13668) by @grokspice in #13925
• fix(duckdb): deny the regexp built-ins DuckDB cannot answer faithfully (fixes #13809) by @claudespice in #13871
• fix(search): drop the chunks a shortened row no longer produces from a chunked index (refs #13717) by @claudespice in #13960
• Reduce Cayenne allocations during primary-key validation and filtering by @lukekim in #14009
• fix: clarify OpenDAL S3 retry warnings by @lukekim in #14040
• feat(cayenne): materialize multi-reference CTEs on the query path by @lukekim in #13918
• fix(smb): pad an empty CREATE buffer so Samba lists the share root (fixes #13293) by @grokspice in #14050
• feat(mcp): support MCP specification 2026-07-28 (dual-era) by @lukekim in #14043
• feat(connectors): add ORC listing format via in-repo FileFormat by @lukekim in #14075
• feat(cloud-connect): add a GetDatasets command that answers the /v1/datasets document (refs #13369) by @grokspice in #14051
• Suppress Cayenne startup logs when no Cayenne dataset is configured by @Jeadie in #14042
• fix(cache): re-bind parameter values when revalidating a stale result (fixes #14099) by @bjchambers in #14100
• fix(deps): remediate open Dependabot alerts by @phillipleblanc in #14111
• feat(caching): Make caching_stale_if_error RFC-5861 compliant (with stale-if-error header) by @Jeadie in #14134
• perf(runtime-table): defer cache-eviction key extraction to entries a delete actually names by @Jeadie in #14138
• fix(runtime): report the acceleration.ready_state deprecation once per component (fixes #13749) by @claudespice in #14006
• fix(runtime): write the inferred Arrow sort order under the prefixed key its validation accepts (fixes #14023) by @claudespice in #14032
• fix(connectors): Fix JSON/Orca files using metadata columns by @Jeadie in #14115
• fix(turso): build a dictionary column directly so a dictionary over a list, map or boolean value reads back (fixes #13033) by @grokspice in #14181
• fix(github): paginate nested GraphQL connections and pace to GitHub's rate limits by @lukekim in #14179
• fix(vortex): stop an IN list holding a NULL from panicking the scan by @krinart in #14163
• fix(runtime-table): stop rebuilding SessionContext on every cache fetch by @Jeadie in #14141
• fix(runtime): count a first load as still loading in the Dataset load summary (fixes #13974) by @claudespice in #14020
• fix(duckdb): push regexp_count down again at a rendering that counts as the kernel does (fixes #13870) by @claudespice in #14153
• fix(cayenne): build the DELETE sink inside the execution-time write lock (fixes #13828) by @claudespice in #14218
• fix(cluster): decide what a Flight message carries by its IPC header, not its body length (refs #13737) by @claudespice in #14212
• fix(runtime): install the Spice function deny-list on the PostgreSQL catalog connector (refs #13664) by @claudespice in #14225
• perf(cayenne): share inline-cache view entries by Arc instead of cloning them per scan by @krinart in #14191
• feat(cache): Spice sharded cache as the sole LruCache engine by @lukekim in #14206
• feat: TypeSafe Jev System One evaluation provider by @lukekim in #14215
• fix(cayenne): tag file statistics bounds as the column's Arrow type (fixes #14280) by @phillipleblanc in #14283
• fix(cayenne): coalesce inline writes into one batch by @sgrebnov in #14279
• fix(cayenne): count each Arrow allocation once in the inline-cache gauge by @krinart in #14272
• feat(s3): SQS event-driven changes for refresh_mode: changes by @lukekim in #14121
• feat(caching): single-flight coalesce concurrent cache-miss fetches by @Jeadie in #14142
• fix(caching): make caching_stale_if_error detect transient HTTP failures on real schemas by @krinart in #14161
• Use Cayenne secondary indexes for dynamic join filters by @phillipleblanc in #14282
• fix(cli): surface the full error chain for spice chat connection failures by @krinart in #14289
• fix(runtime-table): serialize a direct write against acceleration snapshot creation (fixes #13548) by @claudespice in #14310
• fix(duckdb): screen regexp_like and regexp_replace as regexp_count is screened (fixes #14148) by @claudespice in #14321
• fix(http): honor retry budget without an extra origin request by @phillipleblanc in #14313
• Fix SchemaCastScanExec's schema conversion in fn partition_statistics by @Jeadie in #14258
• fix(cluster): recognise a Flight keepalive by its empty envelope, not by what its header declares (fixes #13737) by @claudespice in #14327
• perf(http): defer zero-TTL acceleration lookup until origin failure by @phillipleblanc in #14302
• fix(cayenne): keep a key visible when it is re-inserted over a stale-insert tombstone by @sgrebnov in #14312
• perf(cayenne): read only key and filter columns in filtered key deletes by @sgrebnov in #14374
• fix(cayenne): let small protected-snapshot merges run during a long large-tier merge by @sgrebnov in #14296
• perf(cayenne): serve primary-key lookups on a freshly loaded table in ~1 ms by @lukekim in #14314
• fix(cayenne): skip min/max statistics for nested columns (fixes #14368) by @sgrebnov in #14392
• fix(turso): keep a quantified comparison out of Turso SQL (fixes #14041) by @grokspice in #14393
• perf(cayenne): batch metastore manifest rewrites, upsert in place, and run every write on one writer connection by @lukekim in #14369
• fix(runtime-table): ignore zero-row stale fallback batches by @phillipleblanc in #14331
• Prune object-store file listing by _last_modified predicates by @Jeadie in #14265
• Reading only partition or metadata columns needlessly scans all file contents by @Jeadie in #14116
• fix(duckdb): keep a concat over a binary operand out of the federated plan (fixes #13915) by @claudespice in #14333
• fix(federation): keep a correlated subquery predicate above a join of two sources (refs #8220) by @claudespice in #14372
• fix(caching): snapshot staleness before the origin fetch for stale_if_error by @Jeadie in #14263
• perf(snapshots): skip unchanged snapshot metadata with a conditional GET by @sgrebnov in #14409
• fix(search): prune the rest of a key group from an Elasticsearch chunked index (refs #13717) by @claudespice in #14320
• fix(graphql): debit a LIMIT by the rows a page returned, not the declared page size (fixes #14308) by @claudespice in #14353
• fix(snapshots): resolve snapshot entries relative to the metadata location (#14425) by @sgrebnov in #14426
• Lower GitHub connector default concurrency limit to 4 by @lukekim in #14431
• fix(cache): don't report an invalidation cancelled by runtime shutdown as a failure by @grokspice in #14417
• fix(runtime): keep the built-in date_part so both weekday spellings agree (fixes #13920) by @claudespice in #14154
• fix(cayenne): include the in-memory CDC tier when an overwrite or a retention pass covers the whole table by @lukekim in #14428
• fix: keep NOT IN null-aware through the join reorder and the Cayenne sort-merge rewrite by @lukekim in #14429
• fix(cayenne): discard a compaction whose snapshot an overwrite replaced mid-pass by @lukekim in #14432
• fix(cayenne): stop maintained views, Vortex IN lists and dynamic-filter sharing from returning wrong rows by @lukekim in #14427
• fix(cayenne): hide spilled rows on CDC upsert fallback by @bjchambers in #14416
• fix(cayenne): judge a filtered key delete's captured sources by the index captured with them (refs #13913) by @claudespice in #14455
• fix(cayenne): correctness fixes for the goal-driven adaptive controller, with a closed-loop simulation harness by @lukekim in #14443
• fix(cdc): keep the newest source commit timestamp on a coalesced change batch by @lukekim in #14463
• fix(cayenne): draw a sequence for a current-snapshot append so per-key OCC can order it (fixes #13685) by @claudespice in #14360
• fix(http): name a configured model's load failure instead of reporting it not found (fixes #13303) by @claudespice in #14395
• fix(duckdb): keep inferred source indexes off change-stream accelerations so upserts commit under concurrent reads (refs #13929) by @claudespice in #14396
• fix(duckdb): keep a text cast over a binary operand out of the federated plan (fixes #14355) by @claudespice in #14448
• fix(llms): honor tool_choice required and allowed_tools on mistral.rs-hosted models instead of panicking (fixes #14230) by @claudespice in #14460
• fix(runtime): stop a replaced dataset configuration's load from registering over the new one (fixes #1458) by @claudespice in #14367
• fix(install): stop asking for sudo on a first install into a fresh HOME (fixes #14445) by @claudespice in #14495
• perf(cayenne): serve primary-key point lookups in half the time by @lukekim in #14433
• fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning by @lukekim in #14430
• fix(runtime): size every internal DataFusion session from the CPU budget by @bjchambers in #14412
• fix(runtime): serve nested, zoned and half-float columns from the Iceberg catalog API (fixes #4815) by @claudespice in #14480
• fix(acceleration): keep cached results when a snapshot refresh finds no newer snapshot by @sgrebnov in #14497
• fix(cayenne): stop the small-file compaction of a position-mode PK table from deadlocking on its own write lock (fixes #14420) by @claudespice in #14481
• fix(runtime-table): report refresh bytes for the rows each batch holds by @Jeadie in #14469
• fix(spark,databricks): keep Spice-only functions out of SQL sent to Spark Connect and Databricks SQL Warehouse (refs #13664) by @claudespice in #14498
• fix(vortex): size a cached footer by what it retains, not its serialized bytes (fixes #12917) by @claudespice in #14502
• fix(cayenne,telemetry): Register accelerated sink dataset immediately from existing acceleration by @peasee in #13955
• fix(cayenne): keep a rewrite's count inexact when it retains a late protected snapshot (fixes #14383) by @claudespice in #14385
• fix(postgres): release a shared slot's hold on a table its publication cannot drop (fixes #13032) by @claudespice in #14527
• fix(cayenne): size the build side before sort-merging a non-Cayenne outer join by @krinart in #14520
• fix(deps): move the DataFusion pin past the four unparser fixes, and guard each of them (fixes #13022) by @grokspice in #14570
• fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does by @lukekim in #14419
• feat(snapshots): serve a dataset from published snapshots with from: s3://… and file_format: snapshot by @lukekim in #14529
• perf(cayenne): keep the per-shard PK index across checkpoint flushes and back off futile bakes; fix q17 and two wrong-results races (fixes #14235) by @lukekim in #14555
• fix(graphql): show the parse-failure bytes in JSON decode previews by @Jeadie in #14530
• Defer source-first cache fallback planning by @phillipleblanc in #14556
• perf(cayenne): order whole-table rewrites per scan partition, then merge by @bjchambers in #14488
• fix(graphql): retry an inferred credential refusal instead of failing the refresh by @Jeadie in #14539
• fix(cayenne): serve a widened table's files from their persisted statistics (refs #13829) by @claudespice in #14316
• fix(acceleration): checkpoint DuckDB's write-ahead log before a snapshot copies its file (fixes #13912) by @claudespice in #14325
• fix(models): apply a tool_choice that forces a call to one round of the tool-use loop (fixes #14459) by @claudespice in #14635
• fix(runtime): don't end a tool-use stream on a stale tool_calls finish (fixes #13309) by @claudespice in #14548
• fix(smb): name listed locations under the share so directory datasets load (fixes #14060) by @claudespice in #14549
• fix(models): name a configured model's load failure in ai() and streaming nsql (fixes #14394) by @claudespice in #14632
• fix(cayenne): record sharded CDC keys in the table-wide PK index by @lukekim in #14603
• fix(sqlite): keep TRY_CAST and every cast SQLite answers differently out of the federated plan (fixes #14398) by @claudespice in #14496
• fix(snapshots): retry a failed snapshot attempt with backoff by @sgrebnov in #14676
• fix(snapshots): large snapshots no longer fail bootstrap on slow connection by @sgrebnov in #14571
• fix(refresh): start a refresh's source scan only when the sink reads it, so the full-text index keeps its rows (fixes #14619) by @claudespice in #14663
• Evaluate any chat model via /v1/evaluate by @lukekim in #14568
• fix(cayenne): stop snapshotting datasets with a datalake tier, whose restored copies deleted each other's files by @lukekim in #14583
• fix(cayenne): count each Arrow allocation once for the mem-tier limit and checkpoint write sizing by @krinart in #14675
• fix(runtime): retry a dataset whose source is unreachable at startup (fixes #14609) by @bjchambers in #14623
• fix(cayenne): archive only the snapshot directories a dataset snapshot references (fixes #14605) by @sgrebnov in #14627
• fix(smb): serve every share on a host from the one store registered for it (fixes #14550) by @grokspice in #14689
• fix(runtime): discard cached plans and results when a dataset is unloaded (refs #14251) by @claudespice in #14365
• feat(snapshots): reload snapshot-mode datasets from S3 event notifications (SQS) by @lukekim in #14335
• fix(cayenne): report runs per size tier when protected-snapshot compaction declines (fixes #13622) by @claudespice in #14476
• fix(cayenne): re-run a post-write compaction pass a concurrent append asked for (fixes #13906) by @claudespice in #14479
• fix(http): keep the request URL out of HTTP connector errors (fixes #13534) by @claudespice in #14490
• fix(runtime): load every localpod dataset that reads from one parent at startup (fixes #13087) by @claudespice in #14532
• fix(duckdb): push regexp_replace down again for a one-digit group reference, and refresh the Postgres ClickBench q35 plan (refs #13966) by @claudespice in #14552
• fix(duckdb): keep a cast into binary out of the federated plan (refs #14397) by @claudespice in #14633
• fix(runtime): keep finished async query jobs finished, and delete expired ones by @lukekim in #14585
• fix(federation): keep DataFusion's cast built-ins out of every federated plan (fixes #14444) by @claudespice in #14484
• fix(runtime): decide every Spark/built-in function collision by name, and refuse an undecided one (fixes #14361) by @grokspice in #14400
• fix(runtime-component): skip an inferred Cayenne index on a floating-point column instead of failing the load (fixes #14590) by @grokspice in #14599
• fix(cache): serve stale cached results during frequent table updates (stale_while_revalidate_ttl) by @sgrebnov in #14708
• Update Turso to 0.8.1 and retry write conflicts a metastore statement raises by @lukekim in #14680
• fix(snapshots): stop a snapshot dataset's load when a reload replaces it by @lukekim in #14673
• fix(cayenne): hand non-partition filters to every partition scan (fixes #12959) by @claudespice in #14370
• fix(postgres-accel): resolve secret references in the sidecar connection parameters (fixes #13296) by @claudespice in #14544
• fix(cayenne): drain the in-memory CDC tier before a full rewrite scans it (fixes #14450) by @claudespice in #14486
• feat(caching): warm SQL results cache on first refresh from persisted plan shapes by @lukekim in #14178
• fix(federation): keep a fractional-to-integer cast out of the plans pushed to DuckDB, PostgreSQL, MySQL and BigQuery (fixes #14482) by @grokspice in #14601
• fix(federation): keep arrow_typeof and its plan-introspection siblings local on every backend (fixes #14334) by @grokspice in #14695
• fix(cayenne): back off futile bakes under a violated query goal too by @lukekim in #14736
• fix(snapshots): bootstrap readers from late snapshot publications by @phillipleblanc in #14468
• fix(cayenne): apply retention_sql to a mode: memory acceleration (fixes #14045) by @claudespice in #14344
• fix(duckdb, sqlite): keep the first copy of a key a write repeats under on_conflict: drop (fixes #14629) by @claudespice in #14748
• fix(cache): release a removed dataset's memory when the plan cache discards its plans (fixes #14251) by @claudespice in #14760
• fix(llms): explain an Anthropic model's refusal of a sampling control instead of passing the bare 400 through (fixes #13564) by @grokspice in #14700
• fix(snapshots): allow cayenne_file_path and cayenne_metadata_dir on file_format: snapshot datasets (fixes #14696) by @sgrebnov in #14698
• feat(cayenne): keep the secondary index current across every write by @bjchambers in #14593
• fix(http): stop a non-2xx response body replacing an accelerated table's rows (fixes #13515) by @claudespice in #13538
• fix(chat-api): answer a tool-call-only assistant turn instead of panicking (fixes #13207) by @grokspice in #14232
• fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) by @claudespice in #14670
• fix(spiceai, duckdb, sharepoint): name the Spicepod key a missing-parameter error asks for (refs #14446) by @claudespice in #14769
• fix(cayenne): keep the per-shard PK index when the table-wide index is discarded by @lukekim in #14604
• Upgrade to DataFusion 55.1 and Arrow 59.3 by @krinart in #14612
• Upgrade to iceberg-rust to 0.11.0 and DF 55.2 by @sgrebnov in #14771
• fix(cayenne): write an append into an empty unkeyed table as one load by @phillipleblanc in #14784
• fix(mysql_replication): don't re-send a live member's delivered commits after a reconnect by @lukekim in #14751
• fix(acceleration): accept time_format iso8601 when the time_column is already a timestamp by @lukekim in #14777
• fix(cache): store SQL results that go stale mid-query by @lukekim in #14710
• fix(cayenne): serve filtered and global maintained aggregates by @lukekim in #14761
• perf(cayenne): build the checkpoint's tombstone union after releasing the capture locks by @lukekim in #14759
• fix(cayenne): keep maintenance from deleting files a snapshot is archiving by @sgrebnov in #14789
• fix(snapshots): never overwrite shared snapshot metadata, and publish more than once to a file:// location by @lukekim in #14582
• perf(cdc): build deferred change rows in the reader while the apply runs by @lukekim in #14746
• fix(runtime,data_components): wait for change-data-capture sources to record their positions before shutdown closes the accelerations (fixes #14523) by @grokspice in #14702
• fix(graphql): detect non-JSON responses and treat gateway errors as retryable by @lukekim in #14781
• fix(runtime): require keys for s3_auth: key, honor gs:// state location params, and leave newer rate-control state alone by @lukekim in #14584
• fix(listing): skip zero-byte objects (S3 folder markers) in format-selected listings by @sgrebnov in #14822
• Avoid unnecessary repartitioning in indexed dynamic-filter joins by @bjchambers in #14799
• feat(cayenne): keep one row per primary key automatically and deprecate on_conflict by @bjchambers in #14726
• fix(object-store): stop reading a response body at its first error by @phillipleblanc in #14831
• fix(listing): keep a partition predicate as a residual filter on the _location fast path by @grokspice in #14790
• fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) by @claudespice in #14840
• perf(runtime): skip EnsureRequirements while planning a point lookup by @lukekim in #14807
• feat(acceleration): make Cayenne the default acceleration engine by @phillipleblanc in #14837
• fix(cayenne): keep a join's LIMIT when the oversized-join rewrite makes it a sort-merge join by @lukekim in #14805
• fix(cayenne): fail contradictory write settings once, and order NULL times below every time by @bjchambers in #14847
• fix(runtime): serve an existing acceleration while its source is unavailable (fixes #14610) by @bjchambers in #14624
• Prune object-store file listing by metadata columns (#14264) by @Jeadie in #14303
• fix(vortex): fold partition values into the file-pruning predicate; test null-equal joins under mode:file by @lukekim in #14797
• feat(rate-control): adaptive, bounded and cluster-coordinated HTTP rate controls by @Jeadie in #14143
• fix(sql): align date_part('dow') with EXTRACT(dow) Sunday=0 by @lukekim in #14796
• fix(runtime): carry API-key principal into MCP tools/call by @lukekim in #14828
• fix(llms): un-mute the LLM integration suite and make it pass (fixes #6399; Bedrock deferred to #13558) by @Jeadie in #13656
• perf(cayenne): hold maintained-aggregate retraction state in a compact shared index by @lukekim in #14762
• fix(deps): keep a hash join's order once it has a dynamic filter, so vector_search plans again by @Jeadie in #14857
• feat(cayenne): persist secondary index runs, recorded in the metastore by @bjchambers in #14594
• fix(cayenne): fence schema statistics and control maintenance tests by @bjchambers in #14856
• fix(github): retry nested GraphQL pages in place and fail incomplete nested connections by @sgrebnov in #14862
• fix(cayenne): load an append dataset that has retention_sql and orders versions by time by @phillipleblanc in #14878
• fix(cdc): Cayenne replication needs only a primary key, not on_conflict by @phillipleblanc in #14880
• feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
• fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
• fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
• fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
• fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
• fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
• fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
• fix(cayenne): a hash join over a secondary index reads the indexed table's files once (fixes #14933) by @lukekim in #14934
• fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
• fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
• fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
• fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
• fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
• fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
• fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
• fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
• fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
• fix(llms): stream Anthropic responses that open with a thinking block by @krinart in #14920
• fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
• feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
• fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
• fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
• feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
• fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
• feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
• fix(search): keep named arguments on search calls nested in rrf() (fixes #14923) by @phillipleblanc in #14975
• fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
• fix(vortex): don't push down temporal casts Vortex can't evaluate by @sgrebnov in #14985
• fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
• fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
• fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
• Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
• fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
• fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
• fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
• fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
• fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
• fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
• fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
• fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
• feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
• fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
• fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
• fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
• fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
• fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
• feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
• fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
• feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
• Fixes for MCP server by @Jeadie in #15041
• fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
• fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
• perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
• fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
• fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
• fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
• perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
• fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
• fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
• fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
• feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
• feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
• fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
• perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
• feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
• fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
• fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
• fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
• fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
• feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
• fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
• fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
• feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
• fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
• fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
• fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
• feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
• feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
• fix(models): enable Luna chat and NSQL support by @krinart in #15166
• fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
• fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
• feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176
• fix(iceberg): find the committed Hadoop metadata version by @phillipleblanc in #15142
• feat(iceberg): append refresh follows Iceberg snapshots so late-arriving rows are not missed by @sgrebnov in #15147
• Drive accelerator refresh writes on the refresh runtime by @sgrebnov in #15262
• fix(models): answer ai_* decisions from Bedrock models instead of failing them as refusals by @krinart in #15270
• fix(cache): coalesce Cayenne cache replacements so caching mode keeps up with sustained misses by @krinart in #15253
• feat(cluster): lend a distributed task's slot while it waits on an HTTP source by @peasee in #15123
• fix(cayenne): evict a folded snapshot's cached listing once its superseded files are removed by @bjchambers in #15303
• fix(cache): delete replaced Cayenne cache entries by primary key, not by scanning the table by @krinart in #15312
• fix(cayenne): list snapshot directories under the listing fence by @bjchambers in #15304
• fix(cayenne): checkpoint before a durable CDC write only when a source callback is installed by @phillipleblanc in #15315
• fix(iceberg): read tables whose last commit is schema-only (ADD/RENAME COLUMN) by @lukekim in #15334
• fix(runtime): skip the refresh_append_overlap warning for Iceberg append datasets by @sgrebnov in #15340
• fix(spiced): raise the soft open file limit to the hard limit at startup by @bjchambers in #15339
• fix(iceberg): keyed DuckDB/SQLite append refreshes read by time so late rows don't overwrite newer versions by @sgrebnov in #15346
• fix(acceleration): restore the strict high-water mark in append dedupe by @sgrebnov in #15358
• fix(observability): export a pinned query's execution log records under its trace id by @krinart in #15363
• fix(caching): let an empty origin response replace a cached HTTP entry by @krinart in #15365
• feat: one instance identity for every shared-state feature by @phillipleblanc in #15135
• feat(snapshots): per-dataset snapshot settings; deprecate top-level snapshots and runtime.scheduler state settings by @phillipleblanc in #15136
• fix(state): make every shared-state write in object storage conditional by @phillipleblanc in #15137
• fix(mysql): read the shared binlog on a thread of its own by @lukekim in #15181
• fix(datafusion): keep placeholder type inference linear over deep expressions by @lukekim in #15187
• fix(sql): cast float literals to decimal through their exact spelling by @lukekim in #15082
• fix(federation): keep a Spice-only function local without unfederating the plan around it (fixes #13770) by @krinart in #15078
• fix(snapshots): count every failed snapshot in dataset_acceleration_snapshot_failure_count by @sgrebnov in #15212
• feat(state): clean up unused shared state and keep 14 days of snapshots by @phillipleblanc in #15199
• feat(state): add state-store request, conflict and cleanup-delete metrics by @phillipleblanc in #15203
• fix(sql): keep float token spellings when planning refresh_sql by @lukekim in #15235
• feat(state): one ownership primitive for scheduler membership, job ownership and snapshot writers by @phillipleblanc in #15196
• fix(search): reject empty or whitespace-only text on /v1/search by @lukekim in #15119
• feat(snapshots): keep a dataset's snapshots for 28 days by @phillipleblanc in #15241
• fix(snapshots): release the snapshot writer lease on graceful shutdown by @sgrebnov in #15256
• feat(monitoring): snapshot freshness, Cayenne segment cache and pod health panels on the Datadog dashboard by @sgrebnov in #15309
• fix(observability): report snapshot reader refresh failures in metrics and label them mode=snapshot by @sgrebnov in #15310
• fix(cayenne): leave the Turso metastore log out of snapshots by @phillipleblanc in #15318
• fix(object_store_occ): never let a local version repeat an earlier version's ETag by @claudespice in #15437
• fix(cayenne): complete, version, and validate a dataset's metastore slice (fixes #13241) by @claudespice in #14238
• fix(acceleration): fail and retry a refresh whose source goes silent mid-stream by @sgrebnov in #15457
• fix: emit one HELP/TYPE per derived Prometheus summary family by @lukekim in #15416
• fix: serve append accelerations with snapshots enabled while the source is down by @sgrebnov in #15463
• perf(cayenne): compact an append-only table's small files without stalling its writers by @lukekim in #15227
• fix(cayenne): keep lookups fast under sustained replacement on tables without a primary key by @krinart in #15362
• fix(snapshots): report a snapshot reader that cannot load a newer snapshot by @sgrebnov in #15472
• fix(cayenne): rematerialize autocommit UPDATE inside write_lock by @lukekim in #15490
• feat: deprecate non-Cayenne acceleration engines and the DuckDB vector engine by @bjchambers in #15496
• fix(cayenne): count logical nulls so IS NULL keeps rows in inline and RAM-tier data by @lukekim in #15186
• fix(cayenne): filter a scan view's rows against the deletions it captured by @lukekim in #15169
• fix(snapshots): stop a restarted snapshot reader from serving a partial table by @sgrebnov in #15536
• fix(cluster): keep executor shuffle fetches on cluster mTLS by @phillipleblanc in #15484
• Add table-bound ChangeSink ownership and backends by @phillipleblanc in #14704
• chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
• chore(deps): raise the datafusion-table-providers pin to include the NUMERIC result-column fix by @phillipleblanc in #14254
• feat(key-index): immutable secondary index runs over compound Arrow keys by @bjchambers in #14592
• feat(object_store_occ): add transactional WAL and MVCC snapshots by @lukekim in #14732
• Route CDC refresh and Cayenne cache writes through the table-bound ChangeSink by @phillipleblanc in #14788
• Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
• fix(duckdb): pull in the bundled Brotli decoder fix (CVE-2026-111497) by @phillipleblanc in #15580

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.3...v2.4.0

Spice v2.4.0-rc.2 is now available! 🔥

Spice v2.4.0-rc.2 is the second release candidate of v2.4.0 for early testing and feedback. It introduces a new OpenAI-compatible POST /v1/decisions API backed by an OpenAI decision model, TypeSafe Jev, or any other LLM in the Spicepod that can answer decisions. Five new SQL functions bring the same AI classification, scoring, and decisions into SQL.

This release also adds Iceberg reads without a catalog and improves acceleration snapshots. Cayenne gains runtime-wide adaptive tuning and applies acceleration configuration changes on restart.

Highlights in this release:

• AI Decisions API and decisions in SQL: An OpenAI-compatible /v1/decisions API, plus five SQL functions for filtering, classifying, and scoring data with models.
• Iceberg without a catalog: Read a table directly from its root directory or pin a metadata file to read a fixed version.
• Shared runtime state: Startup validation and a common location for cluster state, rate control, cache warmup, and acceleration snapshots.
• SQL results cache: Cache entries expire by TTL and remain available across data refreshes.
• Cayenne acceleration: Improvements to primary-key loading, secondary indexes, and configuration updates.
• OpenTelemetry log export: Send runtime logs to any OTLP endpoint as structured log records, and choose which failed queries are logged.

These notes cover changes since v2.4.0-rc.1. Review Breaking Changes before upgrading; upgrades from earlier versions also require the rc.1 migration guidance.

What's New in v2.4.0-rc.2​

AI Decisions API and decisions in SQL​

Spice now supports POST /v1/decisions, which is compatible with the OpenAI Decisions API. A decision request asks typed questions about an input. Each answer comes back with its probabilities and a confidence.

Any model in the Spicepod can answer a decision request:

OpenAI decision models use the native OpenAI Decisions API. gpt-6-luna also supports chat and /v1/nsql. TypeSafe Jev returns calibrated probabilities and answers decisions only. Other LLMs answer through chat completions and return their own uncalibrated estimates.

models:
  - from: openai:gpt-6-luna
    name: luna
    params:
      openai_api_key: ${secrets:OPENAI_API_KEY}
  - from: typesafe:jev
    name: jev
    params:
      typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Five new SQL functions make the same decisions available in queries:

SELECT
    id,
    ai_classify(body, ['billing', 'technical', 'account'], model => 'support_model' ) AS team,
    ai_probability(body, 'The customer is requesting a refund', model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI functions for more examples.

Iceberg Without a Catalog​

Read an Iceberg table directly from object storage or the local filesystem, without configuring a catalog. Set from to the table root using file://, s3://, s3a://, gs://, or gcs://:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1

Spice resolves the metadata file on each query and refresh. It supports metadata/version-hint.text as well as a fixed version (a path relative to the table root or a full URL):

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1
      iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Fixed tables are read-only. The earlier metadata_path parameter remains supported with a deprecation warning; see Breaking Changes.

Shared Runtime State​

runtime.state provides a shared, object-store location for cluster state, rate control, SQL results-cache warmup, and acceleration snapshots. Each feature uses its own subdirectory within the provided location.

Acceleration snapshots now default to the snapshots/ subdirectory when runtime.state is configured.

runtime:
  state:
    location: s3://my-bucket/spice/

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled

A dataset can override the snapshot location, its settings, and the bootstrap behavior on its own acceleration:

datasets:
  - from: postgres:events
    name: events
    acceleration:
      snapshots: enabled
      snapshots_location: s3://archive-bucket/snapshots/
      snapshots_params:
        s3_region: eu-west-1
      snapshots_bootstrap_on_failure_behavior: fallback

Other changes to shared state:

• Snapshot queues: An SQS queue that covers the whole runtime.state prefix no longer causes an ERROR log for each scheduler heartbeat. The snapshot reader now deletes messages for the other state folders without a log.

SQL Results Cache​

The SQL results cache now expires entries by TTL, independent of acceleration refreshes, CDC batches, and retention passes no longer invalidate cached results.

Review item_ttl and stale_while_revalidate_ttl when upgrading: a successful refresh no longer makes cached results reflect the refreshed data immediately. See Breaking Changes.

Cayenne Acceleration​

Adaptive tuning is configured once for the runtime. Enable it with runtime.adaptive_tuning and set targets under runtime.params:

runtime:
  adaptive_tuning: enabled
  params:
    target_freshness: 30s

Supported targets are target_replication_lag, target_freshness, target_query_latency, target_convergence_window, and target_qph. Adaptive tuning defaults to disabled. See Breaking Changes and the runtime YAML documentation.

Other Cayenne improvements include:

• Primary-key loading: Keyed tables build their primary-key filter during the initial load and bulk-load without per-row routing or a whole-table fold.
• Position deletes: cayenne_deletion_mode: auto can select position deletes for small primary-key tables on local storage on Linux systems with at least eight physical cores.
• Secondary indexes: Index runs are persisted, indexes remain complete across acceleration snapshots, and CDC maintains indexes for each operation type. Hash joins read each file once when narrowing a scan through an index.
• Partition handling: Cayenne rejects stored data with an incompatible partition_by, protects keyed appends to partitioned tables during publication, and prepares every partition after a refresh.
• Storage and query correctness: Cayenne reclaims snapshot directories superseded by a reload, corrects pruning for NaN values, and fixes deletion-index bounds and reload fencing.
• Snapshot storage growth: A writer with snapshots_trigger: refresh_complete kept one more copy of the table for each refresh. Each snapshot held file deletions, so Cayenne skipped each cleanup pass. Cayenne now runs the cleanup when a snapshot releases its hold. In a 10-minute test with a refresh every 60 seconds, each dataset kept 1 snapshot directory instead of 10.
• Configuration changes on restart: A change to write settings only, such as cayenne_sort_columns or cayenne_compression_strategy, applies to the existing acceleration. A change to primary_key or partition_by rebuilds the acceleration from the source only under specific conditions. See Cayenne Configuration Updates.

OpenTelemetry Log Export​

Spice can now send its logs to an OpenTelemetry Collector, or any OTLP endpoint, as structured OTLP log records. Logs use the same otel_exporter that already sends metrics. A new runtime.query.log_errors setting chooses which failed queries are logged.

runtime:
  telemetry:
    otel_exporter:
      endpoint: otel-collector:4317
      signals: [metrics, logs] # new; default [metrics]
  query:
    log_errors: execution # new; none (default) | execution | all

• Structured records: Each log line becomes an OTLP log record with its severity, the message as the body, typed fields as attributes, and the logging module as the instrumentation scope. Records share the resource attributes of exported metrics.
• Trace correlation: A record logged during a query carries the query's trace ID, the same ID returned in spice-trace-id and stored in runtime.task_history.
• Delivery: Records are batched on a dedicated export runtime, separate from query and HTTP traffic, and flushed at shutdown so the final error and Goodbye! reach the endpoint. For OTLP/HTTP endpoints, logs are sent to <base>/v1/logs.
• Cluster executors: Executors export their logs with the scheduler's telemetry settings, including headers resolved from the scheduler's secrets.

runtime.query.log_errors controls which failed queries are logged:

Each record reads Query failed: <error>, with error_code and datasets fields and the query's trace ID. The SQL text is never logged.

Unknown-column errors are now reported with the correct error_code instead of InternalError, in logs, runtime.task_history, and the query_failures metric.

See OpenTelemetry Logs Export, Exporting logs, and runtime.query.log_errors.

Fixes and Improvements​

AI APIs and Models​

• Responses API: /v1/responses executes Spice tools and honors tool_recursion_limit when streaming. It returns Spice tool calls as mcp_call items in output, allowing clients that manage their own conversation history to replay them.
• Model controls: /v1/nsql accepts reasoning_effort, and rerank() uses evaluation models with a strategy selected for each query.
• Anthropic: Streaming handles responses that begin with a thinking block, empty text blocks are dropped, and anthropic_workspace_id is available as a model parameter.
• OpenAI: Models with an unknown service tier can load and serve requests.
• Luna: gpt-6-luna and its snapshots support chat, /v1/nsql, and native decisions.

Acceleration​

• Caching point lookups: Lookups on a refresh_mode: caching dataset failed with HTTP 400. This occurred when Cayenne accelerated the dataset, the primary key covered the request columns, and the table held two protected snapshots. The point-lookup plan did not combine the partitions of the Cayenne scan. Lookups now return the cached row or fetch it from the source.

Connectors​

• MySQL timestamps: Timestamp literals in queries that the runtime sends to MySQL now keep their fractional seconds.

Dependency Updates​

Breaking Changes​

Shared State Directory Layout and Startup Checks​

State written at the runtime.state root by rc.1 is no longer read from that location. Move existing state objects into the appropriate subdirectories before upgrading to retain them:

Azure locations (abfs:// and abfss://) need one more step. In rc.1, the runtime ignored the path in an Azure URL and wrote state at the container root. For example, with abfs://container/spice/, rc.1 wrote cluster.json at the container root. This release writes spice/cluster/cluster.json. Move Azure state from the container root into the configured path and then into the feature subdirectory.

An explicit runtime.scheduler.state_location retains its earlier layout. Without migration, Spice starts without the state previously stored at the shared root.

The configured store must accept writes and conditional writes. Spice does not become ready until startup validation succeeds and refuses stores with a newer state format. Check storage permissions and compatibility before upgrading.

SQL Results Cache Freshness​

A refresh no longer invalidates SQL results-cache entries. Queries can return data from before the latest refresh until item_ttl expires, plus stale_while_revalidate_ttl when configured. Reduce item_ttl if results need to follow source refreshes more closely.

Cayenne Tuning Configuration​

Replace the Cayenne tuning parameters with runtime settings that apply across datasets and catalogs:

runtime.adaptive_tuning accepts only enabled or disabled. Other values, including true, fail the Spicepod load.

Pinned Iceberg Tables​

Replace the deprecated metadata_path parameter with iceberg_metadata_path. The two settings resolve paths differently. metadata_path uses its value as given. iceberg_metadata_path resolves a value without a URL scheme relative to the table root in from. Convert an absolute local path to a full file:// URL or to a path relative to the table root.

Datasets pinned to a .json metadata file are read-only and fail to load with access: read_write. Remove that setting from pinned datasets before upgrading.

Cayenne Configuration Updates​

A restart now applies changed settings to an existing Cayenne acceleration. The result depends on the setting and the dataset:

• Write settings only: A change to only cayenne_sort_columns, cayenne_cluster_by, cayenne_compression_strategy, cayenne_delta_encoding, or cayenne_shard_key_columns applies to the existing acceleration. Stored rows keep their layout until a refresh or compaction rewrites them.
• Identity settings: A change to primary_key, partition_by, cayenne_pk_conflict_detection from none to auto, or the data location rebuilds the acceleration from the source. The runtime rebuilds only a dataset with refresh_mode: full, mode: file or file_update, and write_mode: write_through. The source must not be sink:, and must be readable with the schema of the dataset.
• All other datasets: The acceleration keeps all of its stored settings. This includes write settings that changed in the same restart. The runtime logs a warning that names the change and the fix.

While the runtime cannot read the source, a dataset that qualifies for a rebuild keeps its existing acceleration. A partition_by change or a conflict-detection change cannot use the stored rows, so it fails the dataset instead. Revert the setting, or restart when the source is readable.

A restart alone does not migrate a dataset that does not meet these conditions. Read the startup warning for the dataset before you delete stored data. Allow time for a rebuild when the conditions apply.

Upgrading​

Once release artifacts are available, select v2.4.0-rc.2 explicitly with the CLI, Docker tag, or Helm chart version.

CLI​

spice upgrade v2.4.0-rc.2

Docker​

docker pull spiceai/spiceai:2.4.0-rc.2

See Docker Hub for available tags.

Helm​

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.2

Homebrew and AWS Marketplace​

Homebrew and AWS Marketplace follow their published versions. To select this release candidate, use an explicit version through the CLI or Docker.

Spice is also available in the AWS Marketplace.

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @spiceemma

Changelog​

See the full changelog from v2.4.0-rc.1 to v2.4.0-rc.2.

• feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
• fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
• fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
• fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
• fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-duckdb[file]-2gib.yaml by @github-actions in #14850
• Update openapi.json by @github-actions in #14858
• test: strengthen assertions and enforce sign-off coverage by @lukekim in #14740
• Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
• fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
• fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
• fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
• test(sqlite): guard the Date32 literal cast through the SQLite acceleration's own scan (fixes #14491) by @grokspice in #14703
• fix(cayenne): read each file once when a hash join narrows an indexed table's scan by @lukekim in #14934
• fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
• fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
• fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
• fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
• fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
• fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
• feat(spicepod): publish pull request reviews by @lukekim in #14964
• fix(ci): bound the REPL expect-script waits on silence, not total response time (fixes #13711) by @grokspice in #14963
• fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
• fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
• fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
• fix(ci): provision DuckLake for the ducklake[catalog] schema test, which has never passed by @grokspice in #14952
• fix(llms): stream Anthropic responses that open with a thinking block (fixes #14909) by @krinart in #14920
• fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
• feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
• fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
• fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
• feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
• fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
• feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
• fix(rrf): keep named arguments on nested search calls by @phillipleblanc in #14975
• fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
• fix(vortex): keep temporal casts Vortex can't evaluate out of the scan by @sgrebnov in #14985
• fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
• test(huggingface): unblock the merge queue: read imdb's parquet conversion through @~parquet and fail fast on a vanished Hub pin by @grokspice in #14998
• test(cayenne): link the integration tests into one test binary by @bjchambers in #14976
• test(vortex): compile the integration tests into a binary the sign-off gate runs by @bjchambers in #14987
• fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
• fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
• Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
• fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
• test: run federated source tests on every supported LTS release line by @lukekim in #14936
• fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
• fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
• fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
• Improve types methods in runtime-rate-control by @Jeadie in #14573
• fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
• Update openapi.json by @github-actions in #14908
• test(cayenne): run the heap-measuring tests in one isolated test binary by @bjchambers in #15010
• test(vortex): run vortex-datafusion's integration tests in the sign-off gate by @bjchambers in #15001
• fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
• Update openapi.json by @github-actions in #15018
• fix(ci): keep the OpenAPI regeneration PR current instead of skipping while one is open by @phillipleblanc in #15023
• fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
• fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
• chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
• feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
• ci: apply whole-binary nextest overrides to the module test sets they select by @bjchambers in #15011
• fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
• test(data_components): compile the integration tests into one test binary by @bjchambers in #14991
• fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
• fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
• test(data_components): make the HTTP provider snapshots independent of the machine by @bjchambers in #15000
• fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
• test(runtime): drive the cdc_cayenne tests through Cayenne's change sink by @bjchambers in #15003
• ci(object_store_occ): run the WAL E2E suite on self-hosted runners against disk, MinIO, RustFS and spiceio by @lukekim in #15031
• test(llms): compile the credential-free tests into the integration binary by @bjchambers in #14990
• fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
• ci(setup-minio): name the missing secret instead of failing in mc's argument parser by @grokspice in #15073
• ci: run Substrait Mode B on the Linux runners that can reach the trunk spiced builds by @grokspice in #15074
• feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
• build(deps): bump the github-actions-dependencies group across 5 directories with 7 updates by @dependabot in #15027
• fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
• feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
• Fixes for MCP server by @Jeadie in #15041
• fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
• fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
• perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
• fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
• fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
• fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
• perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
• fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
• test(runtime,spiced): compile integration tests into declared binaries, run the ones no CI job ran, and remove the uncalled supersede_initial_load by @bjchambers in #15005
• fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
• ci(e2e): don't run rust-cache on the macOS build, whose save deletes cargo from sibling runners by @grokspice in #15100
• feat(testoperator): record per-query search metrics by @Jeadie in #15090
• fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
• feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
• ci: run merge-queue Mode A TPC-H and LLM integration tests on spiceai-macos-large by @lukekim in #15116
• docs(cayenne): the per-file fold merges no source files, but may roll by @bjchambers in #15101
• build(deps): bump async-openai to the fork's merge of upstream main by @krinart in #14980
• feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
• fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
• perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
• test(cayenne): guard per-file statistics sharing for listings outside a scan view by @bjchambers in #15103
• feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
• fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
• Change codeql runner from spiceai-macos to spiceai-macos-large by @lukekim in #15120
• fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
• fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
• fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
• fix: Update Search integration test snapshots by @github-actions in #15050
• test(iceberg): read the Glue fixture unpinned and check the picked metadata file by @sgrebnov in #15115
• test(cayenne): differential and plan coverage for secondary indexes on partitioned accelerations by @bjchambers in #15113
• test: answer results-cache misses from the acceleration while the source is down by @bjchambers in #15112
• ci(lint): fail on test files no target compiles, and document the integration test layout by @bjchambers in #15109
• feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
• fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
• fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
• feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
• fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
• fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
• fix(test): wait for pkd_ctl to load before reading its bootstrap row in the write-back pk_conflict test by @grokspice in #15102
• fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
• test(mysql): skip the SQL results cache in the binlog replication end-to-end tests by @grokspice in #15141
• feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
• feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
• fix(models): enable Luna chat and NSQL support by @krinart in #15166
• fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
• fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
• feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176

Spice v2.1.1 is now available! 🛠️

Spice v2.1.1 is a patch release focused on reliability and performance. It resolves a possible deadlock that affected datasets with partitioned Cayenne accelerators, caches empty SQL result sets so repeat queries are served from cache, speeds up repeated queries on multi-file datasets, and restores Bedrock embedding provider parameters.

What's New in v2.1.1​

Cayenne Partitioned Dataset Deadlock Fix​

Cayenne datasets configured with partition_by could deadlock during their initial refresh and never become ready. Non-partitioned tables and small partitioned tables were unaffected. The root cause was a deadlock between the partition routing and the global Vortex encode budget introduced in v2.1.0.

Cayenne Zero-Row Append Refresh Stability​

An idle append refresh, one where no source rows are newer than the current max(time_column), wrote no Vortex files, so the expected snapshot directory was never created. The subsequent fsync on that directory failed with ENOENT, marking the dataset unhealthy. The fix skips the snapshot sequence record and protected-snapshot publish when the write carried no rows.

Caching of Empty Result Sets​

The SQL results cache now stores empty (zero-row) result sets. Queries that legitimately return no rows (e.g. WHERE 1=0, LIMIT 0) are now served from the results cache on subsequent requests instead of being re-executed against the source, reducing planning and query latency for these patterns.

Faster Repeated Queries on Multi-File Datasets​

Object store datasets using parquet now cache Parquet footer statistics across queries. This reduces the frequency of Parquet footer parsing during planning, subsequently heavily reducing planning latency for certain query patterns (e.g. COUNT(*)).

Embedding Parameter Regression Fixes​

v2.1.0 introduced explicit definitions across embedding component parameters (i.e. .embeddings[].params). This introduced regressions for AWS Bedrock parameters truncation and truncation_mode that caused panics.

Contributors​

• @bjchambers
• @Jeadie
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.1 image:

docker pull spiceai/spiceai:2.1.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(cache): cache empty (zero-row) SQL result sets by @bjchambers in #11699
• fix(cayenne): zero-row append refresh fails with "No such file or directory" by @sgrebnov in #11710
• feat(file): cache ListingTable file statistics to avoid per-query footer re-parse by @phillipleblanc in #11793
• fix(embeddings): restore params broken by #10853 by @Jeadie in #11788
• fix(cayenne): partitioned datasets deadlock against the global encode budget and never become ready by @Jeadie in #11825

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.0...v2.1.1

Spice v2.1.0 is now available! 🎉

Spice v2.1.0 is the next minor release of Spice, headlined by high-throughput Cayenne CDC, scaling and resilience improvements to PostgreSQL logical replication, expanded distributed query with Iceberg catalog scans and broadcast joins, and the upgrade to DataFusion v54 (including v53), Arrow v58.3, and Vortex v0.74. The release also adds experimental adaptive self-tuning for the Cayenne accelerator, distributed GLM inference, and a range of security, search, and connector improvements.

Highlights in v2.1.0 include:

• High-Throughput Cayenne CDC — in-memory CDC tier, dedicated compaction runtime, and write-path optimizations that cut replication lag on high-volume CDC workloads
• PostgreSQL Replication at Scale — multiple changes-mode datasets share a single replication slot, unchanged-TOAST recovery, and resilient reconnects across rolling deploys
• Distributed Query — distributed Ballista scans of Iceberg catalog tables, broadcast joins for small dimension tables, and shared scheduler job state with failover
• DataFusion v54 — upgrade to DataFusion v54 (folding in v53), Arrow v58.3, and Vortex v0.74, bringing faster joins, scans, and planning
• Adaptive Self-Tuning (Experimental) — opt-in closed-loop tuning and maintained aggregates that adapt Cayenne to hardware, schema, and live workload

What's New in v2.1.0​

High-Throughput Cayenne CDC​

A major focus of v2.1 is Spice Cayenne write-path throughput for change-data-capture (HTAP) workloads:

• In-Memory CDC Tier: A new in-memory CDC tier and follow-ups cut replication lag on hot upsert tables, with bounded mem-tier checkpointing and O(1) per-scan deletion views, plus a two-phase off-fence checkpoint on the ingest path.
• Dedicated Compaction Runtime: A dedicated compaction runtime with CDC pipelining and protected snapshots isolates compaction from query and ingest paths, with parallelized deletion-vector writes, per-batch directory-barrier coalescing, and size-aware parallel encode for protected-snapshot compaction.
• Incremental Protected Snapshot Compaction: Incremental compaction of protected snapshots (used in Cayenne's merge-on-read deletion index) reduces disk usage and improves query performance.
• Smaller WAL & Metadata-Only Publish: cayenne_insert_record table IDs are stored as 16-byte raw-UUID BLOBs, cutting CDC WAL volume ~34%; upsert commits publish metadata-only, dropping per-key insert records; transient staged CDC deltas are light-encoded.
• Delta-Write Encoding Levels: A new cayenne_delta_encoding setting (default auto) selects delta-write encoding, and cayenne_compression_strategy: zstd is now fully wired.
• In-Memory CDC Sharding: PK-hash intra-apply sharding parallelizes in-memory CDC apply.
• Scan Safety Under Write: In-flight scans are ref-counted so snapshot GC can't delete Vortex files mid-read; in-RAM scan parallelism, query admission control, and sound scan output ordering improve read behavior under sustained CDC.

Delta-write encoding effort and Vortex compression are tunable per accelerator. cayenne_delta_encoding: auto (the default) size-gates fresh CDC/append writes — small deltas use a light scheme and are re-encoded during compaction — or pin an explicit level 0..10 (7 is the full default cascade); cayenne_compression_strategy selects the Vortex compression:

acceleration:
  engine: cayenne
  refresh_mode: changes
  params:
    cayenne_delta_encoding: auto # 'auto' (default), or pin a level 0..10 (7 = full cascade)
    cayenne_compression_strategy: zstd # 'btrblocks' (default) or 'zstd'

Change Data Capture & HTAP​

PostgreSQL logical replication (CDC, refresh_mode: changes, introduced in v2.0) gets significant scaling and resilience work in v2.1:

• Shared Replication Slot: Multiple refresh_mode: changes PostgreSQL datasets on the same connection can name the same pg_replication_slot to share a single replication slot, walsender decoder, and publication, with decoded changes multiplexed by (schema, table) to each dataset. This collapses the slot count from one-per-dataset to one — staying well under Postgres's default max_replication_slots = 10.

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # shared slot name
    acceleration:
      refresh_mode: changes
  - from: postgres:public.customers
    name: customers
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # same name -> one slot, walsender & publication
    acceleration:
      refresh_mode: changes

• Unchanged-TOAST Recovery: Under REPLICA IDENTITY FULL, when an UPDATE leaves a large TOASTed column unchanged, pgoutput sends an "unchanged" marker; Spice now fills that value from the old tuple — its old value is its current value — so updates no longer error or drop columns. Without an old tuple, the error persists with a hint to enable REPLICA IDENTITY FULL.
• Transient Walsender Contention: Slot-contention errors during rolling deploys — SQLSTATE 55006 ("replication slot is active for PID") and 53300 ("requested standby connections exceeds max_wal_senders") — are now classified as transient and retried with backoff instead of fatally terminating the stream. Replication connections are also released at shutdown start (not process exit), freeing walsender seats for replacement instances.
• Strict CDC Param Validation: PostgreSQL CDC parameters are strictly validated rather than silently defaulted.
• Debezium Schema Evolution: Fixes for Debezium schema-evolution support, including tombstone-message handling and sign-extension of minimal-width base64 decimals.

Distributed Query​

Distributed Query gains:

• Distributed Iceberg Catalog Scans: Ballista distributes scans of Iceberg catalog tables across executors.
• Broadcast Joins: Small dimension tables are broadcast to executors for distributed joins.
• Shared Scheduler Job State with Failover: Ballista job state is shared so the scheduler can fail over without losing in-flight work.

Performance & Query Engine​

Apache DataFusion is upgraded to v54, folding in v53, alongside Arrow v58.3 and Vortex v0.74 (with a pin bump adding intra-file decode split and a per-execution kernel cache). Two DataFusion releases land in this upgrade:

• DataFusion v54 (release notes): adds LATERAL joins, SQL lambda functions (x -> expr with array_transform/array_filter/array_any_match), spilling nested-loop joins, and a faster arrow-avro reader. Performance work includes morsel-driven Parquet scans (up to ~2x faster for skewed scans), 20-50x faster sort-merge semi/anti/mark joins, redundant-sort-key pruning, NDV-based cardinality estimation, and inner_product/cosine_distance functions.
• DataFusion v53 (release notes): adds LIMIT-aware Parquet row-group pruning, broader filter pushdown through joins and UNION, nested-field pushdown (get_field into the scan), faster query planning (some plans dropping from ~4-5ms to ~100us), and 42 faster built-in functions.

Federation deny-list enforcement and catalog DDL are restored after the DataFusion upgrades, and a cost-based left-deep join reordering rule is added for Cayenne acceleration.

AI & LLM​

• Native GLM Support with Distributed Inference: Native GLM model support with surfaced reasoning_content, including tensor-parallel GLM inference. Load a GLM model with model_type: glm4 (glm4moe and glm4moelite are also supported):

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4

For large models, GLM inference can be distributed across nodes (tensor parallelism) via the mistral.rs pure-TCP ring all-reduce backend — no NCCL/system dependency. This is a Spice.ai Enterprise feature requiring the distributed build. Run the same model on each node, changing only node_rank:

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4
      distributed_backend: ring
      nodes: 10.0.4.21,10.0.4.22 # ordered host/IP per rank; the ring backend currently requires exactly 2
      node_rank: 0 # rank of THIS node in [0, world_size); rank 0 serves the API. Set node_rank: 1 on 10.0.4.22

• NSQL Context Endpoint: A new GET /v1/nsql/context endpoint returns the SQL dialect, dataset schemas (with optional sample rows), and registered functions that Spice injects into natural-language-to-SQL (POST /v1/nsql) requests — useful for inspecting or caching exactly what the model sees:

# Inspect the context injected into /v1/nsql requests (examples_limit default 3, max 100)
curl "http://localhost:8090/v1/nsql/context?include_examples=true&examples_limit=3"

Returns the dialect, per-dataset schema (keys, indexes, searchable columns), the registered function inventory, and sample rows (abbreviated):

{
  "context": "# Spice.ai NSQL Context",
  "instructions": [
    "Write SQL for the Spice runtime, which uses Apache DataFusion with the SQL parser configured for the PostgreSQL dialect.",
    "Use table and column descriptions, primary keys, foreign keys, unique constraints, and indexes when choosing joins and filters."
  ],
  "sql": {
    "engine": "Apache DataFusion",
    "version": "54.0.0",
    "dialect": "PostgreSQL",
    "parser": "DataFusion SQL parser configured with PostgreSQL dialect"
  },
  "datasets": [
    {
      "name": "sales.orders",
      "table": "orders",
      "description": "Customer orders",
      "columns": [
        { "name": "order_id", "data_type": "Int64", "nullable": false, "primary_key": true, "indexed": true },
        { "name": "customer_id", "data_type": "Utf8", "nullable": false, "vector_search": true, "full_text_search": true }
      ],
      "primary_key": ["order_id"],
      "foreign_keys": [
        { "columns": ["customer_id"], "foreign_table": "spice.sales.customers", "foreign_columns": ["id"] }
      ]
    }
  ],
  "functions": {
    "summary": "Spice SQL runs on Apache DataFusion ... Run SELECT * FROM list_udfs() to inspect the full registered function inventory",
    "search": [
      { "name": "vector_search", "syntax": "vector_search(dataset, 'query text'[, column])" },
      { "name": "text_search", "syntax": "text_search(dataset, 'query text'[, column])" }
    ]
  },
  "samples": [
    { "title": "Example rows for `sales.orders`", "content": "| order_id | customer_id |\n| --- | --- |\n| 42 | CUST-1 |" }
  ]
}

Search & Vectors​

• S3 Vectors Pagination: QueryVectors paginates for top-K up to 10,000.
• Elasticsearch kNN Candidate Pool: The default kNN candidate pool is raised from 10 to 1000 for better recall.

SQL & Query Engine​

• FlightSQL Substrait Plans: CommandStatementSubstraitPlan support.
• Large Result Streaming: Flight streaming is optimized for large result sets.
• Write Authorization: The SQL tool allows writes for ReadWrite API keys.
• Schema Evolution Policies: on_schema_change supports widening-only evolution and a drop_and_recreate policy.

Security & Connectors​

• Kafka mTLS: Mutual TLS configuration is surfaced in the Kafka data connector.
• Secret Resolution at Startup: Secret references are checked and reported at startup.
• DuckDB HNSW: Upgrade to DuckDB v1.5.3 with the statically linked VSS (HNSW) vector extension.

Adaptive Self-Tuning (Experimental)​

The Spice Cayenne accelerator gains experimental opt-in self-tuning. cayenne_tuning: auto derives configuration from the detected hardware and inferred schema, while adaptive additionally runs a per-table closed-feedback controller that adapts flush caps, the in-memory CDC tier, compaction cadence, and write concurrency toward operator SLOs (replication lag, freshness, query latency, queries-per-hour). Cayenne can also maintain aggregates incrementally — with predicate-aware delta serving and incremental retraction — and fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics. These features are experimental and disabled by default.

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      refresh_mode: changes
      params:
        cayenne_tuning: adaptive # 'auto' (static, env- + schema-derived) or 'adaptive' (closed-loop)

Observability​

• Per-Dataset Query Attribution: The query_executions metric gains a datasets dimension.
• HTAP Diagnostics: Improved HTAP replication diagnostics on non-convergence.
• Cayenne Write Observability: Write-phase observability for the in-memory CDC tier.

Notable Bug Fixes​

• Cayenne Utf8View: The Utf8View read schema avoids a hash-join offset overflow.
• Cayenne metastore: cayenne_metastore: turso is honored for partitioned tables and the dataset checkpoint.
• Dual-write detection: Dual-write accelerated tables are detected behind the metadata-enrichment wrapper.
• digest_many collisions: Values are length-prefixed so column boundaries can't collide.
• Turso WAL checkpoint: WAL checkpoints route through the native Turso connection.
• TLS status probe: The status check probes the metrics endpoint over HTTPS when TLS is enabled.
• Search snippet offsets: Character chunk offsets persist so search snippets aren't shifted or garbled.
• Async query chunk offsets: /v1/queries chunk row_offset uses the cumulative offset rather than chunk_index * chunk_size.

Dependency Updates​

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @melks
• @mvanhorn
• @Oxygen56
• @peasee
• @phillipleblanc
• @sgrebnov
• @v1gnesh

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.0 image:

docker pull spiceai/spiceai:2.1.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Make DuckDB schema cast logic more robust by @sgrebnov in #10991
• perf(cayenne): reduce allocation overheads in hot paths by @lukekim in #10950
• improve error message for 'params.spiceai_region' by @Jeadie in #10954
• fix(acceleration): rename WriteMode variants to fix #10960 by @phillipleblanc in #10974
• add Scylla, bigquery and turso to throughput benchmarking by @Jeadie in #11006
• deps(ballista): pull in shuffle-on-object-store correctness fixes (datafusion-ballista PRs #42 + #43) by @phillipleblanc in #10919
• fix(kafka): seek to sidecar offsets via post_rebalance callback on restart by @ewgenius in #11007
• Propagate source comments into schema metadata by @lukekim in #10944
• feat(chbench-driver): better alignment to BenchBase by @sgrebnov in #11012
• fix: handle EXISTS/NOT EXISTS subqueries in federation analyzer by @sgrebnov in #10996
• Enable filter pushdown in spicepod defined UDTFs by @Jeadie in #11004
• Refactor spice dataset configuration command by @Jeadie in #10999
• fix: ensure HashJoinExec partition counts match when join input is statically empty by @phillipleblanc in #11025
• feat(chbench): Improve OLTP throughput and reduce PostgreSQL CDC overhead by @sgrebnov in #11018
• feat(cluster): add distributed query observability metrics by @phillipleblanc in #10990
• fix: delegate truncate in PolyTableProvider to inner write provider by @claudespice in #11036
• Remove default runtime features - enable explicitly in spiced by @phillipleblanc in #11037
• fix: preserve field and schema metadata in Vortex physical schema calculation by @claudespice in #11013
• Expose metadata descriptions via PostgreSQL UDFs by @lukekim in #11032
• fix: route Turso WAL checkpoint through native turso connection (fixes #10657) by @claudespice in #11048
• fix: add missing truncate delegation and guards for wrapper TableProviders by @claudespice in #11014
• Update DataConnector statuses by @krinart in #11052
• remove redundant readonly checks by @Jeadie in #10975
• Fix Unity Catalog connector compatibility with OSS Unity Catalog by @ewgenius in #11026
• refactor(cdc): reduce CDC sub-batch splits for interleaved upsert/delete workloads by @sgrebnov in #11051
• feat(cayenne): allow inline writes with pending deletions (deletes/upserts) by @sgrebnov in #11031
• fix(sql-tool): defer read-only gate to caller's API key role by @phillipleblanc in #11040
• fix: map Gemini Recitation finish reason to ContentFilter by @claudespice in #11046
• feat(cayenne): fast-path CDC deletes by extracting PK values from filters by @sgrebnov in #11049
• fix(cluster): gate scheduler readiness on executor partition loads by @phillipleblanc in #10992
• fix(snowflake): enforce function deny-list in federation pushdown by @claudespice in #11057
• perf(cdc): Last-write-wins dedup in group_into_sub_batches to reduce sub-batch splits by @sgrebnov in #11059
• [Bug] Timing between reconnect and AllocateInitialPartitions leaves connection without flight_sql_client by @Jeadie in #10805
• Define 'trait QueryEngine' to refactor runtime crate by @Jeadie in #11028
• fix(snowflake): apply Spice function deny-list in extracted connector crate by @claudespice in #11071
• perf(cayenne): keep CDC upsert PK keysets resident to avoid per-batch full-table rebuilds by @lukekim in #11074
• fix(postgres-replication): emit recovery log + reduce reconnect-warn volume by @claudespice in #11084
• fix metadata on search indexing by @Jeadie in #11080
• perf(cayenne): scale CDC inline flush caps with memory + storage class by @lukekim in #11087
• feat(cayenne): merge-on-read position deletes for PK upsert tables + memory-pool accounting by @lukekim in #11085
• Support tuple-IN composite PK extraction in Cayenne delete fast-path by @sgrebnov in #11093
• feat(cluster): report per-executor table statistics so distributed JoinSelection can size joins by @phillipleblanc in #11089
• feat(cluster): NDV-aware executor stats so CDC q18 join swap fires by @phillipleblanc in #11098
• Improve HTAP replication diagnostics on non-convergence by @sgrebnov in #11100
• Normalize DataType::Null to Int32 in acceleration schema for duckdb by @krinart in #11062
• Fix debezium schema evolution support by @ewgenius in #11095
• feat(cayenne): incremental write-path executor statistics for distributed join sizing by @phillipleblanc in #11104
• fix(cache): periodic moka maintenance to drain invalidation predicates (#11077) by @phillipleblanc in #11106
• fix: validate Snowflake account identifiers and auth config by @Jeadie in #11024
• fix: trace external mcp server tool calls by @ewgenius in #11058
• Remove unwrap from test code; drop clippy::unwrap_used suppressions by @phillipleblanc in #11108
• Upgrade to DuckDB 1.5.3 + statically link the VSS (HNSW) extension by @sgrebnov in #11107
• Fix fetched_at for HTTP connector by @Jeadie in #11116
• fix(search): propagate LIMIT to base TableScan in VectorScanTableProvider (fixes #8368) by @claudespice in #11124
• feat(runtime): add spicebench feature to register the Cayenne catalog connector by @phillipleblanc in #11122
• fix(cayenne): tombstone inline-checkpointed rows on upsert to prevent duplicate PKs by @sgrebnov in #11129
• Remove possibility of a deadlock in RuntimeStatus by @krinart in #11114
• Fix Windows build: vendored-vss duckdb-rs + adapt to table-providers mongodb API by @phillipleblanc in #11140
• localpod: synchronize child refreshes when parent uses in-memory (arrow) accelerator by @phillipleblanc in #11139
• Add datasets dimension to the query_executions metric by @phillipleblanc in #11138
• fix(spiceai): keep correlated subqueries out of JOIN ON for Spice Cloud federation by @phillipleblanc in #11143
• fix(duckdb): normalize timestamp columns to microsecond precision (fixes #10627) by @claudespice in #11145
• feat(cayenne): sharded parallel Vortex encode with key/time clustering by @lukekim in #11144
• fix(cluster): prevent DoPut write pipeline self-deadlock under ingest backpressure by @phillipleblanc in #11160
• feat(chbench): configurable HTAP concurrency, DuckDB query overrides, and OLTP rate control by @sgrebnov in #11162
• fix(http): preserve non-JSON response rows instead of crashing nested decomposition (fixes #11155) by @claudespice in #11161
• Use declared schema in DynamoDB/MongoDB/Debezium by @krinart in #11066
• fix(cluster): prevent partitioned datasets from staying Refreshing by @phillipleblanc in #11157
• fix(runtime): don't list postgres as a valid accelerator engine when postgres-accel is disabled by @sgrebnov in #11169
• fix(spark): recover stale or broken Spark Connect sessions on failure by @lukekim in #11171
• fix(secrets): don't abort secret lookup precedence walk on a failing store by @phillipleblanc in #11175
• feat(cayenne): bound aggregate write concurrency, conservative defaults, and write/read observability by @lukekim in #11170
• feat(unity_catalog): support Unity Catalog credential vending for Delta Lake tables by @phillipleblanc in #11180
• fix(secrets): keep failed secret stores registered so lookups report the init root cause by @phillipleblanc in #11181
• fix(debezium): sign-extend minimal-width base64 decimals instead of zero-padding by @claudespice in #11184
• fix(deps): update hickory-resolver to 0.26 (evicts hickory-proto 0.25.x) by @phillipleblanc in #11183
• refactor(secrets): derive secret store metadata from a single registry table by @phillipleblanc in #11188
• perf(cayenne): cut CDC replication lag on hot upsert tables by @lukekim in #11191
• feat(cayenne): async inline-fallback (per-tombstone published flag) + 64c/256GB tuning by @lukekim in #11194
• Add HTTP connector mTLS support by @lukekim in #11127
• feat(snowflake): push AT TIME ZONE as CONVERT_TIMEZONE and pin session to UTC by @lukekim in #11190
• feat(cdc): make cdc_max_coalesce_age_ms a real apply-loop linger by @sgrebnov in #11196
• feat(cayenne): delta-write encoding levels (cayenne_delta_encoding, default auto) + make compression_strategy=zstd real by @lukekim in #11199
• Add NSQL context endpoint by @lukekim in #11075
• fix(federation): respect the Spice function deny-list across all SQL connectors; dialect-aware DuckDB pushdown by @claudespice in #11186
• fix: surface unknown/applied cayenne_* runtime.params at startup (fixes #10970) by @claudespice in #11133
• perf(cayenne): plain-fsync ordering tier on the staged-commit hot path by @lukekim in #11198
• fix(kafka): decode JSON payloads to Arrow directly — fixes lossy Decimal128 + removes double-parse (#11192) by @claudespice in #11207
• feat(cayenne): self-tuning accelerator — hardware + schema + closed-loop adaptive (auto/adaptive modes) by @lukekim in #11213
• perf(cayenne): CDC throughput — SF-100 @10K txn/s toward <5s lag + 5K QPH by @lukekim in #11206
• fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc in #11226
• Improve Cayenne adaptive tuning and schema safety by @lukekim in #11237
• feat: Add cayenne_file_pruning param by @peasee in #11239
• Debezium connector - handle tombstone messages in kafka topic, with schema evolution enabled by @ewgenius in #11197
• feat(cayenne): broadcast small-dimension joins to executors by @phillipleblanc in #11245
• fix: scope request context across the managed query runtime by @phillipleblanc in #11253
• fix: Strip inference columns from table schema on query by @peasee in #11251
• fix(flightsql): don't drop un-pushed FilterExec predicates in distributed pushdown rules by @claudespice in #11256
• feat(postgres): share one replication slot across multiple changes-mode datasets by @phillipleblanc in #11255
• Upgrade to DataFusion v53.1, Arrow v58.3, Vortex v0.74, and dependencies by @lukekim in #11118
• feat(connectors): support file_format: vortex everywhere parquet is supported by @lukekim in #11282
• perf(cayenne): metadata-only publish — drop per-key insert records on upsert commit by @lukekim in #11260
• fix(kafka): harden fetch_latest_message for multi-partition topics by @ewgenius in #11285
• perf(cayenne): bound mem-tier checkpoint churn + O(1) per-scan deletion view by @lukekim in #11249
• fix(udfs): length-prefix digest_many values so column boundaries can't collide (fixes #11272) by @claudespice in #11288
• fix: restore federation deny-list enforcement regressed by the DataFusion 53 upgrade by @claudespice in #11294
• fix(postgres): recover unchanged-TOAST columns from the old tuple; classify walsender contention as transient by @phillipleblanc in #11293
• feat: deepen extended schema inference and wire it into cayenne compaction sharding/sorting by @lukekim in #11284
• perf(cayenne): in-memory CDC tier follow-ups + write-phase observability by @lukekim in #11278
• Support per-dataset CDC tunable overrides by @sgrebnov in #11295
• feat(cayenne): harden adaptive auto-tuner (controller hygiene, mem-tier actuator, delete/burst signals, single opt-in) by @lukekim in #11302
• fix(cayenne): scan inlined-view capture starvation under sustained CDC (analytical QPH) by @lukekim in #11299
• fix(vortex): don't row-evaluate hash-join dynamic filters in the scan by @sgrebnov in #11307
• fix(deps): bump rust-postgres crates (RUSTSEC-2026-0178/0179) by @lukekim in #11313
• fix: strict validation of Postgres CDC params instead of silent defaults (fixes #11274) by @claudespice in #11304
• fix(duckdb): always quote on-refresh sort columns so reserved-word names don't break refresh by @claudespice in #11305
• feat(flightsql): fall back to original connection when endpoint location is unreachable by @melks in #11287
• perf(cayenne): light-encode transient staged CDC deltas by @lukekim in #11311
• feat(acceleration): widening-only schema evolution via on_schema_change by @lukekim in #11261
• deps(vortex): bump pin to spiceai-53 HEAD — intra-file decode split + per-execution kernel cache by @lukekim in #11314
• feat(github): enhance GitHub component validation and error handling by @lukekim in #11259
• feat(cayenne): goal-driven adaptive tuning toward operator SLOs (lag, freshness, query latency, QPH) by @lukekim in #11310
• fix(cayenne): broadcast-join rewrite must bail on ambiguous columns, NULL-equal joins, and residual filters by @claudespice in #11252
• Add Cayenne maintained aggregates by @lukekim in #11235
• perf(cayenne): single-hash composite deletion filter via KeyDeletionIndex::get_batch by @phillipleblanc in #11325
• fix(Vortex): decline only the InList membership conjunct of hash-join dynamic filters by @sgrebnov in #11335
• fix: scope SQL UDF arg inlining to args-table columns (fixes #11273) by @claudespice in #11337
• fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc in #11339
• fix(cayenne): ref-count in-flight scans so GC can't delete Vortex files mid-read by @phillipleblanc in #11321
• fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc in #11342
• feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc in #11347
• fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc in #11345
• fix: detect dual-write accelerated tables behind the metadata-enrichment wrapper by @claudespice in #11351
• feat(cayenne): incremental seq-prefix bake — shrink the merge-on-read deletion index by @lukekim in #11326
• fix(adbc): prevent Spice-specific UDFs from being pushed down to ADBC sources by @krinart in #11297
• fix: Query Redshift schema details from svv_redshift tables by @peasee in #11362
• perf(cayenne): tune Turso connection PRAGMAs + jitter metastore retries by @lukekim in #11359
• Upgrade to DataFusion 54 by @sgrebnov in #11360
• feat(runtime): dedicated CDC-apply tokio runtime + per-runtime tokio metrics by @lukekim in #11370
• fix(cayenne): spill oversized hash joins via sort-merge to avoid OOM by @lukekim in #11371
• fix(cayenne): honor cayenne_metastore: turso for partitioned tables and the dataset checkpoint by @phillipleblanc in #11365
• perf(cayenne): in-RAM scan parallelism, query admission control, skip no-op deletion encode, sound scan output_ordering by @lukekim in #11332
• fix(catalog): restore DDL after DataFusion 54 broke transparent catalog-provider downcasts by @phillipleblanc in #11375
• feat(flightsql): infer schema via SELECT * LIMIT 1 when GetTables is unimplemented by @melks in #11286
• fix(cayenne): Utf8View read schema avoids hash-join offset overflow by @lukekim in #11379
• feat(cluster): support distributed (Ballista) scans of Iceberg tables by @phillipleblanc in #11378
• feat(optimizer): cost-based left-deep join reordering for Cayenne acceleration by @sgrebnov in #11377
• cli - fix service-account auth in spice cloud * commands by @ewgenius in #11316
• fix: Support external Redshift table schema inference and Hive external type parsing by @peasee in #11399
• feat(llms): native GLM support — opt-in flash-attn + surface reasoning_content by @lukekim in #11400
• feat(s3_vectors): paginate QueryVectors for topK up to 10,000 by @bjchambers in #11405
• fix: surface .env parse errors with line numbers instead of silently skipping by @Oxygen56 in #11306
• fix(status): probe metrics endpoint over https when TLS is enabled by @phillipleblanc in #11393
• fix(mcp): record task_history spans for tool calls proxied through /v1/mcp by @phillipleblanc in #11397
• Properly handle date_trunc in BigQueryDialect by @krinart in #11416
• fix(snowflake): honor column scale in Int64 timestamp arm and cast TIME by @claudespice in #11418
• fix: /v1/queries chunk row_offset uses cumulative offset, not chunk_index * chunk_size (fixes #11271) by @claudespice in #11398
• feat(cayenne): incremental retraction for maintained aggregates + anchor bench by @lukekim in #11389
• feat(cluster): support distributed (Ballista) scans of Iceberg catalog tables by @phillipleblanc in #11419
• Simplify chat/responses models by @Jeadie in #10997
• feat(llms): distributed tensor-parallel GLM inference via mistral.rs ring backend by @lukekim in #11406
• Optimize Flight streaming for large result sets by @lukekim in #11420
• fix(deps): evict rustls 0.21 / rustls-webpki 0.101.7 (GHSA-82j2-j2ch-gfr8) by @phillipleblanc in #11428
• feat(cayenne): in-memory CDC intra-apply sharding (PK-hash shards) by @lukekim in #11421
• fix(cayenne): shard CDC upserts with pending deletions so the N>1 slot-ack advances by @lukekim in #11445
• fix(cluster): keep built-in avg over Spark avg (distributed aggregate state schema mismatch) by @phillipleblanc in #11434
• feat(views): support params.file_format for embedding chunking by @Jeadie in #11424
• fix: offload blocking sync calls off the primary async runtime by @phillipleblanc in #11435
• fix(udfs): rebind dot_product alias to Spice's inner_product on DataFusion 54 by @lukekim in #11443
• feat(secrets): add full-fidelity reference iteration and a resolution-status API by @phillipleblanc in #11195
• fix: Deny unsupported array functions for Postgres pushdown by @peasee in #11450
• fix(cli): spice query honors --http-endpoint instead of failing on a Flight connect by @phillipleblanc in #11452
• fix(cayenne): coordinate query-pool + in-memory CDC tier budgets to prevent adaptive OOM by @lukekim in #11449
• Surface mTLS config in Kafka data connector by @v1gnesh in #11372
• feat(secrets): check and report secret references at startup by @phillipleblanc in #11457
• Default cayenne_force_view_types to false by @sgrebnov in #11459
• fix: resolve table-reference qualification in results-cache invalidation (fixes #11266) by @claudespice in #11460
• feat(cayenne): metadata aggregate pushdown — fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics by @bjchambers in #11414
• fix(search): restore numeric trunc and fix SortPreservingMergeExec planning error (DF54) by @Jeadie in #11415
• fix(cluster): route Ballista shuffle/temp to the data PVC by @phillipleblanc in #11454
• fix(search): default Elasticsearch kNN candidate pool to 1000 instead of 10 (fixes #11264) by @claudespice in #11467
• feat(acceleration): add on_schema_change drop_and_recreate policy by @lukekim in #11462
• feat(cayenne): predicate-aware maintained aggregates serve filtered analytical queries from the CDC delta by @lukekim in #11458
• feat(cluster): shared Ballista job state with scheduler failover by @phillipleblanc in #11436
• feat(cayenne): extend HLL NDV sketching to string and date columns by @bjchambers in #11468
• fix(search): persist character chunk offsets so search snippets aren't shifted/garbled (fixes #11269) by @claudespice in #11479
• feat(cayenne): storage-aware adaptive CDC tuning — calibration probe, IMDS, I/O-cliff fast path, infeasible-SLO feedback by @lukekim in #11463
• fix(cayenne): LIMIT N under-delivers on key-deletion tables by @lukekim in #11490
• fix(cayenne): live/tier-accurate join build-side stats (merge-on-read deletes + never-shrink NDV) by @lukekim in #11496
• perf(cayenne): kernel-space I/O hygiene — compaction fadvise + staged-commit barrier reduction by @lukekim in #11495
• feat(cayenne): feed maintained-aggregate IVM from the staged-disk CDC path by @lukekim in #11491
• feat(cayenne): global adaptive-tuning SLOs with per-dataset overrides; QPH global-only by @lukekim in #11497
• Update search snapshots by @sgrebnov in #11473
• fix: Support reading column types longer than 128 chars in Redshift by @peasee in #11500
• fix(cluster): distributed (Ballista) query-execution config + scheduled SF10 bench by @phillipleblanc in #11478
• fix(http): retry transient response-body read failures; de-flake backoff test by @claudespice in #11482
• Re-land orphaned deletion-vector cleanup during retention deletes by @lukekim in #11501
• fix(cayenne): re-upsert over a pending delete tombstone records an insert-record (overwrite resurrection) by @bjchambers in #11469
• fix: Flight DoPut silently dropped client batches on early sink completion by @claudespice in #11507
• Upgrade OpenTelemetry to 0.32 and reqwest to 0.13 by @phillipleblanc in #11506
• fix(queries): run async /v1/queries jobs under the submitting request context by @phillipleblanc in #11505
• fix(cayenne): restore append-only current-snapshot compaction by @Jeadie in #11439
• perf(cayenne): orphaned deletion-vector cleanup off the write path, behind a knob by @bjchambers in #11517
• fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov in #11503
• fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim in #11514
• fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov in #11515
• fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim in #11516
• feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers in #11523
• Fix s3 vectors API by @krinart in #11536
• fix: Placeholder table initialization lock swap by @peasee in #11540
• chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc in #11544
• fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie in #11546
• Bump datafusion to include spiceai/datafusion#181 by @Jeadie in #11563
• Update deny.toml by @krinart in #11571
• Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie in #11598

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.1.0

Spice v2.0-rc.5 is now available! 🔥

v2.0.0-rc.5 is the fifth release candidate for advanced testing of v2.0, building on v2.0.0-rc.4.

This release completes the mTLS implementation across server endpoints and outbound connectors, adds MongoDB Change Streams and durable Kafka offset persistence as new CDC sources, expands DML write-back to PostgreSQL, Snowflake, and Arrow, promotes DuckLake to Beta, introduces user-defined functions, on-demand dataset loading, unified query cancellation, dynamic HTTP request headers and subquery-driven request parameters, provider-aware LLM prompt caching, and a long list of Cayenne performance improvements.

Highlights in this release candidate include:

• Spice Cayenne — CDC throughput, compaction and scan caching, synchronized partition commits, join filter propagation, parallel Vortex writes, lock-free deletion caches
• Mutual TLS (mTLS) — TLS cert hot-reload, public mTLS for HTTP and Flight (channel + identity modes), mTLS client certs for FlightSQL and Spice.ai connectors
• MongoDB Change Streams — native real-time CDC for MongoDB, no Debezium or Kafka required
• Kafka CDC offsets — offsets persisted in sidecar tables for durable, resumable Kafka CDC
• PostgreSQL DML — INSERT, UPDATE, DELETE write-back on PostgreSQL datasets
• Snowflake DML — INSERT, UPDATE, DELETE write-back on Snowflake datasets
• Arrow Primary Key Upserts — native upsert path using primary key matching
• DuckLake promoted to Beta — with INSERT support on catalog tables
• User-Defined Functions — define SQL UDFs in spicepods, plus remote UDFs over HTTP (Spice.ai Enterprise)
• Spatial SQL UDFs — optional geospatial UDFs (ST_*) for geometry workloads
• On-Demand Dataset Loading — datasets can be deferred and loaded on first reference
• Unified Query Cancellation — Ctrl-C and HTTP request cancellation propagate across all execution paths
• Dynamic HTTP Connector — pass-through request headers, subquery-driven params, and JSON schema decomposition
• HTTP Rate-Control persistence — rate-limit state persisted in object storage across restarts
• refresh_mode: snapshot — point-in-time snapshot acceleration with SQLite/Turso WAL flushing
• Storage-profile accelerator tuning — accelerators auto-tune defaults based on local SSD, EBS-class disk, or tmpfs
• Provider-Aware LLM Prompt Caching — automatic prompt caching for OpenAI-compatible providers that support it
• Responses API — support across all model providers with streaming response.output_text.delta, plus Authorization: Bearer header support

What's New in v2.0.0-rc.5​

Cayenne Improvements​

Significant performance work across Spice Cayenne-backed catalogs and accelerators.

• Ingest throughput: End-to-end improvements to CDC ingest, background compaction, and a new scan-result cache for hot reads; parallel Vortex partition writes; lock-free deletion caches with bloom-prefiltered probes; background retention with CDC pipelining; SQLite metastore pool scaled to 32 for high-concurrency mutation workloads.
• Data inlining: Small writes are serialized as Arrow IPC and committed directly into the Cayenne metastore (cayenne_inlined_data), bypassing the staged Vortex write path for low-latency ingest. Inline upserts atomically rewrite existing inline rows instead of emitting side delete markers, and inline data remains query-visible via an in-memory union scan with a generation-keyed decode cache. Inline rows are checkpointed to Vortex when row, segment, or byte thresholds are reached. Defaults are refresh-mode aware: inline writes are enabled by default for high-frequency caching, changes, and fast append workloads and disabled for full, snapshot, and slower append.
• Query planning: Join filter propagation across equi-join keys (gated behind runtime.params.cayenne_filter_propagation), range fallback for large join filters, hot-path clone elimination, and IN-list rewrites for large filter lists.
• Correctness: Synchronized partition commits across partitions, correct NULL-sentinel handling for nullable partition expressions (e.g. bucket(N, col)), Vortex panic fix on highly compressible data, and live reads through expired protected snapshots.
• Catalog and platform: Refresh-mode-aware compaction defaults, rejection of non-distributed Cayenne catalog configurations, and a vendored Vortex DataFusion integration for faster iteration on the Cayenne planner.

Mutual TLS (mTLS)​

Spice.ai Enterprise feature. See Enterprise Security.

Spice now supports full mutual TLS for both HTTP and Arrow Flight endpoints.

TLS cert hot-reload (#10727): The Spice runtime watches for SIGHUP and reloads TLS certificates without restarting, enabling cert rotation with zero downtime.

Public mTLS for HTTP and Flight (#10753): Two client_auth_mode values control how the server handles client certificates:

• request — optional mTLS: the server requests a client cert but accepts connections without one (useful for migration windows).
• required — strict mTLS: the server requires a valid client cert signed by the configured CA.

mTLS client certs for FlightSQL and Spice.ai connectors (#10764): Outbound connections from the FlightSQL and Spice.ai data connectors can now present client certificates for mutual authentication with upstream services.

Example configuration:

runtime:
  tls:
    enabled: true
    certificate_file: /etc/spice/tls/server.crt
    key_file: /etc/spice/tls/server.key
    client_auth_mode: required
    client_auth_ca_file: /etc/spice/tls/client-ca.crt

MongoDB Change Streams​

MongoDB datasets configured with refresh_mode: changes now stream changes from MongoDB Change Streams into any local accelerator (#10813), providing real-time CDC without Debezium or Kafka.

Example configuration:

datasets:
  - from: mongodb:my_collection
    name: my_collection
    params:
      host: my-cluster.mongodb.net
      db: mydb
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: changes

CDC Improvements​

See Change Data Capture (CDC) for an overview of CDC in Spice.

• Kafka CDC offset persistence (#10823): Kafka CDC offsets are persisted in sidecar tables for durable, resumable streams. On restart or failover, Spice resumes from the last committed offset.
• Pipelined CDC ingestion (#10676): Source reads overlap with batch apply, with additional batching, envelope coalescing, and nullability propagation improvements across the apply pipeline.
• Debezium schema evolution fix (#10144): Schema changes in Debezium-sourced datasets no longer break dataset initialization on reload (fixes #9782).

PostgreSQL DML Support​

The PostgreSQL data connector now supports write-back via INSERT, UPDATE, and DELETE operations (#10446). Combined with the existing read-side federation, PostgreSQL-backed datasets can serve as full read/write tables. The PostgreSQL Catalog connector additionally exposes foreign-key metadata for NSQL and query planning (#10849).

Snowflake DML Support​

The Snowflake data connector now supports write-back via INSERT, UPDATE, and DELETE operations (#10747), complementing its existing read capabilities.

Arrow Primary Key Upserts​

Arrow-accelerated tables now support native upsert operations using primary key matching (#10749), providing efficient update-or-insert semantics for in-memory datasets.

DuckLake Promoted to Beta​

The DuckLake Catalog and Data Connector are promoted to Beta quality (#10743).

DuckLake catalog tables with read_write access now support INSERT operations (#10744), enabling full read/write workflows against DuckLake-backed catalogs. The DuckLake connector also gains a series of correctness fixes for downcast, module registration, schema discovery, and S3 credentials (#10650).

User-Defined Functions​

Spice now supports user-defined functions (UDFs) as a first-class spicepod component (#10571), letting you define reusable SQL functions in the spicepod or invoke remote functions over HTTP. The runtime also gains table user functions with HTTP server gating (#10675).

A security fix closes a remote-UDF SSRF vector (#10757).

Spatial SQL UDFs​

Spice now ships an optional set of geospatial SQL UDFs (ST_*) for geometry workloads (#10833). The functions are gated behind a build feature and can be invoked from any SQL surface.

On-Demand Dataset Loading​

Datasets can now be marked for on-demand loading (#10629). Deferred datasets are registered with a declared schema at startup (#10669) and only fully resolve when first referenced, reducing startup time and memory footprint for spicepods with many seldom-used datasets.

Spicepods also gain columns[].type and columns[].nullable (#10661) with a lenient type parser for declaring schemas inline.

Unified Query Cancellation​

All query execution paths — HTTP, Flight, FlightSQL, MCP, and internal — now honour a unified cancellation signal (#10390). When a client disconnects, presses Ctrl-C in the REPL, or cancels an in-flight HTTP request, the corresponding query is cancelled end-to-end, freeing resources promptly.

Dynamic HTTP Connector​

The HTTP data connector gains dynamic request headers parameterised from query predicates (#10604), subquery-driven request parameters for fan-out queries (#10636), HTTP response metadata as queryable columns via JSON schema decomposition (#10679), no-limit pagination (#10673), and shared rate-control across HTTP-based connectors using the same backend host (#10648).

HTTP Rate-Control Persistence​

The HTTP rate-control state (per-endpoint throttle counters) is now persisted in object storage (#10697), ensuring rate limits survive restarts and are consistent across replicas. Rate-control metrics now use an origin label rather than the connector name for cleaner aggregation (#10689).

The metrics HTTP endpoint (/metrics) is also independently rate-limited (#10162) to prevent scraping from impacting query serving.

refresh_mode: snapshot​

Spice.ai Enterprise feature. See Acceleration Snapshots.

A new refresh_mode: snapshot provides point-in-time snapshot acceleration (#10651), with SQLite and Turso WAL flushing and a Cayenne metastore slice integration so accelerated readers see a consistent snapshot while writes continue.

Storage-Profile Accelerator Tuning​

Acceleration configs gain a new storage_profile field (#10913) with values auto (default), local_ssd, ebs, and tmpfs. Under auto, the runtime detects whether the acceleration store is backed by local SSD, EBS-class network disk, or tmpfs, and applies storage-aware defaults across DuckDB, partitioned DuckDB, SQLite, Turso, and Cayenne file-mode accelerators. Explicit per-accelerator parameters always override the profile defaults.

Provider-Aware LLM Prompt Caching​

LLM calls automatically use provider-aware prompt caching (#10645) when the configured model provider supports it (e.g., Anthropic, OpenAI). System prompts and tool descriptions are marked for caching so repeated invocations within the cache window reuse the provider-side cached prefix, reducing latency and cost.

A new searchable registry mode for LLM tools (#10647) lets agents discover tools by semantic search rather than enumerating all tools in the system prompt, which scales to large tool inventories.

Responses API Improvements​

The Responses API is now supported across all configured model providers (#10724). Streaming delta events via response.output_text.delta are also supported (#10828). The runtime now also accepts Authorization: Bearer headers in addition to x-api-key, bumps async-openai, and stops populating FunctionToolCall.id so OpenAI-compatible servers can assign the ID themselves (#10911).

Distributed Cluster Improvements​

Spice.ai Enterprise feature. See High Availability.

• Per-request executor readiness gate (#10860): /v1/ready on schedulers waits for a configurable quorum of executors before returning healthy, enabling proper rolling deployments.
• Ballista S3 shuffle reads under cluster mode (#10910): The shuffle reader builds its S3 client from the executor pod's environment, matching the writer. Async queries with runtime.params.shuffle_location: s3://... now complete instead of failing with AccessDenied on shuffle fetches.
• Flattened scheduler config (#10450): runtime.scheduler.partition_management.* fields are flattened directly onto runtime.scheduler and renamed under the canonical "partition assignment" terminology. See Breaking Changes.

Caching & Search​

Improvements across Caching and Search:

• Per-principal cache namespacing (#10702): SQL, search, and caching-accelerator caches are now namespaced per authenticated principal, so cached results never cross identity boundaries.
• DuckDB HNSW vector indexes (#10695, #10674, #10668): DuckDB-accelerated views support HNSW vector indexes for vector search, vector search SQL is rewritten to activate HNSW_INDEX_SCAN, and HNSW indexes are preserved across data refresh.

Security Improvements​

See Authentication and TLS for configuring Spice security.

• API key timing-position leak and remote-UDF SSRF (#10757): Closed a timing-based position-disclosure leak in API key comparison and blocked SSRF via remote UDF endpoint parameters.
• Configurable allowed_hosts for MCP (#10638): MCP servers can be restricted to an explicit allowlist of upstream hosts.

SQL, Query, and Developer Experience​

See the SQL Reference for the full SQL surface area.

• SQL REPL expanded view (#10797): Toggle \x in the REPL for a vertical key-value layout on wide result sets.
• FlightSQL Substrait plan support (#10761): The Spice runtime now implements CommandStatementSubstraitPlan, enabling clients that submit plans as Substrait-encoded protobuf.
• MCP auth for streamable HTTP tools (#10927): Streamable HTTP MCP tools support native authentication via mcp_auth_token and mcp_headers, both with full Spice secret expansion.
• Elasticsearch FTS engine config and index lifecycle (#10672): Direct FTS engine configuration plus index lifecycle and ingestion controls for the Elasticsearch connector.
• Self-hosted Spice connector (#10546): Connect Spice to another self-hosted Spice runtime as a federated source.

Connector Bug Fixes​

Notable correctness fixes across the Data Connectors: DynamoDB Streams retry on transient errors (#10794) and typed-NULL handling in DML (#10511); ScyllaDB physical filter pushdown disabled to fix incorrect results (#10772); MSSQL TOP N pushdown for non-nullable sort columns (#10621); DuckLake include filter applied (#10738); DuckDB DELETE/UPDATE on full and caching refresh modes (#10632); checked arithmetic for Turso integer-millis and timestamp-to-nanosecond conversions (#10786, #10666); and Flight GetFlightInfo/DoGet schema parity (#10864). See the Changelog for the full list.

Dependency Updates​

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

Flattened runtime.scheduler configuration (#10450): The nested runtime.scheduler.partition_management block has been flattened and renamed to use the canonical "partition assignment" terminology. Migrate as follows:

# Before
runtime:
  scheduler:
    partition_management:
      interval: 30s
      max_assignments_per_cycle: 16
      discovery_timeout: 10s

# After
runtime:
  scheduler:
    partition_assignment_interval: 30s
    max_assignments_per_interval: 16
    partition_discovery_timeout: 10s

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.5, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.5

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.5 image:

docker pull spiceai/spiceai:2.0.0-rc.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.5

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Enable DML support for PostgreSQL data connector by @phillipleblanc in #10446
• feat(postgres): support inline PEM sslrootcert by @claudespice in #10578
• Add foreign key metadata discovery to PostgreSQL Catalog by @sgrebnov in #10849
• Add Snowflake DML support by @lukekim in #10747
• Add MongoDB Change Streams support by @lukekim in #10813
• Add user-defined functions by @lukekim in #10571
• Add table user functions and gate HTTP servers by @lukekim in #10675
• feat: add on-demand dataset loading by @phillipleblanc in #10629
• feat(runtime): declared-schema deferred datasets by @phillipleblanc in #10669
• feat(spicepod, runtime): add columns[].type / nullable + lenient type parser by @phillipleblanc in #10661
• Replace external smb crate with internal SMB 3.1.1 client by @phillipleblanc in #10516
• Add unified query cancellation across all paths by @lukekim in #10390
• Add dynamic HTTP request headers by @lukekim in #10604
• feat(http): Support dynamic HTTP connector request params from subqueries by @lukekim in #10636
• feat(http): pass through HTTP metadata columns with JSON schema decomposition by @lukekim in #10679
• Add nolimit HTTP pagination max pages by @lukekim in #10673
• Add shared HTTP rate control for connectors by @lukekim in #10648
• Use origin label instead of name for HTTP rate control metrics by @lukekim in #10689
• fix(http): reject OR across different HTTP filter columns by @lukekim in #10625
• Add provider-aware LLM prompt caching by @lukekim in #10645
• Add searchable registry mode for LLM tools by @lukekim in #10647
• feat: refresh_mode: snapshot + SQLite/Turso WAL flush + Cayenne metastore slice by @phillipleblanc in #10651
• feat: per-principal cache namespacing for SQL/search/caching-accelerator by @lukekim in #10702
• Add self-hosted Spice connector support by @phillipleblanc in #10546
• Add Delta Lake Azure tenant parameter by @phillipleblanc in #10671
• Support OAuth2 client credentials in 'spice cloud login' by @ewgenius in #10586
• Add configurable allowed_hosts for MCP by @lukekim in #10638
• fix: make Helm chart probes configurable by @peasee in #10696
• Strip high-cardinality datasets dim from anonymous telemetry by @lukekim in #10711
• feat(elasticsearch): direct FTS engine config + index lifecycle and ingestion controls by @lukekim in #10672
• Add DuckDB HNSW vector index support for accelerated views by @sgrebnov in #10695
• Rewrite DuckDB vector search SQL to activate HNSW_INDEX_SCAN by @sgrebnov in #10674
• Fix DuckDB HNSW vector indexes lost after data refresh by @sgrebnov in #10668
• Fix DuckDB DELETE/UPDATE on full and caching refresh mode datasets by @phillipleblanc in #10632
• Fix DuckLake connector: downcast, module registration, schema discovery, and S3 credentials by @sgrebnov in #10650
• Fix federation pushing denied functions inside subqueries to remote engines by @phillipleblanc in #10692
• fix(caching): honour refresh_on_startup: always in caching mode by @phillipleblanc in #10594
• fix(iceberg): rebuild storage factory when Hadoop catalog scheme is inferred by @sgrebnov in #10601
• Pipeline CDC ingestion: overlap source reads with batch apply by @lukekim in #10676
• fix: add NULL check to CDC primary key extraction by @lukekim in #10684
• Properly handle nullability during CDC processing by @krinart in #10803
• Flatten scheduler config and rename partition management → partition assignment by @lukekim in #10450
• Improve NSQL UX and harden internal LLM tools by @lukekim in #10715
• Support Responses API across model providers by @lukekim in #10724
• Update xAI default model and handle Grok model retirements by @Jeadie in #10723
• Improve cli table layout by @krinart in #10725
• TLS cert hot-reload (mTLS plan M1) by @phillipleblanc in #10727
• Fix DuckLake catalog include filter being ignored by @phillipleblanc in #10738
• Promote DuckLake Catalog and Data Connector to Beta quality by @sgrebnov in #10743
• feat(ducklake): Support INSERT on catalog tables with read_write access by @sgrebnov in #10744
• perf(cdc): coalesce envelopes and overlap commits in apply pipeline by @lukekim in #10745
• feat: Allow full version tags in spicepod version by @peasee in #10748
• Add Arrow primary key upserts by @lukekim in #10749
• fix(snapshot): keep refresh_mode snapshot read-only by @phillipleblanc in #10752
• feat(tls): public mTLS for HTTP and Flight (channel + identity modes) by @phillipleblanc in #10753
• perf(cayenne): lock-free deletion caches with bloom-prefiltered probe by @lukekim in #10756
• fix(security): close API key timing-position leak and remote-UDF SSRF by @lukekim in #10757
• Fix 'wait_until_dependent_tables_are_ready' for catalogs by @phillipleblanc in #10758
• Fixes for views and resolved tables on 'spice refresh' CLI by @phillipleblanc in #10759
• Implement FlightSQL CommandStatementSubstraitPlan support by @lukekim in #10761
• feat(connectors): mTLS client cert support for flightsql and spiceai connectors by @phillipleblanc in #10764
• Allow arbitrary filenames when specifying spicepod path + kind validation by @krinart in #10777
• fix: ignore field metadata in schema compatibility check in index_table_scan by @Jeadie in #10778
• Display pushed-down limits in EXPLAIN TREE output by @lukekim in #10779
• fix: enable streaming append for Kafka with Cayenne accelerator by @lukekim in #10780
• fix: bound chunked-index intermediate batch size to prevent OOM by @phillipleblanc in #10783
• fix: label all columns in spice cloud metrics table output by @claudespice in #10784
• fix: use checked arithmetic for Turso integer-millis timestamp read path by @claudespice in #10786
• fix: use checked arithmetic in timestamp-to-nanosecond conversions by @claudespice in #10666
• Upgrade to DuckDB v1.5.2 by @sgrebnov in #10788
• Improve CDC ingestion performance by @lukekim in #10789
• Fix tool_search/tool_invoke spans by @lukekim in #10791
• Add Cayenne inline mutations and benchmark coverage by @lukekim in #10792
• Ensure we always resolve table names in distributed mode/metadata by @Jeadie in #10793
• Remove permanent errors from DynamoDB Streams by @krinart in #10794
• Add expanded view mode for wide table display in SQL REPL by @lukekim in #10797
• Fix Cayenne CDC schema mismatch error by @sgrebnov in #10800
• Executors should create catalog tables on join by @Jeadie in #10807
• Add compressed file support for listing connectors by @lukekim in #10809
• Improve Cayenne mutation, scan, and inline memtable scaling by @lukekim in #10811
• Add range fallback for large join filters by @lukekim in #10816
• Improve Cayenne join filter pushdown by @lukekim in #10818
• Synchronize Cayenne partition commits across partitions by @phillipleblanc in #10819
• fix: Deny nondistributed cayenne catalog by @peasee in #10821
• Enable parallel Cayenne Vortex writes by @lukekim in #10822
• Expand Arrow type handling in formatting and Elasticsearch by @lukekim in #10825
• Add response.output_text.delta to responses API by @krinart in #10828
• feat(cayenne): add join filter propagation and no-spill Q21 planning by @lukekim in #10840
• Upgrade Turso to v0.6.0 by @sgrebnov in #10843
• feat(cli): add spice feedback command to open community Slack by @lukekim in #10856
• Upgrade iceberg to v0.9.1 by @sgrebnov in #10859
• feat(cluster): per-request executor readiness gate on /v1/ready by @phillipleblanc in #10860
• fix: Require dim-side statistics for CayennePropagateFilterAcrossEquiJoinKeys by @sgrebnov in #10863
• fix: Debezium schema evolution breaks dataset init on reload by @claudespice in #10144
• fix(mssql): Push topK limit to SQL Server for non-nullable sort columns by @Jeadie in #10621
• fix(ScyllaDB): disable physical filter pushdown by @sgrebnov in #10772
• fix: handle typed NULLs and prevent overflow in DynamoDB DML type conversions by @krinart in #10511
• fix: use InsertOp::Overwrite in DynamoDB bootstrap scan_and_overwrite_accelerator by @krinart in #10639
• Improve DynamoDB Bootstrap performance by @krinart in #10616
• fix: preserve field and schema metadata in Vortex type transformation by @lukekim in #10628
• fix: GH connector - explicitly use AWS LC RS crypto provider for jwt by @phillipleblanc in #10619
• fix: add snapshot mode guards to delete_from/update and delegate DML in SwappableTableProvider by @phillipleblanc in #10685
• Persist HTTP rate-control state in object storage by @lukekim in #10697
• Rate limit metrics HTTP endpoint by @lukekim in #10162
• feat(geo): add optional spatial SQL UDF support by @lukekim in #10833
• feat(cayenne): CDC throughput, compaction, scan caching, and benchmarks by @lukekim in #10852
• fix(cayenne): fix Vortex panic on highly compressible data by @sgrebnov in #10855
• fix(cayenne): Read live protected snapshots after cleanup grace period by @sgrebnov in #10901
• fix: Disable Cayenne HashJoin rewriter optimizer by @sgrebnov in #10882
• Fix GetFlightInfo vs DoGet Flight Schema by @krinart in #10864
• fix(search): preserve column casing in /v1/search primary key plumbing by @claudespice in #10909
• fix(object-store): dedupe s3 url style auto-detection log by @phillipleblanc in #10898
• Improve Spice CLI manifest editing and direct command modes by @lukekim in #10815
• Persist Kafka CDC offsets in sidecar tables by @lukekim in #10823
• feat(task-history): record Ballista stages for distributed queries by @phillipleblanc in #10831
• Add '#[deny(clippy::missing_trait_methods)]' to wrapper/delegation trait impls by @Jeadie in #10795
• Optimize Cayenne catalog maintenance paths by @lukekim in #10904
• Centralize DuckDB settings for accelerator by @ewgenius in #10895
• deps(ballista): bump to 47e2b494 to fix S3 shuffle reads under cluster mode by @phillipleblanc in #10910
• Authorization header + Bump async-openai + responses_adapter fix by @krinart in #10911
• Tune accelerators by storage profile by @lukekim in #10913
• feat: add dataset-level on_schema_change config by @lukekim in #10908
• Handle NULL sentinel for nullable partition expressions by @Jeadie in #10880
• fix: Remove Cayenne Catalog from catalog registration by @peasee in #10914
• Add catalog name to foreign key metadata in postgres catalog by @Jeadie in #10917
• Cayenne perf: eliminate redundant clones, PK point-lookup fanout fix, IN-list rewrite + microbench coverage by @lukekim in #10916
• fix(turso-shared): retry on Turso BEGIN CONCURRENT "Write-write conflict" by @lukekim in #10946
• Vendor Vortex DataFusion for Cayenne by @lukekim in #10933
• perf(cayenne): background retention + enable CDC pipelining for retention-configured tables by @lukekim in #10936
• feat(cayenne): scale metastore pool to 32 + vs_duckdb_scaling benches (1→128 concurrency, sqlite + turso lanes) by @lukekim in #10943
• feat(mcp): support auth for streamable HTTP tools by @phillipleblanc in #10927
• Explicit error if v1/search requests a table without search index by @Jeadie in #10968
• Fix spicepod loading failure when directory name contains dots by @sgrebnov in #10958
• Extend append tests with arrow engine configurations by @sgrebnov in #10959
• Remove dataset on_schema_change Policy from rc.5 release notes by @sgrebnov in #10964
• Skip tpcds_q78 for Cayenne engine at SF100 by @sgrebnov in #10966
• fix: Update benchmark snapshots May-20 by @app/github-actions in #10952
• Fix #10951: UdtfExec invariant Vec lengths must match children count by @phillipleblanc in #10953
• docs(release): update v2.0.0-rc.5 notes with latest trunk PRs by @lukekim in #10949
• Remove eval related things for v2.0.0 by @Jeadie in #10945
• build(deps): bump ubuntu from 24.04 to 26.04 in the docker-dependencies group by @app/dependabot in #10883
• fix: Add publish = false to chbench-driver by @sgrebnov in #10939

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.4...v2.0.0-rc.5

Announcing the release of Spice v1.11.1! 🛠️

v1.11.1 is a patch release improving Spice Cayenne accelerator reliability and performance, enhancing DynamoDB Streams and HTTP data connectors, and fixing issues in Federated Task History and FlightSQL.

What's New in v1.11.1​

Spice Cayenne Accelerator Improvements​

This release includes stability and performance fixes for the Spice Cayenne accelerator:

• Row-based Deletion Logic: Refactored row-based delete operations to use per-file deletion vectors with RoaringBitmap. Deletion scans now use Vortex-native streaming with filter pushdown and project only row indices, achieving zero data I/O for delete operations.
• Constraints & On Conflict: constraints and on_conflict configurations are now automatically inferred from federated table metadata, enabling datasets like DynamoDB to work without explicitly defining primary_key in the Spicepod.
• Partitioned Table Deletion: Fixed an issue where DELETE operations on partitioned Cayenne tables failed.
• Data Integrity: Fixed two issues with acceleration snapshot handling: protected snapshots are now included in conflict detection keyset scans (preventing duplicate key creation during append refresh), and snapshot cleanup no longer deletes protected snapshots.

Data Connector Improvements​

• DynamoDB Streams: Added automatic re-bootstrapping when the stream lag exceeds DynamoDB shard retention (24h). Configurable via the new lag_exceeds_shard_retention_behavior parameter with values error (default), ready_before_load, or ready_after_load.
• HTTP Connector: HTTP responses now include a response_status column (UInt16). 4xx responses (e.g., 404 Not Found) are treated as valid queryable data and cached normally. 5xx responses are retried with backoff, returned to the user, but excluded from the cache to prevent transient server errors from polluting cached results.

Other Improvements​

• Reliability: Added retries for SnapshotManager operations and general snapshot reliability improvements.
• Reliability: Fixed handling of timestamp precision mismatches in query result caching.
• Reliability: Fixed a double projection issue in federated task history queries that caused Schema error: project index out of bounds errors in cluster mode.
• Developer Experience: Added cookie middleware support to the FlightSQL data connector.

Contributors​

• @krinart
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates. The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.1 image:

docker pull spiceai/spiceai:1.11.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Cayenne: row-based delete logic improvements by @sgrebnov in #9237
• Proper support for constraints/on_conflict in Cayenne Acceleration by @krinart in #9335
• Retries for SnapshotManager by @krinart in #9334
• fix(cayenne): Include protected snapshots in conflict detection keyset scan by @sgrebnov in #9176
• fix(cayenne): Fix data loss by preserving protected snapshots during cleanup by @sgrebnov in #9182
• Simplify retention filter expressions before pushdown by @sgrebnov in #9244
• Fix test_retention_complex_sql by @sgrebnov in #9270
• runtime: avoid double projection in federated task history by @phillipleblanc in #9326
• feat(http): Return all HTTP responses as data, skip caching 5xx by @sgrebnov in #9313
• Snapshots Improvements by @krinart in #9318
• fix(caching): Handle timestamp precision mismatch and add more tests by @sgrebnov in #9315
• DynamoDB Streams Table Rebootstrapping by @krinart in #9305
• Fix Cayenne partitioned table deletion support by @sgrebnov in #9267
• FlightSQL: add cookie middleware support by @phillipleblanc in #9282
• Apply SchemaCastScanExec before applying changes in process_upsert_batch by @krinart in #9297

Announcing the release of Spice v1.10.1! 🚀

v1.10.1 is a patch release with Cayenne accelerator improvements including configurable compression strategies and improved partition ID handling, isolated refresh runtime for better query API responsiveness, and security hardening. In addition, the GO SDK, gospice v8 has been released.

What's New in v1.10.1​

Cayenne Accelerator Improvements​

Several improvements and bug fixes for the Cayenne data accelerator:

• Compression Strategies: The new cayenne_compression_strategy parameter enables choosing between zstd for compact storage or btrblocks for encoding-efficient compression.
• Improved Vortex Defaults: Aligned Cayenne to Vortex footer configuration for better compatibility.
• Partition ID Handling: Improved partition ID generation to avoid potential locking race conditions.

Example spicepod.yaml configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      params:
        cayenne_compression_strategy: zstd # or btrblocks (default)

For more details, refer to the Cayenne Data Accelerator Documentation.

Isolated Refresh Runtime​

Refresh tasks now run on a separate Tokio runtime isolated from the main query API. This prevents long-running or resource-intensive refresh operations from impacting query latency and ensures the /health endpoint remains responsive during heavy refresh workloads.

Security Hardening​

Multiple security improvements have been implemented:

• Recursion Depth Limits: Added limits to DynamoDB and S3 Vectors integrations to prevent stack overflow from deeply nested structures, mitigating potential DoS attacks.
• Spicepod Summary API: The GET /v1/spicepods endpoint now returns summarized information instead of full spicepod.yaml representations, preventing potential sensitive information leakage.

Additional Improvements & Bug Fixes​

• Performance: Fixed double hashing of user supplied cache keys, improving cache lookup efficiency.
• Reliability: Fixed idle DynamoDB Stream handling for more stable CDC operations.
• Reliability: Added warnings when multiple partitions are defined for the same table.
• Performance: Eagerly drop cached records for results larger than max cache size.

Spice Go SDK v8​

The Spice Go SDK has been upgraded to v8 with a cleaner API, parameterized queries, and health check methods: gospice v8.0.0.

Key Features:

• Cleaner API: New Sql() and SqlWithParams() methods with more intuitive naming.
• Parameterized Queries: Safe, SQL-injection-resistant queries with automatic Go-to-Arrow type inference.
• Typed Parameters: Explicit type control with constructors like Decimal128Param, TimestampParam, and more.
• Health Check Methods: New IsSpiceHealthy() and IsSpiceReady() methods for instance monitoring.
• Upgraded Dependencies: Apache Arrow v18 and ADBC Go driver v1.3.0.

Example usage with a local Spice runtime:

import "github.com/spiceai/gospice/v8"

// Initialize client for local runtime
spice := gospice.NewSpiceClient()
defer spice.Close()

if err := spice.Init(
    gospice.WithFlightAddress("grpc://localhost:50051"),
); err != nil {
    panic(err)
}

// Parameterized query (safe from SQL injection)
reader, err := spice.SqlWithParams(
    ctx,
    "SELECT * FROM users WHERE id = $1 AND created_at > $2",
    userId,
    startTime,
)

Upgrade:

go get github.com/spiceai/gospice/[email protected]

For more details, refer to the Go SDK Documentation.

Contributors​

• @phillipleblanc
• @lukekim
• @peasee
• @krinart
• @jeadie
• @sgrebnov
• @mach-kernel

Breaking Changes​

• GET /v1/spicepods no longer returns the full spicepod.yaml JSON representation. A summary is returned instead. See #8404.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 82+ recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.10.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.1 image:

docker pull spiceai/spiceai:1.10.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• Return summarized spicepods from /v1/spicepods by @phillipleblanc in #8404
• DynamoDB tests and fixes by @lukekim in #8491
• Use an isolated Tokio runtime for refresh tasks that is separate from the main query API by @phillipleblanc in #8504
• fix: Avoid double hashing cache key by @peasee in #8511
• fix: Remove unused Cayenne parameters by @peasee in #8500
• feat: Support vortex zstd compressor by @peasee in #8515
• Fix for idle DynamoDB Stream by @krinart in #8506
• fix: Improve Cayenne errors, ID selection for table/partition creation by @peasee in #8523
• Update dependencies by @phillipleblanc in #8513
• Upgrade to gospice v8 by @lukekim in #8524
• fix: Add recursion depth limits to prevent DoS via deeply nested data (DynamoDB + S3 Vectors) by @phillipleblanc in #8544
• fix: Add warning when multiple partitions are defined for the same table by @peasee in #8540
• fix: Eagerly drop cached records for results larger than max by @peasee in #8516
• DDB Streams Integration Test + Memory Acceleration + Improved Warning by @krinart in #8520
• fix(cluster): initialize secrets before object stores in executor by @sgrebnov in #8532
• Show user-friendly error on empty DDB table by @krinart in #8586
• Move 'test_projection_pushdown' to runtime-datafusion by @Jeadie in #8490
• Fix stats for rewritten DistributeFileScanOptimizer plans by @mach-kernel in #8581

Announcing the release of Spice v1.8.2! 🔍

Spice v1.8.2 is a patch release focused on reliability, validation, performance, and bug fixes, with improvements across DuckDB acceleration, S3 Vectors, document tables, and HTTP search.

What's New in v1.8.2​

Support Table Relations in /v1/search HTTP Endpoint​

Spice now supports table relations for the additional_columns and where parameters in the /v1/search endpoint. This enables improved search for multi-dataset use cases, where filters and columns can be used on specific datasets.

Example:

curl 'http://localhost:8090/v1/search' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' -d '{
        "text": "hello world",
        "additional_columns": ["tbl1.foo", "tbl2.bar", "baz"],
        "where": "tbl1.foo > 100000",
        "limit": 5
    }'