10 posts tagged with "caching"

Spice v2.3.0 brings performance improvements, broader query federation, and expanded data connector capabilities. The release improves cache reuse and reduces memory use for cached SQL results. It also extends BigQuery support and adds GitHub review, release, and repository data for SQL analysis. Google models now use Vertex AI, so deployments with credentials for Google AI Studio require migration.

Highlights in v2.3.0 include:

• SQL Federation Improvements — bug fixes for function translation, NULL handling, and timezone declarations
• BigQuery Federation — temporal expressions, recursive CTEs, correlated subqueries, and window aggregates now run as one BigQuery job
• Caching Accelerator Limits — caching_max_size and caching_max_items bound what a refresh_mode: caching accelerator holds
• Stale-While-Revalidate Across a Refresh — configured stale-while-revalidate windows retain cached results across acceleration refreshes
• GitHub Review Data — eight new tables, 17 new pulls columns, and a repo / owner column on every row
• Google Models on Vertex AI — a from: google model authenticates as a GCP service account

What's New in v2.3.0​

SQL Federation Improvements​

This release improves SQL translation and function handling for accelerated and federated datasets. The following bug fixes cover string functions, NULL handling, correlated subqueries, and timezone declarations.

• A trim call failed on DuckDB, SQLite, and MySQL because DataFusion emitted its canonical name, btrim. DuckDB now receives trim, with an explicit space argument for the one-argument form. The native SQLite and MySQL paths evaluate btrim locally.
• DuckDB federation now lower-cases to_hex output to match local evaluation.
• DuckDB federation now decodes sha256 output into a 32-byte digest instead of a hexadecimal string.
• concat uses || on DuckDB to consistently propagate NULL arguments.
• inner_product now returns NULL for an undefined dot product on DuckDB for consistency.
• Operations that DuckDB cannot handle (such as regex with U or R flags) are executed locally.
• User-defined functions registered after startup now execute locally.
• Catalog connectors now use the same list of local Spice-only functions as the data connectors.
• Two EXISTS shapes emitted SQL which evaluated the correlation over the whole relation, so the bound selected nothing. A semi or mark join then reported a match on a row the plan never read. Both shapes now refuse pushdown and run locally.
• DuckDB labels a TIMESTAMPTZ column with the connection's own timezone. The connector built its pool and never pinned that setting. A dataset's schema therefore carried the host timezone, and the same query returned different rows on different machines. A connector session is now pinned to UTC.

BigQuery Federation​

BigQuery federation runs more query shapes as one remote job.

• Temporal expressions, recursive CTEs, and integer division now keep their results and stay in one federated statement. These shapes previously failed remotely, split into several queries, or produced wrong cohort boundaries.
• Three more statement shapes now run. The first is a grouped query that projects a wrapped form of its grouping expression. The second is a query whose federated tables all sit inside a correlated subquery. The third is any aggregate window function.
• A query that reads BigQuery tables from several datasets of one project now runs as one BigQuery query. It also no longer returns rows from the wrong dataset when two datasets hold a table of the same name.
• A dataset that stores JSON in a STRING column now pushes down scalar json_as_text expressions and json_get(...) IS NULL checks.
• The built-in date_trunc is preserved, and the dialect and the federation policy now agree on which aggregate and window calls are eligible.
• regexp_match null checks federate safely.
• A distinct union now renders as UNION DISTINCT. BigQuery rejects a bare UNION, so such a query failed outright. UNION ALL is unchanged.
• A numbering function such as ROW_NUMBER no longer carries a window frame, which BigQuery rejects. An aggregate window function keeps its frame.
• Percentile functions and grouping keys now render in the form BigQuery accepts. A reported 29-statement workload that failed on these shapes now runs in full.
• array_element translates a non-negative integer literal index that fits in Int64 as SAFE_ORDINAL. Other indexes evaluate locally. This preserves DataFusion's end-relative semantics for negative indexes.

Cancellation: A BigQuery query whose client goes away now stops. The BigQuery job ends as cancelled, and the pooled connection returns immediately. Before this release the query ran to completion, and the connection stayed busy for its whole duration. Enough cancellations left an application unable to query at all.

The Caching Accelerator Accepts Explicit Limits​

A refresh_mode: caching accelerator had nothing bounding what it held. Retention was derived from caching_ttl plus caching_stale_while_revalidate_ttl, and only when caching_stale_if_error was disabled. A dataset that set caching_stale_if_error therefore got no policy at all, and nothing was ever evicted. Nothing capped the acceleration by size or by count either.

Two settings now bound the acceleration. Each one refuses an unparseable value rather than falling back to a default:

• caching_max_size — a byte budget, such as 512MiB.
• caching_max_items — a row budget.

caching_ttl is also accepted as caching_item_ttl, which is the spelling the SQL results, search results, and embeddings caches use. Eviction is entry-granular. A cached response can span several rows, so the runtime ranks entries by their oldest page and removes all of an entry's rows together. The storage schema is unchanged, and no existing acceleration needs a rebuild.

datasets:
  - from: https://api.example.com/v1/items
    name: items
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: caching
      primary_key: '(request_query, request_path)'
      params:
        caching_ttl: 5m
        caching_max_size: 512MiB
        caching_max_items: 50000

caching_stale_if_error now fires on the failure it exists for. It keyed off a fetch that returned an error. The HTTP connector reports a failing origin as a successful fetch whose rows carry a 429 or 5xx status once it exhausts max_retries. An operator who enabled the setting received the origin's error body instead of the cached response.

A caching accelerator that has nothing bounding it now says so at startup.

SQL Results Cache Improvements​

Stale results remain available across a refresh when configured. An acceleration refresh evicted every dependent SQL results cache entry, and any successful refresh counted as a change. A refresh_mode: full update still flushed the whole per-table cache. For a workload with consistently high QPS, each refresh turned a population of cached results into simultaneous synchronous misses.

When stale_while_revalidate_ttl is configured, an invalidation now marks dependent entries stale as of the refresh instead of evicting them. Inside the stale window the runtime serves the previous result with Results-Cache-Status: STALE and starts one background revalidation per key. Past the stale window the request is a miss, exactly as before. With no stale window configured, invalidation stays hard.

Cache accounting covers more retained memory. A cache with a million empty results reported 0.09 GiB against 1.85 GiB of retained memory. Its max_size accounting omitted parts of each entry. The cache now shares schemas and input-table sets, copies foreign buffers, and accounts for per-buffer allocation overhead.

In the reported benchmark, each entry retained 1%–80% less memory across 20 combinations of result shape and source. The benchmark ran on macOS/arm64 with snmalloc. Reported size ranged from 0.65x to 1.73x of retained memory after the change. The lowest ratio before the change was 0.31x. These figures compare the fix with its merge base, not v2.2.1, and are not guarantees for every workload.

This release also corrects memory accounting for the search results and embeddings caches.

Pingora cache engine: Table invalidation read every entry with a destructive get, so an invalidation promoted every key in the cache. Scan order replaced recency, and each visited key became a momentary miss to concurrent readers. A read also served a hit destructively, so a second reader reported a miss for a key the cache holds. Both reads are now non-destructive.

GitHub Data Connector: Review, Release, and Repository Tables​

The GitHub Data Connector adds eight tables, 17 columns on pulls, and repository identity on every row. An application can now answer a code-review question in SQL. Before this release, pulls.reviews_count was a bare integer with no state and no reviewer, and pulls.review_comments recorded only inline comments.

pulls adds is_draft, mergeable, merge_state_status, review_decision, status_check_rollup, merge_queue_state, merge_queue_position, merged_by, closed_by, base_ref, head_ref, head_sha, milestone_id, milestone_title, closing_issues_references, closing_issues_count, and reactions_count. issues adds state_reason, closed_by, reactions_count, type, and type_color. Every table returns a repo and an owner column, so a multi-repository UNION ALL keeps its rows apart.

The connector also asks GitHub for a narrower pull request page. A 100-node page now exceeds GitHub's per-request compute budget on a large repository. GitHub rejects that page with Resource limits for this query exceeded and returns every node as null, so the dataset never loaded.

Google Models Move to Vertex AI​

A from: google chat or embedding model now authenticates as a GCP service account against Vertex AI. Spice no longer accepts a Google AI Studio API key. See Breaking Changes for the migration.

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_project: my-project
      google_location: us-central1
      google_service_account_path: /etc/spice/gcp-sa.json

Other AI model fixes in this release:

• An Anthropic model configured without an explicit model id now resolves. The default named claude-3-5-sonnet-latest, which Anthropic has retired, so every such request failed.
• An Anthropic model now refuses an OpenAI top_logprobs request instead of translating it to top_k. The two fields are unrelated. top_logprobs reports log probabilities and top_k narrows sampling, so the translation silently changed the model's output.
• Anthropic streaming failures and provider refusals from openai, xai, and spiceai are now classified from the provider's typed error fields. Each path searched the rendered error text for 401, 429, or rate, and then replaced the provider's own detail with a fixed string.
• A from: huggingface: chat model reads hf_token again. The parameter moved to the prefix huggingface, so hf_token was warned about as unknown and a gated repository was downloaded anonymously.
• An Amazon Bedrock model now names the credential AWS rejected instead of reporting unhandled error.
• The text-embeddings-inference model-load path runs its filesystem and tokenizer work on a blocking thread. That work ran on a Tokio worker thread during model registration, so it could starve /health.

Search Improvements and Bug Fixes​

This release fixes bugs in search result limits, index updates, and deletes.

• A result set larger than the requested limit: vector_search(tbl, 'query', 10) against an Elasticsearch-backed index now respects the requested limit.
• Index writes and deletes kept in sync with the table: Previously, writes for rows with repeated primary keys, no chunks, or non-embeddable chunks could leave the previous index entry. These outdated index entries are now deleted.
• A chunked Elasticsearch delete: The delete filtered on the key columns, and a string key was left to Elasticsearch dynamic mapping as an analyzed text field. The delete now filters on a field that can match the key exactly.
• A partial Elasticsearch delete: _delete_by_query returns 2xx when the request ran, and it reports per-document failures and version conflicts in the body. Spice discarded that body, so a delete could leave documents behind and still report success.
• Full-text index encoding: A full-text upsert is a delete followed by an insert, so both halves must encode the primary key the same way. They disagreed for Float32, Float16, and Binary keys, and both the old and the new row stayed in the index.

Acceleration and Refresh​

• LIMIT on a partitioned scan: PartitionTableProvider::scan passed the scan limit as the skip argument rather than the fetch argument. LIMIT 10 over a three-row partitioned dataset returned zero rows.
• acceleration.enabled: false: A dataset or a view can set enabled: false and leave the rest of the block in place. The runtime read every other setting, accepted it, and then ignored it. The component reported healthy and served federated queries. The runtime now names the settings it discards.
• ready_state on a view: A view's acceleration.ready_state was accepted by the schema and by the parser, and then never applied. The identical key on a dataset was honoured. A view now resolves the key the way a dataset does.
• A retention policy that cannot start: A dataset that set retention_check_enabled: true, a retention_period, and a time_column but no retention_check_interval got no retention task and no diagnostic. The builder now reports the refusal.
• A refresh completion that arrives early: A completion published before a caller registered its wait was dropped with no record, and the caller waited for a refresh that had already happened. The signal is now level-triggered.
• A refresh completion from the wrong refresh: A waiter was satisfied by the next completion recorded on the table, whichever refresh produced it. A refresh that was already running could therefore release a caller. Completions are now correlated with the refresh that a caller triggered.
• A table replaced during a refresh: Two callers acted on a completion for a table that had since been removed or rebuilt. The runtime now re-resolves the table after the refresh lands and before it acts on the completion.
• A schema repair on a checkpoint: Writing a checkpoint's schema also wrote its refresh timestamp, so a schema repair told the scheduler the data was fresh. An overdue dataset then waited a full refresh_check_interval. A schema repair now leaves the freshness clock alone.
• A recorded snapshot schema: A snapshot's recorded schema is a foreign declaration, and a Map that declares its entries field nullable is a declaration no accelerator can hold. The restore path now conforms that declaration to the Arrow map layout.
• A source row durable write-back could not confirm: The delivery worker read a missing point-scan row as a deletion. A short visibility gap in the accelerator therefore deleted that row from the source of record. The worker now withholds a key it cannot read and retries it on a later pass. A delivery cursor advances only after the pass succeeds. Write-back also refuses a configuration it cannot uphold, which is a behavior change. See Breaking Changes.

Arrow and Storage Bug Fixes​

• Decimal128 on write paths: Three conversions could produce a plausible wrong number instead of an error. The sum of two in-range halves wrapped to a large negative decimal at scale 38. A float-to-int cast saturated to i128::MAX, and NaN became 0. The declared precision was never checked, so a value needing more digits than the column declares was appended anyway. All three were reachable from Debezium decimal ingestion, where the input is source-controlled. The runtime now validates the destination precision once, where every input form converges.

• An Iceberg DELETE an equality key cannot express: An Iceberg delete writes an equality delete file, which removes rows whose key columns equal the given values. That statement matches the user's WHERE only when the condition reads key columns alone. A condition on a float or a nested column removed rows that did not match. Spice now refuses the statement and names the offending column.

• A Parquet object overwritten mid-scan: A listing-table Parquet scan decoded two object generations as one file. The scan now pins one generation through a version id or an If-Match header. dataset_acceleration_refresh_errors carries reason=object_generation_changed|parquet_decode|other, so an expected overwrite is distinguishable from corruption.

• An Arrow relabel: relabel_array_data carries an array's values across a type change untouched, and only field names and nested nullability flags may differ. Nothing enforced that contract. It now refuses three kinds of target:

  • a target that changes what the buffers mean
  • a target that declares away nulls the array still holds
  • a target whose same-typed sibling fields are reordered

  The third kind produced silent column-value transposition on the Delta Lake column-mapping path.

• Arrow MAP columns: The Arrow map layout forbids a nullable entries field, and MapArray::try_new refuses one. Nothing enforced it at decode. A producer that declared it that way handed over a column that decoded cleanly and then failed in the first kernel that rebuilt it. The runtime now normalizes entries nullability at every Arrow decode point. A Databricks SQL Warehouse MAP column, which declares entries nullable, no longer panics the runtime. A Cayenne accelerator that has already persisted the non-conforming declaration is now repaired.

• A nested nullability difference: try_cast_to decided its fast paths with Schema::contains, which permits a nested field's nullability to differ. RecordBatch requires the two types to be identical. The shared entry point now aligns the difference instead of publishing it.

• A retired Vortex file: Retiring a Cayenne file released its Vortex segments and left its footer in DataFusion's file-metadata cache, which has no TTL. A file opened during retirement could also repopulate the path that retirement had just cleared. The retirement drain had no ceiling. A stalled put therefore held every caller of the invalidation, and the delete sink is one of them. All three faults are fixed.

• A Cayenne teardown that deleted a shared metastore: Recreation of a Cayenne dataset could delete a shared metastore and leave other datasets unavailable after restart. The guard checked only the metastore named by that dataset's settings, not catalogs inside its data directory. Open file handles hid the loss until restart. The runtime now scans that directory before catalog changes and again before deletion. It refuses recreation if it finds a Cayenne metastore or cannot safely resolve the configured paths.

Observability and Operations​

• OpenTelemetry resource attributes: The OTLP ingest path parsed resource attributes such as service.name and service.instance.id and then dropped them. Data points from two processes were therefore indistinguishable once written. Resource attributes now reach the metric data points. The same change closes four ingest races that dropped data with no error. One of those races let a write publish through a table provider that a schema evolution had already replaced.
• A panicking query: A query whose execution panicked was sometimes returned as an empty HTTP 200 success, which no client can tell apart from "no rows matched". This happened in 20 of 60 identical runs on trunk. A panicking query is now always an error.
• runtime.cpu.cores above the container's ceiling: runtime.cpu.cores was the one CPU entitlement setting taken raw rather than clamped. A pod configured with runtime.cpu.cores: 6 under resources.limits.cpu: 2 sized every derived pool for six cores and was then throttled. The runtime now warns and names both readings. It does not clamp, because an operator may size the runtime for a node the pod has not reached yet.
• HTTP latency: The HTTP server sets TCP_NODELAY, which lowers the latency of a small response body. The Flight SQL server already set it.
• MCP tools in runtime.task_history: A proxied MCP tool call was recorded under two different task values depending on the entry point, so one logical tool split across two rows. Grouping by task gave wrong per-tool counts. Both entry points now use the encoded name.
• A discarded Flight batch: The runtime reported a data_loss count that counted a message by its body length. A batch whose body is empty still carries rows, so the count was wrong. The runtime now reads the IPC header.
• A hot reload that changes functions or catalogs: A cached logical plan embeds the ScalarUDF and the TableSource it was planned against. A hot reload that redefined a SQL function or replaced a catalog left those plans in place. The same SQL then kept answering from the replaced component. The plan cache is installed unconditionally with a one-hour TTL, so no caching configuration was needed to hit this. Both handlers now discard the affected plans.
• Cloud Connect metrics cadence: A Cloud Connect instance exports metrics every 10 seconds rather than every 30, so a chart drawn from the control stream resolves at 10 seconds. The payload is a snapshot of cumulative totals, so this changes chart resolution and not what is recorded.

Other Improvements and Bug Fixes​

• Glue catalog: A discovered Glue table that Spice cannot read, such as an ORC or Avro table, was absent from the catalog with nothing said about it. The catalog connector now reports each such table and the reason.
• Databricks: The connector accepts Unity Catalog streaming tables and views. It also forwards the runtime's spark feature.
• DuckDB index materialization: The DuckDB intermediate index materialization rule reads a table's index list before it rewrites a scan into a materialized CTE. That list had been empty, so the rule never fired and an indexed column never narrowed a scan.
• Vortex list_length pushdown: DataFusion array_length(expr) and array_length(expr, 1) now convert to Vortex list_length and push into the scan. A list length is computed from offsets, and element values are not materialized.
• ScyllaDB: The ScyllaDB Data Connector is out of the default build, alongside ODBC. make install-scylladb or --features scylladb builds it. A Spicepod that names scylladb: on a build without it now says the build lacks the connector rather than offering the closest registered name. The connector also declines a physical sort that CQL cannot serve.
• Turso: The accelerator refuses a stored list whose encoding predates the version marker rather than reading it under the current encoding.
• CLI: spice run resolves spiced beside the CLI before it reaches for the managed install, and never from PATH. A Spice Cloud project listing is attributed to the organization it was requested for. spice query and spice nsql analyze keep the API key on its origin across a redirect. A Cloud Connect managed instance no longer warns about the default pods watcher on every spice run.
• MCP tools: A renamed tool forwards strict() and as_mcp_proxy() to the tool it wraps. runtime-tools declares what its mcp feature needs.
• Connector registries on shutdown: The runtime no longer clears stateless connector registries on shutdown.

Dependency Updates​

Compared with v2.2.1, this release changes the following versions:

DataFusion remains at v54.1.0 and Arrow remains at v58.3.0. Spice updates their fork revisions for the federation, Parquet scan, and cache fixes described above.

Contributors​

• @bjchambers
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @vatsalp2008

Breaking Changes​

Google models authenticate against Vertex AI. A from: google chat or embedding model no longer accepts google_api_key. Every such model now authenticates as a GCP service account.

Update each from: google model and embedding to set google_project, google_location, and exactly one credential setting.

Before:

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_api_key: ${secrets:google_api_key}

After:

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_project: my-project
      google_location: us-central1
      google_service_account_path: /etc/spice/gcp-sa.json

Set exactly one of the three credential settings.

Two behavior changes to note before you upgrade:

• Durable write-back rejects unsafe settings and operations. Before you upgrade, set mode: file, remove retention settings, and declare a single-column primary_key for each durable write-back dataset. The runtime rejects unsupported settings at load time. Submit writes inside a transaction as one BEGIN; ...; COMMIT; request. Write-back datasets reject DELETE and TRUNCATE when you issue those statements.
• The ScyllaDB Data Connector is out of the default build. Build with --features scylladb, or run make install-scylladb, to keep it. ODBC already worked this way.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.3.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.0 image:

docker pull spiceai/spiceai:2.3.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(search): surface an Elasticsearch delete that only partially applied (fixes #12364) by @claudespice in #12720
• fix(ci): bound every integration job, so a wedged one cannot hold the queue (refs #12718) by @grokspice in #12721
• feat(tools): add pdf-parse tool to compare liteparse and pdf-inspector by @Jeadie in #12807
• fix(turso): refuse a stored list whose encoding predates the version marker (fixes #12632) by @grokspice in #12837
• Use TypedParams for reranker parameters by @Jeadie in #13046
• fix(search): correctness, async-safety, and performance fixes across the search subsystem by @Jeadie in #13065
• Decide full-text CDC-attachment at construction, not after by @Jeadie in #13075
• fix(scylladb): decline the physical sort pushdown CQL cannot serve (fixes #10775) by @claudespice in #13107
• fix(cache): stop Pingora table invalidation from promoting every key it reads (fixes #12674) by @claudespice in #13117
• fix(cayenne): stop a failed statistics read publishing a partial count as exact (fixes #13010) by @claudespice in #13125
• fix(cayenne): refuse a widened CDC batch a partitioned acceleration cannot apply (fixes #13051) by @claudespice in #13133
• fix(catalogs): report the Glue tables Spice cannot read instead of dropping them silently (fixes #13102) by @claudespice in #13149
• fix(ci): gate every tracked Rust source tree in the merge queue's change filter (fixes #13120) by @claudespice in #13151
• fix(cache): bound max_size on the memory an entry holds, not its array bytes (fixes #12931) by @claudespice in #13154
• fix(cayenne): honour the sort-merge row floor on the memory-gated path (fixes #12958) by @claudespice in #13157
• fix(cache): serve a Pingora hit without hiding it from a concurrent reader (fixes #12987) by @claudespice in #13158
• fix(vortex): stop a file opened mid-retirement from repopulating the cleared path (fixes #12963) by @claudespice in #13161
• fix(vortex): evict a retired file's footer with its segments (fixes #12953) by @claudespice in #13162
• feat(google): switch from Google AI Studio to Vertex AI by @krinart in #13210
• Run TEI candle model-load filesystem and tokenizer work on a blocking thread by @Jeadie in #13223
• fix(correctness): validate Decimal128 precision and overflow on write paths by @lukekim in #13224
• fix(runtime): stop clearing stateless connector registries on shutdown by @Jeadie in #13225
• fix(search): use one encoding path for full-text index writes and deletes by @Jeadie in #13226
• fix(docker): install ARM64 linker toolchain by @Jeadie in #13266
• ci: build the runtime integration archives with one feature set by @bjchambers in #13270
• ci: archive every integration test target in one invocation by @bjchambers in #13273
• fix(sql): bump the datafusion pin so a bounded EXISTS refuses instead of returning wrong rows (fixes #13277) by @claudespice in #13280
• ci: have the compiler-cache action own its credentials by @bjchambers in #13282
• fix(cpu-budget): warn when a configured cores value exceeds the container's real CPU ceiling (fixes #13275) by @claudespice in #13284
• fix(databricks): publish a MAP column with the non-nullable entries field Arrow requires (fixes #7307) by @claudespice in #13288
• fix(search): bound a search result set by the requested limit, not just the index read (fixes #13274) by @claudespice in #13290
• fix(vortex): bound the segment-cache retirement drain so a stuck put cannot hold a delete open (fixes #12964) by @claudespice in #13300
• Reach the accelerator contract without going through the runtime by @bjchambers in #13304
• fix(postgres): rebuild the datasets already streaming when a join replaces their replication slot (fixes #13229) by @claudespice in #13313
• fix(cli): resolve spiced beside the CLI before the managed install, and never from PATH by @claudespice in #13317
• fix(iceberg): refuse a delete an equality key cannot express by @lukekim in #13322
• Stop the accelerator engines reaching up into the runtime by @bjchambers in #13324
• fix(benchmarks): open snapshot-update PRs against the dispatch branch, not always trunk by @krinart in #13335
• chore(deps): bump datafusion-table-providers for NUMERIC scale fidelity by @lukekim in #13349
• feat(cloud-connect): export metrics every 10s by @phillipleblanc in #13353
• Move the accelerator engines into their own crates by @bjchambers in #13354
• perf(cayenne): shard the encode by what the write is, not by what the table declares by @lukekim in #13356
• fix(cli): attribute a project listing to the org it was requested for by @lukekim in #13357
• Enable Oracle TPC-H result validation, fix the loader that trimmed leading spaces by @sgrebnov in #13360
• Readme: Change PG catalog status from Alpha to Beta by @sgrebnov in #13361
• Upgrade iceberg-rust to v0.10.1 by @krinart in #13365
• perf(cayenne): cut a rewrite's shards at equal row mass, not equal width by @lukekim in #13373
• fix(deps): bump datafusion-table-providers for AVG/division NUMERIC scale rounding by @krinart in #13387
• fix(databricks): forward runtime's spark feature to the databricks connector by @sgrebnov in #13389
• Move the Cayenne accelerator into its own crate by @bjchambers in #13391
• fix(testoperator): compare exact decimals on their mantissas, not through f64 by @bjchambers in #13407
• Configure an accelerator engine through its constructor instead of a published global by @bjchambers in #13409
• fix(opentelemetry): merge resource attributes into metric data points and close OTLP-to-sink ingest races by @peasee in #13412
• fix(arrow): refuse a relabel that changes what the array's buffers mean (fixes #13423) by @claudespice in #13435
• fix(tools): record one task_history task per proxied MCP tool (fixes #13338) by @claudespice in #13437
• fix(release): read release notes from a file and trim them to GitHub's body limit by @sgrebnov in #13457
• fix(ci): probe the cc toolchain before reaching for brew in setup-cc by @claudespice in #13466
• fix(cayenne): refuse a teardown that would delete a metastore no parameter names (fixes #13436) by @claudespice in #13471
• fix(spiced): stop warning about the default pods watcher by @phillipleblanc in #13494
• fix(arrow): align a nested nullability difference instead of advertising it (fixes #13285) by @claudespice in #13496
• fix(acceleration): make the refresh-completion signal level-triggered (fixes #13086) by @claudespice in #13505
• fix(acceleration): apply a partitioned scan's LIMIT as a fetch, not a skip by @vatsalp2008 in #13507
• fix(ci): stage the retention OOM test binary without its debug info by @claudespice in #13511
• fix(ci): let sign-off run from a worktree nested inside the checkout, and add --skip-targeted by @bjchambers in #13520
• Authenticate the stargazers dataset with a PAT and drop qa_analytics by @lukekim in #13529
• fix(ci): isolate sccache per job on shared self-hosted Macs by @lukekim in #13531
• chore(scylladb): take the ScyllaDB connector out of the default build by @lukekim in #13532
• feat(github): add review, release, milestone, user and repo tables, plus repo/owner columns by @lukekim in #13545
• fix(ci): match the allowed refresh-task warning at its current module path by @claudespice in #13547
• fix(flight): normalize MAP entries nullability at every Arrow decode point (fixes #13495) by @claudespice in #13550
• fix(anthropic): default to a model Anthropic still serves (fixes #13557) by @claudespice in #13563
• fix(postgres): rebuild an emptied CDC acceleration rather than resume its surviving position (refs #13546) by @claudespice in #13566
• chore(spicepod): accelerate GitHub datasets with Cayenne instead of DuckDB by @lukekim in #13571
• fix(arrow): refuse a relabel that declares away nulls the array still holds (fixes #13433) by @claudespice in #13585
• fix(cayenne): refuse a catalog whose data directory would hold its metastore (fixes #13105) by @claudespice in #13593
• fix(ci): exempt every App account from the assignee gate, not just Dependabot (fixes #13115) by @grokspice in #13594
• fix(mysql): assign the binlog dump session's net_write_timeout floor as an integer literal (fixes #13307) by @grokspice in #13595
• fix(ci): let the E2E macOS build share the fleet's Cargo home (fixes #13299) by @grokspice in #13596
• fix(ci): call a test binary the runner cannot load an infrastructure failure (fixes #13518) by @grokspice in #13597
• fix(spicepod): say which acceleration settings enabled: false discards (fixes #13514) by @grokspice in #13602
• feat(caching): bound a caching accelerator by size, count and entry lifetime (closes #13525) by @bjchambers in #13604
• fix(ci): catch a stale Cargo.lock before the merge queue, not after a 55-minute build (fixes #13598) by @grokspice in #13606
• fix(bedrock): say which credential AWS rejected instead of "unhandled error" (refs #12396) by @claudespice in #13616
• feat(caching): serve results stale after an acceleration refresh instead of evicting them by @krinart in #13618
• fix(write-back): never delete a source row for a key the accelerator did not return by @phillipleblanc in #13638
• fix(cayenne): fold a staged append's unpublished keys into the PK-keyset rebuild (fixes #13639) by @claudespice in #13644
• fix(tools): forward strict() and as_mcp_proxy() from a renamed tool (fixes #13443) by @claudespice in #13649
• fix(delta_lake): order a column-mapping relabel target the way the scan reads it (fixes #13434) by @claudespice in #13655
• fix(ci): raise the e2e Linux build bound above the worst legitimate run (fixes #13674) by @grokspice in #13675
• fix: comment out refresh_append_overlap in the sample spicepod by @lukekim in #13680
• fix(anthropic): refuse a log-probability request instead of narrowing sampling (fixes #13581) by @claudespice in #13682
• fix(cayenne): record a pipelined non-conflict staged append's primary keys (fixes #13642) by @claudespice in #13686
• fix(schema): stop an illegal Arrow Map entries declaration from being stored or compared (fixes #13549) by @claudespice in #13695
• build(rust): upgrade toolchain to 1.97.1 by @lukekim in #13696
• fix(ci): stop a CI git push from blocking forever on a credential prompt (fixes #13701) by @claudespice in #13702
• fix(search): remove the vector a rejected write left behind (fixes #13504) by @claudespice in #13705
• fix(acceleration): stop a refresh already running from answering a later waiter (refs #13544) by @claudespice in #13709
• fix(search): remove a chunked row's stale chunks when its text goes away (refs #13704) by @claudespice in #13716
• fix(catalogs): install the Spice function deny-list on the SQL catalog connectors (refs #13664) by @claudespice in #13731
• fix(runtime-tools): declare what the mcp feature actually needs (fixes #13648) by @grokspice in #13733
• fix(runtime): re-resolve a table after its refresh lands, before acting on the completion (fixes #13603) by @claudespice in #13735
• fix(flight): count a discarded batch by its IPC header, not its body length (fixes #13636) by @grokspice in #13736
• docs(makefile): say what SPICED_DATA_FEATURES actually is (fixes #13678) by @grokspice in #13738
• fix(anthropic): classify a streaming failure by Anthropic's error type, not its message text (fixes #13562) by @claudespice in #13748
• fix(views): apply a view's acceleration.ready_state instead of dropping it (fixes #13615) by @claudespice in #13750
• fix(ci): resolve a Python 3.11+ interpreter for the lint-rust guards (refs #13754) by @grokspice in #13755
• fix(bigquery): emit valid pushed-down SQL by @phillipleblanc in #13768
• fix(bigquery): safely federate regexp_match null checks by @krinart in #13771
• feat(hash-index): verify the bloom filter's block index with Verus by @lukekim in #13777
• fix(adbc): federate a BigQuery statement spanning datasets as one query by @phillipleblanc in #13780
• fix(adbc): cancel an abandoned query, stop the BigQuery job, free the connection by @phillipleblanc in #13782
• fix(snapshot): conform a recorded snapshot schema to the Arrow map layout (fixes #13694) by @claudespice in #13786
• fix(cayenne): materialize the in-memory CDC tier before a scanning DELETE by @lukekim in #13798
• fix(caching): say when a caching accelerator has nothing bounding it (fixes #13525) by @claudespice in #13805
• fix(bigquery): run three federated statement shapes BigQuery was refusing by @phillipleblanc in #13812
• fix(duckdb): rewrite DataFusion's btrim to DuckDB's trim (fixes #13794) by @claudespice in #13821
• fix(federation): stop pushing btrim to SQLite and MySQL, which have no btrim (fixes #13840) by @claudespice in #13823
• fix(ci): give the throughput workflow the postgres fixtures bench provisions by @krinart in #13830
• fix(acceleration,cayenne): Resolve quoted columns in keys, name which primary key columns are null by @peasee in #13845
• fix: pin listing-table Parquet reads to one object generation by @phillipleblanc in #13847
• fix(duckdb): lower-case the hex digits a federated to_hex gets back (fixes #13818) by @claudespice in #13852
• fix(bigquery): carry the merged unparser fixes through the dialect wrapper by @phillipleblanc in #13853
• fix(databricks): allow Unity Catalog streaming tables and views through the table-type check by @krinart in #13855
• fix(acceleration): report a retention policy that cannot start instead of silently building none (fixes #13804) by @claudespice in #13857
• fix(search): evict a key whose deciding row the index rejected (fixes #13848) by @claudespice in #13859
• fix(duckdb): Restore intermediate index materialization optimization by @sgrebnov in #13864
• fix(federation): refuse a user function the deny-list snapshot was built before (fixes #13726) by @claudespice in #13868
• fix(duckdb): decode a federated sha256 back to the digest's bytes (fixes #13850) by @claudespice in #13869
• fix(runtime): Add TCP_NODELAY to HTTP server by @peasee in #13874
• fix(query): surface a panicking query as an error, never an empty success (fixes #13876) by @claudespice in #13878
• fix(bigquery): keep the built-in date_trunc, forward two dialect renderings, repin the unparser by @phillipleblanc in #13882
• fix(llms): classify a provider refusal from its typed fields, not its message (refs #13747) by @claudespice in #13884
• fix(vortex): keep control-byte field names unescaped in physical schema by @lukekim in #13886
• perf(vortex): push DataFusion array_length down as Vortex list_length by @lukekim in #13888
• fix(duckdb): render a federated concat as || so a NULL argument propagates (fixes #13849) by @claudespice in #13889
• fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
• fix(duckdb): screen a federated inner_product so a non-finite result is NULL (fixes #13787) by @claudespice in #13895
• refactor(postgres): report an acceleration re-read as a refresh, not a bespoke metric by @bjchambers in #13896
• fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
• fix(duckdb): pin a connector's DuckDB session to UTC so a dataset's schema does not carry the host timezone (fixes #13899) by @claudespice in #13903
• fix(cayenne): keep a file's statistics the same whichever source serves them (refs #13829) by @claudespice in #13904
• fix(bigquery): preserve results and federation for temporal and recursive queries by @bjchambers in #13905
• Make the SQL results cache hold what it says it holds by @bjchambers in #13908
• fix(duckdb): keep a call the dialect cannot render out of the federated plan (fixes #13900) by @claudespice in #13909
• fix(functions): discard cached plans when a hot reload changes the function set (refs #13873) by @claudespice in #13911
• fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
• fix(cli): keep the API key on its origin in the SDK-built query client (fixes #12502) by @grokspice in #13923
• fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
• ci: align default and ODBC build features by @phillipleblanc in #13933
• fix(github): bound the pull request page to GitHub's per-request compute budget (refs #13762) by @grokspice in #13938
• fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
• fix(ci): preserve Cargo discovery markers during runner disk sweeps by @phillipleblanc in #13940
• fix(bigquery): push down JSON scalar text and null checks by @phillipleblanc in #13944
• fix(models): read the HuggingFace chat token as hf_token again (fixes #13932) by @claudespice in #13946
• fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
• fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
• ci: upgrade spiceio setup action to v0.9.0 by @lukekim in #13971

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.1...v2.3.0

Spice v2.1.5 is now available! 🛠️

Spice v2.1.5 is a patch release focused on dependable cached data and predictable operation under load. Cached queries now reflect data changes more reliably, Cayenne workloads stay within configured memory limits, health checks remain responsive while cached results are updated, and cache dashboards provide a more complete view of activity.

What's New in v2.1.5​

Cached Queries Stay Fresh as Data Changes​

Cached results are now cleared reliably after refreshes, writes, retention changes, and updates to dependent local datasets. Expired entries are removed promptly, and entries for data that is no longer part of a Cayenne dataset are not reused by later queries.

These improvements prevent a completed data change from being followed by an older cached answer. No configuration changes are required.

More Predictable Cayenne Behavior Under Heavy Load​

Cayenne now keeps track of the memory needed to prepare query results, including when several parts of a query are prepared at once. Multiple accelerated tables also share available memory instead of each planning as though it were the only table in the deployment.

Large and concurrent workloads are therefore less likely to exhaust the available memory. When a query cannot fit within the configured limit, it fails cleanly instead of putting the entire service at risk. Operators can also limit how much work one dataset does at once when it needs a smaller memory footprint, without slowing every query.

Health Checks Remain Responsive During Cache Updates​

Refreshing or writing a dataset with many cached results no longer holds up Spice while it finds old answers that need to be cleared. Health checks and other requests can continue during this work, reducing avoidable service restarts under load.

More Trustworthy Cache Dashboards​

Cache dashboards now show total space, space in use, stored results, requests, and successful reuse whenever the dashboard is refreshed, including for new or lightly used datasets. Counts for expired results, automatic size cleanup, and cleanup after data changes are also reported consistently, so a zero value represents no activity rather than missing information.

Contributors​

• @bjchambers
• @krinart
• @lukekim
• @phillipleblanc

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.5, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.5 image:

docker pull spiceai/spiceai:2.1.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.5

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix: arrow IndexedMemTable serves stale results after DML/retention/sync (fixes #11262) by @claudespice in #11532
• fix(cache): evict the Pingora cache down to max_size instead of only recording it (fixes #12688) by @grokspice in #12694
• fix(cache): count an invalidation as an eviction, and export the counters before one fires (fixes #12687) by @grokspice in #12791
• fix(cache): never serve a result whose tables changed after it read them by @bjchambers in #12703
• Remove unnecessary allocations from results-cache and hot conversion paths by @phillipleblanc in #11895
• fix(cache): read and expire a Pingora entry under one hold of its shard (fixes #12832) by @grokspice in #12839
• fix(runtime): invalidate localpod children's cached results on parent refresh by @krinart in #12897
• fix(cache): run the Pingora invalidation scan off the calling runtime worker (fixes #12806) by @grokspice in #12808
• fix(cache): count the removals the Pingora engine performs itself (fixes #12792) by @grokspice in #12830
• feat(cayenne): charge scan materialization to the query memory pool by @lukekim in #12759
• feat(cayenne): bound the SUM of the per-table PK keyset caches by @lukekim in #12802
• Charge the query memory pool for concurrent Vortex split decodes by @lukekim in #12940
• fix(vortex): invalidate retired segment cache entries by @phillipleblanc in #12943
• fix(vortex): expose segment cache metrics on scrape by @phillipleblanc in #12944

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.4...v2.1.5

Spice v2.1.1 is now available! 🛠️

Spice v2.1.1 is a patch release focused on reliability and performance. It resolves a possible deadlock that affected datasets with partitioned Cayenne accelerators, caches empty SQL result sets so repeat queries are served from cache, speeds up repeated queries on multi-file datasets, and restores Bedrock embedding provider parameters.

What's New in v2.1.1​

Cayenne Partitioned Dataset Deadlock Fix​

Cayenne datasets configured with partition_by could deadlock during their initial refresh and never become ready. Non-partitioned tables and small partitioned tables were unaffected. The root cause was a deadlock between the partition routing and the global Vortex encode budget introduced in v2.1.0.

Cayenne Zero-Row Append Refresh Stability​

An idle append refresh, one where no source rows are newer than the current max(time_column), wrote no Vortex files, so the expected snapshot directory was never created. The subsequent fsync on that directory failed with ENOENT, marking the dataset unhealthy. The fix skips the snapshot sequence record and protected-snapshot publish when the write carried no rows.

Caching of Empty Result Sets​

The SQL results cache now stores empty (zero-row) result sets. Queries that legitimately return no rows (e.g. WHERE 1=0, LIMIT 0) are now served from the results cache on subsequent requests instead of being re-executed against the source, reducing planning and query latency for these patterns.

Faster Repeated Queries on Multi-File Datasets​

Object store datasets using parquet now cache Parquet footer statistics across queries. This reduces the frequency of Parquet footer parsing during planning, subsequently heavily reducing planning latency for certain query patterns (e.g. COUNT(*)).

Embedding Parameter Regression Fixes​

v2.1.0 introduced explicit definitions across embedding component parameters (i.e. .embeddings[].params). This introduced regressions for AWS Bedrock parameters truncation and truncation_mode that caused panics.

Contributors​

• @bjchambers
• @Jeadie
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.1 image:

docker pull spiceai/spiceai:2.1.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(cache): cache empty (zero-row) SQL result sets by @bjchambers in #11699
• fix(cayenne): zero-row append refresh fails with "No such file or directory" by @sgrebnov in #11710
• feat(file): cache ListingTable file statistics to avoid per-query footer re-parse by @phillipleblanc in #11793
• fix(embeddings): restore params broken by #10853 by @Jeadie in #11788
• fix(cayenne): partitioned datasets deadlock against the global encode budget and never become ready by @Jeadie in #11825

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.0...v2.1.1

Announcing the release of Spice v1.11.3! 🛠️

Spice v1.11.3 is a patch release fixing schema consistency issues in the S3 and FlightSQL data connectors, improving CDC cache invalidation, and enhancing the HTTP data connector's error handling and response metadata.

What's New in v1.11.3​

S3 Data Connector Fix​

Fixed an issue where queries using metadata columns (location, last_modified, size) on S3 datasets produced Input field name does not match with the projection expression errors (#9647). This occurred when projecting metadata columns with filters or scalar functions (e.g., SELECT lower(location) FROM table WHERE location = '...'), and when projection returned no matching files.

FlightSQL Schema Consistency​

Fixed an issue where the Flight SQL JDBC driver returned Unsupported ArrowType Utf8View errors when performing ::TEXT type casts (#9253). The FlightSQL endpoint now maps view types (e.g., Utf8View, BinaryView) to their non-view equivalents, ensuring compatibility with JDBC and ODBC clients.

CDC Cache Invalidation​

Fixed an issue where the SQL results cache was invalidated on every change stream poll, even when zero records were returned (#9472). This caused near-total cache miss rates for datasets using refresh_mode: changes (e.g., DynamoDB Streams), effectively rendering the cache useless. Cache invalidation now only occurs when a change batch contains actual data changes.

HTTP Data Connector Improvements​

• HTTP error responses (e.g., 5xx) are now excluded from the cache, preventing transient server errors from polluting cached results.
• Added a response_headers column (Map type) to HTTP responses, providing access to response header metadata in query results.

Contributors​

• @krinart
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.3, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.3 image:

docker pull spiceai/spiceai:1.11.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(s3): Fix metadata column schema mismatches in projected queries by @sgrebnov in #9664
• s3_metadata_columns tests: include test for location outside table prefix by @sgrebnov in #9676
• Fix Flight SQL schema consistency: expand view types and verify field names by @sgrebnov in #9438
• Improve CDC cache invalidation by @krinart in #9651
• Skip caching http error response + add response_headers by @krinart in #9670

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.2...v1.11.3

Announcing the release of Spice v1.11.0-stable! ⚡

In Spice v1.11.0, Spice Cayenne reaches Beta status with acceleration snapshots, Key-based deletion vectors, and Amazon S3 Express One Zone support. DataFusion has been upgraded to v51 along with Arrow v57.2, and iceberg-rust v0.8.0. v1.11 adds several DynamoDB & DynamoDB Streams improvements such as JSON nesting, and adds significant improvements to Distributed Query with active-active schedulers and mTLS for enterprise-grade high-availability and secure cluster communication.

This release also adds new SMB, NFS, and ScyllaDB Data Connectors (Alpha), Prepared Statements with full SDK support (gospice, spice-rs, spice-dotnet, spice-java, spice.js, and spicepy), Google LLM Support for expanded AI inference capabilities, and significant improvements to caching, observability, and Hash Indexing for Arrow Acceleration.

What's New in v1.11.0​

Spice Cayenne Accelerator Reaches Beta​

Spice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous performance and stability improvements.

Key Enhancements:

• Key-based Deletion Vectors: Improved deletion vector support using key-based lookups for more efficient data management and faster delete operations. Key-based deletion vectors are more memory-efficient than positional vectors for sparse deletions.
• S3 Express One Zone Support: Store Cayenne data files in S3 Express One Zone for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence.

Improved Reliability:

• Resolved FuturesUnordered reentrant drop crashes
• Fixed memory growth issues related to Vortex metrics allocation
• Metadata catalog now properly respects cayenne_file_path location
• Added warnings for unparseable configuration values

For more details, refer to the Cayenne Documentation.

DataFusion v51 Upgrade​

Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.

Performance Improvements:

• Faster CASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patterns
• Better Defaults for Remote Parquet Reads: DataFusion now fetches the last 512KB of Parquet files by default, typically avoiding 2 I/O requests per file
• Faster Parquet Metadata Parsing: Leverages Arrow 57's new thrift metadata parser for up to 4x faster metadata parsing

New SQL Features:

• SQL Pipe Operators: Support for |> syntax for inline transforms
• DESCRIBE <query>: Returns the schema of any query without executing it
• Named Arguments in SQL Functions: PostgreSQL-style param => value syntax for scalar, aggregate, and window functions
• Decimal32/Decimal64 Support: New Arrow types supported including aggregations like SUM, AVG, and MIN/MAX

Example pipe operator:

SELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;

Improved Observability:

• Improved EXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdowns

Arrow 57.2 Upgrade​

Apache Arrow has been upgraded to v57.2, bringing major performance improvements and new capabilities.

Key Features:

• 4x Faster Parquet Metadata Parsing: A rewritten thrift metadata parser delivers up to 4x faster metadata parsing, especially beneficial for low-latency use cases and files with large amounts of metadata
• Parquet Variant Support: Experimental support for reading and writing the new Parquet Variant type for semi-structured data, including shredded variant values
• Parquet Geometry Support: Read and write support for Parquet Geometry types (GEOMETRY and GEOGRAPHY) with GeospatialStatistics
• New arrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution support

DynamoDB Connector Enhancements​

• Added JSON nesting for DynamoDB Streams
• Improved batch deletion handling

Distributed Query Improvements​

High Availability Clusters: Spice now supports running multiple active schedulers in an active/active configuration for production deployments. This eliminates the scheduler as a single point of failure and enables graceful handling of node failures.

• Multiple schedulers run simultaneously, each capable of accepting queries
• Schedulers coordinate via a shared S3-compatible object store
• Executors discover all schedulers automatically
• A load balancer distributes client queries across schedulers

Example HA configuration:

runtime:
  scheduler:
    state_location: s3://my-bucket/spice-cluster
    params:
      region: us-east-1

mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.

Credential Propagation: S3, ABFS, and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.

Improved Resilience:

• Exponential backoff for scheduler disconnection recovery
• Increased gRPC message size limit from 16MB to 100MB for large query plans
• HTTP health endpoint for cluster executors
• Automatic executor role inference when --scheduler-address is provided

For more details, refer to the Distributed Query Documentation.

iceberg-rust v0.8.0 Upgrade​

Spice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.

Key Features:

• V3 Metadata Support: Full support for Iceberg V3 table metadata format
• INSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tables
• Improved Delete File Handling: Better support for position and equality delete files, including shared delete file loading and caching
• SQL Catalog Updates: Implement update_table and register_table for SQL catalog
• S3 Tables Catalog: Implement update_table for S3 Tables catalog
• Enhanced Arrow Integration: Convert Arrow schema to Iceberg schema with auto-assigned field IDs, _file column support, and Date32 type support

Acceleration Snapshots​

Acceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.

Key Features:

• Flexible Triggers: Configure when snapshots are created based on time intervals or stream batch counts
• Automatic Compaction: Reduce storage overhead by compacting older snapshots (DuckDB only)
• Bootstrap Integration: Snapshots can reset cache expiry on load for seamless recovery (DuckDB with Caching refresh mode)
• Smart Creation Policies: Only create snapshots when data has actually changed

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled
      snapshots_trigger: time_interval
      snapshots_trigger_threshold: 1h
      snapshots_creation_policy: on_changed

Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically.

CLI Commands:

# List all snapshots for a dataset
spice acceleration snapshots taxi_trips

# Get details of a specific snapshot
spice acceleration snapshot taxi_trips 3

# Set the current snapshot for rollback (requires runtime restart)
spice acceleration set-snapshot taxi_trips 2

HTTP API Endpoints:

For more details, refer to the Acceleration Snapshots Documentation.

Caching Acceleration Mode Improvements​

The Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.

Performance Optimizations:

• Non-blocking Cache Writes: Cache misses no longer block query responses. Data is written to the cache asynchronously after the query returns, reducing query latency for cache miss scenarios.
• Batch Cache Writes: Multiple cache entries are now written in batches rather than individually, significantly improving write throughput for high-volume cache operations.

Reliability Fixes:

• Correct SWR Refresh Behavior: The stale-while-revalidate (SWR) pattern now correctly refreshes only the specific entries that were accessed instead of refreshing all stale rows in the dataset. This prevents unnecessary source queries and reduces load on upstream data sources.
• Deduplicated Refresh Requests: Fixed an issue where JSON array responses could trigger multiple redundant refresh operations. Refresh requests are now properly deduplicated.
• Fixed Cache Hit Detection: Resolved an issue where queries that didn't include fetched_at in their projection would always result in cache misses, even when cached data was available.
• Unfiltered Query Optimization: SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.

For more details, refer to the Caching Acceleration Mode Documentation.

Prepared Statements​

Improved Query Performance and Security: Spice now supports prepared statements, enabling parameterized queries that improve both performance through query plan caching and security by preventing SQL injection attacks.

Key Features:

• Query Plan Caching: Prepared statements cache query plans, reducing planning overhead for repeated queries
• SQL Injection Prevention: Parameters are safely bound, preventing SQL injection vulnerabilities
• Arrow Flight SQL Support: Full prepared statement support via Arrow Flight SQL protocol

SDK Support:

Example (Go):

import "github.com/spiceai/gospice/v8"

client, _ := spice.NewClient()
defer client.Close()

// Parameterized query with typed parameters
results, _ := client.SqlWithParams(ctx,
    "SELECT * FROM products WHERE price > $1 AND category = $2",
    spice.Float64Param(10.0),
    spice.StringParam("electronics"),
)

Example (Java):

import ai.spice.SpiceClient;
import ai.spice.Param;
import org.apache.arrow.adbc.core.ArrowReader;

try (SpiceClient client = new SpiceClient()) {
    // With automatic type inference
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM products WHERE price > $1 AND category = $2",
        10.0, "electronics");

    // With explicit typed parameters
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM products WHERE price > $1 AND category = $2",
        Param.float64(10.0),
        Param.string("electronics"));
}

For more details, refer to the Parameterized Queries Documentation.

Spice Java SDK v0.5.0​

Parameterized Query Support for Java: The Spice Java SDK v0.5.0 introduces parameterized queries using ADBC (Arrow Database Connectivity), providing a safer and more efficient way to execute queries with dynamic parameters.

Key Features:

• SQL Injection Prevention: Parameters are safely bound, preventing SQL injection vulnerabilities
• Automatic Type Inference: Java types are automatically mapped to Arrow types (e.g., double → Float64, String → Utf8)
• Explicit Type Control: Use the new Param class with typed factory methods (Param.int64(), Param.string(), Param.decimal128(), etc.) for precise control over Arrow types
• Updated Dependencies: Apache Arrow Flight SQL upgraded to 18.3.0, plus new ADBC driver support

Example:

import ai.spice.SpiceClient;
import ai.spice.Param;

try (SpiceClient client = new SpiceClient()) {
    // With automatic type inference
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM taxi_trips WHERE trip_distance > $1 LIMIT 10",
        5.0);

    // With explicit typed parameters for precise control
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM orders WHERE order_id = $1 AND amount >= $2",
        Param.int64(12345),
        Param.decimal128(new BigDecimal("99.99"), 10, 2));
}

Maven:

<dependency>
  <groupId>ai.spice</groupId>
  <artifactId>spiceai</artifactId>
  <version>0.5.0</version>
</dependency>

For more details, refer to the Spice Java SDK Repository.

Google LLM Support​

Expanded AI Provider Support: Spice now supports Google embedding and chat models via the Google AI provider, expanding the available LLM options for AI inference workloads alongside existing providers like OpenAI, Anthropic, and AWS Bedrock.

Key Features:

• Google Chat Models: Access Google's Gemini models for chat completions
• Google Embeddings: Generate embeddings using Google's text embedding models
• Unified API: Use the same OpenAI-compatible API endpoints for all LLM providers

Example spicepod.yaml configuration:

models:
  - from: google:gemini-2.0-flash
    name: gemini
    params:
      google_api_key: ${secrets:GOOGLE_API_KEY}

embeddings:
  - from: google:text-embedding-004
    name: google_embeddings
    params:
      google_api_key: ${secrets:GOOGLE_API_KEY}

For more details, refer to the Google LLM Documentation (see docs PR #1286).

URL Tables​

Query data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.

Supported Patterns:

• Single files: SELECT * FROM 's3://bucket/data.parquet'
• Directories/prefixes: SELECT * FROM 's3://bucket/data/'
• Glob patterns: SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'

Key Features:

• Automatic file format detection (Parquet, CSV, JSON, etc.)
• Hive-style partition inference with filter pushdown
• Schema inference from files
• Works with both SQL and DataFrame APIs

Example with hive partitioning:

-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'

Enable via spicepod.yml:

runtime:
  params:
    url_tables: enabled

Cluster Mode Async Query APIs (experimental)​

New asynchronous query APIs for long-running queries in cluster mode:

• /v1/queries endpoint: Submit queries and retrieve results asynchronously

OpenTelemetry Improvements​

Unified Telemetry Endpoint: OTel metrics ingestion has been consolidated to the Flight port (50051), simplifying deployment by removing the separate OTel port (50052). The push-based metrics exporter continues to support integration with OpenTelemetry collectors.

Note: This is a breaking change. Update your configurations if you were using the dedicated OTel port 50052. Internal cluster communication now uses port 50052 exclusively.

Observability Improvements​

Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:

• Snapshot monitoring widgets
• Improved accelerated datasets section
• Renamed ingestion lag charts for clarity

Additional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.

For more details, refer to the Monitoring Documentation.

Hash Indexing for Arrow Acceleration (experimental)​

Arrow-based accelerations now support hash indexing for faster point lookups on equality predicates. Hash indexes provide O(1) average-case lookup performance for columns with high cardinality.

Features:

• Primary key hash index support
• Secondary index support for non-primary key columns
• Composite key support with proper null value handling

Example configuration:

datasets:
  - from: postgres:users
    name: users
    acceleration:
      enabled: true
      engine: arrow
      primary_key: user_id
      indexes:
        '(tenant_id, user_id)': unique  # Composite hash index

For more details, refer to the Hash Index Documentation.

SMB and NFS Data Connectors​

Network-Attached Storage Connectors: New data connectors for SMB (Server Message Block) and NFS (Network File System) protocols enable direct federated queries against network-attached storage without requiring data movement to cloud object stores.

Key Features:

• SMB Protocol Support: Connect to Windows file shares and Samba servers with authentication support
• NFS Protocol Support: Connect to Unix/Linux NFS exports for direct data access
• Federated Queries: Query Parquet, CSV, JSON, and other file formats directly from network storage with full SQL support
• Acceleration Support: Accelerate data from SMB/NFS sources using DuckDB, Spice Cayenne, or other accelerators

Example spicepod.yaml configuration:

datasets:
  # SMB share
  - from: smb://fileserver/share/data.parquet
    name: smb_data
    params:
      smb_username: ${secrets:SMB_USER}
      smb_password: ${secrets:SMB_PASS}

  # NFS export
  - from: nfs://nfsserver/export/data.parquet
    name: nfs_data

For more details, refer to the Data Connectors Documentation.

ScyllaDB Data Connector​

A new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.

Example configuration:

datasets:
  - from: scylladb:my_keyspace.my_table
    name: scylla_data
    acceleration:
      enabled: true
      engine: duckdb

For more details, refer to the ScyllaDB Data Connector Documentation.

Flight SQL TLS Connection Fixes​

TLS Connection Support: Fixed TLS connection issues when using grpc+tls:// scheme with Flight SQL endpoints. Added support for custom CA certificate files via the new flightsql_tls_ca_certificate_file parameter.

Developer Experience Improvements​

• Turso v0.3.2 Upgrade: Upgraded Turso accelerator for improved performance and reliability
• Rust 1.91 Upgrade: Updated to Rust 1.91 for latest language features and performance improvements
• Spice Cloud CLI: Added spice cloud CLI commands for cloud deployment management
• Improved Spicepod Schema: Improved JSON schema generation for better IDE support and validation
• Acceleration Snapshots: Added configurable snapshots_create_interval for periodic acceleration snapshots independent of refresh cycles
• Tiered Caching with Localpod: The Localpod connector now supports caching refresh mode, enabling multi-layer acceleration where a persistent cache feeds a fast in-memory cache
• GitHub Data Connector: Added workflows and workflow runs support for GitHub repositories
• NDJSON/LDJSON Support: Added support for Newline Delimited JSON and Line Delimited JSON file formats

Additional Improvements & Bug Fixes​

• Model Listing: New functionality to list available models across multiple AI providers
• DuckDB Partitioned Tables: Primary key constraints now supported in partitioned DuckDB table mode
• Post-refresh Sorting: New on_refresh_sort_columns parameter for DuckDB enables data ordering after writes
• Improved Install Scripts: Removed jq dependency and improved cross-platform compatibility
• Better Error Messages: Improved error messaging for bucket UDF arguments and deprecated OpenAI parameters
• Reliability: Fixed DynamoDB IAM role authentication with new dynamodb_auth: iam_role parameter
• Reliability: Fixed cluster executors to use scheduler's temp_directory parameter for shuffle files
• Reliability: Initialize secrets before object stores in cluster executor mode
• Reliability: Added page-level retry with backoff for transient GitHub GraphQL errors
• Performance: Improved statistics for rewritten DistributeFileScanOptimizer plans
• Developer Experience: Added max_message_size configuration for Flight service

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @mach-kernel
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

OTel Ingestion Port Change​

OTel ingestion has been moved to the Flight port (50051), removing the separate OTel port 50052. Port 50052 is now used exclusively for internal cluster communication. Update your configurations if you were using the dedicated OTel port.

Distributed Query Cluster Mode Requires mTLS​

Distributed query cluster mode now requires mTLS for secure communication between cluster nodes. This is a security enhancement to prevent unauthorized nodes from joining the cluster and accessing secrets.

Migration Steps:

1. Generate certificates using spice cluster tls init and spice cluster tls add
2. Update scheduler and executor startup commands with --node-mtls-* arguments
3. For development/testing, use --allow-insecure-connections to opt out of mTLS

Renamed CLI Arguments:

Removed CLI Arguments:

• --cluster-api-key: Replaced by mTLS authentication

Cookbook Updates​

New ScyllaDB Data Connector Recipe: New recipe demonstrating how to use the ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.

New SMB Data Connector Recipe: New recipe demonstrating how to use the SMB Data Connector. See SMB Data Connector Recipe for details.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.0 image:

docker pull spiceai/spiceai:1.11.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

Dependencies​

• DataFusion: Upgraded to v51 (release notes)
• Arrow: Upgraded to v57.2 (release notes)
• iceberg-rust: Upgraded to v0.8.0 (release notes)

What's Changed​

Changelog​

• OTel exporter for push metrics by @lukekim in #8442
• fix: Update benchmark snapshots by @app/github-actions in #8448
• Add TPCH append tests to scheduled dispatch workflow by @sgrebnov in #8451
• Add snapshot creation logging by @krinart in #8469
• Fix PeriodicReader panic by @krinart in #8471
• Benchmarks: increase readiness timeout for turso acceleration (TPC-H) by @sgrebnov in #8470
• fix: Pin CUDA build actions to commits by @peasee in #8477
• Add Criterion benchmarking to chunking crate. by @Jeadie in #8431
• DuckDB agg pushdown: gate behind accelerator parameter by @mach-kernel in #8474
• Rename aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485
• Handle throttling exception for DynamoDB streams by @phillipleblanc in #8492
• docs: Add release notes by @peasee in #8478
• Update spicepod.schema.json by @app/github-actions in #8496
• Move 'test_projection_pushdown' to runtime-datafusion by @Jeadie in #8490
• Fix OTEL metrics HTTP exporter client setup by @phillipleblanc in #8489
• Update endgame to include new caching accelerator cookbook by @phillipleblanc in #8487
• DynamoDB tests and fixes by @lukekim in #8491
• Align make lint-rust-fix with make lint-rust by @Jeadie in #8499
• fix: Remove unused Cayenne parameters by @peasee in #8500
• Force task history captured_plan outputs to be captured even if they would be filtered out otherwise by @phillipleblanc in #8501
• release: post-release updates by @peasee in #8503
• CI: Fix E2E models dispatch by @mach-kernel in #8505
• Use an isolated Tokio runtime for refresh tasks that is separate from the main query API by @phillipleblanc in #8504
• Update openapi.json by @app/github-actions in #8512
• Update dependencies by @phillipleblanc in #8513
• fix: Avoid double hashing cache key by @peasee in #8511
• fix: Eagerly drop cached records for results larger than max by @peasee in #8516
• feat: Support vortex zstd compressor by @peasee in #8515
• warning if column is defined in spicepod but is non-existant by @Jeadie in #8498
• Return summarized spicepods from /v1/spicepods by @phillipleblanc in #8404
• Fix for idle DynamoDB Stream by @krinart in #8506
• Use Datafusion::Plan over Datafusion::Internal for user-facing search errors. by @Jeadie in #8484
• DDB Streams Integration Test + Memory Acceleration + Improved Warning by @krinart in #8520
• Upgrade to gospice v8 by @lukekim in #8524
• Vortex file format for object store by @Jeadie in #8525
• docs: Add missing cookbooks to endgame, focus area section by @peasee in #8527
• ListingTableConnector: Drop partition columns that reoccur in file schema by @mach-kernel in #8519
• fix(cluster): initialize secrets before object stores in executor by @sgrebnov in #8532
• Use separate Tokio runtime for SWR refreshes by @phillipleblanc in #8530
• fix: SQL Results Cache SWR triggers incorrect cache miss metric by @phillipleblanc in #8529
• feat: testoperator: OTLP streaming metrics / connect to existing instance / infinite mode / query timeout by @phillipleblanc in #8537
• Update openapi.json by @app/github-actions in #8518
• Add better attributes for search testing by @Jeadie in #8531
• fix: Improve Cayenne errors, ID selection for table/partition creation by @peasee in #8523
• Percent-encode Kubernetes secret name path segment by @phillipleblanc in #8522
• Initial parameterisation of Search integration tests by @Jeadie in #8066
• fix: Add warning when multiple partitions are defined for the same table by @peasee in #8540
• Add DuckDB file-mode support for search test parameterization. by @Jeadie in #8541
• fix: Add recursion depth limits to prevent DoS via deeply nested data (DynamoDB + S3 Vectors) by @phillipleblanc in #8544
• Remove the clippy::too_many_lines lint by @phillipleblanc in #8549
• feat: Add spice cluster tls commands by @phillipleblanc in #8550
• Move OTel ingestion to Flight port, remove separate OTel port 50052 by @phillipleblanc in #8551
• Move verbose tool init messages to trace by @phillipleblanc in #8552
• Add SANs to spice cluster tls add certificates by @phillipleblanc in #8554
• Add cpu, gpu, and memory to telemetry by @lukekim in #8483
• feat: Add workflows and workflow runs to GitHub Data Connector by @peasee in #8548
• Add S3Vectors option for paramterised search tests by @Jeadie in #8555
• Distributed query: TLS + API key by @mach-kernel in #8468
• Refactor RRF SQL to use LogicalPlanBuilder by @Jeadie in #7968
• Require mTLS for distributed query cluster mode by @phillipleblanc in #8580
• Fix stats for rewritten DistributeFileScanOptimizer plans by @mach-kernel in #8581
• Bump actions/cache from 4.3.0 to 5.0.1 by @app/dependabot in #8573
• Show user-friendly error on empty DDB table by @krinart in #8586
• Add (Deprecated) labels to deprecated spice sql params by @krinart in #8588
• Fix kafka warning when security.protocol is set to PLAINTEXT by @krinart in #8587
• Upgrade dependencies by @phillipleblanc in #8593
• Release notes for v1.10.1 by @Jeadie in #8568
• Run search benchmarks twice a week. by @Jeadie in #8592
• Add cayenne data accelerator by @Jeadie in #8553
• SQL allowlist for tools: sql, list_datasets, search, table_schema by @Jeadie in #8449
• use rstest for llms integration tests by @Jeadie in #8566
• Post v1.10.1 housekeeping by @Jeadie in #8600
• Add checklist for SDK publication in end_game.md by @Jeadie in #8602
• Fix IMAGE_TAG assignment for Docker compatibility by @Jeadie in #8599
• Modify cluster arguments to spiced for UX review by @phillipleblanc in #8603
• Update QA analytics with new release data by @Jeadie in #8601
• Remove 'tract-core' dependency. by @Jeadie in #8605
• Google embedding and chat models. by @Jeadie in #8423
• Fix test_github_workflows integration test by @sgrebnov in #8607
• Update openapi.json by @app/github-actions in #8604
• ci: Upload artifacts to MinIO eagerly after each build step by @phillipleblanc in #8615
• fix: SQLite accelerator decimal/date handling by @phillipleblanc in #8606
• Configure mTLS for executor-to-executor gRPC connections by @sgrebnov in #8617
• feat: Enable localpod with caching mode accelerator for tiered caching by @phillipleblanc in #8621
• Add Cayenne S3 Express One Zone support for data files by @lukekim in #8502
• Add snapshot interval for acceleration snapshots by @phillipleblanc in #8627
• Add dataset_load_parallelism parameter to spicepod.yml by @peasee in #8630
• Json Nesting for DynamoDB by @krinart in #8623
• Restore deprecated open-telemetry flag in spiced by @phillipleblanc in #8629
• Implement batching for Kafka/Debezium + null Decimal handling by @krinart in #8622
• fix: Status field in /v1/datasets & /v1/models by @lukekim in #8633
• Add Spice test operator improvements by @sgrebnov in #8625
• Add v1.10.2 release notes by @sgrebnov in #8640
• Align object-store vortex with runtime feature flagging by @Jeadie in #8620
• Extended LLM & search tests on cron by @Jeadie in #8624
• Test-operator: emit main metrics as part of load tests by @sgrebnov in #8639
• Build a local docker image from an existing Spice binary by @phillipleblanc in #8619
• fix: Use runtime-rate-control for GitHub Data Connector by @peasee in #8638
• Upgrade dependencies by @phillipleblanc in #8655
• Bump headers-accept from 0.1.4 to 0.3.0 by @app/dependabot in #8644
• Update openapi.json by @app/github-actions in #8637
• Update SECURITY.md - Include v1.10.2 by @sgrebnov in #8661
• Serialize acceleration snapshots with refresh writes by @phillipleblanc in #8652
• Update AI Installation test to use minilm_l6_v2 by @sgrebnov in #8659
• Fixes for search integration test CI by @Jeadie in #8656
• fix: Use a GitHub rate controller per auth context by @peasee in #8662
• fix: Update Search integration test snapshots by @app/github-actions in #8654
• Make E2E Test Release Installation (AI, Local HF model) test more robust by @sgrebnov in #8666
• Fix issue with location predicate for custom S3 endpoints + regression integration test by @phillipleblanc in #8668
• fix: Validate schema match before projection pushdown in UnionProjectionPushdownOptimizer by @phillipleblanc in #8669
• Proper batch commit for kafka/debezium by @krinart in #8671
• Improve spicepod json schema generation by @ewgenius in #8547
• Start the anonymous telemetry exporter asynchronously by @phillipleblanc in #8679
• fix: Move enforce-pulls to hosted runner by @phillipleblanc in #8686
• Update QA analytics with 1.10.2 release data by @sgrebnov in #8667
• fix: Azure does not support suffix range requests by @phillipleblanc in #8685
• Remove spicepod-validator cargo build from build-dev target by @Jeadie in #8684
• fix: Update test snapshots by @app/github-actions in #8680
• fix: Update Search integration test snapshots by @app/github-actions in #8681
• SMB and NFS Data Connectors by @lukekim in #8674
• Upgrade to openai-async v0.32 by @lukekim in #8635
• v1.10.3 release notes by @phillipleblanc in #8693
• Upgrade dependencies by @phillipleblanc in #8704
• fix: Support NDJSON and LDJSON by @lukekim in #8649
• move OpenAI overrides to non-prefixed by @Jeadie in #8678
• Update Google LLM param: google_dimensions -> dimensions by @Jeadie in #8677
• Make cluster mTLS optional with insecure flag by @phillipleblanc in #8703
• Revert "fix: Move enforce-pulls to hosted runner (#8686)" by @phillipleblanc in #8709
• Initial 'testoperator run text-to-sql' by @Jeadie in #8618
• Add support for abfss by @krinart in #8706
• Add testoperator TPCH dispatch for ABFS with hierarchical namespace disabled + versioning enabled by @phillipleblanc in #8711
• Update openapi.json by @app/github-actions in #8692
• cluster: validate --role argument by @phillipleblanc in #8717
• Upgrade to Turso v0.3.2 by @lukekim in #8716
• Rename --insecure to --allow-insecure-connections to be consistent with existing naming by @lukekim in #8720
• Remove 'testoperator run http-consistency/http-overhead' by @Jeadie in #8708
• refactor: Remove cluster feature flag by @phillipleblanc in #8718
• Docs: Distributed query ADR by @mach-kernel in #8608
• Use model.datasets to allowlist on tools by @Jeadie in #8714
• cluster: quality of life improvements to starting cluster mode locally by @phillipleblanc in #8719
• Docs: Ballista extension ADR by @mach-kernel in #8616
• Improve deprecation messages when going from prefixed -> non-prefixed. by @Jeadie in #8724
• Remove tools from auto-defaults by @Jeadie in #8725
• Make distinct providers for vector spilling, vector partitioning. by @Jeadie in #8546
• cluster: default scheduler address port by @phillipleblanc in #8728
• Add Makefile targets for testoperator by @Jeadie in #8729
• text-to-sql dispatch in testoperator by @Jeadie in #8705
• DR-006: High Availability Distributed Query with Stateless Schedulers by @lukekim in #8721
• DR-007: mTLS for Distributed Query Cluster Communication by @lukekim in #8722
• SMB and NFS improvements by @lukekim in #8710
• fix: Cluster executors use scheduler's temp_directory for shuffle files by @phillipleblanc in #8733
• use 'max_message_size' in flight service too by @Jeadie in #8730
• Add page-level retry for transient GraphQL errors with backoff and increase GitHub rate limit buffer up to 100 by @ewgenius in #8726
• Make testoperator Dockerfile; CI to build docker image to ghcr.io. by @Jeadie in #8732
• cluster: UnionProjectionPushdownOptimizer: Add projection pushdown diagnostics for union children by @phillipleblanc in #8734
• Fix column projection order mismatch with location metadata columns by @phillipleblanc in #8738
• Fixes for testoperator. by @Jeadie in #8737
• Improve Cayenne Deletion Vectors with KeyBased support by @lukekim in #8713
• Fix testoperator_dispatch.yaml by @Jeadie in #8740
• Add spice cloud CLI commands by @lukekim in #8528
• Add FTP, NFS, & SMB TPCH SF1 spicepods by @lukekim in #8739
• Prepared Statements by @lukekim in #7588
• Schedule dispatch of testoperator run text-to-sql. by @Jeadie in #8745
• Fix minio for ai benchmark CI by @Jeadie in #8743
• Upgrade to Rust 1.91 by @phillipleblanc in #8749
• fix: Update benchmark snapshots by @app/github-actions in #8763
• Benchmarks: make row count validation skip logic configurable by scale factor, query set, and overrides by @sgrebnov in #8756
• Make benchmark tests more robust by @sgrebnov in #8766
• Add parameter to force using iam_role for DynamoDB by @krinart in #8767
• fix: Update Search integration test snapshots by @app/github-actions in #8735
• v1.10.4 release notes by @phillipleblanc in #8790
• Trace metrics export errors by @sgrebnov in #8791
• v1.10.4 SECURITY.md update by @phillipleblanc in #8800
• Add timezone database to Docker image to fix Cayenne acceleration panic by @sgrebnov in #8799
• Upgrade dependencies by @phillipleblanc in #8801
• Fix table_allowlist for table sampling and NSQL by @Jeadie in #8789
• Cayenne primary key on-conflict handling by @lukekim in #8788
• fix: Update benchmark snapshots by @app/github-actions in #8773
• fix: correctly identify deprecated openai_* parameters by @phillipleblanc in #8809
• fix: Update benchmark snapshots by @app/github-actions in #8812
• Use workspace version for cayenne crate by @phillipleblanc in #8811
• Don't CAST strings which breaks push down optimizer by @lukekim in #8810
• fix: Update benchmark snapshots by @app/github-actions in #8815
• Update async-openai to latest revision 4dcd633aad6f - brings fix for openai compatible model providers by @ewgenius in #8816
• Add auth/iam_role_source to DynamoDB connector by @krinart in #8808
• DynamoDB fixes: JSON nesting for Streams, proper batch deletions by @krinart in #8821
• Rough roadmap for 2026-2027 by @lukekim in #8805
• Release notes for v1.11.0-rc1 by @ewgenius in #8786
• Make S3V integration tests prepare_for_aws_tests more robust by @sgrebnov in #8820
• Bump rsa from 0.9.9 to 0.9.10 in the cargo group across 1 directory by @app/dependabot in #8819
• Add timezone database to Release and CUDA Docker images to fix Cayene panic by @sgrebnov in #8832
• fix: UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831
• Update grafana/datadog example dashboards by @krinart in #8833
• Add Dev bird bench as text-to-sql queryset in CI. by @Jeadie in #8753
• Update testoperator scheduler to use release/1.11 branch by @ewgenius in #8829
• Spice Cayenne fixes and test spicepods for Beta & RC by @lukekim in #8787
• testoperator dispatch all bird-bench database variants by @Jeadie in #8835
• feat: Improve column statistics handling with safe access and defaults by @phillipleblanc in #8836
• cluster: mTLS verification by @phillipleblanc in #8837
• fix: 8770: Unsupported ScalarFunctionExpr in ORDER BY by @lukekim in #8838
• Workflow tweaks by @lukekim in #8845
• Cayenne: metadata catalog should respect cayenne_file_path location by @sgrebnov in #8844
• Expand Cayenne feature coverage by @lukekim in #8848
• docs: HA distributed query decisions by @phillipleblanc in #8817
• fix(optimizer): Fix correctness issues in UnionProjectionPushdownOptimizer by @phillipleblanc in #8851
• Pin reqwest to 0.12.24 to fix HuggingFace embedding model download by @ewgenius in #8853
• Fix builds and pin to Ubuntu 22.04 by @lukekim in #8856
• Revert "Fix builds and pin to Ubuntu 22.04" by @lukekim in #8861
• Ensure setup Rust is run by @lukekim in #8862
• fix: Ubuntu 24.04+ renamed libaio1 to libaio1t64 by @lukekim in #8865
• Upgrade to Pulls with Spice v2 by @lukekim in #8866
• Add limit and configuration name to 'testoperator run text-to-sql' by @Jeadie in #8839
• PR check and test optimization by @lukekim in #8868
• Upgrade S3 Vectors SDK and improve test robustness by @lukekim in #8867
• [Testoperator] Query level and improved aggregate level for NSQL by @Jeadie in #8840
• Add docker build for private branches for ghcr.io/spiceai/spiceai-dev by @phillipleblanc in #8873
• Expand the data acceleration round-trip test coverage by @lukekim in #8855
• fix: Provide a better error for improper bucket UDF arguments by @peasee in #8849
• ScyllaDB Data Connector by @lukekim in #8827
• Use tokio-rusqlite for Spice Cayenne SQLite by @lukekim in #8857
• Cayenne: fix FuturesUnordered reentrant drop crash by @sgrebnov in #8863
• Bump github/codeql-action from 4.31.9 to 4.31.10 by @app/dependabot in #8884
• Bump golang.org/x/sys from 0.39.0 to 0.40.0 by @app/dependabot in #8881
• Bump github.com/spiceai/gospice/v8 from 8.0.0 to 8.0.1 by @app/dependabot in #8883
• Bump roaring from 0.11.2 to 0.11.3 by @app/dependabot in #8885
• Bump golang.org/x/mod from 0.31.0 to 0.32.0 by @app/dependabot in #8882
• Bump aws-sdk-s3 from 1.115.0 to 1.119.0 by @app/dependabot in #8887
• Bump libc from 0.2.177 to 0.2.180 by @app/dependabot in #8886
• Bump tokio-util from 0.7.17 to 0.7.18 by @app/dependabot in #8889
• Bump governor from 0.10.2 to 0.10.4 by @app/dependabot in #8888
• fix: flaky test test_concurrent_partition_creation by @phillipleblanc in #8898
• Update Cayenne snapshots for TPC-DS by @lukekim in #8890
• Add more buckets to histogram metrics by @krinart in #8850
• feat: Add HTTP health endpoint for cluster executors by @phillipleblanc in #8899
• feat: Implement model listing functionality for multiple providers by @lukekim in #8901
• feat: Initial HA schedulers distributed query implementation by @phillipleblanc in #8852
• fix: infer executor role from --scheduler-address when --role is omitted by @phillipleblanc in #8903
• Improve install scripts and remove jq dependency by @lukekim in #8847
• Benchmarks: sort PartitionedUnionExec children for deterministic snapshot comparison by @sgrebnov in #8877
• Cayenne: share VortexFileCache across partitions via CayenneContext by @sgrebnov in #8880
• Update ballista to add exponential backoff for scheduler disconnection by @phillipleblanc in #8905
• Configurably add BirdBench evidence to testoperator text-to-SQL. by @Jeadie in #8904
• Helm: Allow command override via values.yaml by @sgrebnov in #8906
• Fix distributed query gRPC message size limit (16MB -> 100MB) by @phillipleblanc in #8900
• OS specific setup actions by @lukekim in #8909
• Cayenne should warn if unable to parse configuration value by @sgrebnov in #8907
• Add snapshots widgets to example dashboard by @krinart in #8910
• Add quality criteria for the features by @krinart in #8897
• Improve Accelerated Datasets section for Grafana/Datadog dashboards by @krinart in #8915
• Use HTTP traceparent in NSQL to support concurrency in 'testoperator run text-to-SQL' by @Jeadie in #8912
• Remove setup for cc from integration_models.yml by @Jeadie in #8917
• Propagate Azure and GCS credentials to executors in cluster mode by @phillipleblanc in #8918
• Cayenne: fix memory growth due to vortex metrics allocation by @sgrebnov in #8908
• fix(caching): Deduplicate refresh requests for JSON array responses by @sgrebnov in #8921
• fix(caching): Return cached data directly for unfiltered queries (SELECT *) by @sgrebnov in #8919
• Correct MinIO path syntax for spiced download by @Jeadie in #8916
• Acceleration snapshots compaction + Improved Snapshots UX by @krinart in #8858
• Change base image from bookworm-slim to trixie-slim by @Jeadie in #8923
• Add testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895
• Fix spicepod dependencies in testoperator by @Jeadie in #8875
• Update copilot instructions for data correctness by @lukekim in #8922
• Add BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926
• fix: add missing feature-gate for AWS Secrets Manager error variant by @phillipleblanc in #8928
• refactor: make ConnectorParams fields public for external connectors by @phillipleblanc in #8929
• fix(caching): SWR refreshes only accessed entry instead of all stale rows by @sgrebnov in #8931
• Cayenne: include cayenne_metadata_dir to known params by @sgrebnov in #8933
• Rename Ingestion Lag chart in example dashboards by @krinart in #8932
• fix(caching): Fix HTTP caching always MISS when projection excludes fetched_at by @sgrebnov in #8930
• Reset expiry after snapshot bootstraping for Caching by @krinart in #8925
• Set use_ssl=false for sccache by @lukekim in #8945
• Hash indexing for Arrow Acceleration by @lukekim in #8924
• [Cayenne] Acceleration snapshots support by @lukekim in #7973
• perf(caching): Non-blocking cache writes on cache miss by @sgrebnov in #8948
• Update NSQL models by @lukekim in #8951
• Hash Index Key verification by @lukekim in #8949
• Add snapshots_creation_policy param by @krinart in #8954
• Remove candle & cudarc from non-models build by @lukekim in #8955
• Acceleration Snapshots API and CLI by @lukekim in #8934
• Ignore test for data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962
• fix: Update benchmark snapshots by @app/github-actions in #8965
• Hash Index secondary index support by @lukekim in #8958
• fix: Support primary key constraints in partitioned DuckDB tables mode by @sgrebnov in #8966
• perf(caching): Batch cache writes by @sgrebnov in #8959
• CI perf optimizations by @lukekim in #8968
• Fix Makefile linting by @Jeadie in #8970
• Fixes in testoperator run text-to-sql. by @Jeadie in #8927
• implement Chat::as_sql for xAI anthropic by @Jeadie in #8957
• Fix duckdb_file_path in search integration test by @Jeadie in #8972
• fix: Update benchmark snapshots by @app/github-actions in #8971
• Maintenance updates to Anthropic API by @Jeadie in #8956
• Add CacheBackend Trait, implement pingora-lru, and add throughput tests by @lukekim in #8080
• fix: Update benchmark snapshots by @app/github-actions in #8974
• fix: Update benchmark snapshots by @app/github-actions in #8975
• Make accelerator shutdown more robust by @lukekim in #8969
• feat(duckdb): Add on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964
• Proper handling for initial snapshot by @krinart in #8911
• fix: Remove --no-default-features from cargo-hack command in features workflow by @phillipleblanc in #8977
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 by @app/dependabot in #8983
• build(deps): bump actions/checkout from 4 to 6 by @app/dependabot in #8982
• build(deps): bump actions/setup-go from 6.1.0 to 6.2.0 by @app/dependabot in #8984
• build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 by @app/dependabot in #8979
• build(deps): bump github.com/klauspost/compress from 1.18.2 to 1.18.3 by @app/dependabot in #8980
• Add /v1/queries and Arrow Flight async APIs by @lukekim in #8946
• build(deps): bump Vampire/setup-wsl from 5 to 6 by @app/dependabot in #8981
• fix: Update Search integration test snapshots by @app/github-actions in #8973
• build(deps): bump insta from 1.46.0 to 1.46.1 by @app/dependabot in #8988
• build(deps): bump schemars from 1.1.0 to 1.2.0 by @app/dependabot in #8985
• fix: Update benchmark snapshots by @app/github-actions in #8978
• fix: Data correctness edge cases by @lukekim in #8953
• Correct MinIO path syntax for spiced download (Part 2) by @Jeadie in #8995
• Make .spice/data in search integration tests by @Jeadie in #8992
• fix: Hash index composite keys null values by @lukekim in #9001
• Update Cayenne status to Beta by @lukekim in #9002
• fix: Disable TPC-DS result validation (not yet supported) by @sgrebnov in #9004
• feat: Upgrade to DataFusion v51 and dependencies by @lukekim in #8864
• Improvements for snapshots_creation_policy by @krinart in #9003
• fix(ci): restore cached spicepod-validator binary instead of lookup-only by @phillipleblanc in #9007
• Update version by @krinart in #9010
• Update lock file - https://github.com/spiceai/spiceai/commit/53babbf07ca8c1c7b2e1da42ce58c465d9bc9276/
• fix: Enable Cayenne acceleration snapshots by @lukekim in #9020
• Add TPC-DS integration tests with S3 source and PostgreSQL acceleration by @phillipleblanc in #9006
• fix(tests): fix flaky/slow/failing unit tests by @phillipleblanc in #9009
• fix: Update benchmark snapshots for DF51 upgrade by @app/github-actions in #9008
• fix: add feature gate to rrf TEST_EMBEDDING_MODEL by @phillipleblanc in #9017
• fix: features check by @phillipleblanc in #9014
• URL table support by @lukekim in #9018
• ScyllaDB key filter by @lukekim in #8997
• fix: Schema mismatch when using column projection with HTTP caching by @phillipleblanc in #9021
• Add more tests for HTTP caching with columns selection by @sgrebnov in #9025
• HTTP cache snapshots: default to time_interval and fix snapshots_creation_policy: on_change by @sgrebnov in #9026
• Fix duplicate snapshot creation on startup by @sgrebnov in #9029
• Remove waiting for runtime to be ready before creating snapshot by @krinart in #9033
• Fix snapshot on_change policy to skip when no writes occurred by @sgrebnov in #9028
• Release notes for release release/1.11.0-rc.2 by @krinart in #9016
• ci: use arduino/setup-protoc for official protobuf compiler by @phillipleblanc in #9036
• ci: install unzip on aarch64 runner for arduino/setup-protoc by @phillipleblanc in #9038
• fix: don't fail release if upload to minio fails by @phillipleblanc in #9039
• Improve validation and logging for hash indexes by @lukekim in #9047
• Pin to ubuntu-22.04 by @lukekim in #9068
• Fix broken telemetry for testoperator by @krinart in #9054
• Fix release builds by @lukekim in #9069
• Spice 1.11.0-rc3 release notes by @krinart in #9070
• Update spicepod.schema.json by @app/github-actions in #9071
• Add missing protoc step to setup-cc action by @krinart in #9041
• Fix TLS connection for grpc+tls:// Flight SQL endpoints and add custom CA certificate support by @phillipleblanc in #9073
• Update 1.11.0-rc.3 release notes by @krinart in #9082
• Fix formula_1 and codebase_community in bird-bench by @Jeadie in #9000
• Cayenne S3 Express One Zone improvements by @lukekim in #9015
• Add zlib1g-dev to CI by @lukekim in #9052
• Upgrade Vortex with CASE-WHEN by @lukekim in #9051
• fix: Cayenne CatalogError handling for constraint violations by @lukekim in #9050
• Fix Docker build failing to copy shared libraries due to ldd output parsing by @phillipleblanc in #9058
• feat: Change /v1/sql and FlightSQL to use local execution in cluster mode by @phillipleblanc in #9055
• Remove unmaintained dependencies by @lukekim in #9045
• Enable cayenne + changes stream by @Jeadie in #9053
• feat(cli): add spice query command for async queries REPL by @phillipleblanc in #9057
• Remove unncessary allocations by @lukekim in #9059
• Add dataset_acceleration_size_bytes metric by @krinart in #9062
• Fix tracing of sql_query beneath tool_use::sample_data. by @Jeadie in #9043
• Basic script to run distributed spice by @Jeadie in #9049
• Add integration tests for Acceleration Snapshots by @krinart in #9067
• Upgrade CUDA toolkit to 12.6.0 by @sgrebnov in #9079
• Install required protoc dependency for CUDA build by @sgrebnov in #9080
• feat(cluster): add executor control stream heartbeat by @phillipleblanc in #9072
• feat: Fix async queries API and integrate Ballista shuffle improvements by @lukekim in #9075
• Remove models variant (now default) & Windows builds (use WSL) by @lukekim in #9063
• Cayenne: share upload semaphore across partitions to bound memory growth and optimize I/O by @sgrebnov in #9078
• Snowflake data connector - add snowflake_private_key parameter by @ewgenius in #9085
• Rewrite Go CLI in Rust by @phillipleblanc in #9061
• GCS Data Connector (Alpha) by @lukekim in #9084
• Skip 'latest' Docker tag for pre-release versions by @sgrebnov in #9077
• Add HTTP endpoints for acceleration snapshots API by @phillipleblanc in #9065
• Cayenne: Allow append mode with both primary_key and time_column by @sgrebnov in #9090
• Add Cluster Observability (Metrics+Dashboard) by @phillipleblanc in #9066
• proto for 'CayenneAccelerationExec' by @Jeadie in #9094
• Add 'anthropic-beta' header for structured outputs by @Jeadie in #9093
• Cayenne: refactor write path to use insert_into() as single entry point (part 1) by @sgrebnov in #9088
• Fix testoperator dispatch by @sgrebnov in #9097
• Fix setup-spiced GH action (_models suffix does not exist anymore) by @sgrebnov in #9102
• build(deps): bump github/codeql-action from 4.31.10 to 4.31.11 by @app/dependabot in #9108
• Remove DistributeFileScanOptimizer and UnionProjectionPushdownOptimizer & set target_partitions dynamically based on cluster capacity by @phillipleblanc in #9100
• build(deps): bump zip from 2.4.2 to 6.0.0 by @app/dependabot in #9111
• fix: Preserve query parameter order in HTTP connector to match filter values by @sgrebnov in #9114
• Add PollNow interrupt for Ballista executors to reduce task scheduling latency by @phillipleblanc in #9098
• Revert "GCS Data Connector (Alpha) " by @lukekim in #9084
• Fix stack overflow for CDC batching by @krinart in #9115
• fix: update Ballista fork to include executor timeout fix by @phillipleblanc in #9124
• Properly propagate SIGINT/SIGTERM from CLI to runtime by @krinart in #9127
• fix: Use the same vortex dependency as ballista by @peasee in #9123
• release: Bump version to 1.11.0 for stable - https://github.com/spiceai/spiceai/commit/14d09f8e262008df69ded898ed3bebee08471508/
• Cayenne snapshots with shared metadata by @lukekim in #9118
• Improve error handling for URL tables with Azure URLs by @phillipleblanc in #9129
• Add missing Windows build step for spice CLI in build_and_release workflow by @phillipleblanc in #9143
• Fix install-dev to use debug build path for spice binary by @phillipleblanc in #9142
• fix: CLI builds by @peasee in #9145
• Always create initial snapshots (unless bootstrapped) + when no snapshots exist by @krinart in #9119
• fix(cayenne): Fix upsert with pending deletions causing duplicate PKs by @sgrebnov in #9152
• fix(flightrepl): Add chrono-tz feature to flightrepl for timezone formatting by @sgrebnov in #9153
• fix(delta_lake): Preserve container name in ABFSS URLs for Azure Delta Lake tables by @sgrebnov in #9155
• fix: Make CLI system and asset type detection more robust by @peasee in #9148
• fix: Set query set properly on benchmarks telemetry metrics attributes by @peasee in #9162
• fix: Download _models variant - https://github.com/spiceai/spiceai/commit/27f3058d0007595b02c198755c3b22319032ff30/
• fix: Helm chart image tag - https://github.com/spiceai/spiceai/commit/7405c8df0db4ecce0ed6d4a4d424553d604b9036/
• Revert "Remove models variant (now default) & Windows builds (use WSL) " by @lukekim in #9063
• fix(cli): Several CLI fixes from the Go to Rust migration by @lukekim in #9157

Announcing the release of Spice v1.11.0-rc.2! ⭐

v1.11.0-rc.2 is the second release candidate for advanced test of v1.11. It brings Spice Cayenne to Beta status with acceleration snapshots support, a new ScyllaDB Data Connector, upgrades to DataFusion v51, Arrow 57.2, and iceberg-rust v0.8.0. It includes significant improvements to distributed query, caching, and observability.

What's New in v1.11.0-rc.2​

Spice Cayenne Accelerator Reaches Beta​

Spice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous stability improvements.

Improved Reliability:

• Fixed timezone database issues in Docker images that caused acceleration panics
• Resolved FuturesUnordered reentrant drop crashes
• Fixed memory growth issues related to Vortex metrics allocation
• Metadata catalog now properly respects cayenne_file_path location
• Added warnings for unparseable configuration values

Example configuration with snapshots:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file

DataFusion v51 Upgrade​

Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.

Performance Improvements:

• Faster CASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patterns
• Better Defaults for Remote Parquet Reads: DataFusion now fetches the last 512KB of Parquet files by default, typically avoiding 2 I/O requests per file
• Faster Parquet Metadata Parsing: Leverages Arrow 57's new thrift metadata parser for up to 4x faster metadata parsing

New SQL Features:

• SQL Pipe Operators: Support for |> syntax for inline transforms
• DESCRIBE <query>: Returns the schema of any query without executing it
• Named Arguments in SQL Functions: PostgreSQL-style param => value syntax for scalar, aggregate, and window functions
• Decimal32/Decimal64 Support: New Arrow types supported including aggregations like SUM, AVG, and MIN/MAX

Example pipe operator:

SELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;

Improved Observability:

• Improved EXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdowns

Arrow 57.2 Upgrade​

Spice has been upgraded to Apache Arrow Rust 57.2.0, bringing major performance improvements and new capabilities.

Key Features:

• 4x Faster Parquet Metadata Parsing: A rewritten thrift metadata parser delivers up to 4x faster metadata parsing, especially beneficial for low-latency use cases and files with large amounts of metadata
• Parquet Variant Support: Experimental support for reading and writing the new Parquet Variant type for semi-structured data, including shredded variant values
• Parquet Geometry Support: Read and write support for Parquet Geometry types (GEOMETRY and GEOGRAPHY) with GeospatialStatistics
• New arrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution support

iceberg-rust v0.8.0 Upgrade​

Spice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.

Key Features:

• V3 Metadata Support: Full support for Iceberg V3 table metadata format
• INSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tables
• Improved Delete File Handling: Better support for position and equality delete files, including shared delete file loading and caching
• SQL Catalog Updates: Implement update_table and register_table for SQL catalog
• S3 Tables Catalog: Implement update_table for S3 Tables catalog
• Enhanced Arrow Integration: Convert Arrow schema to Iceberg schema with auto-assigned field IDs, _file column support, and Date32 type support

Acceleration Snapshots​

Acceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.

Key Feature Improvements in v1.11:

• Flexible Triggers: Configure when snapshots are created based on time intervals or stream batch counts
• Automatic Compaction: Reduce storage overhead by compacting older snapshots (DuckDB only)
• Bootstrap Integration: Snapshots can reset cache expiry on load for seamless recovery (DuckDB with Caching refresh mode)
• Smart Creation Policies: Only create snapshots when data has actually changed

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled
      snapshots_trigger: time_interval
      snapshots_trigger_threshold: 1h
      snapshots_creation_policy: on_changed

Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically. List, create, and restore snapshots directly from the command line or via HTTP.

For more details, refer to the Acceleration Snapshots Documentation.

ScyllaDB Data Connector​

A new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.

Example configuration:

datasets:
  - from: scylladb:my_keyspace.my_table
    name: scylla_data
    acceleration:
      enabled: true
      engine: duckdb

For more details, refer to the ScyllaDB Data Connector Documentation.

Distributed Query Improvements​

mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.

Credential Propagation: Azure and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.

Improved Resilience:

• Exponential backoff for scheduler disconnection recovery
• Increased gRPC message size limit from 16MB to 100MB for large query plans
• HTTP health endpoint for cluster executors
• Automatic executor role inference when --scheduler-address is provided

For more details, refer to the Distributed Query Documentation.

Caching Acceleration Mode Improvements​

The Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.

Performance Optimizations:

• Non-blocking Cache Writes: Cache misses no longer block query responses. Data is written to the cache asynchronously after the query returns, reducing query latency for cache miss scenarios.
• Batch Cache Writes: Multiple cache entries are now written in batches rather than individually, significantly improving write throughput for high-volume cache operations.

Reliability Fixes:

• Correct SWR Refresh Behavior: The stale-while-revalidate (SWR) pattern now correctly refreshes only the specific entries that were accessed instead of refreshing all stale rows in the dataset. This prevents unnecessary source queries and reduces load on upstream data sources.
• Deduplicated Refresh Requests: Fixed an issue where JSON array responses could trigger multiple redundant refresh operations. Refresh requests are now properly deduplicated.
• Fixed Cache Hit Detection: Resolved an issue where queries that didn't include fetched_at in their projection would always result in cache misses, even when cached data was available.
• Unfiltered Query Optimization: SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.

For more details, refer to the Caching Acceleration Mode Documentation.

DynamoDB Connector Enhancements​

• Added JSON nesting for DynamoDB Streams
• Proper batch deletion handling

URL Tables​

Query data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.

Supported Patterns:

• Single files: SELECT * FROM 's3://bucket/data.parquet'
• Directories/prefixes: SELECT * FROM 's3://bucket/data/'
• Glob patterns: SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'

Key Features:

• Automatic file format detection (Parquet, CSV, JSON, etc.)
• Hive-style partition inference with filter pushdown
• Schema inference from files
• Works with both SQL and DataFrame APIs

Example with hive partitioning:

-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'

Enable via spicepod.yml:

runtime:
  params:
    url_tables: enabled

Cluster Mode Async Query APIs (experimental)​

New asynchronous query APIs for long-running queries in cluster mode:

• /v1/queries endpoint: Submit queries and retrieve results asynchronously
• Arrow Flight async support: Non-blocking query execution via Arrow Flight protocol

Observability Improvements​

Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:

• Snapshot monitoring widgets
• Improved accelerated datasets section
• Renamed ingestion lag charts for clarity

Additional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.

For more details, refer to the Monitoring Documentation.

Additional Improvements​

• Model Listing: New functionality to list available models across multiple AI providers
• DuckDB Partitioned Tables: Primary key constraints now supported in partitioned DuckDB table mode
• Post-refresh Sorting: New on_refresh_sort_columns parameter for DuckDB enables data ordering after writes
• Improved Install Scripts: Removed jq dependency and improved cross-platform compatibility
• Better Error Messages: Improved error messaging for bucket UDF arguments and deprecated OpenAI parameters

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

New ScyllaDB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.

New SMB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See SMB Data Connector Recipe for details.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.0-rc.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:v1.11.0-rc.2 image:

docker pull spiceai/spiceai:v1.11.0-rc.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

Spice is available in the AWS Marketplace.

Dependencies​

• DataFusion: Upgraded to v51 (release notes)
• Arrow: Upgraded to v57 (release notes)
• iceberg-rust: Upgraded to v0.8.0 (release notes)

Changelog​

• Add timezone database to Docker image to fix Cayenne acceleration panic by @sgrebnov in #8799
• Upgrade dependencies by @phillipleblanc in #8801
• Fix table_allowlist for table sampling and NSQL by @Jeadie in #8789
• Cayenne primary key on-conflict handling by @lukekim in #8788
• fix: Update benchmark snapshots by @app/github-actions in #8773
• fix: correctly identify deprecated openai_* parameters by @phillipleblanc in #8809
• fix: Update benchmark snapshots by @app/github-actions in #8812
• Use workspace version for cayenne crate by @phillipleblanc in #8811
• Don't CAST strings which breaks push down optimizer by @lukekim in #8810
• fix: Update benchmark snapshots by @app/github-actions in #8815
• Update async-openai to latest revision 4dcd633aad6f - brings fix for openai compatible model providers by @ewgenius in #8816
• Add auth/iam_role_source to DynamoDB connector by @krinart in #8808
• DynamoDB fixes: JSON nesting for Streams, proper batch deletions by @krinart in #8821
• Rough roadmap for 2026-2027 by @lukekim in #8805
• Release notes for v1.11.0-rc1 by @ewgenius in #8786
• Make S3V integration tests prepare_for_aws_tests more robust by @sgrebnov in #8820
• Bump rsa from 0.9.9 to 0.9.10 in the cargo group across 1 directory by @app/dependabot in #8819
• Add timezone database to Release and CUDA Docker images to fix Cayene panic by @sgrebnov in #8832
• fix: UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831
• Update grafana/datadog example dashboards by @krinart in #8833
• Add Dev bird bench as text-to-sql queryset in CI. by @Jeadie in #8753
• Update testoperator scheduler to use release/1.11 branch by @ewgenius in #8829
• Spice Cayenne fixes and test spicepods for Beta & RC by @lukekim in #8787
• testoperator dispatch all bird-bench database variants by @Jeadie in #8835
• feat: Improve column statistics handling with safe access and defaults by @phillipleblanc in #8836
• cluster: mTLS verification by @phillipleblanc in #8837
• fix: 8770: Unsupported ScalarFunctionExpr in ORDER BY by @lukekim in #8838
• Workflow tweaks by @lukekim in #8845
• Cayenne: metadata catalog should respect cayenne_file_path location by @sgrebnov in #8844
• Expand Cayenne feature coverage by @lukekim in #8848
• docs: HA distributed query decisions by @phillipleblanc in #8817
• fix(optimizer): Fix correctness issues in UnionProjectionPushdownOptimizer by @phillipleblanc in #8851
• Pin reqwest to 0.12.24 to fix HuggingFace embedding model download by @ewgenius in #8853
• Fix builds and pin to Ubuntu 22.04 by @lukekim in #8856
• Revert "Fix builds and pin to Ubuntu 22.04" by @lukekim in #8861
• Ensure setup Rust is run by @lukekim in #8862
• fix: Ubuntu 24.04+ renamed libaio1 to libaio1t64 by @lukekim in #8865
• Upgrade to Pulls with Spice v2 by @lukekim in #8866
• Add limit and configuration name to 'testoperator run text-to-sql' by @Jeadie in #8839
• PR check and test optimization by @lukekim in #8868
• Upgrade S3 Vectors SDK and improve test robustness by @lukekim in #8867
• [Testoperator] Query level and improved aggregate level for NSQL by @Jeadie in #8840
• Add docker build for private branches for ghcr.io/spiceai/spiceai-dev by @phillipleblanc in #8873
• Expand the data acceleration round-trip test coverage by @lukekim in #8855
• fix: Provide a better error for improper bucket UDF arguments by @peasee in #8849
• ScyllaDB Data Connector by @lukekim in #8827
• Use tokio-rusqlite for Spice Cayenne SQLite by @lukekim in #8857
• Cayenne: fix FuturesUnordered reentrant drop crash by @sgrebnov in #8863
• Bump github/codeql-action from 4.31.9 to 4.31.10 by @app/dependabot in #8884
• Bump golang.org/x/sys from 0.39.0 to 0.40.0 by @app/dependabot in #8881
• Bump github.com/spiceai/gospice/v8 from 8.0.0 to 8.0.1 by @app/dependabot in #8883
• Bump roaring from 0.11.2 to 0.11.3 by @app/dependabot in #8885
• Bump golang.org/x/mod from 0.31.0 to 0.32.0 by @app/dependabot in #8882
• Bump aws-sdk-s3 from 1.115.0 to 1.119.0 by @app/dependabot in #8887
• Bump libc from 0.2.177 to 0.2.180 by @app/dependabot in #8886
• Bump tokio-util from 0.7.17 to 0.7.18 by @app/dependabot in #8889
• Bump governor from 0.10.2 to 0.10.4 by @app/dependabot in #8888
• fix: flaky test test_concurrent_partition_creation by @phillipleblanc in #8898
• Update Cayenne snapshots for TPC-DS by @lukekim in #8890
• Add more buckets to histogram metrics by @krinart in #8850
• feat: Add HTTP health endpoint for cluster executors by @phillipleblanc in #8899
• feat: Implement model listing functionality for multiple providers by @lukekim in #8901
• feat: Initial HA schedulers distributed query implementation by @phillipleblanc in #8852
• fix: infer executor role from --scheduler-address when --role is omitted by @phillipleblanc in #8903
• Improve install scripts and remove jq dependency by @lukekim in #8847
• Benchmarks: sort PartitionedUnionExec children for deterministic snapshot comparison by @sgrebnov in #8877
• Cayenne: share VortexFileCache across partitions via CayenneContext by @sgrebnov in #8880
• Update ballista to add exponential backoff for scheduler disconnection by @phillipleblanc in #8905
• Configurably add BirdBench evidence to testoperator text-to-SQL. by @Jeadie in #8904
• Helm: Allow command override via values.yaml by @sgrebnov in #8906
• Fix distributed query gRPC message size limit (16MB -> 100MB) by @phillipleblanc in #8900
• OS specific setup actions by @lukekim in #8909
• Cayenne should warn if unable to parse configuration value by @sgrebnov in #8907
• Add snapshots widgets to example dashboard by @krinart in #8910
• Add quality criteria for the features by @krinart in #8897
• Improve Accelerated Datasets section for Grafana/Datadog dashboards by @krinart in #8915
• Use HTTP traceparent in NSQL to support concurrency in 'testoperator run text-to-SQL' by @Jeadie in #8912
• Remove setup for cc from integration_models.yml by @Jeadie in #8917
• Propagate Azure and GCS credentials to executors in cluster mode by @phillipleblanc in #8918
• Cayenne: fix memory growth due to vortex metrics allocation by @sgrebnov in #8908
• fix(caching): Deduplicate refresh requests for JSON array responses by @sgrebnov in #8921
• fix(caching): Return cached data directly for unfiltered queries (SELECT *) by @sgrebnov in #8919
• Correct MinIO path syntax for spiced download by @Jeadie in #8916
• Acceleration snapshots compaction + Improved Snapshots UX by @krinart in #8858
• Change base image from bookworm-slim to trixie-slim by @Jeadie in #8923
• Add testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895
• Fix spicepod dependencies in testoperator by @Jeadie in #8875
• Update copilot instructions for data correctness by @lukekim in #8922
• Add BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926
• fix: add missing feature-gate for AWS Secrets Manager error variant by @phillipleblanc in #8928
• refactor: make ConnectorParams fields public for external connectors by @phillipleblanc in #8929
• fix(caching): SWR refreshes only accessed entry instead of all stale rows by @sgrebnov in #8931
• Cayenne: include cayenne_metadata_dir to known params by @sgrebnov in #8933
• Rename Ingestion Lag chart in example dashboards by @krinart in #8932
• fix(caching): Fix HTTP caching always MISS when projection excludes fetched_at by @sgrebnov in #8930
• Reset expiry after snapshot bootstraping for Caching by @krinart in #8925
• Set use_ssl=false for sccache by @lukekim in #8945
• Hash indexing for Arrow Acceleration by @lukekim in #8924
• [Cayenne] Acceleration snapshots support by @lukekim in #7973
• perf(caching): Non-blocking cache writes on cache miss by @sgrebnov in #8948
• Update NSQL models by @lukekim in #8951
• Hash Index Key verification by @lukekim in #8949
• Add snapshots_creation_policy param by @krinart in #8954
• Remove candle & cudarc from non-models build by @lukekim in #8955
• Acceleration Snapshots API and CLI by @lukekim in #8934
• Ignore test for data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962
• fix: Update benchmark snapshots by @app/github-actions in #8965
• Hash Index secondary index support by @lukekim in #8958
• fix: Support primary key constraints in partitioned DuckDB tables mode by @sgrebnov in #8966
• perf(caching): Batch cache writes by @sgrebnov in #8959
• CI perf optimizations by @lukekim in #8968
• Fix Makefile linting by @Jeadie in #8970
• Fixes in testoperator run text-to-sql. by @Jeadie in #8927
• implement Chat::as_sql for xAI anthropic by @Jeadie in #8957
• Fix duckdb_file_path in search integration test by @Jeadie in #8972
• fix: Update benchmark snapshots by @app/github-actions in #8971
• Maintenance updates to Anthropic API by @Jeadie in #8956
• Add CacheBackend Trait, implement pingora-lru, and add throughput tests by @lukekim in #8080
• fix: Update benchmark snapshots by @app/github-actions in #8974
• fix: Update benchmark snapshots by @app/github-actions in #8975
• Make accelerator shutdown more robust by @lukekim in #8969
• feat(duckdb): Add on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964
• Proper handling for initial snapshot by @krinart in #8911
• fix: Remove --no-default-features from cargo-hack command in features workflow by @phillipleblanc in #8977
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 by @app/dependabot in #8983
• build(deps): bump actions/checkout from 4 to 6 by @app/dependabot in #8982
• build(deps): bump actions/setup-go from 6.1.0 to 6.2.0 by @app/dependabot in #8984
• build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 by @app/dependabot in #8979
• build(deps): bump github.com/klauspost/compress from 1.18.2 to 1.18.3 by @app/dependabot in #8980
• Add /v1/queries and Arrow Flight async APIs by @lukekim in #8946
• build(deps): bump Vampire/setup-wsl from 5 to 6 by @app/dependabot in #8981
• fix: Update Search integration test snapshots by @app/github-actions in #8973
• build(deps): bump insta from 1.46.0 to 1.46.1 by @app/dependabot in #8988
• build(deps): bump schemars from 1.1.0 to 1.2.0 by @app/dependabot in #8985
• fix: Update benchmark snapshots by @app/github-actions in #8978
• fix: Data correctness edge cases by @lukekim in #8953
• Correct MinIO path syntax for spiced download (Part 2) by @Jeadie in #8995
• Make .spice/data in search integration tests by @Jeadie in #8992
• fix: Hash index composite keys null values by @lukekim in #9001
• Update Cayenne status to Beta by @lukekim in #9002
• fix: Disable TPC-DS result validation (not yet supported) by @sgrebnov in #9004
• feat: Upgrade to DataFusion v51 and dependencies by @lukekim in #8864
• Improvements for snapshots_creation_policy by @krinart in #9003
• fix(ci): restore cached spicepod-validator binary instead of lookup-only by @phillipleblanc in #9007
• Update version by @krinart in #9010

Announcing the release of Spice v1.10.2! 🔥

v1.10.2 introduces Tiered Caching Acceleration with Localpod for multi-layer acceleration architectures, Periodic Acceleration Snapshots with configurable intervals, DynamoDB JSON Nesting for column consolidation, and Kafka/Debezium Batching for faster data ingestion. This release also includes fixes for SQLite accelerator decimal/date handling and real-time status reporting for the /v1/datasets and /v1/models API endpoints.

What's New in v1.10.2​

Tiered Caching with Localpod​

Multi-Layer Acceleration Architecture: The Localpod connector now supports caching refresh mode, enabling tiered acceleration where a persistent cache (e.g., file-mode DuckDB) feeds a fast in-memory cache (e.g., Arrow, memory-mode DuckDB).

Key Features:

• Automatic Cache Propagation: New cache entries automatically propagate from parent to child accelerators
• Warm Startup: Child accelerators initialize from existing parent data on startup, eliminating cold-start latency
• Flexible Tiering: Combine any accelerator engines (DuckDB, SQLite, Cayenne) across tiers

Example spicepod.yaml configuration:

datasets:
  # Parent: persistent file-mode cache
  - from: https://api.example.com
    name: api_cache
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file

  # Child: fast in-memory cache fed by parent
  - from: localpod:api_cache
    name: api_cache_memory
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: arrow
      mode: memory

For more details, refer to the Localpod Data Connector Documentation.

Periodic Acceleration Snapshots​

Configurable Snapshot Intervals: A new snapshots_create_interval parameter enables periodic snapshot creation for accelerated datasets across all refresh modes. This provides better control over snapshot frequency and ensures consistent recovery points for accelerated data.

Example spicepod.yaml configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: caching
      snapshots: enabled
      params:
        snapshots_create_interval: 60s # Write a snapshot every 60 seconds

For more details, refer to the Data Acceleration Documentation.

DynamoDB JSON Nesting​

Consolidate Columns into JSON: The DynamoDB Data Connector now supports consolidating columns into a single JSON column using the json_object: "*" metadata option. This is useful when only a few columns are needed as discrete fields while the rest can be accessed as nested JSON.

Example spicepod.yaml configuration:

datasets:
  - from: dynamodb:my_table
    name: my_table
    columns:
      - name: PK
      - name: SK
      - name: data_json
        metadata:
          json_object: '*' # Captures all other columns as JSON

Example Output: Given a DynamoDB table with columns PK, SK, name, email, and status, the resulting table schema consolidates all non-specified columns into the data_json column:

For more details, refer to the DynamoDB JSON Nesting Documentation.

Kafka/Debezium Batching​

Faster Data Ingestion: Configure message batching for Kafka and Debezium connectors to improve data ingestion throughput. Batching reduces processing overhead by grouping multiple messages together before insertion.

Key Features:

• Configurable Batch Size: Control the maximum number of records per batch (default: 10,000)
• Configurable Batch Duration: Set the maximum wait time before flushing a partial batch (default: 1s)

Example spicepod.yaml configuration:

datasets:
  - from: debezium:kafka-server.public.my_table
    name: my_table
    params:
      batch_max_size: 10000 # Max records per batch (default: 10000)
      batch_max_duration: 1s # Max wait time per batch (default: 1s)

For more details, refer to the Kafka Data Connector Documentation and Debezium Data Connector Documentation.

Additional Improvements & Bug Fixes​

• Reliability: Fixed SQLite accelerator decimal and date type handling for improved data type accuracy.
• Reliability: Fixed real-time status reporting for /v1/datasets and /v1/models API endpoints.
• Reliability: Fixed Kafka warning when security.protocol is set to PLAINTEXT.

Contributors​

• @phillipleblanc
• @lukekim
• @krinart

Breaking Changes​

No breaking changes.

Cookbook Updates​

New Cayenne Data Accelerator Recipe: New recipe demonstrating how to accelerate a local copy of the taxi trips dataset using Cayenne as the data accelerator engine. See Cayenne Data Accelerator Recipe for details.

New Dataset Partitioning Recipe: New recipe demonstrating how to partition accelerated datasets to improve query performance. See Dataset Partitioning for details.

The Spice Cookbook includes 84 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.10.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.2 image:

docker pull spiceai/spiceai:1.10.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• Fix kafka warning when security.protocol is set to PLAINTEXT by @krinart in #8587
• fix: SQLite accelerator decimal/date handling by @phillipleblanc in #8606
• feat: Enable localpod with caching mode accelerator for tiered caching by @phillipleblanc in #8621
• Remove the clippy::too_many_lines lint by @phillipleblanc in #8549
• Add snapshot interval for acceleration snapshots by @phillipleblanc in #8627
• Json Nesting for DynamoDB by @krinart in #8623
• Implement batching for Kafka/Debezium + null Decimal handling by @krinart in #8622
• fix: Status field in /v1/datasets & /v1/models by @lukekim in #8633

Announcing the release of Spice v1.10.0! ⚡

Spice v1.10.0 introduces a new Caching Acceleration Mode with stale-while-revalidate (SWR) semantics for disk-persisted, low-latency queries with background refresh. This release also adds the TinyLFU eviction policy for the SQL results cache, a preview of the DynamoDB Streams connector for real-time CDC, S3 location predicate pruning for faster partitioned queries, improved distributed query execution, and multiple security hardening improvements.

What's New in v1.10.0​

Caching Acceleration Mode​

Low-Latency Queries with Background Refresh: This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern. Queries return cached results immediately while data refreshes asynchronously in the background, eliminating query latency spikes during refresh cycles. Cached data persists to disk using DuckDB, SQLite, or Cayenne file modes.

Key Features:

• Stale-While-Revalidate (SWR): Returns cached data immediately while refreshing in the background, reducing query latency
• Disk Persistence: Cached results persist across restarts using DuckDB, SQLite, or Cayenne file modes
• Configurable Refresh: Control refresh intervals with refresh_check_interval to balance freshness and source load

Recommendation: Use retention configuration with caching acceleration to ensure stale data is cleaned up over time.

Example spicepod.yaml configuration:

datasets:
  - from: http://localhost:7400
    name: cached_data
    time_column: fetched_at
    acceleration:
      enabled: true
      engine: duckdb
      mode: file # Persist cache to disk
      refresh_mode: caching
      refresh_check_interval: 10m
      retention_check_enabled: true
      retention_period: 24h
      retention_check_interval: 1h

For more details, refer to the Data Acceleration Documentation.

TinyLFU Cache Eviction Policy​

Higher Cache Hit Rates for SQL Results Cache: A new TinyLFU cache eviction policy is now available for the SQL results cache. TinyLFU is a probabilistic cache admission policy that maintains higher hit rates than LRU while keeping memory usage predictable, making it ideal for workloads with varying query frequency patterns.

Example spicepod.yaml configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      eviction_policy: tiny_lfu # default: lru

For more details, refer to the Caching Documentation and the Moka TinyLFU Documentation for details of the algorithm.

DynamoDB Streams Data Connector (Preview)​

Real-Time Change Data Capture for DynamoDB: The DynamoDB connector now integrates with DynamoDB Streams for real-time change data capture (CDC). This enables continuous synchronization of DynamoDB table changes into Spice for real-time query, search, and LLM-inference.

Key Features:

• Real-Time CDC: Automatically captures inserts, updates, and deletes from DynamoDB tables as they occur
• Table Bootstrapping: Performs an initial full table scan before streaming changes, ensuring complete data consistency
• Acceleration Integration: Works with refresh_mode: changes to incrementally update accelerated datasets

Note: DynamoDB Streams must be enabled on your DynamoDB table. This feature is in preview.

Example spicepod.yaml configuration:

datasets:
  - from: dynamodb:my_table
    name: orders_stream
    acceleration:
      enabled: true
      refresh_mode: changes # Enable Streams capture

For more details, refer to the DynamoDB Connector Documentation.

OpenTelemetry Metrics Exporter​

Spice can now push metrics to an OpenTelemetry collector, enabling integration with platforms such as Jaeger, New Relic, Honeycomb, and other OpenTelemetry-compatible backends.

Key Features:

• Protocol Support: Supports the gRPC (default port 4317) protocol
• Configurable Push Interval: Control how frequently metrics are pushed to the collector

Example spicepod.yaml configuration for gRPC:

runtime:
  telemetry:
    enabled: true
    otel_exporter:
      endpoint: 'localhost:4317'
      push_interval: '30s'

For more details, refer to the Observability & Monitoring Documentation.

S3 Connector Improvements​

S3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down location filter predicates to S3 listing operations. For partitioned datasets (e.g., year=2025/month=12/), Spice now skips listing irrelevant partitions entirely, significantly reducing query latency and S3 API costs.

AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling direct integration with AWS's managed table format for S3. Use standard SQL INSERT INTO to write data.

For more details, refer to the S3 Data Connector Documentation and Glue Data Connector Documentation.

Faster Distributed Query Execution​

Distributed query planning and execution have been significantly improved:

• Fixed executor registration in cluster mode for more reliable distributed deployments
• Improved hostname resolution for Flight server binding, enabling better executor discovery
• Distributed accelerator registration: Data accelerators now properly register in distributed mode
• Optimized query planning: DistributeFileScanOptimizer improvements for faster planning with large datasets

For more details, refer to the Distributed Query Documentation.

Search Improvements​

Search capabilities have been improved with several performance and reliability enhancements:

• Fixed FTS query blocking: Full-text search queries no longer block unnecessarily, improving query responsiveness
• Optimized vector index operations: Eliminated unnecessary list_vectors calls for better performance
• Improved limit pushdown: IndexerExec now properly handles limit pushdown for more efficient searches

For more details, refer to the Search Documentation.

Security Hardening​

Multiple security improvements have been implemented:

• SQL Identifier Quoting: Hardened SQL identifier quoting across all database connectors (PostgreSQL, MySQL, DuckDB, etc.) to prevent SQL injection attacks through table or column names
• Token Redaction: Sensitive authentication tokens are now fully redacted in debug and error output, preventing accidental credential exposure in logs
• Path Traversal Prevention: Fixed tar extraction operations to prevent directory traversal vulnerabilities when processing archived files
• Input Sanitization: Added strict validation for top_n_sample order_by clause parsing to prevent injection attacks
• Glue Credential Handling: Prevented automatic loading of AWS credentials from environment in Glue connector, ensuring explicit credential configuration

Developer Experience Improvements​

• Health probe metrics: Added health probe latency metrics for better observability
• CLI improvements: Fixed .clear history command in the REPL to fully clear persisted history

Contributors​

• @lukekim
• @sgrebnov
• @peasee
• @krinart
• @Jeadie
• @mach-kernel
• @phillipleblanc
• @ewgenius

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.10.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.0 image:

docker pull spiceai/spiceai:1.10.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• Test-operator: Add tpcds_q8 to the default row-count validation skip list by @sgrebnov in #8185
• fix: Remove unwrap_used from test by @peasee in #8212
• Run glue_iceberg_integration_test_catalog as part of main integration tests by @sgrebnov in #8222
• Add TPCH sf100 testoperator spicepods with dispatch by @Jeadie in #8192
• Build with CPU native flags by @lukekim in #8224
• fix: Apply assertion clippy in CI/Makefile only by @peasee in #8229
• feat: Support running queries only in testoperator by @peasee in #8211
• DuckDB query planning: aggregate pushdown by @mach-kernel in #8174
• install.sh improvements by @lukekim in #8252
• Fix .clear history by @lukekim in #8254
• Improve the output of dataset loading by @lukekim in #8256
• Refactor view validation by @lukekim in #8258
• Upgrade AWS crates by @lukekim in #8259
• fix: Pushdown dynamic filters to partition scans by @peasee in #8240
• Harden SQL identifier quoting in connectors by @phillipleblanc in #8276
• Cayenne sort_columns on insert by @lukekim in #8091
• Redact token debug output by @phillipleblanc in #8280
• fix: Cayenne configuration options by @lukekim in #8281
• Prevent path traversal in untar by @phillipleblanc in #8284
• Fix cluster mode executor registration by @mach-kernel in #8292
• Unignore s3_vectors_kafka_stream test by @Jeadie in #8289
• Post-release house keeping by @krinart in #8293
• Improve generate_changelog script by @krinart in #8273
• Acceleration mode caching by @lukekim in #8237
• Sanitization and security checks by @lukekim in #7854
• Add health probe latency metric by @phillipleblanc in #8300
• Add distributed registration for data accelerators by @phillipleblanc in #8299
• Pass IndexedTableProvider down in 'changes_stream' and 'append_stream' by @Jeadie in #8295
• Add dynamodb-streams crate by @krinart in #8283
• Distributed query: resolve executor hostname when determining Flight server binding by @mach-kernel in #8304
• Return computed embeddings from index for partitioned S3Vectors by @Jeadie in #8306
• [DDB Streams] Skeleton for DynamoDB Streams by @krinart in #8296
• DistributeFileScanOptimizer: Improve planning performance by @mach-kernel in #8305
• feat: Add an ExactLeftAccumulator implementation by @peasee in #8302
• deps: Upgrade Vortex to 0.56 by @peasee in #8311
• DynamoDB table bootstrapping + streaming by @krinart in #8312
• Avoid calling S3Vector list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie in #8282
• Add on_conflict testing support to append benchmark by @sgrebnov in #8314
• docker: Add valid home directory to fix duckdb extension loading issue by @phillipleblanc in #8318
• Add GH Workflow to run Append benchmark test by @sgrebnov in #8321
• Exclude MySQL SF100 from test-operator dispatch by @sgrebnov in #8320
• feat: Update clippy lints by @peasee in #8317
• Add S3 location predicate pruning to listing connector by @phillipleblanc in #8319
• Review feedback for caching mode accelerator by @phillipleblanc in #8326
• Also include Dockerfile home changes for release build by @phillipleblanc in #8327
• Change communication channel from Discord to Slack by @Jeadie in #8330
• Replace Discord link with Slack link in README by @Jeadie in #8331
• fix(glue): Prevent OpenDAL from automatic loading of AWS credentials from environment by @sgrebnov in #8337
• Block on index read for FTS queries by @Jeadie in #8339
• Fix search query provider by @Jeadie in #8343
• Support for writing into AWS S3 Tables by @sgrebnov in #8344
• Acceleration file_create mode by @lukekim in #8347
• Don't block on lock in FTS query path by @Jeadie in #8348
• feat: Add an optimizer rule to replace join accumulator for Cayenne by @peasee in #8316
• S3 Vectors limit updates by @lukekim in #8352
• Sanitize top_n_sample order_by parsing by @phillipleblanc in #8356
• Add distributed registration for data connectors by @phillipleblanc in #8354
• Improve IndexerExec to properly handle limit pushdown by @sgrebnov in #8366
• Fix Cayenne partition_by metadata flaky integration test by @phillipleblanc in #8367
• Rework caching accelerator to use the stale-while-revalidate pattern. by @phillipleblanc in #8365
• Add TinyLFU caching policy by @lukekim in #8370
• Make Arrow acceleration on_conflict verification more robust by @sgrebnov in #8375
• Add additional test for verify_on_conflict_matches_primary_key (Arrow acceleration) by @sgrebnov in #8376
• Add v1.10.0-rc1 release notes by @mach-kernel in #8373
• docs: Remove DuckDB agg pushdown from release notes by @peasee in #8383
• Testoperator dispatch: add Append support and test configurations by @sgrebnov in #8360
• fix: Increase TPCDS DuckDB connection pool size by @peasee in #8386
• fix: Update benchmark snapshots by @app/github-actions in #8385
• fix: Update benchmark snapshots by @app/github-actions in #8389
• Fix Windows build by @phillipleblanc in #8391
• DuckDB aggregate pushdown: fix partitioning and schema rewrite bugs by @mach-kernel in #8397
• Delta table: Store current snapshot ref with table instance by @mach-kernel in #8358
• GetAppDefinition: Check if executor is part of cluster by @mach-kernel in #8396
• 1.10.0-rc1 housekeeping by @mach-kernel in #8394
• Change debug log to warning for vector engine config by @Jeadie in #8378
• Clarify /v1/nsql datasets sampling hint by @phillipleblanc in #8395
• Use bmi1 target feature for x86_64 by @phillipleblanc in #8401
• benchmarks: Default to update snapshots when run on a non-release branch by @phillipleblanc in #8402
• Update threat model for v1.9.2 by @phillipleblanc in #8400
• Fix iceberg tables metadata - assign ids to all fields, including nested by @ewgenius in #8351
• Fix databricks_spark_connect_m2m_integration_test_catalog snapshot by @ewgenius in #8403
• Move all GitHub Actions workflows to use exact commit sha by @phillipleblanc in #8409
• Upgrade datafusion-tableproviders (df v50) by @lukekim in #8261
• Batching for CDC by @krinart in #8359
• fix: make DuckDB attachments logic more robust by @sgrebnov in #8411
• Persistent checkpoints for DynamoDB Streams by @krinart in #8345
• Distributed query: Support AsyncFuncExec and Spice UDFs in Ballista by @mach-kernel in #8414
• Pin GitHub Actions to fix Testoperator build action by @sgrebnov in #8416
• Watermarks support for DynamoDB by @krinart in #8417
• Fix typo in .vscode/launch.json by @sgrebnov in #8415
• DuckSqlExec: Update equivalence properties when rewriting schema by @phillipleblanc in #8420
• Validate that the commit for datafusion-table-providers exists on the spiceai branch by @phillipleblanc in #8421
• Append Tests: add support for retention testing by @sgrebnov in #8419
• New crate google-genai by @Jeadie in #8390
• federation: Improve error message and add debug logging for cast failures by @phillipleblanc in #8422
• DynamoDB Streams Error Handling by @krinart in #8418
• Append tests: add support for with_retention_data to dispatch by @sgrebnov in #8430
• Append tests: add support for test metrics reporting by @sgrebnov in #8432
• test-operator: fix metrics reporting by @sgrebnov in #8435
• Follow-up improvements and bug fixes by @krinart in #8433
• Periodic snapshots for append/changes streams by @krinart in #8407
• Add support for caching_stale_if_error to caching accelerator; fix multiple upstream requests during SWR; fix Arrow accelerator by @phillipleblanc in #8425
• Disable dataset health monitor for dynamic HTTP connector by @phillipleblanc in #8441
• Metrics + snapshots_trigger_threshold for DynamoDB Streams by @krinart in #8437
• Clear the in-flight revalidations cache after a revalidation has completed. by @phillipleblanc in #8443
• dont build 'spicepod-validator' on 'make install' by @Jeadie in #8426
• fix: Update benchmark snapshots by @app/github-actions in #8436
• fix: Disable Cayenne HashJoin rewriter optimizer by @peasee in #8439
• Testoperator: add duckdb-partitioned query override by @sgrebnov in #8446
• Add a check to validate that results cache SWR and caching accelerator SWR are not both set. by @phillipleblanc in #8445
• OTel exporter for push metrics by @lukekim in #8442
• fix: Update benchmark snapshots by @app/github-actions in #8448
• Add snapshot creation logging by @krinart in #8469
• Fix PeriodicReader panic by @krinart in #8471
• fix: Pin CUDA build actions to commits by @peasee in #8477
• DuckDB agg pushdown: gate behind accelerator parameter by @mach-kernel in #8474
• Rename aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485
• Handle throttling exception for DynamoDB streams by @phillipleblanc in #8492

Announcing the release of Spice v1.10.0-rc.1! ⚡

v1.10.0-rc1 is a release candidate for early testing of v1.10 features including an all new caching acceleration mode, tiny_lfu caching policy, a new DynamoDB Streams connector (Preview), improvements to the DynamoDB connector, faster distributed query execution, S3 connector improvements, and security hardening for v1.10.0-stable.

What's New in v1.10.0-rc1​

Caching Acceleration Mode with SWR and TinyLFU​

This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern using Data Accelerators such as DuckDB or Cayenne, enabling queries to return file-persisted cached results immediately while asynchronously refreshing data in the background. Combined with the new TinyLFU cache eviction policy, Spice can now maintain higher cache hit rates while keeping memory usage predictable.

Key Features:

• Stale-While-Revalidate (SWR): Returns cached data immediately while refreshing in the background
• Data Accelerator Support: Cached accelerators can persist data to disk using DuckDB, SQLite, or Cayenne file modes.
• TinyLFU Cache Policy: Probabilistic cache admission policy that maintains high hit rates with minimal overhead
• Predictable Memory Usage: Configurable memory limits with automatic eviction of less frequently used entries

Example Spicepod.yml configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      eviction_policy: tiny_lfu # default lru

datasets:
  - from: s3://my-bucket/data.parquet
    name: cached_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file # Persist cache to disk
      refresh_mode: caching
      refresh_check_interval: 10m

For more details, refer to the Data Acceleration Documentation and Caching Documentation.

DynamoDB Streams Data Connector in Preview​

DynamoDB Connector now integrates with DynamoDB Streams which enables real-time streaming with support for both table bootstrapping and continuous change data capture (CDC). This connector automatically detects changes in DynamoDB tables and streams them into Spice for real-time query, search, and LLM-inference.

Key Features:

• Real-Time CDC: Automatically captures inserts, updates, and deletes from DynamoDB tables
• Table Bootstrapping: Initial full table load before streaming changes

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: orders_stream
    acceleration:
      enabled: true
      refresh_mode: changes

For more details, refer to the DynamoDB Connector Documentation.

Cayenne Accelerator Enhancements​

The Cayenne data accelerator now supports:

• Sort Columns Configuration: Optimize inserts by pre-sorting data on specified columns for improved query performance

Example Spicepod.yml configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: sorted_data
    acceleration:
      enabled: true
      engine: cayenne
      mode: file_create
      params:
        sort_columns: timestamp,region

For more details, refer to the Cayenne Documentation.

S3 Connector Improvements​

S3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down predicates to S3 listing operations. This optimization is especially effective for partitioned datasets stored in S3.

AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling fast integration with AWS's table format for S3.

For more details, refer to the S3 Tables Data Connector Documentation and Glue Data Connection Documentation.

Faster Distributed Query Execution​

Distributed query planning and execution have been significantly improved:

• Fixed executor registration in cluster mode for more reliable distributed deployments
• Improved hostname resolution for Flight server binding, enabling better executor discovery
• Distributed accelerator registration: Data accelerators now properly register in distributed mode
• Optimized query planning: DistributeFileScanOptimizer improvements for faster planning with large datasets

For more details, refer to the Distributed Query Documentation.

Search Improvements​

Search capabilities have been improved with several performance and reliability enhancements:

• Fixed FTS query blocking: Full-text search queries no longer block unnecessarily, improving query responsiveness
• Optimized vector index operations: Eliminated unnecessary list_vectors calls for better performance
• Improved limit pushdown: IndexerExec now properly handles limit pushdown for more efficient searches

For more details, refer to the Search Documentation.

Security Hardening​

Multiple security improvements have been implemented:

• SQL identifier quoting: Hardened SQL identifier quoting across all connectors to prevent injection attacks
• Token redaction: Sensitive tokens are now fully redacted in debug output to prevent credential leakage
• Path traversal prevention: Fixed tar extraction to prevent path traversal vulnerabilities
• Input sanitization: Added validation for top_n_sample order_by parsing
• Improved credential handling: Improved credential management in Glue connector

Developer Experience Improvements​

• Health probe metrics: Added health probe latency metrics for better observability
• CLI improvements: Fixed .clear history command in the REPL to fully clear persisted history

Contributors​

• @lukekim
• @sgrebnov
• @peasee
• @krinart
• @Jeadie
• @mach-kernel
• @phillipleblanc
• @ewgenius

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates. The Spice Cookbook still offers 82+ recipes to help you prototype quickly.

Upgrading​

To try v1.10.0-rc1, use one of the following methods:

CLI:

spice upgrade --version 1.10.0-rc1

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.0-rc1 image:

docker pull spiceai/spiceai:1.10.0-rc1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.10.0-rc1

AWS Marketplace:

🎉 Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Test-operator: Add tpcds_q8 to the default row-count validation skip list by @sgrebnov in #8185
• fix: Remove unwrap_used from test by @peasee in #8212
• Run glue_iceberg_integration_test_catalog as part of main integration tests by @sgrebnov in #8222
• Add TPCH sf100 testoperator spicepods with dispatch by @Jeadie in #8192
• Build with CPU native flags by @lukekim in #8224
• Make copilot check for empty copyright header by @krinart in #8245
• fix: Apply assertion clippy in CI/Makefile only by @peasee in #8229
• feat: Support running queries only in testoperator by @peasee in #8211
• DuckDB query planning: aggregate pushdown by @mach-kernel in #8174
• install.sh improvements by @lukekim in #8252
• Fix .clear history by @lukekim in #8254
• fix: Pushdown dynamic filters to partition scans by @peasee in #8240
• Harden SQL identifier quoting in connectors by @phillipleblanc in #8276
• Cayenne sort_columns on insert by @lukekim in #8091
• Redact token debug output by @phillipleblanc in #8280
• fix: Cayenne configuration options by @lukekim in #8281
• Prevent path traversal in untar by @phillipleblanc in #8284
• Fix cluster mode executor registration by @mach-kernel in #8292
• Unignore s3_vectors_kafka_stream test by @Jeadie in #8289
• Post-release house keeping by @krinart in #8293
• Improve generate_changelog script by @krinart in #8273
• Acceleration mode caching by @lukekim in #8237
• Sanitization and security checks by @lukekim in #7854
• Add health probe latency metric by @phillipleblanc in #8300
• Add distributed registration for data accelerators by @phillipleblanc in #8299
• Pass IndexedTableProvider down in 'changes_stream' and 'append_stream' by @Jeadie in #8295
• Add dynamodb-streams crate by @krinart in #8283
• Distributed query: resolve executor hostname when determining Flight server binding by @mach-kernel in #8304
• Return computed embeddings from index for partitioned S3Vectors by @Jeadie in #8306
• [DDB Streams] Skeleton for DynamoDB Streams by @krinart in #8296
• DistributeFileScanOptimizer: Improve planning performance by @mach-kernel in #8305
• feat: Add an ExactLeftAccumulator implementation by @peasee in #8302
• deps: Upgrade Vortex to 0.56 by @peasee in #8311
• DynamoDB table bootstrapping + streaming by @krinart in #8312
• Avoid calling S3Vector list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie in #8282
• Add on_conflict testing support to append benchmark by @sgrebnov in #8314
• docker: Add valid home directory to fix duckdb extension loading issue by @phillipleblanc in #8318
• Add GH Workflow to run Append benchmark test by @sgrebnov in #8321
• Exclude MySQL SF100 from test-operator dispatch by @sgrebnov in #8320
• feat: Update clippy lints by @peasee in #8317
• Add S3 location predicate pruning to listing connector by @phillipleblanc
• Review feedback for caching mode accelerator by @phillipleblanc in #8326
• Also include Dockerfile home changes for release build by @phillipleblanc in #8327
• Change communication channel from Discord to Slack by @Jeadie in #8330
• Replace Discord link with Slack link in README by @Jeadie in #8331
• fix(glue): Prevent OpenDAL from automatic loading of AWS credentials from environment by @sgrebnov in #8337
• Block on index read for FTS queries by @Jeadie in #8339
• Fix search query provider by @Jeadie in #8343
• Support for writing into AWS S3 Tables by @sgrebnov in #8344
• Acceleration file_create mode by @lukekim in #8347
• Don't block on lock in FTS query path by @Jeadie in #8348
• feat: Add an optimizer rule to replace join accumulator for Cayenne by @peasee in #8316
• S3 Vectors limit updates by @lukekim in #8352
• Sanitize top_n_sample order_by parsing by @phillipleblanc in #8356
• Update version to v1.10.0-rc.1 by @ewgenius in #8362
• Improve IndexerExec to properly handle limit pushdown by @sgrebnov in #8366
• Fix Cayenne partition_by metadata flaky integration test by @phillipleblanc in #8367
• Rework caching accelerator to use the stale-while-revalidate pattern. by @phillipleblanc in #8365
• Add TinyLFU caching policy by @lukekim in #8370

Announcing the release of Spice v1.1.1! 📊

Spice v1.1.1 introduces several key updates, including a new Component Metrics System, improved Delta Data Connector performance, improved MCP tool descriptions, and expanded runtime results caching options. This release also adds detailed MySQL connection pool metrics for better observability. Component Metrics are Prometheus-compatible and accessible via the metrics endpoint.

Highlights v1.1.1​

• Component Metrics System: A new system for monitoring components, starting with MySQL connection pool metrics. These metrics provide insights into MySQL connection performance and can be selectively enabled in the dataset configuration. Metrics are exposed in Prometheus format via the metrics endpoint.

For more details, see the Component Metrics documentation.

• Results Caching Enhancements: Added a cache_key_type option for runtime results caching. Options include:
  • plan (Default): Uses the query's logical plan as the cache key. Matches semantically equivalent queries but requires query parsing.
  • sql: Uses the raw SQL string as the cache key. Provides faster lookups but requires exact string matches. Use sql for predictable queries without dynamic functions like NOW().

Example spicepod.yaml configuration:

runtime:
  results_cache:
    enabled: true
    cache_max_size: 128MiB
    cache_key_type: sql # Use SQL for the results cache key
    item_ttl: 1s

For more details, see the runtime configuration documentation.

• Delta Data Connector: Improved scan performance for faster query performance.

• MCP Tools: Improved descriptions for built-in MCP tools to improve usability.

• MySQL Component Metrics: Added detailed metrics for monitoring MySQL connections, such as connection count and pool activity.

Example spicepod.yaml configuration:

datasets:
  - from: mysql:my_table
    name: my_dataset
    metrics:
      - name: connection_count
        enabled: true
      - name: connections_in_pool
        enabled: true
      - name: active_wait_requests
        enabled: true
    params:
      mysql_host: localhost
      mysql_tcp_port: 3306
      mysql_user: root
      mysql_pass: ${secrets:MYSQL_PASS}

For more details, see the MySQL Data Connector documentation.

• spice.js SDK: The spice.js SDK has been updated to v2.0.1 and includes several important security updates.

New Contributors 🎉​

• @kczimm made their first contribution in #5243

Contributors​

• @peasee
• @lukekim
• @phillipleblanc
• @Sevenannn
• @sgrebnov
• @ewgenius
• @kczimm

Breaking Changes​

No breaking changes in this release.

Cookbook Updates​

The Spice Cookbook now includes 65 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.1.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.1.1 image:

docker pull spiceai/spiceai:1.1.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

• No major dependency changes.

Changelog​

- fix: Testoperator DuckDB, SQLite, Postgres, Spicecloud by [@peasee](https://github.com/peasee) in [#5190](https://github.com/spiceai/spiceai/pull/5190)
- Update Helm Chart and SECURITY.md to v1.1.0 by [@lukekim](https://github.com/lukekim) in [#5223](https://github.com/spiceai/spiceai/pull/5223)
- Update version.txt to v1.1.1-unstable by [@lukekim](https://github.com/lukekim) in [#5224](https://github.com/spiceai/spiceai/pull/5224)
- Update Cargo.lock to v1.1.1-unstable by [@lukekim](https://github.com/lukekim) in [#5225](https://github.com/spiceai/spiceai/pull/5225)
- Add tests for `verify_schema_source_path` in `ListingTableConnector` by [@phillipleblanc](https://github.com/phillipleblanc) in [#5221](https://github.com/spiceai/spiceai/pull/5221)
- Reduce noise from debug logging by [@phillipleblanc](https://github.com/phillipleblanc) in [#5227](https://github.com/spiceai/spiceai/pull/5227)
- Improve `openai_test_chat_messages` integration test reliability by [@Sevenannn](https://github.com/Sevenannn) in [#5222](https://github.com/spiceai/spiceai/pull/5222)
- Verify the checkpoints existence before shutting down runtime in integration tests directly querying checkpoint by [@Sevenannn](https://github.com/Sevenannn) in [#5232](https://github.com/spiceai/spiceai/pull/5232)
- Fix CORS support for json content-type api by [@sgrebnov](https://github.com/sgrebnov) in [#5241](https://github.com/spiceai/spiceai/pull/5241)
- Fix ModelGradedScorer error: The 'metadata' parameter is only allowed when 'store' is enabled. by [@sgrebnov](https://github.com/sgrebnov) in [#5231](https://github.com/spiceai/spiceai/pull/5231)
- fix: Use `pulls-with-spice-action` and switch to `spiceai-macos` runners by [@peasee](https://github.com/peasee) in [#5238](https://github.com/spiceai/spiceai/pull/5238)
- Use v1.0.3 pulls with spice action by [@lukekim](https://github.com/lukekim) in [#5244](https://github.com/spiceai/spiceai/pull/5244)
- feat: Build ODBC binaries, run testoperator on ODBC by [@peasee](https://github.com/peasee) in [#5237](https://github.com/spiceai/spiceai/pull/5237)
- Bump timeout for several integration test runtime load_components & readiness check by [@Sevenannn](https://github.com/Sevenannn) in [#5229](https://github.com/spiceai/spiceai/pull/5229)
- Validate port is available before binding port for docker container in integration tests by [@Sevenannn](https://github.com/Sevenannn) in [#5248](https://github.com/spiceai/spiceai/pull/5248)
- Update datafusion-table-providers to fix the schema for PostgreSQL materialized views by [@ewgenius](https://github.com/ewgenius) in [#5259](https://github.com/spiceai/spiceai/pull/5259)
- Verify flight server is ready for flight integration tests by [@Sevenannn](https://github.com/Sevenannn) in [#5240](https://github.com/spiceai/spiceai/pull/5240)
- fix: Publish to MinIO inside of matrix on build_and_release by [@peasee](https://github.com/peasee) in [#5258](https://github.com/spiceai/spiceai/pull/5258)
- fix: TPCDS on zero results benchmarks by [@peasee](https://github.com/peasee) in [#5263](https://github.com/spiceai/spiceai/pull/5263)
- Use model as a judge scorer for Financebench by [@sgrebnov](https://github.com/sgrebnov) in [#5264](https://github.com/spiceai/spiceai/pull/5264)
- Fix FinanceBench llm scorer secret name by [@sgrebnov](https://github.com/sgrebnov) in [#5276](https://github.com/spiceai/spiceai/pull/5276)
- Implements support for `runtime.results_cache.cache_key_type` by [@phillipleblanc](https://github.com/phillipleblanc) in [#5265](https://github.com/spiceai/spiceai/pull/5265)
- fix: Testoperator MS SQL, query overrides, dispatcher by [@peasee](https://github.com/peasee) in [#5279](https://github.com/spiceai/spiceai/pull/5279)
- refactor: Delete old benchmarks by [@peasee](https://github.com/peasee) in [#5283](https://github.com/spiceai/spiceai/pull/5283)
- Imporve embedding column parsing performance test by [@Sevenannn](https://github.com/Sevenannn) in [#5268](https://github.com/spiceai/spiceai/pull/5268)
- Add Support for AWS Session Token in S3 Data Connector by [@kczimm](https://github.com/kczimm) in [#5243](https://github.com/spiceai/spiceai/pull/5243)
- Implement Component Metrics system + MySQL connection pool metrics by [@phillipleblanc](https://github.com/phillipleblanc) in [#5290](https://github.com/spiceai/spiceai/pull/5290)
- Add default descriptions to built-in MCP tools by [@lukekim](https://github.com/lukekim) in [#5293](https://github.com/spiceai/spiceai/pull/5293)
- fix: Vector search with cased columns by [@peasee](https://github.com/peasee) in [#5295](https://github.com/spiceai/spiceai/pull/5295)
- Run delta kernel scan in a blocking Tokio thread. by [@phillipleblanc](https://github.com/phillipleblanc) in [#5296](https://github.com/spiceai/spiceai/pull/5296)
- Expose the `mysql_pool_min` and `mysql_pool_max` connection pool parameters by [@phillipleblanc](https://github.com/phillipleblanc) in [#5297](https://github.com/spiceai/spiceai/pull/5297)
- use patched pdf-extract by [@kczimm](https://github.com/kczimm) in [#5270](https://github.com/spiceai/spiceai/pull/5270)