10 posts tagged with "github"

Spice v2.3.0 brings performance improvements, broader query federation, and expanded data connector capabilities. The release improves cache reuse and reduces memory use for cached SQL results. It also extends BigQuery support and adds GitHub review, release, and repository data for SQL analysis. Google models now use Vertex AI, so deployments with credentials for Google AI Studio require migration.

Highlights in v2.3.0 include:

• SQL Federation Improvements — bug fixes for function translation, NULL handling, and timezone declarations
• BigQuery Federation — temporal expressions, recursive CTEs, correlated subqueries, and window aggregates now run as one BigQuery job
• Caching Accelerator Limits — caching_max_size and caching_max_items bound what a refresh_mode: caching accelerator holds
• Stale-While-Revalidate Across a Refresh — configured stale-while-revalidate windows retain cached results across acceleration refreshes
• GitHub Review Data — eight new tables, 17 new pulls columns, and a repo / owner column on every row
• Google Models on Vertex AI — a from: google model authenticates as a GCP service account

What's New in v2.3.0​

SQL Federation Improvements​

This release improves SQL translation and function handling for accelerated and federated datasets. The following bug fixes cover string functions, NULL handling, correlated subqueries, and timezone declarations.

• A trim call failed on DuckDB, SQLite, and MySQL because DataFusion emitted its canonical name, btrim. DuckDB now receives trim, with an explicit space argument for the one-argument form. The native SQLite and MySQL paths evaluate btrim locally.
• DuckDB federation now lower-cases to_hex output to match local evaluation.
• DuckDB federation now decodes sha256 output into a 32-byte digest instead of a hexadecimal string.
• concat uses || on DuckDB to consistently propagate NULL arguments.
• inner_product now returns NULL for an undefined dot product on DuckDB for consistency.
• Operations that DuckDB cannot handle (such as regex with U or R flags) are executed locally.
• User-defined functions registered after startup now execute locally.
• Catalog connectors now use the same list of local Spice-only functions as the data connectors.
• Two EXISTS shapes emitted SQL which evaluated the correlation over the whole relation, so the bound selected nothing. A semi or mark join then reported a match on a row the plan never read. Both shapes now refuse pushdown and run locally.
• DuckDB labels a TIMESTAMPTZ column with the connection's own timezone. The connector built its pool and never pinned that setting. A dataset's schema therefore carried the host timezone, and the same query returned different rows on different machines. A connector session is now pinned to UTC.

BigQuery Federation​

BigQuery federation runs more query shapes as one remote job.

• Temporal expressions, recursive CTEs, and integer division now keep their results and stay in one federated statement. These shapes previously failed remotely, split into several queries, or produced wrong cohort boundaries.
• Three more statement shapes now run. The first is a grouped query that projects a wrapped form of its grouping expression. The second is a query whose federated tables all sit inside a correlated subquery. The third is any aggregate window function.
• A query that reads BigQuery tables from several datasets of one project now runs as one BigQuery query. It also no longer returns rows from the wrong dataset when two datasets hold a table of the same name.
• A dataset that stores JSON in a STRING column now pushes down scalar json_as_text expressions and json_get(...) IS NULL checks.
• The built-in date_trunc is preserved, and the dialect and the federation policy now agree on which aggregate and window calls are eligible.
• regexp_match null checks federate safely.
• A distinct union now renders as UNION DISTINCT. BigQuery rejects a bare UNION, so such a query failed outright. UNION ALL is unchanged.
• A numbering function such as ROW_NUMBER no longer carries a window frame, which BigQuery rejects. An aggregate window function keeps its frame.
• Percentile functions and grouping keys now render in the form BigQuery accepts. A reported 29-statement workload that failed on these shapes now runs in full.
• array_element translates a non-negative integer literal index that fits in Int64 as SAFE_ORDINAL. Other indexes evaluate locally. This preserves DataFusion's end-relative semantics for negative indexes.

Cancellation: A BigQuery query whose client goes away now stops. The BigQuery job ends as cancelled, and the pooled connection returns immediately. Before this release the query ran to completion, and the connection stayed busy for its whole duration. Enough cancellations left an application unable to query at all.

The Caching Accelerator Accepts Explicit Limits​

A refresh_mode: caching accelerator had nothing bounding what it held. Retention was derived from caching_ttl plus caching_stale_while_revalidate_ttl, and only when caching_stale_if_error was disabled. A dataset that set caching_stale_if_error therefore got no policy at all, and nothing was ever evicted. Nothing capped the acceleration by size or by count either.

Two settings now bound the acceleration. Each one refuses an unparseable value rather than falling back to a default:

• caching_max_size — a byte budget, such as 512MiB.
• caching_max_items — a row budget.

caching_ttl is also accepted as caching_item_ttl, which is the spelling the SQL results, search results, and embeddings caches use. Eviction is entry-granular. A cached response can span several rows, so the runtime ranks entries by their oldest page and removes all of an entry's rows together. The storage schema is unchanged, and no existing acceleration needs a rebuild.

datasets:
  - from: https://api.example.com/v1/items
    name: items
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: caching
      primary_key: '(request_query, request_path)'
      params:
        caching_ttl: 5m
        caching_max_size: 512MiB
        caching_max_items: 50000

caching_stale_if_error now fires on the failure it exists for. It keyed off a fetch that returned an error. The HTTP connector reports a failing origin as a successful fetch whose rows carry a 429 or 5xx status once it exhausts max_retries. An operator who enabled the setting received the origin's error body instead of the cached response.

A caching accelerator that has nothing bounding it now says so at startup.

SQL Results Cache Improvements​

Stale results remain available across a refresh when configured. An acceleration refresh evicted every dependent SQL results cache entry, and any successful refresh counted as a change. A refresh_mode: full update still flushed the whole per-table cache. For a workload with consistently high QPS, each refresh turned a population of cached results into simultaneous synchronous misses.

When stale_while_revalidate_ttl is configured, an invalidation now marks dependent entries stale as of the refresh instead of evicting them. Inside the stale window the runtime serves the previous result with Results-Cache-Status: STALE and starts one background revalidation per key. Past the stale window the request is a miss, exactly as before. With no stale window configured, invalidation stays hard.

Cache accounting covers more retained memory. A cache with a million empty results reported 0.09 GiB against 1.85 GiB of retained memory. Its max_size accounting omitted parts of each entry. The cache now shares schemas and input-table sets, copies foreign buffers, and accounts for per-buffer allocation overhead.

In the reported benchmark, each entry retained 1%–80% less memory across 20 combinations of result shape and source. The benchmark ran on macOS/arm64 with snmalloc. Reported size ranged from 0.65x to 1.73x of retained memory after the change. The lowest ratio before the change was 0.31x. These figures compare the fix with its merge base, not v2.2.1, and are not guarantees for every workload.

This release also corrects memory accounting for the search results and embeddings caches.

Pingora cache engine: Table invalidation read every entry with a destructive get, so an invalidation promoted every key in the cache. Scan order replaced recency, and each visited key became a momentary miss to concurrent readers. A read also served a hit destructively, so a second reader reported a miss for a key the cache holds. Both reads are now non-destructive.

GitHub Data Connector: Review, Release, and Repository Tables​

The GitHub Data Connector adds eight tables, 17 columns on pulls, and repository identity on every row. An application can now answer a code-review question in SQL. Before this release, pulls.reviews_count was a bare integer with no state and no reviewer, and pulls.review_comments recorded only inline comments.

pulls adds is_draft, mergeable, merge_state_status, review_decision, status_check_rollup, merge_queue_state, merge_queue_position, merged_by, closed_by, base_ref, head_ref, head_sha, milestone_id, milestone_title, closing_issues_references, closing_issues_count, and reactions_count. issues adds state_reason, closed_by, reactions_count, type, and type_color. Every table returns a repo and an owner column, so a multi-repository UNION ALL keeps its rows apart.

The connector also asks GitHub for a narrower pull request page. A 100-node page now exceeds GitHub's per-request compute budget on a large repository. GitHub rejects that page with Resource limits for this query exceeded and returns every node as null, so the dataset never loaded.

Google Models Move to Vertex AI​

A from: google chat or embedding model now authenticates as a GCP service account against Vertex AI. Spice no longer accepts a Google AI Studio API key. See Breaking Changes for the migration.

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_project: my-project
      google_location: us-central1
      google_service_account_path: /etc/spice/gcp-sa.json

Other AI model fixes in this release:

• An Anthropic model configured without an explicit model id now resolves. The default named claude-3-5-sonnet-latest, which Anthropic has retired, so every such request failed.
• An Anthropic model now refuses an OpenAI top_logprobs request instead of translating it to top_k. The two fields are unrelated. top_logprobs reports log probabilities and top_k narrows sampling, so the translation silently changed the model's output.
• Anthropic streaming failures and provider refusals from openai, xai, and spiceai are now classified from the provider's typed error fields. Each path searched the rendered error text for 401, 429, or rate, and then replaced the provider's own detail with a fixed string.
• A from: huggingface: chat model reads hf_token again. The parameter moved to the prefix huggingface, so hf_token was warned about as unknown and a gated repository was downloaded anonymously.
• An Amazon Bedrock model now names the credential AWS rejected instead of reporting unhandled error.
• The text-embeddings-inference model-load path runs its filesystem and tokenizer work on a blocking thread. That work ran on a Tokio worker thread during model registration, so it could starve /health.

Search Improvements and Bug Fixes​

This release fixes bugs in search result limits, index updates, and deletes.

• A result set larger than the requested limit: vector_search(tbl, 'query', 10) against an Elasticsearch-backed index now respects the requested limit.
• Index writes and deletes kept in sync with the table: Previously, writes for rows with repeated primary keys, no chunks, or non-embeddable chunks could leave the previous index entry. These outdated index entries are now deleted.
• A chunked Elasticsearch delete: The delete filtered on the key columns, and a string key was left to Elasticsearch dynamic mapping as an analyzed text field. The delete now filters on a field that can match the key exactly.
• A partial Elasticsearch delete: _delete_by_query returns 2xx when the request ran, and it reports per-document failures and version conflicts in the body. Spice discarded that body, so a delete could leave documents behind and still report success.
• Full-text index encoding: A full-text upsert is a delete followed by an insert, so both halves must encode the primary key the same way. They disagreed for Float32, Float16, and Binary keys, and both the old and the new row stayed in the index.

Acceleration and Refresh​

• LIMIT on a partitioned scan: PartitionTableProvider::scan passed the scan limit as the skip argument rather than the fetch argument. LIMIT 10 over a three-row partitioned dataset returned zero rows.
• acceleration.enabled: false: A dataset or a view can set enabled: false and leave the rest of the block in place. The runtime read every other setting, accepted it, and then ignored it. The component reported healthy and served federated queries. The runtime now names the settings it discards.
• ready_state on a view: A view's acceleration.ready_state was accepted by the schema and by the parser, and then never applied. The identical key on a dataset was honoured. A view now resolves the key the way a dataset does.
• A retention policy that cannot start: A dataset that set retention_check_enabled: true, a retention_period, and a time_column but no retention_check_interval got no retention task and no diagnostic. The builder now reports the refusal.
• A refresh completion that arrives early: A completion published before a caller registered its wait was dropped with no record, and the caller waited for a refresh that had already happened. The signal is now level-triggered.
• A refresh completion from the wrong refresh: A waiter was satisfied by the next completion recorded on the table, whichever refresh produced it. A refresh that was already running could therefore release a caller. Completions are now correlated with the refresh that a caller triggered.
• A table replaced during a refresh: Two callers acted on a completion for a table that had since been removed or rebuilt. The runtime now re-resolves the table after the refresh lands and before it acts on the completion.
• A schema repair on a checkpoint: Writing a checkpoint's schema also wrote its refresh timestamp, so a schema repair told the scheduler the data was fresh. An overdue dataset then waited a full refresh_check_interval. A schema repair now leaves the freshness clock alone.
• A recorded snapshot schema: A snapshot's recorded schema is a foreign declaration, and a Map that declares its entries field nullable is a declaration no accelerator can hold. The restore path now conforms that declaration to the Arrow map layout.
• A source row durable write-back could not confirm: The delivery worker read a missing point-scan row as a deletion. A short visibility gap in the accelerator therefore deleted that row from the source of record. The worker now withholds a key it cannot read and retries it on a later pass. A delivery cursor advances only after the pass succeeds. Write-back also refuses a configuration it cannot uphold, which is a behavior change. See Breaking Changes.

Arrow and Storage Bug Fixes​

• Decimal128 on write paths: Three conversions could produce a plausible wrong number instead of an error. The sum of two in-range halves wrapped to a large negative decimal at scale 38. A float-to-int cast saturated to i128::MAX, and NaN became 0. The declared precision was never checked, so a value needing more digits than the column declares was appended anyway. All three were reachable from Debezium decimal ingestion, where the input is source-controlled. The runtime now validates the destination precision once, where every input form converges.

• An Iceberg DELETE an equality key cannot express: An Iceberg delete writes an equality delete file, which removes rows whose key columns equal the given values. That statement matches the user's WHERE only when the condition reads key columns alone. A condition on a float or a nested column removed rows that did not match. Spice now refuses the statement and names the offending column.

• A Parquet object overwritten mid-scan: A listing-table Parquet scan decoded two object generations as one file. The scan now pins one generation through a version id or an If-Match header. dataset_acceleration_refresh_errors carries reason=object_generation_changed|parquet_decode|other, so an expected overwrite is distinguishable from corruption.

• An Arrow relabel: relabel_array_data carries an array's values across a type change untouched, and only field names and nested nullability flags may differ. Nothing enforced that contract. It now refuses three kinds of target:

  • a target that changes what the buffers mean
  • a target that declares away nulls the array still holds
  • a target whose same-typed sibling fields are reordered

  The third kind produced silent column-value transposition on the Delta Lake column-mapping path.

• Arrow MAP columns: The Arrow map layout forbids a nullable entries field, and MapArray::try_new refuses one. Nothing enforced it at decode. A producer that declared it that way handed over a column that decoded cleanly and then failed in the first kernel that rebuilt it. The runtime now normalizes entries nullability at every Arrow decode point. A Databricks SQL Warehouse MAP column, which declares entries nullable, no longer panics the runtime. A Cayenne accelerator that has already persisted the non-conforming declaration is now repaired.

• A nested nullability difference: try_cast_to decided its fast paths with Schema::contains, which permits a nested field's nullability to differ. RecordBatch requires the two types to be identical. The shared entry point now aligns the difference instead of publishing it.

• A retired Vortex file: Retiring a Cayenne file released its Vortex segments and left its footer in DataFusion's file-metadata cache, which has no TTL. A file opened during retirement could also repopulate the path that retirement had just cleared. The retirement drain had no ceiling. A stalled put therefore held every caller of the invalidation, and the delete sink is one of them. All three faults are fixed.

• A Cayenne teardown that deleted a shared metastore: Recreation of a Cayenne dataset could delete a shared metastore and leave other datasets unavailable after restart. The guard checked only the metastore named by that dataset's settings, not catalogs inside its data directory. Open file handles hid the loss until restart. The runtime now scans that directory before catalog changes and again before deletion. It refuses recreation if it finds a Cayenne metastore or cannot safely resolve the configured paths.

Observability and Operations​

• OpenTelemetry resource attributes: The OTLP ingest path parsed resource attributes such as service.name and service.instance.id and then dropped them. Data points from two processes were therefore indistinguishable once written. Resource attributes now reach the metric data points. The same change closes four ingest races that dropped data with no error. One of those races let a write publish through a table provider that a schema evolution had already replaced.
• A panicking query: A query whose execution panicked was sometimes returned as an empty HTTP 200 success, which no client can tell apart from "no rows matched". This happened in 20 of 60 identical runs on trunk. A panicking query is now always an error.
• runtime.cpu.cores above the container's ceiling: runtime.cpu.cores was the one CPU entitlement setting taken raw rather than clamped. A pod configured with runtime.cpu.cores: 6 under resources.limits.cpu: 2 sized every derived pool for six cores and was then throttled. The runtime now warns and names both readings. It does not clamp, because an operator may size the runtime for a node the pod has not reached yet.
• HTTP latency: The HTTP server sets TCP_NODELAY, which lowers the latency of a small response body. The Flight SQL server already set it.
• MCP tools in runtime.task_history: A proxied MCP tool call was recorded under two different task values depending on the entry point, so one logical tool split across two rows. Grouping by task gave wrong per-tool counts. Both entry points now use the encoded name.
• A discarded Flight batch: The runtime reported a data_loss count that counted a message by its body length. A batch whose body is empty still carries rows, so the count was wrong. The runtime now reads the IPC header.
• A hot reload that changes functions or catalogs: A cached logical plan embeds the ScalarUDF and the TableSource it was planned against. A hot reload that redefined a SQL function or replaced a catalog left those plans in place. The same SQL then kept answering from the replaced component. The plan cache is installed unconditionally with a one-hour TTL, so no caching configuration was needed to hit this. Both handlers now discard the affected plans.
• Cloud Connect metrics cadence: A Cloud Connect instance exports metrics every 10 seconds rather than every 30, so a chart drawn from the control stream resolves at 10 seconds. The payload is a snapshot of cumulative totals, so this changes chart resolution and not what is recorded.

Other Improvements and Bug Fixes​

• Glue catalog: A discovered Glue table that Spice cannot read, such as an ORC or Avro table, was absent from the catalog with nothing said about it. The catalog connector now reports each such table and the reason.
• Databricks: The connector accepts Unity Catalog streaming tables and views. It also forwards the runtime's spark feature.
• DuckDB index materialization: The DuckDB intermediate index materialization rule reads a table's index list before it rewrites a scan into a materialized CTE. That list had been empty, so the rule never fired and an indexed column never narrowed a scan.
• Vortex list_length pushdown: DataFusion array_length(expr) and array_length(expr, 1) now convert to Vortex list_length and push into the scan. A list length is computed from offsets, and element values are not materialized.
• ScyllaDB: The ScyllaDB Data Connector is out of the default build, alongside ODBC. make install-scylladb or --features scylladb builds it. A Spicepod that names scylladb: on a build without it now says the build lacks the connector rather than offering the closest registered name. The connector also declines a physical sort that CQL cannot serve.
• Turso: The accelerator refuses a stored list whose encoding predates the version marker rather than reading it under the current encoding.
• CLI: spice run resolves spiced beside the CLI before it reaches for the managed install, and never from PATH. A Spice Cloud project listing is attributed to the organization it was requested for. spice query and spice nsql analyze keep the API key on its origin across a redirect. A Cloud Connect managed instance no longer warns about the default pods watcher on every spice run.
• MCP tools: A renamed tool forwards strict() and as_mcp_proxy() to the tool it wraps. runtime-tools declares what its mcp feature needs.
• Connector registries on shutdown: The runtime no longer clears stateless connector registries on shutdown.

Dependency Updates​

Compared with v2.2.1, this release changes the following versions:

DataFusion remains at v54.1.0 and Arrow remains at v58.3.0. Spice updates their fork revisions for the federation, Parquet scan, and cache fixes described above.

Contributors​

• @bjchambers
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @vatsalp2008

Breaking Changes​

Google models authenticate against Vertex AI. A from: google chat or embedding model no longer accepts google_api_key. Every such model now authenticates as a GCP service account.

Update each from: google model and embedding to set google_project, google_location, and exactly one credential setting.

Before:

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_api_key: ${secrets:google_api_key}

After:

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_project: my-project
      google_location: us-central1
      google_service_account_path: /etc/spice/gcp-sa.json

Set exactly one of the three credential settings.

Two behavior changes to note before you upgrade:

• Durable write-back rejects unsafe settings and operations. Before you upgrade, set mode: file, remove retention settings, and declare a single-column primary_key for each durable write-back dataset. The runtime rejects unsupported settings at load time. Submit writes inside a transaction as one BEGIN; ...; COMMIT; request. Write-back datasets reject DELETE and TRUNCATE when you issue those statements.
• The ScyllaDB Data Connector is out of the default build. Build with --features scylladb, or run make install-scylladb, to keep it. ODBC already worked this way.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.3.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.0 image:

docker pull spiceai/spiceai:2.3.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(search): surface an Elasticsearch delete that only partially applied (fixes #12364) by @claudespice in #12720
• fix(ci): bound every integration job, so a wedged one cannot hold the queue (refs #12718) by @grokspice in #12721
• feat(tools): add pdf-parse tool to compare liteparse and pdf-inspector by @Jeadie in #12807
• fix(turso): refuse a stored list whose encoding predates the version marker (fixes #12632) by @grokspice in #12837
• Use TypedParams for reranker parameters by @Jeadie in #13046
• fix(search): correctness, async-safety, and performance fixes across the search subsystem by @Jeadie in #13065
• Decide full-text CDC-attachment at construction, not after by @Jeadie in #13075
• fix(scylladb): decline the physical sort pushdown CQL cannot serve (fixes #10775) by @claudespice in #13107
• fix(cache): stop Pingora table invalidation from promoting every key it reads (fixes #12674) by @claudespice in #13117
• fix(cayenne): stop a failed statistics read publishing a partial count as exact (fixes #13010) by @claudespice in #13125
• fix(cayenne): refuse a widened CDC batch a partitioned acceleration cannot apply (fixes #13051) by @claudespice in #13133
• fix(catalogs): report the Glue tables Spice cannot read instead of dropping them silently (fixes #13102) by @claudespice in #13149
• fix(ci): gate every tracked Rust source tree in the merge queue's change filter (fixes #13120) by @claudespice in #13151
• fix(cache): bound max_size on the memory an entry holds, not its array bytes (fixes #12931) by @claudespice in #13154
• fix(cayenne): honour the sort-merge row floor on the memory-gated path (fixes #12958) by @claudespice in #13157
• fix(cache): serve a Pingora hit without hiding it from a concurrent reader (fixes #12987) by @claudespice in #13158
• fix(vortex): stop a file opened mid-retirement from repopulating the cleared path (fixes #12963) by @claudespice in #13161
• fix(vortex): evict a retired file's footer with its segments (fixes #12953) by @claudespice in #13162
• feat(google): switch from Google AI Studio to Vertex AI by @krinart in #13210
• Run TEI candle model-load filesystem and tokenizer work on a blocking thread by @Jeadie in #13223
• fix(correctness): validate Decimal128 precision and overflow on write paths by @lukekim in #13224
• fix(runtime): stop clearing stateless connector registries on shutdown by @Jeadie in #13225
• fix(search): use one encoding path for full-text index writes and deletes by @Jeadie in #13226
• fix(docker): install ARM64 linker toolchain by @Jeadie in #13266
• ci: build the runtime integration archives with one feature set by @bjchambers in #13270
• ci: archive every integration test target in one invocation by @bjchambers in #13273
• fix(sql): bump the datafusion pin so a bounded EXISTS refuses instead of returning wrong rows (fixes #13277) by @claudespice in #13280
• ci: have the compiler-cache action own its credentials by @bjchambers in #13282
• fix(cpu-budget): warn when a configured cores value exceeds the container's real CPU ceiling (fixes #13275) by @claudespice in #13284
• fix(databricks): publish a MAP column with the non-nullable entries field Arrow requires (fixes #7307) by @claudespice in #13288
• fix(search): bound a search result set by the requested limit, not just the index read (fixes #13274) by @claudespice in #13290
• fix(vortex): bound the segment-cache retirement drain so a stuck put cannot hold a delete open (fixes #12964) by @claudespice in #13300
• Reach the accelerator contract without going through the runtime by @bjchambers in #13304
• fix(postgres): rebuild the datasets already streaming when a join replaces their replication slot (fixes #13229) by @claudespice in #13313
• fix(cli): resolve spiced beside the CLI before the managed install, and never from PATH by @claudespice in #13317
• fix(iceberg): refuse a delete an equality key cannot express by @lukekim in #13322
• Stop the accelerator engines reaching up into the runtime by @bjchambers in #13324
• fix(benchmarks): open snapshot-update PRs against the dispatch branch, not always trunk by @krinart in #13335
• chore(deps): bump datafusion-table-providers for NUMERIC scale fidelity by @lukekim in #13349
• feat(cloud-connect): export metrics every 10s by @phillipleblanc in #13353
• Move the accelerator engines into their own crates by @bjchambers in #13354
• perf(cayenne): shard the encode by what the write is, not by what the table declares by @lukekim in #13356
• fix(cli): attribute a project listing to the org it was requested for by @lukekim in #13357
• Enable Oracle TPC-H result validation, fix the loader that trimmed leading spaces by @sgrebnov in #13360
• Readme: Change PG catalog status from Alpha to Beta by @sgrebnov in #13361
• Upgrade iceberg-rust to v0.10.1 by @krinart in #13365
• perf(cayenne): cut a rewrite's shards at equal row mass, not equal width by @lukekim in #13373
• fix(deps): bump datafusion-table-providers for AVG/division NUMERIC scale rounding by @krinart in #13387
• fix(databricks): forward runtime's spark feature to the databricks connector by @sgrebnov in #13389
• Move the Cayenne accelerator into its own crate by @bjchambers in #13391
• fix(testoperator): compare exact decimals on their mantissas, not through f64 by @bjchambers in #13407
• Configure an accelerator engine through its constructor instead of a published global by @bjchambers in #13409
• fix(opentelemetry): merge resource attributes into metric data points and close OTLP-to-sink ingest races by @peasee in #13412
• fix(arrow): refuse a relabel that changes what the array's buffers mean (fixes #13423) by @claudespice in #13435
• fix(tools): record one task_history task per proxied MCP tool (fixes #13338) by @claudespice in #13437
• fix(release): read release notes from a file and trim them to GitHub's body limit by @sgrebnov in #13457
• fix(ci): probe the cc toolchain before reaching for brew in setup-cc by @claudespice in #13466
• fix(cayenne): refuse a teardown that would delete a metastore no parameter names (fixes #13436) by @claudespice in #13471
• fix(spiced): stop warning about the default pods watcher by @phillipleblanc in #13494
• fix(arrow): align a nested nullability difference instead of advertising it (fixes #13285) by @claudespice in #13496
• fix(acceleration): make the refresh-completion signal level-triggered (fixes #13086) by @claudespice in #13505
• fix(acceleration): apply a partitioned scan's LIMIT as a fetch, not a skip by @vatsalp2008 in #13507
• fix(ci): stage the retention OOM test binary without its debug info by @claudespice in #13511
• fix(ci): let sign-off run from a worktree nested inside the checkout, and add --skip-targeted by @bjchambers in #13520
• Authenticate the stargazers dataset with a PAT and drop qa_analytics by @lukekim in #13529
• fix(ci): isolate sccache per job on shared self-hosted Macs by @lukekim in #13531
• chore(scylladb): take the ScyllaDB connector out of the default build by @lukekim in #13532
• feat(github): add review, release, milestone, user and repo tables, plus repo/owner columns by @lukekim in #13545
• fix(ci): match the allowed refresh-task warning at its current module path by @claudespice in #13547
• fix(flight): normalize MAP entries nullability at every Arrow decode point (fixes #13495) by @claudespice in #13550
• fix(anthropic): default to a model Anthropic still serves (fixes #13557) by @claudespice in #13563
• fix(postgres): rebuild an emptied CDC acceleration rather than resume its surviving position (refs #13546) by @claudespice in #13566
• chore(spicepod): accelerate GitHub datasets with Cayenne instead of DuckDB by @lukekim in #13571
• fix(arrow): refuse a relabel that declares away nulls the array still holds (fixes #13433) by @claudespice in #13585
• fix(cayenne): refuse a catalog whose data directory would hold its metastore (fixes #13105) by @claudespice in #13593
• fix(ci): exempt every App account from the assignee gate, not just Dependabot (fixes #13115) by @grokspice in #13594
• fix(mysql): assign the binlog dump session's net_write_timeout floor as an integer literal (fixes #13307) by @grokspice in #13595
• fix(ci): let the E2E macOS build share the fleet's Cargo home (fixes #13299) by @grokspice in #13596
• fix(ci): call a test binary the runner cannot load an infrastructure failure (fixes #13518) by @grokspice in #13597
• fix(spicepod): say which acceleration settings enabled: false discards (fixes #13514) by @grokspice in #13602
• feat(caching): bound a caching accelerator by size, count and entry lifetime (closes #13525) by @bjchambers in #13604
• fix(ci): catch a stale Cargo.lock before the merge queue, not after a 55-minute build (fixes #13598) by @grokspice in #13606
• fix(bedrock): say which credential AWS rejected instead of "unhandled error" (refs #12396) by @claudespice in #13616
• feat(caching): serve results stale after an acceleration refresh instead of evicting them by @krinart in #13618
• fix(write-back): never delete a source row for a key the accelerator did not return by @phillipleblanc in #13638
• fix(cayenne): fold a staged append's unpublished keys into the PK-keyset rebuild (fixes #13639) by @claudespice in #13644
• fix(tools): forward strict() and as_mcp_proxy() from a renamed tool (fixes #13443) by @claudespice in #13649
• fix(delta_lake): order a column-mapping relabel target the way the scan reads it (fixes #13434) by @claudespice in #13655
• fix(ci): raise the e2e Linux build bound above the worst legitimate run (fixes #13674) by @grokspice in #13675
• fix: comment out refresh_append_overlap in the sample spicepod by @lukekim in #13680
• fix(anthropic): refuse a log-probability request instead of narrowing sampling (fixes #13581) by @claudespice in #13682
• fix(cayenne): record a pipelined non-conflict staged append's primary keys (fixes #13642) by @claudespice in #13686
• fix(schema): stop an illegal Arrow Map entries declaration from being stored or compared (fixes #13549) by @claudespice in #13695
• build(rust): upgrade toolchain to 1.97.1 by @lukekim in #13696
• fix(ci): stop a CI git push from blocking forever on a credential prompt (fixes #13701) by @claudespice in #13702
• fix(search): remove the vector a rejected write left behind (fixes #13504) by @claudespice in #13705
• fix(acceleration): stop a refresh already running from answering a later waiter (refs #13544) by @claudespice in #13709
• fix(search): remove a chunked row's stale chunks when its text goes away (refs #13704) by @claudespice in #13716
• fix(catalogs): install the Spice function deny-list on the SQL catalog connectors (refs #13664) by @claudespice in #13731
• fix(runtime-tools): declare what the mcp feature actually needs (fixes #13648) by @grokspice in #13733
• fix(runtime): re-resolve a table after its refresh lands, before acting on the completion (fixes #13603) by @claudespice in #13735
• fix(flight): count a discarded batch by its IPC header, not its body length (fixes #13636) by @grokspice in #13736
• docs(makefile): say what SPICED_DATA_FEATURES actually is (fixes #13678) by @grokspice in #13738
• fix(anthropic): classify a streaming failure by Anthropic's error type, not its message text (fixes #13562) by @claudespice in #13748
• fix(views): apply a view's acceleration.ready_state instead of dropping it (fixes #13615) by @claudespice in #13750
• fix(ci): resolve a Python 3.11+ interpreter for the lint-rust guards (refs #13754) by @grokspice in #13755
• fix(bigquery): emit valid pushed-down SQL by @phillipleblanc in #13768
• fix(bigquery): safely federate regexp_match null checks by @krinart in #13771
• feat(hash-index): verify the bloom filter's block index with Verus by @lukekim in #13777
• fix(adbc): federate a BigQuery statement spanning datasets as one query by @phillipleblanc in #13780
• fix(adbc): cancel an abandoned query, stop the BigQuery job, free the connection by @phillipleblanc in #13782
• fix(snapshot): conform a recorded snapshot schema to the Arrow map layout (fixes #13694) by @claudespice in #13786
• fix(cayenne): materialize the in-memory CDC tier before a scanning DELETE by @lukekim in #13798
• fix(caching): say when a caching accelerator has nothing bounding it (fixes #13525) by @claudespice in #13805
• fix(bigquery): run three federated statement shapes BigQuery was refusing by @phillipleblanc in #13812
• fix(duckdb): rewrite DataFusion's btrim to DuckDB's trim (fixes #13794) by @claudespice in #13821
• fix(federation): stop pushing btrim to SQLite and MySQL, which have no btrim (fixes #13840) by @claudespice in #13823
• fix(ci): give the throughput workflow the postgres fixtures bench provisions by @krinart in #13830
• fix(acceleration,cayenne): Resolve quoted columns in keys, name which primary key columns are null by @peasee in #13845
• fix: pin listing-table Parquet reads to one object generation by @phillipleblanc in #13847
• fix(duckdb): lower-case the hex digits a federated to_hex gets back (fixes #13818) by @claudespice in #13852
• fix(bigquery): carry the merged unparser fixes through the dialect wrapper by @phillipleblanc in #13853
• fix(databricks): allow Unity Catalog streaming tables and views through the table-type check by @krinart in #13855
• fix(acceleration): report a retention policy that cannot start instead of silently building none (fixes #13804) by @claudespice in #13857
• fix(search): evict a key whose deciding row the index rejected (fixes #13848) by @claudespice in #13859
• fix(duckdb): Restore intermediate index materialization optimization by @sgrebnov in #13864
• fix(federation): refuse a user function the deny-list snapshot was built before (fixes #13726) by @claudespice in #13868
• fix(duckdb): decode a federated sha256 back to the digest's bytes (fixes #13850) by @claudespice in #13869
• fix(runtime): Add TCP_NODELAY to HTTP server by @peasee in #13874
• fix(query): surface a panicking query as an error, never an empty success (fixes #13876) by @claudespice in #13878
• fix(bigquery): keep the built-in date_trunc, forward two dialect renderings, repin the unparser by @phillipleblanc in #13882
• fix(llms): classify a provider refusal from its typed fields, not its message (refs #13747) by @claudespice in #13884
• fix(vortex): keep control-byte field names unescaped in physical schema by @lukekim in #13886
• perf(vortex): push DataFusion array_length down as Vortex list_length by @lukekim in #13888
• fix(duckdb): render a federated concat as || so a NULL argument propagates (fixes #13849) by @claudespice in #13889
• fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
• fix(duckdb): screen a federated inner_product so a non-finite result is NULL (fixes #13787) by @claudespice in #13895
• refactor(postgres): report an acceleration re-read as a refresh, not a bespoke metric by @bjchambers in #13896
• fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
• fix(duckdb): pin a connector's DuckDB session to UTC so a dataset's schema does not carry the host timezone (fixes #13899) by @claudespice in #13903
• fix(cayenne): keep a file's statistics the same whichever source serves them (refs #13829) by @claudespice in #13904
• fix(bigquery): preserve results and federation for temporal and recursive queries by @bjchambers in #13905
• Make the SQL results cache hold what it says it holds by @bjchambers in #13908
• fix(duckdb): keep a call the dialect cannot render out of the federated plan (fixes #13900) by @claudespice in #13909
• fix(functions): discard cached plans when a hot reload changes the function set (refs #13873) by @claudespice in #13911
• fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
• fix(cli): keep the API key on its origin in the SDK-built query client (fixes #12502) by @grokspice in #13923
• fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
• ci: align default and ODBC build features by @phillipleblanc in #13933
• fix(github): bound the pull request page to GitHub's per-request compute budget (refs #13762) by @grokspice in #13938
• fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
• fix(ci): preserve Cargo discovery markers during runner disk sweeps by @phillipleblanc in #13940
• fix(bigquery): push down JSON scalar text and null checks by @phillipleblanc in #13944
• fix(models): read the HuggingFace chat token as hf_token again (fixes #13932) by @claudespice in #13946
• fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
• fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
• ci: upgrade spiceio setup action to v0.9.0 by @lukekim in #13971

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.1...v2.3.0

Announcing the release of Spice v1.5.1! 🔑

Spice v1.5.1 expands the GitHub data connector to include pull-request comments, adds a configurable rate limiting for AWS Bedrock embedding models, expands partition pruning with inequality operators, and adds client-supplied cache keys for granular caching control in the HTTP and Arrow Flight SQL APIs.

What's New in v1.5.1​

GitHub Data Connector Pull Request Comments: Configure GitHub pulls datasets to include comments.

Example Spicepod.yaml:

datasets:
  - from: github:github.com/spiceai/spiceai/pulls
    name: spiceai.pulls
    params:
      github_include_comments: all # 'review', 'discussion', or 'none'. Defaults to 'none'.
      github_max_comments_fetched: '25' # Defaults to 100
      # ...

For details, see the GitHub Data Connector documentation.

AWS Bedrock Embedding Models Invocation Control: Improved rate limiting control for AWS Bedrock embedding models with max_concurrent_invocations configuration.

embeddings:
  - from: bedrock:cohere.embed-english-v3
    name: cohere-embeddings
    params:
      max_concurrent_invocations: '41'
      # ...

For details, see the AWS Bedrock Embeddings Model Provider documentation.

Improved Query Partitioning: Expanded partition pruning support with additional inequality operators (e.g. >, >=, <, <=).

For details, see the Query Partitioning documentation.

Client-Supplied Cache Keys: Support for a new Spice-Cache-Key header/metadata-key in the HTTP and Arrow Flight SQL query APIs to for fine-grained client-side caching control.

Example HTTP API usage:

$ curl -vvS -XPOST http://localhost:8090/v1/sql \
-H"spice-cache-key: 1851400_20170216_north_america" \
-d "select * from scihub_journals_accessed
    where user_id = '1851400'
      and date_trunc('DAY', timestamp) = '2017-02-16'
      and city = 'New York';"

Example Response:

< HTTP/1.1 200 OK
< content-type: application/json
< x-cache: Hit from spiceai
< results-cache-status: HIT
< vary: Spice-Cache-Key
< vary: origin, access-control-request-method, access-control-request-headers
< content-length: 604
< date: Wed, 23 Jul 2025 20:26:12 GMT
<
[{
"timestamp": "2017-02-16 09:55:06",
"doi": "10.1155/2012/650929",
"ip_identifier": 1000856,
"user_id": 1851400,
"country": "United States",
"city": "New York",
"longitude": 40.7830603,
"latitude": -73.9712488
},
...
]

For details, see the Cache Control documentation.

Contributors​

• @Jeadie
• @Advayp
• @sgrebnov
• @kczimm
• @lukekim
• @phillipleblanc
• @mach-kernel
• @varunguleriaCodes
• @peasee
• @ewgenius

New Contributors​

• @varunguleriaCodes made their first contribution in github.com/spiceai/spiceai/pull/6383

Breaking Changes​

• N/A

Cookbook Updates​

No new recipes added in this release.

The Spice Cookbook includes 74 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.5.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.5.1 image:

docker pull spiceai/spiceai:1.5.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

No major dependency updates.

Changelog​

• Fix refresh via Api when dataset is already accelerated and no refresh interval is set by @sgrebnov in #6549
• Add support for custom GraphQL unnesting behavior by @Advayp in #6540
• Regex Update to disallow hyphens dataset names by @varunguleriaCodes in #6383
• Enforce max limit on comments fetched per PR by @Advayp in #6580
• Fix accelerated refresh issue by @Advayp in #6590
• Enable configurations of max invocations for Bedrock models by @Advayp in #6592
• Client-supplied cache keys (Spice-Cache-Key) by @mach-kernel in #6579
• Improved partition pruning by @kczimm in #6582
• Fix retention filter when both retention_sql and period are set by @sgrebnov in #6595
• Initial support for PR comments by @Advayp in #6569
• chore: Update croner by @peasee in #6547
• fix databricks streaming for Claude model by @peasee in #6601
• Remove FullTextUDTFAnalyzerRule and move FTS code into search crate by @jeadie in #6596
• Remove download of legacy sentence transformers config by @jeadie in #6605
• re-add snapshot tests by @jeadie
• Embedding column config to support client-specified vector sizes by @mach-kernel in #6610
• Fix mismatch in columns for the GitHub PR table type by @Advayp in #6616
• bump version to 1.5.1 by @phillipleblanc
• fix issues with cherry-picking by @jeadie
• Add integration tests for GitHub PRs with comments by @Advayp in #6581
• Add view name to view creation errors by @lukekim in #6611
• CDC: Compute embeddings on ingest by @mach-kernel in #6612

Announcing the release of Spice v1.5.0! 🔍

Spice v1.5.0 brings major upgrades to search and retrieval. It introduces native support for Amazon S3 Vectors, enabling petabyte scale vector search directly from S3 vector buckets, alongside SQL-integrated vector and tantivy-powered full-text search, partitioning for DuckDB acceleration, and automated refreshes for search indexes and views. It includes the AWS Bedrock Embeddings Model Provider, the Oracle Database connector, and the now-stable Spice.ai Cloud Data Connector, and the upgrade to DuckDB v1.3.2.

What's New in v1.5.0​

Amazon S3 Vectors Support: Spice.ai now integrates with Amazon S3 Vectors, launched in public preview on July 15, 2025, enabling vector-native object storage with built-in indexing and querying. This integration supports semantic search, recommendation systems, and retrieval-augmented generation (RAG) at petabyte scale with S3’s durability and elasticity. Spice.ai manages the vector lifecycle—ingesting data, creating embeddings with models like Amazon Titan or Cohere via AWS Bedrock, or others available on HuggingFace, and storing it in S3 Vector buckets.

Example Spicepod.yml configuration for S3 Vectors:

datasets:
  - from: s3://my_data_bucket/data/
    name: my_vectors
    params:
      file_format: parquet
    acceleration:
      enabled: true
    vectors:
      engine: s3_vectors
      params:
        s3_vectors_aws_region: us-east-2
        s3_vectors_bucket: my-s3-vectors-bucket
    columns:
      - name: content
        embeddings:
          - from: bedrock_titan
            row_id:
              - id

Example SQL query using S3 Vectors:

SELECT *
FROM vector_search(my_vectors, 'Cricket bats', 10)
WHERE price < 100
ORDER BY score

For more details, refer to the S3 Vectors Documentation.

SQL-integrated Search: Vector and BM25-scored full-text search capabilities are now natively available in SQL queries, extending the power of the POST v1/search endpoint to all SQL workflows.

Example Vector-Similarity-Search (VSS) using the vector_search UDTF on the table reviews for the search term "Cricket bats":

SELECT review_id, review_text, review_date, score
FROM vector_search(reviews, "Cricket bats")
WHERE country_code="AUS"
LIMIT 3

Example Full-Text-Search (FTS) using the text_search UDTF on the table reviews for the search term "Cricket bats":

SELECT review_id, review_text, review_date, score
FROM text_search(reviews, "Cricket bats")
LIMIT 3

DuckDB v1.3.2 Upgrade: Upgraded DuckDB engine from v1.1.3 to v1.3.2. Key improvements include support for adding primary keys to existing tables, resolution of over-eager unique constraint checking for smoother inserts, and 13% reduced runtime on TPC-H SF100 queries through extensive optimizer refinements. The v1.2.x release of DuckDB was skipped due to a regression in indexes.

• Read the DuckDB v1.2.0 announcement.
• Read the DuckDB v1.3.0 announcement.

Partitioned Acceleration: DuckDB file-based accelerations now support partition_by expressions, enabling queries to scale to large datasets through automatic data partitioning and query predicate pruning. New UDFs, bucket and truncate, simplify partition logic.

New UDFs useful for partition_by expressions:

• bucket(num_buckets, col): Partitions a column into a specified number of buckets based on a hash of the column value.
• truncate(width, col): Truncates a column to a specified width, aligning values to the nearest lower multiple (e.g., truncate(10, 101) = 100).

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/some_large_table/
    name: my_table
    params:
      file_format: parquet
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      partition_by: bucket(100, account_id) # Partition account_id into 100 buckets

Full-Text-Search (FTS) Index Refresh: Accelerated datasets with search indexes maintain up-to-date results with configurable refresh intervals.

Example refreshing search indexes on body every 10 seconds:

datasets:
  - from: github:github.com/spiceai/docs/pulls
    name: spiceai.doc.pulls
    params:
      github_token: ${secrets:GITHUB_TOKEN}
    acceleration:
      enabled: true
      refresh_mode: full
      refresh_check_interval: 10s
    columns:
      - name: body
        full_text_search:
          enabled: true
          row_id:
            - id

Scheduled View Refresh: Accelerated Views now support cron-based refresh schedules using refresh_cron, automating updates for accelerated data.

Example Spicepod.yml configuration:

views:
  - name: my_view
    sql: SELECT 1
    acceleration:
      enabled: true
      refresh_cron: '0 * * * *' # Every hour

For more details, refer to Scheduled Refreshes.

Multi-column Vector Search: For datasets configured with embeddings on more than one column, POST v1/search and similarity_search perform parallel vector search on each column, aggregating results using reciprocal rank fusion.

Example Spicepod.yml for multi-column search:

datasets:
  - from: github:github.com/apache/datafusion/issues
    name: datafusion.issues
    params:
      github_token: ${secrets:GITHUB_TOKEN}
    columns:
      - name: title
        embeddings:
          - from: hf_minilm
      - name: body
        embeddings:
          - from: openai_embeddings

AWS Bedrock Embeddings Model Provider: Added support for AWS Bedrock embedding models, including Amazon Titan Text Embeddings and Cohere Text Embeddings.

Example Spicepod.yml:

embeddings:
  - from: bedrock:cohere.embed-english-v3
    name: cohere-embeddings
    params:
      aws_region: us-east-1
      input_type: search_document
      truncate: END
  - from: bedrock:amazon.titan-embed-text-v2:0
    name: titan-embeddings
    params:
      aws_region: us-east-1
      dimensions: '256'

For more details, refer to the AWS Bedrock Embedding Models Documentation.

Oracle Data Connector: Use from: oracle: to access and accelerate data stored in Oracle databases, deployed on-premises or in the cloud.

Example Spicepod.yml:

datasets:
  - from: oracle:"SH"."PRODUCTS"
    name: products
    params:
      oracle_host: 127.0.0.1
      oracle_username: scott
      oracle_password: tiger

See the Oracle Data Connector documentation.

GitHub Data Connector: The GitHub data connector supports query and acceleration of members, the users of an organization.

Example Spicepod.yml configuration:

datasets:
  - from: github:github.com/spiceai/members # General format: github.com/[org-name]/members
    name: spiceai.members
    params:
      # With GitHub Apps (recommended)
      github_client_id: ${secrets:GITHUB_SPICEHQ_CLIENT_ID}
      github_private_key: ${secrets:GITHUB_SPICEHQ_PRIVATE_KEY}
      github_installation_id: ${secrets:GITHUB_SPICEHQ_INSTALLATION_ID}
      # With GitHub Tokens
      # github_token: ${secrets:GITHUB_TOKEN}

See the GitHub Data Connector Documentation

Spice.ai Cloud Data Connector: Graduated to Stable.

spice-rs SDK Release: The Spice Rust SDK has updated to v3.0.0. This release includes optimizations for the Spice client API, adds robust query retries, and custom metadata configurations for spice queries.

Contributors​

• @Jeadie
• @peasee
• @sgrebnov
• @Sevenannn
• @kczimm
• @phillipleblanc
• @Advayp
• @lukekim
• @ewgenius
• @mach-kernel
• @suhuruli

Breaking Changes​

• Search HTTP API Response: POST v1/search response payload has changed. See the new API documentation for details.
• Model Provider Parameter Prefixes: Model Provider parameters use provider-specific prefixes instead of openai_ prefixes (e.g., hf_temperature for HuggingFace, anthropic_max_completion_tokens for Anthropic, perplexity_tool_choice for Perplexity). The openai_ prefix remains supported for backward compatibility but is deprecated and will be removed in a future release.

Cookbook Updates​

• Added Oracle Data Connector cookbook: Connect to tables in Oracle databases.
• Added Hashed Partitioning with DuckDB cookbook: Accelerate data on large datasets by partitioning data into a fixed number of buckets.

The Spice Cookbook now includes 72 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.5.0, download and install the specific binary from github.com/spiceai/spiceai/releases/tag/v1.5.0 or pull the v1.5.0 Docker image (spiceai/spiceai:1.5.0).

What's Changed​

Dependencies​

• delta_kernel: Upgraded to v0.12.1
• DuckDB: Upgraded from v1.1.3 to v1.3.2
• iceberg-rust: Upgraded from v0.4.0 to v0.5.1

Changelog​

• fix: openai model endpoint (#6394) by @Sevenannn in #6394
• Enable configuring otel endpoint from spice run (#6360) by @Advayp in #6360
• Enable Oracle connector in default build configuration (#6395) by @sgrebnov in #6395
• fix llm integraion test (#6398) by @Sevenannn in #6398
• Promote spice cloud connector to stable quality (#6221) by @Sevenannn in #6221
• v1.5.0-rc.1 release notes (#6397) by @lukekim in #6397
• Fix model nsql integration tests (#6365) by @Sevenannn in #6365
• Fix incorrect UDTF name and SQL query (#6404) by @lukekim in #6404
• Update v1.5.0-rc.1.md (#6407) by @sgrebnov in #6407
• Improve error messages (#6405) by @lukekim in #6405
• build(deps): bump Jimver/cuda-toolkit from 0.2.25 to 0.2.26 (#6388) by @app/dependabot in #6388
• Upgrade dependabot dependencies (#6411) by @phillipleblanc in #6411
• Fix projection pushdown issues for document based file connector (#6362) by @Advayp in #6362
• Add a PartitionedDuckDB Accelerator (#6338) by @kczimm in #6338
• Use vector_search() UDTF in HTTP APIs (#6417) by @Jeadie in #6417
• add supported types (#6409) by @kczimm in #6409
• Enable session time zone override for MySQL (#6426) by @sgrebnov in #6426
• Acceleration-like indexing for full text search indexes. (#6382) by @Jeadie in #6382
• Provide error message when partition by expression changes (#6415) by @kczimm in #6415
• Add support for Oracle Autonomous Database connections (Oracle Cloud) (#6421) by @sgrebnov in #6421
• prune partitions for exact and in list with and without UDFs (#6423) by @kczimm in #6423
• Fixes and reenable FTS tests (#6431) by @Jeadie in #6431
• Upgrade DuckDB to 1.3.2 (#6434) by @phillipleblanc in #6434
• Fix issue in limit clause for the Github Data connector (#6443) by @Advayp in #6443
• Upgrade iceberg-rust to 0.5.1 (#6446) by @phillipleblanc in #6446
• v1.5.0-rc.2 release notes (#6440) by @lukekim in #6440
• Oracle: add automated TPC-H SF1 benchmark tests (#6449) by @sgrebnov in #6449
• fix: Update benchmark snapshots (#6455) by @app/github-actions in #6455
• Preserve ArrowError in arrow_tools::record_batch (#6454) by @mach-kernel in #6454
• fix: Update benchmark snapshots (#6465) by @app/github-actions in #6465
• Add option to preinstall Oracle ODPI-C library in Docker image (#6466) by @sgrebnov in #6466
• Include Oracle connector (federated mode) in automated benchmarks (#6467) by @sgrebnov in #6467
• Update crates/llms/src/bedrock/embed/mod.rs by @lukekim in #6468
• v1.5.0-rc.3 release notes (#6474) by @lukekim in #6474
• Add integration tests for S3 Vectors filters pushdown (#6469) by @sgrebnov in #6469
• check for indexedtableprovider when finding tables to search on (#6478) by @Jeadie in #6478
• Parse fully qualified table names in UDTFs (#6461) by @Jeadie in #6461
• Add integration test for S3 Vectors to cover data update (overwrite) (#6480) by @sgrebnov in #6480
• Add 'Run all tests' option for models tests and enable Bedrock tests (#6481) by @sgrebnov in #6481
• Add support for a members table type for the GitHub Data Connector (#6464) by @Advayp in #6464
• S3 vector data cannot be null (#6483) by @Jeadie in #6483
• Don't infer FixedSizeList size during indexing vectors. (#6487) by @Jeadie in #6487
• Add support for retention_sql acceleration param (#6488) by @sgrebnov in #6488
• Make dataset refresh progress tracing less verbose (#6489) by @sgrebnov in #6489
• Use RwLock on tantivy index in FullTextDatabaseIndex for update concurrency (#6490) by @Jeadie in #6490
• Add tests for dataset retention logic and refactor retention code (#6495) by @sgrebnov in #6495
• Upgade dependabot dependencies (#6497) by @phillipleblanc in #6497
• Add periodic tracing of data loading progress during dataset refresh (#6499) by @sgrebnov in #6499
• Promote Oracle Data Connector to Alpha (#6503) by @sgrebnov in #6503
• Use AWS SDK to provide credentials for Iceberg connectors (#6498) by @phillipleblanc in #6498
• Add integration tests for partitioning (#6463) by @kczimm in #6463
• Use top-level table in full-text search JOIN ON (#6491) by @Jeadie in #6491
• Use accelerated table in vector_search JOIN operations when appropriate (#6516) by @Jeadie in #6516
• Fix 'additional_column' for quoted columns (fix for qualified columns broke it) (#6512) by @Jeadie in #6512
• Also use AWS SDK for inferring credentials for S3/Delta/Databricks Delta data connectors (#6504) by @phillipleblanc in #6504
• Add per-dataset availability monitor configuration (#6482) by @phillipleblanc in #6482
• Suppress the warning from the AWS SDK if it can't load credentials (#6533) by @phillipleblanc in #6533
• Change default value of check_availability from default to auto (#6534) by @lukekim in #6534
• README.md improvements for v1.5.0 (#6539) by @lukekim in #6539
• Temporary disable s3_vectors_basic (#6537) by @sgrebnov in #6537
• Ensure binder errors show before query and other (#6374) by @suhuruli in #6374
• Update spiceai/duckdb-rs -> DuckDB 1.3.2 + index fix (#6496) by @mach-kernel in #6496
• Update table-providers to latest version with DuckDB fixes (#6535) by @phillipleblanc in #6535
• S3: default to public access if no auth is provided (#6532) by @sgrebnov in #6532

Happy New Year 🎆!

Announcing the release of Spice v1.0-rc.4 🌟

Spice v1.0.0-rc.4 is the fourth release candidate for the first major version of Spice.ai OSS. This release continues the focus on production readiness. In addition, xAI has been added as a model provider.

Highlights in v1.0-rc.4​

• xAI Model Provider: Adds support for xAI hosted models.

models:
  - from: xai:grok2-latest
    name: xai
    params:
      xai_api_key: ${secrets:SPICE_XAI_API_KEY}

• Spicepod Spec Version: Spicepod spec version v1 is now by default. v1beta1 will continue to work.

version: v1
kind: Spicepod
name: my_pod

• GitHub Data Connector: Graduated to Stable.

• PostgreSQL Data Accelerator: Graduated to Release Candidate (RC).

Cookbook​

• Added xAI model provider recipe.

Dependencies​

No major dependency changes.

Contributors​

• @lukekim
• @phillipleblanc
• @peasee
• @karifabri
• @sgrebnov
• @Jeadie
• @ewgenius

What's Changed​

- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4087>
- Update Helm chart for v1.0.0-rc.3 (v0.2.2) by @lukekim in <https://github.com/spiceai/spiceai/pull/4088>
- Rev version to v1.0.0-rc.4 by @lukekim in <https://github.com/spiceai/spiceai/pull/4090>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4089>
- Fix OpenAI Models Integration tests by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4084>
- fix: Update Postgres TPCDS and ClickBench queries by @peasee in <https://github.com/spiceai/spiceai/pull/4092>
- fix: Check Postgres acceleration schema on insert by @peasee in <https://github.com/spiceai/spiceai/pull/4094>
- Update v1.0.0-rc.3.md by @karifabri in <https://github.com/spiceai/spiceai/pull/4096>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4093>
- First-class TSV for file data connector by @lukekim in <https://github.com/spiceai/spiceai/pull/4098>
- Allow Flight DoPut only for write api-keys by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4010>
- Only create tables `eval.runs` and `eval.results` when an eval is defined by @Jeadie in <https://github.com/spiceai/spiceai/pull/4099>
- Update Copyright year to include 2025 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4100>
- feat: add postgres clickbench accelerator, release postgres accelerator by @peasee in <https://github.com/spiceai/spiceai/pull/4111>
- Add spice binaries with metal to releases; detect metal device in `spice install/upgrade`. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4097>
- docs: Clarify connector release criteria by @peasee in <https://github.com/spiceai/spiceai/pull/4112>
- Update datafusion-federation to fix LIMIT with OFFSET handling in logical plan rewrite by @ewgenius in <https://github.com/spiceai/spiceai/pull/4115>
- Support Grok AI. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4113>
- Fix `spice chat` usage bar. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4119>
- Set unified max encoding and decoding message size for all flight client configurations across runtime by @ewgenius in <https://github.com/spiceai/spiceai/pull/4116>
- feat: Add the file connector as an appendable benchmark connector by @peasee in <https://github.com/spiceai/spiceai/pull/4120>
- Add `spice eval` command by @lukekim in <https://github.com/spiceai/spiceai/pull/4118>
- Support multi-level table nesting for Dremio by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4129>
- feat: run append TPCH benchmarks in workflow (Arrow, DuckDB) by @peasee in <https://github.com/spiceai/spiceai/pull/4131>
- Fix bug in Iceberg tables selecting a subset of columns by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4132>
- feat: Run append TPCDS benchmarks in workflow (Arrow, DuckDB) by @peasee in <https://github.com/spiceai/spiceai/pull/4141>
- Setup spice.ai clickbench by @ewgenius in <https://github.com/spiceai/spiceai/pull/4134>
- Data is streamed when reading from the GitHub connector (GraphQL tables) by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4142>
- Mark the GitHub Data Connector as Stable by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4143>
- Fix table quoting for Databricks Spark connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4145>
- Extend flight compute context for spice.ai connector with org and app names, to fix federated queries from different spice.ai data sources by @ewgenius in <https://github.com/spiceai/spiceai/pull/4144>
- Enforce Flight DoPut policies: Rate Limiting, Read Timeout, and Max Records per Batch by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4117>
- Fix bug Changes in catalog.yaml would require saving in spicepod.yaml to apply by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4147>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/4137>
- Add `test-framework` crate to contain all common benchmark, E2E, integration testing logic. by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4157>
- Fix `platform_option` variable in `build_and_release.yml`. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4154>
- feat: Add Clickbench append benchmark for DuckDB and Arrow by @peasee in <https://github.com/spiceai/spiceai/pull/4160>
- Upload artifacts to Minio on build_and_release by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4159>
- feat: add on zero results benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/4164>
- Update spice.ai connector tests by @ewgenius in <https://github.com/spiceai/spiceai/pull/4161>

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v1.0.0-rc.3...v1.0.0-rc.4>
```text

## Resources

- [Getting started with Spice.ai](https://docs.spiceai.org/getting-started/)
- [Documentation](https://docs.spiceai.org/)

## Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

- Twitter: [@spice_ai](https://twitter.com/spice_ai)
- Slack: [spiceai.org/slack](https://spiceai.org/slack)
- Telegram: [Spice AI Discussion](https://t.me/spiceaichat)
- Reddit: [https://www.reddit.com/r/spiceai](https://www.reddit.com/r/spiceai)
- Email: [[email protected]](mailto:[email protected])

Announcing the release of Spice v0.20-beta 🧩

Spice v0.20.0-beta improves federated query performance with column pruning and adds support for Metal (Apple Silicon) and CUDA (NVidia) accelerators. The S3, PostgreSQL, MySQL, and GitHub Data Connectors have graduated from Beta to Release Candidates. The Arrow, DuckDB, and SQLite Data Accelerators have graduated from Alpha to Beta.

Highlights in v0.20.0-beta​

Data Connectors: The S3, PostgreSQL, MySQL, and GitHub Data Connectors have graduated from beta to release candidate.

Data Accelerators: The Arrow, DuckDB, and SQLite Data Accelerators have graduated from alpha to beta.

Metal and CUDA Support: Added support for Metal (Apple Silicon) and CUDA (NVidia) for AI/ML workloads including embeddings and local LLM inference.

For instructions on compiling a Meta or CUDA binary, see the Installation Docs.

Breaking Changes​

• The ODBC Data Connector now requires ODBC drivers specified in connection strings are registered in the system ODBC driver manager.

Example invalid connection string:

DRIVER={/path/to/driver.so};SERVER=localhost;DATABASE=master

Example valid connection string:

DRIVER={My ODBC Driver};SERVER=localhost;DATABASE=master

Where My ODBC Driver is the name of an ODBC driver registered in the ODBC driver manager.

Contributors​

• @ewgenius
• @peasee
• @phillipleblanc
• @sgrebnov
• @Jeadie
• @barracudarin
• @Sevenannn

What's Changed​

- Update Helm for v0.19.4-beta and add release notes by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3310>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/3311>
- `metal` & `cuda` flags for spice by @Jeadie in <https://github.com/spiceai/spiceai/pull/3212>
- Promote postgres connector to RC quality by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3305>
- docs: Update ROADMAP.md by @peasee in <https://github.com/spiceai/spiceai/pull/3322>
- feat: Enable federation for in-memory accelerators by @peasee in <https://github.com/spiceai/spiceai/pull/3325>
- fix: Only allow env files from the current dir by @peasee in <https://github.com/spiceai/spiceai/pull/3327>
- Always read TimezoneTZ from PostgreSQL as UTC by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3330>
- For multi-sink acceleration refreshes, ensure parent table completes before the children. by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3329>
- Update TPC-DS Q49 (Decimal to Float) to match SQLite's type system by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3323>
- Enable parquet pushdown in Spice by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3245>
- Use spice object_store fork to fix S3 ambiguous error by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3304>
- Don't mix commented out queries for s3 connectors and accelerators by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3331>
- Allow only valid WHERE conditions in vector searches by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3335>
- fix: Allow only ODBC profiles by @peasee in <https://github.com/spiceai/spiceai/pull/3324>
- Track how many times an acceleration falls back during initialization by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3339>
- Anthropic model regex and fix tool parsing aggregation bug by @Jeadie in <https://github.com/spiceai/spiceai/pull/3334>
- Upgrade runtime along with CLI on `spice upgrade` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3341>
- Update upcoming Roadmap by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3343>
- fix: Prevent acceleration files outside of working directory by @peasee in <https://github.com/spiceai/spiceai/pull/3340>
- Document S3 connector limitations by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3333>
- Update Object Store Patch by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3361>
- Promote SQLite Data Accelerator to Beta by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3365>
- Promote S3 connector to RC quality by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3362>
- Revert "fix: Only allow env files from the current dir" by @peasee in <https://github.com/spiceai/spiceai/pull/3368>
- docs: Fix typo for S3 release status in README.md by @peasee in <https://github.com/spiceai/spiceai/pull/3370>
- Include unnecessary columns pruning step during federated plan creation by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3363>

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v0.19.4-beta...v0.20.0-beta>

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]

Announcing the release of Spice v0.19.3-beta 📈

Spice v0.19.3-beta improves the performance and stability of data connectors and accelerators, including faster queries across multiple federated sources by optimizing how filters are applied. Anthropic has also been added as a LLM model provider.

Highlights in v0.19.3​

DataFusion Fixes: Resolved bugs in DataFusion and DataFusion Table Providers, expanding TPC-DS coverage and correctness.

GitHub Data Connector Beta Milestone: The GitHub Data Connector has graduated to Beta after extensive testing, stability, and performance improvements.

Anthropic Models Provider: Anthropic has been added as an LLM provider, including support for streaming.

Example spicepod.yml:

models:
  - from: anthropic:claude-3-5-sonnet-20240620
    name: claude_3_5_sonnet
    params:
      anthropic_api_key: ${ secrets:SPICE_ANTHROPIC_API_KEY }

Breaking changes​

None.

Contributors​

• @Jeadie
• @Sevenannn
• @phillipleblanc
• @peasee
• @sgrebnov
• @nlamirault
• @barracudarin
• @lukekim
• @slyons

New Contributors​

• @nlamirault made their first contribution in https://github.com/spiceai/spiceai/pull/3207
• @barracudarin made their first contribution in https://github.com/spiceai/spiceai/pull/3228

What's Changed​

- Make Anthropic OpenAI compatible. by @Jeadie in https://github.com/spiceai/spiceai/pull/3087
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/3200
- Bump version to 1.0.0-rc.1 by @Sevenannn in https://github.com/spiceai/spiceai/pull/3202
- Fix clickhouse schema inference for non-default database by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3201
- Update endgame template by @Sevenannn in https://github.com/spiceai/spiceai/pull/3198
- Upgrade dependencies by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3197
- fix: dataset refresh defaults properties to None by @peasee in https://github.com/spiceai/spiceai/pull/3205
- Upgrade OTEL to v0.26 and make seconds based metrics reported precisely by @sgrebnov in https://github.com/spiceai/spiceai/pull/3203
- use `text_embedding_inference::Infer` for more complete embedding solution by @Jeadie in https://github.com/spiceai/spiceai/pull/3199
- Add S3 parquet file - arrow accelerator e2e test by @Sevenannn in https://github.com/spiceai/spiceai/pull/3154
- feat: Add script to setup clickbench on mysql by @peasee in https://github.com/spiceai/spiceai/pull/3176
- Update helm chart version to v0.19.2 by @Sevenannn in https://github.com/spiceai/spiceai/pull/3210
- Add sample dataset option in `v1/nsql`. by @Jeadie in https://github.com/spiceai/spiceai/pull/3105
- Split spiced_docker build across architectures by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3206
- feat(helm): do not install demo dataset by default by @nlamirault in https://github.com/spiceai/spiceai/pull/3207
- Split integration test across build/run steps by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3215
- feat(helm): Refactoring Kubernetes labels by @nlamirault in https://github.com/spiceai/spiceai/pull/3208
- Define 'tool_recursion_limit' for LLMs, and limit internal tool calling recursion. by @Jeadie in https://github.com/spiceai/spiceai/pull/3214
- Improve filters pushdown for federated queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/3183
- Implement native schema inference for PostgreSQL by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3209
- docs: Update release criteria by @peasee in https://github.com/spiceai/spiceai/pull/3219
- Run SQLite acceleration TPC-DS tests using smaller scale by @sgrebnov in https://github.com/spiceai/spiceai/pull/3227
- bind the serviceAccount if a name is given or if we're creating one by @barracudarin in https://github.com/spiceai/spiceai/pull/3228
- Only emit channel send error log when its not a closed channel error by @Jeadie in https://github.com/spiceai/spiceai/pull/3230
- Enable Parquet Exec filter pushdown in Spice by @Sevenannn in https://github.com/spiceai/spiceai/pull/3216
- Add snapshots for SQLite TPC-DS benchmark (file mode) by @sgrebnov in https://github.com/spiceai/spiceai/pull/3234
- docs: Add SDK release checks to endgame by @peasee in https://github.com/spiceai/spiceai/pull/3256
- Implement `localpod` Data Connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3249
- Revert "Enable Parquet Exec filter pushdown in Spice (#3216)" by @Sevenannn in https://github.com/spiceai/spiceai/pull/3244
- refactor: Use existing action for detecting changes by @peasee in https://github.com/spiceai/spiceai/pull/3255
- feat: Add GitHub integration test by @peasee in https://github.com/spiceai/spiceai/pull/3226
- Add get_readiness tool to retrieve status of all registered components by @lukekim in https://github.com/spiceai/spiceai/pull/3035
- Improve CLI error output when REPL can't connect to the Flight endpoint by @slyons in https://github.com/spiceai/spiceai/pull/3188
- Fixing FTP link in Endgame by @slyons in https://github.com/spiceai/spiceai/pull/3267
- Update version to 0.19.3-beta by @sgrebnov in https://github.com/spiceai/spiceai/pull/3269
- add service type and annotation customizations in https://github.com/spiceai/spiceai/pull/3268

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.19.2-beta...v0.19.3-beta

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]

Announcing the release of Spice v0.19.1-beta 🔥

Spice v0.19.1 brings further performance and stability improvements to data connectors, including improved query push-down for file-based connectors (s3, abfs, file, ftp, sftp) that use Hive-style partitioning.

Highlights in v0.19.1​

TPC-H and TPC-DS Coverage: Expanded coverage for TPC-H and TPC-DS benchmarking suites across accelerators and connectors.

GitHub Connector Array Filter: The GitHub connector now supports filter push down for the array_contains function in SQL queries using search query mode.

NSQL CLI Command: A new spice nsql CLI command has been added to easily query datasets with natural language from the command line.

Breaking changes​

None

Contributors​

• @peasee
• @Sevenannn
• @sgrebnov
• @karifabri
• @phillipleblanc
• @lukekim
• @Jeadie
• @slyons

Dependencies​

• DataFusion Table Providers: Upgraded to rev f22b96601891856e02a73d482cca4f6100137df8.

What's Changed​

- release: Update helm chart for v0.19.0-beta by @peasee in https://github.com/spiceai/spiceai/pull/3024
- Set fail-fast = true for benchmark test by @Sevenannn in https://github.com/spiceai/spiceai/pull/2997
- release: Update next version and ROADMAP by @peasee in https://github.com/spiceai/spiceai/pull/3033
- Verify TPCH benchmark query results for Spark connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/2993
- feat: Add x-spice-user-agent header to Spice REPL by @peasee in https://github.com/spiceai/spiceai/pull/2979
- Update to object store file formats documentation link by @karifabri in https://github.com/spiceai/spiceai/pull/3036
- Use teraswitch-runners for Linux x64 workflows + builds by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3042
- feat: Support array contains in GitHub pushdown by @peasee in https://github.com/spiceai/spiceai/pull/2983
- Bump text-splitter from 0.16.1 to 0.17.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2987
- Revert integration tests back to hosted runner by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3046
- Tune Github runner resources to allow in memory TPCDS benchmark to run by @Sevenannn in https://github.com/spiceai/spiceai/pull/3025
- fix: add winver by @peasee in https://github.com/spiceai/spiceai/pull/3054
- refactor: Use is modifier for checking GitHub state filter by @peasee in https://github.com/spiceai/spiceai/pull/3056
- Enable `merge_group` checks for PR workflows by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3058
- Fix issues with merge group by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3059
- Validate in-memory arrow accelertion TPCDS result correctness by @Sevenannn in https://github.com/spiceai/spiceai/pull/3044
- Fix rev parsing for PR checks by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3060
- Use 'Accept' header for `/v1/sql/` and `/v1/nsql` by @Jeadie in https://github.com/spiceai/spiceai/pull/3032
- Verify Postgres acceleration TPCDS result correctness by @Sevenannn in https://github.com/spiceai/spiceai/pull/3043
- Add NSQL CLI REPL command by @lukekim in https://github.com/spiceai/spiceai/pull/2856
- Preserve query results order and add TPCH benchmark results verification for duckdb:file mode by @sgrebnov in https://github.com/spiceai/spiceai/pull/3034
- Refactor benchmark to include MySQL tpcds bench, tweaks to makefile target for generating mysql tpcds data by @Sevenannn in https://github.com/spiceai/spiceai/pull/2967
- Support runtime parameter for `sql_query_keep_partition_by_columns` & enable by default by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3065
- Document TPC-DS limitations: `EXCEPT`, `INTERSECT`, duplicate names by @sgrebnov in https://github.com/spiceai/spiceai/pull/3069
- Adding ABFS benchmark by @slyons in https://github.com/spiceai/spiceai/pull/3062
- Add support for GitHub app installation auth for GitHub connector by @ewgenius in https://github.com/spiceai/spiceai/pull/3063
- docs: Document stack overflow workaround, add helper script by @peasee in https://github.com/spiceai/spiceai/pull/3070
- Tune MySQL TPCDS image to allow for successful benchmark test run by @Sevenannn in https://github.com/spiceai/spiceai/pull/3067
- Automatically infer partitions for hive-style partitioned files for object store based connectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3073
- Support `hf_token` from params/secrets by @Jeadie in https://github.com/spiceai/spiceai/pull/3071
- Inherit embedding columns from source, when available. by @Jeadie in https://github.com/spiceai/spiceai/pull/3045
- Validate identifiers for component names by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3079
- docs: Add workaround for TPC-DS Q97 in MySQL by @peasee in https://github.com/spiceai/spiceai/pull/3080
- Document TPC-DS Postgres column alias in a CASE statement limitation by @sgrebnov in https://github.com/spiceai/spiceai/pull/3083
- Update plan snapshots for TPC-H bench queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/3088
- Update Datafusion crate to include recent unparsing fixes by @sgrebnov in https://github.com/spiceai/spiceai/pull/3089
- Sample SQL table data tool and API by @Jeadie in https://github.com/spiceai/spiceai/pull/3081
- chore: Update datafusion-table-providers by @peasee in https://github.com/spiceai/spiceai/pull/3090
- Add `hive_infer_partitions` to remaining object store connectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3086
- deps: Update datafusion-table-providers by @peasee in https://github.com/spiceai/spiceai/pull/3093
- For local embedding models, return usage input tokens. by @Jeadie in https://github.com/spiceai/spiceai/pull/3095
- Update end_game.md with Accelerator/Connector criteria check by @slyons in https://github.com/spiceai/spiceai/pull/3092
- Update TPC-DS Q90 by @sgrebnov in https://github.com/spiceai/spiceai/pull/3094
- docs: Add RC connector criteria by @peasee in https://github.com/spiceai/spiceai/pull/3026
- Update version to 0.19.1-beta by @sgrebnov in https://github.com/spiceai/spiceai/pull/3101

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.19.0-beta...v0.19.1-beta

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]

Announcing the release of Spice v0.18.3-beta 🛠️

The Spice v0.18.3-beta release includes several quality-of-life improvements including verbosity flags for spiced and the Spice CLI, vector search over larger documents with support for chunking dataset embeddings, and multiple performance enhancements. Additionally, the release includes several bug fixes, dependency updates, and optimizations, including updated table providers and significantly improved GitHub data connector performance for issues and pull requests.

Highlights in v0.18.3-beta​

GitHub Query Mode: A new github_query_mode: search parameter has been added to the GitHub Data Connector, which uses the GitHub Search API to enable faster and more efficient query of issues and pull requests when using filters.

Example spicepod.yml:

- from: github:github.com/spiceai/spiceai/issues/trunk
  name: spiceai.issues
  params:
    github_query_mode: search # Use GitHub Search API
    github_token: ${secrets:GITHUB_TOKEN}

Output Verbosity: Higher verbosity output levels can be specified through flags for both spiced and the Spice CLI.

Example command line:

spice -v
spice --very-verbose

spiced -vv
spiced --verbose

Embedding Chunking: Chunking can be enabled and configured to preprocess input data before generating dataset embeddings. This improves the relevance and precision for larger pieces of content.

Example spicepod.yml:

- name: support_tickets
  embeddings:
    - column: conversation_history
      use: openai_embeddings
      chunking:
        enabled: true
        target_chunk_size: 128
        overlap_size: 16
        trim_whitespace: true

For details, see the Search Documentation.

Dependencies​

• DataFusion Table Providers: Upgraded to rev b0af91992699ecbf5adf2036a07122578f06150e.

Contributors​

• @Sevenannn
• @peasee
• @Jeadie
• @sgrebnov
• @phillipleblanc
• @ewgenius
• @slyons

What's Changed​

- Update datafusion table provider patch by @Sevenannn in https://github.com/spiceai/spiceai/pull/2817
- refactor: Set max_rows_per_batch for ODBC to 4000 by @peasee in https://github.com/spiceai/spiceai/pull/2822
- Use User message for health check by @Jeadie in https://github.com/spiceai/spiceai/pull/2823
- Upgrade Helm chart (Spice v0.18.2-beta) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2820
- Add verbosity flags for spiced, spice: `-v`, `-vv`, `--verbose`, `--very-verbose`. by @Jeadie in https://github.com/spiceai/spiceai/pull/2831
- Rename `spiceai` data connector to `spice.ai` by @sgrebnov in https://github.com/spiceai/spiceai/pull/2680
- Prepare for v0.19.0-beta release (version bump) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2821
- Bump clap from 4.5.17 to 4.5.18 (#2801) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2848
- Enable "rc" feature for serde in spicepod crate by @ewgenius in https://github.com/spiceai/spiceai/pull/2851
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2852
- chore: update table providers by @peasee in https://github.com/spiceai/spiceai/pull/2858
- fix: Use GitHub search for issues in GraphQL by @peasee in https://github.com/spiceai/spiceai/pull/2845
- fix: Use GitHub search for pull_requests by @peasee in https://github.com/spiceai/spiceai/pull/2847
- Support chunking dataset embeddings by @Jeadie in https://github.com/spiceai/spiceai/pull/2854
- refactor: Update GraphQL client to be more robust for filter push down by @peasee in https://github.com/spiceai/spiceai/pull/2864
- docs: Update accelerator beta criteria by @peasee in https://github.com/spiceai/spiceai/pull/2865
- Change `BytesProcessedRule` to be an optimizer rather than an analyzer rule by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2867
- Don't run E2E or PR tests on documentation by @Jeadie in https://github.com/spiceai/spiceai/pull/2869
- Verify benchmark query results using snapshot testing (spice.ai connector) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2866
- feat: Add GraphQLOptimizer by @peasee in https://github.com/spiceai/spiceai/pull/2868
- Update quickstarts for Endgame by @Jeadie in https://github.com/spiceai/spiceai/pull/2863
- Update version to v0.18.3-beta by @sgrebnov in https://github.com/spiceai/spiceai/pull/2882
- Update DataFusion: fix coalesce, Aggregation with Window functions unparsing support by @sgrebnov in https://github.com/spiceai/spiceai/pull/2884
- Revert "Rename `spiceai` data connector to `spice.ai`" by @sgrebnov in https://github.com/spiceai/spiceai/pull/2881
- Adding integration test for DuckDB read functions by @slyons in https://github.com/spiceai/spiceai/pull/2857
- Show more informative mysql error message by @Sevenannn in https://github.com/spiceai/spiceai/pull/2883
- Fix `no process-level CryptoProvider available` when using REPL and TLS by @sgrebnov in https://github.com/spiceai/spiceai/pull/2887
- Change UX for chunking and enable overlap_size in chunking by @Jeadie in https://github.com/spiceai/spiceai/pull/2890
- Add `log/slog` to spice CLI tool by @Jeadie in https://github.com/spiceai/spiceai/pull/2859
- feat: Add GitHub GraphQLOptimizer by @peasee in https://github.com/spiceai/spiceai/pull/2870
- Fix mysql invalid tablename error message by @Sevenannn in https://github.com/spiceai/spiceai/pull/2896
- fix: Remove login column rename in pulls and update Optimizer by @peasee in https://github.com/spiceai/spiceai/pull/2897
- Fix require check checking. by @Jeadie in https://github.com/spiceai/spiceai/pull/2898

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.18.2-beta...v0.18.3-beta

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]

Announcing the release of Spice v0.18-beta.

The v0.18.0-beta release adds new Sharepoint and File data connectors, introduces AWS Identity and Access Management (IAM) support for the S3 Data Connector, improves performance of the GitHub connector, and increases the overall reliability of all data accelerators. The /ready API endpoint was enhanced to report as ready only when all components, including loaded data, have successfully reported readiness.

Highlights in v0.18.0-beta​

Sharepoint Data Connector: Use from: sharepoint: to access and accelerate documents stored in Microsoft 365 OneDrive for Business (Sharepoint). The CLI also includes a new spice login sharepoint to aid in local development and testing.

Example spicepod.yml:

datasets:
  - from: sharepoint:drive:Documents/path:/important_documents/
    name: important_documents
    params:
      sharepoint_client_id: ${secrets:SPICE_SHAREPOINT_CLIENT_ID}
      sharepoint_tenant_id: ${secrets:SPICE_SHAREPOINT_TENANT_ID}
      sharepoint_client_secret: ${secrets:SPICE_SHAREPOINT_CLIENT_SECRET}

See the Sharepoint Data Connector documentation.

AWS Identity and Access Management (IAM) for S3: A new s3_auth parameter for the s3 data connector to configure the authentication method to use when connecting to S3. Supported values are public, key, and iam_role. Use s3_auth: iam_role to assume the instance IAM role.

Example spicepod.yml:

datasets:
  - from: s3://my-bucket
    name: bucket
    params:
      s3_auth: iam_role # Assume IAM role of instance

See the S3 Data Connector documentation.

File Data Connector Use from: file: to query files stored by locally accessible filesystems.

Example spicepod.yml:

datasets:
  - from: file://path/to/customer.parquet
    name: customer
    params:
      file_format: parquet

See the File Data Connector documentation.

Improved /ready Api Now includes the initial data load for accelerated datasets in addition to component readiness to ensure readiness is only reported when data has loaded and can be successfully queried.

Breaking Changes​

• GitHub Data Connector: The data type for time-related columns has changed from Utf8 to Timestamp. To upgrade, data type references to timestamp. For example, if using time_format:, change uses of time_format: ISO8601 to time_format: timestamp.

• Ready API: The /ready API reports ready only when all components have reported ready and data is fully loaded. To upgrade, evaluate uses of the Ready API (such as Kubernetes readiness probes) and consider how it might affect system behavior.

Dependencies​

No major dependencies updates.

Contributors​

• @phillipleblanc
• @Jeadie
• @lukekim
• @sgrebnov
• @peasee
• @eltociear
• @Sevenannn
• @ewgenius
• @karifabri

New Contributors​

• @karifabri made their first contribution in https://github.com/spiceai/spiceai/pull/2601

What's Changed​

- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2585
- Set helm to v0.17.4-beta by @ewgenius in https://github.com/spiceai/spiceai/pull/2595
- Bump to next v0.18.0-beta version by @ewgenius in https://github.com/spiceai/spiceai/pull/2596
- Add snapshot test docs / Update beta criteria for data accelerators by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2594
- Enable federation for accelerated queries (sqlite, duckdb, postgres) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2598
- spelling updates on v0.17.4 release notes by @karifabri in https://github.com/spiceai/spiceai/pull/2601
- Update endgame template by @ewgenius in https://github.com/spiceai/spiceai/pull/2591
- fix: Re-attach DuckDB attachments on each query by @peasee in https://github.com/spiceai/spiceai/pull/2602
- Speed up sqlite accelerator benchmark test with indexes by @Sevenannn in https://github.com/spiceai/spiceai/pull/2597
- Fix refresh API using `refresh_mode: append` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2609
- Tweak `/ready` to only report ready when components have all reported Ready by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2600
- Add `s3_auth` parameter to configure IAM role authentication by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2611
- Bump fundu from 2.0.0 to 2.0.1 by @dependabot in https://github.com/spiceai/spiceai/pull/2576
- fix: Remove comments from SQL files by @peasee in https://github.com/spiceai/spiceai/pull/2627
- Utilize runtime.status().is_ready() to check acceleration dataset readiness in benchmark test by @Sevenannn in https://github.com/spiceai/spiceai/pull/2614
- Allow for prefix to be kept in internal Parameters by @Jeadie in https://github.com/spiceai/spiceai/pull/2603
- Bump itertools from 0.12.1 to 0.13.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2572
- Bump golang.org/x/mod from 0.20.0 to 0.21.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2571
- Add initial threat model using OWASP Threat Dragon by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2599
- fix: Explicitly error for duplicate duckdb file accelerators by @peasee in https://github.com/spiceai/spiceai/pull/2628
- Benchmark test binary can parse command line option by @Sevenannn in https://github.com/spiceai/spiceai/pull/2626
- Snapshot tests shouldn't crash the Spice benchmark test by @Sevenannn in https://github.com/spiceai/spiceai/pull/2613
- Bump anyhow from 1.0.86 to 1.0.87 by @dependabot in https://github.com/spiceai/spiceai/pull/2573
- Upgrade datafusion to improve SQLite subquery tables aliasing support by @sgrebnov in https://github.com/spiceai/spiceai/pull/2634
- Run benchmark separately using workflow by @Sevenannn in https://github.com/spiceai/spiceai/pull/2631
- Sharepoint UX changes by @Jeadie in https://github.com/spiceai/spiceai/pull/2633
- Improve `/ready` to only mark a dataset ready iff the initial refresh completed by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2630
- Support relative paths for file connector by @Jeadie in https://github.com/spiceai/spiceai/pull/2637
- Fix `error decoding response body` GitHub file connector bug by @sgrebnov in https://github.com/spiceai/spiceai/pull/2645
- GraphQL pagination and robustness. by @Jeadie in https://github.com/spiceai/spiceai/pull/2632
- docs: Update bug template by @peasee in https://github.com/spiceai/spiceai/pull/2629
- Define GitHub `issues` data connector schema upfront by @sgrebnov in https://github.com/spiceai/spiceai/pull/2646
- Add support for loading from Sharepoint Group's default drive. by @Jeadie in https://github.com/spiceai/spiceai/pull/2642
- Fix typo in workflow, fix the postgres connector container readiness check by @Sevenannn in https://github.com/spiceai/spiceai/pull/2654
- Fix check all features by @Sevenannn in https://github.com/spiceai/spiceai/pull/2653
- Enable Warn/Error traces from dependency components by @sgrebnov in https://github.com/spiceai/spiceai/pull/2655
- Use lower case iso8601 for time_column by @Sevenannn in https://github.com/spiceai/spiceai/pull/2551
- Add basic integration test for Spice spill-to-disk and re-hydration scenario by @sgrebnov in https://github.com/spiceai/spiceai/pull/2643
- Add 'RefreshOverrides::max_jitter' to 'POST /v1/datasets/:name/acceleration/refresh' by @Jeadie in https://github.com/spiceai/spiceai/pull/2641
- Bump rustls-pemfile from 1.0.4 to 2.1.3 by @dependabot in https://github.com/spiceai/spiceai/pull/2575
- Update dependencies to support querying postgres enum types by @Sevenannn in https://github.com/spiceai/spiceai/pull/2657
- Upgrade table-providers by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2659
- Improve `spill_to_disk_and_rehydration` integration test by @sgrebnov in https://github.com/spiceai/spiceai/pull/2658
- Enhance GitHub connector robustness with explicit table schema definitions by @sgrebnov in https://github.com/spiceai/spiceai/pull/2661
- Rename sharepoint fields by @Jeadie in https://github.com/spiceai/spiceai/pull/2668
- Disable dataset checkpoint for DuckDB acceleration by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2676
- Revert "Enable federation for accelerated queries (sqlite, duckdb, postgres) (#2598) by @Sevenannn in https://github.com/spiceai/spiceai/pull/2683

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.17.4-beta...v0.18.0-beta

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]

Announcing the release of Spice v0.17.3-beta.

The v0.17.3-beta release further improves data accelerator robustness and adds a new github data connector that makes accelerating GitHub Issues, Pull Requests, Commits, and Blobs easy.

Highlights in v0.17.3-beta​

Improved benchmarking, testing, and robustness of data accelerators: Continued improvements to benchmarking and testing of data accelerators, leading to more robust and reliable data accelerators.

GitHub Connector (alpha): Connect to GitHub and accelerate Issues, Pull Requests, Commits, and Blobs.

datasets:
  # Fetch all rust and golang files from spiceai/spiceai
  - from: github:github.com/spiceai/spiceai/files/trunk
    name: spiceai.files
    params:
      include: '**/*.rs; **/*.go'
      github_token: ${secrets:GITHUB_TOKEN}

    # Fetch all issues from spiceai/spiceai. Similar for pull requests, commits, and more.
  - from: github:github.com/spiceai/spiceai/issues
    name: spiceai.issues
    params:
      github_token: ${secrets:GITHUB_TOKEN}

Breaking Changes​

None.

Upgrade Instructions​

• CLI: Run spice upgrade
• Docker: docker pull spiceai/spiceai:latest
• Container image tag: spiceai/spiceai:latest or spiceai/spiceai:0.17.3-beta

Contributors​

• @phillipleblanc
• @Jeadie
• @peasee
• @sgrebnov
• @Sevenannn
• @lukekim
• @dependabot
• @ewgenius

What's Changed​

Dependencies​

• delta_kernel from 0.2.0 to 0.3.0.

Commits​

- Prepare version for v0.17.3-beta by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2388
- Add a basic Github Connector by @Jeadie in https://github.com/spiceai/spiceai/pull/2365
- task: Re-enable federation by @peasee in https://github.com/spiceai/spiceai/pull/2389
- fix: Implement custom PartialEq for Dataset by @peasee in https://github.com/spiceai/spiceai/pull/2390
- GitHub Data Connector `files` support (basic fields) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2393
- Add a `--force` flag to `spice install` to force it to install the latest released version by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2395
- Improve experience of using `spice chat` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2396
- Fix view loading on startup by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2398
- Add `include` param support to GitHub Data Connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/2397
- Postgres integration test to cover on-conflict behavior by @Sevenannn in https://github.com/spiceai/spiceai/pull/2359
- Create dependabot.yml by @lukekim in https://github.com/spiceai/spiceai/pull/2399
- Add `content` column to GitHub Connector when dataset is accelerated by @sgrebnov in https://github.com/spiceai/spiceai/pull/2400
- Fix dependabot indentation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2402
- Bump docker/setup-buildx-action from 1 to 3 by @dependabot in https://github.com/spiceai/spiceai/pull/2403
- Bump github/codeql-action from 2 to 3 by @dependabot in https://github.com/spiceai/spiceai/pull/2404
- Bump docker/login-action from 1 to 3 by @dependabot in https://github.com/spiceai/spiceai/pull/2405
- Bump yogevbd/enforce-label-action from 2.1.0 to 2.2.2 by @dependabot in https://github.com/spiceai/spiceai/pull/2406
- Bump actions/checkout from 3 to 4 by @dependabot in https://github.com/spiceai/spiceai/pull/2407
- Bump go.uber.org/zap from 1.21.0 to 1.27.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2408
- Bump github.com/prometheus/client_model from 0.6.0 to 0.6.1 by @dependabot in https://github.com/spiceai/spiceai/pull/2409
- Bump github.com/spf13/cobra from 1.6.0 to 1.8.1 by @dependabot in https://github.com/spiceai/spiceai/pull/2412
- Bump chrono-tz from 0.8.6 to 0.9.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2413
- Bump tokio from 1.39.2 to 1.39.3 by @dependabot in https://github.com/spiceai/spiceai/pull/2414
- Bump tokenizers from 0.19.1 to 0.20.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2415
- Bump serde from 1.0.207 to 1.0.209 by @dependabot in https://github.com/spiceai/spiceai/pull/2416
- Bump gopkg.in/natefinch/lumberjack.v2 from 2.0.0 to 2.2.1 by @dependabot in https://github.com/spiceai/spiceai/pull/2410
- Bump ndarray from 0.15.6 to 0.16.1 by @dependabot in https://github.com/spiceai/spiceai/pull/2417
- Bump golang.org/x/mod from 0.14.0 to 0.20.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2411
- Add correct labels to dependabot.yml by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2418
- Fix build break by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2430
- Dependabot updates by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2431
- Bump github.com/stretchr/testify from 1.8.1 to 1.9.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2422
- Preserve timezone information in constructing expr by @Sevenannn in https://github.com/spiceai/spiceai/pull/2392
- Bump github.com/spf13/viper from 1.12.0 to 1.19.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2420
- Fix repeated base table data in acceleration with embeddings by @Sevenannn in https://github.com/spiceai/spiceai/pull/2401
- Fix tool calling with Groq (and potentially other tool-enabled models) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2435
- Remove candle from `crates/llms/src/chat/` by @Jeadie in https://github.com/spiceai/spiceai/pull/2439
- fix: Only attach successfully initialized accelerators by @peasee in https://github.com/spiceai/spiceai/pull/2433
- Support overriding OpenAI default values in a model param; add token usage telemetry to task_history. by @Jeadie in https://github.com/spiceai/spiceai/pull/2434
- Enable message chains and tool calls for local LLMs by @Jeadie in https://github.com/spiceai/spiceai/pull/2180
- DuckDB on-conflict integration test by @Sevenannn in https://github.com/spiceai/spiceai/pull/2437
- Fix MySQL E2E tests and include MySQL acceleration testing by @sgrebnov in https://github.com/spiceai/spiceai/pull/2441
- Use rtcontext for proper cloud/local context in `spice chat` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2442
- Fix MySQL connector to respect the source column's decimal precision by @sgrebnov in https://github.com/spiceai/spiceai/pull/2443
- Improve Github Data Connector tables schema by @sgrebnov in https://github.com/spiceai/spiceai/pull/2448
- Improve GitHub Connector error msg when invalid token or permissions by @sgrebnov in https://github.com/spiceai/spiceai/pull/2449
- Proper error tracking across tracing spans by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2454
- task: Disable and update federation by @peasee in https://github.com/spiceai/spiceai/pull/2457
- GitHub connector: convert `labels` and `hashes` to primitive arrays by @sgrebnov in https://github.com/spiceai/spiceai/pull/2452
- Bump `datafusion` version to the latest by @sgrebnov in https://github.com/spiceai/spiceai/pull/2456
- Trim trailing `/` for S3 data connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2458
- Add `accelerated_refresh` to `task_history` table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2459
- Add `assignees` and `labels` fields to github issues and github pulls datasets by @ewgenius in https://github.com/spiceai/spiceai/pull/2467
- Native clickhouse schema inference by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2466
- List GitHub connector in readme by @ewgenius in https://github.com/spiceai/spiceai/pull/2468
- Fix LLMs health check; Add `updatedAt` field to GitHub connector by @ewgenius in https://github.com/spiceai/spiceai/pull/2474
- Remove non existing updated_at from github.pulls dataset by @ewgenius in https://github.com/spiceai/spiceai/pull/2475
- GitHub connector: add pulls labels and rm duplicate milestoneId and milestoneTitle for issues by @sgrebnov in https://github.com/spiceai/spiceai/pull/2477
- Bump delta_kernel from 0.2.0 to 0.3.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2472
- Add back GitHub connector Pull Request `updated_at` by @lukekim in https://github.com/spiceai/spiceai/pull/2479
- Update ROADMAP Sep 2, 2024. by @lukekim in https://github.com/spiceai/spiceai/pull/2478

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v0.17.2-beta...v0.17.3-beta>

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]