Skip to main content
Version: Next

Changes Refresh Mode

The changes refresh mode applies incremental inserts, updates, and deletes from a Change Data Capture (CDC) source. Unlike append, changes mode reflects modifications and deletions in the acceleration, keeping it consistent with sources where rows mutate over time.

Use changes when:

  • The source supports CDC (e.g., a database with a transaction log).
  • Rows in the source are updated or deleted, not just inserted.
  • The acceleration must reflect the current state of the source row-for-row.

Configuration​

refresh_mode: changes requires a CDC-capable data connector. Spice supports CDC via PostgreSQL Logical Replication, MySQL Binlog Replication, MongoDB Change Streams, DynamoDB Streams, and Debezium (over Kafka). An accelerated S3 prefix can also use refresh_mode: changes with S3 Event Notifications delivered to SQS, which appends new objects rather than applying row-level changes. See Supported Data Connectors for details.

note

Apache Kafka is a real-time streaming source but is append-only — it uses refresh_mode: append, not changes.

Iceberg delete files are not a CDC stream. refresh_mode: changes does not apply Iceberg v2 position or equality delete files to an acceleration — those files are applied on federated scans and on refresh_mode: full only. See Delete files on federated reads.

Any accelerator engine that supports writes can be a changes sink — arrow, duckdb, sqlite, and cayenne. Spice Cayenne is recommended for large-scale CDC (incremental materialized views, in-memory CDC tier, and replication-lag/freshness SLOs).

datasets:
  - from: debezium:cdc.public.customer_orders
    name: customer_orders
    acceleration:
      enabled: true
      refresh_mode: changes
      engine: duckdb
      mode: file

The Debezium connector streams change events from a Kafka topic produced by Debezium. Each event is applied to the acceleration in order, preserving inserts, updates, and deletes from the source.

Behavior​

  • The acceleration is bootstrapped from the source snapshot, then continuously updated from the change stream.
  • refresh_check_interval, refresh_cron, on-demand refresh, refresh_data_window, and retention_period do not apply — updates are driven by the change stream rather than periodic polling.
  • refresh_sql can only modify selected columns in changes mode and cannot apply row filters.

Native CDC​

Use refresh_mode: changes when an interval full or append refresh is not fresh enough and the source can emit row-level changes. For PostgreSQL and MySQL, the native connectors are the path to use: PostgreSQL logical replication and MySQL binlog replication. Debezium remains the path for databases without a native Spice CDC connector, and for deployments that already run that pipeline. The debezium connector consumes a Kafka topic (debezium_transport: kafka is the only supported transport); Debezium push ingest accepts change events over HTTP without Kafka.

Point the dataset at a replica when that replica still exposes the log the connector reads — Postgres logical decoding (wal_level=logical), or the MySQL binary log — so the log read stays off the primary.

Spice Cayenne is the accelerator for high-throughput CDC. arrow, duckdb, and sqlite can apply a change stream; they are a better fit when the table is small. An append-only acceleration still needs an application tombstone (a deleted or deleted_at column, filtered in a view) when the source does not emit deletes. changes applies source deletes into the accelerator.

Schema changes on the source are not applied silently. See on_schema_change, MySQL schema changes, and PostgreSQL CDC limitations.