One post tagged with "localpod"

Announcing the release of Spice v0.19.4-beta ☘️

Spice v0.19.4-beta introduces a new localpod Data Connector, improvements to accelerator resiliency and control, and a new configuration to control when accelerated datasets are considered ready.

Highlights in v0.19.4​

localpod Connector: Implement a "tiered" acceleration strategy with a new localpod Data Connector that can be used to accelerate datasets from other datasets registered in Spice.

datasets:
  - from: s3://my_bucket/my_dataset
    name: my_dataset
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_check_interval: 60s
  - from: localpod:my_dataset
    name: my_localpod_dataset
    acceleration:
      enabled: true

Refreshes on the localpod's parent dataset will automatically be synchronized with the localpod dataset.

Improved Accelerator Resiliency: When Spice is restarted, if the federated source for a dataset configured with a file-based accelerator is not available, the dataset will still load from the existing file data and will attempt to connect to the federated source in the background for future refreshes.

Accelerator Ready State: Control when an accelerated dataset is considered "ready" by the runtime with the new ready_state parameter.

datasets:
  - from: s3://my_bucket/my_dataset
    name: my_dataset
    acceleration:
      enabled: true
      ready_state: on_load # or on_registration

• ready_state: on_load: Default. The dataset is considered ready after the initial load of the accelerated data. For file-based accelerated datasets that have existing data, this means the dataset is ready immediately.
• ready_state: on_registration: The dataset is considered ready when the dataset is registered in Spice. Queries against this dataset before the data is loaded will fallback to the federated source.