Announcing the release of Spice v1.0-rc.2 🔗

Spice v1.0.0-rc.2 is the second release candidate for the first major version of Spice.ai OSS. This release continues to build on the stability of Spice for production use, including key Data Connector graduations, bug fixes, and AI features.

Highlights in v1.0-rc.2​

• MS SQL and File Data Connectors: Graduated from Alpha to Beta.

• GraphQL and Databricks Delta Lake Data Connectors: Graduated from Beta to Release Candidate.

• gospice SDK Release: The Spice Go SDK has updated to v7.0, adding support for refreshing datasets and upgrading dependencies.

• Azure AI Support: Added support for both LLMs and embedding models. Example spicepod.yml configuration:

embeddings:
  - name: azure
    from: azure:text-embedding-3-small
    params:
      endpoint: https://your-resource-name.openai.azure.com
      azure_api_version: 2024-08-01-preview
      azure_deployment_name: text-embedding-3-small
      azure_api_key: ${ secrets:SPICE_AZURE_API_KEY }
models:
  - name: azure
    from: azure:gpt-4o-mini
    params:
      endpoint: https://your-resource-name.openai.azure.com
      azure_api_version: 2024-08-01-preview
      azure_deployment_name: gpt-4o-mini
      azure_api_key: ${ secrets:SPICE_AZURE_TOKEN }

Accelerate subsets of columns: Spice now supports acceleration for specific columns from a federated source. Specify the desired columns directly in the Refresh SQL for more selective and efficient data acceleration.

Example spicepod.yaml configuration:

datasets:
  - from: s3://spiceai-demo-datasets/taxi_trips/2024/
    name: taxi_trips
    params:
      file_format: parquet
    acceleration:
      refresh_sql: SELECT tpep_pickup_datetime, tpep_dropoff_datetime, trip_distance, total_amount FROM taxi_trips

Announcing the release of Spice v1.0-rc.1 🚀

Spice v1.0.0-rc.1 marks the release candidate for the first major version of Spice.ai OSS. This milestone includes key Connector and Accelerator graduations and bug fixes, positioning Spice for a stable and production-ready release.

Highlights in v1.0-rc.1​

API Key Authentication: Spice now supports optional authentication for API endpoints via configurable API keys, for additional security and control over runtime access.

Example Spicepod.yml configuration:

runtime:
  auth:
    api-key:
      enabled: true
      keys:
        - ${ secrets:api_key } # Load from a secret store
        - my-api-key # Or specify directly

Usage:

• HTTP API: Include the API key in the X-API-Key header.
• Flight SQL: Use the API key in the Authorization header as a Bearer token.
• Spice CLI: Provide the --api-key flag for CLI commands.

For more details on using API Key auth, refer to the API Auth documentation.

DuckDB Data Connector: Has graduated from Beta to Release Candidate.

Arrow and DuckDB Data Accelerators: Both have graduated from Beta to Release Candidates.

Debezium Kafka Integration: Spice now supports secure authentication and encryption options for Kafka connections when using Debezium for Change Data Capture (CDC). The previous limitation of PLAINTEXT protocol-only connections has been lifted. Spice now supports the following Kafka security configurations:

• Security protocol: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
• SASL mechanisms: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512

Example Spicepod.yml configuration:

datasets:
  - from: debezium:my_kafka_topic_with_debezium_changes
    name: my_dataset
    params:
      kafka_security_protocol: SASL_SSL
      kafka_sasl_mechanism: SCRAM-SHA-512
      kafka_sasl_username: kafka
      kafka_sasl_password: ${secrets:kafka_sasl_password}
      kafka_ssl_ca_location: ./certs/kafka_ca_cert.pem

Announcing the release of Spice v0.20-beta 🧩

Spice v0.20.0-beta improves federated query performance with column pruning and adds support for Metal (Apple Silicon) and CUDA (NVidia) accelerators. The S3, PostgreSQL, MySQL, and GitHub Data Connectors have graduated from Beta to Release Candidates. The Arrow, DuckDB, and SQLite Data Accelerators have graduated from Alpha to Beta.

Highlights in v0.20.0-beta​

Data Connectors: The S3, PostgreSQL, MySQL, and GitHub Data Connectors have graduated from beta to release candidate.

Data Accelerators: The Arrow, DuckDB, and SQLite Data Accelerators have graduated from alpha to beta.

Metal and CUDA Support: Added support for Metal (Apple Silicon) and CUDA (NVidia) for AI/ML workloads including embeddings and local LLM inference.

For instructions on compiling a Meta or CUDA binary, see the Installation Docs.

Announcing the release of Spice v0.19.4-beta ☘️

Spice v0.19.4-beta introduces a new localpod Data Connector, improvements to accelerator resiliency and control, and a new configuration to control when accelerated datasets are considered ready.

Highlights in v0.19.4​

localpod Connector: Implement a "tiered" acceleration strategy with a new localpod Data Connector that can be used to accelerate datasets from other datasets registered in Spice.

datasets:
  - from: s3://my_bucket/my_dataset
    name: my_dataset
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_check_interval: 60s
  - from: localpod:my_dataset
    name: my_localpod_dataset
    acceleration:
      enabled: true

Refreshes on the localpod's parent dataset will automatically be synchronized with the localpod dataset.

Improved Accelerator Resiliency: When Spice is restarted, if the federated source for a dataset configured with a file-based accelerator is not available, the dataset will still load from the existing file data and will attempt to connect to the federated source in the background for future refreshes.

Accelerator Ready State: Control when an accelerated dataset is considered "ready" by the runtime with the new ready_state parameter.

datasets:
  - from: s3://my_bucket/my_dataset
    name: my_dataset
    acceleration:
      enabled: true
      ready_state: on_load # or on_registration

• ready_state: on_load: Default. The dataset is considered ready after the initial load of the accelerated data. For file-based accelerated datasets that have existing data, this means the dataset is ready immediately.
• ready_state: on_registration: The dataset is considered ready when the dataset is registered in Spice. Queries against this dataset before the data is loaded will fallback to the federated source.

Announcing the release of Spice v0.19.3-beta 📈

Spice v0.19.3-beta improves the performance and stability of data connectors and accelerators, including faster queries across multiple federated sources by optimizing how filters are applied. Anthropic has also been added as a LLM model provider.

Highlights in v0.19.3​

DataFusion Fixes: Resolved bugs in DataFusion and DataFusion Table Providers, expanding TPC-DS coverage and correctness.

GitHub Data Connector Beta Milestone: The GitHub Data Connector has graduated to Beta after extensive testing, stability, and performance improvements.

Anthropic Models Provider: Anthropic has been added as an LLM provider, including support for streaming.

Example spicepod.yml:

models:
  - from: anthropic:claude-3-5-sonnet-20240620
    name: claude_3_5_sonnet
    params:
      anthropic_api_key: ${ secrets:SPICE_ANTHROPIC_API_KEY }