9 posts tagged with "duckdb"

🎉 After 47 releases, Spice.ai OSS has reached production readiness with the 1.0-stable milestone!

The core runtime and features such as query federation, query acceleration, catalog integration, search and AI-inference have all graduated to stable status along with key component graduations across data connectors, data accelerators, catalog connectors, and AI model providers.

Highlights in v1.0-stable​

• Stable Data Connectors: The following data connectors have graduated to Stable:

  • Delta Lake
  • MySQL
  • Dremio
  • PostgreSQL
  • Databricks (mode: delta_lake)
  • DuckDB
  • S3

• Stable Data Accelerators: The following data accelerators have graduated to Stable:

  • DuckDB
  • Arrow

• Unity Catalog Connector: Graduated to Stable.

• Databricks (mode: spark_connect) Data Connector: Graduated to Beta.

• Beta Catalog Connectors: The Iceberg and Databricks catalog connectors graduated to Beta.

• OpenAI Model & Embeddings Provider: Graduated to Release Candidate (RC).

• Alpha Model Providers: The Anthropic and xAI (Grok) model providers graduated to Alpha.

Breaking Changes​

• Default Runtime Version: The CLI will install the GPU accelerated AI-capable Runtime by default (if supported), when running spice install or spice run. To force-install the non-GPU version, run spice install ai --cpu.

• Default OpenAI Model: The default OpenAI model has updated to gpt-4o-mini.

• Identifier Normalization: Unquoted identifiers such as table names are no longer normalized to lowercase. Identifiers will now retain their exact case as provided.

• Sandboxed Docker Image: The Runtime Docker Image now runs the spiced process as the nobody user in a minimal chroot sandbox.

• Insecure S3 and ABFS endpoints: The S3 and ABFS connectors now enforce insecure endpoint checks, preventing HTTP endpoints unless allow_http is explicitly enabled. Refer to the documentation for details.

Announcing the release of Spice v1.0-rc.3 🧊

Spice v1.0.0-rc.3 is the third release candidate for the first major version of Spice.ai OSS. This release continues the focus on production readiness and includes new Iceberg Catalog APIs, DuckDB improvements, and a new Iceberg Catalog Connector.

Highlights in v1.0-rc.3​

• Iceberg Catalog APIs: Spice now functions as an Iceberg Catalog provider, implementing a core subset of the Iceberg Catalog APIs. This enables Iceberg Catalog clients native discovery of datasets and schemas through Spice APIs.

• GET /v1/namespaces - List all catalogs registered in Spice.

• GET /v1/namespaces?parent=catalog - List schemas registered under a given catalog.

• GET /v1/namespaces/:catalog_schema/tables - List tables registered under a given schema.

• GET /v1/namespaces/:catalog_schema/tables/:table - Get the schema of a given table.

• Iceberg Catalog Connector: The Iceberg Catalog Connector is a new integration to discover and query datasets from a remote Iceberg Catalog.

Example connecting to a remote Iceberg Catalog with tables stored in S3:

catalogs:
  - from: iceberg:https://my-iceberg-catalog.com/v1/namespaces
    name: ice
    params:
      iceberg_s3_access_key_id: ${secrets:ICEBERG_S3_ACCESS_KEY_ID}
      iceberg_s3_secret_access_key: ${secrets:ICEBERG_S3_SECRET_ACCESS_KEY}
      iceberg_s3_region: us-east-1

View the Iceberg Catalog Connector documentation for more details.

• DuckDB Improvements: Added cosine_distance support for DuckDB-backed vector search, improved unnest nested type handling for array_element and lists, and optimized query performance.

• SQLite Data Accelerator: Graduated to Release Candidate (RC).

• File Data Accelerator: Graduated to Release Candidate (RC).

Announcing the release of Spice v1.0-rc.1 🚀

Spice v1.0.0-rc.1 marks the release candidate for the first major version of Spice.ai OSS. This milestone includes key Connector and Accelerator graduations and bug fixes, positioning Spice for a stable and production-ready release.

Highlights in v1.0-rc.1​

API Key Authentication: Spice now supports optional authentication for API endpoints via configurable API keys, for additional security and control over runtime access.

Example Spicepod.yml configuration:

runtime:
  auth:
    api-key:
      enabled: true
      keys:
        - ${ secrets:api_key } # Load from a secret store
        - my-api-key # Or specify directly

Usage:

• HTTP API: Include the API key in the X-API-Key header.
• Flight SQL: Use the API key in the Authorization header as a Bearer token.
• Spice CLI: Provide the --api-key flag for CLI commands.

For more details on using API Key auth, refer to the API Auth documentation.

DuckDB Data Connector: Has graduated from Beta to Release Candidate.

Arrow and DuckDB Data Accelerators: Both have graduated from Beta to Release Candidates.

Debezium Kafka Integration: Spice now supports secure authentication and encryption options for Kafka connections when using Debezium for Change Data Capture (CDC). The previous limitation of PLAINTEXT protocol-only connections has been lifted. Spice now supports the following Kafka security configurations:

• Security protocol: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
• SASL mechanisms: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512

Example Spicepod.yml configuration:

datasets:
  - from: debezium:my_kafka_topic_with_debezium_changes
    name: my_dataset
    params:
      kafka_security_protocol: SASL_SSL
      kafka_sasl_mechanism: SCRAM-SHA-512
      kafka_sasl_username: kafka
      kafka_sasl_password: ${secrets:kafka_sasl_password}
      kafka_ssl_ca_location: ./certs/kafka_ca_cert.pem

Announcing the release of Spice v0.20-beta 🧩

Spice v0.20.0-beta improves federated query performance with column pruning and adds support for Metal (Apple Silicon) and CUDA (NVidia) accelerators. The S3, PostgreSQL, MySQL, and GitHub Data Connectors have graduated from Beta to Release Candidates. The Arrow, DuckDB, and SQLite Data Accelerators have graduated from Alpha to Beta.

Highlights in v0.20.0-beta​

Data Connectors: The S3, PostgreSQL, MySQL, and GitHub Data Connectors have graduated from beta to release candidate.

Data Accelerators: The Arrow, DuckDB, and SQLite Data Accelerators have graduated from alpha to beta.

Metal and CUDA Support: Added support for Metal (Apple Silicon) and CUDA (NVidia) for AI/ML workloads including embeddings and local LLM inference.

For instructions on compiling a Meta or CUDA binary, see the Installation Docs.

Announcing the release of Spice v0.17.4-beta.

The v0.17.4-beta release adds compatibility, performance, and reliability improvements to the DuckDB and SQLite accelerators. The GitHub data connector adds a Stargazers table, Snowflake and Clickhouse data connectors have improved resiliency for empty tables, and core data processing and quality has been improved.

Highlights in v0.17.4-beta​

Improved benchmarking, testing, and robustness of data accelerators: Continued compatibility, performance, and reliability improvements for SQLite and DuckDB data accelerators and expanded performance and quality testing.

GitHub Stargazers: The GitHub Data Connector adds support for a /stargazers table making it easy to query GitHub Stargazers using SQL!