HuggingFace Text Embedding Models

To use an embedding model from HuggingFace with Spice, specify the huggingface path in the from field of your configuration. The model and its related files will be automatically downloaded, loaded, and served locally by Spice.

The following parameters are specific to HuggingFace models:

Parameter	Description	Default
hf_token	The Huggingface access token.	-
pooling	The pooling method for embedding models. Supported values are cls, mean, splade, last_token	-

Here is an example configuration in spicepod.yaml:

embeddings:
  - from: huggingface:huggingface.co/sentence-transformers/all-MiniLM-L6-v2
    name: all_minilm_l6_v2

Supported models include:

• All models tagged as text-embeddings-inference on Huggingface
• Any Huggingface repository with the correct files to be loaded as a local embedding model.

Model Weights

Spice downloads the repository's safetensors weights. When a repository publishes none, it falls back to pytorch_model.bin and logs a warning at startup:

safetensors weights not found; falling back to `pytorch_model.bin`. Model loading is significantly slower.

A repository that publishes neither format fails to load. Prefer a repository with safetensors weights where one is available.

With the same semantics as language models, spice can run private HuggingFace embedding models:

embeddings:
  - from: huggingface:huggingface.co/secret-company/awesome-embedding-model
    name: top_secret
    params:
      hf_token: ${ secrets:HF_TOKEN }