MultilingualE5LargeEmbedding
Multilingual E5, large.
Parameters
- device : string, default=
CPU - Device the model runs on. A GPU is much faster than the CPU.
- batch_size : integer, default=
32 - Number of texts encoded together. Lower it if memory runs out.
- normalize : boolean, default=
True - Scale every vector to length one, as cosine similarity expects.
- overflow_strategy : string, default=
truncate - What to do with a text longer than the model accepts: cut it (truncate) or encode it in pieces and average them (aggregate).
Methods
build_report(self, data: 'Dict[str, Any]') -> 'Dict[str, Any]'
BaseConverterWrap converter-specific data with the producer converter name.
component_dir(cls) -> pathlib.Path
DownloadableMixinReturn this component's own storage directory.
Returns
- pathlib.Path
<COMPONENT_PATH>/<ClassName>.
delete(cls) -> None
DownloadableMixinRemove the component's downloaded artifacts.
download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None
HFDownloadableMixinDownload all repos listed in hf_repos() into component_dir().
Parameters
- report : ProgressReporter, optional
- Callback invoked before each repo download with
report(None, "Downloading <repo_id>").Nonemeans no progress reporting.
encode(self, texts: List[str], role: str = 'passage') -> numpy.ndarray
TextEmbeddingConverterEncode texts into dense vectors.
Parameters
- texts : list of str
- Texts to encode; None becomes an empty string.
- role : str, optional
"passage"for indexed text,"query"for a search query.
Returns
- np.ndarray
(len(texts), dim)float32 array.
fit(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'TextEmbeddingConverter'
TextEmbeddingConverterCheck that every column holds text.
Parameters
- x : DashAIDataset
- The columns to embed.
- y : DashAIDataset, optional
- Ignored.
Returns
- TextEmbeddingConverter
- This converter.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_metadata(cls) -> 'Dict[str, Any]'
BaseConverterGet metadata for the converter, used by the DashAI frontend.
Parameters
- cls : type
- The converter class (injected automatically by Python for classmethods).
Returns
- Dict[str, Any]
- Dictionary containing display name, short description, image preview path, category, icon, color, and whether the converter is supervised.
get_output_type(self, column_name: Optional[str] = None) -> DashAI.back.types.dashai_data_type.DashAIDataType
TextEmbeddingConverterReturn the type of every embedding column.
Parameters
- column_name : str, optional
- Unused.
Returns
- DashAIDataType
Floatbacked bypyarrow.float32().
get_report(self) -> 'Union[Dict[str, Any], None]'
BaseConverterReturn the converter report produced after execution, if any.
Returns
- Dict[str, Any] or None
- JSON-serializable report dict, or
Nonewhen the converter does not produce one.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
hf_repos(cls)
HFPretrainedDownloadMixinDerive the single repo entry from MODEL_NAME.
Returns
- list of tuple of (str, str)
[(MODEL_NAME, "model")]or an empty list when unset.
infer_output_columns(self, inputs: 'List[StateItem]') -> 'StructureDelta'
BaseConverterEstimate what this converter does to its scope, without any data.
Parameters
- inputs : list of ColumnItem | BlockItem
- The dataset state items in this converter's scope.
Returns
- StructureDelta
- The scope items that survive (maybe retyped) and the new items.
is_downloaded(cls) -> bool
HFDownloadableMixinReturn whether all repo directories exist and are non-empty.
Returns
- bool
Truewhen every repo listed inhf_repos()has a non-empty local directory;Falseotherwise (including when the list is empty).
load_pretrained(self, source: str) -> Any
TextEmbeddingConverterLoad the Sentence Transformers model.
Parameters
- source : str
- A local directory or a Hub repo id.
Returns
- SentenceTransformer
- The model on the chosen device.
pretrained(self) -> Any
PretrainedToolReturn the loaded model, loading it on first use.
Returns
- Any
- The value :meth:
load_pretrainedreturned.
prompt(self, role: str) -> Optional[str]
TextEmbeddingConverterReturn the text prepended to inputs of a role.
Parameters
- role : str
"passage"or"query".
Returns
- str or None
- The prompt, or None when the model takes the text as is.
rows_to_keep(self, x: "'DashAIDataset'") -> 'List[int]'
BaseConverterPositions of the rows of x that survive this converter.
Parameters
- x : DashAIDataset
- The converter's scope columns of one split.
Returns
- list of int
- 0-based positions of the kept rows, in order.
setup_device(self, label: Optional[str]) -> None
PretrainedToolChoose the device and forget any loaded model.
Parameters
- label : str, optional
- The
deviceparameter.
transform(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DashAIDataset'
TextEmbeddingConverterReplace every text column with its embedding columns.
Parameters
- x : DashAIDataset
- The text columns.
- y : DashAIDataset, optional
- Ignored.
Returns
- DashAIDataset
- One float32 column
emb_{column}_{i}per dimension of every input column, and no input column.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.