TFIDFConverter

Converter

DashAI.back.converters.scikit_learn.TFIDFConverter

---

Convert raw text documents into a matrix of TF-IDF weighted features.

TF-IDF (Term Frequency - Inverse Document Frequency) re-weights raw token counts so that terms that appear frequently in a specific document but rarely across the whole corpus receive a higher score, while common stop-like terms are down-weighted. Each document is represented as a floating-point vector of TF-IDF scores, one dimension per vocabulary term.

The TF-IDF score for term t in document d is:

tfidf(t, d) = tf(t, d) x log((1 + n) / (1 + df(t))) + 1

where n is the total number of documents and df(t) is the number of documents containing t (scikit-learn's smooth_idf=True default).

Optional preprocessing steps include lower-casing, stop-word removal, and n-gram extraction. The result is a floating-point DashAI dataset with one column per vocabulary term.

Internally wraps sklearn.feature_extraction.text.TfidfVectorizer.

The TF-IDF weighting scheme was introduced as a core technique in information retrieval by Salton & McGill (1983) [2].

References

• [1] https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
• [2] Salton, G. & McGill, M. J. (1983). Introduction to Modern Information Retrieval. McGraw-Hill.

Parameters

max_features, default=1000

Maximum number of terms to keep, the most frequent first. Leave empty to keep every term.

lowercase : boolean, default=True

Whether to convert all characters to lowercase before tokenizing.

strip_accents, default=None

Remove accents during preprocessing.

analyzer : string, default=word

Whether the feature should be made of word or character n-grams.

stop_words, default=None

Stop word set to remove. Use 'english' or None.

custom_stop_words, default=None

Extra words to ignore, added to the stop word set above.

lower_bound_ngrams : integer, default=1

Lower bound for n-grams to be extracted. Must be <= upper bound.

upper_bound_ngrams : integer, default=1

Upper bound for n-grams to be extracted. Must be >= lower bound.

min_df : number, default=0.0

Ignore terms with document frequency below this threshold.

max_df : number, default=1.0

Ignore terms with document frequency above this threshold.

norm, default=l2

Norm used to normalize term vectors.

use_idf : boolean, default=True

Enable inverse-document-frequency reweighting.

smooth_idf : boolean, default=True

Smooth IDF weights to prevent zero divisions.

sublinear_tf : boolean, default=False

Apply sublinear TF scaling (1 + log(tf)).

Methods

build_vectorizer(self, params: Dict[str, Any]) -> Any

Defined on TFIDFConverter

Return an unfitted TfidfVectorizer.

Parameters

params : dict

Converter parameters.

Returns

TfidfVectorizer

The configured vectorizer.

get_output_type(self, column_name: Optional[str] = None) -> DashAI.back.types.dashai_data_type.DashAIDataType

Defined on TFIDFConverter

Return the DashAI data type produced by this converter for a column.

Parameters

column_name : str, optional

The column name to look up in the fitted vectoriser. When provided and the vectoriser has been fitted, the returned type reflects the actual fitted vocabulary. Defaults to None.

Returns

DashAIDataType

A Float type for each TF-IDF weight column.

build_report(self, data: 'Dict[str, Any]') -> 'Dict[str, Any]'

Defined on BaseConverter

Wrap converter-specific data with the producer converter name.

fit(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'SparseTextVectorizer'

Defined on SparseTextVectorizer

Learn the vocabulary of the first column.

Parameters

x : DashAIDataset

Input dataset; only the first column is used.

y : DashAIDataset, optional

Ignored.

Returns

SparseTextVectorizer

This converter.

fit_matrix(self, texts: List[str]) -> Any

Defined on SparseTextVectorizer

Learn the vocabulary and return the matrix of the texts.

Parameters

texts : list of str

The corpus.

Returns

scipy.sparse.csr_matrix

One row per text, one column per term.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str

Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential

An instance of the requested credential component.

get_metadata(cls) -> 'Dict[str, Any]'

Defined on BaseConverter

Get metadata for the converter, used by the DashAI frontend.

Parameters

cls : type

The converter class (injected automatically by Python for classmethods).

Returns

Dict[str, Any]

Dictionary containing display name, short description, image preview path, category, icon, color, and whether the converter is supervised.

get_report(self) -> 'Union[Dict[str, Any], None]'

Defined on BaseConverter

Return the converter report produced after execution, if any.

Returns

Dict[str, Any] or None

JSON-serializable report dict, or None when the converter does not produce one.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict

Dictionary representing the Json Schema of the component.

infer_output_columns(self, inputs: 'List[StateItem]') -> 'StructureDelta'

Defined on BaseConverter

Estimate what this converter does to its scope, without any data.

Parameters

inputs : list of ColumnItem | BlockItem

The dataset state items in this converter's scope.

Returns

StructureDelta

The scope items that survive (maybe retyped) and the new items.

rows_to_keep(self, x: "'DashAIDataset'") -> 'List[int]'

Defined on BaseConverter

Positions of the rows of x that survive this converter.

Parameters

x : DashAIDataset

The converter's scope columns of one split.

Returns

list of int

0-based positions of the kept rows, in order.

transform(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DashAIDataset'

Defined on SparseTextVectorizer

Append one column per vocabulary term to the dataset.

Parameters

x : DashAIDataset

Input dataset; the first column is vectorized and kept.

y : DashAIDataset, optional

Ignored.

Returns

DashAIDataset

The input columns plus {COLUMN_PREFIX}{term} columns.

transform_matrix(self, texts: List[str]) -> Any

Defined on SparseTextVectorizer

Return the matrix of texts over the learned vocabulary.

Parameters

texts : list of str

Texts to vectorize.

Returns

scipy.sparse.csr_matrix

One row per text, one column per term.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict

A dictionary with the data provided by the user to initialize the model.

Returns

dict

A validated dictionary with the necessary objects.