Download the PHP package serafim/tf-idf without Composer

Introduction

TF-IDF is a method of information retrieval that is used to rank the importance of words in a document. It is based on the idea that words that appear in a document more often are more relevant to the document.

TF-IDF is the product of Term Frequency and Inverse Document Frequency. Here’s the formula for TF-IDF calculation.

Term Frequency

the ratio of the number of occurrences of a certain word to the total number of words in the document. Thus, the importance of the word $t_{{i}}$ within a single document is evaluated

$\mathrm{tf}(t, d) = \frac{n_t}{\sum _kn_k}$

where $n_t$ is the number of occurrences of the word $t$ in the document, and the denominator is the total number of words in the document.

Inverse Document Frequency

The inverse of the frequency with which a certain word occurs in the documents of the collection. The founder of this concept is Karen Spark Jones. Accounting for IDF reduces the weight of commonly used words. There is only one IDF value for each unique word within a given collection of documents.

$\mathrm{idf}(t, D) = \log \frac {|D|}{| {\,d{i}\in D\mid t\in d{i}\,} |}$

where

• $|D|$ — The number of documents in the collection;
• ${\displaystyle |{d{i}\in D\mid t\in d{i}}|}$ — the number of documents in collection $D$ where $t$ occurs (when ${\displaystyle n_{t}\neq 0}$).

The choice of the base of the logarithm in the formula does not matter, since changing the base changes the weight of each word by a constant factor, which does not affect the weight ratio.

Thus, the TF-IDF measure is the product of two factors:

$\mathrm{tf-idf}(t, d, D) = \mathrm{tf}(t,d)\times \mathrm{idf}(t,D)$

High weight in TF-IDF will be given to words with high frequency within a particular document and low frequency in other documents.

Installation

TF-IDF is available as composer repository and can be installed using the following command in a root of your project:

Quick Start

Getting information about words:

Example Result:

Adding Documents

The IDF (Inverse Document Frequency) calculation requires several documents in the corpus. To do this, you can use several methods:

Creating Documents

Computing

To calculate TF-IDF between loaded documents, use the compute(): iterable method:

To calculate the TF-IDF between the loaded documents and the passed one, use the computeFor(StreamingDocumentInterface|TextDocumentInterface): iterable method:

Custom Memory Driver

By default, all operations are calculated in memory. This happens pretty quickly, but it can overflow it. You can write your own driver if you need to save memory.

Custom Stop Words

In the case that it is required that some set of "stop words", which would not be taken into account in the result, a custom implementation should be specified.

Please note that by default, the list of stop words from the voku/stop-words package is used.

Custom Locale

Custom Tokenizer

If for some reason the analysis of words in the text does not suit you, you can write your own tokenizer.