Download the PHP package crscheid/php-article-extractor without Composer

PHP Article extractor

This is a web article parsing and language detection library for PHP. This library reads the article content from a web page, removing all HTML and providing just the raw text, suitable for text to speech or machine learning processes.

For a project I have developed, I found many existing open source solutions good starting points, but each had unique failures. This library aggregates three different approaches into a single solution while adding the additional functionality of language detection.

How To Use

This library is distributed via packagist.org, so you can use composer to retrieve the dependency

Calling via URL

This library will attempt to retrieve the HTML for you. You need simply to create an ArticleExtractor class and call the parseURL function on it, passing in the URL desired.

The function processURL returns an array containing the title, text, and meta data associated with the request. If the text is null then this indicates a failed parsing. Below should be the output of the above code.

The field result_url will be different if the library followed redirects. This field represents the final page actually retrieved after redirects.

Calling with HTML

If you already have HTML, you can use the parseHTML function and use your HTML processed through the same logic.

The function parseHTML returns an array containing the title, text, and meta data associated with the request. If the text is null then this indicates a failed parsing. Below should be the output of the above code.

The field result_url will not be included in this case since we are not attempting to get the HTML during the process call.

You can also create the ArticleExtractor class by passing in a key for the language detection service as well as a custom User-Agent string. See more information below.

Options

Language Detection Methods

Language detection is handled by either looking for language specifiers within the HTML meta data or by utilizing the Detect Language service.

If it is possible to detect the language of the article, the language code in ISO 639-1 format as well as the detection method are returned in the fields language and language_method respectively. The language_method field, if found successfully, may be either html or service.

If language detection fails or is not available, both of these fields will be returned as null.

Detect Language requires the use of an API KEY which you can sign up for. However, you can also use this library without it. If the HTML meta data do not contain information about the language of the article, then language and language_method will be returned as null values.

To utilize this library utilizing the language detection service, create the ArticleExtractor object by passing in your API KEY for Detect Language.

Setting User Agent

It is possible to set the user-agent for outgoing requests. To do so pass the desired user agent string to the constructor as follows:

Force Reading Method

It is possible to force the method by which the reading is attempted, either with Readability, Goose, or Goose with our custom processing. This can come in handy where Readability or Goose have particular issues with particular websites.

To force the method, simply provide a third argument to the constructor as such. The four valid methods are readability, goose, goosecustom, or custom.

Output Format

As of version 1.0, the output format has been altered to provide newline breaks for headings. This is important especially for natural language processing applications in determining sentence boundaries. If this behavior is not desired, simply strip out the additional newlines where needed.

This change was made due the fact that when header and paragraph HTML elements are simply stripped out, there often occurs issues where there is no separation between the heading and the proceeding sentence.

Example of Output Format for Text Field

Running tests

Unit tests are included in this distribution and can be run utilizing PHPUnit after installing dependencies. The recommended approach is to use Docker for this purpose, so you then don't even need to have dependencies installed on your system.

Note: Please set the environment variable DETECT_LANGUAGE_KEY with your Detect Language key in order for language detection in unit tests to work properly.

Installing Dependencies

This will use the composer docker image to download the requirements. Note the use of the --ignore-platform-reqs since some of our dependencies do not yet support PHP 8.

Running Unit Tests

This runs the phpunit dependency that we downloaded within the php 7.4 command line environment.