Download the PHP package batnieluyo/extractor without Composer

Extractor: AI-Powered Data Extraction Library for Laravel.

Effortlessly extract structured data from various sources, including images, PDFs, and emails, using OpenAI within your Laravel application.

Features

• A convenient wrapper around OpenAI Chat and Completion endpoints.
• Supports multiple input formats such as Plain Text, PDF, Rtf, Images, Word documents and Web content.
• Includes a flexible Field Extractor that can extract any arbitrary data without writing custom logic.
• Can return a regular array or a Spatie/data object.
• Integrates with Textract for OCR functionality.
• Uses JSON Mode from the latest GPT-3.5 and GPT-4 models.

Example

Installation

Install the package via composer:

Publish the configuration file:

You can find all the configuration options in the configuration file.

Since this package relies on the OpenAI Laravel Package, you also need to publish their configuration and add the OPENAI_API_KEY to your .env file:

Usage

Extracting plain text from documents

Extracting structured data

The Extractor package includes a set of pre-built extractors designed to simplify the extraction of structured data from various types of text. Each extractor is optimized for specific data formats, making it easy to process different types of information. Below is a list of the included extractors along with brief descriptions and convenient shortened methods for each:

These extractors are provided out of the box and offer a convenient way to extract specific types of structured data from text. You can use the shortened methods to easily access the functionality of each extractor.

Using the Field extractor

The field extractor is great if you don't need much custom logic or validation and just want to extract out some structured data from a piece of text.

Here is an example of extracting information from a CV, note that providing a description to guide the AI model is supported, as well as nested items (which is useful for lists of sub-items, like work history, line items, comments on a product etc )

Using GPT-4-Vision with Extractor

Note: This feature is still WIP.

The Extractor package also integrates with OpenAI's new Vision API, leveraging the powerful gpt-4-vision-preview model to extract structured data from images. This feature enables you to analyze and interpret visual content with ease, whether it's reading text from images, extracting data from charts, or understanding complex visual scenarios.

How to Use OpenAI's Vision API with ImageContent

To use the Vision features in Extractor, you need to provide an image as input. This can be done in a few different ways:

1. Using a File Path: Load an image from a file path.
2. Using Raw Image Data: Use the raw data of an image, for example, from an uploaded file.
3. Using an Image URL: Load an image directly from a URL.

Here's how you can use each method:

Using a File Path

Using Raw Image Data

Using an Image URL

Extracting Data from Images with OpenAI's Vision API

After preparing your ImageContent object, you can pass it to the Extractor::fields method to extract structured data using OpenAI's Vision API. For example:

Creating Custom Extractors

Custom extractors in Extractor allow for tailored data extraction to meet specific needs. Here's how you can create and use a custom extractor, using the example of a Job Posting Extractor.

Implementing a Custom Extractor

Create a new class for your custom extractor by extending the Extractor class. In this example, we'll create a JobPostingExtractor to extract key information from job postings:

Note: Adding an instruction on which $outputKey key to nest the data under is recommended, as the JsonMode response from OpenAI end to want to put everything under a root key, by overriding the expectedOutputKey() method, it will tell the base Extractor class which key to pull the data from.

Registering the Custom Extractor

After defining your custom extractor, register it with the main Extractor class using the extend method:

Using the Custom Extractor

Once registered, you can use your custom extractor just like the built-in ones. Here's an example of how to use the JobPostingExtractor:

With the JobPostingExtractor, you can efficiently parse and extract key information from job postings, structuring it in a way that's easy to manage and use within your Laravel application.

Adding Validation to the Job Posting Extractor

To ensure the integrity of the extracted data, you can add validation rules to your Job Posting Extractor. This is done by using the HasValidation trait and defining validation rules in the rules method:

This will ensure that each key field in the job posting data meets the specified criteria, enhancing the reliability of your data extraction.

Extracting Data into a DTO

Extractor can integrate with spatie/data to cast the extracted data into a Data Transfer Object (DTO) of your choosing. To do this, add the HasDto trait to your extractor and specify the DTO class in the dataClass method:

OCR Configuration with AWS Textract

To use AWS Textract for extracting text from large images and multi-page PDFs, the package needs to upload the file to S3 and pass the s3 object location along to the textract service.

So you need to configure your AWS Credentials in the config/extractor.php file as follows:

You also need to configure a seperate Textract disk where the files will be stored, open your config/filesystems.php configuration file and add the following:

Ensure the textract_disk setting in config/extractor.php is the same as your disk name in the filesystems.php config, you can change it with the .env value TEXTRACT_DISK.

.env

Delete the files after processing them with Textract

Using S3 Lifecycle Rules

You can configure a lifecycle rule on your S3 bucket to delete the files after a certain amount of time, see the AWS docs for more info:

https://repost.aws/knowledge-center/s3-empty-bucket-lifecycle-rule

Using the cleanupFileUsing hook

By default, the package will NOT delete the files that has been uploaded in the textract S3 bucket, if you want to delete these files, you can implement this using the TextractUsingS3Upload::cleanupFileUsing(Closure) hook.

Note

Textract is not available in all regions:

Q: In which AWS regions is Amazon Textract available? Amazon Textract is currently available in the US East (Northern Virginia), US East (Ohio), US West (Oregon), US West ( N. California), AWS GovCloud (US-West), AWS GovCloud (US-East), Canada (Central), EU (Ireland), EU (London), EU ( Frankfurt), EU (Paris), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Seoul), and Asia Pacific ( Mumbai) Regions.

See: https://aws.amazon.com/textract/faqs/

All Parameters and Their Functions

$input (TextContent|string)

The input text or data that needs to be processed. It accepts either a TextContent object or a string.

$model (Model)

This parameter specifies the OpenAI model used for the extraction process.

It accepts a string value. Different models have different speed/accuracy characteristics and use cases, for convenience, most of the accepted models are provided as constants on the Engine class.

Available Models:

$maxTokens (int)

The maximum number of tokens that the model will process. The default value is 2000, and adjusting this value may be necessary for very long text. A value of 2000 is usually sufficient.

$temperature (float)

Controls the randomness/creativity of the model's output.

A higher value (e.g., 0.8) makes the output more random, which is usually not desired in this context. A recommended value is 0.1 or 0.2; anything over 0.5 tends to be less useful. The default is 0.1.

License

This package is licensed under the MIT License. For more details, refer to the License File.