Download the PHP package acdh-oeaw/repo-file-checker without Composer

# repo-filechecker

Functionality

• Analyzes the data structure and creates a json/ndjson output providing:
  • Files list
  • Directory list
  • File type list
  • Errors list
• Can also create HTML reports from the generated JSON file.
• When run as a docker container, performs antivirus check on files.

Implemented error checks

• File and directory names don't contain forbidden characters.
• File extension matches MIME type deteced based on the file content (MIME-extensions mapping based on the PRONOM database with some tuning for not fully reliable content-based MIME type recognition).
• MIME type of a file must be accepted by the ARCHE (as reported by the arche-assets).
• Text files don't contain the byte order mark.
• BagIt archives are correct (based on checks performed by the whikloj/bagittools library; bagit archives can be uncompressed of zip/tar gz/tar bz2 files).
• ZIP, XLSX, DOCX, ODS, ODT and PDF files aren't password protected.
  • To avoid memory limit problems only files up to a configuration-determined size are checked.
• XML files provide XML declaration and schema declaration and validate against the schema.
• Image files aren't corrupted.
• No duplicated files (compared by hash).
• No filenames conflicts on case-insensitive filesystems.

Installation

Locally

The filechecker depends on presence of some external tools in your system (e.g. gdal) so trying to run it locally can be a painful experience. If you want to try, just:

• Install PHP and composer

• Run:

• Install any other missing software based on errors you get while running the filechecker.

As a docker image

• Install docker.

On ACDH Cluster

Nothing to be done. It is installed there already.

Usage

General remarks

• You can test if the check was successful by reading the exit code of the arche-filechecker command. 0 indicates a successful check and non-zero value that at least one error was found.

• To get a list of all available parameters run:

• If you have bagit files, place them into a folder called bagit and also compress them into a tgz file.

On ACDH cluster

First, get the arche-ingestion workload console as described here

Then:

• filechecker

• virus scan

Locally

As a docker container

• Consider downloading fresh signatures for the antivirus software

  • If you're running inside a CI/CD workflow and don't want to be a bad guy causing unnecessary load on the server storing the signature, store the downloaded database in a cache, e.g. on Github Actions you may perform the db update using following build steps:

  • On localhost (just adjust the path to the directory with the virus signatures)

• To run a virus check

  e.g.

• To run the filechecker

  e.g.

Remarks:

• If you're processing data in parts you can save some time by running the container in the daemonized mode. That way you can avoid loading the virus signatures database every time you run the check. The database load takes 2-5 seconds. In the daemonized setup:

  • Run the container with

  • Wait a few seconds for the AV software to load the viruses database (you can look at docker logs to check if it's ready).
  • Perform the checks with

Test Files:

Test files are stored in the tests/data folder.