Download the PHP package wp-php-toolkit/data-liberation without Composer

slug: dataliberation title: DataLiberation install: wp-php-toolkit/data-liberation

see_also:

• ../learn/03-importing-content.html | Tutorial — Markdown to WXR | The chapter that walks through importing a folder of Markdown files into WordPress via the toolkit.
• markdown | Markdown | Use Markdown as a source or destination format.
• blockparser | BlockParser | Analyze serialized blocks inside post content.

• httpclient | HttpClient | Download media and remote source data while importing.

Streaming WordPress import/export. WXR, SQL, block markup — process entities one at a time instead of building whole-dataset object graphs.

Why this exists

WordPress content should be portable, but real migrations cross several formats. A site export might arrive as WXR, a Markdown folder, or entities from another CMS. URLs can hide in block attributes, HTML, CSS, feeds, GUIDs, and post meta. Importers must also resume after a failed media download or upload.

The DataLiberation component streams WordPress-shaped data through readers, transformers, and writers. It models posts, terms, comments, attachments, and metadata as ImportEntity objects, then lets a pipeline rewrite each entity without loading the full export into memory.

The API reflects specific migration bugs: relative URLs in known block attributes, URLs inside inline CSS, self-closing block comments that must keep their shape, and origin-only URLs whose trailing slash style should not change during a rewrite.

Reach for it when the job combines formats: build WXR from another CMS, rewrite a staging export for production, frontload remote assets, or compose Markdown, XML, HTML, CSS, and URL rewriting into one pipeline.

Write a WXR file in five lines

Stream a single post into a WXR document via WXRWriter. The writer emits each entity to the output stream and only keeps the small amount of state needed for the current document.

Build a WXR programmatically from any source

The writer doesn't care where entities come from. Loop over rows from a CMS, a CSV, or a Notion API dump and emit posts plus their meta and comments.

Read entities from a WXR file incrementally

WXREntityReader emits one entity at a time. Memory use is driven by the current entity and parser buffers rather than the total file size.

Streaming transform: rewrite URLs while copying WXR

Wire reader to writer to rewrite a WXR file on the fly. This pattern is how you migrate a staging export to production: swap staging.example.com for example.com while holding only the current entity and output buffers.

Render Markdown into a WXR import in one pipeline

Compose MarkdownConsumer with WXRWriter to publish a folder of Markdown directly as a WordPress import file.

Replace a CSS value prefix without changing its suffix

CSSProcessor::measure_value_prefix() finds how many source bytes represent a decoded prefix, including CSS escapes and string line continuations. Replace those bytes with escape_value_prefix() output to keep the existing quotes, url() wrapper, and unmatched suffix unchanged. The escaped replacement also works in unquoted URLs. Whole-value set_token_value() still adds quotes and normalizes CRLF to one newline without dropping text after a lone CR.

Prefix measurement scans ordinary URL bytes with strspn() and decodes escapes and UTF-8 separately. Replacement escaping uses strtr() rather than a PHP character loop, with an early return when no bytes need escaping.

The prefix-edit caller and the whole-URL caller show each operation separately, without streamed input or saved cursors.

Find CSS URLs in imports and image sets

CSSURLProcessor::next_url() recognizes url(), bare @import strings, and strings used as images in image-set() or -webkit-image-set(). Comments, displayed text, MIME-type strings, and malformed string or URL tokens are not returned. More than 128 nested image sets throw an error.

The file-rewrite caller uses the whole-string iterator and writes the output only after iteration completes.

Stream CSS with the existing scan-and-edit API

For example, one read can end at url(https://old.exa and the next can supply mple/photo.png). The processor waits for the rest of that URL before returning it. The caller then reads and edits it with the same methods used for a complete CSS string.

Both CSSProcessor and CSSURLProcessor follow the XML streaming API:

1. Call create_for_streaming($css = '', $cursor = null) to supply initial bytes or start with an empty buffer.
2. Call append_bytes($bytes) when more source bytes arrive.
3. Use next_token() or next_url() and the existing getters and setters. A false result can mean the processor needs more input. Check is_paused_at_incomplete_input() to distinguish that from completion.
4. Call input_finished() only at the actual source end. Continue scanning to read the last tokens. A stopped download is not the source end.
5. is_finished() becomes true when input has ended and no current or unread token remains. is_expecting_more_input() says whether more bytes can still be appended.

The whole-string API is unchanged: CSSProcessor::create($css) and new CSSURLProcessor($css) still take a complete stylesheet. Getters, setters, and get_updated_css() work the same way with either input mode. There is no separate rewrite_chunk() API or built-in URL mapping policy.

The caller chooses which URLs to change. set_raw_url() uses the existing whole-value setter: it quotes an unquoted URL and escapes the replacement for CSS. For example, replacing old.png with new.png changes url(old.png) to url("new.png"). It does not apply a separate prefix-only rewrite rule.

Write and release completed output

flush_processed_css() returns completed CSS with edits applied and removes those source bytes from memory. An unfinished token remains for the next read. Flushing clears the current token or URL, so edit it before flushing.

Appending input does not require a flush. Without flushing, get_updated_css() returns all supplied CSS with edits applied. After a flush, it returns only the retained CSS. Choose when to flush based on how much output the caller wants to keep in memory. For example, flushing after each edited URL avoids storing many large replacements. Flush once more after scanning stops to get completed CSS after the last URL. There is no automatic output-size threshold or byte slicing.

There is no token-size cap. A large comment, string, identifier, or embedded image can use a lot of memory even with small input chunks. Each read reparses the unfinished token. Small chunks therefore do not bound the largest token's memory use or parsing work. The cursor does not copy these bytes. More than 128 open, nested image-set() functions causes an error.

Resume in a new process

Suppose the source is a{src:url(https://old.example/photo.png)}. A read ends inside the URL. After flushing a{src:, the saved source offset points at url(, not at the end of that read. A new process reads the unfinished URL again from url(. It does not repeat the flushed prefix.

get_reentrancy_cursor() returns an opaque string. Save it with get_token_byte_offset_in_the_input_stream(). Supply source bytes from that offset to create_for_streaming($css, $cursor). The cursor contains parsing state, including the URL position after @import or inside image-set(), but no source bytes or edits. Do not inspect or change its internal format.

A cursor saved while a token or URL is current reads that token again on resume, as XML does. If input_finished() was already called, pass all remaining source bytes to the factory; appending after the source end is rejected. For file rewrites, save a checkpoint after flushing completed output instead:

1. Read a source chunk and append it. Mark the source end when it is reached.
2. Scan and edit using the ordinary token or URL methods.
3. Write the string from flush_processed_css() and flush the output file.
4. Save the processor's source byte offset, the output file offset, and the parser cursor together. Do not use the input file handle's current offset: it can be past bytes that the processor still needs to read again.

On resume, seek the source to its saved offset. Remove output bytes after the saved output offset, then append there. Those extra bytes may have been written before the previous process stopped, but after its last checkpoint. Removing them prevents duplicate output when the corresponding source is read again. Keep the source file and the caller's edit rules unchanged between runs. The processor does not check either of them.

If a write fails, discard the processor and resume from the last checkpoint. The token file caller and URL file caller show how to save and restore both file positions and the parser state. Their tests stop on both sides of a checkpoint and start a fresh PHP process to finish the output.