Extensions

Building a Custom Parser Extension

Build a wiki link extension that recognizes [[Page Name]] and [[Page Name|Display Text]], then query and transform the resulting nodes. The completed extension includes tokenization, node building, querying, configuration, diagnostics, and rewriting.

For the API reference covering all extension types and context methods, see Parser Extensions.

#Planning the Extension

A wiki link uses double-bracket syntax to reference other pages. The simplest form is [[Page Name]], and an optional pipe separator provides display text: [[Page Name|Display Text]]. The extension needs:

  • A trigger character ([) that tells the lexer when to consult the extension
  • A token type for the wiki link token
  • A node kind for the wiki link AST node
  • A custom node class with methods to extract the target and label

#Defining the Node

Start by creating a GenericNode subclass with methods to extract structured data from the raw source. Extension nodes should extend GenericNode rather than Node directly so that XPath queries can discover them. The getDocumentContent method (inherited from Node) returns the exact source text covered by the node:

<?php



use Forte\Ast\GenericNode;



class WikiLinkNode extends GenericNode

{

    public function target(): string

    {

        $inner = substr($this->getDocumentContent(), 2, -2);



        return trim(explode('|', $inner, 2)[0]);

    }



    public function label(): ?string

    {

        $inner = substr($this->getDocumentContent(), 2, -2);

        $parts = explode('|', $inner, 2);



        return isset($parts[1]) ? trim($parts[1]) : null;

    }



    public function hasLabel(): bool

    {

        return str_contains($this->getDocumentContent(), '|');

    }

}

The target method strips the [[ and ]] delimiters, splits on the pipe character, and returns the first part. The label method returns the second part if a pipe is present, or null otherwise.

#The Extension Class

Extend AbstractExtension to combine lexer and tree builder support in one class. Four methods are required: id, triggerCharacters, registerTypes, and doTokenize. The optional registerKinds method wires your custom node class into the tree builder:

<?php



use Forte\Extensions\AbstractExtension;

use Forte\Lexer\Extension\LexerContext;

use Forte\Lexer\Tokens\TokenTypeRegistry;

use Forte\Parser\NodeKindRegistry;



class WikiLinkExtension extends AbstractExtension

{

    private int $wikiLinkType;



    public function id(): string

    {

        return 'wiki-links';

    }



    public function triggerCharacters(): string

    {

        return '[';

    }



    protected function registerTypes(TokenTypeRegistry $registry): void

    {

        $this->wikiLinkType = $this->registerType($registry, 'WikiLink');

    }



    protected function registerKinds(NodeKindRegistry $registry): void

    {

        $this->registerKind($registry, 'WikiLink', WikiLinkNode::class);

    }



    protected function doTokenize(LexerContext $ctx): bool

    {

        if ($ctx->current() !== '[' || $ctx->peek(1) !== '[') {

            return false;

        }



        $start = $ctx->position();

        $ctx->advance(2);



        while (! $ctx->isAtEnd()) {

            if ($ctx->current() === ']' && $ctx->peek(1) === ']') {

                $ctx->advance(2);

                $ctx->emit($this->wikiLinkType, $start, $ctx->position());



                return true;

            }



            $ctx->advance();

        }



        return false;

    }

}

#How Tokenization Works

When the lexer encounters the trigger character [, it calls doTokenize. The method checks for a second [ and, if found, scans forward until it finds the closing ]]. Three outcomes are possible:

  1. Match: The method emits a token spanning from the opening [[ to the closing ]] and returns true. The lexer records the token and moves past it
  2. No match at prefix: The first character is [ but the next is not [. The method returns false and the lexer handles [ with its built-in logic
  3. No closing delimiter: The method reaches the end of source without finding ]]. It returns false, and the lexer resets to the original position

This "try and fall back" pattern is common across all extensions. Return true only after emitting a valid token. Return false to let the lexer proceed normally.

#Using the Extension

Pass the extension class to ParserOptions::withExtensions and use the resulting options with Forte::parse. Extension nodes appear alongside regular nodes and can be found with findAll:

<?php



use Forte\Facades\Forte;

use Forte\Parser\ParserOptions;



$options = ParserOptions::withExtensions(WikiLinkExtension::class);

$doc = Forte::parse('See [[Getting Started]] for details.', $options);



$links = $doc->findAll(fn ($n) => $n instanceof WikiLinkNode);



count($links);            // 1

$links[0]->target();      // "Getting Started"

$links[0]->hasLabel();    // false

#Parsing Labels

When a wiki link contains a pipe, the label method returns the display text:

<?php



use Forte\Facades\Forte;

use Forte\Parser\ParserOptions;



$options = ParserOptions::withExtensions(WikiLinkExtension::class);

$doc = Forte::parse('Read the [[Getting Started|intro guide]] first.', $options);



$link = $doc->findAll(fn ($n) => $n instanceof WikiLinkNode)[0];



$link->target();    // "Getting Started"

$link->label();     // "intro guide"

$link->hasLabel();  // true

#Source Preservation

Extension nodes preserve the original source, so render always reproduces the template exactly:

<?php



use Forte\Facades\Forte;

use Forte\Parser\ParserOptions;



$template = 'Visit [[Home]] and [[FAQ|questions]].';

$options = ParserOptions::withExtensions(WikiLinkExtension::class);

$doc = Forte::parse($template, $options);



$doc->render(); // "Visit [[Home]] and [[FAQ|questions]]."

#Adding Configuration

All extensions built with AbstractExtension include the HasConfiguration trait. Use configure to pass options and option to read them:

<?php



$ext = new WikiLinkExtension;

$ext->configure(['baseUrl' => '/wiki/', 'strict' => true]);



$ext->option('baseUrl');           // "/wiki/"

$ext->option('strict');            // true

$ext->option('missing', 'default'); // "default"

$ext->hasOption('baseUrl');        // true

$ext->hasOption('missing');        // false

You can read configuration values inside doTokenize to change tokenization behavior. For example, a strict option could reject wiki links with certain characters.

#Reporting Diagnostics

The HasDiagnostics trait (included in all abstract extension classes) lets you report warnings and errors during tokenization. Call warn, error, or info with a message and the byte range:

<?php



use Forte\Extensions\AbstractExtension;

use Forte\Lexer\Extension\LexerContext;

use Forte\Lexer\Tokens\TokenTypeRegistry;

use Forte\Parser\NodeKindRegistry;



class StrictWikiLinkExtension extends AbstractExtension

{

    private int $wikiLinkType;



    public function id(): string

    {

        return 'wiki-links';

    }



    public function triggerCharacters(): string

    {

        return '[';

    }



    protected function registerTypes(TokenTypeRegistry $registry): void

    {

        $this->wikiLinkType = $this->registerType($registry, 'WikiLink');

    }



    protected function registerKinds(NodeKindRegistry $registry): void

    {

        $this->registerKind($registry, 'WikiLink', WikiLinkNode::class);

    }



    protected function doTokenize(LexerContext $ctx): bool

    {

        if ($ctx->current() !== '[' || $ctx->peek(1) !== '[') {

            return false;

        }



        $start = $ctx->position();

        $ctx->advance(2);



        while (! $ctx->isAtEnd()) {

            if ($ctx->current() === ']' && $ctx->peek(1) === ']') {

                $end = $ctx->position() + 2;



                if ($end - $start === 4) {

                    $this->warn('Empty wiki link', $start, $end);

                }



                $ctx->advance(2);

                $ctx->emit($this->wikiLinkType, $start, $ctx->position());



                return true;

            }



            $ctx->advance();

        }



        return false;

    }

}

After parsing, retrieve diagnostics from the extension instance:

<?php



use Forte\Facades\Forte;

use Forte\Parser\ParserOptions;



$ext = new StrictWikiLinkExtension;

$options = ParserOptions::withExtensions($ext);

Forte::parse('See [[]] for details.', $options);



$diagnostics = $ext->getDiagnostics();



count($diagnostics);              // 1

$diagnostics[0]->message;         // "Empty wiki link"

$diagnostics[0]->isWarning();     // true

Pass an instance (not a class string) when you need to read diagnostics after parsing. A class string causes Forte to instantiate the extension internally, and you would not have a reference to retrieve the results from.

#Rewriting Extension Nodes

Extension nodes work with Forte's rewriting system just like built-in nodes. The rewriteWith method accepts a callback that receives a NodePath for each node in the tree. Use replaceWith to swap extension nodes for HTML:

<?php



use Forte\Facades\Forte;

use Forte\Parser\ParserOptions;

use Forte\Rewriting\NodePath;

use Illuminate\Support\Str;



$options = ParserOptions::withExtensions(WikiLinkExtension::class);

$doc = Forte::parse('See [[Getting Started|intro]] for details.', $options);



$result = $doc->rewriteWith(function (NodePath $path) {

    $node = $path->node();



    if ($node instanceof WikiLinkNode) {

        $target = $node->target();

        $label = $node->label() ?? $target;

        $slug = Str::slug($target);



        $path->replaceWith("<a href=\"/wiki/{$slug}\">{$label}</a>");

    }

});



$result->render(); // 'See <a href="/wiki/getting-started">intro</a> for details.'

Because extensions produce standard AST nodes, they compose naturally with rewrite passes, pipelines, and enclaves.

#See Also

Continue with these related Forte guides:

  • Parser Extensions: Full API reference for all extension types and context methods
  • Parser Options: Configure the parser and register extensions
  • Rewriters: Transform documents using the visitor pattern
  • Traversal: Navigate and query the document tree