Skip to content
Get packs

Doctrine Lexer

A PHP base library for splitting text into tokens, used to build parsers.doctrine/lexer

GitHub Download 3.0.3
Stars
11.1k
Forks
68
Licence
MIT
Language
PHP
Last update
19 days ago
Type
Library

What it is

Doctrine Lexer is a PHP library that breaks a string of text into a sequence of tokens, the first step in reading a structured language. Developers use it as a building block when writing top-down, recursive descent parsers. According to the README, it is used inside Doctrine Annotations and Doctrine ORM's query language (DQL).

  • Tokenizing the text of a custom configuration or query syntax before parsing it
  • Reusing the same lexer design that Doctrine Annotations uses for annotation parsing
  • Building a recursive descent parser on top of the token stream

Great for

  • PHP developers building a parser for a small custom language or syntax
  • Maintainers of PHP projects who want to reuse the lexer that Doctrine uses internally
  • Developers who need a lexer for recursive descent parsing rather than a full parser generator

Not ideal for

  • People who are not writing PHP code, since the library is PHP-only
  • Anyone wanting an end-user application; this is a code dependency with no interface
  • Projects needing a ready-made parser, because the README describes only the lexing base layer
Step-by-step guide: how to use Doctrine LexerInstall, first run, examples, tips and FAQs

How to get it

  1. 1

    Install with a package manager

    You'll need: PHP · Composer

    Install via Composer

    composer require doctrine/lexer
  2. 2

    Run it

    • Install the package with Composer.
    • Create a lexer class that extends the library's base lexer and defines token types.
    • Feed your input string to the lexer and iterate over the resulting tokens.
    • Pass the tokens into a recursive descent parser of your own design.
  3. 3

    Get the most out of it

    • Review the Doctrine ORM source code to see a real-world example of the lexer in use
    • Read the project homepage at doctrine-project.org for documentation beyond the README
    • Check the repository's continuous integration results to confirm the version you use is tested

Downloads

Source code (.zip)3.0.x branch · straight from GitHubRelease 3.0.3Release notes and sourceOfficial website & docswww.doctrine-project.org/projects/lexer.html

Strengths

  • Comes from the Doctrine project, which the README says uses it in Doctrine ORM and Annotations
  • Published under the permissive MIT license
  • Has a large community following, with about 11,000 stars on GitHub

Trade-offs

  • The README is very short and does not document the API or show example code
  • Scope is narrow: it only provides the lexing base, so you must write the parser yourself
  • Requires working knowledge of parsing concepts to use effectively

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.