Make tokens optional in AST for handcrafting

[Copilot]

Handcrafting AST nodes required providing token arrays and position metadata even when not needed, creating unnecessary boilerplate.

Changes

• Type system: Made tokens field optional in TextNode, Scheme, Authority, Query, Fragment, and UriAst. Renamed delimiter/colon token fields with "Token" suffix for clarity (colon → colonToken, delimiter → delimiterToken) and made them optional.

• Parser: Uses nullish coalescing to handle optional tokens. Still populates all token fields when parsing from strings (backward compatible, lossless tokenization maintained).

• Validation: Removed token presence checks to support handcrafted AST.

• Tests: Added 17 tests covering handcrafted AST creation and stringification. Updated all existing tests to use new field names.

• Documentation: Added handcrafting examples to README.

Field Naming

The "Token" suffix makes it clear these fields store token data (not just semantic information), enabling true lossless tokenization while keeping fields optional for handcrafting:

• Scheme.colonToken (optional) - stores the ':' token
• Query.delimiterToken (optional) - stores the '?' token
• Fragment.delimiterToken (optional) - stores the '#' token

Example

Before:

const ast: UriAst = {
  kind: 'uri',
  scheme: {
    kind: 'scheme',
    name: { kind: 'text', text: 'http', tokens: [] },
    colon: { type: 'Colon', value: ':', pos: 4 },
    tokens: []
  },
  path: { kind: 'text', text: '/path', tokens: [] },
  tokens: []
};

After:

const ast: UriAst = {
  kind: 'uri',
  scheme: {
    kind: 'scheme',
    name: { kind: 'text', text: 'http' }
    // colonToken is optional - no boilerplate needed!
  },
  path: { kind: 'text', text: '/path' }
};

Original prompt

---

This section details on the original issue you should resolve

<issue_title>req: Makes token optional in AST</issue_title>
<issue_description>Currently, handcrafting AST is difficult due to the need to restore tokens. This is unnecessary DX burden and we should make it optional.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes req: Makes token optional in AST #3

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[lxcid]

@copilot

for colon or delimiter, have a separate storing probably store token under token suffix.

this should allow true lossless tokenization

[Copilot]

@copilot

for colon or delimiter, have a separate storing probably store token under token suffix....

Done! Renamed fields to use "Token" suffix for clarity:

• colon → colonToken
• delimiter → delimiterToken

All fields remain optional for handcrafting while maintaining lossless tokenization when parsing. Changes in commit 279fabd.