Infra: Add custom lexers for PEP 823 + 824

[cdce8p]

Even minimal syntax highlighting can improve the readability of code snippets and examples in a PEPs. For syntax changes the default Python lexer provided by pygments can sometimes fail to parse the code having to fall back to pure text.

This PR adds new custom lexers which extends the default one to support the examples in PEP 823 and 824. The sphinx directives for these will be .. code-block:: py823 and py824 respectively.

• PEP 823: None-aware access operators #4798
• PEP 824: None-coalescing operators #4799

	Before	After
823
824

[cdce8p]

@hugovk Would you mind taking a look at this change from an infra perspective? Is it fine to add the custom lexers within pep_sphinx_extensions/lexers or should they be moved somewhere else? As a followup it might be great to add the ASDL lexer from the CPython repo here as well. Just wasn't sure about the licensing which is why I didn't include it. https://github.com/python/cpython/blob/main/Doc/tools/extensions/lexers/asdl_lexer.py

Note: The custom lexers themselves don't attempt to be feature complete, they work well enough to enable syntax highlighting in the PEPs though.

[hugovk]

Hi! I'll come back to this, but a quick note: might be some overlap with #5084? cc @ZeroIntensity

[ZeroIntensity]

Yeah, the new keywords look like they can be covered by #5084.

I think it would also be a good idea to make this generic; rather than making a custom lexer for every new syntax PEP, we should add a way to customize code blocks if necessary (my PR lets you do .. code-block:: python+soft-keywords:my_keyword, for example).

[cdce8p]

While I do like a good general solution, I'm not sure it's the best option here. Syntax changes in itself are difficult to generalize. Sure there are different categories, like adding a new (soft-)keyword or operator, but how would you make sure it works in every case? And if it doesn't work and requires changes, you would either have to go back and check every PEP that nothing broke or add a small test framework as well.

Writing a custom lexer is fairly strait forward and only a few lines of code. This will be even easier once there are a few examples you just need to copy and modify. It's also important to keep in mind that they don't have to be perfect. It's enough if they work for the examples in the PEP and nothing more.

[hugovk]

A benefit of #5084 is that it only applies to the PEPs you want it to.

It could be confusing if all the custom syntax started showing up in all PEPs.

[cdce8p]

A benefit of #5084 is that it only applies to the PEPs you want it to.

It could be confusing if all the custom syntax started showing up in all PEPs.

Unless I'm missing something, that applies to this PR as well. It also requires the author to explicitly request the lexer e.g. .. code-block:: py823, compared to .. code-block:: python+soft-keywords:my_keyword for #5084.

[hugovk]

Sorry, you're right, it was me missing something :)

[gvanrossum]

Also, PEP 823/4 don't add new keywords, they add new operators.

[ZeroIntensity]

Could we perhaps allow PEPs to specify custom PEG grammar and then generate a parser for it at build time?

[cdce8p]

Could we perhaps allow PEPs to specify custom PEG grammar and then generate a parser for it at build time?

That might be possible, but is it worth the effort? Tbh I only considered custom lexers recently after I saw how easy it was to write one for the examples in the PEPs. I'd like to point out that they are by no means perfect und sure there are likely edge cases which aren't covered but that isn't the point.

They are a quick and dirty solution which work well enough to add some basic syntax highlighting, nothing more. It was so simple to write that I even include highlighting for rejected ideas (maybe for 823 and otherwise for 824).

[cdce8p]

How can we move forward with this one? Would it be an acceptable (temporary) solution, maybe until someone builds something more general?

PEPs 823 and 824 are on track to be ready early next week. While it would be awesome to have syntax highlighting for the examples, in case this PR is unacceptable it would help knowing that sooner than later so we don't wait around for it.

[ZeroIntensity]

I'd personally be fine with this as a temporary solution if it helps PEP 823/824 reach discussion quicker.

[hugovk]

I'm a big fan of syntax highlighting in docs, but my concern is that once PEPs are implemented, the PEP becomes a historic document. Or maybe it's rejected, and becomes even more historic and less interesting.

We'd be maintaining this custom Sphinx code "forever", for PEPs that no one reads any more (and for some: no one cares about any more).

And maintaining custom Sphinx code can be a bit tricky, especially as our resident Sphinx maintainer is on an extended break.

Please don't wait on this PR (or the other one), to begin discussion of PEPs 823 and 824.

[cdce8p]

I understand the maintenance concerns. That's actually one of the reasons why I decided to not do a general plugin. The lexers are each only a few lines and at worst disabling / removing them will just revert back to a state without highlighting like we have now.

Personally, I'm prepared to deal with any issues for at least the duration of the PEP deliberations. Beyond that I don't think it's a bit deal either way. If the PEP is accepted, the syntax will be implemented and at some point the official Python lexer will support it anyway at which point the custom ones could be removed. Or it get's rejected and becomes historic. At that time the value proposition for syntax highlighting isn't there anymore, so if it breaks it could also be removed.

The custom lexers provide the most value during the actual discussion when folks are reading the PEPs for the first time. That's why I'm so keen on adding them. Hope that makes sense.

[ZeroIntensity]

We'd be maintaining this custom Sphinx code "forever", for PEPs that no one reads any more (and for some: no one cares about any more).

Yeah, I get this argument, but if the code breaks for whatever reason on a PEP we don't care about, we can just remove it. For accepted PEPs, we'll be able to remove it once Pygments supports the new syntax anyway.