Prototype of getting active vscode theme in the extension

[vezwork]

The problem

As reported in #525, the visual editor does not use code highlighting colors from the user's vscode color theme.

The extension does get some colors from the theme, but it does not get any code highlighting related colors. I believe the extension includes its own code highlighting color themes for each of the default vscode themes. If the user is using a built-in vscode theme (e.g. Monokai, Solarized, etc.) then the extension is able to guess that and select a matching theme definition from its own list. However, if the user is not using a built-in vscode theme, the extension somehow picks one of the built-in themes to use as the code highlighting theme, which is not great.

You can see this maybe most glaringly with a monochrome theme from the extension store:

Source Editor with Monochromatic theme	Visual Editor with same Monochromatic theme

---

Can we get the user's theme colors?

Yes, to some extent at least. See the discussion on the vscode repo here.

This PR so far gives a prototype of getting and logging code highlighting theme data from the user's vscode theme. Here's an example of what we can get:

Here's an example of the data that is logged in this PR

'[["comment",[["foreground","#6a737d"]]],["punctuation.definition.comment",[["foreground","#6a737d"]]],["string.comment",[["foreground","#6a737d"]]],["constant",[["foreground","#005cc5"]]],["entity.name.constant",[["foreground","#005cc5"]]],["variable.other.constant",[["foreground","#005cc5"]]],["variable.language",[["foreground","#005cc5"]]],["keyword.operator.symbole",[["foreground","#000000"]]],["keyword.other.mark",[["foreground","#000000"]]],["entity",[["foreground","#6f42c1"]]],["entity.name",[["foreground","#6f42c1"]]],["variable.parameter.function",[["foreground","#000000"]]],["entity.name.tag",[["foreground","#22863a"]]],["keyword",[["foreground","#d73a49"]]],["storage",[["foreground","#d73a49"]]],["storage.type",[["foreground","#d73a49"]]],["storage.modifier.package",[["foreground","#000000"]]],["storage.modifier.import",[["foreground","#000000"]]],["storage.type.java",[["foreground","#000000"]]],["string",[["foreground","#032f62"]]],["punctuation.definition.string",[["foreground","#032f62"]]],["string punctuation.section.embedded source",[["foreground","#032f62"]]],["string.unquoted.import.ada",[]],["support",[["foreground","#005cc5"]]],["meta.property-name",[["foreground","#005cc5"]]],["variable",[["foreground","#e36209"]]],["variable.other",[["foreground","#000000"]]],["invalid.broken",[["fontStyle","bold italic underline"],["foreground","#b31d28"]]],["invalid.deprecated",[["fontStyle","bold italic underline"],["foreground","#b31d28"]]],["invalid.illegal",[["fontStyle","italic underline"],["foreground","#b31d28"]]],["carriage-return",[["fontStyle","italic underline"],["foreground","#d73a49"]]],["invalid.unimplemented",[["fontStyle","bold italic underline"],["foreground","#b31d28"]]],["message.error",[["foreground","#b31d28"]]],["string source",[["foreground","#000000"]]],["string variable",[["foreground","#005cc5"]]],["source.regexp",[["foreground","#032f62"]]],["string.regexp",[["foreground","#032f62"]]],["string.regexp.character-class",[["foreground","#032f62"]]],["string.regexp constant.character.escape",[["fontStyle","bold"],["foreground","#22863a"]]],["string.regexp source.ruby.embedded",[["foreground","#032f62"]]],["string.regexp string.regexp.arbitrary-repitition",[["foreground","#032f62"]]],["support.constant",[["foreground","#005cc5"]]],["support.variable",[["foreground","#005cc5"]]],["meta.module-reference",[["foreground","#005cc5"]]],["markup.list",[["foreground","#735c0f"]]],["markup.heading",[["fontStyle","bold"],["foreground","#005cc5"]]],["markup.heading entity.name",[["fontStyle","bold"],["foreground","#005cc5"]]],["markup.quote",[["foreground","#22863a"]]],["markup.italic",[["fontStyle","italic"],["foreground","#000000"]]],["markup.bold",[["fontStyle","bold"],["foreground","#000000"]]],["markup.raw",[["foreground","#005cc5"]]],["markup.deleted",[["foreground","#b31d28"]]],["meta.diff.header.from-file",[["foreground","#b31d28"]]],["punctuation.definition.deleted",[["foreground","#b31d28"]]],["markup.inserted",[["foreground","#22863a"]]],["meta.diff.header.to-file",[["foreground","#22863a"]]],["punctuation.definition.inserted",[["foreground","#22863a"]]],["markup.changed",[["foreground","#e36209"]]],["punctuation.definition.changed",[["foreground","#e36209"]]],["markup.ignored",[["foreground","#005cc5"]]],["markup.untracked",[["foreground","#005cc5"]]],["meta.diff.range",[["foreground","#6f42c1"],["fontStyle","bold"]]],["meta.diff.header",[["foreground","#005cc5"]]],["meta.separator",[["fontStyle","bold"],["foreground","#005cc5"]]],["meta.output",[["foreground","#005cc5"]]],["brackethighlighter.tag",[["foreground","#586069"]]],["brackethighlighter.curly",[["foreground","#586069"]]],["brackethighlighter.round",[["foreground","#586069"]]],["brackethighlighter.square",[["foreground","#586069"]]],["brackethighlighter.angle",[["foreground","#586069"]]],["brackethighlighter.quote",[["foreground","#586069"]]],["brackethighlighter.unmatched",[["foreground","#b31d28"]]],["sublimelinter.mark.error",[["foreground","#b31d28"]]],["sublimelinter.mark.warning",[["foreground","#e36209"]]],["sublimelinter.gutter-mark",[["foreground","#959da5"]]],["constant.other.reference.link",[["foreground","#032f62"],["fontStyle","underline"]]],["string.other.link",[["foreground","#032f62"],["fontStyle","underline"]]],["meta.function-call support.function",[["foreground","#005cc5"]]],["meta.function-call entity.name.function",[["foreground","#005cc5"]]],["keyword.operator",[["foreground","#000000"]]]]'

Do the code colors we get from VSCode map to code colors for the Visual Editor?

No, not directly. Compare the keys in the code in the expandable section above with this code color theme type (from packages/editor/src/editor/editor-theme.ts) for the Visual Editor:

export interface CodeTheme {
  keywordColor: string;
  atomColor: string;
  numberColor: string;
  variableColor: string;
  defColor: string;
  operatorColor: string;
  commentColor: string;
  stringColor: string;
  metaColor: string;
  builtinColor: string;
  bracketColor: string;
  tagColor: string;
  attributeColor: string;
  hrColor: string;
  linkColor: string;
  errorColor: string;
}

The keys do not directly correspond to eachother. That being said, we are getting quite a bit of data from the VSCode theme, I would think that we could manually come up with a reasonable mapping to this set of 16 colors for CodeTheme.

Open Questions

• How do source editor code blocks get highlighted?
• Does the vscode color theme data we are getting in this PR change when the user changes (to a non-default) theme?

[vezwork]

Where this is currently at:

Before	After
without_themes.mp4 (note the glaring differences with custom themes)	with_themes.mp4

As was before, the code syntax highlighting in the Visual Editor simply does not match the Source Editor; Though this PR is an overall improvement in using the user's theme colors.

Notable exception: the dark blue abyss theme looks very bad in the After. It should be possible to address this particular exception with some finessing, but I'm sure other custom themes would lead to other exceptions.

How could we make the Visual Editor code block syntax highlighting actually match the Source Editor?

To answer that, let me give some context: I have learned that

• we use Codemirror with Lezer for syntax highlighting;
• lezer supports a very limited number of "tags" for highlighting compared to VSCode's themes which have a great abundance of different parts of source code that you can style and a great abundance of ways to express how things are styled, from scope cascading to logical operators on scopes;
• as such, we have to finesse and make heuristics if we want to map VSCode themes to our code block syntax highlighting. I've given a good initial attempt at that here (sorry this link doesn't seem to scroll to the right part of the code, trying ctrl+f'ing for tokenColorsLookup).
• This all means that, using Lezer, the VE syntax highlighting will never match the SE. It also means there will always be cases where our heuristics fail to produce a readable themes for some custom themes (unless we process the colors to force them to have contrast or something, but that would push us even further from VE and SE matching).

So, we would have to stop using Lezer.

We may be able to replace Lezer with Shiki which seems to have support for VSCode themes.

Another alternative: if we really care about consistency with the source editor, we could attempt to replace CodeMirror with Monaco. That scares me a bit, but should be do-able with some time and care. I talked to @softwarenerd and that is the approach that the Databot extension is using for this problem.

What should I do from here?

Options:

• [Easy-ish] Finesse this PR a bit more and land it as an incremental improvement (that still kinda sucks)
• [Medium, maybe-easy-maybe-not] Use Shiki for syntax highlighting
• [Hard] Use Monaco for code cells

[juliasilge]

I know you put this as "hard" but I do think in the long term it might be the right call to use Monaco for the code cells. 😬

I am very open to landing something incremental and better now, based on your current work, but the consistency that using Monaco would give us would be HUGE. I know it's not super relevant to your work, but that would also mean it would be possible to be consistent across regular source editors, Databot, the Positron console, the new Positron notebook frontend (also Monaco), and more.