Generate the `vgplot-python` API using Python

[dangotbanned]

Why?

Why should we do it?

IMO, a strong point of altair is that because API generation takes place in the language of
it's users - the bar is relatively low for new contributors.
For example, there are 12 authors for the most recent 30 commits to vega/altair/tools

Making this change would lower the barrier to who can maintain this code ¹ and hopefully
increase the chances of a bug reporter being able to fix their issue themselves.

Surely LLMs can bridge the gap for those unfamiliar with JS?

Aha, got you!

If a potential contributor doesn't know JS, are they in a position to judge the correctness of LLM
output in that language?

Why did I already do it?

Note

Everything from here onwards is explaining my rabbit hole(s)
Feel free to stop reading now!

A couple of weeks ago I was looking into fixing (#1095 (comment)) as part of (#1088).
I'm embarrassed to admit it, but it took me quite some time to wrap my head around generate-python-api.js.
The lack of typing made the process pretty painful, as my IDE wasn't able to fill in the gaps.

To get myself out of that pickle, I added some JSDoc ² and then began porting the rest in python to figure out what I needed to change.

Note

This first rabbit hole led to a second one - which sidetracked me for over a week -
before continuing on this one a couple days ago (2d87f54a6cf435e47c1ea600b48124c7a6873cf8)

So anyway, back to rabbit hole 1!

Equipped with my recent knowledge of mosaic-schema.json - the rest fell out without too much of battle 😄

See changes (compare/main...dangotbanned:mosaic:vgplot-python/fix-dunder-all-2)

I was able to fix __all__!

Findings

I've found some other oddities in the process, which I wanted to mention before opening a PR ³

TL;DR: (1) I unintentionally fixed; (2) I haven't repeated but still can; (3) I have repeated but would prefer not to.

1 - type -> type_?

This code is responsible for renaming type:

mosaic/bin/generate-python-api.js

Lines 26 to 30 in 7afc4ec

	/** Python identifier for a schema (camelCase) name, keyword-safe. */
	function ident(name) {
	const s = camelCaseToSnake(name);
	return PYTHON_KEYWORDS.has(s) ? s + '_' : s;
	}

Which appears here:

mosaic/packages/vgplot/vgplot-python/vgplot/_generated/marks.py

Line 1329 in 7afc4ec

type_: ChannelValue | UNSET = UNSET,

Despite it already having been documented that this is not needed, here:

mosaic/packages/vgplot/spec/src/python-codegen.js

Lines 16 to 20 in 7afc4ec

	// Keywords that cannot appear as a bare keyword-argument name. `type` is a
	// builtin, not a reserved word, so it stays valid as a kwarg.
	const PYTHON_KWARG_UNSAFE = new Set(
	[...PYTHON_KEYWORDS].filter(k => k !== 'type')
	);

I only noticed this because I used keyword.iskeyword ⁴, which correctly allows type as a
keyword, but that also made me realize there is no coverage for either 😬

2 - transform-keys.js

This one confused me, because it lists every transform as they all have "a generated vgplot Python
API function."

mosaic/bin/generate-python-api.js

Lines 212 to 220 in 7afc4ec

	// Shared artifact for ast-to-python.js: the transform keys that have a
	// generated Python function, so the spec emitter needs no hand-kept list.
	writeFileSync(resolve(SPEC_GEN_DIR, 'transform-keys.js'),
	'// DO NOT EDIT. Generated from the Mosaic JSON schema by bin/generate-python-api.js.\n' +
	'// Regenerate with: pnpm run generate:python-api\n\n' +
	'/** Transform keys with a generated vgplot Python API function. */\n' +
	'export const TRANSFORM_KEYS = new Set([\n' +
	transforms.map(({ key }) => ` '${key}',`).join('\n') +
	'\n]);\n');

To figure out if it did anything at all, I deleted this line where TRANSFORM_KEYS is used:

Show me the diff

diff --git a/docs/public/specs/python/wnba-shots.py b/docs/public/specs/python/wnba-shots.py
index abd185a6..d1f74657 100644
--- a/docs/public/specs/python/wnba-shots.py
+++ b/docs/public/specs/python/wnba-shots.py
@@ -32,7 +32,7 @@ view = vg.vconcat(
             y="y_position",
             fill=vg.avg("score_value"),
             r=vg.count(),
-            tip={"format": {"x": False, "y": False}},
+            tip=vg.format({"x": False, "y": False}),
         ),
         vg.line(court, stroke_linecap="butt", stroke_opacity=0.5, x="x", y="y", z="z"),
         vg.name("shot-chart"),
diff --git a/packages/vgplot/spec/src/ast-to-python.js b/packages/vgplot/spec/src/ast-to-python.js
index 345992de..546beebb 100644
--- a/packages/vgplot/spec/src/ast-to-python.js
+++ b/packages/vgplot/spec/src/ast-to-python.js
@@ -207,7 +207,6 @@ function emitEncoding(v, ctx) {
   if (!keys.length) return literal(v, 0, ctx);
   const key = keys[0];
   if (key === 'sql' && keys.length === 1) return `vg.sql(${literal(v.sql, 0, ctx)})`;
-  if (!TRANSFORM_KEYS.has(key)) return literal(v, 0, ctx);
   const { [key]: val, ...opts } = v;
   const args = (val === '' || val == null) ? []
     : (Array.isArray(val) ? val : [val]).map(x => literal(x, 0, ctx));
diff --git a/specs/python/wnba-shots.py b/specs/python/wnba-shots.py
index abd185a6..d1f74657 100644
--- a/specs/python/wnba-shots.py
+++ b/specs/python/wnba-shots.py
@@ -32,7 +32,7 @@ view = vg.vconcat(
             y="y_position",
             fill=vg.avg("score_value"),
             r=vg.count(),
-            tip={"format": {"x": False, "y": False}},
+            tip=vg.format({"x": False, "y": False}),
         ),
         vg.line(court, stroke_linecap="butt", stroke_opacity=0.5, x="x", y="y", z="z"),
         vg.name("shot-chart"),

I'm very confused by this solution. Nothing about the code
suggests that this is what's happening. Does this make sense to anyone else?
My questions here are:

1. Why isn't there a vg.format?
   1. If there was, then TRANSFORM_KEYS would be a noop
2. Can we drop this part of the API generation and handle whatever this is doing in a clearer
   way?
   1. Either by adding the missing format helper
   2. Or identifying when {"format": {... should be generated by using something named FORMAT*

3 - Docstrings

I hope it isn't too controversial to say that this is a tough read 😉

mosaic/bin/generate-python-api.js

Lines 32 to 43 in 7afc4ec

	/** First sentence of a schema description, with markdown links stripped and
	* escaped for a docstring. */
	function docline(desc, fallback) {
	let text = (desc || fallback || '')
	.replace(/\[([^\]]+)\]\([^)]*\)/g, '$1') // [text](url) -> text
	.replace(/\[([^\]]+)\]\[[^\]]*\]/g, '$1') // [text][ref] -> text
	.replace(/\[(\d+)\]/g, '') // bare footnote [1] -> (removed)
	.replace(/\[([^\]]+)\]/g, '$1') // [text] shortcut -> text
	.replace(/\s+/g, ' ').trim();
	const first = text.split(/(?<=\.)\s/)[0] || fallback;
	return first.replace(/\\/g, '\\\\').replace(/"""/g, '\\"\\"\\"');
	}

But the larger issue to me is I don't understand why this is happening in the first place.

The docs in vgplot/_generated/attributes.py could be so much better if they just kept things
as they are in the original (PlotAttribute.ts)

How do I know? Because here's the docs and typing sitting pretty in python!

If you made it this far

Apologies for the essay! 😅
This one leads into why packages/vgplot/spec-python even exists - which I need to get around to writing more about ⏲️

Footnotes

1. understanding 1 language vs 2 ↩

2. which I removed in 7e1ab15013e38e2d42c3cb14045dcf535649247a ↩

3. if there's interest ↩

4. Python stdlib function ↩

[domoritz]

We actually had a python version of the generator before but I suggested we switch to js because we already had the api generator for esm written in js. It was nice not having to repeat some logic we already had.

That's not to say I'm against writing the generator in python but just to offer an explanation.

Honestly, i think the best way forward would be a generic api generator that maybe does not even start with json schema since json schema is so lossy when it comes to types.

[dangotbanned]

Thanks for the quick response @domoritz

We actually had a python version of the generator before but I suggested we switch to js because we already had the api generator for esm written in js.
It was nice not having to repeat some logic we already had.

That's not to say I'm against writing the generator in python but just to offer an explanation.

Ah I didn't know this part, so the explanation is appreciated!
I can see how my suggestion carries a bit of whiplash with it 😅

Honestly, i think the best way forward would be a generic api generator that maybe does not even
start with json schema since json schema is so lossy when it comes to types.

I'm also interested in pursuing this direction, but in my head I'm seeing it as more of a long-term goal.
I see 3 related but distinct ideas and creating a generic api generator is most in need of
it's own issue to define it's scope.

Overview

Note

TL;DR: These all feel different to me and not like an either/or situation
(1): Small potatoes, doesn't really change the world and is almost ready to go.
(2): More scope, can survive (and likely benefit) from switching to a source that is not JSON Schema - but can be improved regardless.
(3): Maximum scope, I want to sink my teeth into it but see many unknowns which I'm looking forward to finding the answers too!

1 (#1154)

Moving generate-python-api.js to python has the smallest scope; is entirely non-user facing and could always be reverted later if it turns out to be the wrong move.

(Selfishly) it has the immediate benefit of allowing me to work faster on the python side 😉.
In the medium-term, I hope that this move can attract more python contributors, since JS experience wouldn't be needed for a larger slice of Mosaic.

Alternative

Something I'd benefit from - would be adding JSDoc annotations anywhere in generate-python-api.js that calls a method or accesses a property on a symbol that is currently inferred as any.
I'd have more visibility into what is happening and where changes propagate to through types.

I'd still prefer it in python, but I'm open to solving the problem I had in other ways

2 (#1075)

Adding a python equivalent of mosaic-spec is a (relatively) big leap in scope.
My vision of it would be a package that vgplot-python could build on top of, while also unlocking more ways other packages can integrate with Mosaic's python side.

My current draft (#1075 (comment)) uses JSON Schema as that's what is available now.
I needed to massage the schema a bit to get things into a reasonably functional state, but the idea can work.
In the process, I've identified several improvements ¹ we can make, without leaving JSON schema behind spec-python/roadmap.md#mosaic-spec-feedback.

With that being said, I would happily take on using a source that is not JSON Schema.
I briefly explored the idea of first lowering mosaic-schema.json into a new IR written for this specific codegen task.
Ultimately, I went in a "simpler" ² direction of operating directly on the schema for now.
Transitioning to a new source would be easier if I reoriented things around generating from an IR, so I'm keen to do that anyway!

I will save you from any more gushing about this until I update (#1075) 😂.

3 (A new API generator)

The most exciting idea IMO, but it currently has an unbounded scope.
Here are some things we could discuss to narrow that (probably in a series of issues):

• What does a generic API generator mean to you?
  • Source
    • Are you comfortable with defining the API in a language that is not TypeScript? ³
  • Target
    • A per-language equivalent of a TS Interface?
    • Functionality like a builder API?
  • Usage
    • Is there a need for extensibility, or is the audience limited to Mosaic?
• Which areas of JSON Schema are causing the most pain for us?
  • Discussed a bit in #1075
  • Would be helpful to know if things are/can be worked around and the cost of doing so?
• What needs does a new API generator have, that are/are not met by existing an Interface Definition Language?
  • I looked into each of these after opening (#1075), but none seemed expressive enough.
  • Perhaps there are things to learn from them though?
    • Google FlatBuffers
    • Google Protobuf
    • Apache Thrift
    • Apache Fory
• Is there any other interesting prior art, which solves a subset of the problem?
  • https://github.com/vega/ts-json-schema-generator
  • https://github.com/koxudaxi/datamodel-code-generator
  • https://github.com/apioo/typeschema
  • https://github.com/ccxt/ast-transpiler

Footnotes

1. E.g. less work for me in the "build step"; representing the same types while
   generating less code; improving LSP performance (particularly for Pylance w/ Pyright). ↩

2. "Simple" if changes to the schema don't happen to break anything - but less than ideal for a
   long-term solution. ↩

3. AFAIK, TypeScript is quite unique in how expressive it is. That expression is a huge positive
   for writing a definition but comes at a cost when translating it to other languages. ↩