feat: add schema-compare command to test harness#2891
Draft
markphelps wants to merge 1 commit intomainfrom
Draft
feat: add schema-compare command to test harness#2891markphelps wants to merge 1 commit intomainfrom
markphelps wants to merge 1 commit intomainfrom
Conversation
2c85d07 to
a3e6902
Compare
Add a new `schema-compare` command that builds each model twice — once with COG_STATIC_SCHEMA=1 (Go tree-sitter) and once without (Python runtime) — then compares the resulting OpenAPI schemas for exact JSON equality. Differences are reported with a structured diff showing the exact JSON paths that diverge. Also add 7 local fixture models covering the full input type matrix: scalar types, optional types (PEP 604 + typing.Optional), list types, optional list types, constraints/choices, File/Path types, and structured BaseModel output.
a3e6902 to
1ce7b49
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
schema-comparecommand to the test harness (tools/test-harness/) that compares static (Go tree-sitter) vs runtime (Python) schema generation for regression detection.How it works
For each model,
schema-compare:COG_STATIC_SCHEMA=1→ extracts OpenAPI schema from Docker labelLocal fixture models
7 local fixture models (
fixtures/models/) covering the full input type matrix:scalar-typesoptional-typesX | NoneandOptional[X]for all typeslist-typeslist[X]andList[X]for str, int, Path, Fileoptional-list-typeslist[X] | NoneandOptional[List[X]]constraints-and-choicesfile-path-typescomplex-outputThese use
repo: localin the manifest and are loaded fromfixtures/models/without cloning.Example output
No normalization is applied yet — the comparison is strict equality so we can see all differences before deciding which to fix vs which to accept.