Savaity/search skill experiment#48655
Draft
samvaity wants to merge 29 commits intoAzure:mainfrom
Draft
Conversation
….java for CI compliance
This reverts commit fbd62ca.
- Add find-package-skill registry at .github/skills/ (matches JS/Python/.NET pattern) - Fix search SKILL.md: add breaking change detection (Phase 3.6), import cleanup (Phase 3.7), test recording guidance (Phase 4), structured changelog format (Phase 5) - Fix search SKILL.md: rename to search-documents, improve trigger phrases - Fix search SKILL.md: add error categorization guidance (generated vs hand-written files) - Remove .NET comparison section from customizations.md (cross-language pollution) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
- Phase 3.5: Add dedup guard for includeOldApiVersions() — prevents duplicate enum constants when generator produces a version that was previously only in the customization list - Phase 3: Add accept-header type renumbering guidance (CreateOrUpdateRequestAccept* types change numbers between spec versions) - Customizations.md: Update includeOldApiVersions() code example with existence check and .toUpperCase() fix Eval: mock_search_2026-05-01-preview produced 18 build errors. These fixes address 2 of the 3 root causes found. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root cause analysis showed the import errors are NOT from type renumbering — they're from hideWithResponseBinaryDataApis() customization creating cross-package references. Updated Phase 3 guidance to point agent to SearchCustomizations.java instead of suggesting manual import edits on generated files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Skill fixes based on eval where with-skill agent performed worse on 2 of 3 errors: 1. SKILL.md: Add 'Common Pitfalls' section (Maor's one-pager pattern) calling out the 5 most dangerous mistakes upfront 2. SKILL.md Phase 3: Add error categorization table with 3 categories and explicit 'check SearchCustomizations.java FIRST' directive 3. SKILL.md Phase 3.5: Reframe from 'add versions' to 'verify and remove duplicates' 4. architecture.md: Add @generated annotation guidance — explains that generated files contain both @generated (auto-updated) and non-@generated (hand-written, needs manual update) methods 5. SKILL.md description: Use semicolons for trigger phrases (Maor's YAML convention) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lly lint vally lint requires directory name to match skill name in frontmatter. Before: Azure.Search.Documents/ with name: search-documents → name-directory-mismatch After: search-documents/ → 3/3 checks passed (spec-compliance, valid-refs, orphan-files) Also updated find-package-skill registry path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stdlib-only Python script that extracts code references (file paths, method names, class names) from skill markdown and verifies they still exist in the codebase. Catches the most common staleness pattern: code renamed/deleted but skill still references old names. Tested against search-documents skill: 51 anchors passed, 0 false positives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Interactive wizard (modeled after repo-bootstrap from ai-first-dev-starter-pack) that walks service teams through creating a package-specific skill: Phase 0: Scan package — detect architecture, customizations, key files Phase 1: Scaffold SKILL.md — generate with pitfalls, architecture, workflow Phase 2: Generate references — architecture.md and customizations.md Phase 3: Validate — vally lint + anchor checker Phase 4: Register — add to find-package-skill table Includes references for: - Skill template with required sections and structural rules - Validation tools guide (vally lint + validate_skill_anchors.py + CI workflow) - Cross-language examples with eval data (1/3 → 3/3 improvement) Key design decisions based on eval evidence: - Common Pitfalls MUST be first section (biggest eval impact) - Error categorization tables over prose - 'Check X FIRST' directives change agent behavior - Scaffold stubs, not finished products Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make explicit that package skills complement (not replace) the Azure SDK MCP tools and shared skills like generate-sdk-locally. Added: - 'Relationship to existing SDK tools' guardrail in SKILL.md - MCP tool references in Phase 1 scaffold (azsdk_package_generate_code, etc.) - 'Reference existing tools, don't redefine them' directive This prevents service teams from creating skills that duplicate or conflict with the deterministic SDK generation pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The generate-sdk-locally shared skill (synced via PR Azure#48773) and Azure SDK MCP tools already handle generation, build, test, changelog, and versioning. The search package skill was re-documenting these steps, creating overlap. Refactored to: - Defer generation/build/test to generate-sdk-locally skill and MCP tools - Keep ONLY search-specific content: error categorization table, customization patterns, ServiceVersion management, breaking change detection, import cleanup - Add explicit 'use generate-sdk-locally for generation/build/test' directive Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. Skill name = Maven artifactId (azure-search-documents, not search-documents) 2. No versioning in skills -- keep content static (design, patterns, not release state) 3. TypeSpec-first customization guardrail -- prefer @@clientName/@@access over Java AST 4. Note azsdk_customized_code_update MCP tool overlap with post-regen checklist 5. Breaking change detection context (eval found 29 removed types not caught) 6. Removed duplicated generation workflow steps (shared skill handles those) 7. Content principles: static, TypeSpec-first, don't re-doc MCP, focus on convenience layer Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.