diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 0999af523..3c33fd607 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -13,18 +13,17 @@ Applicable helpers: Applicable plugins: -- [ ] allure +- [ ] aiTrace - [ ] autoDelay - [ ] autoLogin - [ ] customLocator -- [ ] pauseOnFail +- [ ] pause - [ ] coverage +- [ ] heal - [ ] retryFailedStep -- [ ] screenshotOnFail +- [ ] screenshot - [ ] selenoid -- [ ] stepByStepReport - [ ] stepTimeout -- [ ] wdio - [ ] subtitles ## Type of change diff --git a/docs/commands.md b/docs/commands.md index 8e087ae6c..dff6a23b6 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -115,10 +115,10 @@ Plugins listed via `-p` are activated even when their config has `enabled: false A few examples: ```sh -npx codeceptjs run -p pauseOnFail # pause on first failure -npx codeceptjs run -p pauseOn:step # pause before every step -npx codeceptjs run -p pauseOn:url:/checkout/* # pause on URL match -npx codeceptjs run -p stepByStepReport # produce a step-by-step HTML report +npx codeceptjs run -p pause # pause on first failure (default on=fail) +npx codeceptjs run -p pause:on=step # pause before every step +npx codeceptjs run -p pause:on=url:pattern=/checkout/* # pause on URL match +npx codeceptjs run -p "screenshot:on=step;slides=true" # produce a step-by-step HTML report ``` ### Browser Control diff --git a/docs/configuration.md b/docs/configuration.md index 8aa05c3ad..73d50992b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -194,13 +194,14 @@ import { setCommonPlugins } from '@codeceptjs/configure' setCommonPlugins() ``` -| Plugin | Default | Notes | -| :----------------- | :------------- | :----------------------------------------------------------------------------- | -| `retryFailedStep` | enabled | Retry steps that fail with transient errors | -| `screenshotOnFail` | enabled | Capture a screenshot when a test fails | -| `pauseOn` | registered | Pause on failure / step / file / URL — `-p pauseOn:fail`, `-p pauseOn:step`, `-p pauseOn:file:tests/login_test.js`, `-p pauseOn:url:/checkout/*` | -| `browser` | registered | CLI overrides for browser helpers — `-p browser:show`, `-p browser:browser=firefox`, see [commands](/commands#browser-control) | -| `aiTrace` | registered | Capture AI traces — `-p aiTrace` | +| Plugin | Default | Notes | +| :---------------- | :------------- | :----------------------------------------------------------------------------- | +| `retryFailedStep` | enabled | Retry steps that fail with transient errors | +| `screenshot` | enabled | Screenshot on `fail` (default) / `test` / `step` / `file` / `url` | +| `pause` | registered | Pause on failure / step / file / URL — `-p pause:on=fail`, `-p pause:on=step`, `-p pause:on=file:path=tests/login_test.js`, `-p pause:on=url:pattern=/checkout/*` | +| `browser` | registered | CLI overrides for browser helpers — `-p browser:show`, `-p browser:browser=firefox`, see [commands](/commands#browser-control) | +| `aiTrace` | registered | Capture AI traces — `-p aiTrace`, narrow with `on=fail|test|step|file|url` | +| `heal` | registered | Self-heal failing steps — `-p heal`, narrow with `on=file|url` | > `eachElement`, `tryTo`, and `retryTo` are no longer plugins in 4.x — import them from `codeceptjs/effects`. diff --git a/docs/debugging.md b/docs/debugging.md index 81b8cd66a..9368423dc 100644 --- a/docs/debugging.md +++ b/docs/debugging.md @@ -107,28 +107,30 @@ After(({ I }) => { }) ``` -## Pause On Plugin +## Pause Plugin -For automated debugging without modifying test code, use the `pauseOn` plugin. It pauses tests based on different triggers, controlled entirely from the command line. +For automated debugging without modifying test code, use the `pause` plugin. It pauses tests based on different triggers, controlled entirely from the command line. The default is `on=fail`. ### Pause on Failure Automatically enters interactive pause when a step fails: ```bash -npx codeceptjs run -p pauseOn:fail +npx codeceptjs run -p pause +# or, explicit: +npx codeceptjs run -p pause:on=fail ``` This is the most common debug workflow — run your tests, and when one fails, you land in the interactive shell with the browser in the exact state of the failure. You can inspect elements, try different selectors, and figure out what went wrong. -> The older `pauseOnFail` plugin still works: `npx codeceptjs run -p pauseOnFail` +> The legacy `pauseOnFail` plugin still works as a deprecated alias. ### Pause on Every Step Enters interactive pause at the start of the test. Use *ENTER* to advance step by step: ```bash -npx codeceptjs run -p pauseOn:step +npx codeceptjs run -p pause:on=step ``` This gives you full step-by-step execution. After each step, you're back in the interactive shell where you can inspect the page before pressing ENTER to continue. @@ -138,13 +140,13 @@ This gives you full step-by-step execution. After each step, you're back in the Pauses when execution reaches a specific file: ```bash -npx codeceptjs run -p pauseOn:file:tests/login_test.js +npx codeceptjs run -p pause:on=file:path=tests/login_test.js ``` With a specific line number: ```bash -npx codeceptjs run -p pauseOn:file:tests/login_test.js:43 +npx codeceptjs run -p pause:on=file:path=tests/login_test.js;line=43 ``` This works like a breakpoint — the test runs normally until it hits a step defined at that file and line, then opens the interactive shell. @@ -154,14 +156,14 @@ This works like a breakpoint — the test runs normally until it hits a step def Pauses when the browser navigates to a matching URL: ```bash -npx codeceptjs run -p pauseOn:url:/users/1 +npx codeceptjs run -p pause:on=url:pattern=/users/1 ``` Supports `*` wildcards: ```bash -npx codeceptjs run -p pauseOn:url:/api/*/edit -npx codeceptjs run -p pauseOn:url:/checkout/* +npx codeceptjs run -p pause:on=url:pattern=/api/*/edit +npx codeceptjs run -p pause:on=url:pattern=/checkout/* ``` This is useful when you want to inspect a specific page regardless of which test step navigates there. @@ -243,15 +245,16 @@ Enabled by default. Saves a screenshot when a test fails: ```js plugins: { - screenshotOnFail: { + screenshot: { enabled: true, + on: 'fail', uniqueScreenshotNames: true, fullPageScreenshots: true, } } ``` -Screenshots are saved in the `output` directory. +Screenshots are saved in the `output` directory. The same plugin also supports `on=test`, `on=step`, `on=file`, and `on=url` to capture screenshots in other situations. ### Page Info on Failure @@ -267,12 +270,14 @@ plugins: { ### Step-by-Step Report -Generates a slideshow of screenshots taken after every step — a visual replay of what the test did: +Generates a slideshow of screenshots taken after every step — a visual replay of what the test did. Set `slides: true` on the `screenshot` plugin (with `on=step`): ```js plugins: { - stepByStepReport: { + screenshot: { enabled: true, + on: 'step', + slides: true, deleteSuccessful: true, // keep only failed tests fullPageScreenshots: true, } @@ -280,7 +285,7 @@ plugins: { ``` ```bash -npx codeceptjs run -p stepByStepReport +npx codeceptjs run -p screenshot:on=step;slides=true ``` After the run, open `output/records.html` to browse through the slideshows. diff --git a/docs/migration-4.md b/docs/migration-4.md index 145428755..bda2baa7c 100644 --- a/docs/migration-4.md +++ b/docs/migration-4.md @@ -187,10 +187,21 @@ plugins: { Inject `login` and call `login('admin')` — same as before. +### Renamed Plugins + +4.x unifies four plugins (`screenshot`, `pause`, `aiTrace`, `heal`) under a shared `on=` parameter. The old names live on as deprecated aliases that emit a warning and forward to the new plugin. + +| Old plugin | New plugin | Notes | +| :------------------ | :------------------------------- | :------------------------------------------------- | +| `screenshotOnFail` | `screenshot` | Default `on='fail'`, same behavior | +| `pauseOnFail` | `pause` | Default `on='fail'`, same behavior | +| `stepByStepReport` | `screenshot` with `slides: true` | Use `on=step` to capture every step | + ### New Plugins You Can Enable - **`aiTrace`** — captures failure traces (DOM, console, network, screenshots) for AI debugging. See [AI Trace](/aitrace). -- **`pauseOn`** — pauses execution on a chosen event or on failure. See [Debugging](/debugging). +- **`pause`** — pauses execution on a chosen event or on failure. See [Debugging](/debugging). +- **`heal`** — self-heals failing steps with AI; narrow with `on=file|url`. ## 5. Update Removed and Changed APIs @@ -455,13 +466,16 @@ I.fillField('input', 'x', step.opts({ elementIndex: -1 })) `-p` accepts colon-chained arguments, so plugins can be enabled and configured from the command line without editing config: ```bash -npx codeceptjs run -p pauseOn:fail # pause on every failure -npx codeceptjs run -p pauseOn:url:/checkout/* # pause when URL matches -npx codeceptjs run -p browser:show # force visible browser +npx codeceptjs run -p pause # pause on every failure +npx codeceptjs run -p pause:on=url:pattern=/checkout/* # pause when URL matches +npx codeceptjs run -p screenshot:on=step # screenshot every step +npx codeceptjs run -p browser:show # force visible browser npx codeceptjs run -p browser:browser=firefox:windowSize=1024x768 -npx codeceptjs run -p plugin1,plugin2:arg # multiple plugins +npx codeceptjs run -p plugin1,plugin2:arg # multiple plugins ``` +Each argument after the plugin name is a `key=value` pair. `:` separates pairs. `;` is an inline alternative for visually grouping related pairs (e.g. `path=...;line=...`). Reserved keys: `on`, `path`, `line`, `pattern`. + The `browser` plugin is new in 4.x — it overrides the active browser helper (Playwright, Puppeteer, WebDriver, Appium) from the CLI, useful for ad-hoc local runs and CI matrices. See [Commands](/commands). The old `-p all` magic keyword is gone (it conflicted with the colon syntax). Enable specific plugins explicitly: `-p pluginA,pluginB`. diff --git a/docs/plugins.md b/docs/plugins.md index bed98c8cc..21278f98a 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -595,47 +595,42 @@ Additional config options: * `config` (optional, default `{}`) -## pauseOn +## pause -Pauses test execution in different modes. Unlike `pauseOnFail`, this plugin supports -multiple triggers for pausing and is controlled via CLI arguments. +Pauses test execution interactively. Replaces the legacy `pauseOnFail` plugin. +Default `on=fail` matches the old `pauseOnFail` behavior. -Enable it via `-p` option with a mode: - - npx codeceptjs run -p pauseOn:fail - npx codeceptjs run -p pauseOn:step - npx codeceptjs run -p pauseOn:file:tests/login_test.js - npx codeceptjs run -p pauseOn:file:tests/login_test.js:43 - npx codeceptjs run -p pauseOn:url:/users/* - -#### Modes - -* **fail** — pause when a step fails (same as `pauseOnFail` plugin) -* **step** — pause before first step, use `next` to advance step-by-step -* **file** — pause when execution reaches a specific file (and optionally line) -* **url** — pause when the browser URL matches a pattern (supports `*` wildcards) +```js +plugins: { + pause: { + enabled: false, + on: 'fail', + } +} +``` -### Parameters +#### `on=` modes -* `config` (optional, default `{}`) +* **fail** — pause when a step fails (default) +* **test** — pause after each test +* **step** — pause before the first step (interactive walk-through) +* **file** — pause when execution reaches `path=...[;line=...]` +* **url** — pause when the current URL matches `pattern=...` -## pauseOnFail +CLI examples: -Automatically launches [interactive pause][5] when a test fails. + npx codeceptjs run -p pause + npx codeceptjs run -p pause:on=step + npx codeceptjs run -p pause:on=file:path=tests/login_test.js + npx codeceptjs run -p pause:on=file:path=tests/login_test.js;line=43 + npx codeceptjs run -p pause:on=url:pattern=/users/* -Useful for debugging flaky tests on local environment. -Add this plugin to config file: +> The legacy `pauseOnFail` plugin remains as a deprecated alias that emits a +> deprecation warning and forwards to `pause` with `on=fail`. -```js -plugins: { - pauseOnFail: {}, -} -``` - -Unlike other plugins, `pauseOnFail` is not recommended to be enabled by default. -Enable it manually on each run via `-p` option: +### Parameters - npx codeceptjs run -p pauseOnFail +* `config` (optional, default `{}`) ## retryFailedStep @@ -705,70 +700,60 @@ Scenario('scenario tite', { disableRetryFailedStep: true }, () => { * `config` -## screenshotOnFail - -Creates screenshot on failure. Screenshot is saved into `output` directory. +## screenshot -Initially this functionality was part of corresponding helper but has been moved into plugin since 1.4 +Saves screenshots from the browser at points triggered by `on=`. Replaces the +legacy `screenshotOnFail` plugin. Default `on=fail` preserves the old behavior. This plugin is **enabled by default**. -#### Configuration - -Configuration can either be taken from a corresponding helper (deprecated) or a from plugin config (recommended). - ```js plugins: { - screenshotOnFail: { - enabled: true - } + screenshot: { + enabled: true, + on: 'fail', + } } ``` -Possible config options: - -* `uniqueScreenshotNames`: use unique names for screenshot. Default: false. -* `fullPageScreenshots`: make full page screenshots. Default: false. - -### Parameters +#### `on=` modes -* `config` - -## stepByStepReport - -![step-by-step-report][6] +* **fail** — screenshot when a test fails (default) +* **test** — screenshot at the end of every test +* **step** — screenshot after every step +* **file** — screenshot for steps in `path=...[;line=...]` +* **url** — screenshot when the current URL matches `pattern=...` -Generates step by step report for a test. -After each step in a test a screenshot is created. After test executed screenshots are combined into slideshow. -By default, reports are generated only for failed tests. +CLI examples: -Run tests with plugin enabled: + npx codeceptjs run -p screenshot + npx codeceptjs run -p screenshot:on=step + npx codeceptjs run -p screenshot:on=step;slides=true + npx codeceptjs run -p screenshot:on=file:path=tests/login_test.js + npx codeceptjs run -p screenshot:on=url:pattern=/users/* - npx codeceptjs run --plugins stepByStepReport +Possible config options: -#### Configuration +* `uniqueScreenshotNames`: use unique names for screenshot. Default: false. +* `fullPageScreenshots`: make full page screenshots. Default: false. +* `slides`: generate a step-by-step slideshow report (works with `on=step|file|url`). Replaces the legacy `stepByStepReport` plugin. Default: false. +* `deleteSuccessful`: when `slides=true`, drop slideshow folders for passing tests. Default: true. +* `animateSlides`: when `slides=true`, animate transitions between slides. Default: true. +* `ignoreSteps`: when `slides=true`, RegExps of step names to skip in the slideshow. -```js -"plugins": { - "stepByStepReport": { - "enabled": true - } -} -``` +#### Step-by-step slideshow -Possible config options: +`screenshot:on=step;slides=true` writes a per-test `record_/index.html` +slideshow and a top-level `records.html` index. The output is a self-contained +HTML page with keyboard / dot navigation — no external assets, no JavaScript +frameworks. -* `deleteSuccessful`: do not save screenshots for successfully executed tests. Default: true. -* `animateSlides`: should animation for slides to be used. Default: true. -* `ignoreSteps`: steps to ignore in report. Array of RegExps is expected. Recommended to skip `grab*` and `wait*` steps. -* `fullPageScreenshots`: should full page screenshots be used. Default: false. -* `output`: a directory where reports should be stored. Default: `output`. -* `screenshotsForAllureReport`: If Allure plugin is enabled this plugin attaches each saved screenshot to allure report. Default: false. -* \`disableScreenshotOnFail : Disables the capturing of screeshots after the failed step. Default: true. +> The legacy `screenshotOnFail` plugin remains as a deprecated alias. The legacy +> `stepByStepReport` plugin has been replaced by `screenshot:slides=true`. ### Parameters -* `config` **any** +* `config` ## stepTimeout diff --git a/docs/tutorial.md b/docs/tutorial.md index f117628ac..13a44397a 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -153,13 +153,13 @@ Please note, that you shouldn't use a real credit card number here. Good news, y Run the test with next command: ``` -npx codeceptjs run --debug -p pauseOnFail +npx codeceptjs run --debug -p pause ``` What are special options here? * `--debug` flag is used to output additional information to the console, such as the details of each step in the test, the values of variables, and the results of test assertions. This can help you to identify and fix any issues in your tests. -* `-p pauseOnFail` option is also used to keep the browser opened even if a test fails. It will help us to identify to which point test was executed and what can be improved. +* `-p pause` option is also used to keep the browser opened even if a test fails (default `on=fail`). It will help us to identify to which point test was executed and what can be improved. Add more test steps if needed, update locators, and notify business owners that all that purchases are made by you so your collegues won't call you in the night asking when you want to get a coffee cup 😀 Also the good idea is to run tests on staging website, to not interfere with business process. diff --git a/examples/codecept.config.js b/examples/codecept.config.js index c59e980be..c6da729d7 100644 --- a/examples/codecept.config.js +++ b/examples/codecept.config.js @@ -50,7 +50,7 @@ export const config = { enabled: false, services: ['selenium-standalone'], }, - stepByStepReport: {}, + screenshot: { on: 'step', slides: true }, autoDelay: { enabled: false, }, diff --git a/lib/config.js b/lib/config.js index 0a33d24fd..7779c0cd3 100644 --- a/lib/config.js +++ b/lib/config.js @@ -15,8 +15,9 @@ const defaultConfig = { hooks: [], gherkin: {}, plugins: { - screenshotOnFail: { - enabled: true, // will be disabled by default in 2.0 + screenshot: { + enabled: true, + on: 'fail', }, }, stepTimeout: 0, diff --git a/lib/plugin/aiTrace.js b/lib/plugin/aiTrace.js index f2668afbc..e24f8db49 100644 --- a/lib/plugin/aiTrace.js +++ b/lib/plugin/aiTrace.js @@ -11,10 +11,17 @@ import event from '../event.js' import output from '../output.js' import { deleteDir, clearString } from '../utils.js' import colors from 'chalk' +import { + parsePluginArgs, + resolveTrigger, + matchStepFile, + matchUrl, +} from '../utils/pluginParser.js' const supportedHelpers = Container.STANDARD_ACTING_HELPERS const defaultConfig = { + on: 'step', deleteSuccessful: false, fullPageScreenshots: false, output: store.outputDir, @@ -53,13 +60,26 @@ const defaultConfig = { * * `captureHTTP`: capture HTTP requests (requires `trace` or `recordHar` enabled in helper config). Default: true. * * `captureDebugOutput`: capture CodeceptJS debug output. Default: true. * * `ignoreSteps`: steps to ignore in trace. Array of RegExps is expected. + * * `on`: trigger mode — `step` (default), `fail`, `test`, `file`, `url`. + * + * #### `on=` modes + * + * * **step** — persist every step (default) + * * **fail** — persist only the failed step + * * **test** — persist only the last step of each test + * * **file** — persist steps from `path=...[;line=...]` + * * **url** — persist when the current URL matches `pattern=...` * * @param {*} config */ -export default function (config) { +export default function (config = {}) { const helpers = Container.helpers() let helper + const cliArgs = parsePluginArgs(config._args) + const trigger = resolveTrigger(cliArgs, config, { on: defaultConfig.on }, { name: 'aiTrace' }) + if (!trigger) return + config = Object.assign(defaultConfig, config) for (const helperName of supportedHelpers) { @@ -141,6 +161,24 @@ export default function (config) { output.debug(`aiTrace: Skipping failed step "${step.toString()}" - already handled by step.failed event`) return } + + // on= filtering + if (trigger.on === 'fail') return // failed steps handled by step.failed + if (trigger.on === 'file' && !matchStepFile(step, trigger.path, trigger.line)) return + if (trigger.on === 'url') { + recorder.add('aiTrace:url check', async () => { + try { + if (!helper.grabCurrentUrl) return + const url = await helper.grabCurrentUrl() + if (!matchUrl(url, trigger.pattern)) return + await persistStep(step) + } catch (err) { + output.debug(`aiTrace: Error in url-mode step persistence: ${err.message}`) + } + }, true) + return + } + const stepPersistPromise = persistStep(step).catch(err => { output.debug(`aiTrace: Error saving step: ${err.message}`) }) @@ -361,6 +399,12 @@ export default function (config) { return } + // on=test: only render the last step in markdown; artifacts of earlier steps + // remain on disk unreferenced. + if (trigger.on === 'test') { + steps = steps.slice(-1) + } + const testDuration = ((Date.now() - testStartTime) / 1000).toFixed(2) let markdown = `file: ${test.file || 'unknown'}\n` diff --git a/lib/plugin/heal.js b/lib/plugin/heal.js index c5163c64d..c8237dcf4 100644 --- a/lib/plugin/heal.js +++ b/lib/plugin/heal.js @@ -10,21 +10,30 @@ import output from '../output.js' import healModule from '../heal.js' const heal = healModule.default || healModule import store from '../store.js' +import { + parsePluginArgs, + resolveTrigger, + matchStepFile, + matchUrl, + getBrowserHelper, +} from '../utils/pluginParser.js' const defaultConfig = { + on: 'fail', healLimit: 2, } /** * Self-healing tests with AI. * - * Read more about heaking in [Self-Healing Tests](https://codecept.io/heal/) + * Read more about healing in [Self-Healing Tests](https://codecept.io/heal/) * * ```js * plugins: { * heal: { * enabled: true, + * on: 'fail', * } * } * ``` @@ -32,7 +41,17 @@ const defaultConfig = { * More config options are available: * * * `healLimit` - how many steps can be healed in a single test (default: 2) + * * `on` - trigger mode. `fail` (default), `file` (filter to a path), `url` (filter to a URL pattern). * + * #### `on=` modes + * + * Heal always runs on step failures; `on=` narrows when it engages. + * + * * **fail** — heal any failing step (default) + * * **file** — heal only failures in `path=...[;line=...]` + * * **url** — heal only failures when the current URL matches `pattern=...` + * + * `on=step` and `on=test` are not supported and are rejected with an error. */ export default function (config = {}) { if (store.debugMode && !process.env.DEBUG) { @@ -42,6 +61,13 @@ export default function (config = {}) { return } + const cliArgs = parsePluginArgs(config._args) + const trigger = resolveTrigger(cliArgs, config, { on: defaultConfig.on }, { + name: 'heal', + validModes: ['fail', 'file', 'url'], + }) + if (!trigger) return + let currentTest = null let currentStep = null let healedSteps = 0 @@ -65,6 +91,8 @@ export default function (config = {}) { if (!heal.hasCorrespondingRecipes(step)) return + if (trigger.on === 'file' && !matchStepFile(step, trigger.path, trigger.line)) return + recorder.catchWithoutStop(async err => { isHealing = true if (caughtError === err) throw err // avoid double handling @@ -72,6 +100,21 @@ export default function (config = {}) { const test = currentTest + if (trigger.on === 'url') { + try { + const helper = getBrowserHelper() + const url = helper && helper.grabCurrentUrl ? await helper.grabCurrentUrl() : null + if (!matchUrl(url, trigger.pattern)) { + isHealing = false + throw err + } + } catch (e) { + if (e === err) throw e + isHealing = false + throw err + } + } + recorder.session.start('heal') debug('Self-healing started', step.toCode()) diff --git a/lib/plugin/pause.js b/lib/plugin/pause.js new file mode 100644 index 000000000..9bec1cfe7 --- /dev/null +++ b/lib/plugin/pause.js @@ -0,0 +1,131 @@ +import event from '../event.js' +import pause from '../pause.js' +import recorder from '../recorder.js' +import output from '../output.js' +import { + parsePluginArgs, + resolveTrigger, + matchStepFile, + matchUrl, + getBrowserHelper, +} from '../utils/pluginParser.js' + +/** + * Pauses test execution interactively. Replaces the legacy `pauseOnFail` + * plugin. The default `on=fail` matches the old `pauseOnFail` behavior. + * + * #### Configuration + * + * ```js + * plugins: { + * pause: { + * enabled: false, + * on: 'fail', + * } + * } + * ``` + * + * #### `on=` modes + * + * * **fail** — pause when a step fails (default) + * * **test** — pause after each test + * * **step** — pause before the first step (interactive walk-through) + * * **file** — pause when execution reaches `path=...[;line=...]` + * * **url** — pause when the browser URL matches `pattern=...` + * + * CLI examples: + * + * ``` + * npx codeceptjs run -p pause + * npx codeceptjs run -p pause:on=step + * npx codeceptjs run -p pause:on=file:path=tests/login_test.js;line=43 + * npx codeceptjs run -p pause:on=url:pattern=/users/* + * ``` + */ +export default function (config = {}) { + const cliArgs = parsePluginArgs(config._args) + const trigger = resolveTrigger(cliArgs, config, { on: 'fail' }, { name: 'pause' }) + if (!trigger) return + + switch (trigger.on) { + case 'fail': + return initFailMode() + case 'test': + return initTestMode() + case 'step': + return initStepMode() + case 'file': + return initFileMode(trigger.path, trigger.line) + case 'url': + return initUrlMode(trigger.pattern) + } +} + +function initFailMode() { + let failed = false + + event.dispatcher.on(event.test.started, () => { + failed = false + }) + + event.dispatcher.on(event.step.failed, () => { + failed = true + }) + + event.dispatcher.on(event.test.after, () => { + if (failed) pause() + }) +} + +function initTestMode() { + event.dispatcher.on(event.test.after, () => pause()) +} + +function initStepMode() { + let activated = false + + event.dispatcher.on(event.test.before, () => { + if (activated) return + activated = true + recorder.add('pause:step', () => pause()) + }) +} + +function initFileMode(targetPath, targetLine) { + let paused = false + + event.dispatcher.on(event.step.before, step => { + if (paused) return + if (!matchStepFile(step, targetPath, targetLine)) return + paused = true + recorder.add('pause:file', () => pause()) + }) +} + +function initUrlMode(pattern) { + const helper = getBrowserHelper() + + if (!helper) { + output.error('pause:on=url requires a browser helper (Playwright, WebDriver, Puppeteer, Appium)') + return + } + + let paused = false + + event.dispatcher.on(event.step.after, () => { + if (paused) return + + recorder.add('pause:url check', async () => { + if (paused) return + try { + const currentUrl = await helper.grabCurrentUrl() + if (matchUrl(currentUrl, pattern)) { + paused = true + return pause() + } + } catch (err) { + // page may not be loaded yet + } + }) + }) +} diff --git a/lib/plugin/pauseOn.js b/lib/plugin/pauseOn.js deleted file mode 100644 index dea25dea5..000000000 --- a/lib/plugin/pauseOn.js +++ /dev/null @@ -1,167 +0,0 @@ -import event from '../event.js' -import pause from '../pause.js' -import recorder from '../recorder.js' -import Container from '../container.js' -import output from '../output.js' - -const supportedHelpers = Container.STANDARD_ACTING_HELPERS - -/** - * Pauses test execution in different modes. Unlike `pauseOnFail`, this plugin supports - * multiple triggers for pausing and is controlled via CLI arguments. - * - * Enable it via `-p` option with a mode: - * - * ``` - * npx codeceptjs run -p pauseOn:fail - * npx codeceptjs run -p pauseOn:step - * npx codeceptjs run -p pauseOn:file:tests/login_test.js - * npx codeceptjs run -p pauseOn:file:tests/login_test.js:43 - * npx codeceptjs run -p pauseOn:url:/users/* - * ``` - * - * #### Modes - * - * * **fail** — pause when a step fails (same as `pauseOnFail` plugin) - * * **step** — pause before first step, use `next` to advance step-by-step - * * **file** — pause when execution reaches a specific file (and optionally line) - * * **url** — pause when the browser URL matches a pattern (supports `*` wildcards) - * - */ -export default function (config = {}) { - const args = config._args || [] - const mode = args[0] || 'fail' - - switch (mode) { - case 'fail': - return initFailMode() - case 'step': - return initStepMode() - case 'file': - return initFileMode(args.slice(1)) - case 'url': - return initUrlMode(args.slice(1)) - default: - output.error(`pauseOn: unknown mode "${mode}". Available: fail, step, file, url`) - } -} - -function initFailMode() { - let failed = false - - event.dispatcher.on(event.test.started, () => { - failed = false - }) - - event.dispatcher.on(event.step.failed, () => { - failed = true - }) - - event.dispatcher.on(event.test.after, () => { - if (failed) pause() - }) -} - -function initStepMode() { - let activated = false - - event.dispatcher.on(event.test.before, () => { - if (activated) return - activated = true - recorder.add('pauseOn:step', () => pause()) - }) -} - -function initFileMode(fileArgs) { - if (fileArgs.length === 0) { - output.error('pauseOn:file requires a path. Usage: -p pauseOn:file:[:]') - return - } - - const targetFile = fileArgs[0] - const targetLine = fileArgs[1] ? parseInt(fileArgs[1], 10) : null - let paused = false - - event.dispatcher.on(event.step.before, (step) => { - if (paused) return - - const stepLine = step.line() - if (!stepLine) return - - const match = parseStepLine(stepLine) - if (!match) return - - const fileMatches = match.file.includes(targetFile) || match.file.endsWith(targetFile) - if (!fileMatches) return - - if (targetLine !== null && match.line !== targetLine) return - - paused = true - recorder.add('pauseOn:file', () => pause()) - }) -} - -function initUrlMode(urlArgs) { - if (urlArgs.length === 0) { - output.error('pauseOn:url requires a pattern. Usage: -p pauseOn:url:') - return - } - - const urlPattern = urlArgs.join(':') - - const helpers = Container.helpers() - let helper = null - for (const helperName of supportedHelpers) { - if (Object.keys(helpers).indexOf(helperName) > -1) { - helper = helpers[helperName] - } - } - - if (!helper) { - output.error('pauseOn:url requires a browser helper (Playwright, WebDriver, Puppeteer, Appium)') - return - } - - const regex = patternToRegex(urlPattern) - let paused = false - - event.dispatcher.on(event.step.after, () => { - if (paused) return - - recorder.add('pauseOn:url check', async () => { - if (paused) return - try { - const currentUrl = await helper.grabCurrentUrl() - if (regex.test(currentUrl)) { - paused = true - return pause() - } - } catch (err) { - // page may not be loaded yet - } - }) - }) -} - -function parseStepLine(stepLine) { - let line = stepLine.trim() - if (line.startsWith('at ')) line = line.substring(3).trim() - - const lastColon = line.lastIndexOf(':') - if (lastColon < 0) return null - const secondLastColon = line.lastIndexOf(':', lastColon - 1) - if (secondLastColon < 0) return null - - const file = line.substring(0, secondLastColon) - const lineNum = parseInt(line.substring(secondLastColon + 1, lastColon), 10) - - if (isNaN(lineNum)) return null - - return { file, line: lineNum } -} - -function patternToRegex(pattern) { - const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, '\\$&') - const regexStr = escaped.replace(/\*/g, '.*') - return new RegExp(regexStr) -} diff --git a/lib/plugin/pauseOnFail.js b/lib/plugin/pauseOnFail.js index 36511ff1b..625b5a0bb 100644 --- a/lib/plugin/pauseOnFail.js +++ b/lib/plugin/pauseOnFail.js @@ -1,39 +1,15 @@ -import event from '../event.js' +import output from '../output.js' +import pause from './pause.js' -import pause from '../pause.js' +let warned = false /** - * Automatically launches [interactive pause](/basics/#pause) when a test fails. - * - * Useful for debugging flaky tests on local environment. - * Add this plugin to config file: - * - * ```js - * plugins: { - * pauseOnFail: {}, - * } - * ``` - * - * Unlike other plugins, `pauseOnFail` is not recommended to be enabled by default. - * Enable it manually on each run via `-p` option: - * - * ``` - * npx codeceptjs run -p pauseOnFail - * ``` - * + * @deprecated Use the `pause` plugin with `on: 'fail'` (the default). */ -export default function() { - let failed = false - - event.dispatcher.on(event.test.started, () => { - failed = false - }) - - event.dispatcher.on(event.step.failed, () => { - failed = true - }) - - event.dispatcher.on(event.test.after, () => { - if (failed) pause() - }) +export default function (config = {}) { + if (!warned) { + output.error('pauseOnFail is deprecated; use the `pause` plugin (default on=fail).') + warned = true + } + return pause({ ...config, on: 'fail' }) } diff --git a/lib/plugin/screenshot.js b/lib/plugin/screenshot.js new file mode 100644 index 000000000..ca58d7367 --- /dev/null +++ b/lib/plugin/screenshot.js @@ -0,0 +1,563 @@ +import crypto from 'crypto' +import fs from 'fs' +import path from 'path' +import { mkdirp } from 'mkdirp' + +import Container from '../container.js' +import recorder from '../recorder.js' +import event from '../event.js' +import output from '../output.js' +import store from '../store.js' + +import { fileExists, deleteDir, template } from '../utils.js' +import Codeceptjs from '../index.js' +import { testToFileName } from '../mocha/test.js' +import { + parsePluginArgs, + resolveTrigger, + matchStepFile, + matchUrl, + getBrowserHelper, +} from '../utils/pluginParser.js' + +const defaultConfig = { + on: 'fail', + slides: false, + uniqueScreenshotNames: false, + disableScreenshots: false, + fullPageScreenshots: false, + animateSlides: true, + deleteSuccessful: true, + ignoreSteps: [], +} + +/** + * Saves screenshots from the browser at points triggered by `on=`. + * + * Replaces the legacy `screenshotOnFail` plugin. Default `on=fail` preserves the + * old behavior (screenshot when a test fails). Pass `slides=true` (with `on=step`) + * to generate a step-by-step slideshow report — replaces the legacy + * `stepByStepReport` plugin. + * + * #### Configuration + * + * ```js + * plugins: { + * screenshot: { + * enabled: true, + * on: 'fail', + * } + * } + * ``` + * + * #### `on=` modes + * + * * **fail** — screenshot when a test fails (default) + * * **test** — screenshot at the end of every test + * * **step** — screenshot after every step + * * **file** — screenshot for steps in `path=...[;line=...]` + * * **url** — screenshot when the current browser URL matches `pattern=...` + * + * Other config options: + * + * * `uniqueScreenshotNames`: use unique names for screenshot. Default: false. + * * `fullPageScreenshots`: make full page screenshots. Default: false. + * * `disableScreenshots`: legacy switch to skip the plugin entirely. + * * `slides`: generate a step-by-step slideshow report (requires `on=step`). Default: false. + * * `deleteSuccessful`: when `slides=true`, drop slideshow directories of passing tests. Default: true. + * * `animateSlides`: when `slides=true`, animate transitions between slides. Default: true. + * * `ignoreSteps`: when `slides=true`, RegExps of step names to skip in the slideshow. + * + * CLI examples: + * + * ``` + * npx codeceptjs run -p screenshot + * npx codeceptjs run -p screenshot:on=step + * npx codeceptjs run -p screenshot:on=step;slides=true + * npx codeceptjs run -p screenshot:on=file:path=tests/login_test.js + * npx codeceptjs run -p screenshot:on=url:pattern=/users/* + * ``` + */ +export default function (config = {}) { + const helper = getBrowserHelper() + if (!helper) return + + const cliArgs = parsePluginArgs(config._args) + const trigger = resolveTrigger(cliArgs, config, { on: defaultConfig.on }, { name: 'screenshot' }) + if (!trigger) return + + const helpers = Container.helpers() + const options = Object.assign({}, defaultConfig, helper.options, config) + options.slides = cliArgs.slides ?? config.slides ?? defaultConfig.slides + + if (helpers.Mochawesome?.config) { + options.uniqueScreenshotNames = helpers.Mochawesome.config.uniqueScreenshotNames + } + + if (Codeceptjs.container.mocha()) { + options.reportDir = Codeceptjs.container.mocha()?.options?.reporterOptions + && Codeceptjs.container.mocha()?.options?.reporterOptions?.reportDir + } + + if (options.disableScreenshots) return + + if (options.slides) { + return wireSlides(options, trigger) + } + + switch (trigger.on) { + case 'fail': + return wireOnFail(options) + case 'test': + return wireOnTest(options) + case 'step': + return wireOnStep(options, () => true) + case 'file': + return wireOnStep(options, step => matchStepFile(step, trigger.path, trigger.line)) + case 'url': + return wireOnUrl(options, trigger.pattern) + } +} + +function wireOnFail(options) { + let currentTest = null + event.dispatcher.on(event.test.before, test => { + currentTest = test + }) + event.dispatcher.on(event.test.failed, (test, _err, hookName) => { + if (hookName === 'BeforeSuite' || hookName === 'AfterSuite') return + const t = test || currentTest + if (!t) return + scheduleScreenshot(t, suffix(t, options, 'failed'), options) + }) +} + +function wireOnTest(options) { + event.dispatcher.on(event.test.after, test => { + if (!test) return + scheduleScreenshot(test, suffix(test, options, 'test'), options) + }) +} + +function wireOnStep(options, filter) { + let currentTest = null + let stepCount = 0 + event.dispatcher.on(event.test.before, test => { + currentTest = test + stepCount = 0 + }) + event.dispatcher.on(event.step.after, step => { + if (!currentTest) return + if (!filter(step)) return + stepCount++ + const name = `${testToFileName(currentTest, { suffix: '', unique: options.uniqueScreenshotNames })}.step_${stepCount}.png` + scheduleScreenshot(currentTest, name, options) + }) +} + +function wireOnUrl(options, pattern) { + let currentTest = null + let stepCount = 0 + event.dispatcher.on(event.test.before, test => { + currentTest = test + stepCount = 0 + }) + event.dispatcher.on(event.step.after, () => { + if (!currentTest) return + const helper = getBrowserHelper() + if (!helper) return + recorder.add('screenshot:url check', async () => { + try { + const url = await helper.grabCurrentUrl() + if (!matchUrl(url, pattern)) return + stepCount++ + const name = `${testToFileName(currentTest, { suffix: '', unique: options.uniqueScreenshotNames })}.url_${stepCount}.png` + await takeScreenshot(currentTest, name, options) + } catch (err) { + // page may not be ready + } + }) + }) +} + +function suffix(test, options, kind) { + const base = testToFileName(test, { suffix: '', unique: options.uniqueScreenshotNames }) + return `${base}.${kind}.png` +} + +function scheduleScreenshot(test, fileName, options) { + recorder.add( + 'screenshot capture', + async () => takeScreenshot(test, fileName, options), + true, + ) +} + +async function takeScreenshot(test, fileName, options) { + const quietMode = !store.outputDir + if (!quietMode) { + output.plugin('screenshot', `Saving screenshot ${fileName}`) + } + + const helper = getBrowserHelper() + if (!helper || typeof helper.saveScreenshot !== 'function') return + + try { + if (options.reportDir) { + fileName = path.join(options.reportDir, fileName) + const mochaReportDir = path.resolve(process.cwd(), options.reportDir) + if (!fileExists(mochaReportDir)) fs.mkdirSync(mochaReportDir) + } + + if (helper.page && helper.page.isClosed && helper.page.isClosed()) { + throw new Error('Browser page has been closed') + } + if (helper.browser && helper.browser.isConnected && !helper.browser.isConnected()) { + throw new Error('Browser has been disconnected') + } + + const screenshotPromise = helper.saveScreenshot(fileName, options.fullPageScreenshots) + const timeoutPromise = new Promise((_, reject) => { + setTimeout(() => reject(new Error('Screenshot timeout after 5 seconds')), 5000) + }) + + await Promise.race([screenshotPromise, timeoutPromise]) + + if (!test.artifacts) test.artifacts = {} + const baseOutputDir = store.outputDir || null + if (baseOutputDir) { + test.artifacts.screenshot = path.join(baseOutputDir, fileName) + const mocha = Container.mocha() + const junit = mocha?.options?.reporterOptions?.['mocha-junit-reporter'] + if (junit?.options?.attachments) { + test.attachments = [path.join(baseOutputDir, fileName)] + } + } else { + test.artifacts.screenshot = fileName + } + } catch (err) { + if (!quietMode) { + output.plugin('screenshot', `Failed to save screenshot: ${err.message}`) + } + if ( + err + && ((err.message + && (err.message.includes('Target page, context or browser has been closed') + || err.message.includes('Browser page has been closed') + || err.message.includes('Browser has been disconnected') + || err.message.includes('was terminated due to') + || err.message.includes('no such window: target window already closed') + || err.message.includes('Screenshot timeout after'))) + || (err.type && err.type === 'RuntimeError')) + ) { + output.log(`Can't make screenshot, ${err.message}`) + helper.isRunning = false + } + } +} + +function wireSlides(options, trigger) { + const reportDir = options.output + ? path.resolve(store.codeceptDir, options.output) + : (store.outputDir || './_output') + + const stepFilter = makeStepFilter(trigger, options) + const recordedTests = {} + + let dir + let stepNum + let slides = {} + let savedStep = null + let currentTest = null + let scenarioFailed = false + + event.dispatcher.on(event.suite.before, () => { + stepNum = -1 + }) + + event.dispatcher.on(event.test.before, test => { + const hash = crypto.createHash('sha256').update(test.file + test.title).digest('hex') + dir = path.join(reportDir, `record_${hash}`) + mkdirp.sync(dir) + stepNum = 0 + slides = {} + savedStep = null + currentTest = test + scenarioFailed = false + }) + + event.dispatcher.on(event.step.failed, step => { + recorder.add('slides: failed step', async () => persistStep(step), true) + }) + + event.dispatcher.on(event.step.after, step => { + recorder.add('slides: step', async () => persistStep(step), true) + }) + + event.dispatcher.on(event.test.passed, test => { + if (options.deleteSuccessful) { + deleteDir(dir) + return + } + persist(test) + }) + + event.dispatcher.on(event.test.failed, (test, _err, hookName) => { + if (hookName === 'BeforeSuite' || hookName === 'AfterSuite') return + persist(test) + }) + + event.dispatcher.on(event.all.result, () => { + if (Object.keys(recordedTests).length === 0) return + writeIndex(reportDir, recordedTests) + }) + + if (event.workers && event.workers.result) { + event.dispatcher.on(event.workers.result, async () => { + await recorder.add(() => { + const tests = scanRecordDirs(reportDir) + if (Object.keys(tests).length) writeIndex(reportDir, tests) + }) + }) + } + + async function persistStep(step) { + if (stepNum === -1) return + if (savedStep === step) return + if (scenarioFailed) return + if (step.metaStep && step.metaStep.name === 'BeforeSuite') return + if (!currentTest) return + if (!stepFilter(step)) return + if (isStepIgnored(step, options.ignoreSteps)) return + + const fileName = `${String(stepNum).padStart(4, '0')}.png` + if (step.status === 'failed') scenarioFailed = true + stepNum++ + slides[fileName] = step + + const helper = getBrowserHelper() + if (!helper || typeof helper.saveScreenshot !== 'function') return + + try { + const screenshotPath = path.join(dir, fileName) + await helper.saveScreenshot(screenshotPath, options.fullPageScreenshots) + step.artifacts = step.artifacts || {} + step.artifacts.screenshot = screenshotPath + + currentTest.artifacts = currentTest.artifacts || {} + currentTest.artifacts.screenshots = currentTest.artifacts.screenshots || [] + currentTest.artifacts.screenshots.push(screenshotPath) + } catch (err) { + output.plugin('screenshot', `Can't save step screenshot: ${err.message}`) + } finally { + savedStep = step + } + } + + function persist(test) { + if (!Object.keys(slides).length) return + + const slideHtml = Object.keys(slides) + .sort() + .map((fileName, idx) => { + const step = slides[fileName] + const caption = step.toString().replace(/\[\d{2}m/g, '') + const failed = step.status === 'failed' ? ' is-failed' : '' + return template(SLIDE_TEMPLATE, { + image: fileName, + caption, + index: idx + 1, + activeClass: idx === 0 ? ' is-active' : '', + failed, + }) + }) + .join('') + + const dotHtml = Object.keys(slides) + .map((_, idx) => ``) + .join('') + + const html = template(SLIDESHOW_TEMPLATE, { + title: test.title, + feature: (test.parent && test.parent.title) || '', + slides: slideHtml, + dots: dotHtml, + animate: options.animateSlides ? 'true' : 'false', + }) + + const indexFile = path.join(dir, 'index.html') + fs.writeFileSync(indexFile, html) + recordedTests[`${(test.parent && test.parent.title) || ''}: ${test.title}`] = path.relative(reportDir, indexFile) + } +} + +function makeStepFilter(trigger, options) { + if (trigger.on === 'file' && trigger.path) { + return step => matchStepFile(step, trigger.path, trigger.line) + } + if (trigger.on === 'fail') { + return step => step.status === 'failed' + } + return () => true +} + +function isStepIgnored(step, patterns) { + if (!patterns || !patterns.length) return false + for (const pattern of patterns) { + if (step.name && step.name.match(pattern)) return true + } + return false +} + +function scanRecordDirs(reportDir) { + const out = {} + try { + for (const item of fs.readdirSync(reportDir, { withFileTypes: true })) { + if (!item.isDirectory() || !item.name.startsWith('record_')) continue + const indexFile = path.join(reportDir, item.name, 'index.html') + if (!fs.existsSync(indexFile)) continue + const html = fs.readFileSync(indexFile, 'utf-8') + const titleMatch = html.match(/([^<]*)<\/title>/) + const label = titleMatch ? titleMatch[1].replace(/^Slides — /, '') : item.name + out[label] = `${item.name}/index.html` + } + } catch (err) { + // ignore + } + return out +} + +function writeIndex(reportDir, recordedTests) { + const items = Object.entries(recordedTests) + .map(([name, href]) => `<li><a href="${href}">${escapeHtml(name)}</a></li>`) + .join('\n') + + const html = template(INDEX_TEMPLATE, { + time: new Date().toString(), + records: items, + }) + + const indexPath = path.join(reportDir, 'records.html') + fs.writeFileSync(indexPath, html) + output.print(`Step-by-step preview: file://${indexPath}`) +} + +function escapeHtml(s) { + return String(s).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c])) +} + +const SLIDE_TEMPLATE = ` +<figure class="slides__slide{{activeClass}}{{failed}}" data-index="{{index}}"> + <img class="slides__image" src="{{image}}" alt=""> + <figcaption class="slides__caption"> + <span class="slides__step">{{index}}</span> + <span class="slides__text">{{caption}}</span> + </figcaption> +</figure> +` + +const SLIDESHOW_TEMPLATE = `<!DOCTYPE html> +<html lang="en"> +<head> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<title>Slides — {{feature}}: {{title}} + + + +
+ « back + {{feature}} + {{title}} +
+
+ {{slides}} + + +
+ +

Use ← / → to navigate, click sides of image, or use the dots below.

+ + + +` + +const INDEX_TEMPLATE = ` + + + + +Step-by-step Reports + + + +

Step-by-step Reports

+
{{time}}
+
    + {{records}} +
+ + +` diff --git a/lib/plugin/screenshotOnFail.js b/lib/plugin/screenshotOnFail.js index 57c181d0f..eae177f7b 100644 --- a/lib/plugin/screenshotOnFail.js +++ b/lib/plugin/screenshotOnFail.js @@ -1,177 +1,15 @@ -import fs from 'fs' -import path from 'path' - -import Container from '../container.js' - -import recorder from '../recorder.js' - -import event from '../event.js' - import output from '../output.js' -import store from '../store.js' - -import { fileExists } from '../utils.js' -import Codeceptjs from '../index.js' -import { testToFileName } from '../mocha/test.js' - -const defaultConfig = { - uniqueScreenshotNames: false, - disableScreenshots: false, - fullPageScreenshots: false, -} +import screenshot from './screenshot.js' -const supportedHelpers = Container.STANDARD_ACTING_HELPERS +let warned = false /** - * Creates screenshot on failure. Screenshot is saved into `output` directory. - * - * Initially this functionality was part of corresponding helper but has been moved into plugin since 1.4 - * - * This plugin is **enabled by default**. - * - * #### Configuration - * - * Configuration can either be taken from a corresponding helper (deprecated) or a from plugin config (recommended). - * - * ```js - * plugins: { - * screenshotOnFail: { - * enabled: true - * } - * } - * ``` - * - * Possible config options: - * - * * `uniqueScreenshotNames`: use unique names for screenshot. Default: false. - * * `fullPageScreenshots`: make full page screenshots. Default: false. - * - * + * @deprecated Use the `screenshot` plugin with `on: 'fail'` (the default). */ -export default function (config) { - const helpers = Container.helpers() - let helper - - for (const helperName of supportedHelpers) { - if (Object.keys(helpers).indexOf(helperName) > -1) { - helper = helpers[helperName] - } - } - - if (!helper) return // no helpers for screenshot - - const options = Object.assign(defaultConfig, helper.options, config) - - if (helpers.Mochawesome) { - if (helpers.Mochawesome.config) { - options.uniqueScreenshotNames = helpers.Mochawesome.config.uniqueScreenshotNames - } - } - - if (Codeceptjs.container.mocha()) { - options.reportDir = Codeceptjs.container.mocha()?.options?.reporterOptions && Codeceptjs.container.mocha()?.options?.reporterOptions?.reportDir - } - - if (options.disableScreenshots) { - // old version of disabling screenshots - return +export default function (config = {}) { + if (!warned) { + output.error('screenshotOnFail is deprecated; use the `screenshot` plugin (default on=fail).') + warned = true } - - event.dispatcher.on(event.test.failed, (test, _err, hookName) => { - if (hookName === 'BeforeSuite' || hookName === 'AfterSuite') { - // no browser here - return - } - - recorder.add( - 'screenshot of failed test', - async () => { - const dataType = 'image/png' - // This prevents data driven to be included in the failed screenshot file name - let fileName - - if (options.uniqueScreenshotNames && test) { - fileName = `${testToFileName(test, { suffix: '', unique: true })}.failed.png` - } else { - fileName = `${testToFileName(test, { suffix: '', unique: false })}.failed.png` - } - const quietMode = !store.outputDir - if (!quietMode) { - output.plugin('screenshotOnFail', 'Test failed, try to save a screenshot') - } - - // Re-check helpers at runtime in case they weren't ready during plugin init - const runtimeHelpers = Container.helpers() - let runtimeHelper = null - for (const helperName of supportedHelpers) { - if (Object.keys(runtimeHelpers).indexOf(helperName) > -1) { - runtimeHelper = runtimeHelpers[helperName] - break - } - } - - if (runtimeHelper && typeof runtimeHelper.saveScreenshot === 'function') { - helper = runtimeHelper - } - - try { - if (options.reportDir) { - fileName = path.join(options.reportDir, fileName) - const mochaReportDir = path.resolve(process.cwd(), options.reportDir) - if (!fileExists(mochaReportDir)) { - fs.mkdirSync(mochaReportDir) - } - } - - // Check if browser/page is still available before attempting screenshot - if (helper.page && helper.page.isClosed && helper.page.isClosed()) { - throw new Error('Browser page has been closed') - } - if (helper.browser && helper.browser.isConnected && !helper.browser.isConnected()) { - throw new Error('Browser has been disconnected') - } - - // Add timeout wrapper to prevent hanging with shorter timeout for ESM - const screenshotPromise = helper.saveScreenshot(fileName, options.fullPageScreenshots) - const timeoutPromise = new Promise((_, reject) => { - setTimeout(() => reject(new Error('Screenshot timeout after 5 seconds')), 5000) - }) - - await Promise.race([screenshotPromise, timeoutPromise]) - - if (!test.artifacts) test.artifacts = {} - const baseOutputDir = store.outputDir || null - if (baseOutputDir) { - test.artifacts.screenshot = path.join(baseOutputDir, fileName) - if (Container.mocha().options.reporterOptions['mocha-junit-reporter'] && Container.mocha().options.reporterOptions['mocha-junit-reporter'].options.attachments) { - test.attachments = [path.join(baseOutputDir, fileName)] - } - } else { - // Fallback: just store the file name to keep tests stable without triggering path errors - test.artifacts.screenshot = fileName - } - } catch (err) { - if (!quietMode) { - output.plugin('screenshotOnFail', `Failed to save screenshot: ${err.message}`) - } - // Enhanced error handling for browser closed scenarios - if ( - err && - ((err.message && - (err.message.includes('Target page, context or browser has been closed') || - err.message.includes('Browser page has been closed') || - err.message.includes('Browser has been disconnected') || - err.message.includes('was terminated due to') || - err.message.includes('no such window: target window already closed') || - err.message.includes('Screenshot timeout after'))) || - (err.type && err.type === 'RuntimeError')) - ) { - output.log(`Can't make screenshot, ${err.message}`) - helper.isRunning = false - } - } - }, - true, - ) - }) + return screenshot({ ...config, on: 'fail' }) } diff --git a/lib/plugin/stepByStepReport.js b/lib/plugin/stepByStepReport.js deleted file mode 100644 index 390f7a635..000000000 --- a/lib/plugin/stepByStepReport.js +++ /dev/null @@ -1,432 +0,0 @@ -import colors from 'chalk' -import crypto from 'crypto' -import figures from 'figures' -import fs from 'fs' -import { mkdirp } from 'mkdirp' -import path from 'path' -import * as cheerio from 'cheerio' - -import store from '../store.js' -import Container from '../container.js' -import recorder from '../recorder.js' -import event from '../event.js' -import output from '../output.js' -import { template, deleteDir } from '../utils.js' - -const supportedHelpers = Container.STANDARD_ACTING_HELPERS - -const defaultConfig = { - deleteSuccessful: true, - animateSlides: true, - ignoreSteps: [], - fullPageScreenshots: false, - output: store.outputDir, - screenshotsForAllureReport: false, - disableScreenshotOnFail: true, -} - -const templates = {} - -/** - * ![step-by-step-report](https://codecept.io/img/codeceptjs-slideshow.gif) - * - * Generates step by step report for a test. - * After each step in a test a screenshot is created. After test executed screenshots are combined into slideshow. - * By default, reports are generated only for failed tests. - * - * - * Run tests with plugin enabled: - * - * ``` - * npx codeceptjs run --plugins stepByStepReport - * ``` - * - * #### Configuration - * - * ```js - * "plugins": { - * "stepByStepReport": { - * "enabled": true - * } - * } - * ``` - * - * Possible config options: - * - * * `deleteSuccessful`: do not save screenshots for successfully executed tests. Default: true. - * * `animateSlides`: should animation for slides to be used. Default: true. - * * `ignoreSteps`: steps to ignore in report. Array of RegExps is expected. Recommended to skip `grab*` and `wait*` steps. - * * `fullPageScreenshots`: should full page screenshots be used. Default: false. - * * `output`: a directory where reports should be stored. Default: `output`. - * * `screenshotsForAllureReport`: If Allure plugin is enabled this plugin attaches each saved screenshot to allure report. Default: false. - * * `disableScreenshotOnFail : Disables the capturing of screeshots after the failed step. Default: true. - * - * @param {*} config - */ - -export default function (config) { - const helpers = Container.helpers() - let helper - - config = Object.assign(defaultConfig, config) - - for (const helperName of supportedHelpers) { - if (Object.keys(helpers).indexOf(helperName) > -1) { - helper = helpers[helperName] - } - } - - if (!helper) return // no helpers for screenshot - - let dir - let stepNum - let slides = {} - let error - let savedStep = null - let currentTest = null - let scenarioFailed = false - - const recordedTests = {} - const pad = '0000' - const reportDir = config.output ? path.resolve(store.codeceptDir, config.output) : defaultConfig.output - - event.dispatcher.on(event.suite.before, suite => { - stepNum = -1 - }) - - event.dispatcher.on(event.test.before, test => { - const sha256hash = crypto - .createHash('sha256') - .update(test.file + test.title) - .digest('hex') - dir = path.join(reportDir, `record_${sha256hash}`) - mkdirp.sync(dir) - stepNum = 0 - error = null - slides = {} - savedStep = null - currentTest = test - }) - - event.dispatcher.on(event.step.failed, step => { - recorder.add('screenshot of failed test', async () => persistStep(step), true) - }) - - event.dispatcher.on(event.step.after, step => { - recorder.add('screenshot of step of test', async () => persistStep(step), true) - }) - - event.dispatcher.on(event.test.passed, test => { - if (!config.deleteSuccessful) return persist(test) - // cleanup - deleteDir(dir) - }) - - event.dispatcher.on(event.test.failed, (test, _err, hookName) => { - if (hookName === 'BeforeSuite' || hookName === 'AfterSuite') { - // no browser here - return - } - - persist(test) - }) - - event.dispatcher.on(event.all.result, () => { - if (Object.keys(recordedTests).length === 0 || !Object.keys(slides).length) return - generateRecordsHtml(recordedTests) - }) - - event.dispatcher.on(event.workers.result, async () => { - await recorder.add(() => { - const recordedTests = getRecordFoldersWithDetails(reportDir) - generateRecordsHtml(recordedTests) - }) - }) - - function getRecordFoldersWithDetails(dirPath) { - let results = {} - - try { - const items = fs.readdirSync(dirPath, { withFileTypes: true }) - - items.forEach(item => { - if (item.isDirectory() && item.name.startsWith('record_')) { - const recordFolderPath = path.join(dirPath, item.name) - const indexPath = path.join(recordFolderPath, 'index.html') - - let name = '' - if (fs.existsSync(indexPath)) { - try { - const htmlContent = fs.readFileSync(indexPath, 'utf-8') - const $ = cheerio.load(htmlContent) - name = $('.navbar-brand').text().trim() - } catch (err) { - console.error(`Error reading index.html in ${recordFolderPath}:`, err.message) - } - } - - results[name || 'Unkown'] = `${item.name}/index.html` - } - }) - } catch (err) { - console.error(`Error reading directory ${dirPath}:`, err.message) - } - - return results - } - - function generateRecordsHtml(recordedTests) { - let links = '' - - for (const link in recordedTests) { - links += `
  • ${link}
    • \n` - } - - const indexHTML = template(templates.index, { - time: Date().toString(), - records: links, - }) - - fs.writeFileSync(path.join(reportDir, 'records.html'), indexHTML) - - output.print(`${figures.circleFilled} Step-by-step preview: ${colors.white.bold(`file://${reportDir}/records.html`)}`) - } - - async function persistStep(step) { - if (stepNum === -1) return // Ignore steps from BeforeSuite function - if (isStepIgnored(step)) return - if (savedStep === step) return // already saved - // Ignore steps from BeforeSuite function - if (scenarioFailed && config.disableScreenshotOnFail) return - if (step.metaStep && step.metaStep.name === 'BeforeSuite') return - if (!currentTest) return // Ignore steps from AfterSuite - - const fileName = `${pad.substring(0, pad.length - stepNum.toString().length) + stepNum.toString()}.png` - if (step.status === 'failed') { - scenarioFailed = true - } - stepNum++ - slides[fileName] = step - try { - const screenshotPath = path.join(dir, fileName) - await helper.saveScreenshot(screenshotPath, config.fullPageScreenshots) - - step.artifacts = step.artifacts || {} - step.artifacts.screenshot = screenshotPath - } catch (err) { - output.plugin(`Can't save step screenshot: ${err}`) - error = err - return - } finally { - savedStep = step - } - - if (!currentTest.artifacts.screenshots) currentTest.artifacts.screenshots = [] - // added attachments to test - currentTest.artifacts.screenshots.push(path.join(dir, fileName)) - - const allureReporter = Container.plugins('allure') - if (allureReporter && config.screenshotsForAllureReport) { - output.plugin('stepByStepReport', 'Adding screenshot to Allure') - allureReporter.addAttachment(`Screenshot of step ${step}`, fs.readFileSync(path.join(dir, fileName)), 'image/png') - } - } - - function persist(test) { - if (error) return - - let indicatorHtml = '' - let slideHtml = '' - - for (const i in slides) { - const step = slides[i] - const stepNum = parseInt(i, 10) - indicatorHtml += template(templates.indicator, { - step: stepNum, - isActive: stepNum ? '' : 'class="active"', - }) - - slideHtml += template(templates.slides, { - image: i, - caption: step.toString().replace(/\[\d{2}m/g, ''), // remove ANSI escape sequence - isActive: stepNum ? '' : 'active', - isError: step.status === 'failed' ? 'error' : '', - }) - } - - const html = template(templates.global, { - indicators: indicatorHtml, - slides: slideHtml, - feature: test.parent && test.parent.title, - test: test.title, - carousel_class: config.animateSlides ? ' slide' : '', - }) - - const index = path.join(dir, 'index.html') - fs.writeFileSync(index, html) - recordedTests[`${test.parent.title}: ${test.title}`] = path.relative(reportDir, index) - } - - function isStepIgnored(step) { - if (!config.ignoreSteps) return - for (const pattern of config.ignoreSteps || []) { - if (step.name.match(pattern)) return true - } - return false - } -} - -templates.slides = ` -
    -
    - -
    -
    -

    {{caption}}

    - scroll up and down to see the full page -
    -
    -` - -templates.indicator = ` -
  • -` - -templates.index = ` - - - - - - Step by Steps Report - - - - - - -
    -

    Recorded @ {{time}}

    -
      - {{records}} -
    -
    - - - -` - -templates.global = ` - - - - - - Recorder Result - - - - - - - - - -
    - -
      - {{indicators}} -
    - - -
    - {{slides}} -
    - - - - - - - - - -
    - - - - - - - - - - - -` diff --git a/lib/utils/pluginParser.js b/lib/utils/pluginParser.js new file mode 100644 index 000000000..8bcf8a838 --- /dev/null +++ b/lib/utils/pluginParser.js @@ -0,0 +1,151 @@ +import Container from '../container.js' +import output from '../output.js' + +const supportedHelpers = Container.STANDARD_ACTING_HELPERS + +const RESERVED_KEYS = new Set(['on', 'path', 'line', 'pattern']) +const ALL_MODES = ['fail', 'test', 'step', 'file', 'url'] + +/** + * Parse a plugin's _args (from CLI `-p plugin:key=value:key=value`) into a flat dict. + * Each entry is split on `;` then on the first `=`. Bare segments become `{ key: true }`. + * + * Examples: + * parsePluginArgs(['on=fail']) + * → { on: 'fail' } + * parsePluginArgs(['on=file', 'path=tests/foo.js;line=43']) + * → { on: 'file', path: 'tests/foo.js', line: '43' } + * parsePluginArgs(['on=file', 'path=tests/foo.js', 'line=43']) + * → { on: 'file', path: 'tests/foo.js', line: '43' } + * parsePluginArgs(['show']) + * → { show: true } + */ +export function parsePluginArgs(args = []) { + const opts = {} + for (const arg of args) { + if (!arg) continue + for (const segment of arg.split(';')) { + if (!segment) continue + if (segment.includes('=')) { + const eq = segment.indexOf('=') + const key = segment.slice(0, eq) + const value = segment.slice(eq + 1) + opts[key] = coerce(value) + } else { + opts[segment] = true + } + } + } + return opts +} + +function coerce(v) { + if (v === 'true') return true + if (v === 'false') return false + return v +} + +/** + * Compose CLI args > config > defaults into a normalized trigger spec, then + * validate it. Returns `{ on, path, line, pattern, ...rest }` with `line` + * coerced to a number, or `null` if validation failed (an error is printed). + * + * @param {object} cliArgs — output of parsePluginArgs(config._args) + * @param {object} config — full plugin config object + * @param {object} defaults — fallback values, e.g. `{ on: 'fail' }` + * @param {object} options + * @param {string} options.name — plugin name, used in error messages + * @param {string[]} [options.validModes] — accepted values for `on` + * (default: fail, test, step, file, url) + */ +export function resolveTrigger(cliArgs = {}, config = {}, defaults = {}, options = {}) { + const { name = 'plugin', validModes = ALL_MODES } = options + const merged = { ...defaults, ...pickKnown(config), ...cliArgs } + if (merged.line != null) merged.line = parseInt(merged.line, 10) + + const valid = new Set(validModes) + if (!valid.has(merged.on)) { + output.error(`${name}: unknown on="${merged.on}". Valid: ${validModes.join(', ')}`) + return null + } + if (merged.on === 'file' && !merged.path) { + output.error(`${name}:on=file requires path=. Example: -p ${name}:on=file:path=tests/foo.js`) + return null + } + if (merged.on === 'url' && !merged.pattern) { + output.error(`${name}:on=url requires pattern=. Example: -p ${name}:on=url:pattern=/users/*`) + return null + } + + return merged +} + +function pickKnown(config) { + const out = {} + for (const key of Object.keys(config || {})) { + if (RESERVED_KEYS.has(key)) out[key] = config[key] + } + return out +} + +/** + * Match a step's source location against a `path` (substring/suffix) and optional `line`. + * Reads the step's stack via `step.line()` to get `file:row:col`. + */ +export function matchStepFile(step, targetPath, targetLine) { + if (!targetPath) return false + const stepLine = step.line && step.line() + if (!stepLine) return false + + const parsed = parseStepLine(stepLine) + if (!parsed) return false + + const fileMatches = parsed.file.includes(targetPath) || parsed.file.endsWith(targetPath) + if (!fileMatches) return false + + if (targetLine != null && !Number.isNaN(targetLine) && parsed.line !== targetLine) return false + return true +} + +function parseStepLine(stepLine) { + let line = stepLine.trim() + if (line.startsWith('at ')) line = line.substring(3).trim() + + const lastColon = line.lastIndexOf(':') + if (lastColon < 0) return null + const secondLastColon = line.lastIndexOf(':', lastColon - 1) + if (secondLastColon < 0) return null + + const file = line.substring(0, secondLastColon) + const lineNum = parseInt(line.substring(secondLastColon + 1, lastColon), 10) + + if (Number.isNaN(lineNum)) return null + return { file, line: lineNum } +} + +/** + * Match a URL string against a glob-style pattern (supports `*` wildcards). + */ +export function matchUrl(currentUrl, pattern) { + if (!pattern || !currentUrl) return false + return patternToRegex(pattern).test(currentUrl) +} + +function patternToRegex(pattern) { + const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, '\\$&') + const regexStr = escaped.replace(/\*/g, '.*') + return new RegExp(regexStr) +} + +/** + * Return the first available standard browser helper, or null. + */ +export function getBrowserHelper() { + const helpers = Container.helpers() + for (const name of supportedHelpers) { + if (Object.keys(helpers).indexOf(name) > -1) { + return helpers[name] + } + } + return null +} diff --git a/test/acceptance/codecept.Playwright.coverage.js b/test/acceptance/codecept.Playwright.coverage.js index 3fbb21074..f189d75ae 100644 --- a/test/acceptance/codecept.Playwright.coverage.js +++ b/test/acceptance/codecept.Playwright.coverage.js @@ -32,9 +32,6 @@ export const config = { bootstrap: installCodeceptjs, mocha: {}, plugins: { - screenshotOnFail: { - enabled: true, - }, coverage: { enabled: true, debug: true, diff --git a/test/acceptance/codecept.Playwright.js b/test/acceptance/codecept.Playwright.js index c190315c3..dedbfe19f 100644 --- a/test/acceptance/codecept.Playwright.js +++ b/test/acceptance/codecept.Playwright.js @@ -35,9 +35,6 @@ export const config = { bootstrap: installCodeceptjs, mocha: {}, plugins: { - screenshotOnFail: { - enabled: true, - }, retryFailedStep: { enabled: true, }, diff --git a/test/acceptance/codecept.Puppeteer.js b/test/acceptance/codecept.Puppeteer.js index a821c4fb3..b21abdea4 100644 --- a/test/acceptance/codecept.Puppeteer.js +++ b/test/acceptance/codecept.Puppeteer.js @@ -24,11 +24,6 @@ export const config = { include: {}, bootstrap: installCodeceptjs, mocha: {}, - plugins: { - screenshotOnFail: { - enabled: true, - }, - }, name: 'acceptance', gherkin: { features: './gherkin/*.feature', diff --git a/test/acceptance/codecept.WebDriver.js b/test/acceptance/codecept.WebDriver.js index 8a9950619..019011f29 100644 --- a/test/acceptance/codecept.WebDriver.js +++ b/test/acceptance/codecept.WebDriver.js @@ -33,11 +33,6 @@ export const config = { }, mocha: {}, name: 'acceptance', - plugins: { - screenshotOnFail: { - enabled: true, - }, - }, gherkin: { features: './gherkin/*.feature', steps: ['./gherkin/steps.js'], diff --git a/test/acceptance/codecept.els.Playwright.js b/test/acceptance/codecept.els.Playwright.js index 3e45755b0..d80fd351d 100644 --- a/test/acceptance/codecept.els.Playwright.js +++ b/test/acceptance/codecept.els.Playwright.js @@ -22,10 +22,5 @@ export const config = { include: {}, bootstrap: false, mocha: {}, - plugins: { - screenshotOnFail: { - enabled: true, - }, - }, name: 'acceptance', } diff --git a/test/bdd/codecept.faker.js b/test/bdd/codecept.faker.js index be1b1b688..d587e1669 100644 --- a/test/bdd/codecept.faker.js +++ b/test/bdd/codecept.faker.js @@ -20,9 +20,6 @@ export const config = { bootstrap: false, mocha: {}, plugins: { - screenshotOnFail: { - enabled: true, - }, fakerTransform: { enabled: true, }, diff --git a/test/plugin/plugin_test.js b/test/plugin/plugin_test.js index 02020824c..2f5d41890 100644 --- a/test/plugin/plugin_test.js +++ b/test/plugin/plugin_test.js @@ -31,7 +31,7 @@ describe('CodeceptJS plugin', function () { try { // Check that the coverage plugin is loaded and starts attempting coverage collection - expect(lines).toEqual(expect.arrayContaining([expect.stringContaining('Plugins: screenshotOnFail, coverage'), expect.stringContaining('writing output/coverage')])) + expect(lines).toEqual(expect.arrayContaining([expect.stringContaining('Plugins: screenshot, coverage'), expect.stringContaining('writing output/coverage')])) // Test should pass regardless of whether coverage data is found (depends on external site) // The test may fail due to network issues or browser problems, but the coverage plugin should still initialize