Umbraco-CMS/tests/Umbraco.Tests.AcceptanceTest

Umbraco Acceptance Tests

End-to-end acceptance tests for Umbraco CMS using Playwright.

You can watch a video following these instructions here and a longer UmbraCollab recording here. Make sure to use the latest recommended main branch rather than v10 that's mentioned in the video.

---

Prerequisites

• Node.js 22+
• A running installed Umbraco instance on URL: https://localhost:44339 (default development port)
  • Install using SqlServer/LocalDb as the tests execute too fast for SQLite to handle

---

Getting Started

1. Navigate to the test project folder:

   cd tests/Umbraco.Tests.AcceptanceTest
   

2. Install dependencies and Playwright browsers:

   npm ci
   npx playwright install
   

3. The setup script will prompt you to enter credentials for a superadmin user of your Umbraco CMS.

---

Executing Tests

Available NPM Scripts

Command	Description
npm run test	Execute DefaultConfig tests headlessly
npm run ui	Open Playwright UI mode with browser
npm run smokeTest	Run quick smoke tests (@smoke tagged)
npm run releaseTest	Run comprehensive release tests (@release tagged)
npm run all	Run all test suites
npm run testSqlite	Run tests excluding User tests (SQLite limitation)
npm run testWindows	Run tests excluding RelationType tests
npm run createTest <name>	Generate a new test file template
npm run config	Reconfigure environment settings

Running Single Tests

Run a specific test file:

npx playwright test tests/DefaultConfig/Content/Content.spec.ts

Run a single test by name:

npx playwright test -g "can create content with the document link"

Run tests with visible browser:

npx playwright test --headed tests/DefaultConfig/Content/Content.spec.ts

UI Mode

For an interactive testing experience with step-by-step visualization:

npx playwright test --ui

Or specify a test directory:

npx playwright test --ui tests/DefaultConfig

Note

: In UI mode, if you only see the authenticate test, click on 'Projects' and select 'defaultConfig' to see all tests.

---

Test Helpers and Fixtures

Tests use the @umbraco/playwright-testhelpers package which provides three main fixtures:

umbracoUi - UI Interaction Helper

For browser interactions organized by section:

// Navigation
await umbracoUi.goToBackOffice();
await umbracoUi.content.goToSection(ConstantHelper.sections.content);

// Interactions
await umbracoUi.content.enterContentName('My Content');
await umbracoUi.content.clickSaveButton();
await umbracoUi.content.clickSaveAndPublishButton();

// Assertions
await umbracoUi.content.isSuccessStateVisibleForSaveButton();
await umbracoUi.content.doesSuccessNotificationHaveText('Content saved');

umbracoApi - REST API Helper

For server-side operations (setup/teardown):

// Create test data
const docTypeId = await umbracoApi.documentType.createDefaultDocumentType('TestType');
const dataTypeId = await umbracoApi.dataType.createTextstringDataType('TestDataType');

// Query data
const exists = await umbracoApi.document.doesNameExist('MyContent');
const data = await umbracoApi.document.getByName('MyContent');

// Cleanup (idempotent - won't fail if not exists)
await umbracoApi.documentType.ensureNameNotExists('TestType');

// Publishing
await umbracoApi.document.publish(documentId);

page - Raw Playwright Page

Direct access to Playwright's Page object for custom interactions:

await page.pause();  // Pause for debugging
await page.screenshot({ path: 'debug.png' });

Helper Constants

import { ConstantHelper, NotificationConstantHelper, AliasHelper } from '@umbraco/playwright-testhelpers';

// Section names
ConstantHelper.sections.content
ConstantHelper.sections.media
ConstantHelper.sections.settings

// Notification messages
NotificationConstantHelper.success.published
NotificationConstantHelper.success.saved

// String utilities
AliasHelper.toAlias('Test Document Type')  // → 'testDocumentType'

---

Writing Tests

Test Structure (AAA Pattern)

All tests follow the Arrange-Act-Assert pattern:

import { ConstantHelper, test } from '@umbraco/playwright-testhelpers';
import { expect } from '@playwright/test';

const documentTypeName = 'TestDocumentType';
const contentName = 'TestContent';

test.beforeEach(async ({ umbracoApi, umbracoUi }) => {
  // Clean up any existing test data (idempotent)
  await umbracoApi.documentType.ensureNameNotExists(documentTypeName);
  await umbracoApi.document.ensureNameNotExists(contentName);

  // Navigate to backoffice
  await umbracoUi.goToBackOffice();
});

test.afterEach(async ({ umbracoApi }) => {
  // Always clean up after tests
  await umbracoApi.documentType.ensureNameNotExists(documentTypeName);
  await umbracoApi.document.ensureNameNotExists(contentName);
});

test('can create content', { tag: '@smoke' }, async ({ umbracoApi, umbracoUi }) => {
  // Arrange - Setup test data via API
  const documentTypeId = await umbracoApi.documentType.createDefaultDocumentType(documentTypeName);

  // Act - Perform UI actions
  await umbracoUi.content.goToSection(ConstantHelper.sections.content);
  await umbracoUi.content.clickActionsMenuAtRoot();
  await umbracoUi.content.clickCreateButton();
  await umbracoUi.content.chooseDocumentType(documentTypeName);
  await umbracoUi.content.enterContentName(contentName);
  await umbracoUi.content.clickSaveButton();

  // Assert - Verify results
  await umbracoUi.content.isSuccessStateVisibleForSaveButton();
  expect(await umbracoApi.document.doesNameExist(contentName)).toBeTruthy();
});

Test Tags

Tag tests for selective execution:

test('critical path test', { tag: '@smoke' }, async ({ umbracoApi, umbracoUi }) => {
  // Quick smoke test
});

test('comprehensive test', { tag: '@release' }, async ({ umbracoApi, umbracoUi }) => {
  // Full release validation
});

Creating a New Test

Use the generator script:

npm run createTest MyFeatureName

This creates tests/MyFeatureName.spec.ts with a template.

Test Conventions

1. Idempotent cleanup: Use ensureNameNotExists() instead of delete() - won't fail if item doesn't exist
2. API for setup: Create test data via API (faster than UI)
3. UI for validation: Test actual user workflows through the UI
4. Test independence: Each test should run standalone without depending on other tests
5. Descriptive names: Use clear, descriptive test and variable names
6. Clean up: Always clean up test data in afterEach

---

Environment Configuration

The environment configuration is set up by the npm installation script, creating a .env file (git-ignored):

UMBRACO_USER_LOGIN=email@example.com
UMBRACO_USER_PASSWORD=yourpassword
URL=https://localhost:44339

To reconfigure:

npm run config

---

Debugging Tests

Pause Execution

await page.pause();  // Opens Playwright Inspector

Take Screenshots

await page.screenshot({ path: 'debug.png' });

View Traces

Failed tests automatically save traces. View them with:

npx playwright show-trace results/trace.zip

Run in Debug Mode

PWDEBUG=1 npx playwright test tests/DefaultConfig/Content/Content.spec.ts

---

Test Projects

The test suite is organized into multiple Playwright projects (see playwright.config.ts):

Project	Description
setup	Authentication setup (runs first)
defaultConfig	Main test suite (depends on setup)
extensionRegistry	Extension registry tests
entityDataPicker	Entity data picker tests
deliveryApi	Delivery API tests
externalLoginAzureADB2C	Azure AD B2C authentication tests
unattendedInstallConfig	Installation tests (no auth required)
smtp	Email/SMTP tests

---

Configuration Details

Key settings in playwright.config.ts:

• Test timeout: 30 seconds per test
• Expect timeout: 5 seconds for assertions
• Retries: 2 retries on CI (0 locally)
• Workers: 1 (sequential execution for state consistency)
• Browser: Desktop Chrome with HTTPS
• Test identifier: data-mark attribute

---

Documentation

• Playwright Documentation
• Umbraco Documentation
• @umbraco/playwright-testhelpers
• @umbraco/json-models-builders