yv01p/Umbraco-CMS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

V16 QA cherry picked acceptance tests with different configuration (#20106)

Browse Source 
* Added appsettings

* Added test setup for different config

* Added appsettings for external login

* Added acceptance tests

* Updated pipelines

* Updated solution file

* V15 QA Added external login provider tests and split pipeline into templates (#20049)

* Added setup for external login

* Started on yaml

* Added test file

* Updated pipeline

* Use env vars

* Added env variables and commented out test we don't need to run

* Removed list from matrix

* Updated condition

* Updated package path

* Updated testFolder

* double slash

* Updated condition

* Updated condition again

* Added port

* Removed redundant values

* Set as env vars

* Added env vars beneath matrix

* Get env

* Updated naming

* Updated usage of values

* Added a check for client id, to see if value set

* Moved env out of pool

* Tried moving env

* Trying to fix the env being empty

* Removed env

* Updated name of variable

* Fixed A cyclical reference

* Updated typo

* More logging

* Reverted change

* Added env

* Added env to tests

* Cleaned up

* Added yaml template files

* Updated nighly pipepline to use templates

* Updated sln

* Split yaml into templates for e2e setup

* Updated pipeline

* Updated solution file

* Set value

* Added if statement

* Added variables

* Set default values

* Updated values

* Updated condition

* Run multiple tests

* Added env

* Updated from parameter to variable

* Fixed condition

* Fixed condition to use actual value

* Updated npx wait on command

* Updated pwsh

* Updated port again

* Updated port value

* Updated wait on

* Updated condition

* Restructured

* Updated var

* Updated run application steps

* Added echo

* Updated to boolean

* Updated conditions

* Updated test template usage

* Added databaseType

* Added another databaseType

* Split up templates

* Fixed indentation

* Updated condition

* updated path

* removed build from path

* Updated conditions for azureAd

* Fixed indentation

* Updated to single qoutes

* Cleaned up

* Removed unused file

* Clarified namin

* Moved

* Updated pipeline, not done

* Updated locator

* Updated pipelines

* Updated test helpers package

* Skipped build stage for default app settings tests

* Updated password var

* Updated locators

* Updated defaultconfig build setup

* Split E2E stage in two

* Added parameter for skipping integration tests

* Cleaned up

* Added ASPNETCORE_URLS

* V15 QA acceptance tests with appsettings (#19550)

* Start of appsetting

* Updated setup of playwright

* Adjusted the pipeline

* Updated appsetting

* Added install test

* Added comments

* Updated pipeline

* Updated development app settings

* Commented tests out

* comment

* Added if statement

* Updated pipeline

* Fixed condition

* Changed to production

* Added a log

* Updated copy item

* Added

* Updated app settings

* Updated pipeline

* Moved playwright login

* Updated pipeline

* Updated app setting

* Updated nightly

* Updated appsettings

* Updated get

* Updated wait on

* Updated appsettings

* Updated connection string

* Updates

* Skips code

* Updated variable

* Updated pipeline

* We want to always retain the trace, to see if the test runs as expected on the pipeline

* Added a temporary wait till port is open

* Fixed condition

* Added missing tcp for wait on

* Updated URL env

* Updated setup

* Fixed string

* Updated locator

* Split tests into SQLite and SQLServer

* Updated pipeline to run all tests

* Retain trace on failure

* Added testFolder var

* Added appsettings and program for delivery api tests

* Updated playwright config

* Split test runners into defaultconfig and different app settings

* Added delivery api tests

* Cleaned up tests

* Bumped version

* Updated pipeline

* Small fixes

* Added password

* Updated connection string

* Fixed

* Removed quotes

* Removed unnecessary connection string

* Added missing password

* Cleaned up

* Cleaned up

* Cleaned up

* Updated to use helpers

* Bumped version

* Updated helper usage

* Added password to variables and a condition

* Added check

* Indented value

* Fixed condition

* More updates

* Updated variable

* Removed settings

* Updated delivery api tests

* Bumped version

* Updated test

* Removed unnecessary variables

* Updates based on copilot comments

* Fixed merge conflict

* Fixed env creation step

* Bumped version

* Updated tests to use new helper

* Updated helper

* Updated to string

* Moved logic to conditions

* bumped version

* Use new name for helper

* Remove echo

* Added variable
This commit is contained in:
Andreas Zerbst
2025-09-18 10:58:54 +02:00
committed by GitHub
parent fd0ccc529b
commit f23050d5c6
28 changed files with 1357 additions and 725 deletions
Download Patch File Download Diff File Expand all files Collapse all files

506
build/azure-pipelines.yml
View File

@@ -522,160 +522,72 @@ stages:
# Connection string
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=Umbraco;Mode=Memory;Cache=Shared;Foreign Keys=True;Pooling=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.Sqlite
DatabaseType: SQLite
additionalEnvironmentVariables: false
strategy:
matrix:
LinuxPart1Of3:
vmImage: "ubuntu-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTestSqlite -- --shard=1/3"
LinuxPart2Of3:
vmImage: "ubuntu-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTestSqlite -- --shard=2/3"
LinuxPart3Of3:
vmImage: "ubuntu-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTestSqlite -- --shard=3/3"
WindowsPart1Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTestSqlite -- --shard=1/3"
WindowsPart2Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTestSqlite -- --shard=2/3"
WindowsPart3Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTestSqlite -- --shard=3/3"
pool:
vmImage: $(vmImage)
steps:
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download NuGet artifacts
inputs:
artifact: nupkg
path: $(Agent.BuildDirectory)/app/nupkg
- task: NodeTool@0
displayName: Use Node.js $(nodeVersion)
retryCountOnTaskFailure: 3
inputs:
versionSpec: $(nodeVersion)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
# Setup test environment Template
- template: nightly-E2E-setup-template.yml
parameters:
nodeVersion: ${{ variables.nodeVersion }}
PlaywrightUserEmail: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL }}
PlaywrightPassword: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD }}
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
npm_config_cache: ${{ variables.npm_config_cache }}
- pwsh: |
"UMBRACO_USER_LOGIN=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL)
UMBRACO_USER_PASSWORD=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
URL=$(ASPNETCORE_URLS)
STORAGE_STAGE_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/playwright/.auth/user.json
CONSOLE_ERRORS_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/console-errors.json" | Out-File .env
displayName: Generate .env
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
# Cache and restore NPM packages
- task: Cache@2
displayName: Cache NPM packages
inputs:
key: 'npm_e2e | "$(Agent.OS)" | $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/package-lock.json'
restoreKeys: |
npm_e2e | "$(Agent.OS)"
npm_e2e
path: $(npm_config_cache)
- script: npm ci --no-fund --no-audit --prefer-offline
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
displayName: Restore NPM packages
# Build application
- pwsh: |
$cmsVersion = "$(Build.BuildNumber)" -replace "\+",".g"
dotnet new nugetconfig
dotnet nuget add source ./nupkg --name Local
dotnet new install Umbraco.Templates::$cmsVersion
dotnet new umbraco --name UmbracoProject --version $cmsVersion --exclude-gitignore --no-restore --no-update-check
dotnet restore UmbracoProject
cp $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest.UmbracoProject/*.cs UmbracoProject
dotnet build UmbracoProject --configuration $(buildConfiguration) --no-restore
displayName: Restore project
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
dotnet build UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-restore
dotnet dev-certs https
displayName: Build application
workingDirectory: $(Agent.BuildDirectory)/app
condition: succeeded()
# Run application
- bash: |
nohup dotnet run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
echo "##vso[task.setvariable variable=AcceptanceTestProcessId]$!"
displayName: Run application (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
workingDirectory: $(Agent.BuildDirectory)/app
# Run application Template
- template: nightly-E2E-run-application-template.yml
parameters:
DatabaseType: ${{ variables.DatabaseType }}
buildConfiguration: ${{ variables.buildConfiguration }}
additionalEnvironmentVariables: ${{ variables.additionalEnvironmentVariables }}
- pwsh: |
$process = Start-Process dotnet "run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
Write-Host "##vso[task.setvariable variable=AcceptanceTestProcessId]$($process.Id)"
displayName: Run application (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
workingDirectory: $(Agent.BuildDirectory)/app
# Wait for application to start responding to requests
- pwsh: npx wait-on -v --interval 1000 --timeout 120000 $(ASPNETCORE_URLS)
displayName: Wait for application
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Install Playwright and dependencies
- pwsh: npx playwright install chromium
displayName: Install Playwright only with Chromium browser
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Test
- pwsh: $(testCommand)
displayName: Run Playwright tests
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
env:
CI: true
CommitId: $(Build.SourceVersion)
AgentOs: $(Agent.OS)
# Stop application
- bash: kill -15 $(AcceptanceTestProcessId)
displayName: Stop application (Linux)
condition: and(ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Linux'))
- pwsh: Stop-Process -Id $(AcceptanceTestProcessId)
displayName: Stop application (Windows)
condition: and(ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Windows_NT'))
# Copy artifacts
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/results/*) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/results/* $(Build.ArtifactStagingDirectory) -Recurse
}
displayName: Copy Playwright results
condition: succeededOrFailed()
# Copy console error log
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/console-errors.json) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/console-errors.json $(Build.ArtifactStagingDirectory)
}
displayName: Copy console error log
condition: succeededOrFailed()
# Publish test artifacts
- task: PublishPipelineArtifact@1
displayName: Publish test artifacts
condition: succeededOrFailed()
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: "Acceptance Test Results - $(Agent.JobName) - Attempt #$(System.JobAttempt)"
# Publish test results
- task: PublishTestResults@2
displayName: "Publish test results"
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: "tests/Umbraco.Tests.AcceptanceTest/results"
testRunTitle: "$(Agent.JobName)"
# Run tests Template
- template: nightly-E2E-run-tests-template.yml
parameters:
testCommand: $(testCommand)
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
DatabaseType: ${{ variables.DatabaseType }}
- job:
displayName: E2E Smoke Tests (SQL Server)
@@ -683,354 +595,78 @@ stages:
# Connection string
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=(localdb)\MSSQLLocalDB;AttachDbFilename=|DataDirectory|\Umbraco.mdf;Integrated Security=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.SqlClient
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
DatabaseType: SQLServer
additionalEnvironmentVariables: false
strategy:
matrix:
${{ if eq(parameters.sqlServerLinuxAcceptanceTests, True) }}:
LinuxPart1Of3:
testCommand: "npm run smokeTest -- --shard=1/3"
vmImage: "ubuntu-latest"
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
testFolder: "DefaultConfig"
CONNECTIONSTRINGS__UMBRACODBDSN: "Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);TrustServerCertificate=True"
LinuxPart2Of3:
testCommand: "npm run smokeTest -- --shard=2/3"
vmImage: "ubuntu-latest"
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
testFolder: "DefaultConfig"
CONNECTIONSTRINGS__UMBRACODBDSN: "Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);TrustServerCertificate=True"
LinuxPart3Of3:
testCommand: "npm run smokeTest -- --shard=3/3"
vmImage: "ubuntu-latest"
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
testFolder: "DefaultConfig"
CONNECTIONSTRINGS__UMBRACODBDSN: "Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);TrustServerCertificate=True"
WindowsPart1Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTest -- --shard=1/3"
WindowsPart2Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTest -- --shard=2/3"
WindowsPart3Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run smokeTest -- --shard=3/3"
pool:
vmImage: $(vmImage)
steps:
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download NuGet artifacts
inputs:
artifact: nupkg
path: $(Agent.BuildDirectory)/app/nupkg
- task: NodeTool@0
displayName: Use Node.js $(nodeVersion)
inputs:
versionSpec: $(nodeVersion)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
# Setup test environment Template
- template: nightly-E2E-setup-template.yml
parameters:
nodeVersion: ${{ variables.nodeVersion }}
PlaywrightUserEmail: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL }}
PlaywrightPassword: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD }}
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
npm_config_cache: ${{ variables.npm_config_cache }}
- pwsh: |
"UMBRACO_USER_LOGIN=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL)
UMBRACO_USER_PASSWORD=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
URL=$(ASPNETCORE_URLS)
STORAGE_STAGE_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/playwright/.auth/user.json
CONSOLE_ERRORS_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/console-errors.json" | Out-File .env
displayName: Generate .env
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
# Cache and restore NPM packages
- task: Cache@2
displayName: Cache NPM packages
inputs:
key: 'npm_e2e | "$(Agent.OS)" | $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/package-lock.json'
restoreKeys: |
npm_e2e | "$(Agent.OS)"
npm_e2e
path: $(npm_config_cache)
- script: npm ci --no-fund --no-audit --prefer-offline
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
displayName: Restore NPM packages
# Build application
- pwsh: |
$cmsVersion = "$(Build.BuildNumber)" -replace "\+",".g"
dotnet new nugetconfig
dotnet nuget add source ./nupkg --name Local
dotnet new install Umbraco.Templates::$cmsVersion
dotnet new umbraco --name UmbracoProject --version $cmsVersion --exclude-gitignore --no-restore --no-update-check
dotnet restore UmbracoProject
cp $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest.UmbracoProject/*.cs UmbracoProject
dotnet build UmbracoProject --configuration $(buildConfiguration) --no-restore
displayName: Restore project
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
dotnet build UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-restore
dotnet dev-certs https
displayName: Build application
workingDirectory: $(Agent.BuildDirectory)/app
condition: succeeded()
# Start SQL Server
- powershell: docker run --name mssql -d -p 1433:1433 -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=$(SA_PASSWORD)" mcr.microsoft.com/mssql/server:2022-latest
displayName: Start SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
# Run application Template
- template: nightly-E2E-run-application-template.yml
parameters:
SA_PASSWORD: ${{ variables.SA_PASSWORD }}
buildConfiguration: ${{ variables.buildConfiguration }}
DatabaseType: ${{ variables.DatabaseType }}
additionalEnvironmentVariables: ${{ variables.additionalEnvironmentVariables }}
- pwsh: SqlLocalDB start MSSQLLocalDB
displayName: Start SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
# Run application
- bash: |
nohup dotnet run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
echo "##vso[task.setvariable variable=AcceptanceTestProcessId]$!"
displayName: Run application (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
$process = Start-Process dotnet "run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
Write-Host "##vso[task.setvariable variable=AcceptanceTestProcessId]$($process.Id)"
displayName: Run application (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
workingDirectory: $(Agent.BuildDirectory)/app
# Wait for application to start responding to requests
- pwsh: npx wait-on -v --interval 1000 --timeout 120000 $(ASPNETCORE_URLS)
displayName: Wait for application
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Install Playwright and dependencies
- pwsh: npx playwright install chromium
displayName: Install Playwright only with Chromium browser
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Test
- pwsh: $(testCommand)
displayName: Run Playwright tests
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
env:
CI: true
CommitId: $(Build.SourceVersion)
AgentOs: $(Agent.OS)
# Stop application
- bash: kill -15 $(AcceptanceTestProcessId)
displayName: Stop application (Linux)
condition: and(ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Linux'))
- pwsh: Stop-Process -Id $(AcceptanceTestProcessId)
displayName: Stop application (Windows)
condition: and(ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Windows_NT'))
# Stop SQL Server
- pwsh: docker stop mssql
displayName: Stop SQL Server Docker image (Linux)
condition: eq(variables['Agent.OS'], 'Linux')
- pwsh: SqlLocalDB stop MSSQLLocalDB
displayName: Stop SQL Server LocalDB (Windows)
condition: eq(variables['Agent.OS'], 'Windows_NT')
# Copy artifacts
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/results/*) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/results/* $(Build.ArtifactStagingDirectory) -Recurse
}
displayName: Copy Playwright results
condition: succeededOrFailed()
# Copy console error log
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/console-errors.json) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/console-errors.json $(Build.ArtifactStagingDirectory)
}
displayName: Copy console error log
condition: succeededOrFailed()
# Publish test artifacts
- task: PublishPipelineArtifact@1
displayName: Publish test artifacts
condition: succeededOrFailed()
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: "Acceptance Test Results - $(Agent.JobName) - Attempt #$(System.JobAttempt)"
# Publish test results
- task: PublishTestResults@2
displayName: "Publish test results"
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: "tests/Umbraco.Tests.AcceptanceTest/results"
testRunTitle: "$(Agent.JobName)"
- job:
displayName: E2E Release Tests (SQL Server)
variables:
# Connection string
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=(localdb)\MSSQLLocalDB;AttachDbFilename=|DataDirectory|\Umbraco.mdf;Integrated Security=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.SqlClient
condition: eq(dependencies.Build.outputs['A.build.NBGV_PublicRelease'], 'True')
strategy:
matrix:
WindowsPart1Of3:
vmImage: "windows-latest"
testCommand: "npm run releaseTest -- --shard=1/3"
WindowsPart2Of3:
vmImage: "windows-latest"
testCommand: "npm run releaseTest -- --shard=2/3"
WindowsPart3Of3:
vmImage: "windows-latest"
testCommand: "npm run releaseTest -- --shard=3/3"
pool:
vmImage: $(vmImage)
steps:
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download NuGet artifacts
inputs:
artifact: nupkg
path: $(Agent.BuildDirectory)/app/nupkg
- task: NodeTool@0
displayName: Use Node.js $(nodeVersion)
inputs:
versionSpec: $(nodeVersion)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
- pwsh: |
"UMBRACO_USER_LOGIN=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL)
UMBRACO_USER_PASSWORD=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
URL=$(ASPNETCORE_URLS)
STORAGE_STAGE_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/playwright/.auth/user.json
CONSOLE_ERRORS_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/console-errors.json" | Out-File .env
displayName: Generate .env
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
# Cache and restore NPM packages
- task: Cache@2
displayName: Cache NPM packages
inputs:
key: 'npm_e2e | "$(Agent.OS)" | $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/package-lock.json'
restoreKeys: |
npm_e2e | "$(Agent.OS)"
npm_e2e
path: $(npm_config_cache)
- script: npm ci --no-fund --no-audit --prefer-offline
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
displayName: Restore NPM packages
# Build application
- pwsh: |
$cmsVersion = "$(Build.BuildNumber)" -replace "\+",".g"
dotnet new nugetconfig
dotnet nuget add source ./nupkg --name Local
dotnet new install Umbraco.Templates::$cmsVersion
dotnet new umbraco --name UmbracoProject --version $cmsVersion --exclude-gitignore --no-restore --no-update-check
dotnet restore UmbracoProject
cp $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest.UmbracoProject/*.cs UmbracoProject
dotnet build UmbracoProject --configuration $(buildConfiguration) --no-restore
dotnet dev-certs https
displayName: Build application
workingDirectory: $(Agent.BuildDirectory)/app
# Start SQL Server
- powershell: docker run --name mssql -d -p 1433:1433 -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=$(SA_PASSWORD)" mcr.microsoft.com/mssql/server:2022-latest
displayName: Start SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
- pwsh: SqlLocalDB start MSSQLLocalDB
displayName: Start SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
# Run application
- bash: |
nohup dotnet run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
echo "##vso[task.setvariable variable=AcceptanceTestProcessId]$!"
displayName: Run application (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
$process = Start-Process dotnet "run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
Write-Host "##vso[task.setvariable variable=AcceptanceTestProcessId]$($process.Id)"
displayName: Run application (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
workingDirectory: $(Agent.BuildDirectory)/app
# Wait for application to start responding to requests
- pwsh: npx wait-on -v --interval 1000 --timeout 120000 $(ASPNETCORE_URLS)
displayName: Wait for application
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Install Playwright and dependencies
- pwsh: npx playwright install chromium
displayName: Install Playwright only with Chromium browser
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Test
- pwsh: $(testCommand)
displayName: Run Playwright tests
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
env:
CI: true
CommitId: $(Build.SourceVersion)
AgentOs: $(Agent.OS)
# Stop application
- bash: kill -15 $(AcceptanceTestProcessId)
displayName: Stop application (Linux)
condition: and(ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Linux'))
- pwsh: Stop-Process -Id $(AcceptanceTestProcessId)
displayName: Stop application (Windows)
condition: and(ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Windows_NT'))
# Stop SQL Server
- pwsh: docker stop mssql
displayName: Stop SQL Server Docker image (Linux)
condition: eq(variables['Agent.OS'], 'Linux')
- pwsh: SqlLocalDB stop MSSQLLocalDB
displayName: Stop SQL Server LocalDB (Windows)
condition: eq(variables['Agent.OS'], 'Windows_NT')
# Copy artifacts
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/results/*) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/results/* $(Build.ArtifactStagingDirectory) -Recurse
}
displayName: Copy Playwright results
condition: succeededOrFailed()
# Copy console error log
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/console-errors.json) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/console-errors.json $(Build.ArtifactStagingDirectory)
}
displayName: Copy console error log
condition: succeededOrFailed()
# Publish test artifacts
- task: PublishPipelineArtifact@1
displayName: Publish test artifacts
condition: succeededOrFailed()
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: "Acceptance Test Results - $(Agent.JobName) - Attempt #$(System.JobAttempt)"
# Publish test results
- task: PublishTestResults@2
displayName: "Publish test results"
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: "tests/Umbraco.Tests.AcceptanceTest/results"
testRunTitle: "$(Agent.JobName)"
# Run tests Template
- template: nightly-E2E-run-tests-template.yml
parameters:
testCommand: $(testCommand)
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
DatabaseType: ${{ variables.DatabaseType }}
###############################################
## Release
@@ -1237,4 +873,4 @@ stages:
storage: umbracoapidocs
ContainerName: "$web"
BlobPrefix: v$(umbracoMajorVersion)/ui-api
CleanTargetBeforeCopy: true
CleanTargetBeforeCopy: true

74
build/nightly-E2E-build-template.yml Normal file
View File

@@ -0,0 +1,74 @@
parameters:
- name: testFolder
type: string
default: ''
- name: buildConfiguration
type: string
default: ''
- name: additionalEnvironmentVariables
type: string
default: 'false'
steps:
- pwsh: |
dotnet restore UmbracoProject
cp $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest.UmbracoProject/*.cs UmbracoProject
displayName: Restore project
workingDirectory: $(Agent.BuildDirectory)/app
# Update application to use necessary app settings
- pwsh: |
$sourcePath = "$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/tests/${{ parameters.testFolder }}/AdditionalSetup"
$destinationPath = "UmbracoProject"
$jsonFiles = Get-ChildItem -Path $sourcePath -Filter "*.json"
if ($jsonFiles) {
$jsonFiles | ForEach-Object {
Write-Host "Copying: $($_.FullName)"
Copy-Item -Path $_.FullName -Destination $destinationPath -Force
}
} else {
Write-Host "No JSON files found."
}
displayName: Update application to use necessary app settings
workingDirectory: $(Agent.BuildDirectory)/app
# Update application to use necessary App_Plugins
- pwsh: |
$sourcePath = "$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/tests/${{ parameters.testFolder }}/AdditionalSetup"
$destinationPath = "UmbracoProject"
$appPluginsFolders = Get-ChildItem -Path $sourcePath -Directory -Filter "App_Plugins"
if ($appPluginsFolders) {
foreach ($folder in $appPluginsFolders) {
Write-Host "Copying folder: $($folder.FullName)"
Copy-Item -Path $folder.FullName -Destination $destinationPath -Recurse -Force
}
} else {
Write-Host "No App_Plugins found."
}
displayName: Update application to use necessary app plugins
workingDirectory: $(Agent.BuildDirectory)/app
# Update application to use necessary classes
- pwsh: |
$sourcePath = "$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/tests/${{ parameters.testFolder }}/AdditionalSetup"
$destinationPath = "UmbracoProject"
$csharpFiles = Get-ChildItem -Path $sourcePath -Filter "*.cs"
if ($csharpFiles) {
$csharpFiles | ForEach-Object {
Write-Host "Copying: $($_.FullName)"
Copy-Item -Path $_.FullName -Destination $destinationPath -Force
}
} else {
Write-Host "No C# files found."
}
displayName: Update application to use necessary classes
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
dotnet build UmbracoProject --configuration ${{ parameters.buildConfiguration }} --no-restore
dotnet dev-certs https
displayName: Build application
workingDirectory: $(Agent.BuildDirectory)/app
condition: and(succeeded(), eq(variables['additionalEnvironmentVariables'], 'false'))

45
build/nightly-E2E-run-application-template.yml Normal file
View File

@@ -0,0 +1,45 @@
parameters:
- name: SA_PASSWORD
type: string
default: ''
- name: buildConfiguration
type: string
default: ''
- name: additionalEnvironmentVariables
type: string
default: 'false'
- name: DatabaseType
type: string
default: ''
steps:
# Skips the SQLServer setup if the databaseType does not match
- ${{ if eq(parameters.DatabaseType, 'SQLServer') }}:
# Start SQL Server Linux
- powershell: docker run --name mssql -d -p 1433:1433 -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=${{ parameters.SA_PASSWORD }}" mcr.microsoft.com/mssql/server:2022-latest
displayName: Start SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
# Start SQL Server LocalDB Windows
- pwsh: SqlLocalDB start MSSQLLocalDB
displayName: Start SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
# Run application for Linux
- bash: |
nohup dotnet run --project UmbracoProject --configuration ${{ parameters.buildConfiguration }} --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
echo "##vso[task.setvariable variable=AcceptanceTestProcessId]$!"
displayName: Run application (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'), eq(variables['additionalEnvironmentVariables'], 'false'))
workingDirectory: $(Agent.BuildDirectory)/app
# Run application for Windows
- pwsh: |
$process = Start-Process dotnet "run --project UmbracoProject --configuration ${{ parameters.buildConfiguration }} --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
Write-Host "##vso[task.setvariable variable=AcceptanceTestProcessId]$($process.Id)"
displayName: Run application (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'), eq(variables['additionalEnvironmentVariables'], 'false'))
workingDirectory: $(Agent.BuildDirectory)/app

106
build/nightly-E2E-run-tests-template.yml Normal file
View File

@@ -0,0 +1,106 @@
parameters:
- name: ASPNETCORE_URLS
type: string
default: ''
- name: testCommand
type: string
default: ''
- name: port
type: string
default: ''
- name: AZUREB2CTESTUSEREMAIL
type: string
default: ''
- name: AZUREB2CTESTUSERPASSWORD
type: string
default: ''
- name: DatabaseType
type: string
default: ''
steps:
# Ensures we have the package wait-on installed
- pwsh: npm install wait-on
displayName: Install wait-on package
# Wait for either the port of the aspnetcore url
- pwsh: |
$Port = "${{ parameters.port }}"
$Url = "${{ parameters.ASPNETCORE_URLS }}"
if ($Port -ne "") {
Write-Host "Waiting on TCP port $Port"
npx wait-on -v --interval 1000 --timeout 120000 "tcp:$Port"
} else {
Write-Host "Waiting on URL $Url"
npx wait-on -v --interval 1000 --timeout 120000 "$Url"
}
displayName: Wait for application
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Install Playwright and dependencies
- pwsh: npx playwright install chromium
displayName: Install Playwright only with Chromium browser
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Test
- pwsh: ${{ parameters.testCommand }}
displayName: Run Playwright tests
continueOnError: true
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
env:
CI: true
CommitId: $(Build.SourceVersion)
AgentOs: $(Agent.OS)
AZUREADB2CTESTUSEREMAIL: ${{ parameters.AZUREB2CTESTUSEREMAIL }}
AZUREADB2CTESTUSERPASSWORD: ${{ parameters.AZUREB2CTESTUSERPASSWORD }}
# Stop application
- bash: kill -15 $(AcceptanceTestProcessId)
displayName: Stop application (Linux)
condition: and(succeededOrFailed(), ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Linux'))
- pwsh: Stop-Process -Id $(AcceptanceTestProcessId)
displayName: Stop application (Windows)
condition: and(succeededOrFailed(), ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Windows_NT'))
- ${{ if eq(parameters.DatabaseType, 'SQLServer') }}:
# Stop SQL Server
- pwsh: docker stop mssql
displayName: Stop SQL Server Docker image (Linux)
condition: and(succeededOrFailed(), eq(variables['Agent.OS'], 'Linux'))
- pwsh: SqlLocalDB stop MSSQLLocalDB
displayName: Stop SQL Server LocalDB (Windows)
condition: and(succeededOrFailed(), eq(variables['Agent.OS'], 'Windows_NT'))
# Copy artifacts
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/results/*) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/results/* $(Build.ArtifactStagingDirectory) -Recurse
}
displayName: Copy Playwright results
condition: succeededOrFailed()
# Publish
- task: PublishPipelineArtifact@1
displayName: Publish test artifacts
condition: succeededOrFailed()
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: "Acceptance Test Results - $(Agent.JobName) - Attempt #$(System.JobAttempt)"
# Publish test results
- task: PublishTestResults@2
displayName: "Publish test results"
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: "tests/Umbraco.Tests.AcceptanceTest/results"
testRunTitle: "$(Agent.JobName)"

70
build/nightly-E2E-setup-template.yml Normal file
View File

@@ -0,0 +1,70 @@
parameters:
- name: nodeVersion
type: string
default: ''
- name: PlaywrightUserEmail
type: string
default: ''
- name: PlaywrightPassword
type: string
default: ''
- name: ASPNETCORE_URLS
type: string
default: ''
- name: npm_config_cache
type: string
default: ''
steps:
- task: DownloadPipelineArtifact@2
displayName: Download NuGet artifacts
inputs:
artifact: nupkg
path: $(Agent.BuildDirectory)/app/nupkg
- task: NodeTool@0
displayName: Use Node.js $(nodeVersion)
inputs:
versionSpec: $(nodeVersion)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
- pwsh: |
"UMBRACO_USER_LOGIN=${{ parameters.PlaywrightUserEmail }}
UMBRACO_USER_PASSWORD=${{ parameters.PlaywrightPassword }}
URL=${{ parameters.ASPNETCORE_URLS }}
STORAGE_STAGE_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/playwright/.auth/user.json
CONSOLE_ERRORS_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/console-errors.json" | Out-File .env
displayName: Generate .env
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
# Cache and restore NPM packages
- task: Cache@2
displayName: Cache NPM packages
inputs:
key: 'npm_e2e | "$(Agent.OS)" | $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/package-lock.json'
restoreKeys: |
npm_e2e | "$(Agent.OS)"
npm_e2e
path: ${{ parameters.npm_config_cache }}
- script: npm ci --no-fund --no-audit --prefer-offline
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
displayName: Restore NPM packages
# Install Template
- pwsh: |
$cmsVersion = "$(Build.BuildNumber)" -replace "\+",".g"
dotnet new nugetconfig
dotnet nuget add source ./nupkg --name Local
dotnet new install Umbraco.Templates::$cmsVersion
dotnet new umbraco --name UmbracoProject --version $cmsVersion --exclude-gitignore --no-restore --no-update-check
displayName: Install Template
workingDirectory: $(Agent.BuildDirectory)/app

677
build/nightly-E2E-test-pipelines.yml
View File

@@ -12,9 +12,18 @@ schedules:
- main
parameters:
# Skipped due to DB locks
- name: sqliteAcceptanceTests
displayName: Run SQLite Acceptance Tests
- name: skipIntegrationTests
displayName: Skip integration tests
type: boolean
default: false
- name: differentAppSettingsAcceptanceTests
displayName: Run acceptance tests with different app settings
type: boolean
default: false
- name: skipDefaultConfigAcceptanceTests
displayName: Skip tests with DefaultConfig
type: boolean
default: false
@@ -100,8 +109,191 @@ stages:
targetPath: $(Build.ArtifactStagingDirectory)/npm
artifactName: npm
- stage: E2E
displayName: E2E Tests
- stage: Integration
displayName: Integration Tests
dependsOn: Build
condition: ${{ eq(parameters.skipIntegrationTests, false) }}
jobs:
# Integration Tests (SQLite)
- job:
timeoutInMinutes: 180
displayName: Integration Tests (SQLite)
strategy:
matrix:
# Windows:
# vmImage: 'windows-latest'
# We split the tests into 3 parts for each OS to reduce the time it takes to run them on the pipeline
LinuxPart1Of3:
vmImage: "ubuntu-latest"
# Filter tests that are part of the Umbraco.Infrastructure namespace but not part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure) & (FullyQualifiedName!~Umbraco.Infrastructure.Service)"
LinuxPart2Of3:
vmImage: "ubuntu-latest"
# Filter tests that are part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure.Service)"
LinuxPart3Of3:
vmImage: "ubuntu-latest"
# Filter tests that are not part of the Umbraco.Infrastructure namespace. So this will run all tests that are not part of the Umbraco.Infrastructure namespace
testFilter: "(FullyQualifiedName!~Umbraco.Infrastructure)"
macOSPart1Of3:
vmImage: "macOS-latest"
# Filter tests that are part of the Umbraco.Infrastructure namespace but not part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure) & (FullyQualifiedName!~Umbraco.Infrastructure.Service)"
macOSPart2Of3:
vmImage: "macOS-latest"
# Filter tests that are part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure.Service)"
macOSPart3Of3:
vmImage: "macOS-latest"
# Filter tests that are not part of the Umbraco.Infrastructure namespace.
testFilter: "(FullyQualifiedName!~Umbraco.Infrastructure)"
pool:
vmImage: $(vmImage)
variables:
Tests__Database__DatabaseType: "Sqlite"
steps:
- checkout: self
submodules: false
lfs: false,
fetchDepth: 1
fetchFilter: tree:0
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download build artifacts
inputs:
artifact: build_output
path: $(Build.SourcesDirectory)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
# Test
- task: DotNetCoreCLI@2
displayName: Run dotnet test
inputs:
command: test
projects: "tests/Umbraco.Tests.Integration/Umbraco.Tests.Integration.csproj"
testRunTitle: Integration Tests SQLite - $(Agent.OS)
arguments: '--filter "$(testFilter)" --configuration $(buildConfiguration) --no-build'
# Integration Tests (SQL Server)
- job:
timeoutInMinutes: 180
displayName: Integration Tests (SQL Server)
variables:
SA_PASSWORD: UmbracoAcceptance123!
strategy:
matrix:
# We split the tests into 3 parts for each OS to reduce the time it takes to run them on the pipeline
WindowsPart1Of3:
vmImage: "windows-latest"
Tests__Database__DatabaseType: LocalDb
Tests__Database__SQLServerMasterConnectionString: N/A
# Filter tests that are part of the Umbraco.Infrastructure namespace but not part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure) & (FullyQualifiedName!~Umbraco.Infrastructure.Service)"
WindowsPart2Of3:
vmImage: "windows-latest"
Tests__Database__DatabaseType: LocalDb
Tests__Database__SQLServerMasterConnectionString: N/A
# Filter tests that are part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure.Service)"
WindowsPart3Of3:
vmImage: "windows-latest"
Tests__Database__DatabaseType: LocalDb
Tests__Database__SQLServerMasterConnectionString: N/A
# Filter tests that are not part of the Umbraco.Infrastructure namespace. So this will run all tests that are not part of the Umbraco.Infrastructure namespace
testFilter: "(FullyQualifiedName!~Umbraco.Infrastructure)"
LinuxPart1Of3:
vmImage: "ubuntu-latest"
Tests__Database__DatabaseType: SqlServer
Tests__Database__SQLServerMasterConnectionString: "Server=(local);User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True"
# Filter tests that are part of the Umbraco.Infrastructure namespace but not part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure) & (FullyQualifiedName!~Umbraco.Infrastructure.Service)"
LinuxPart2Of3:
vmImage: "ubuntu-latest"
Tests__Database__DatabaseType: SqlServer
Tests__Database__SQLServerMasterConnectionString: "Server=(local);User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True"
# Filter tests that are part of the Umbraco.Infrastructure.Service namespace
testFilter: "(FullyQualifiedName~Umbraco.Infrastructure.Service)"
LinuxPart3Of3:
vmImage: "ubuntu-latest"
Tests__Database__DatabaseType: SqlServer
Tests__Database__SQLServerMasterConnectionString: "Server=(local);User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True"
# Filter tests that are not part of the Umbraco.Infrastructure namespace. So this will run all tests that are not part of the Umbraco.Infrastructure namespace
testFilter: "(FullyQualifiedName!~Umbraco.Infrastructure)"
pool:
vmImage: $(vmImage)
steps:
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download build artifacts
inputs:
artifact: build_output
path: $(Build.SourcesDirectory)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
# Start SQL Server
- powershell: docker run --name mssql -d -p 1433:1433 -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=$(SA_PASSWORD)" mcr.microsoft.com/mssql/server:2022-latest
displayName: Start SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
- powershell: |
$maxAttempts = 12
$attempt = 0
$status = ""
while (($status -ne 'running') -and ($attempt -lt $maxAttempts)) {
Start-Sleep -Seconds 5
# We use the docker inspect command to check the status of the container. If the container is not running, we wait 5 seconds and try again. And if reaches 12 attempts, we fail the build.
$status = docker inspect -f '{{.State.Status}}' mssql
if ($status -ne 'running') {
Write-Host "Waiting for SQL Server to be ready... Attempt $($attempt + 1)"
$attempt++
}
}
if ($status -eq 'running') {
Write-Host "SQL Server container is running"
docker ps -a
} else {
Write-Host "SQL Server did not become ready in time. Last known status: $status"
docker logs mssql
exit 1
}
displayName: Wait for SQL Server to be ready (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
- pwsh: SqlLocalDB start MSSQLLocalDB
displayName: Start SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
# Test
- task: DotNetCoreCLI@2
displayName: Run dotnet test
inputs:
command: test
projects: "tests/Umbraco.Tests.Integration/Umbraco.Tests.Integration.csproj"
testRunTitle: Integration Tests SQL Server - $(Agent.OS)
arguments: '--filter "$(testFilter)" --configuration $(buildConfiguration) --no-build'
# Stop SQL Server
- pwsh: docker stop mssql
displayName: Stop SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
- pwsh: SqlLocalDB stop MSSQLLocalDB
displayName: Stop SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
- stage: DefaultConfigE2E
displayName: Default Config E2E Tests
dependsOn: Build
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm_e2e
@@ -128,369 +320,304 @@ stages:
# E2E Tests
- job:
displayName: E2E Tests (SQLite)
condition: eq(${{parameters.sqliteAcceptanceTests}}, True)
timeoutInMinutes: 180
condition: ${{ eq(parameters.skipDefaultConfigAcceptanceTests, false) }}
variables:
# Connection string
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=Umbraco;Mode=Memory;Cache=Shared;Foreign Keys=True;Pooling=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.Sqlite
DatabaseType: SQLite
additionalEnvironmentVariables: false
strategy:
matrix:
LinuxPart1Of3:
vmImage: "ubuntu-latest"
testFolder: "DefaultConfig"
testCommand: "npm run testSqlite -- --shard=1/3"
LinuxPart2Of3:
vmImage: "ubuntu-latest"
testFolder: "DefaultConfig"
testCommand: "npm run testSqlite -- --shard=2/3"
LinuxPart3Of3:
vmImage: "ubuntu-latest"
testFolder: "DefaultConfig"
testCommand: "npm run testSqlite -- --shard=3/3"
WindowsPart1Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run testSqlite -- --shard=1/3"
WindowsPart2Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run testSqlite -- --shard=2/3"
WindowsPart3Of3:
vmImage: "windows-latest"
testFolder: "DefaultConfig"
testCommand: "npm run testSqlite -- --shard=3/3"
pool:
vmImage: $(vmImage)
steps:
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download NuGet artifacts
inputs:
artifact: nupkg
path: $(Agent.BuildDirectory)/app/nupkg
- task: NodeTool@0
displayName: Use Node.js $(nodeVersion)
retryCountOnTaskFailure: 3
inputs:
versionSpec: $(nodeVersion)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
# Setup test environment Template
- template: nightly-E2E-setup-template.yml
parameters:
nodeVersion: ${{ variables.nodeVersion }}
PlaywrightUserEmail: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL }}
PlaywrightPassword: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD }}
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
npm_config_cache: ${{ variables.npm_config_cache }}
- pwsh: |
"UMBRACO_USER_LOGIN=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL)
UMBRACO_USER_PASSWORD=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
URL=$(ASPNETCORE_URLS)
STORAGE_STAGE_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/playwright/.auth/user.json
CONSOLE_ERRORS_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/console-errors.json" | Out-File .env
displayName: Generate .env
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
# Cache and restore NPM packages
- task: Cache@2
displayName: Cache NPM packages
inputs:
key: 'npm_e2e | "$(Agent.OS)" | $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/package-lock.json'
restoreKeys: |
npm_e2e | "$(Agent.OS)"
npm_e2e
path: $(npm_config_cache)
- script: npm ci --no-fund --no-audit --prefer-offline
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
displayName: Restore NPM packages
# Build application
- pwsh: |
$cmsVersion = "$(Build.BuildNumber)" -replace "\+",".g"
dotnet new nugetconfig
dotnet nuget add source ./nupkg --name Local
dotnet new install Umbraco.Templates::$cmsVersion
dotnet new umbraco --name UmbracoProject --version $cmsVersion --exclude-gitignore --no-restore --no-update-check
dotnet restore UmbracoProject
cp $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest.UmbracoProject/*.cs UmbracoProject
dotnet build UmbracoProject --configuration $(buildConfiguration) --no-restore
displayName: Restore project
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
dotnet build UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-restore
dotnet dev-certs https
displayName: Build application
workingDirectory: $(Agent.BuildDirectory)/app
condition: succeeded()
# Run application
- bash: |
nohup dotnet run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
echo "##vso[task.setvariable variable=AcceptanceTestProcessId]$!"
displayName: Run application (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
workingDirectory: $(Agent.BuildDirectory)/app
# Run application Template
- template: nightly-E2E-run-application-template.yml
parameters:
DatabaseType: ${{ variables.DatabaseType }}
buildConfiguration: ${{ variables.buildConfiguration }}
additionalEnvironmentVariables: ${{ variables.additionalEnvironmentVariables }}
- pwsh: |
$process = Start-Process dotnet "run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
Write-Host "##vso[task.setvariable variable=AcceptanceTestProcessId]$($process.Id)"
displayName: Run application (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
workingDirectory: $(Agent.BuildDirectory)/app
# Ensures we have the package wait-on installed
- pwsh: npm install wait-on
displayName: Install wait-on package
# Wait for application to start responding to requests
- pwsh: npx wait-on -v --interval 1000 --timeout 120000 $(ASPNETCORE_URLS)
displayName: Wait for application
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Install Playwright and dependencies
- pwsh: npx playwright install chromium
displayName: Install Playwright only with Chromium browser
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Test
- pwsh: $(testCommand)
displayName: Run Playwright tests
continueOnError: true
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
env:
CI: true
CommitId: $(Build.SourceVersion)
AgentOs: $(Agent.OS)
# Stop application
- bash: kill -15 $(AcceptanceTestProcessId)
displayName: Stop application (Linux)
condition: and(succeeded(), ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Linux'))
- pwsh: Stop-Process -Id $(AcceptanceTestProcessId)
displayName: Stop application (Windows)
condition: and(succeeded(), ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Windows_NT'))
# Copy artifacts
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/results/*) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/results/* $(Build.ArtifactStagingDirectory) -Recurse
}
displayName: Copy Playwright results
condition: succeededOrFailed()
# Copy console error log
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/console-errors.json) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/console-errors.json $(Build.ArtifactStagingDirectory)
}
displayName: Copy console error log
condition: succeededOrFailed()
# Publish
- task: PublishPipelineArtifact@1
displayName: Publish test artifacts
condition: succeededOrFailed()
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: "Acceptance Test Results - $(Agent.JobName) - Attempt #$(System.JobAttempt)"
# Publish test results
- task: PublishTestResults@2
displayName: "Publish test results"
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: "tests/Umbraco.Tests.AcceptanceTest/results"
testRunTitle: "$(Agent.JobName)"
# Run tests Template
- template: nightly-E2E-run-tests-template.yml
parameters:
testCommand: $(testCommand)
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
DatabaseType: ${{ variables.DatabaseType }}
- job:
displayName: E2E Tests (SQL Server)
timeoutInMinutes: 180
condition: ${{ eq(parameters.skipDefaultConfigAcceptanceTests, false) }}
variables:
# Connection string
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=(localdb)\MSSQLLocalDB;AttachDbFilename=|DataDirectory|\Umbraco.mdf;Integrated Security=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.SqlClient
DatabaseType: SQLServer
SA_PASSWORD: UmbracoAcceptance123!
additionalEnvironmentVariables: false
strategy:
matrix:
LinuxPart1Of3:
testCommand: "npm run test -- --shard=1/3"
testFolder: "DefaultConfig"
vmImage: "ubuntu-latest"
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
CONNECTIONSTRINGS__UMBRACODBDSN: "Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True"
LinuxPart2Of3:
testCommand: "npm run test -- --shard=2/3"
testFolder: "DefaultConfig"
vmImage: "ubuntu-latest"
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
CONNECTIONSTRINGS__UMBRACODBDSN: "Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True"
LinuxPart3Of3:
testCommand: "npm run test -- --shard=3/3"
testFolder: "DefaultConfig"
vmImage: "ubuntu-latest"
SA_PASSWORD: $(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
CONNECTIONSTRINGS__UMBRACODBDSN: "Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True"
WindowsPart1Of3:
vmImage: "windows-latest"
testCommand: "npm run test -- --shard=1/3"
testFolder: "DefaultConfig"
vmImage: "windows-latest"
WindowsPart2Of3:
vmImage: "windows-latest"
testCommand: "npm run test -- --shard=2/3"
WindowsPart3Of3:
testFolder: "DefaultConfig"
vmImage: "windows-latest"
WindowsPart3Of3:
testCommand: "npm run test -- --shard=3/3"
testFolder: "DefaultConfig"
vmImage: "windows-latest"
pool:
vmImage: $(vmImage)
steps:
# Setup test environment
- task: DownloadPipelineArtifact@2
displayName: Download NuGet artifacts
inputs:
artifact: nupkg
path: $(Agent.BuildDirectory)/app/nupkg
- task: NodeTool@0
displayName: Use Node.js $(nodeVersion)
inputs:
versionSpec: $(nodeVersion)
- task: UseDotNet@2
displayName: Use .NET SDK from global.json
inputs:
useGlobalJson: true
# Setup test environment Template
- template: nightly-E2E-setup-template.yml
parameters:
nodeVersion: ${{ variables.nodeVersion }}
PlaywrightUserEmail: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL }}
PlaywrightPassword: ${{ variables.UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD }}
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
npm_config_cache: ${{ variables.npm_config_cache }}
- pwsh: |
"UMBRACO_USER_LOGIN=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSEREMAIL)
UMBRACO_USER_PASSWORD=$(UMBRACO__CMS__UNATTENDED__UNATTENDEDUSERPASSWORD)
URL=$(ASPNETCORE_URLS)
STORAGE_STAGE_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/playwright/.auth/user.json
CONSOLE_ERRORS_PATH=$(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/console-errors.json" | Out-File .env
displayName: Generate .env
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
# Cache and restore NPM packages
- task: Cache@2
displayName: Cache NPM packages
inputs:
key: 'npm_e2e | "$(Agent.OS)" | $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest/package-lock.json'
restoreKeys: |
npm_e2e | "$(Agent.OS)"
npm_e2e
path: $(npm_config_cache)
- script: npm ci --no-fund --no-audit --prefer-offline
workingDirectory: $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest
displayName: Restore NPM packages
# Build application
- pwsh: |
$cmsVersion = "$(Build.BuildNumber)" -replace "\+",".g"
dotnet new nugetconfig
dotnet nuget add source ./nupkg --name Local
dotnet new install Umbraco.Templates::$cmsVersion
dotnet new umbraco --name UmbracoProject --version $cmsVersion --exclude-gitignore --no-restore --no-update-check
dotnet restore UmbracoProject
cp $(Build.SourcesDirectory)/tests/Umbraco.Tests.AcceptanceTest.UmbracoProject/*.cs UmbracoProject
dotnet build UmbracoProject --configuration $(buildConfiguration) --no-restore
displayName: Restore project
workingDirectory: $(Agent.BuildDirectory)/app
- pwsh: |
dotnet build UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-restore
dotnet dev-certs https
displayName: Build application
workingDirectory: $(Agent.BuildDirectory)/app
condition: succeeded()
# Start SQL Server
- powershell: docker run --name mssql -d -p 1433:1433 -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=$(SA_PASSWORD)" mcr.microsoft.com/mssql/server:2022-latest
displayName: Start SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
# Run application Template
- template: nightly-E2E-run-application-template.yml
parameters:
SA_PASSWORD: ${{ variables.SA_PASSWORD }}
buildConfiguration: ${{ variables.buildConfiguration }}
DatabaseType: ${{ variables.DatabaseType }}
additionalEnvironmentVariables: ${{ variables.additionalEnvironmentVariables }}
- pwsh: SqlLocalDB start MSSQLLocalDB
displayName: Start SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
# Run tests Template
- template: nightly-E2E-run-tests-template.yml
parameters:
testCommand: $(testCommand)
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
DatabaseType: ${{ variables.DatabaseType }}
# Run application
- stage: AdditionalConfigE2E
displayName: Additional Config E2E Tests
dependsOn: Build
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm_e2e
ASPNETCORE_URLS: https://localhost:44331
PlaywrightPassword: UmbracoAcceptance123!
PlaywrightUserEmail: playwright@umbraco.com
jobs:
- job:
displayName: E2E Tests with Different App settings (SQL Server)
condition: ${{ or(eq(parameters.differentAppSettingsAcceptanceTests, true), eq(parameters.skipDefaultConfigAcceptanceTests, true)) }}
timeoutInMinutes: 180
variables:
SA_PASSWORD: UmbracoAcceptance123!
DatabaseType: SQLServer
strategy:
matrix:
# UnattendedInstallConfig
WindowsUnattendedInstallConfig:
vmImage: "windows-latest"
testFolder: "UnattendedInstallConfig"
testCommand: "npx playwright test --project=unattendedInstallConfig --grep=InstallSQLServer"
port: 44331
additionalEnvironmentVariables: false
# DeliveryApiConfig
WindowsDeliveryApiConfig:
vmImage: "windows-latest"
testFolder: "DeliveryApi"
port: ''
testCommand: "npx playwright test --project=deliveryApi"
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=(localdb)\MSSQLLocalDB;AttachDbFilename=|DataDirectory|\Umbraco.mdf;Integrated Security=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.SqlClient
additionalEnvironmentVariables: false
LinuxDeliveryApiConfig:
vmImage: "ubuntu-latest"
testFolder: "DeliveryApi"
port: ''
testCommand: "npx playwright test --project=deliveryApi"
CONNECTIONSTRINGS__UMBRACODBDSN: Server=(local);Database=Umbraco;User Id=sa;Password=$(SA_PASSWORD);Encrypt=True;TrustServerCertificate=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.SqlClient
additionalEnvironmentVariables: false
# ExternalLogin AzureADB2C
WindowsExternalLoginAzureADB2C:
vmImage: "windows-latest"
testFolder: "ExternalLogin\\AzureADB2C"
testCommand: "npx playwright test --project=externalLoginAzureADB2C"
port: 44331
packageName: "Microsoft.AspNetCore.Authentication.OpenIdConnect"
packageVersion: "9.0.8"
CONNECTIONSTRINGS__UMBRACODBDSN: Data Source=(localdb)\MSSQLLocalDB;AttachDbFilename=|DataDirectory|\Umbraco.mdf;Integrated Security=True
CONNECTIONSTRINGS__UMBRACODBDSN_PROVIDERNAME: Microsoft.Data.SqlClient
additionalEnvironmentVariables: true
pool:
vmImage: $(vmImage)
steps:
# Setup test environment Template
- template: nightly-E2E-setup-template.yml
parameters:
nodeVersion: ${{ variables.nodeVersion }}
PlaywrightUserEmail: ${{ variables.PlaywrightUserEmail }}
PlaywrightPassword: ${{ variables.PlaywrightPassword }}
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
npm_config_cache: ${{ variables.npm_config_cache }}
# Install NuGet package if specified in the matrix
- pwsh: |
Write-Host "Installing package $(packageName) version $(packageVersion)"
dotnet add package $(packageName) --version $(packageVersion)
displayName: "Install NuGet package: $(packageName)"
workingDirectory: $(Agent.BuildDirectory)/app/UmbracoProject
condition: and(succeeded(), ne(variables['packageName'], ''), ne(variables['packageVersion'], ''))
# Build application Template
- template: nightly-E2E-build-template.yml
parameters:
testFolder: $(testFolder)
buildConfiguration: ${{ variables.buildConfiguration }}
additionalEnvironmentVariables: $(additionalEnvironmentVariables)
# Build application for AzureADB2C
- pwsh: |
dotnet build UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-restore
dotnet dev-certs https
displayName: Build application for AzureADB2C
workingDirectory: $(Agent.BuildDirectory)/app
env:
AZUREADB2CDOMAIN: $(AZUREB2CDOMAIN)
AZUREADB2CTENANT: $(AZUREB2CTENANT)
AZUREADB2CPOLICY: $(AZUREB2CPOLICY)
AZUREADB2CCLIENTID: $(AZUREB2CCLIENTID)
AZUREADB2CCLIENTSECRET: $(AZUREB2CCLIENTSECRET)
condition: and(succeeded(), eq(variables['testFolder'], 'ExternalLogin\AzureADB2C'))
# Run application Template
- template: nightly-E2E-run-application-template.yml
parameters:
SA_PASSWORD: ${{ variables.SA_PASSWORD }}
additionalEnvironmentVariables: $(additionalEnvironmentVariables)
buildConfiguration: ${{ variables.buildConfiguration }}
DatabaseType: ${{ variables.DatabaseType }}
# Run application for Linux with additional Environment Variables for Azure AD
- bash: |
nohup dotnet run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
nohup dotnet run --project UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-build --no-launch-profile > $(Build.ArtifactStagingDirectory)/playwright.log 2>&1 &
echo "##vso[task.setvariable variable=AcceptanceTestProcessId]$!"
displayName: Run application (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'), eq(variables['testFolder'], 'ExternalLogin\AzureADB2C'))
workingDirectory: $(Agent.BuildDirectory)/app
env:
AZUREADB2CDOMAIN: $(AZUREB2CDOMAIN)
AZUREADB2CTENANT: $(AZUREB2CTENANT)
AZUREADB2CPOLICY: $(AZUREB2CPOLICY)
AZUREADB2CCLIENTID: $(AZUREB2CCLIENTID)
AZUREADB2CCLIENTSECRET: $(AZUREB2CCLIENTSECRET)
# Run application for Windows with additional Environment Variables for Azure AD
- pwsh: |
$process = Start-Process dotnet "run --project UmbracoProject --configuration $(buildConfiguration) --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
$process = Start-Process dotnet "run --project UmbracoProject --configuration ${{ variables.buildConfiguration }} --no-build --no-launch-profile 2>&1" -PassThru -NoNewWindow -RedirectStandardOutput $(Build.ArtifactStagingDirectory)/playwright.log
Write-Host "##vso[task.setvariable variable=AcceptanceTestProcessId]$($process.Id)"
displayName: Run application (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'), eq(variables['testFolder'], 'ExternalLogin\AzureADB2C'))
workingDirectory: $(Agent.BuildDirectory)/app
# Ensures we have the package wait-on installed
- pwsh: npm install wait-on
displayName: Install wait-on package
# Wait for application to start responding to requests
- pwsh: npx wait-on -v --interval 1000 --timeout 120000 $(ASPNETCORE_URLS)
displayName: Wait for application
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Install Playwright and dependencies
- pwsh: npx playwright install chromium
displayName: Install Playwright only with Chromium browser
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
# Test
- pwsh: $(testCommand)
displayName: Run Playwright tests
continueOnError: true
workingDirectory: tests/Umbraco.Tests.AcceptanceTest
env:
CI: true
CommitId: $(Build.SourceVersion)
AgentOs: $(Agent.OS)
AZUREADB2CDOMAIN: $(AZUREB2CDOMAIN)
AZUREADB2CTENANT: $(AZUREB2CTENANT)
AZUREADB2CPOLICY: $(AZUREB2CPOLICY)
AZUREADB2CCLIENTID: $(AZUREB2CCLIENTID)
AZUREADB2CCLIENTSECRET: $(AZUREB2CCLIENTSECRET)
# Stop application
- bash: kill -15 $(AcceptanceTestProcessId)
displayName: Stop application (Linux)
condition: and(succeeded(), ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Linux'))
- pwsh: Stop-Process -Id $(AcceptanceTestProcessId)
displayName: Stop application (Windows)
condition: and(succeeded(), ne(variables.AcceptanceTestProcessId, ''), eq(variables['Agent.OS'], 'Windows_NT'))
# Stop SQL Server
- pwsh: docker stop mssql
displayName: Stop SQL Server Docker image (Linux)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
- pwsh: SqlLocalDB stop MSSQLLocalDB
displayName: Stop SQL Server LocalDB (Windows)
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
# Copy artifacts
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/results/*) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/results/* $(Build.ArtifactStagingDirectory) -Recurse
}
displayName: Copy Playwright results
condition: succeededOrFailed()
# Copy console error log
- pwsh: |
if (Test-Path tests/Umbraco.Tests.AcceptanceTest/console-errors.json) {
Copy-Item tests/Umbraco.Tests.AcceptanceTest/console-errors.json $(Build.ArtifactStagingDirectory)
}
displayName: Copy console error log
condition: succeededOrFailed()
# Publish
- task: PublishPipelineArtifact@1
displayName: Publish test artifacts
condition: succeededOrFailed()
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
artifact: "Acceptance Test Results - $(Agent.JobName) - Attempt #$(System.JobAttempt)"
# Publish test results
- task: PublishTestResults@2
displayName: "Publish test results"
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: "tests/Umbraco.Tests.AcceptanceTest/results"
testRunTitle: "$(Agent.JobName)"
# Run tests Template
- template: nightly-E2E-run-tests-template.yml
parameters:
testCommand: $(testCommand)
ASPNETCORE_URLS: ${{ variables.ASPNETCORE_URLS }}
port: $(port)
AZUREB2CTESTUSEREMAIL: $(AZUREB2CTESTUSEREMAIL)
AZUREB2CTESTUSERPASSWORD: $(AZUREB2CTESTUSERPASSWORD)
DatabaseType: ${{ variables.DatabaseType }}
- stage: NotifySlackBot
displayName: Notify Slack on Failure
dependsOn: E2E
dependsOn: DefaultConfigE2E
# This stage will only run if the E2E tests fail or succeed with issues
condition: or(
eq(dependencies.E2E.result, 'failed'),
eq(dependencies.E2E.result, 'succeededWithIssues'))
condition: or(eq(dependencies.DefaultConfigE2E.result, 'failed'), eq(dependencies.DefaultConfigE2E.result, 'succeededWithIssues'))
jobs:
- job: PostToSlack
displayName: Send Slack Notification