* New Backoffice: Refactor migrations to allow for unscoped migrations (#13654)
* Remove PostMigrations
These should be replaced with Notification usage
* Remove outer scope from Upgrader
* Remove unececary null check
* Add marker base class for migrations
* Enable scopeless migrations
* Remove unnecessary state check
The final state of the migration is no longer necessarily the final state of the plan.
* Extend ExecutedMigrationPlan
* Ensure that MigrationPlanExecutor.Execute always returns a result.
* Always save final state, regardless of errors
* Remove obsolete Execute
* Add Umbraco specific migration notification
* Publish notification after umbraco migration
* Throw the exception that failed a migration after publishing notification
* Handle notification publishing in DatabaseBuilder
* Fix tests
* Remember to complete scope
* Clean up MigrationPlanExecutor
* Run each package migration in a separate scope
* Add PartialMigrationsTests
* Add unhappy path test
* Fix bug shown by test
* Move PartialMigrationsTests into the correct folder
* Comment out refresh cache in data type migration
Need to add this back again as a notification handler or something.
* Start working on a notification test
* Allow migrations to request a cache rebuild
* Set RebuildCache from MigrateDataTypeConfigurations
* Clean MigrationPlanExecutor
* Add comment explaining the need to partial migration success
* Fix tests
* Allow overriding DefinePlan of UmbracoPlan
This is needed to test the DatabaseBuilder
* Fix notification test
* Don't throw exception to be immediately re-caught
* Assert that scopes notification are always published
* Ensure that scopes are created when requested
* Make test classes internal.
It doesn't really matter, but this way it doesn't show up in intellisense
* Add notification handler for clearing cookies
* Add CompatibilitySuppressions
* Rename Execute to ExecutePlan
We have to do this to be able to obsolete :(
* Update CompatibilitySuppressions
* Update src/Umbraco.Infrastructure/Migrations/MigrationPlanExecutor.cs
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
* generate compatability suppresion file
---------
Co-authored-by: Mole <nikolajlauridsen@protonmail.ch>
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
Co-authored-by: Zeegaan <nge@umbraco.dk>
* Fix CanForceCreateDatabase method and add some unit tests
* Fixed an old copy/paste error
* A little nitpicking over wording and formatting
---------
Co-authored-by: kjac <kja@umbraco.dk>
* Obsolete chings changed in v13
* Fix tests
* Obsolete Execute and add ExecutePlan with default implementation.
This just calls the old implementation and creates an ExecutedMigrationPlan from the result
In V13 this has its own implementation.
* Mention notification in obsolete message
* Removing can connect to database check so that Umbraco errors out on startup rather than trying to re-install when unable to connect to database.
* Revert breaking change
* Support installs of missing databases if the provider CanForceCreateDatabase. Else not
Co-authored-by: Austin Gilbert <AGilbert@rbaconsulting.com>
* Add new BackOfficeApi project
* Add swagger
* Add and route new install controller
* Add new install steps
* Add Setup endpoint
* Add missing RequiresExecution methods
* Fix nullability of databasemodel
* Move user information to separate model
* Remove ping method
* Add view models install data
* Move mapping folder
* Move ViewModels
* Add settings endpoint
* Remove unused binderprovider
* Postfix RequiresExecution with async
* Update NewDatabaseUpgradeStep to not depend on install step
* Add installstep collection
* Move registration into backoffice project
* Add InstallService
* Use service in controller
* Add upgrade to install service and use in controller
* Correctly check is database is configured
* Reorganize
* Reorganize into new core and infrastructure
* Rename steps
* Rename BackofficeApi to MangementApi
* Make install step an interface instead of abstract class
* Rename InstallStep to create CreateUserStep
* Move restart runtime and sign in user into install steps
* Move install service into new core project
* Map controllers in composer
* Restrict access to installcontroller based on runtime level
* Use FireAndForget when logging install
* Use actionresult instead of iactionresult
* Set new projects as not packable
* Link to backoffice in 201 response when installed
* Register installations
* Add custom backoffice routing template token
* Move umbraco path trimming out of application convention
* Make it easier to route to backoffice api
* Make swagger version aware and move behind backoffice path
* Obsolete old install classes
* Move maps into single file
This is all mappint to/from viewmodels in some manner
* Remove usage of InstallSetupResult
* Move new projects to the src folder
* Remove InstallationType from IInstallStep
This upgrade steps should implement their own IUpgradeStep interface
* Remove upgrade from service and controller
This should be its own service and controller
* Add xml docs
* Remove internals visible to
* Disable package validation for new projects
Quite the gotcha here, if the projects are brand new, there is no nuget packages to compare with, this causes the build to fail.
* Add ValidateDatabase endpoint
* Remove project references to new backoffice
We don't actually want to depend on this yet, it's just needed for testing/development
* Obsolete installationtype
* Add DatabaseSettingsFactory tests
* Add InstallServiceTests
* Fix InstallServiceTests
* Test RequireRuntimeLevelAttribute
* Implement new backoffice upgrader (#12818)
* Add UpgradeSettingsModel and viewmodel
* Add upgrade/settings endpoint
* Implement upgrade steps
* Add upgrade step collection
* Add UpgradeService
* Add authorize endpoint to UpgradeController
* Fix interface
* Add upgrade service tests
* Remove runtime check in databaseinstallstep
* Move RequireRuntimeLevel to controller
* Add a readme to the new backoffice part
* BackOffice not Backoffice
* Add conditional project references
* Fixes based on review
* Fix up
* Move running of steps into its own method in UpgradeService
* Make services transient
* More fixup
* Log exceptions when running steps
* Add slider to user view
* fix typo 'constentLevels'
* Add TelemetryLevel to UserModel
* Extend NewInstallStep to include descriptions of each TelemetryLevel
* Add nouislider as dependency for installer view
* Add telemetry levels
Add raw nouislider to select a telemetry level + set chosen level onChange to the user model + show detailed description of each level as a tooltip
* Copy over basic styling of nouislider
* Save consent level in installer
* Fix detailed key to not contain <br>
* Fix MetricsConsentService to log correctly when we set analyticsLevel in installer
* Fix breaking change and obsolete messages
* reinstate saved value of the subscription field if you traverse the steps
* calculate the initial slider level based on the saved value of the model
Co-authored-by: Nikolaj Geisle <niko737@edu.ucl.dk>
Co-authored-by: Jacob Overgaard <752371+iOvergaard@users.noreply.github.com>
* Created Persistence.SQLite project skeleton.
* SQLite database initialization
* Various changes and hacks to make things work.
* WIP integration tests
* Fix thread safety tests
* Fix tests that relied on tie breaker sorting.
Spent a fair amount of time looking for a less lazy fix but gave up.
* Convert right join to left join ContentTypeRepository.PerformGetByQuery
SQLite doesn't support right join
* Fix test Can_Generate_Delete_SubQuery_Statement
Worth noting that NPoco.DatabaseTypes.SQLiteDatabaseType doesn't override
EscapeSqlIdentifier so NPoco will escape with [].
SQLite docs say > "A keyword enclosed in square brackets is an identifier.
This is not standard SQL.
This quoting mechanism is used by MS Access and SQL Server and is
included in SQLite for compatibility."
Also could have updated SqliteSyntaxProvider to match npoco but
decided against it.
* Fixes for paginated custom order by
* Fix tests broken by lack of unique indexes.
* Fix SqlServerTableByTableTest tests.
These tests didn't actually do anything as the tables already exist so schema creator just returned.
Did however point out that the default implementation for DoesTableExist just returns false so added a default naive implementation.
* Fix ValidateLoginSession - SelectTop must come later
* dry up database cleanup
* Fix up db migration tests.
We can't drop pk in sqlite without recreating table.
Test looks to be testing that add column works as intended which we can test.
* Prevent schema creation errors.
* SQLite ignore lock tests, WAL back on.
* Fix package schema tests
* Fix NPocoFetchTests - case sensitivity not under test
* Fix AdvancedMigrationTests (where possible)
Migrations probably need a good look later.
Maybe nuke old migrations and only support moving to v10 from v9.
If we do that can do some cleanup.
* Cleanup test database configuration
* Run integration tests against SQLite on build agent.
* Drop MS.Data.SQLite
System.Data.SQLite was quicker to roll out due to more CLR type mapping
* YAML
* Skip Umbraco.Tests.Integration.SqlCe
* Drop SqlServerTableByTable tests.
Until this week they did nothing anyway as they with NewSchemaPerTest
so the tests all passed as CreateTable was no op (already exists).
Also all of the tables are created in an empty database by SchemaValidationTest.cs
DatabaseSchemaCreation_Produces_DatabaseSchemaResult_With_Zero_Errors
* Might aswell run against macOS also.
* Copy azure pipelines task header layout
* Delete SQLCe projects
* Remove SQL CE specific code.
* Remove SQL CE NuSpec, template params, build script setup
* Delete umbraco-netcore-only.sln
* Add SkipTests solution configuration and use for codeql
* Remove reference to deleted nuspec file.
* Refactor ConnectionStrings WRT DataDirectory placeholder & ProviderName.
At this point you can try out SQLite support by setting the following
in appsettings.json and then completing the install process.
"ConnectionStrings": {
"umbracoDbDSN": "Data Source=|DataDirectory|/umbraco.sqlite",
"umbracoDbDSN_ProviderName": "System.Data.SQLite"
},
Not currently possible via installer UI without provider name pre-set in
configuration.
* Switch to Microsoft.Data.Sqlite
Some gross hacks but will be good to find out if this works
with apple silicon.
* Enable selection of SQLite via installer UI (also quick install)
* Remove SqlServerDbProviderFactoryCreator to cleanup a TODO
* Move SQL Server support to its own class library
* Add persistence dependencies to Umbraco.CMS metapackage
* Bugfix packages delete query
Created invalid query for SQLite.
* Try out cypress tests Linux + SQLite
* Prevent cypress test artifact upload failure on attempt 2+
* LocalDb bugfixes
* Drop redundant enum
* Move SqlClient constant
* Misc whitespace
* Remove IsSqlCe extension (TODO: drop non 9->10 migrations later).
* Umbraco.Persistence.* -> Umbraco.Cms.Persistence.*
* Display quick install defaults and per provider default database name.
* Misc remove old comment
* little re-arrange
* Remove almost all usages of IsSqlite extension.
* visual adjustments
* Custom Database Configuration is last step and should then say Install.
* use text instead of disabled inputs
* move legend, rename to Install
* Update SqlMainDomLock to work without distributed locks.
* Added IDistributedLockingMechanism interface and in memory impl.
* Drop locking from ISqlSyntaxProvider & wire up scope to abstraction.
* Added SqlServerDistributedLockingMechanism
* Move distributed locking interfaces and exceptions to Core + xmldocs.
* Fix tests, Misc cleanup, Add SQL distributed locking integration tests
* Provide mechanism to specify DistributedLockingMechanism in config
(even if added by composer)
* Nomplementation -> NoImplementation
* Fix misleading comment
* Integration tests use SqlServerDistributedLockingMechanism when possible
* Handle up-gradable locks SqlServerDistributedLockingMechanism.
TODO: InMemoryDistributedLockingMechanism.
Note: Nuked SqlServerDistributedLockingMechanismTests, will still sleep
at night.
Is covered by Umbraco.Cms.Tests.Integration.Umbraco.Infrastructure.Persistence.LockTests
* Make tests pass for InMemoryDistributedLockingMechanism, pretty hacky.
* Tweak constraints on WithCollectionBuilder so i can drop bad constructor
* Added SqliteDistributedLockingMechanism
* Dropped InMemoryDistributedMechanism + magic
InMemoryDistributedMechanism was pretty rubbish and now we have
a decent implementation for SQLite as we no longer block readers
see 8d1f42b.
Also drop the CollectionBuilder setup, instead do the same as we do
for syntax providers etc, it's more automagical so we never require an
explicit selection although we are allowing for it.
However keeping the optional IUmbracoBuilder constructor param for
CollectionBuilders as it's extremely useful.
* Fix quick install "" database name.
* Hide Database Configuration section when a connection string is pre-set.
Doesn't seem worth it to extract db name from connection string.
* Ensure wal test 2+
* Fix logging inconsistencies.
* Ensure in transaction when obtaining locks + no-op the SQLite read lock.
There's no point in running the query just to make a single test pass.
* Fix installer database display names
* Allow SQLite shared cache without losing deferred transactions
* Opt into shared cache for new SQLite databases + fix filename
* Fix misc inconsistency in .gitignore
* Prefer our interceptor interface
* Restore DEBUG_DATABASES code OnConnectionOpened in case it's used.
* Back to private cache.
* Added retry strategy for SQLite + refactor out SQL server specific stuff
* Fix SQL server tests.
* Misc - Orphaned comment, incorrect casing.
* InMemory SQLite test database & turn shared cache back on everywhere.
Co-authored-by: Niels Lyngsø <niels.lyngso@gmail.com>
* Allow changing UmbracoMediaPath to an absolute path. Also ensure Imagesharp are handing requests outside of the wwwroot folder.
* Let UmbracoMediaUrl fallback to UmbracoMediaPath when empty
* Add FileSystemFileProvider to expose an IFileSystem as IFileProvider
* Replace IUmbracoMediaFileProvider with IFileProviderFactory implementation
* Fix issue resolving relative paths when media URL has changed
* Remove FileSystemFileProvider and require explicitly implementing IFileProviderFactory
* Update tests (UnauthorizedAccessException isn't thrown anymore for rooted files)
* Update test to use UmbracoMediaUrl
* Add UmbracoMediaPhysicalRootPath global setting
* Remove MediaFileManagerImageProvider and use composited file providers
* Move CreateFileProvider to IFileSystem extension method
* Add rooted path tests
Co-authored-by: Ronald Barendse <ronald@barend.se>
The issue was discovered when testing Umbraco Id. If you have an existing install and there is a pending Umbraco migration (upgrade), but you clear out the connection string to force a new install screen, fill out the existing connection string details, it all works and redirects to the back office. This will throw exceptions because the runtime state is in an upgrade state (which is fixed in #11064) but if you then logout and try to log in with an active OAuth provider, it means there is a redirect outside of Umbraco and back again and you'll end up back on the installer screen - but it will not show you the upgrade screen, instead it shows you the normal install screen. This is because we are not using IOptionsMonitor for connection string settings which means it's already read the original empty connection string setting fom the very beginning and isn't reading the current/updated value.
We need to review all IOptions usages. Most of them should be IOptionsMonitor unless its impossible to change the app behavior at runtime with a particular config option.
* Clean up and changes to backoffice for the nuget only packages
* temp commit of package logic removal
* Lots of package code cleanup and removal
* Removes old package data from the test package xml
* Updates packaging code to take in XDocument instead of a file since we'll not be dealing with files, starts creating expressions for the package migrations scripting.
* fixing tests
* Fixes runtime state and boot failed middleware so that it actually runs. Separates out unattended install/upgrade into notification handlers.
* Gets unattended package migrations working and running
* Gets embedded package.xml resources able to install from package migration.
* Implements automatic package migrations for package that just declare an xml data manifest.
* fix build
* small cleanups
* fix build
* adds some tests
* Fix export test
* Fix newlines in test for linux
* Typo
* removes old todos and updates AutomaticPackgeMigrationPlan to use getter with backing field.
* Update dictionary package data to use GUID
* Ensures macros are packaged and used with their GUID
* Ensures the GUID for doc types and media types remains consistent for package installation based on what is in the xml.
* fix automatic migrations to not validate initial state, fixes packaging GUIDs for multiple entities.
* Added guids to embedded test packages (Some tests are still failing)
* Fix one more test
* Fixes up Key vs Id, moves tests to correct namespace, fix tests
* Fixes Dictionary packaging to ensure an xml hierarchy
* Fixes tests
* fixes package xml
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
* Clean up and changes to backoffice for the nuget only packages
* temp commit of package logic removal
* Lots of package code cleanup and removal
* Removes old package data from the test package xml
* Updates packaging code to take in XDocument instead of a file since we'll not be dealing with files, starts creating expressions for the package migrations scripting.
* fixing tests
* Fixes runtime state and boot failed middleware so that it actually runs. Separates out unattended install/upgrade into notification handlers.
* Gets unattended package migrations working and running
* Gets embedded package.xml resources able to install from package migration.
* Implements automatic package migrations for package that just declare an xml data manifest.
* fix build
* small cleanups
* fix build
* adds some tests
* Fix export test
* Fix newlines in test for linux
* Typo
* removes old todos and updates AutomaticPackgeMigrationPlan to use getter with backing field.
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
