* V16: Cache Version Mechanism (#19747)
* Add RepositoryCacheVersion table
* Add repository
* Add Cache version lock
* Add GetAll method to repository
* Add RepositoryCacheVersionService
* Remember to add lock in data creator
* Work my way out of constructor hell
This is why we use DI folks. 🤦
* Add checks to specific cache policies
* Fix migration
* Add to schema creator
* Fix database access
* Initialize the cache version on in memory miss
* Make cache version service internal
* Add tests
* Apply suggestions from code review
Co-authored-by: Andy Butland <abutland73@gmail.com>
* Add missing obsoletions
* Prefer full name
---------
Co-authored-by: Andy Butland <abutland73@gmail.com>
* fixed merge
* V16/feature/move last synced id to db (#19884)
* Foundation work for moving last synced id
* register manager and repo in dependency injection
* Fixing to make tests work
* Replacing the use of the old LastSyncedFileManager.cs with the new LastSyncedManager.cs
* Testing to delete out of sync id and old entries
* changing some stuff to please the reviewer.
* Inverted saving methods id check and fixed documentation mishaps
* Loadbalancing: Add Cache Sync service to allow us to roll forward isolated caches when backoffice is load balanced. (#20398)
* Split cache refreshers into internal and external caches
* Add obsolete constructor for CacheInstructionsPruningJob
* Add xml docs
* Move lastID management into CacheInstructionService
* Cache last synced ids in memory
* Lock when processing instructions
* Sync caches when out of sync
* Fix constructors for ICacheSyncService
* Cache version on request
* Register caches as synced when instructions are processed
* Rename CacheVersionAccessor to IRepositoryCacheVersionAccessor
* Set caches as synced before actually syncing the caches
* Set caches as synced before syncing, within scope, this should also lock the cache version from being written to whilst updating caches
* Only check version for backoffice requests
* Clear request cache when caches are syned
* Default to using NOOP cache version service
* Don't generate local identity in database server messenger anymore
* Fix ambiguous constructor
* Add helper method to switch to load balanced isolated caches
* Fix LastSyncedManagerTests
* Fix RepositoryCacheVersionServiceTests
* Fix DefaultCachePolicyTests
* Use correct constructor in FullDataSetRepositoryCachePolicy
* Minor cleanup
* Add XML docs
* Add more xml docs
* Apply suggestions from code review
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
---------
Co-authored-by: Zeegaan <skrivdetud@gmail.com>
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
* Fix migration plan
* fix tests
* Fix integration tests
* Fix changes from github review
* Move premigrations to v17
* Make lock constantws sequential
* Fix comment
* Make IRepositoryCacheVersionService and ICacheSyncService protected on EntityRepositoryBase
---------
Co-authored-by: Andy Butland <abutland73@gmail.com>
Co-authored-by: Nicklas Kramer <nik@umbraco.dk>
Co-authored-by: NillasKA <kramernicklas@gmail.com>
Co-authored-by: Zeegaan <skrivdetud@gmail.com>
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
* Initial implementation of non existing property editor
* Adjust `MissingPropertyEditor` to not require registering in PropertyEditorCollection
* Add `MissingPropertyEditor.name` back
* Remove unused dependencies from DataTypeService
* Removed reference to non existing property
* Add parameterless constructor back to MissingPropertyEditor
* Add validation error on document open to property with missing editor
* Update labels
* Removed public editor alias const
* Update src/Umbraco.Web.UI.Client/src/packages/property-editors/missing/manifests.ts
* Add test that checks whether the new MissingPropertyEditor is returned when an editor is not found
* Also check if the editor UI alias is correct in the test
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Share property editor instances between properties
* Only store missing property editors in memory in `ContentMapDefinition.MapValueViewModels()`
* Add value converter for the missing property editor to always return a string (same as the Label did previously)
* Small improvements to code block
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Persist and expose Umbraco system dates as UTC (#19705)
* Updated persistence DTOs defining default dates to use UTC.
* Remove ForceToUtc = false from all persistence DTO attributes (default when not specified is true).
* Removed use of SpecifyKind setting dates to local.
* Removed unnecessary Utc suffixes on properties.
* Persist current date time with UtcNow.
* Removed further necessary Utc suffixes and fixed failing unit tests.
* Added migration for SQL server to update database date default constraints.
* Added comment justifying not providing a migration for SQLite default date constraints.
* Ensure UTC for datetimes created from persistence DTOs.
* Ensure UTC when creating dates for published content rendering in Razor and outputting in delivery API.
* Fixed migration SQL syntax.
* Introduced AuditItemFactory for creating entries for the backoffice document history, so we can control the UTC setting on the retrieved persisted dates.
* Ensured UTC dates are retrieved for document versions.
* Ensured UTC is returned for backoffice display of last edited and published for variant content.
* Fixed SQLite syntax for default current datetime.
* Apply suggestions from code review
Co-authored-by: Laura Neto <12862535+lauraneto@users.noreply.github.com>
* Further updates from code review.
---------
Co-authored-by: Laura Neto <12862535+lauraneto@users.noreply.github.com>
* Migrate system dates from local server time to UTC (#19798)
* Add settings for the migration.
* Add migration and implement for SQL server.
* Implement for SQLite.
* Fixes from testing with SQL Server.
* Fixes from testing with SQLite.
* Code tidy.
* Cleaned up usings.
* Removed audit log date from conversion.
* Removed webhook log date from conversion.
* Updated update date initialization on saving dictionary items.
* Updated filter on log queries.
* Use timezone ID instead of system name to work cross-culture.
---------
Co-authored-by: Laura Neto <12862535+lauraneto@users.noreply.github.com>
* Add support for programmatic creation of property types providing the data type key (#19720)
* Add support for programmatic creation of property types providing the data type key.
* Add integration tests
---------
Co-authored-by: kjac <kja@umbraco.dk>
* Don't use Lazy
---------
Co-authored-by: Andy Butland <abutland73@gmail.com>
* Added logging and try/catch around retrieval of references, so we don't block critical operations following an incompatible data type change.
* Added a little more detail to the log message.
* Added a little more detail to the log message.
* Fix unittest mock dependency
---------
Co-authored-by: Migaroez <geusens@gmail.com>
* Block level variance - initial commit
* Remove TODOs
* Only convert RTEs with blocks
* Fix JSON paths for block level property validation
* Rename Properties to Values
* Correct the JSON path of block level validation errors
* Make it possible to skip content migration + ensure backwards compat for the new block format
* Partial culture variance publishing at property level
* UDI to key conversion for block editors - draft, WIP, do NOT merge 😄 (#16970)
* Convert block UDIs to GUIDs
* Fix merge
* Fix merge issues
* Rework nested layout item key parsing for backwards compatibility
* Clean-up
* Reverse block layout item key calculation
* Review
* Use IOptions to skip content migrations
* Remove "published" from data editor feature naming, as it can be used in other contexts too
* Parallel migration
* Don't use deprecated constructor
* Ensure that layout follows structure for partial publishing
* Block Grid element level variance + tests (incl. refactor of element level variation tests)
* Rollback unintended changes to Program.cs
* Fix bad casing
* Minor formatting
* RTE element level variance + tests
* Remove obsoleted constructors
* Use Umbraco.RichText instead of Umbraco.TinyMCE as layout alias for blocks in the RTE
* Fix bad merge
* Temporary fix for new cache in integration tests
* Add EditorAlias to block level properties
* Remove the unintended PropertyEditorAlias output for block values
* Add EditorAlias to Datatype Item model
* Update OpenApi.json
* Introduce "expose" for blocks
* Strict (explicit) handling for Expose
* Improve handling of document and element level variance changes
* Refactor variance alignment for published rendering
* Block UDI to Key conversion should also register as a conversion
* Convert newly added RTE unit test to new RTE blocks format
* Minor review changes
* Run memory intensive tests on Linux only
* Add tests proving that AllowEditInvariantFromNonDefault has effect for block level variance too
* Fix the Platform annotations
* Removed Platform annotations for tests.
* Fix merge
* Obsolete old PublishCulture extension
* More fixing bad merge
---------
Co-authored-by: Niels Lyngsø <niels.lyngso@gmail.com>
Co-authored-by: nikolajlauridsen <nikolajlauridsen@protonmail.ch>
* Add configuration/code to not run certain tests based on variables/release builds
* Applied longrunning testAttribute to the worst offenders (>200ms on my machine)
* Fix yaml notation
* split up windows/non windows test runs
* Added supression for moved tests
* Fix yaml validation issues
* Change yaml string parameter null value to empty string
* Convert empty strings to whitespace strings
* Rename and cleanup some paramater to better reflect why we use them
* Nightly build test
* Change nightly build authentication type
* template paramater fix
* Update nightly pipeline name
---------
Co-authored-by: Sven Geusens <sge@umbraco.dk>
* Add GetAsync method
* Fix up delete document type controller
* Add scope to delete async
* Add some scaffolding
* Add create model
* Start working on validation
* Move validation to its own service
* Use GetAllAsync instead of GetAsync
* Add initial composition support
Still need to figure out some kinks
* Validate compositions when creating
* Add initial folder support
* Initial handling of generic properties
* Add operation status responses
* Move create operation into service
* Add first test
* Fix issued shown by test
* Ensure a specific key can be specified when creating
* Rename container id to container key
Let's try and be consistent
* Create basic composition test
* Ensure new property groups are created with the correct key
* Add test showing property type issue
* Fix property types not using the expected key.
* Validate against model fetched from content type service
Just to make sure nothing explodes on the round trip
* Make helper for creating create models
* Add helper for creating container
* Make helper methods simpler to use
* Add test for compositions using compositions
* Add more composition tests
* Fix bug allowing element types to be composed by non element types
* Remove validators
This can just be a part of the editing service
* Minor cleanup
* Ensure that multiple levels of inheritance is possible
* Ensure doctype cannot be used as both composition and inheritance on the same doctype
* Ensure no duplicate aliases from composition and that compositions exists
* Minor cleanup
* Address todos
* Add SaveAsync method
* Renamed some models
* Rename from DocumentType to ContentType
* Clarify ParentKey as being container only + untangle things a tiny bit
* Clean out another TODO (less duplicate code) + more tests
* Refactor for reuse across different content types + add media type editing service + unit tests
* Refactor in preparation for update handling
* More tests + fixed bugs found while testing
* Simplify things a bit
* Content type update + a lot of unit tests + some refactor + fix bugs found while testing
* Begin building presentation factories for mapping view models to editing models
* Use async save
* Mapping factories and some clean-up
* Rename Key to Id (ParentKey to ParentId)
* Fix slight typo
* Use editing service in document type controllers and introduce media type controllers
* Validate containers and align container aliases with the current backoffice
* Remove ParentId from response
* Fix scope handling in DeleteAsync
* Refactor ContentTypeSort
* A little renaming for clarity + safeguard against changes to inheritance
* Persist allowed content types
* Fix bad merge + update controller response annotations
* Update OpenAPI JSON
* Update src/Umbraco.Cms.Api.Management/Controllers/DocumentType/DocumentTypeControllerBase.cs
Co-authored-by: Mole <nikolajlauridsen@protonmail.ch>
* Fix review comments
* Update usage of MapCreateAsync to ValidateAndMapForCreationAsync
---------
Co-authored-by: Nikolaj <nikolajlauridsen@protonmail.ch>
* Add alias to document item response
* Add master template key to detailed model
* Add mater template key as optiona parameter to Scaffolding
* Check for duplicate alias when creating templates directly
* Clean
* Ensure integration tests creates templates with unique aliases
* Perform mapping in presentation factory
* Update language models to get and set manual name
* Save custom language name in controller
* Rewrite AngularJS language edit view and controller
* Cleanup language overview
* Remove icon from language overview
* Make styling of control group the same as properties
* Ensure both ISO code and culture name are set in language model
* Use new language model constructor
* Update tests to use new language constructor
* Update culture name in dictionary package export
* Use language name in dictionary
* Fix language nullability issues
* Cleanup GetAllCultures and added null checks
* Re-add obsolete constructors
* Make language name required and update Cypress test
* Fix routing/saveNewLanguages Cypress test
* Make language name optional (improved backwards compatibility)
Co-authored-by: Ronald Barendse <ronald@panoramastudios.nl>
* Created Persistence.SQLite project skeleton.
* SQLite database initialization
* Various changes and hacks to make things work.
* WIP integration tests
* Fix thread safety tests
* Fix tests that relied on tie breaker sorting.
Spent a fair amount of time looking for a less lazy fix but gave up.
* Convert right join to left join ContentTypeRepository.PerformGetByQuery
SQLite doesn't support right join
* Fix test Can_Generate_Delete_SubQuery_Statement
Worth noting that NPoco.DatabaseTypes.SQLiteDatabaseType doesn't override
EscapeSqlIdentifier so NPoco will escape with [].
SQLite docs say > "A keyword enclosed in square brackets is an identifier.
This is not standard SQL.
This quoting mechanism is used by MS Access and SQL Server and is
included in SQLite for compatibility."
Also could have updated SqliteSyntaxProvider to match npoco but
decided against it.
* Fixes for paginated custom order by
* Fix tests broken by lack of unique indexes.
* Fix SqlServerTableByTableTest tests.
These tests didn't actually do anything as the tables already exist so schema creator just returned.
Did however point out that the default implementation for DoesTableExist just returns false so added a default naive implementation.
* Fix ValidateLoginSession - SelectTop must come later
* dry up database cleanup
* Fix up db migration tests.
We can't drop pk in sqlite without recreating table.
Test looks to be testing that add column works as intended which we can test.
* Prevent schema creation errors.
* SQLite ignore lock tests, WAL back on.
* Fix package schema tests
* Fix NPocoFetchTests - case sensitivity not under test
* Fix AdvancedMigrationTests (where possible)
Migrations probably need a good look later.
Maybe nuke old migrations and only support moving to v10 from v9.
If we do that can do some cleanup.
* Cleanup test database configuration
* Run integration tests against SQLite on build agent.
* Drop MS.Data.SQLite
System.Data.SQLite was quicker to roll out due to more CLR type mapping
* YAML
* Skip Umbraco.Tests.Integration.SqlCe
* Drop SqlServerTableByTable tests.
Until this week they did nothing anyway as they with NewSchemaPerTest
so the tests all passed as CreateTable was no op (already exists).
Also all of the tables are created in an empty database by SchemaValidationTest.cs
DatabaseSchemaCreation_Produces_DatabaseSchemaResult_With_Zero_Errors
* Might aswell run against macOS also.
* Copy azure pipelines task header layout
* Delete SQLCe projects
* Remove SQL CE specific code.
* Remove SQL CE NuSpec, template params, build script setup
* Delete umbraco-netcore-only.sln
* Add SkipTests solution configuration and use for codeql
* Remove reference to deleted nuspec file.
* Refactor ConnectionStrings WRT DataDirectory placeholder & ProviderName.
At this point you can try out SQLite support by setting the following
in appsettings.json and then completing the install process.
"ConnectionStrings": {
"umbracoDbDSN": "Data Source=|DataDirectory|/umbraco.sqlite",
"umbracoDbDSN_ProviderName": "System.Data.SQLite"
},
Not currently possible via installer UI without provider name pre-set in
configuration.
* Switch to Microsoft.Data.Sqlite
Some gross hacks but will be good to find out if this works
with apple silicon.
* Enable selection of SQLite via installer UI (also quick install)
* Remove SqlServerDbProviderFactoryCreator to cleanup a TODO
* Move SQL Server support to its own class library
* Add persistence dependencies to Umbraco.CMS metapackage
* Bugfix packages delete query
Created invalid query for SQLite.
* Try out cypress tests Linux + SQLite
* Prevent cypress test artifact upload failure on attempt 2+
* LocalDb bugfixes
* Drop redundant enum
* Move SqlClient constant
* Misc whitespace
* Remove IsSqlCe extension (TODO: drop non 9->10 migrations later).
* Umbraco.Persistence.* -> Umbraco.Cms.Persistence.*
* Display quick install defaults and per provider default database name.
* Misc remove old comment
* little re-arrange
* Remove almost all usages of IsSqlite extension.
* visual adjustments
* Custom Database Configuration is last step and should then say Install.
* use text instead of disabled inputs
* move legend, rename to Install
* Update SqlMainDomLock to work without distributed locks.
* Added IDistributedLockingMechanism interface and in memory impl.
* Drop locking from ISqlSyntaxProvider & wire up scope to abstraction.
* Added SqlServerDistributedLockingMechanism
* Move distributed locking interfaces and exceptions to Core + xmldocs.
* Fix tests, Misc cleanup, Add SQL distributed locking integration tests
* Provide mechanism to specify DistributedLockingMechanism in config
(even if added by composer)
* Nomplementation -> NoImplementation
* Fix misleading comment
* Integration tests use SqlServerDistributedLockingMechanism when possible
* Handle up-gradable locks SqlServerDistributedLockingMechanism.
TODO: InMemoryDistributedLockingMechanism.
Note: Nuked SqlServerDistributedLockingMechanismTests, will still sleep
at night.
Is covered by Umbraco.Cms.Tests.Integration.Umbraco.Infrastructure.Persistence.LockTests
* Make tests pass for InMemoryDistributedLockingMechanism, pretty hacky.
* Tweak constraints on WithCollectionBuilder so i can drop bad constructor
* Added SqliteDistributedLockingMechanism
* Dropped InMemoryDistributedMechanism + magic
InMemoryDistributedMechanism was pretty rubbish and now we have
a decent implementation for SQLite as we no longer block readers
see 8d1f42b.
Also drop the CollectionBuilder setup, instead do the same as we do
for syntax providers etc, it's more automagical so we never require an
explicit selection although we are allowing for it.
However keeping the optional IUmbracoBuilder constructor param for
CollectionBuilders as it's extremely useful.
* Fix quick install "" database name.
* Hide Database Configuration section when a connection string is pre-set.
Doesn't seem worth it to extract db name from connection string.
* Ensure wal test 2+
* Fix logging inconsistencies.
* Ensure in transaction when obtaining locks + no-op the SQLite read lock.
There's no point in running the query just to make a single test pass.
* Fix installer database display names
* Allow SQLite shared cache without losing deferred transactions
* Opt into shared cache for new SQLite databases + fix filename
* Fix misc inconsistency in .gitignore
* Prefer our interceptor interface
* Restore DEBUG_DATABASES code OnConnectionOpened in case it's used.
* Back to private cache.
* Added retry strategy for SQLite + refactor out SQL server specific stuff
* Fix SQL server tests.
* Misc - Orphaned comment, incorrect casing.
* InMemory SQLite test database & turn shared cache back on everywhere.
Co-authored-by: Niels Lyngsø <niels.lyngso@gmail.com>
