* V16: Cache Version Mechanism (#19747)
* Add RepositoryCacheVersion table
* Add repository
* Add Cache version lock
* Add GetAll method to repository
* Add RepositoryCacheVersionService
* Remember to add lock in data creator
* Work my way out of constructor hell
This is why we use DI folks. 🤦
* Add checks to specific cache policies
* Fix migration
* Add to schema creator
* Fix database access
* Initialize the cache version on in memory miss
* Make cache version service internal
* Add tests
* Apply suggestions from code review
Co-authored-by: Andy Butland <abutland73@gmail.com>
* Add missing obsoletions
* Prefer full name
---------
Co-authored-by: Andy Butland <abutland73@gmail.com>
* fixed merge
* V16/feature/move last synced id to db (#19884)
* Foundation work for moving last synced id
* register manager and repo in dependency injection
* Fixing to make tests work
* Replacing the use of the old LastSyncedFileManager.cs with the new LastSyncedManager.cs
* Testing to delete out of sync id and old entries
* changing some stuff to please the reviewer.
* Inverted saving methods id check and fixed documentation mishaps
* Loadbalancing: Add Cache Sync service to allow us to roll forward isolated caches when backoffice is load balanced. (#20398)
* Split cache refreshers into internal and external caches
* Add obsolete constructor for CacheInstructionsPruningJob
* Add xml docs
* Move lastID management into CacheInstructionService
* Cache last synced ids in memory
* Lock when processing instructions
* Sync caches when out of sync
* Fix constructors for ICacheSyncService
* Cache version on request
* Register caches as synced when instructions are processed
* Rename CacheVersionAccessor to IRepositoryCacheVersionAccessor
* Set caches as synced before actually syncing the caches
* Set caches as synced before syncing, within scope, this should also lock the cache version from being written to whilst updating caches
* Only check version for backoffice requests
* Clear request cache when caches are syned
* Default to using NOOP cache version service
* Don't generate local identity in database server messenger anymore
* Fix ambiguous constructor
* Add helper method to switch to load balanced isolated caches
* Fix LastSyncedManagerTests
* Fix RepositoryCacheVersionServiceTests
* Fix DefaultCachePolicyTests
* Use correct constructor in FullDataSetRepositoryCachePolicy
* Minor cleanup
* Add XML docs
* Add more xml docs
* Apply suggestions from code review
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
---------
Co-authored-by: Zeegaan <skrivdetud@gmail.com>
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
* Fix migration plan
* fix tests
* Fix integration tests
* Fix changes from github review
* Move premigrations to v17
* Make lock constantws sequential
* Fix comment
* Make IRepositoryCacheVersionService and ICacheSyncService protected on EntityRepositoryBase
---------
Co-authored-by: Andy Butland <abutland73@gmail.com>
Co-authored-by: Nicklas Kramer <nik@umbraco.dk>
Co-authored-by: NillasKA <kramernicklas@gmail.com>
Co-authored-by: Zeegaan <skrivdetud@gmail.com>
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
* Start work
* Introduce dto
* Start making repository
* Add migrations
* Implement fetchable first job
* Fix up to also finish tasks
* Refactor jobs to distributed background jobs
* Filter jobs correctly on LastRun
* Hardcode delay
* Add settings to configure delay and period
* Fix formatting
* Add default data
* Add update on startup, which will update periods on startup
* Refactor service to return job directly
* Update src/Umbraco.Infrastructure/Services/Implement/DistributedJobService.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/Umbraco.Infrastructure/BackgroundJobs/DistributedBackgroundJobHostedService.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/Umbraco.Infrastructure/Migrations/Install/DatabaseDataCreator.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/Umbraco.Infrastructure/Migrations/Install/DatabaseDataCreator.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/Umbraco.Infrastructure/BackgroundJobs/DistributedBackgroundJobHostedService.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Remove unused
* Move jobs and make internal
* make OpenIddictCleanupJob.cs public, as it is used elsewhere
* Minor docstring changes
* Update src/Umbraco.Core/Persistence/Constants-Locks.cs
Co-authored-by: Mole <nikolajlauridsen@protonmail.ch>
* ´Throw correct exceptions
* Update xml doc
* Remove business logic from repository
* Remove more business logic from repository into service
* Remove adding jobs from migration
* fix creation
* Rename to ExecuteAsync
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: mole <nikolajlauridsen@protonmail.ch>
* Persist and expose Umbraco system dates as UTC (#19705)
* Updated persistence DTOs defining default dates to use UTC.
* Remove ForceToUtc = false from all persistence DTO attributes (default when not specified is true).
* Removed use of SpecifyKind setting dates to local.
* Removed unnecessary Utc suffixes on properties.
* Persist current date time with UtcNow.
* Removed further necessary Utc suffixes and fixed failing unit tests.
* Added migration for SQL server to update database date default constraints.
* Added comment justifying not providing a migration for SQLite default date constraints.
* Ensure UTC for datetimes created from persistence DTOs.
* Ensure UTC when creating dates for published content rendering in Razor and outputting in delivery API.
* Fixed migration SQL syntax.
* Introduced AuditItemFactory for creating entries for the backoffice document history, so we can control the UTC setting on the retrieved persisted dates.
* Ensured UTC dates are retrieved for document versions.
* Ensured UTC is returned for backoffice display of last edited and published for variant content.
* Fixed SQLite syntax for default current datetime.
* Apply suggestions from code review
Co-authored-by: Laura Neto <12862535+lauraneto@users.noreply.github.com>
* Further updates from code review.
---------
Co-authored-by: Laura Neto <12862535+lauraneto@users.noreply.github.com>
* Migrate system dates from local server time to UTC (#19798)
* Add settings for the migration.
* Add migration and implement for SQL server.
* Implement for SQLite.
* Fixes from testing with SQL Server.
* Fixes from testing with SQLite.
* Code tidy.
* Cleaned up usings.
* Removed audit log date from conversion.
* Removed webhook log date from conversion.
* Updated update date initialization on saving dictionary items.
* Updated filter on log queries.
* Use timezone ID instead of system name to work cross-culture.
---------
Co-authored-by: Laura Neto <12862535+lauraneto@users.noreply.github.com>
* Started implementing new LongRunningOperationService and adjusting tasks to use this service
This service will manage operations that require status to be synced between servers (load balanced setup).
* Missing migration to add new lock. Other simplifications.
* Add job to cleanup the LongRunningOperations entries
* Add new DatabaseCacheRebuilder.RebuildAsync method
This is both async and returns an attempt, which will fail if a rebuild operation is already running.
* Missing LongRunningOperation database table creation on clean install
* Store expire date in the long running operation. Better handling of non-background operations.
Storing an expiration date allows setting different expiration times depending on the type of operation, and whether it is running in the background or not.
* Added integration tests for LongRunningOperationRepository
* Added unit tests for LongRunningOperationService
* Add type as a parameter to more repository calls. Distinguish between expiration and deletion in `LongRunningOperationRepository.CleanOperations`.
* Fix failing unit test
* Fixed `PerformPublishBranchAsync` result not being deserialized correctly
* Remove unnecessary DatabaseCacheRebuildResult value
* Add status to `LongRunningOperationService.GetResult` attempt to inform on why a result could not be retrieved
* General improvements
* Missing rename
* Improve the handling of long running operations that are not in background and stale operations
* Fix failing unit tests
* Fixed small mismatch between interface and implementation
* Use a fire and forget task instead of the background queue
* Apply suggestions from code review
Co-authored-by: Andy Butland <abutland73@gmail.com>
* Make sure exceptions are caught when running in the background
* Alignment with other repositories (async + pagination)
* Additional fixes
* Add Async suffix to service methods
* Missing adjustment
* Moved hardcoded settings to IOptions
* Fix issue in SQL Server where 0 is not accepted as requested number of rows
* Fix issue in SQL Server where query provided to count cannot contain orderby
* Additional SQL Server fixes
---------
Co-authored-by: Andy Butland <abutland73@gmail.com>
* Introduce new AuditEntryService
- Moved logic related to the IAuditEntryRepository from the AuditService to the new service
- Introduced new Async methods
- Using ids (for easier transition from the previous Write method)
- Using keys
- Moved and updated integration tests related to the audit entries to a new test class `AuditEntryServiceTests`
- Added unit tests class `AuditEntryServiceTests` and added a few unit tests
- Added migration to add columns for `performingUserKey` and `affectedUserKey` and convert existing user ids
- Adjusted usages of the old AuditService.Write method to use the new one (mostly notification handlers)
* Audit service rework
- Added new async and paged methods
- Marked (now) redundant methods as obsolete
- Updated all of the usages to use the non-obsolete methods
- Added unit tests class `AuditServiceTests` and some unit tests
- Updated existing integration test
* Apply suggestions from code review
* Small improvement
* Update src/Umbraco.Core/Services/AuditService.cs
* Some minor adjustments following the merge
* Delete unnecessary file
* Small cleanup on the tests
* Fix warnings SA1111, SA1028, SA1500, IDE1270 in Umbraco.Web.Website, and updated rules.
* Remove warnings: IDE0270: Null check can be simplified
* More SqlServer project warnings resolved
* CS0105 namespace appeared already
* Suppress warning until implementation:
#pragma warning disable CS0162 // Unreachable code detected
#pragma warning disable CS0618 // Type or member is obsolete
CS0162 remove unreachable code
SA1028 remove trailing whitespace
SA1106 no empty statements
CS1570 malformed XML
CS1572 corrected xml parameter
CS1573 param tag added
IDE0007 var not explicit
IDE0008 explicit not var
IDE0057 simplify substring
IDE0074 compound assignment
CA1825 array.empty
Down to 3479 warnings
* - SA1116, SA117 params on same line
- IDE0057 substring simplified
Specific warnings for Umbraco.Tests.Benchmarks
* Fixed IDE0074 compound assignment and added specific warnings for Umbraco.Tests.Common
* Specific warnings for Umbraco.Tests.Integration and Umbraco.Tests.Common
Fixed:
- SA1111, SA1116, SA117 params and line formatting (not all as there are many)
- SA1122 string.Empty
- IDE0057 simplify substring
- IDE0044,IDE0044 make field readonly
- IDE1006 naming rule violation (add _)
- SA1111 closing parenthesis on line of last parameter
- SA1649 filename match type name
- SA1312,SA1306 lowercase variable and field names
* Fixed various warnings where they are more straight-forward, including:
- SA1649 file name match type name
- SA111 parenthesis on line of last parameter
- IDE0028 simplify collection initializer
- SA1306 lower-case letter field
- IDE044 readonly field
- SA1122 string.Empty
- SA1116 params same line
- IDE1006 upper casing
- IDE0041 simplify null check
Updated the following projects to only list their remaining specific warning codes:
- Umbraco.Tests.UnitTests
Typo in `Umbraco.Web.Website` project
* Reverted test change
* Now 1556 warnings.
Fixed various warnings where they are more straight-forward, including:
- SA1111/SA1116/SA1119 parenthesis
- SA1117 params
- SA1312 lowercase variable
- SA1121 built-in type
- SA1500/SA1513/SA1503 formatting braces
- SA1400 declare access modifier
- SA1122 string.Empty
- SA1310 no underscore
- IDE0049 name simplified
- IDE0057 simplify substring
- IDE0074 compound assignment
- IDE0032 use auto-property
- IDE0037 simplify member name
- IDE0008 explicit type not var
- IDE0016/IDE0270/IDE0041 simplify null checks
- IDE0048/SA1407 clarity in arithmetic
- IDE1006 correct param names
- IDE0042 deconstruct variable
- IDE0044 readonly
- IDE0018 inline variable declarations
- IDE0074/IDE0054 compound assignment
- IDE1006 naming
- CS1573 param XML
- CS0168 unused variable
Comment formatting in project files for consistency.
Updated all projects to only list remaining specific warning codes as warnings instead of errors (errors is now default).
* Type not var, and more warning exceptions
* Tweaked merge issue, readded comment about rollback
* Readded comment re rollback.
* Readded comments
* Comment tweak
* Comment tweak
* Fix warnings SA1111, SA1028, SA1500, IDE1270 in Umbraco.Web.Website, and updated rules.
* Remove warnings: IDE0270: Null check can be simplified
* More SqlServer project warnings resolved
* CS0105 namespace appeared already
* Suppress warning until implementation:
#pragma warning disable CS0162 // Unreachable code detected
#pragma warning disable CS0618 // Type or member is obsolete
CS0162 remove unreachable code
SA1028 remove trailing whitespace
SA1106 no empty statements
CS1570 malformed XML
CS1572 corrected xml parameter
CS1573 param tag added
IDE0007 var not explicit
IDE0008 explicit not var
IDE0057 simplify substring
IDE0074 compound assignment
CA1825 array.empty
Down to 3479 warnings
* Include umbraco-package.json manifests in package telemetry and add ID
* Add package ID to ManifestResponseModel
* Use async service method and HTTP client factory
* Avoid breaking changes
* Obsoletions related to Delivery API
* Fix TypeLoader and TypeFinder tests
* Remove obsolete and default implementations of IFileSource and IFileTypeCollection
* More Delivery API related obsoletions
* VariationContextAccessor related
* ValueFactories obsoletion and fix references
* ValueSetBuilders obsoletions
* ValueConverters obsoletions
* Other obsolete ctors and methods
* Forgotten VariationContextAccessor obsoletion
* More obsoletions
* XPath related obsoletions
* Revert XmlHelper changes
* Delete RenamedRootNavigator and its tests
* Fix test
* XmlHelper obsoletion
* Return null instead of GetXPathValue
* Obsolete entire class instead
* Remove XPath obsoletions from IPublishedCache
* Remove XPath-related if-block that is no longer needed
* Change obsolete msg for classes needed for NuCache
* Moving classes to NuCache and making them internal
* Remove more XPath-related obsoletions
* Remove NavigableNavigator and its tests
* Cleanup
* Remove Xpath references from tests
* Revert interface deletion in MediaCache
* Using XOR operation
Co-authored-by: Nuklon <Nuklon@users.noreply.github.com>
---------
Co-authored-by: Nuklon <Nuklon@users.noreply.github.com>
* Only use datetimeoffset in API
* Updated publish endpoint to take schedule information
* Fixed test builds
* OpenApi updates
* Update OpenApi schema
* Fixed issues with publishing.
* Added validation before publishing. Had to move a lot of classes to core.
* added missing files
* Added validation info to error
* Typo
* Clean up and adding missing result
* Updating to AvailableCultures instead of PublishedCultures
* Handle time needs to be in the future
* validate only cultures that needs to be published
* Fix typos
* Filter out cultures that we are not trying to publish from the validation errors
* Don't filter out only the available cultures, to accommodate non-created variants
* Only allow publish of existing cultures
* Fixed issues found in test
* fixed build
---------
Co-authored-by: Elitsa <elm@umbraco.dk>
* Use named HttpClients
* Register as named client
* Set headers on request message.
* Add User agent header to http client
* Add headers within Try-catch
* Add using for HttpResponseMessage
* Update src/Umbraco.Infrastructure/BackgroundJobs/Jobs/WebhookFiring.cs
Co-authored-by: Ronald Barendse <ronald@barend.se>
* Update src/Umbraco.Infrastructure/BackgroundJobs/Jobs/WebhookFiring.cs
Co-authored-by: Kenn Jacobsen <kja@umbraco.dk>
* Log key instead of entire webhook
* Update src/Umbraco.Web.Common/DependencyInjection/UmbracoBuilderExtensions.cs
Co-authored-by: Ronald Barendse <ronald@barend.se>
* Create constant for User agent header name
* Update src/Umbraco.Infrastructure/BackgroundJobs/Jobs/WebhookFiring.cs
Co-authored-by: Ronald Barendse <ronald@barend.se>
* Use datetime.now
---------
Co-authored-by: Zeegaan <nge@umbraco.dk>
Co-authored-by: Ronald Barendse <ronald@barend.se>
Co-authored-by: Kenn Jacobsen <kja@umbraco.dk>
* Implement persistence
* Start implementing repository
* Implement repository
* Implement request service
* Dont run WebhookFiring if not in runtime mode run
* Refactor repository and service to have full CRUD
* add tests for Request service
* Implement WebhookRequest lock
* Register hosted service
* Add try catch when firing HttpRequest
* Add migration
* Refactor to use renamed IWebhookService
* Refactor tests too
* Add setting to configure webhook firing period
* Update docs
* Review fixes
* Add column renaming migration
* Remove unused service
* run request in parralel
* Refactor to fire parallel in background
---------
Co-authored-by: Zeegaan <nge@umbraco.dk>
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
* Refactor jobs from HostedServices into BackgroundJobs
* Clean up generics and DI setup
* Add RecurringBackgroundJob Unit Tests
* Add ServiceCollection helper
* Add Obsolete attributes
* Add Notification Classes
* Add UnitTests for RecurringBackgroundJob HostedService
* Add NotificationEvents
* Add state to notifications
* Update UnitTests
* Add Obsolete Attributes to old hosted service classes
* Updated xmldoc in IRecurringBackgroundJob.cs
* Update Obsolete attribute messages to indicate classes will be removed in Umbraco 14
(cherry picked from commit c30ffa9ac3)
* Refactor jobs from HostedServices into BackgroundJobs
* Clean up generics and DI setup
* Add RecurringBackgroundJob Unit Tests
* Add ServiceCollection helper
* Add Obsolete attributes
* Add Notification Classes
* Add UnitTests for RecurringBackgroundJob HostedService
* Add NotificationEvents
* Add state to notifications
* Update UnitTests
* Add Obsolete Attributes to old hosted service classes
* Updated xmldoc in IRecurringBackgroundJob.cs
* Update Obsolete attribute messages to indicate classes will be removed in Umbraco 14
