V15: Cache Seeding (#17102)
* Update to dotnet 9 and update nuget packages * Update umbraco code version * Update Directory.Build.props Co-authored-by: Elitsa Marinovska <21998037+elit0451@users.noreply.github.com> * Include preview version in pipeline * update template projects * update global json with specific version * Update version.json to v15 * Rename TrimStart and TrimEnd to string specific * Rename to Exact * Update global.json Co-authored-by: Ronald Barendse <ronald@barend.se> * Remove includePreviewVersion * Rename to trim exact * Add new Hybridcache project * Add tests * Start implementing PublishedContent.cs * Implement repository for content * Refactor to use async everywhere * Add cache refresher * make public as needed for serialization * Use content type cache to get content type out * Refactor to use ContentCacheNode model, that goes in the memory cache * Remove content node kit as its not needed * Implement tests for ensuring caching * Implement better asserts * Implement published property * Refactor to use mapping * Rename to document tests * Update to test properties * Create more tests * Refactor mock tests into own file * Update property test * Fix published version of content * Change default cache level to elements * Refactor to always have draft * Refactor to not use PublishedModelFactory * Added tests * Added and updated tests * Fixed tests * Don't return empty object with id * More tests * Added key * Another key * Refactor CacheService to be responsible for using the hybrid cache * Use notification handler to remove deleted content from cache * Add more tests for missing functions * Implement missing methods * Remove HasContent as it pertains to routing * Fik up test * formatting * refactor variable names * Implement variant tests * Map all the published content properties * Get item out of cache first, to assert updated * Implement member cache * Add member test * Implement media cache * Implement property tests for media tests * Refactor tests to use extension method * Add more media tests * Refactor properties to no longer have element caching * Don't use property cache level * Start implementing seeding * Only seed when main * Add Immutable for performance * Implement permanent seeding of content * Implement cache settings * Implement tests for seeding * Update package version * start refactoring nurepo * Refactor so draft & published nodes are cached individually * Refactor RefreshContent to take node instead of IContent * Refactor media to also use cache nodes * Remove member from repo as it isn't cached * Refactor media to not include preview, as media has no draft * create new benchmark project * POC Integration benchmarks with custom api controllers * Start implementing content picker tests * Implement domain cache * Rework content cache to implement interface * Start implementing elements cache * Implement published snapshot service * Publish snapshot tests * Use snapshot for elements cache * Create test proving we don't clear cache when updating content picker * Clear entire elements cache * Remove properties from element cache, when content gets updated. * Rename methods to async * Refactor to use old cache interfaces instead of new ones * Remove snapshot, as it is no longer needed * Fix tests building * Refactor domaincache to not have snapshots * Delete benchmarks * Delete benchmarks * Add HybridCacheProject to Umbraco * Add comment to route value transformer * Implement is draft * remove snapshot from property * V15 updated the hybrid caching integration tests to use ContentEditingService (#16947) * Added builder extension withParentKey * Created builder with ContentEditingService * Added usage of the ContentEditingService to SETUP * Started using ContentEditingService builder in tests * Updated builder extensions * Fixed builder * Clean up * Clean up, not done * Added Ids * Remove entries from cache on delete * Fix up seeding logic * Don't register hybrid cache twice * Change seeded entry options * Update hybrid cache package * Fix up published property to work with delivery api again * Fix dependency injection to work with tests * Fix naming * Dont make caches nullable * Make content node sealed * Remove path and other unused from content node * Remove hacky 2 phase ctor * Refactor to actually set content templates * Remove umbraco context * Remove "HasBy" methods * rename property data * Delete obsolete legacy stuff * Add todo for making expiration configurable * Add todo in UmbracoContext * Add clarifying comment in content factory * Remove xml stuff from published property * Fix according to review * Make content type cache injectible * Make content type cache injectible * Rename to database cache repository * Rename to document cache * Add TODO * Refactor to async * Rename to async * Make everything async * Remove duplicate line from json schema * Move Hybrid cache project * Remove leftover file * Refactor to use keys * Refactor published content to no longer have content data, as it is on the node itself * Refactor to member to use proper content node ctor * Move tests to own folder * Add immutable objects to property and content data for performance * Make property data public * Fix member caching to be singleton * Obsolete GetContentType * Remove todo * Fix naming * Fix lots of exposed errors due to scope test * Add final scope tests * Rename to document cache service * Rename test files * Create new doc type tests * Add ignore to tests * Start implementing refresh for content type save * Clear contenttype cache when contenttype is updated * Fix test Teh contenttype is not upated unless the property is dirty * Updated tests * Added tests * Use init for ContentSourceDto * Startup of setup * Fix get by key in PublishedContentTypeCache * Remove ContentType from PublishedContentTypeCache when contenttype is deleted * Created interfaces for the builder with the necessary properties * Created builder for PropertyTypeContainer * Created builder for PropertyTypeEditing * Created builder for PropertyTypeValidationEditing * Made adjustments to the builder * Updated name of usage * Commented out to test * Cleaned up builders * Updated integration test setup * Moved tests * Added interface * Add IDocumentSeedKeyProvider and migrate existing logic to seed key provider * Added functionality to the INavigationQueryService to get root keys * Fixed issue with navigation * Created helper to Convert a IContentType to ContentTypeUpdateModel * Added interfaces * Added builder * Cleaned up builders and added fixes * Added tests for PublishedContentTypeCache * Applied changes in builder * Add BreadthFirstKeyProvider * Use ISet for seedkey providers * Implement GetContentSource by key * Seed the cache with keys provided by seed key providers * Builder updates * Test setup updates * Updated tests * Dont require contenttype keys for seeding * Fix cache settings * Don't inject cache settings into SeedingNotificationHandler * Fix tests * Use enlistment for setting updated cache item * Pin seeded nodes for longer * Fix BreadthFirstKeyProvider * Fix ContentTypeSeedKeyProvider * Fix tests * Only seed published documents * Only cache published if contentCacheNode is not draft * Fix incorrect templateId * Removed unnecessary setup * initialized value * Fixed template test * Removed test * Updated tests * Removed code that was not used * Removed unused cacheSettings * Re-organize to support media cache seeding * Add MediaBreadthFirstKeyProvider * Seed media * Don't use IdKeyMap when removing content from cache * Don't clear IdKeyMap in DocumentCacheService * Add unit tests * Don't use IdKeyMap when deleting media * Add default value to timespan * Use cancellation tokens when doing loop * Fixed Models Builder error --------- Co-authored-by: Zeegaan <skrivdetud@gmail.com> Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com> Co-authored-by: Elitsa Marinovska <21998037+elit0451@users.noreply.github.com> Co-authored-by: Ronald Barendse <ronald@barend.se> Co-authored-by: Andreas Zerbst <andr317c@live.dk> Co-authored-by: Sven Geusens <sge@umbraco.dk> Co-authored-by: Andreas Zerbst <73799582+andr317c@users.noreply.github.com> Co-authored-by: Bjarke Berg <mail@bergmania.dk>
This commit is contained in:
Mole
@@ -1,4 +1,7 @@
|
||||
using Microsoft.Extensions.Caching.Hybrid;
|
||||
using System.Diagnostics;
|
||||
using Microsoft.Extensions.Caching.Hybrid;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Umbraco.Cms.Core;
|
||||
using Umbraco.Cms.Core.Models;
|
||||
using Umbraco.Cms.Core.Models.PublishedContent;
|
||||
@@ -6,6 +9,7 @@ using Umbraco.Cms.Core.Scoping;
|
||||
using Umbraco.Cms.Core.Services;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Factories;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Persistence;
|
||||
using Umbraco.Extensions;
|
||||
|
||||
namespace Umbraco.Cms.Infrastructure.HybridCache.Services;
|
||||
|
||||
@@ -17,7 +21,30 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
private readonly Microsoft.Extensions.Caching.Hybrid.HybridCache _hybridCache;
|
||||
private readonly IPublishedContentFactory _publishedContentFactory;
|
||||
private readonly ICacheNodeFactory _cacheNodeFactory;
|
||||
private readonly IEnumerable<IDocumentSeedKeyProvider> _seedKeyProviders;
|
||||
private readonly IPublishedModelFactory _publishedModelFactory;
|
||||
private readonly CacheSettings _cacheSettings;
|
||||
|
||||
private HashSet<Guid>? _seedKeys;
|
||||
private HashSet<Guid> SeedKeys
|
||||
{
|
||||
get
|
||||
{
|
||||
if (_seedKeys is not null)
|
||||
{
|
||||
return _seedKeys;
|
||||
}
|
||||
|
||||
_seedKeys = [];
|
||||
|
||||
foreach (IDocumentSeedKeyProvider provider in _seedKeyProviders)
|
||||
{
|
||||
_seedKeys.UnionWith(provider.GetSeedKeys());
|
||||
}
|
||||
|
||||
return _seedKeys;
|
||||
}
|
||||
}
|
||||
|
||||
public DocumentCacheService(
|
||||
IDatabaseCacheRepository databaseCacheRepository,
|
||||
@@ -25,7 +52,10 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
ICoreScopeProvider scopeProvider,
|
||||
Microsoft.Extensions.Caching.Hybrid.HybridCache hybridCache,
|
||||
IPublishedContentFactory publishedContentFactory,
|
||||
ICacheNodeFactory cacheNodeFactory)
|
||||
ICacheNodeFactory cacheNodeFactory,
|
||||
IEnumerable<IDocumentSeedKeyProvider> seedKeyProviders,
|
||||
IOptions<CacheSettings> cacheSettings,
|
||||
IPublishedModelFactory publishedModelFactory)
|
||||
{
|
||||
_databaseCacheRepository = databaseCacheRepository;
|
||||
_idKeyMap = idKeyMap;
|
||||
@@ -33,25 +63,21 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
_hybridCache = hybridCache;
|
||||
_publishedContentFactory = publishedContentFactory;
|
||||
_cacheNodeFactory = cacheNodeFactory;
|
||||
_seedKeyProviders = seedKeyProviders;
|
||||
_publishedModelFactory = publishedModelFactory;
|
||||
_cacheSettings = cacheSettings.Value;
|
||||
}
|
||||
|
||||
// TODO: Stop using IdKeyMap for these, but right now we both need key and id for caching..
|
||||
public async Task<IPublishedContent?> GetByKeyAsync(Guid key, bool preview = false)
|
||||
{
|
||||
Attempt<int> idAttempt = _idKeyMap.GetIdForKey(key, UmbracoObjectTypes.Document);
|
||||
if (idAttempt.Success is false)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
|
||||
ContentCacheNode? contentCacheNode = await _hybridCache.GetOrCreateAsync(
|
||||
GetCacheKey(key, preview), // Unique key to the cache entry
|
||||
async cancel => await _databaseCacheRepository.GetContentSourceAsync(idAttempt.Result, preview));
|
||||
async cancel => await _databaseCacheRepository.GetContentSourceAsync(key, preview));
|
||||
|
||||
scope.Complete();
|
||||
return contentCacheNode is null ? null : _publishedContentFactory.ToIPublishedContent(contentCacheNode, preview);
|
||||
return contentCacheNode is null ? null : _publishedContentFactory.ToIPublishedContent(contentCacheNode, preview).CreateModel(_publishedModelFactory);
|
||||
}
|
||||
|
||||
public async Task<IPublishedContent?> GetByIdAsync(int id, bool preview = false)
|
||||
@@ -67,37 +93,55 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
GetCacheKey(keyAttempt.Result, preview), // Unique key to the cache entry
|
||||
async cancel => await _databaseCacheRepository.GetContentSourceAsync(id, preview));
|
||||
scope.Complete();
|
||||
return contentCacheNode is null ? null : _publishedContentFactory.ToIPublishedContent(contentCacheNode, preview);
|
||||
return contentCacheNode is null ? null : _publishedContentFactory.ToIPublishedContent(contentCacheNode, preview).CreateModel(_publishedModelFactory);;
|
||||
}
|
||||
|
||||
public async Task SeedAsync(IReadOnlyCollection<Guid> contentTypeKeys)
|
||||
public async Task SeedAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
IEnumerable<ContentCacheNode> contentCacheNodes = _databaseCacheRepository.GetContentByContentTypeKey(contentTypeKeys);
|
||||
foreach (ContentCacheNode contentCacheNode in contentCacheNodes)
|
||||
|
||||
foreach (Guid key in SeedKeys)
|
||||
{
|
||||
if (contentCacheNode.IsDraft)
|
||||
if(cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
continue;
|
||||
break;
|
||||
}
|
||||
|
||||
// TODO: Make these expiration dates configurable.
|
||||
// Never expire seeded values, we cannot do TimeSpan.MaxValue sadly, so best we can do is a year.
|
||||
var entryOptions = new HybridCacheEntryOptions
|
||||
{
|
||||
Expiration = TimeSpan.FromDays(365),
|
||||
LocalCacheExpiration = TimeSpan.FromDays(365),
|
||||
};
|
||||
var cacheKey = GetCacheKey(key, false);
|
||||
|
||||
await _hybridCache.SetAsync(
|
||||
GetCacheKey(contentCacheNode.Key, false),
|
||||
contentCacheNode,
|
||||
entryOptions);
|
||||
// We'll use GetOrCreateAsync because it may be in the second level cache, in which case we don't have to re-seed.
|
||||
ContentCacheNode? cachedValue = await _hybridCache.GetOrCreateAsync<ContentCacheNode?>(
|
||||
cacheKey,
|
||||
async cancel =>
|
||||
{
|
||||
ContentCacheNode? cacheNode = await _databaseCacheRepository.GetContentSourceAsync(key, false);
|
||||
|
||||
// We don't want to seed drafts
|
||||
if (cacheNode is null || cacheNode.IsDraft)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return cacheNode;
|
||||
},
|
||||
GetSeedEntryOptions());
|
||||
|
||||
// If the value is null, it's likely because
|
||||
if (cachedValue is null)
|
||||
{
|
||||
await _hybridCache.RemoveAsync(cacheKey);
|
||||
}
|
||||
}
|
||||
|
||||
scope.Complete();
|
||||
}
|
||||
|
||||
private HybridCacheEntryOptions GetSeedEntryOptions() => new()
|
||||
{
|
||||
Expiration = _cacheSettings.SeedCacheDuration,
|
||||
LocalCacheExpiration = _cacheSettings.SeedCacheDuration
|
||||
};
|
||||
|
||||
public async Task<bool> HasContentByIdAsync(int id, bool preview = false)
|
||||
{
|
||||
Attempt<Guid> keyAttempt = _idKeyMap.GetKeyForId(id, UmbracoObjectTypes.Document);
|
||||
@@ -122,34 +166,67 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
|
||||
bool isSeeded = SeedKeys.Contains(content.Key);
|
||||
|
||||
// Always set draft node
|
||||
// We have nodes seperate in the cache, cause 99% of the time, you are only using one
|
||||
// and thus we won't get too much data when retrieving from the cache.
|
||||
var draftCacheNode = _cacheNodeFactory.ToContentCacheNode(content, true);
|
||||
await _hybridCache.RemoveAsync(GetCacheKey(content.Key, true));
|
||||
ContentCacheNode draftCacheNode = _cacheNodeFactory.ToContentCacheNode(content, true);
|
||||
|
||||
await _databaseCacheRepository.RefreshContentAsync(draftCacheNode, content.PublishedState);
|
||||
_scopeProvider.Context?.Enlist($"UpdateMemoryCache_Draft_{content.Key}", completed =>
|
||||
{
|
||||
if(completed is false)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RefreshHybridCache(draftCacheNode, GetCacheKey(content.Key, true), isSeeded).GetAwaiter().GetResult();
|
||||
}, 1);
|
||||
|
||||
if (content.PublishedState == PublishedState.Publishing)
|
||||
{
|
||||
var publishedCacheNode = _cacheNodeFactory.ToContentCacheNode(content, false);
|
||||
await _hybridCache.RemoveAsync(GetCacheKey(content.Key, false));
|
||||
|
||||
await _databaseCacheRepository.RefreshContentAsync(publishedCacheNode, content.PublishedState);
|
||||
_scopeProvider.Context?.Enlist($"UpdateMemoryCache_{content.Key}", completed =>
|
||||
{
|
||||
if(completed is false)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RefreshHybridCache(publishedCacheNode, GetCacheKey(content.Key, false), isSeeded).GetAwaiter().GetResult();
|
||||
}, 1);
|
||||
}
|
||||
|
||||
scope.Complete();
|
||||
}
|
||||
|
||||
private async Task RefreshHybridCache(ContentCacheNode cacheNode, string cacheKey, bool isSeeded)
|
||||
{
|
||||
// If it's seeded we want it to stick around the cache for longer.
|
||||
if (isSeeded)
|
||||
{
|
||||
await _hybridCache.SetAsync(
|
||||
cacheKey,
|
||||
cacheNode,
|
||||
GetSeedEntryOptions());
|
||||
}
|
||||
else
|
||||
{
|
||||
await _hybridCache.SetAsync(cacheKey, cacheNode);
|
||||
}
|
||||
}
|
||||
|
||||
private string GetCacheKey(Guid key, bool preview) => preview ? $"{key}+draft" : $"{key}";
|
||||
|
||||
public async Task DeleteItemAsync(int id)
|
||||
public async Task DeleteItemAsync(IContentBase content)
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
await _databaseCacheRepository.DeleteContentItemAsync(id);
|
||||
Attempt<Guid> keyAttempt = _idKeyMap.GetKeyForId(id, UmbracoObjectTypes.Document);
|
||||
await _hybridCache.RemoveAsync(GetCacheKey(keyAttempt.Result, true));
|
||||
await _hybridCache.RemoveAsync(GetCacheKey(keyAttempt.Result, false));
|
||||
_idKeyMap.ClearCache(keyAttempt.Result);
|
||||
_idKeyMap.ClearCache(id);
|
||||
await _databaseCacheRepository.DeleteContentItemAsync(content.Id);
|
||||
await _hybridCache.RemoveAsync(GetCacheKey(content.Key, true));
|
||||
await _hybridCache.RemoveAsync(GetCacheKey(content.Key, false));
|
||||
scope.Complete();
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user