V15 QA Hybrid Cache added media cache tests and fixed media cache refreshing (#17160)
* Add cache refresher * make public as needed for serialization * Use content type cache to get content type out * Refactor to use ContentCacheNode model, that goes in the memory cache * Remove content node kit as its not needed * Implement tests for ensuring caching * Implement better asserts * Implement published property * Refactor to use mapping * Rename to document tests * Update to test properties * Create more tests * Refactor mock tests into own file * Update property test * Fix published version of content * Change default cache level to elements * Refactor to always have draft * Refactor to not use PublishedModelFactory * Added tests * Added and updated tests * Fixed tests * Don't return empty object with id * More tests * Added key * Another key * Refactor CacheService to be responsible for using the hybrid cache * Use notification handler to remove deleted content from cache * Add more tests for missing functions * Implement missing methods * Remove HasContent as it pertains to routing * Fik up test * formatting * refactor variable names * Implement variant tests * Map all the published content properties * Get item out of cache first, to assert updated * Implement member cache * Add member test * Implement media cache * Implement property tests for media tests * Refactor tests to use extension method * Add more media tests * Refactor properties to no longer have element caching * Don't use property cache level * Start implementing seeding * Only seed when main * Add Immutable for performance * Implement permanent seeding of content * Implement cache settings * Implement tests for seeding * Update package version * start refactoring nurepo * Refactor so draft & published nodes are cached individually * Refactor RefreshContent to take node instead of IContent * Refactor media to also use cache nodes * Remove member from repo as it isn't cached * Refactor media to not include preview, as media has no draft * create new benchmark project * POC Integration benchmarks with custom api controllers * Start implementing content picker tests * Implement domain cache * Rework content cache to implement interface * Start implementing elements cache * Implement published snapshot service * Publish snapshot tests * Use snapshot for elements cache * Create test proving we don't clear cache when updating content picker * Clear entire elements cache * Remove properties from element cache, when content gets updated. * Rename methods to async * Refactor to use old cache interfaces instead of new ones * Remove snapshot, as it is no longer needed * Fix tests building * Refactor domaincache to not have snapshots * Delete benchmarks * Delete benchmarks * Add HybridCacheProject to Umbraco * Add comment to route value transformer * Implement is draft * remove snapshot from property * V15 updated the hybrid caching integration tests to use ContentEditingService (#16947) * Added builder extension withParentKey * Created builder with ContentEditingService * Added usage of the ContentEditingService to SETUP * Started using ContentEditingService builder in tests * Updated builder extensions * Fixed builder * Clean up * Clean up, not done * Added Ids * Remove entries from cache on delete * Fix up seeding logic * Don't register hybrid cache twice * Change seeded entry options * Update hybrid cache package * Fix up published property to work with delivery api again * Fix dependency injection to work with tests * Fix naming * Dont make caches nullable * Make content node sealed * Remove path and other unused from content node * Remove hacky 2 phase ctor * Refactor to actually set content templates * Remove umbraco context * Remove "HasBy" methods * rename property data * Delete obsolete legacy stuff * Add todo for making expiration configurable * Add todo in UmbracoContext * Add clarifying comment in content factory * Remove xml stuff from published property * Fix according to review * Make content type cache injectible * Make content type cache injectible * Rename to database cache repository * Rename to document cache * Add TODO * Refactor to async * Rename to async * Make everything async * Remove duplicate line from json schema * Move Hybrid cache project * Remove leftover file * Refactor to use keys * Refactor published content to no longer have content data, as it is on the node itself * Refactor to member to use proper content node ctor * Move tests to own folder * Add immutable objects to property and content data for performance * Make property data public * Fix member caching to be singleton * Obsolete GetContentType * Remove todo * Fix naming * Fix lots of exposed errors due to scope test * Add final scope tests * Rename to document cache service * Rename test files * Create new doc type tests * Add ignore to tests * Start implementing refresh for content type save * Clear contenttype cache when contenttype is updated * Fix test Teh contenttype is not upated unless the property is dirty * Updated tests * Added tests * Use init for ContentSourceDto * Startup of setup * Fix get by key in PublishedContentTypeCache * Remove ContentType from PublishedContentTypeCache when contenttype is deleted * Created interfaces for the builder with the necessary properties * Created builder for PropertyTypeContainer * Created builder for PropertyTypeEditing * Created builder for PropertyTypeValidationEditing * Made adjustments to the builder * Updated name of usage * Commented out to test * Cleaned up builders * Updated integration test setup * Moved tests * Added interface * Add IDocumentSeedKeyProvider and migrate existing logic to seed key provider * Added functionality to the INavigationQueryService to get root keys * Fixed issue with navigation * Created helper to Convert a IContentType to ContentTypeUpdateModel * Added interfaces * Added builder * Cleaned up builders and added fixes * Added tests for PublishedContentTypeCache * Applied changes in builder * Add BreadthFirstKeyProvider * Use ISet for seedkey providers * Implement GetContentSource by key * Seed the cache with keys provided by seed key providers * Builder updates * Test setup updates * Updated tests * Dont require contenttype keys for seeding * Fix cache settings * Don't inject cache settings into SeedingNotificationHandler * Fix tests * Use enlistment for setting updated cache item * Pin seeded nodes for longer * Fix BreadthFirstKeyProvider * Fix ContentTypeSeedKeyProvider * Fix tests * Only seed published documents * Only cache published if contentCacheNode is not draft * Fix incorrect templateId * Removed unnecessary setup * initialized value * Fixed template test * Removed test * Updated tests * Removed code that was not used * Removed unused cacheSettings * Re-organize to support media cache seeding * Add MediaBreadthFirstKeyProvider * Seed media * Don't use IdKeyMap when removing content from cache * Don't clear IdKeyMap in DocumentCacheService * Add unit tests * Don't use IdKeyMap when deleting media * Add default value to timespan * Use cancellation tokens when doing loop * Fixed Models Builder error * Builder testing * Media test * Created builders * Updated method from getByKey to getById to maintain structure * Updated test setup * Moved tests * Cleaned up builders * Added helper * Added space * Updated tests * Cleaned setup * Added tests for MediaTypes * Uncommented tests * Added builder extensions * Added interfaces for builder methods * Updated to use interface pattern * Updated name * Used builder * Cleaned up test * Added media scope tests * Removed PropertyTypeModelBuilder * moved PropertyTypeModelBuilder to PropertyTypeEditingBuilder * Removed constructor * Removed duplicate * Fixed naming * Reverted * Removed space * Added todos * Removed only run on linux * Commet out DocumentHybridCacheTemplateTest again This is fixed in the release branch * Update media when media type is updated * Remove todo the tests pass now --------- Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com> Co-authored-by: Zeegaan <skrivdetud@gmail.com> Co-authored-by: Sven Geusens <sge@umbraco.dk> Co-authored-by: nikolajlauridsen <nikolajlauridsen@protonmail.ch> Co-authored-by: Bjarke Berg <mail@bergmania.dk>
This commit is contained in:
Andreas Zerbst
@@ -65,6 +65,8 @@ public static class UmbracoBuilderExtensions
|
||||
builder.AddNotificationAsyncHandler<MediaDeletedNotification, CacheRefreshingNotificationHandler>();
|
||||
builder.AddNotificationAsyncHandler<ContentTypeRefreshedNotification, CacheRefreshingNotificationHandler>();
|
||||
builder.AddNotificationAsyncHandler<ContentTypeDeletedNotification, CacheRefreshingNotificationHandler>();
|
||||
builder.AddNotificationAsyncHandler<MediaTypeRefreshedNotification, CacheRefreshingNotificationHandler>();
|
||||
builder.AddNotificationAsyncHandler<MediaTypeDeletedNotification, CacheRefreshingNotificationHandler>();
|
||||
builder.AddNotificationAsyncHandler<UmbracoApplicationStartedNotification, SeedingNotificationHandler>();
|
||||
builder.AddCacheSeeding();
|
||||
return builder;
|
||||
|
||||
@@ -18,17 +18,17 @@ public class MediaCache : IPublishedMediaCache
|
||||
|
||||
public async Task<IPublishedContent?> GetByIdAsync(int id) => await _mediaCacheService.GetByIdAsync(id);
|
||||
|
||||
public async Task<IPublishedContent?> GetByKeyAsync(Guid key) => await _mediaCacheService.GetByKeyAsync(key);
|
||||
public async Task<IPublishedContent?> GetByIdAsync(Guid key) => await _mediaCacheService.GetByKeyAsync(key);
|
||||
|
||||
public IPublishedContent? GetById(bool preview, int contentId) => GetByIdAsync(contentId).GetAwaiter().GetResult();
|
||||
|
||||
public IPublishedContent? GetById(bool preview, Guid contentId) =>
|
||||
GetByKeyAsync(contentId).GetAwaiter().GetResult();
|
||||
GetByIdAsync(contentId).GetAwaiter().GetResult();
|
||||
|
||||
|
||||
public IPublishedContent? GetById(int contentId) => GetByIdAsync(contentId).GetAwaiter().GetResult();
|
||||
|
||||
public IPublishedContent? GetById(Guid contentId) => GetByKeyAsync(contentId).GetAwaiter().GetResult();
|
||||
public IPublishedContent? GetById(Guid contentId) => GetByIdAsync(contentId).GetAwaiter().GetResult();
|
||||
|
||||
|
||||
public IPublishedContentType? GetContentType(Guid key) => _publishedContentTypeCache.Get(PublishedItemType.Media, key);
|
||||
|
||||
@@ -19,7 +19,9 @@ internal sealed class CacheRefreshingNotificationHandler :
|
||||
INotificationAsyncHandler<MediaRefreshNotification>,
|
||||
INotificationAsyncHandler<MediaDeletedNotification>,
|
||||
INotificationAsyncHandler<ContentTypeRefreshedNotification>,
|
||||
INotificationAsyncHandler<ContentTypeDeletedNotification>
|
||||
INotificationAsyncHandler<ContentTypeDeletedNotification>,
|
||||
INotificationAsyncHandler<MediaTypeRefreshedNotification>,
|
||||
INotificationAsyncHandler<MediaTypeDeletedNotification>
|
||||
{
|
||||
private readonly IDocumentCacheService _documentCacheService;
|
||||
private readonly IMediaCacheService _mediaCacheService;
|
||||
@@ -126,7 +128,7 @@ internal sealed class CacheRefreshingNotificationHandler :
|
||||
var contentTypeIds = notification.Changes.Where(x => x.ChangeTypes.HasTypesAny(types)).Select(x => x.Item.Id)
|
||||
.ToArray();
|
||||
|
||||
if (contentTypeIds.Length != 0)
|
||||
if (contentTypeIds.Length > 0)
|
||||
{
|
||||
foreach (var contentTypeId in contentTypeIds)
|
||||
{
|
||||
@@ -148,4 +150,33 @@ internal sealed class CacheRefreshingNotificationHandler :
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task HandleAsync(MediaTypeRefreshedNotification notification, CancellationToken cancellationToken)
|
||||
{
|
||||
const ContentTypeChangeTypes types // only for those that have been refreshed
|
||||
= ContentTypeChangeTypes.RefreshMain | ContentTypeChangeTypes.RefreshOther;
|
||||
var mediaTypeIds = notification.Changes.Where(x => x.ChangeTypes.HasTypesAny(types)).Select(x => x.Item.Id)
|
||||
.ToArray();
|
||||
|
||||
if (mediaTypeIds.Length > 0)
|
||||
{
|
||||
foreach (var mediaTypeId in mediaTypeIds)
|
||||
{
|
||||
_publishedContentTypeCache.ClearContentType(mediaTypeId);
|
||||
}
|
||||
|
||||
_mediaCacheService.Rebuild(mediaTypeIds);
|
||||
}
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task HandleAsync(MediaTypeDeletedNotification notification, CancellationToken cancellationToken)
|
||||
{
|
||||
foreach (IMediaType deleted in notification.DeletedEntities )
|
||||
{
|
||||
_publishedContentTypeCache.ClearContentType(deleted.Id);
|
||||
}
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -237,7 +237,7 @@ AND cmsContentNu.nodeId IS NULL
|
||||
return CreateContentNodeKit(dto, serializer, preview);
|
||||
}
|
||||
|
||||
private IEnumerable<ContentSourceDto> GetContentSourceByDocumentTypeKey(IEnumerable<Guid> documentTypeKeys)
|
||||
private IEnumerable<ContentSourceDto> GetContentSourceByDocumentTypeKey(IEnumerable<Guid> documentTypeKeys, Guid objectType)
|
||||
{
|
||||
Guid[] keys = documentTypeKeys.ToArray();
|
||||
if (keys.Any() is false)
|
||||
@@ -248,19 +248,27 @@ AND cmsContentNu.nodeId IS NULL
|
||||
Sql<ISqlContext>? sql = SqlContentSourcesSelect()
|
||||
.InnerJoin<NodeDto>("n")
|
||||
.On<NodeDto, ContentDto>((n, c) => n.NodeId == c.ContentTypeId, "n", "umbracoContent")
|
||||
.Append(SqlObjectTypeNotTrashed(SqlContext, Constants.ObjectTypes.Document))
|
||||
.Append(SqlObjectTypeNotTrashed(SqlContext, objectType))
|
||||
.WhereIn<NodeDto>(x => x.UniqueId, keys,"n")
|
||||
.Append(SqlOrderByLevelIdSortOrder(SqlContext));
|
||||
|
||||
return GetContentNodeDtos(sql);
|
||||
}
|
||||
|
||||
public IEnumerable<ContentCacheNode> GetContentByContentTypeKey(IEnumerable<Guid> keys)
|
||||
public IEnumerable<ContentCacheNode> GetContentByContentTypeKey(IEnumerable<Guid> keys, ContentCacheDataSerializerEntityType entityType)
|
||||
{
|
||||
IEnumerable<ContentSourceDto> dtos = GetContentSourceByDocumentTypeKey(keys);
|
||||
Guid objectType = entityType switch
|
||||
{
|
||||
ContentCacheDataSerializerEntityType.Document => Constants.ObjectTypes.Document,
|
||||
ContentCacheDataSerializerEntityType.Media => Constants.ObjectTypes.Media,
|
||||
ContentCacheDataSerializerEntityType.Member => Constants.ObjectTypes.Member,
|
||||
_ => throw new ArgumentOutOfRangeException(nameof(entityType), entityType, null),
|
||||
};
|
||||
|
||||
IEnumerable<ContentSourceDto> dtos = GetContentSourceByDocumentTypeKey(keys, objectType);
|
||||
|
||||
IContentCacheDataSerializer serializer =
|
||||
_contentCacheDataSerializerFactory.Create(ContentCacheDataSerializerEntityType.Document);
|
||||
_contentCacheDataSerializerFactory.Create(entityType);
|
||||
|
||||
foreach (ContentSourceDto row in dtos)
|
||||
{
|
||||
@@ -269,8 +277,8 @@ AND cmsContentNu.nodeId IS NULL
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IEnumerable<Guid> GetContentKeysByContentTypeKeys(IEnumerable<Guid> keys, bool published = false)
|
||||
=> GetContentSourceByDocumentTypeKey(keys).Where(x => x.Published == published).Select(x => x.Key);
|
||||
public IEnumerable<Guid> GetDocumentKeysByContentTypeKeys(IEnumerable<Guid> keys, bool published = false)
|
||||
=> GetContentSourceByDocumentTypeKey(keys, Constants.ObjectTypes.Document).Where(x => x.Published == published).Select(x => x.Key);
|
||||
|
||||
public async Task<ContentCacheNode?> GetMediaSourceAsync(int id)
|
||||
{
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using Umbraco.Cms.Core.Models;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Serialization;
|
||||
|
||||
namespace Umbraco.Cms.Infrastructure.HybridCache.Persistence;
|
||||
|
||||
@@ -15,14 +16,14 @@ internal interface IDatabaseCacheRepository
|
||||
Task<ContentCacheNode?> GetMediaSourceAsync(Guid key);
|
||||
|
||||
|
||||
IEnumerable<ContentCacheNode> GetContentByContentTypeKey(IEnumerable<Guid> keys);
|
||||
IEnumerable<ContentCacheNode> GetContentByContentTypeKey(IEnumerable<Guid> keys, ContentCacheDataSerializerEntityType entityType);
|
||||
|
||||
/// <summary>
|
||||
/// Gets all content keys of specific document types
|
||||
/// </summary>
|
||||
/// <param name="keys">The document types to find content using.</param>
|
||||
/// <returns>The keys of all content use specific document types.</returns>
|
||||
IEnumerable<Guid> GetContentKeysByContentTypeKeys(IEnumerable<Guid> keys, bool published = false);
|
||||
IEnumerable<Guid> GetDocumentKeysByContentTypeKeys(IEnumerable<Guid> keys, bool published = false);
|
||||
|
||||
/// <summary>
|
||||
/// Refreshes the nucache database row for the given cache node />
|
||||
|
||||
@@ -24,7 +24,7 @@ internal sealed class ContentTypeSeedKeyProvider : IDocumentSeedKeyProvider
|
||||
public ISet<Guid> GetSeedKeys()
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
var documentKeys = _databaseCacheRepository.GetContentKeysByContentTypeKeys(_cacheSettings.ContentTypeKeys, published: true).ToHashSet();
|
||||
var documentKeys = _databaseCacheRepository.GetDocumentKeysByContentTypeKeys(_cacheSettings.ContentTypeKeys, published: true).ToHashSet();
|
||||
scope.Complete();
|
||||
|
||||
return documentKeys;
|
||||
|
||||
@@ -7,6 +7,7 @@ using Umbraco.Cms.Core.Scoping;
|
||||
using Umbraco.Cms.Core.Services;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Factories;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Persistence;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Serialization;
|
||||
using Umbraco.Extensions;
|
||||
|
||||
namespace Umbraco.Cms.Infrastructure.HybridCache.Services;
|
||||
@@ -97,7 +98,7 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
public IEnumerable<IPublishedContent> GetByContentType(IPublishedContentType contentType)
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
IEnumerable<ContentCacheNode> nodes = _databaseCacheRepository.GetContentByContentTypeKey([contentType.Key]);
|
||||
IEnumerable<ContentCacheNode> nodes = _databaseCacheRepository.GetContentByContentTypeKey([contentType.Key], ContentCacheDataSerializerEntityType.Document);
|
||||
scope.Complete();
|
||||
|
||||
return nodes
|
||||
@@ -239,11 +240,11 @@ internal sealed class DocumentCacheService : IDocumentCacheService
|
||||
scope.Complete();
|
||||
}
|
||||
|
||||
public void Rebuild(IReadOnlyCollection<int> contentTypeKeys)
|
||||
public void Rebuild(IReadOnlyCollection<int> contentTypeIds)
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
_databaseCacheRepository.Rebuild(contentTypeKeys.ToList());
|
||||
IEnumerable<ContentCacheNode> contentByContentTypeKey = _databaseCacheRepository.GetContentByContentTypeKey(contentTypeKeys.Select(x => _idKeyMap.GetKeyForId(x, UmbracoObjectTypes.DocumentType).Result));
|
||||
_databaseCacheRepository.Rebuild(contentTypeIds.ToList());
|
||||
IEnumerable<ContentCacheNode> contentByContentTypeKey = _databaseCacheRepository.GetContentByContentTypeKey(contentTypeIds.Select(x => _idKeyMap.GetKeyForId(x, UmbracoObjectTypes.DocumentType).Result), ContentCacheDataSerializerEntityType.Document);
|
||||
|
||||
foreach (ContentCacheNode content in contentByContentTypeKey)
|
||||
{
|
||||
|
||||
@@ -17,7 +17,7 @@ public interface IDocumentCacheService
|
||||
|
||||
Task DeleteItemAsync(IContentBase content);
|
||||
|
||||
void Rebuild(IReadOnlyCollection<int> contentTypeKeys);
|
||||
void Rebuild(IReadOnlyCollection<int> contentTypeIds);
|
||||
|
||||
internal IEnumerable<IPublishedContent> GetByContentType(IPublishedContentType contentType);
|
||||
}
|
||||
|
||||
@@ -16,4 +16,6 @@ public interface IMediaCacheService
|
||||
Task DeleteItemAsync(IContentBase media);
|
||||
|
||||
Task SeedAsync(CancellationToken cancellationToken);
|
||||
|
||||
void Rebuild(IReadOnlyCollection<int> contentTypeIds);
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ using Umbraco.Cms.Core.Scoping;
|
||||
using Umbraco.Cms.Core.Services;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Factories;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Persistence;
|
||||
using Umbraco.Cms.Infrastructure.HybridCache.Serialization;
|
||||
|
||||
namespace Umbraco.Cms.Infrastructure.HybridCache.Services;
|
||||
|
||||
@@ -164,6 +165,26 @@ internal class MediaCacheService : IMediaCacheService
|
||||
scope.Complete();
|
||||
}
|
||||
|
||||
public void Rebuild(IReadOnlyCollection<int> contentTypeIds)
|
||||
{
|
||||
using ICoreScope scope = _scopeProvider.CreateCoreScope();
|
||||
_databaseCacheRepository.Rebuild(contentTypeIds.ToList());
|
||||
|
||||
IEnumerable<Guid> mediaTypeKeys = contentTypeIds.Select(x => _idKeyMap.GetKeyForId(x, UmbracoObjectTypes.MediaType))
|
||||
.Where(x => x.Success)
|
||||
.Select(x => x.Result);
|
||||
|
||||
IEnumerable<ContentCacheNode> mediaCacheNodesByContentTypeKey =
|
||||
_databaseCacheRepository.GetContentByContentTypeKey(mediaTypeKeys, ContentCacheDataSerializerEntityType.Media);
|
||||
|
||||
foreach (ContentCacheNode media in mediaCacheNodesByContentTypeKey)
|
||||
{
|
||||
_hybridCache.RemoveAsync(GetCacheKey(media.Key, false));
|
||||
}
|
||||
|
||||
scope.Complete();
|
||||
}
|
||||
|
||||
private HybridCacheEntryOptions GetSeedEntryOptions() => new()
|
||||
{
|
||||
Expiration = _cacheSettings.SeedCacheDuration, LocalCacheExpiration = _cacheSettings.SeedCacheDuration,
|
||||
|
||||
Reference in New Issue
Block a user