Examine 2.0 integration (#10241)
* Init commit for examine 2.0 work, most old umb examine tests working, probably a lot that doesn't
* Gets Umbraco Examine tests passing and makes some sense out of them, fixes some underlying issues.
* Large refactor, remove TaskHelper, rename Notifications to be consistent, Gets all examine/lucene indexes building and startup ordered in the correct way, removes old files, creates new IUmbracoIndexingHandler for abstracting out all index operations for umbraco data, abstracts out IIndexRebuilder, Fixes Stack overflow with LiveModelsProvider and loading assemblies, ports some changes from v8 for startup handling with cold boots, refactors out LastSyncedFileManager
* fix up issues with rebuilding and management dashboard.
* removes old files, removes NetworkHelper, fixes LastSyncedFileManager implementation to ensure the machine name is used, fix up logging with cold boot state.
* Makes MainDom safer to use and makes PublishedSnapshotService lazily register with MainDom
* lazily acquire application id (fix unit tests)
* Fixes resource casing and missing test file
* Ensures caches when requiring internal services for PublishedSnapshotService, UseNuCache is a separate call, shouldn't be buried in AddWebComponents, was also causing issues in integration tests since nucache was being used for the Id2Key service.
* For UmbracoTestServerTestBase enable nucache services
* Fixing tests
* Fix another test
* Fixes tests, use TestHostingEnvironment, make Tests.Common use net5, remove old Lucene.Net.Contrib ref.
* Fixes up some review notes
* Fixes issue with doubly registering PublishedSnapshotService meanig there could be 2x instances of it
* Checks for parseexception when executing the query
* Use application root instead of duplicating functionality.
* Added Examine project to netcore only solution file
* Fixed casing issue with LazyLoad, that is not lowercase.
* uses cancellationToken instead of bool flag, fixes always reading lastId from the LastSyncedFileManager, fixes RecurringHostedServiceBase so that there isn't an overlapping thread for the same task type
* Fix tests
* remove legacy test project from solution file
* Fix test
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2021-05-18 18:31:38 +10:00
|
|
|
using System;
|
|
|
|
|
using System.Collections.Generic;
|
|
|
|
|
using Umbraco.Cms.Core;
|
|
|
|
|
using Umbraco.Cms.Core.Cache;
|
|
|
|
|
using Umbraco.Cms.Core.Events;
|
|
|
|
|
using Umbraco.Cms.Core.Models;
|
|
|
|
|
using Umbraco.Cms.Core.Notifications;
|
|
|
|
|
using Umbraco.Cms.Core.Services;
|
|
|
|
|
using Umbraco.Cms.Core.Services.Changes;
|
|
|
|
|
using Umbraco.Cms.Core.Sync;
|
|
|
|
|
using Umbraco.Extensions;
|
|
|
|
|
|
|
|
|
|
namespace Umbraco.Cms.Infrastructure.Search
|
|
|
|
|
{
|
|
|
|
|
public sealed class ContentIndexingNotificationHandler : INotificationHandler<ContentCacheRefresherNotification>
|
|
|
|
|
{
|
|
|
|
|
private readonly IUmbracoIndexingHandler _umbracoIndexingHandler;
|
|
|
|
|
private readonly IContentService _contentService;
|
|
|
|
|
|
|
|
|
|
public ContentIndexingNotificationHandler(IUmbracoIndexingHandler umbracoIndexingHandler, IContentService contentService)
|
|
|
|
|
{
|
|
|
|
|
_umbracoIndexingHandler = umbracoIndexingHandler ?? throw new ArgumentNullException(nameof(umbracoIndexingHandler));
|
|
|
|
|
_contentService = contentService ?? throw new ArgumentNullException(nameof(contentService));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// <summary>
|
|
|
|
|
/// Updates indexes based on content changes
|
|
|
|
|
/// </summary>
|
|
|
|
|
/// <param name="sender"></param>
|
|
|
|
|
/// <param name="args"></param>
|
|
|
|
|
public void Handle(ContentCacheRefresherNotification args)
|
|
|
|
|
{
|
|
|
|
|
if (!_umbracoIndexingHandler.Enabled)
|
|
|
|
|
{
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
if (Suspendable.ExamineEvents.CanIndex == false)
|
|
|
|
|
{
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (args.MessageType != MessageType.RefreshByPayload)
|
|
|
|
|
{
|
|
|
|
|
throw new NotSupportedException();
|
|
|
|
|
}
|
|
|
|
|
|
2021-09-09 16:04:18 -06:00
|
|
|
// Used to track permanent deletions so we can bulk delete from the index
|
|
|
|
|
// when needed. For example, when emptying the recycle bin, else it will
|
|
|
|
|
// individually update the index which will be much slower.
|
2022-02-22 13:35:32 +01:00
|
|
|
HashSet<int>? deleteBatch = null;
|
2021-09-09 16:04:18 -06:00
|
|
|
|
Examine 2.0 integration (#10241)
* Init commit for examine 2.0 work, most old umb examine tests working, probably a lot that doesn't
* Gets Umbraco Examine tests passing and makes some sense out of them, fixes some underlying issues.
* Large refactor, remove TaskHelper, rename Notifications to be consistent, Gets all examine/lucene indexes building and startup ordered in the correct way, removes old files, creates new IUmbracoIndexingHandler for abstracting out all index operations for umbraco data, abstracts out IIndexRebuilder, Fixes Stack overflow with LiveModelsProvider and loading assemblies, ports some changes from v8 for startup handling with cold boots, refactors out LastSyncedFileManager
* fix up issues with rebuilding and management dashboard.
* removes old files, removes NetworkHelper, fixes LastSyncedFileManager implementation to ensure the machine name is used, fix up logging with cold boot state.
* Makes MainDom safer to use and makes PublishedSnapshotService lazily register with MainDom
* lazily acquire application id (fix unit tests)
* Fixes resource casing and missing test file
* Ensures caches when requiring internal services for PublishedSnapshotService, UseNuCache is a separate call, shouldn't be buried in AddWebComponents, was also causing issues in integration tests since nucache was being used for the Id2Key service.
* For UmbracoTestServerTestBase enable nucache services
* Fixing tests
* Fix another test
* Fixes tests, use TestHostingEnvironment, make Tests.Common use net5, remove old Lucene.Net.Contrib ref.
* Fixes up some review notes
* Fixes issue with doubly registering PublishedSnapshotService meanig there could be 2x instances of it
* Checks for parseexception when executing the query
* Use application root instead of duplicating functionality.
* Added Examine project to netcore only solution file
* Fixed casing issue with LazyLoad, that is not lowercase.
* uses cancellationToken instead of bool flag, fixes always reading lastId from the LastSyncedFileManager, fixes RecurringHostedServiceBase so that there isn't an overlapping thread for the same task type
* Fix tests
* remove legacy test project from solution file
* Fix test
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2021-05-18 18:31:38 +10:00
|
|
|
foreach (var payload in (ContentCacheRefresher.JsonPayload[])args.MessageObject)
|
|
|
|
|
{
|
|
|
|
|
if (payload.ChangeTypes.HasType(TreeChangeTypes.Remove))
|
|
|
|
|
{
|
2021-09-09 16:04:18 -06:00
|
|
|
if (deleteBatch == null)
|
|
|
|
|
{
|
|
|
|
|
deleteBatch = new HashSet<int>();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
deleteBatch.Add(payload.Id);
|
Examine 2.0 integration (#10241)
* Init commit for examine 2.0 work, most old umb examine tests working, probably a lot that doesn't
* Gets Umbraco Examine tests passing and makes some sense out of them, fixes some underlying issues.
* Large refactor, remove TaskHelper, rename Notifications to be consistent, Gets all examine/lucene indexes building and startup ordered in the correct way, removes old files, creates new IUmbracoIndexingHandler for abstracting out all index operations for umbraco data, abstracts out IIndexRebuilder, Fixes Stack overflow with LiveModelsProvider and loading assemblies, ports some changes from v8 for startup handling with cold boots, refactors out LastSyncedFileManager
* fix up issues with rebuilding and management dashboard.
* removes old files, removes NetworkHelper, fixes LastSyncedFileManager implementation to ensure the machine name is used, fix up logging with cold boot state.
* Makes MainDom safer to use and makes PublishedSnapshotService lazily register with MainDom
* lazily acquire application id (fix unit tests)
* Fixes resource casing and missing test file
* Ensures caches when requiring internal services for PublishedSnapshotService, UseNuCache is a separate call, shouldn't be buried in AddWebComponents, was also causing issues in integration tests since nucache was being used for the Id2Key service.
* For UmbracoTestServerTestBase enable nucache services
* Fixing tests
* Fix another test
* Fixes tests, use TestHostingEnvironment, make Tests.Common use net5, remove old Lucene.Net.Contrib ref.
* Fixes up some review notes
* Fixes issue with doubly registering PublishedSnapshotService meanig there could be 2x instances of it
* Checks for parseexception when executing the query
* Use application root instead of duplicating functionality.
* Added Examine project to netcore only solution file
* Fixed casing issue with LazyLoad, that is not lowercase.
* uses cancellationToken instead of bool flag, fixes always reading lastId from the LastSyncedFileManager, fixes RecurringHostedServiceBase so that there isn't an overlapping thread for the same task type
* Fix tests
* remove legacy test project from solution file
* Fix test
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2021-05-18 18:31:38 +10:00
|
|
|
}
|
|
|
|
|
else if (payload.ChangeTypes.HasType(TreeChangeTypes.RefreshAll))
|
|
|
|
|
{
|
|
|
|
|
// ExamineEvents does not support RefreshAll
|
|
|
|
|
// just ignore that payload
|
|
|
|
|
// so what?!
|
|
|
|
|
|
|
|
|
|
// TODO: Rebuild the index at this point?
|
|
|
|
|
}
|
|
|
|
|
else // RefreshNode or RefreshBranch (maybe trashed)
|
|
|
|
|
{
|
2021-09-09 16:04:18 -06:00
|
|
|
if (deleteBatch != null && deleteBatch.Contains(payload.Id))
|
|
|
|
|
{
|
|
|
|
|
// the same node has already been deleted, to ensure ordering is
|
|
|
|
|
// handled, we'll need to execute all queued deleted items now
|
|
|
|
|
// and reset the deleted items list.
|
|
|
|
|
_umbracoIndexingHandler.DeleteIndexForEntities(deleteBatch, false);
|
|
|
|
|
deleteBatch = null;
|
|
|
|
|
}
|
|
|
|
|
|
Examine 2.0 integration (#10241)
* Init commit for examine 2.0 work, most old umb examine tests working, probably a lot that doesn't
* Gets Umbraco Examine tests passing and makes some sense out of them, fixes some underlying issues.
* Large refactor, remove TaskHelper, rename Notifications to be consistent, Gets all examine/lucene indexes building and startup ordered in the correct way, removes old files, creates new IUmbracoIndexingHandler for abstracting out all index operations for umbraco data, abstracts out IIndexRebuilder, Fixes Stack overflow with LiveModelsProvider and loading assemblies, ports some changes from v8 for startup handling with cold boots, refactors out LastSyncedFileManager
* fix up issues with rebuilding and management dashboard.
* removes old files, removes NetworkHelper, fixes LastSyncedFileManager implementation to ensure the machine name is used, fix up logging with cold boot state.
* Makes MainDom safer to use and makes PublishedSnapshotService lazily register with MainDom
* lazily acquire application id (fix unit tests)
* Fixes resource casing and missing test file
* Ensures caches when requiring internal services for PublishedSnapshotService, UseNuCache is a separate call, shouldn't be buried in AddWebComponents, was also causing issues in integration tests since nucache was being used for the Id2Key service.
* For UmbracoTestServerTestBase enable nucache services
* Fixing tests
* Fix another test
* Fixes tests, use TestHostingEnvironment, make Tests.Common use net5, remove old Lucene.Net.Contrib ref.
* Fixes up some review notes
* Fixes issue with doubly registering PublishedSnapshotService meanig there could be 2x instances of it
* Checks for parseexception when executing the query
* Use application root instead of duplicating functionality.
* Added Examine project to netcore only solution file
* Fixed casing issue with LazyLoad, that is not lowercase.
* uses cancellationToken instead of bool flag, fixes always reading lastId from the LastSyncedFileManager, fixes RecurringHostedServiceBase so that there isn't an overlapping thread for the same task type
* Fix tests
* remove legacy test project from solution file
* Fix test
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2021-05-18 18:31:38 +10:00
|
|
|
// don't try to be too clever - refresh entirely
|
|
|
|
|
// there has to be race conditions in there ;-(
|
|
|
|
|
|
|
|
|
|
var content = _contentService.GetById(payload.Id);
|
|
|
|
|
if (content == null)
|
|
|
|
|
{
|
|
|
|
|
// gone fishing, remove entirely from all indexes (with descendants)
|
|
|
|
|
_umbracoIndexingHandler.DeleteIndexForEntity(payload.Id, false);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
2022-02-22 13:35:32 +01:00
|
|
|
IContent? published = null;
|
Examine 2.0 integration (#10241)
* Init commit for examine 2.0 work, most old umb examine tests working, probably a lot that doesn't
* Gets Umbraco Examine tests passing and makes some sense out of them, fixes some underlying issues.
* Large refactor, remove TaskHelper, rename Notifications to be consistent, Gets all examine/lucene indexes building and startup ordered in the correct way, removes old files, creates new IUmbracoIndexingHandler for abstracting out all index operations for umbraco data, abstracts out IIndexRebuilder, Fixes Stack overflow with LiveModelsProvider and loading assemblies, ports some changes from v8 for startup handling with cold boots, refactors out LastSyncedFileManager
* fix up issues with rebuilding and management dashboard.
* removes old files, removes NetworkHelper, fixes LastSyncedFileManager implementation to ensure the machine name is used, fix up logging with cold boot state.
* Makes MainDom safer to use and makes PublishedSnapshotService lazily register with MainDom
* lazily acquire application id (fix unit tests)
* Fixes resource casing and missing test file
* Ensures caches when requiring internal services for PublishedSnapshotService, UseNuCache is a separate call, shouldn't be buried in AddWebComponents, was also causing issues in integration tests since nucache was being used for the Id2Key service.
* For UmbracoTestServerTestBase enable nucache services
* Fixing tests
* Fix another test
* Fixes tests, use TestHostingEnvironment, make Tests.Common use net5, remove old Lucene.Net.Contrib ref.
* Fixes up some review notes
* Fixes issue with doubly registering PublishedSnapshotService meanig there could be 2x instances of it
* Checks for parseexception when executing the query
* Use application root instead of duplicating functionality.
* Added Examine project to netcore only solution file
* Fixed casing issue with LazyLoad, that is not lowercase.
* uses cancellationToken instead of bool flag, fixes always reading lastId from the LastSyncedFileManager, fixes RecurringHostedServiceBase so that there isn't an overlapping thread for the same task type
* Fix tests
* remove legacy test project from solution file
* Fix test
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2021-05-18 18:31:38 +10:00
|
|
|
if (content.Published && _contentService.IsPathPublished(content))
|
|
|
|
|
{
|
|
|
|
|
published = content;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (published == null)
|
|
|
|
|
{
|
|
|
|
|
_umbracoIndexingHandler.DeleteIndexForEntity(payload.Id, true);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// just that content
|
|
|
|
|
_umbracoIndexingHandler.ReIndexForContent(content, published != null);
|
|
|
|
|
|
|
|
|
|
// branch
|
|
|
|
|
if (payload.ChangeTypes.HasType(TreeChangeTypes.RefreshBranch))
|
|
|
|
|
{
|
|
|
|
|
var masked = published == null ? null : new List<int>();
|
|
|
|
|
const int pageSize = 500;
|
|
|
|
|
var page = 0;
|
|
|
|
|
var total = long.MaxValue;
|
|
|
|
|
while (page * pageSize < total)
|
|
|
|
|
{
|
|
|
|
|
var descendants = _contentService.GetPagedDescendants(content.Id, page++, pageSize, out total,
|
|
|
|
|
//order by shallowest to deepest, this allows us to check it's published state without checking every item
|
|
|
|
|
ordering: Ordering.By("Path", Direction.Ascending));
|
|
|
|
|
|
|
|
|
|
foreach (var descendant in descendants)
|
|
|
|
|
{
|
|
|
|
|
published = null;
|
|
|
|
|
if (masked != null) // else everything is masked
|
|
|
|
|
{
|
|
|
|
|
if (masked.Contains(descendant.ParentId) || !descendant.Published)
|
|
|
|
|
{
|
|
|
|
|
masked.Add(descendant.Id);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
published = descendant;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
_umbracoIndexingHandler.ReIndexForContent(descendant, published != null);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// NOTE
|
|
|
|
|
//
|
|
|
|
|
// DeleteIndexForEntity is handled by UmbracoContentIndexer.DeleteFromIndex() which takes
|
|
|
|
|
// care of also deleting the descendants
|
|
|
|
|
//
|
|
|
|
|
// ReIndexForContent is NOT taking care of descendants so we have to reload everything
|
|
|
|
|
// again in order to process the branch - we COULD improve that by just reloading the
|
|
|
|
|
// XML from database instead of reloading content & re-serializing!
|
|
|
|
|
//
|
|
|
|
|
// BUT ... pretty sure it is! see test "Index_Delete_Index_Item_Ensure_Heirarchy_Removed"
|
|
|
|
|
}
|
2021-09-09 16:04:18 -06:00
|
|
|
|
|
|
|
|
if (deleteBatch != null)
|
|
|
|
|
{
|
|
|
|
|
// process the delete batch
|
|
|
|
|
_umbracoIndexingHandler.DeleteIndexForEntities(deleteBatch, false);
|
|
|
|
|
}
|
Examine 2.0 integration (#10241)
* Init commit for examine 2.0 work, most old umb examine tests working, probably a lot that doesn't
* Gets Umbraco Examine tests passing and makes some sense out of them, fixes some underlying issues.
* Large refactor, remove TaskHelper, rename Notifications to be consistent, Gets all examine/lucene indexes building and startup ordered in the correct way, removes old files, creates new IUmbracoIndexingHandler for abstracting out all index operations for umbraco data, abstracts out IIndexRebuilder, Fixes Stack overflow with LiveModelsProvider and loading assemblies, ports some changes from v8 for startup handling with cold boots, refactors out LastSyncedFileManager
* fix up issues with rebuilding and management dashboard.
* removes old files, removes NetworkHelper, fixes LastSyncedFileManager implementation to ensure the machine name is used, fix up logging with cold boot state.
* Makes MainDom safer to use and makes PublishedSnapshotService lazily register with MainDom
* lazily acquire application id (fix unit tests)
* Fixes resource casing and missing test file
* Ensures caches when requiring internal services for PublishedSnapshotService, UseNuCache is a separate call, shouldn't be buried in AddWebComponents, was also causing issues in integration tests since nucache was being used for the Id2Key service.
* For UmbracoTestServerTestBase enable nucache services
* Fixing tests
* Fix another test
* Fixes tests, use TestHostingEnvironment, make Tests.Common use net5, remove old Lucene.Net.Contrib ref.
* Fixes up some review notes
* Fixes issue with doubly registering PublishedSnapshotService meanig there could be 2x instances of it
* Checks for parseexception when executing the query
* Use application root instead of duplicating functionality.
* Added Examine project to netcore only solution file
* Fixed casing issue with LazyLoad, that is not lowercase.
* uses cancellationToken instead of bool flag, fixes always reading lastId from the LastSyncedFileManager, fixes RecurringHostedServiceBase so that there isn't an overlapping thread for the same task type
* Fix tests
* remove legacy test project from solution file
* Fix test
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2021-05-18 18:31:38 +10:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|