Commit Graph

41204 Commits

Author SHA1 Message Date
Paul Johnson
3961c4c233 v10 SQLite support + distributed locking abstractions (#11922)
* Created Persistence.SQLite project skeleton.

* SQLite database initialization

* Various changes and hacks to make things work.

* WIP integration tests

* Fix thread safety tests

* Fix tests that relied on tie breaker sorting.

Spent a fair amount of time looking for a less lazy fix but gave up.

* Convert right join to left join ContentTypeRepository.PerformGetByQuery
SQLite doesn't support right join

* Fix test Can_Generate_Delete_SubQuery_Statement

Worth noting that NPoco.DatabaseTypes.SQLiteDatabaseType doesn't override
EscapeSqlIdentifier so NPoco will escape with [].

SQLite docs say > "A keyword enclosed in square brackets is an identifier.
This is not standard SQL.
This quoting mechanism is used by MS Access and SQL Server and is
included in SQLite for compatibility."

Also could have updated SqliteSyntaxProvider to match npoco but
decided against it.

* Fixes for paginated custom order by

* Fix tests broken by lack of unique indexes.

* Fix SqlServerTableByTableTest tests.

These tests didn't actually do anything as the tables already exist so schema creator just returned.

Did however point out that the default implementation for DoesTableExist just returns false so added a default naive implementation.

* Fix ValidateLoginSession - SelectTop must come later

* dry up database cleanup

* Fix up db migration tests.
We can't drop pk in sqlite without recreating table.
Test looks to be testing that add column works as intended which we can test.

* Prevent schema creation errors.

* SQLite ignore lock tests, WAL back on.

* Fix package schema tests

* Fix NPocoFetchTests - case sensitivity not under test

* Fix AdvancedMigrationTests (where possible)

Migrations probably need a good look later.
Maybe nuke old migrations and only support moving to v10 from v9.

If we do that can do some cleanup.

* Cleanup test database configuration

* Run integration tests against SQLite on build agent.

* Drop MS.Data.SQLite
System.Data.SQLite was quicker to roll out due to more CLR type mapping

* YAML

* Skip Umbraco.Tests.Integration.SqlCe

* Drop SqlServerTableByTable tests.

Until this week they did nothing anyway as they with NewSchemaPerTest
so the tests all passed as CreateTable was no op (already exists).

Also all of the tables are created in an empty database by SchemaValidationTest.cs
DatabaseSchemaCreation_Produces_DatabaseSchemaResult_With_Zero_Errors

* Might aswell run against macOS also.

* Copy azure pipelines task header layout

* Delete SQLCe projects

* Remove SQL CE specific code.

* Remove SQL CE NuSpec, template params, build script setup

* Delete umbraco-netcore-only.sln

* Add SkipTests solution configuration and use for codeql

* Remove reference to deleted nuspec file.

* Refactor ConnectionStrings WRT DataDirectory placeholder & ProviderName.

At this point you can try out SQLite support by setting the following
in appsettings.json and then completing the install process.

"ConnectionStrings": {
  "umbracoDbDSN": "Data Source=|DataDirectory|/umbraco.sqlite",
  "umbracoDbDSN_ProviderName": "System.Data.SQLite"
},

Not currently possible via installer UI without provider name pre-set in
configuration.

* Switch to Microsoft.Data.Sqlite

Some gross hacks but will be good to find out if this works
with apple silicon.

* Enable selection of SQLite via installer UI (also quick install)

* Remove SqlServerDbProviderFactoryCreator to cleanup a TODO

* Move SQL Server support to its own class library

* Add persistence dependencies to Umbraco.CMS metapackage

* Bugfix packages delete query

Created invalid query for SQLite.

* Try out cypress tests Linux + SQLite

* Prevent cypress test artifact upload failure on attempt 2+

* LocalDb bugfixes

* Drop redundant enum

* Move SqlClient constant

* Misc whitespace

* Remove IsSqlCe extension (TODO: drop non 9->10 migrations later).

* Umbraco.Persistence.* -> Umbraco.Cms.Persistence.*

* Display quick install defaults and per provider default database name.

* Misc remove old comment

* little re-arrange

* Remove almost all usages of IsSqlite extension.

* visual adjustments

* Custom Database Configuration is last step and should then say Install.

* use text instead of disabled inputs

* move legend, rename to Install

* Update SqlMainDomLock to work without distributed locks.

* Added IDistributedLockingMechanism interface and in memory impl.

* Drop locking from ISqlSyntaxProvider & wire up scope to abstraction.

* Added SqlServerDistributedLockingMechanism

* Move distributed locking interfaces and exceptions to Core + xmldocs.

* Fix tests, Misc cleanup, Add SQL distributed locking integration tests

* Provide mechanism to specify DistributedLockingMechanism in config
(even if added by composer)

* Nomplementation -> NoImplementation

* Fix misleading comment

* Integration tests use SqlServerDistributedLockingMechanism when possible

* Handle up-gradable locks SqlServerDistributedLockingMechanism.
TODO: InMemoryDistributedLockingMechanism.

Note: Nuked SqlServerDistributedLockingMechanismTests, will still sleep
at night.
Is covered by Umbraco.Cms.Tests.Integration.Umbraco.Infrastructure.Persistence.LockTests

* Make tests pass for InMemoryDistributedLockingMechanism, pretty hacky.

* Tweak constraints on WithCollectionBuilder so i can drop bad constructor

* Added SqliteDistributedLockingMechanism

* Dropped InMemoryDistributedMechanism + magic

InMemoryDistributedMechanism was pretty rubbish and now we have
a decent implementation for SQLite as we no longer block readers
see 8d1f42b.

Also drop the CollectionBuilder setup, instead do the same as we do
for syntax providers etc, it's more automagical so we never require an
explicit selection although we are allowing for it.

However keeping the optional IUmbracoBuilder constructor param for
CollectionBuilders as it's extremely useful.

* Fix quick install "" database name.

* Hide Database Configuration section when a connection string is pre-set.

Doesn't seem worth it to extract db name from connection string.

* Ensure wal test 2+

* Fix logging inconsistencies.

* Ensure in transaction when obtaining locks + no-op the SQLite read lock.

There's no point in running the query just to make a single test pass.

* Fix installer database display names

* Allow SQLite shared cache without losing deferred transactions

* Opt into shared cache for new SQLite databases + fix filename

* Fix misc inconsistency in .gitignore

* Prefer our interceptor interface

* Restore DEBUG_DATABASES code OnConnectionOpened in case it's used.

* Back to private cache.

* Added retry strategy for SQLite + refactor out SQL server specific stuff

* Fix SQL server tests.

* Misc - Orphaned comment, incorrect casing.

* InMemory SQLite test database & turn shared cache back on everywhere.

Co-authored-by: Niels Lyngsø <niels.lyngso@gmail.com>
2022-03-11 17:14:20 +01:00
Bjarke Berg
a30d24130f Merge pull request #12126 from umbraco/v10/feature/merge_v9.4rc_to_v10
Merge v9.4rc to v10
2022-03-11 11:06:02 +01:00
Bjarke Berg
78b85a44db Merge remote-tracking branch 'origin/v9/dev' into v10/dev 2022-03-10 22:10:00 +01:00
Paul Johnson
cc05428640 fix additional legacy password formats (#12120)
* Added failing test to demo issue.

* Handle old machine key default.
2022-03-10 14:59:19 +01:00
Bjarke Berg
981e66a673 Merge remote-tracking branch 'origin/v9/dev' into v10/dev
# Conflicts:
#	src/Umbraco.Core/Telemetry/TelemetryService.cs
#	src/Umbraco.Infrastructure/DependencyInjection/UmbracoBuilder.CoreServices.cs
#	src/Umbraco.Infrastructure/DependencyInjection/UmbracoBuilder.Services.cs
#	src/Umbraco.Infrastructure/HostedServices/ScheduledPublishing.cs
#	src/Umbraco.Infrastructure/Migrations/Upgrade/UmbracoPlan.cs
#	src/Umbraco.Infrastructure/Runtime/SqlMainDomLock.cs
#	src/Umbraco.Web.BackOffice/Controllers/ContentController.cs
#	src/Umbraco.Web.BackOffice/Controllers/DashboardController.cs
#	tests/Umbraco.Tests.UnitTests/Umbraco.Infrastructure/HostedServices/ScheduledPublishingTests.cs
2022-03-10 14:36:30 +01:00
Nikolaj Geisle
1ac77680e0 Merge pull request #12114 from umbraco/v9/feature/move-created-packages
Move created packages out of webroot
2022-03-09 15:20:05 +01:00
Nikolaj
fbb80c4a94 Merge remote-tracking branch 'origin/v9/9.4' into v9/dev 2022-03-09 14:50:47 +01:00
Ronald Barendse
7e8f23e514 Properly clean existing package files 2022-03-09 11:38:52 +01:00
Ronald Barendse
3afad96aa7 Move created packages out of webroot 2022-03-09 11:06:30 +01:00
Andy Butland
5d9e2b0bb8 Updates JSON schema to reference latest Forms and Deploy. (#12106) 2022-03-09 09:19:34 +01:00
Bjarke Berg
c20f45dcfc Fixed issue with pageindex (0-indexed) vs pagenumber (1 indexed) (#12113) 2022-03-09 08:53:18 +01:00
Andy Butland
215bf69cb0 Updates JSON schema to reference latest Forms and Deploy. (#12106) 2022-03-08 16:33:43 +01:00
Mole
fff3e5e9d4 Fix saving content not updating the page (#12108)
* Fix relation type tests

* Fix saving content not updating the UI

We have to map to display after we've saved content or we'll be missing information generated when saving content such as ID

* Dont map to display to get isBlueprint
2022-03-08 11:33:22 +01:00
Mole
413a278373 V9: Fix missing EditedCultures and Name in content saving notifications (#12070)
* Map dirty culture to EditedCultures before saving

And set the name of the content when setting default culture variant name

* Update src/Umbraco.Web.BackOffice/Controllers/ContentController.cs

Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>

Co-authored-by: nikolajlauridsen <nel@umbraco.dk>
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
2022-03-08 09:23:44 +01:00
Bjarke Berg
5d6fc6005c Unbreak breaking change 2022-03-08 07:45:53 +01:00
Elitsa Marinovska
d9d5dc5626 Item tracking improvements (#11919)
* Adding TrackedReferencesController

* Adding/changing views

* Adding/Editing js files

* RelationService changes

* RelationRepository changes

* Adding missing translations

* Adding/Modifying tests

* Beginning of #9125 migration

* Introducing a new component + refactoring based on that

* Abstracting + refactoring

* Work on content unpublishing

* Work on media.delete

* Various small changes

* Beginning of #9119 migration

* Changes on content.delete

* Various fixes

* Adding new keys used in the listview bulk actions

* Adding methods to get the items used in relations from array of ids

* Adding the checkLinkedItems function to the trackedReferencesResource

* Passing the selected items from a listview to unpublish and delete

* Adding umb-tracked-references-listview

* Adding umb-tracked-references-listview-table with language column

* Fixes for tracked references

* Changes in listview unpublish dialog

* Changes in listview delete dialog

* Removing Variants logic as it is not currently supported

* Visual fixes

* Closing dialogs on click

* Fix wording

* Fix breaking changes

* Change to a single title "Items in use" instead of 2 different for Content and Media

* No need for obsoleting because we can change new controllers

* Return ActionResult from actions

* V9:  Prevent delete or unpublish of items that have references (#12047)

* Introducing config settings that prevent delete or unpublish of items referenced by other items

* Disable deletion of content items and show a new warning

* Disable deletion of media items and show a new warning

* Disable deletion of list view items

* Disable unpublish and bulk unpublish

* Add a new warning

* V9: Displaying descendants in use as part of item tracking (#12039)

* Replace HasReferencesInDescendants with GetPagedDescendantsInReferences

* Display descendants in use on parent's info tab

* Add getPagedDescendantsInReferences to trackedReferencesResource

* Add lang keys for Descendants in use

* Refactoring controller actions

* Don't call check descendants usage when it is a new item

* rename busfy to busy

* always show references

* rearrange for scrollbar to appear at the edge of the dialog

* use the word referenced instead of used

* change fallback texts

* Added "IsDependency" to relation types

* refactor of umb-tracked-references

* rename checkLinkedItems to getPagedReferencedItems

* rename check to load, to be consistent with the rest.

* Refactored backend . Needs frontend fixes

* Cleanup

* Use filters

* Front-end refactor to match refactored end-points

* Fixed bug + warning

* Fixed query (more then 2100 descensdants) and optimized it (using nested select instead of inner join).

* remove comment

* hideNoneDependencies including varying text for the configuration

* Hack for SqlCE :(

* some final adjustments for item tracking ui

* Unbreak change

Co-authored-by: Niels Lyngsø <niels.lyngso@gmail.com>
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2022-03-07 22:46:16 +01:00
Bjarke Berg
2ff2ae3f0a Unbreak a change.. 2022-03-07 22:14:21 +01:00
Bjarke Berg
cfacf4e150 Rollback unnecessary breaking change 2022-03-07 22:07:50 +01:00
Sebastiaan Janssen
d8551e4640 Merge branch 'v9/dev' into v9/contrib 2022-03-07 11:41:25 +01:00
Owain Williams
373138d70f Make GridPropertyEditor dirty if a control is removed
This is the same fix that @inetzo made for a PR I checked earlier today except the fix was for U8.  Since I didn't notice it was U8 until I had merged, I thought I'd fix it here but credit should go to @inetzo 

#12097
2022-03-05 17:24:19 +00:00
Callum Whyte
10d252819d Adding null check to IsCulturePublished on IContent 2022-03-05 03:05:31 +01:00
Paul Johnson
ca5c85e656 v9 fix misc issues external member login (#12093)
* Add missing override for SetTokenAsync

* Fix mismatch between expected scheme prefix and exception message

* Store tokens on member update
2022-03-03 17:18:39 +00:00
Anders Bjerner
a6aaf4b0b1 Introduced new "large" overlay size (#11945)
* Introduced new "large" overlay size (1/2)

* Introduced new "large" overlay size (2/2)

* #H5IS

Co-authored-by: Owain <owilliams@spindogs.com>
2022-03-03 12:53:58 +00:00
Matt Brailsford
44e8808079 Check form and querystring when validating ufprt in ValidateUmbracoFormRouteStringAttribute (#11957)
* Check form and querystring when validating ufprt

Checks to see if the request has form data before validating the `ufprt` parameter, and if it doesn't assumes it must be on the querystring

* Create GetUfprt extension method

* Use GetUfprt extension

* Update UmbracoRouteValueTransformer to use GetUfrpt()

* Added missing using statement

* Check for StringValues.Empty
2022-03-03 10:42:14 +00:00
dependabot[bot]
59b0224eff Bump karma from 6.3.14 to 6.3.16 in /src/Umbraco.Web.UI.Client (#12090)
Bumps [karma](https://github.com/karma-runner/karma) from 6.3.14 to 6.3.16.
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v6.3.14...v6.3.16)

---
updated-dependencies:
- dependency-name: karma
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-02 15:06:45 +01:00
Nikolaj Geisle
65723ea9ec Merge pull request #12087 from umbraco/v9/bugfix/history-cleanup-make-contenttype-dirty
V9: Fix history cleanup not making content type dirty
2022-03-02 12:23:24 +01:00
Nikolaj Geisle
50a239a016 Merge pull request #12081 from umbraco/v9/bugfix/fix-invariant-redirect-tracking
Fix invariant redirect tracking
2022-03-02 12:20:15 +01:00
nikolajlauridsen
5ef013ca62 Ensure ContentType is only marked as dirty when it's actually changed 2022-03-02 10:49:06 +01:00
Mole
ff477c3930 Update src/Umbraco.Infrastructure/Routing/RedirectTrackingHandler.cs
Co-authored-by: Nikolaj Geisle <70372949+Zeegaan@users.noreply.github.com>
2022-03-02 10:26:23 +01:00
nikolajlauridsen
1cdc6f0fd2 Make HistoryCleanup register as dirty in ContentType 2022-03-02 10:15:50 +01:00
nikolajlauridsen
49e0e6c1c9 Implement BeingDirtyBase on HistoryCleanup 2022-03-02 10:11:48 +01:00
nikolajlauridsen
7e9def9df4 Add tests 2022-03-02 10:11:33 +01:00
Nikolaj Geisle
009b36b49e Send key from frontend to postsave (#12080)
Co-authored-by: Nikolaj Geisle <niko737@edu.ucl.dk>
2022-03-02 09:12:07 +01:00
Nikolaj Geisle
44ddf7acef Add extra check (#12075)
Co-authored-by: Nikolaj Geisle <niko737@edu.ucl.dk>
2022-03-02 08:47:35 +01:00
Chad Currie
6b3a589aa0 Check Debug logs are enabled for hot path 2022-03-02 02:06:52 +01:00
nikolajlauridsen
95f1ed33ec Make empty string invariant culture in DefaultUrlProvider 2022-03-01 14:14:48 +01:00
nikolajlauridsen
993c582bd9 Handle empty string as invariant when generating cache key 2022-03-01 13:51:21 +01:00
Bjarne Fyrstenborg
a72698e1a9 Transition of lazy loaded image 2022-03-01 17:16:32 +10:00
Martin Bentancour
c905fdd9b5 Validate that imageUrl param is a relative path (#11606)
* Validate that imageUrl param is a relative path

To prevent open redirects, the imagePath should point to a relative path (i.e. not point to a different domain).

* LocalRedirectResult instead of RedirectResult

Uri.Relative does not prevent paths like //google.com/test.jpg
Checking for both relative and local seems to cover all the cases.
2022-02-28 23:59:18 +01:00
Anders Bjerner
34bfd3a8f3 Fixes issue with OrderByDescending generating invalid SQL (#11673)
* Fixes issue with OrderByDescending generating invalid SQL

* Update OrderByDexcending with single field for consistecy

Co-authored-by: Michael <michael@crossingpaths.be>
2022-02-28 23:06:59 +01:00
nikolajlauridsen
dfbb182a94 Handle invariant culture in RedirectTrackingHandler
GetRouteById and RedirectUrlService expects the culture to be null if it's invariant, however, Cultures in IPublished content uses empty string for invariant culture
2022-02-28 17:29:11 +01:00
Corné Hoskam
aba4f02bad (umbraCollab) Check media Parent for permissions when setting correct MediaType (#11858) 2022-02-28 16:29:23 +01:00
Erik-Jan Westendorp
86b7ea0145 v10: Get all ServerRegistrations (#12072) 2022-02-28 15:54:19 +01:00
patrickdemooij9
a27da0fced Content app for dictionary items (#11553) 2022-02-28 15:01:18 +01:00
Mole
0c7ef06031 V9: Fix missing site identifier (#12040)
* Add SiteIdentifierService

* Use SiteIdentifierService in TelemetryService

* Use SiteIdentifierService when installing

* Remove timeout

* Use TryGetOrCreateSiteIdentifier in TelemetryService

* Add site identifier to dashboard url

* Fix and add tests

* Don't accept empty guid as valid site identifier

* Fix dashboard controller

* Fix site id query parameter

* Use Optionsmonitor onchange

Co-authored-by: nikolajlauridsen <nel@umbraco.dk>
Co-authored-by: Bjarke Berg <mail@bergmania.dk>
2022-02-28 13:59:39 +01:00
Mehmet
e6d6dce3a8 Fix mssing image caption in RTE (#11807) 2022-02-28 13:17:17 +01:00
Blake Irwin
24d2dcf363 Fix JsonConfigManipulator causing upgrades to fail when appsettings.json files are not present (#11222) 2022-02-28 10:52:03 +01:00
Scott Brady
25ea5cdb1b Prevent non-backoffice auth schemes being overridden (#11630) 2022-02-28 10:40:51 +01:00
Bjarne Fyrstenborg
cf3d697bc5 Update RemoveTreeController to use TController (#11795) 2022-02-28 09:36:35 +01:00
Warren Buckley
bd6334ecdc Dependancy Update: Switch to Serilog.Expressions away from deprecated Serilog.Filters.Expressions (#12000)
* Uninstall Serilog.Filters.Expressions as it is deprecated by the Serilog team

NOTE: This package brought in a dependenacy on a package called SuperPower from Datalust

* Install replacement package Serilog.Expressions

* Update implementation of Serilog.Expressions

* Add new test cases to verify expressions still working

Currently Serilog.Expressions only supports short names such as @x @l @m and @mt as opposed @Exception @Level @Message and @MessageTemplate

* Use Serilog.Expressions NamedFilters extension point, to plug the missing Has() function in expressions that was not ported across by the library

* Update to dev build of Serilog.Expressions to verify new method we can override works

* Update to release build of Serilog.Expressions 3.3.0 now its been pushed to Nuget
2022-02-28 08:48:10 +01:00