Reduce lookups needed in ConcurrentDictionaries and sort using List.Sort, make key removal O(1) by using hashsets and avoid duplicates, remove unneeded .ToList() and other minor tweaks
Reduce lookups needed in ConcurrentDictionaries and sort using List.Sort, make key removal O(1) by using hashsets and avoid duplicates, remove unneeded .ToList() and other minor tweaks