I found this when searching for something else. This slowness has been a problem in the past. I don't know if the procedure will work.
The link to the procedure is:
https://answers.microsoft.com/en-us/windows/forum/windows_10-files/file-explorer-quick-access-very-slow-to-open/5d8e30ee-0e9c-4b11-bf3d-0118ce259f9e
The author contends that running the following script in an administrative Power Shell will fix it:
Get-AppXPackage -AllUsers | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\AppXManifest.xml"}