Recommended configuration for ScreamingFrog
In order for Screaming Frog to crawl your site properly, it needs to be set up as follows:
Go to “configuration” -> spider.
On the “Crawl” tab, set the following:
On the “Extraction” tab, set the following:
On the “Limits” tab, set the following:
On the “Rendering” tab, set the following:
(Best for most platforms, such as WordPress, Drupal, Magneto etc):
For JS-based sites (Angular, React, Vue etc):
On the “Advanced” tab, set the following:
On the “Preferences” tab, set the following:
Go to Configuration -> System –> Memory
Look at the number that appears and set it at 2 GB less than the maximum.
Click “OK.” DO NOT RESTART Screaming Frog YET!
Go to Configuration -> System -> Storage
Set mode to “Database Storage”
Click OK – DO NOT RESTART Screaming frog YET!
Go to File -> Configuration -> Save current configuration as default
Now you can close and restart Screaming Frog. Your configuration will be active.
That’s all! You are ready to start crawling!