In order for Screaming Frog to crawl your site properly, it needs to be set up as follows:

Go to “configuration” -> spider.

On the “Crawl” tab, set the following:

On the “Extraction” tab, set the following:

On the “Limits” tab, set the following:

On the “Rendering” tab, set the following:

(Best for most platforms, such as WordPress, Drupal, Magneto etc):

For JS-based sites (Angular, React, Vue etc):

On the “Advanced” tab, set the following:

On the “Preferences” tab, set the following:

Click “OK”

Go to Configuration -> System –> Memory

Look at the number that appears and set it at 2 GB less than the maximum.

Click “OK.” DO NOT RESTART Screaming Frog YET!

Go to Configuration -> System -> Storage

Set mode to “Database Storage”

Click OK – DO NOT RESTART Screaming frog YET!

Go to File -> Configuration -> Save current configuration as default

Now you can close and restart Screaming Frog. Your configuration will be active.

That’s all! You are ready to start crawling!