Navigating and Waiting

Every Script we write may do this three things:

1. Navigation to some web page.
2. Waiting for something
3. Possible getting a timeout

Navigating

Initial navigation to any page is pretty much the same for both frameworks and can happen in multiple ways.

• Whenever your code does a page.goto(), or a page.click() on a link, you explicitly trigger navigation.
• The webpage you are on can also trigger navigation by executing location.href= 'https://example.com' or using the history.pushState() API.

In the example below we trigger two navigations:

1. The initial load of the page.
2. Navigate to the shopping cart by clicking a link.

Waiting

In your code, you have a range of options to wait for different things to happen on your browser session

page.waitForSelector()

This method waits for an element to appear on the page. This is your bread and butter and should be used whenever something needs to be loaded after clicking, hovering, navigating, etc. You can pass an object with an timeout attribute to override the default 30 seconds.

page.waitForNavigation()

In your scripts, you can click on a link that triggers navigation to a new page. You can use Puppeteer’s page.waitForNavigation() method here to explicitly wait for this event to happen and then continue your script. The accepted notation in Puppeteer's case is by using the Promise.all() method to wait for the click to happen and the navigation to happen before continuing.

Timeouts

The page.waitForNavigation() method — but also similar methods like page.reload() and page.goBack() — all take some options that determine "how" it should wait and what the timeout limits are.

These options come in two flavors:

1. Hard timeout

The time in milliseconds passed as the timeout property, e.g. page.waitForNavigation({ timeout: 2000 }). We do not recommend using this if you do not explicitly need to.

2a. DOM event-based

These two options are directly related to the events your browser emits when it has reached a certain loading stage. These events are not specific to Puppeteer and are used in almost all browsers.

• load: This is the default and very strict: your whole page including all dependent resources, i.e. images, scripts, CSS, etc.
• domcontentloaded: less strict: when your HTML has loaded.

Note: the load option is the default.

2b. Heuristic-based

These two options are based on the heuristic that if (almost) all network connections your browser has are no longer active, your page has probably finished loading.

• networkidle0: consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
• networkidle2: consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.

Both options 2a and 2b are passed using the waitUntil property, e.g. page.waitForNavigation({ waitUntil: 'networkidle2' })

Which of these options is useful to you depends on your situation:

• Does your SPA need to be fully rendered and finish all XHR calls? Go with load
• You server render and load in some non-crucial element in a lazy fashion? go for one of the networkidle variants.