Note: You need to download a browser add-on tool called XPath Helper. Copy and paste the current page URL ( ) to your browser (e.g., Chrome). Click the pagination section on the webpage and click Loop click single element.To follow through, you may use the link below: Sounds complicated? No worries, let's dive into an example.
#Octoparse xpath pagination how to
Check out What is XPath and how to use it in Octoparse to learn more about using XPath to create the perfect web scraper. TIP: XPath knowledge is not mandatory but is extremely helpful to create a task that does exactly what you need in Octoparse. STEP 2: Revise the XPath of the Pagination in the workflow in Octoparse.
STEP 1: Write/find the XPath of the page element that takes you to the next page (e.g., if you are on page 1, then you would want to click page 2 if you are on page 2, then you would like to click page 3, so on and so forth). Once you have the links generated, Octoparse will go on to scrape all the pages automatically.Įven if the Auto-detect fails to work and page URLs do not show a pattern, you can still manually create a pagination action. If you see a similar pattern to the example above, with only the page number changing in the URLs of the different pages, you can easily batch generate all the page URLs and scrape as many pages as needed. Take a closer look at the web page URLs for the different pages. Use "Batch Generate" to create URLs for all pagesĪn alternative but very effective way to approach scraping multiple pages of a website is to first collect the URLs of all the pages you would need to scrape and build a task using the list of URLs collected. In this case, you can turn to one of the solutions below.Ģ. Auto-detect will sometimes fail to detect pagination links or have the wrong links detected. You can click Check to see the link detected by Octoparse or click Edit to edit the link if it is not correct.Īs we all know, web pages come in many different forms. If Octoparse detects any pagination links on the web page, there will be a pagination option in the Tips panel upon completion of the Auto-detect Process. It can be enabled in your settings to initiate the auto-detect process automatically every time. With the Auto-detect function, Octoparse will automatically scan the web page for data and pagination links. Use "Batch Generate" to create URLs for all pages.Let's explore how you can create a pagination action with no next page button in Octoparse. More specifically, one that requires clicking the numbered links when you want to turn the page, like the ones below. In this tutorial, we will focus on how to create a pagination action when there is no next page button on the page. The first step is to identify the pagination you are dealing with and work from there. If you are seriously looking into scraping a website, you may need to navigate the website's different pages and extract data from each page.
0 kommentar(er)
