A guide to understanding and managing full-page monitors

Distill supports tracking both parts of a webpage and full page. In this article, we’ll dive deeper into full-page monitors, how to use them, and learn to tackle any challenges you’ll face while setting them up. We cover the following topics, please feel free to navigate to read further.

1. Set up a full-page monitor using the extension
2. Set up a full-page monitor using the Distill Web App
3. Bulk set up: How to set up multiple full-page monitors using CSV import
4. Adding conditions to the monitor (Global conditions and monitor-sepcific conditions)
5. Challenges and troubleshooting steps for issues faced while using full-page monitor
   • Steps for countering false positives
   • Adding a delay to the monitor
   • Handling the cookie pop-ups
   • Not detecting changes in hyperlinks (or other HTML attributes) within the page
   • Other issues
6. Set up full-page monitor using Distill’s Enterprise API

What is a full-page monitor in Distill?

If you’d like to monitor the entire webpage for changes, you can set up a full-page monitor in Distill. It’s quick to set up and advantageous when you want to be notified whenever changes occur on the tracking page.

They are also quite handy when you want to set up monitors for many pages. Throughout this short article, we’ll discuss how, where, and when to use full-page monitors. But first, let’s understand how you can monitor a full-page.

How to set up a full-page monitor

Although the steps for setting up a full-page monitor in the Distill Web App and Extension are similar, we’ve segmented this article to detail instructions for both. Look at both sections to familiarize yourself with the UI and where to access the Monitor full-page option.

Set up a full-page monitor using the extension

Step 1: Navigate to the website you want to track. For example, we will track the All Deals page on Amazon.com.

Step 2: Click on the Distill browser extension on the top right corner of the browser and click the drop-down to select Monitor full-page.

Note: Click the pin icon next to Distill and once pinned it will be accessible from your browser tool bar.

Alternatively, you can also right-click the webpage, select Distill Web Monitor and then select Monitor full-page.

Step 3: You will be led to the options page.

Step 4: Under the options page, you can select the device, set an interval for checking the webpage and configure the alert mode using add action.

Step 5: Click Save to view the monitor on your watchlist.

You’ll have successfully set up a full-page monitor from your extension. Anytime there are changes on the page, Distill will send you an alert.

Set up a full-page monitor using the Distill Web App

Step 1: Log in to the Distill web app and click Add Monitor.

Step 2: Select Webpage.

Step 3: Paste the URL. For this example, we will be using this URL.

Step 4: Click Go.

Step 5: Click More Options (three dots) on the top right side of the panel.

Step 6: Click Select full-page.

Step 7: You will be directed to the options page where you can select the device, set an interval for checking the webpage, and configure the alert mode.

Step 8: Click Save to view the monitor on your watchlist.

Bulk set up: How to set up multiple full-page monitors using CSV import

In many cases, when you have a huge set of webpages to monitor, it will be a tedious task to set up monitors one by one, manually. In this case, you can bulk import the URLs to set up full-page monitors on the set of web pages that you want to track.

The bulk import is an efficient and scalable solution for those who particularly want to set up monitors on several e-commerce product pages and for those tracking their or competitor’s websites for updates.

Step 1: Log in to your Distill web app and click the hamburger option.

Step 2: Click Import and select CSV.

Step 3: Create a .csv file with the monitor’s name in column 1 and the web page’s URL in column 2.

Given below is the template of the .csv file that you will have to import in Distill to create the full-page monitors.

Optional: You can also use labels in the following columns.

Step 4: Click Select CSV file to import monitor and select the saved .csv file.

Step 5: Select the column headers. In this case, we will select Name as the first column header and URL as the second.

Note: The Name in the first column indicates the Name of the monitor, while the URL in the second column indicates the URL of the webpage you want to track.

Step 6: Click Next. You will be directed to the options page to select the device, set an interval for checking the webpage and configure the alert mode.

Step 7: Click import. You will now be able to see the imported monitor under your watchlist.

One key advantage of the bulk import feature is the convenience of setting up full-page monitors by effortlessly importing a list of URLs and automatically creating the monitors for each URL, capturing the entire content and layout of the web pages.

You can refer to this template while preparing your file for import.

Troubleshooting steps for issues during CSV import

Follow the below steps in case you face errors while importing the .csv file:

• Make sure that the URL begins with http:// (or https://), else the import will give a malformed URL error.

• It’s important to note that if Distill displays an error while importing the monitors, they will not be successfully imported. In such cases, you can seek assistance from the Distill team by contacting us on the Distill community forum.

Adding conditions to the monitor

Once you set up your full page monitor, you can add conditions while you configure it. You can use conditions to ensure the monitor notifies you only when the criteria are met. You can use conditions to match keywords, regular expressions, track price points and more. You can even add multiple conditions using AND/ OR logic.

With conditions you can check existing text, newly added text and previous text on the page you are tracking. There are two types of conditions that you can set:

1. Global conditions: This will be checked for all the monitors in your watchlist
2. Conditions within the monitor: This is monitor-specific and will need to be added individually.

Steps to add global conditions

Step 1: Click the hamburger icon and choose Global conditions

Step 2: Click Add Condition.

Step 3: Click Add rule if you are operating on the Version 1 User Interface, or simply begin adding your condition if you are working on the Version 2 User interface.

Step 4: Click Save

Once global conditions are added, it is applied by default to all the monitors in the watchlist. The change alerts are now triggered only when the condition is true. To add multiple global conditions among different watchlists, you can use the Teams feature. This is available the enterprise plan, and it allows you to create multiple watchlists.

Steps to add monitor-specific conditions

Step 1: When you are on the Options Page, click on Add Condition

Step 2: Set up your condition as per your requirement. You have the option to add compound condition using AND / OR.

Step 3: Click Save.

You can read more about using conditions here.

Challenges and troubleshooting steps for issues faced while using full-page monitors

While beneficial, full-page monitoring also brings challenges. Firstly, evaluate if such a comprehensive monitoring approach suits your needs, given it tracks all changes and triggers alerts accordingly. If tracking specific parts of a page is your case, full-page monitors may not be suitable. In such instances, we recommend utilizing the visual selector tool to refine your selections.

If you decide it’s appropriate to proceed with a full-page monitor, here are some tips and solutions to help navigate potential issues.

False positives associated with tracking the full page

While the full-page monitoring feature in Distill offers comprehensive insights, it does come with certain limitations. One notable challenge is the possibility of false positives in tracking the full page.

What is a false positive?

False positives occur when the monitoring system detects changes or updates that may not be relevant or significant to the user. Since the full-page monitoring captures the entire content and layout of the webpage, any minor changes, such as insignificant text edits, date counters, or layout adjustments, can trigger alerts that can become overwhelming.

Typical scenarios include:

• Date counters

• Layout adjustments

• Cookie popups (which load sometimes & do not otherwise)

• All elements fail to load before the check is performed

In some cases, all elements on a page may fail to load, which could result in false alarms triggered by missing content. These false positives can lead to unnecessary notifications, exhaustion of alert limits and may require additional effort to filter out irrelevant information.

Troubleshooting steps for countering false positives

1. Deselecting noisy elements on the page

To mitigate the challenges mentioned above, Distill offers a solution by allowing you to deselect sections they do not want to track or have frequent unrelated changes.

You can follow the steps below to deselect parts of the page:

Step 1: Once you are navigated to the options page (after selecting the Monitor full-page option), click Open Webpage Selector under the source section. You will be navigated to the webpage.

Step 2: Click on the play button to activate the visual selector, which allows you to deselect the elements.

Step 3: Select the sections you do not want to track. These sections will be bordered with a red rectangle, indicating that the sections have been deselected from the monitor.

You can also see the deselected sections XPath under the deselected section in the Distill dialogue box.

Step 4: Click Save Selections.

This customization of selections allows you to focus on the elements critical to their monitoring needs, reducing the occurrence of false positives and enhancing the accuracy of alerts.

2. Adding a delay to the monitor

To address the issue of the page failing to load, Distill provides the option to configure delays in the monitoring process.

Setting a delay in Distill is crucial because certain elements on a webpage may take time to load or are dynamically generated, which may result in missing elements and triggering false change alerts. By configuring an appropriate delay, Distill ensures accurate monitoring by allowing sufficient time for all relevant elements to load before capturing changes. This way, false positives stemming from incomplete page loads can be minimized.

You can refer to Distill docs to know more on how to how to add a delay to your monitors.

3. Handling the cookie pop-ups

Another limitation of Distill is its inability to navigate to webpages when the preview doesn’t load. This happens due to cookie gating or other pop-ups that require manual user action to access the webpage. In such cases, where a webpage is gated behind cookie consent or other interactive pop-ups, Distill’s automated monitoring process may not proceed further to capture the page content. Don’t worry, we have a solution to tackle this.

This limitation arises because Distill operates based on automated processes directly accessing the webpage content. Distill cannot overcome these obstacles automatically if cookies or pop-ups block/restrict the page preview.

Troubleshooting Steps

There can be different solutions to address the cookie pop-ups challenge:

1. If there is javascript code that loads additional dynamic content (like popups etc.), then Disabling Javascript can be a simple solution to address the challenge.

Note that this method will typically work in a scenario where the default page loads all the content & subsequently, there is javascript which adds the cookie pop-ups, etc.

2. Try using a local monitor (via the browser extension). Usually, most browsers will store your cookies & once you accept them, they should typically not pop-up again while performing the monitoring.

3. In more complicated cases, especially if you need to monitor via the cloud, we recommend trying out the profiles, as detailed below

Profiles enable you to save and manage cookie information for monitors running in the cloud. By configuring profiles with the necessary cookie acceptance settings, Distill can effectively handle cookie pop-ups and gain access to the gated web pages in the cloud.

Steps to create a profile within Distill

Step 1: Log in to your Distill web app and click the hamburger option.

Step 2: Select Profile. You will be directed to the profiles page.

Step 3: Enter a name for your profile and click Add Profile. You will be navigated to a remote browser.

Step 4: Enter the URL within the browser and click Go. For this example, we will be using this URL.

Step 5: Accept the cookie preferences and click Save cookies and close.

Step 6: You can now see the profile under Profiles tab. You can edit options to change the device type, set the interval, and configure your alert mode.

Disclaimer: To ensure Distill can access and monitor websites that frequently update their cookies, it is recommended to open the browser under the profile section and accept any changed cookies.

Steps to utilize the created profile

Step 1: Click Add Monitor and select Webpage.

Step 2: Paste the URL of the webpage(for which you had accepted your cookie preferences while creating a profile in Distill). For example, in this case we will paste this URL.

Step 3: Select the profile under the profiles tab.

Step 4: You can then use the 3 dots symbol to select the Select full page option or click on the visual selector to select sections to monitor.

Step 5: Click Save Selections to navigate to the options page where you can select the device type, set interval and configure the alert mode for the monitor.

Note: Using Local Monitors as your monitor device type is highly recommended. Some websites have strict policies against automated traffic; running a page monitor can potentially violate those terms. Consequently, websites may block or restrict requests from your IP address if it detects a high volume of requests quickly.

4. Not detecting changes in hyperlinks (or other HTML attributes) within the page

One limitation of using Distill’s full-page monitor feature is the inability to track changes in the content of individual URLs added to the webpage. While Distill effectively monitors entire web pages, it doesn’t provide detailed tracking of the specific URLs within those pages. However, if the text associated with a URL is modified, Distill by default promptly alerts the user. This presents a challenge as it requires you to rely on broader page-level monitoring rather than granular tracking of individual URLs.

You can employ alternative strategies to track individual links to track the href properties and attributes by using the Monitor parts of the section feature. This approach allows you to overcome the challenge of not being able to track changes in individual URLs while still benefiting from Distill’s broader page-level monitoring capabilities.

Troubleshooting Steps

Follow the steps below to set up a monitor for tracking multiple URLs. For this example, we will track the Investor Relations page for JP Morgan Chase & Co.

Step 1: Navigate to the webpage. Click the Distill icon in the top right corner and click Monitor parts of page.

Step 2: Select the links you want to track.

Step 3: You will find a list of URLs you want to track under the watchlist section. To the right of the URL you can find a drop-down under the watchlist where you can choose the specific fields you wish to monitor within the URL. In this example, we will be tracking the href attribute.

Step 4: Click Save Selections. You will be navigated to the options page.

Step 5: Under the options page, choose a device to run your monitor and set an interval for checking the webpage for alerts. You can also configure your alert mode via email/SMS/ Discord etc.

Step 6: Click Save.

Refer to this video for step-by-step instructions on tracking href property associated with the text you wish to track on the page.

Troubleshooting other issues

It’s important to note that full-page monitors typically do not throw errors like SELECTION_EMPTY since the default selector is set to the entire HTML itself. However, when using the full-page monitoring feature in Distill, you may encounter certain issues that require troubleshooting.

Here are other common issues you may encounter and some suggested solutions:

• Blocking: When encountering blocking issues during monitoring on Distill, you can employ the following strategies:

  • Use a local monitor: Running the monitor locally allows you to perform monitoring within your network environment, bypassing any access restrictions or blocking mechanisms set by the monitored webpage.

  • Utilize premium proxies: Consider using premium proxies during monitoring. Premium proxies help mask your IP address, making it more difficult for the monitored webpage to detect and block your requests. This can provide a smoother and uninterrupted monitoring experience.

  Proxies are available for cloud servers only. Click options under the device section to select the proxy.

  

• Full-page monitoring behind login/password: When monitoring webpages that require login credentials, you may face additional challenges. If possible, use local monitors, as cookies are already stored in your browser profile. This allows you to authenticate and access the page seamlessly. Alternatively, you can leverage the use of Profiles which allows you to save cookie information and utilize cloud monitoring to track pages that require logins. This enables you to automate the monitoring process even for pages behind login/password barriers.

You can refer to this video below, for more information on how to use Distill for Cloud-Based Monitoring of Password-Protected Web Pages

Set up full-page monitor using Distill’s Enterprise API

Utilizing Distill’s Enterprise API, you can create full-page monitors for a selected range of target URLs. The API allows you to make both GET and POST requests.

GET requests to retrieve the data, including:

1. Various monitors within the account

2. Details associated with historical data pertaining to a specific monitor

This data can then be fed into any downstream system (as a data pipeline) for further analysis and visualization.

On the other hand, POST requests allow you to create new monitors (as well as make updates to existing monitors) with various configuration parameters. Some basic parameters include the frequency of checks, delays, and actions to be performed when changes are detected.

The Distill API offers a robust and efficient approach to web monitoring at scale, enabling you to add/update or delete monitors via other applications.

Explore the Distill Enterprise plan to understand more about API access. You can also schedule a call with our product expert to discuss your use case.

Conclusion

Maximize your Distill experience and effectively track web page changes by following these troubleshooting approaches. Adapt your monitoring strategies, utilize available features and options, and ensure accurate and uninterrupted monitoring.

When setting up monitors, it’s crucial to understand when to use the Full Page and Monitor Parts of Section features. Here’s how you can leverage them effectively:

1. Full Page Monitoring: Capture comprehensive views of webpage changes, including content and layout. Optimize user experiences and track modifications across the entire page.

2. Monitor Parts of Section: Track specific elements like prices, availability, or sections of interest within larger webpages. Focus on targeted areas and reduce unnecessary alerts.

By understanding these scenarios and choosing the appropriate feature based on your tracking needs, you can maximize the benefits of Distill’s monitoring capabilities and stay informed about the changes that matter most to you.

For further guidance and a complete understanding of Distill’s features, we recommend you to refer to our Distill docs.

Related reading: Guide to use webpage monitors for data-driven competitive intelligence, Sitemap monitor.