How to use webpage monitors for data-driven competitive intelligence

Have you ever wondered if there is a secret window into your competitors’ world, where you can observe every tweak in competitors’ pricing, every shift in their content, and every new product they launch? We’re here to tell you it’s not a secret; all that you need to know is publicly available on their websites and other digital platforms.

You can quietly observe their every move by simply watching their website. We are here to tell you that website monitoring is your covert ally in staying ahead of the competition. Today’s business is heavily focused on DATA. This article aims to broaden your understanding of how timely data is key for your business. We discuss the importance of gathering competitive intelligence from various data sources and how you can set this up yourself. How? You guessed it — we’ll dive into using web monitoring tools for your competitive intelligence pipeline.

Gathering comprehensive competitor insights

If you need to stay on top of what is relevant to your business, you need a multifaceted approach, one that gives you vital insights into market dynamics, competitor strategies, and customer sentiment. All this stems from a data-driven approach that combines various sources of information. To do this, you can rely on website monitoring tools.

How do website monitoring tools fit into your competitive intelligence pipeline?

These tools play a pivotal role in the competitive intelligence pipeline by providing both real-time and historical data on competitors’ online activities. This trove of data not only helps you take quick, decisive actions but also helps you in analyzing trends in competitors’ activities over time. The list of use cases is quite extensive. However, we’ll point out the ones you MUST HAVE in place.

In this article we explore how you can utilize Distill for gathering competitive intelligence in different scenarios. First, you’ll have to download the Distill extension for your web browser. It is available on Chrome, Firefox, Opera, and Microsoft Edge. It is also available as a Web App, Desktop App, iOS app and Android app.

The following sections will walk you through steps to set up tracking and how the insights from Distill can be consumed for your business strategy. Feel free to navigate to the sections using the table of contents listed below

1. Monitoring competitor’s website
2. SEO and Keywords
3. Social listening
4. Monitor leadership careers pages
5. Monitor review platforms
6. Watch investor relation pages and financial reports
7. Get the latest updates on research and development
8. Price monitoring
9. Monitor legal, compliance and policy related updates
10. Track roadmaps and product updates

1. Monitoring competitor website

Sitemap monitoring

By tracking the entire website with a sitemap you have a centralized list of URLs of your competitors’ websites by indexing the pages within their domain. It helps you stay updated about any new pages added to their websites. Sitemap even allows you to exclude links for crawling.

Steps to set up sitemap monitor:

Step 1: From your Distill watchlist view click on Add Monitor and choose Sitemap from the list.

Step 2: Here, enter the starting URL of the website you want to monitor. This can be /blog, /solutions, /press, or you may even want to monitor the entire website.

Optional: You can exclude any page that you do not wish to crawl

Step 3: Click Done. Now you will be directed to a page where you can configure a monitor. You can schedule checks at a desired frequency, add conditions and choose an alert mode.

Step 4: Click Save.

To learn more, check out our docs on sitemap monitor.

Competitive insights from sitemap:

• Get updates whenever a new webpage is added to the competitors’ website.
• Stay updated with alerts every time a new blog or an article is published.
• Get an understanding of the type of webpages being added into the website. Be it blogs, product landing pages, help articles, new job post pages ,etc.
• The sitemap alerts will be an indicator to set up new individual page monitors, which will be handy in tracking page-level changes.
• Understand the frequency at which new webpages are generated. (Eg: Publication frequency for /blog pages for content strategy, tracking new /jobs pages created to track hiring cycles)

2. Monitoring SEO and keywords

With Distill you can set up conditions to get alerts whenever a new keyword is added to a competitor’s webpage. In order to understand the position of the websites ranking for a particular keyword you may also monitor the SERP results pages. First, let’s take a look at how you can use conditions.

Steps to set up conditions for keyword alerts:

Step 1: In the Options Page you can click on Add Condition. Then you can add any keyword you want to monitor in the text box. You have the option to add compound conditions by using the AND / OR logic. You can track keyword updates related to newly added text, previous text or the edits to the existing text. You can choose the set of conditions available from the dropdown as per your requirement. Step 2: Click Save. Now, Distill will give you alerts only when the condition is satisfied.

Steps to get updates on the rankings in the SERP for a keyword search

Step 1: Key in a search term and land on the search results page. Click the Distill extension and choose Monitor Parts of Page.

Step 2: Use the Visual selector and select the SERP feed. Use the expand/contract selection option. Verify the preview of the SERP feed selected.

Step 3: Click Save

Competitive insights from Keyword and SERP tracking:

• Every time a competitor publishes a new blog or uses a specific keyword, you gain insight into their content strategy.
• You can set up monitors for specific keywords and understand how they are building their keyword cluster and content focus areas and SEO efforts
• Tracking the SERP results page helps you keep a pulse on the ads being run, sponsored content and ranking of domains for a particular search phrase.

3. Social listening

You can closely track channels relevant to your industry. These could include monitoring the Youtube channels of tech influencers, industry experts and even your competition. You can keep an eye on the frequency at which they publish content, metrics like no. of subscribers, views on the videos and the comments on their videos.

Twitter, now X, is also a very important channel for social listening. Here you might want to do more than just set up notifications for your favorite channels. You can fine tune your tracking by getting alerts whenever a certain keyword comes up in the posts of the accounts you follow. You can set up multiple monitors for different accounts and track them all from a single watchlist. Take a look at how you can set up Distill to track Twitter posts.

Similarly, you can set up monitors on social media platforms like Facebook and Reddit.

Competitive insights

• The change alerts are the real time updates from social listening e.g: new subscribers, followers, views, comments, number of likes etc,.
• Keeping an eye on competitors’ social media activities and customer engagements.
• Engage in partnerships or influencer marketing to expand your reach by identifying social media activity related your business area.
• On YouTube, track the results for keywords related to your business to find trending videos and topics to stay current and produce relevant content for your audience.
• Set up a watchlist with monitors on relevant subreddits and, twitter accounts to track discussions involving specific keywords about your industry, brand, user opinions or competitors’ products all in one place. You can even organize these monitors using Distill’s labels.

4. Monitor leadership and careers pages

You will need to have a pulse on the trail blazers in the business area you operate in. Keeping track of the leadership and board members of the key players in your industry will be important competitive intelligence. You can even keep a look on About Us page on the websites to see shifts in leadership focus, their vision, mission and values.

Monitoring job portals and career pages will give you an idea about the hiring focus and the talent pool your competitors are looking to induct into their workforce. You will also gain an understanding of the new technologies and skills important for your industry.

You can track job boards on Glassdoor, Indeed, Monster, and more.

Steps to set up monitor for leadership and career pages:

Step 1: Once you are on the URL you want to monitor, from your browser click the Distill extension and click Monitor parts of Page.

Step 2: The visual selector opens up, here you can view your selections and refine them using the expand/ shrink option and

Step 3: You can select parts of the page as shown below using the visual selector. And save your selections.

Reference for selections on a leadership page

Reference for making selections to track the job search results to get alerts whenever a new job crops up. You may have to sort your results page prior to making selections

Step 4: Click Save Selection. Later, in the Options Page you can add configure the alert mode and check frequency and Save your monitor

*Optional: You can choose to add conditions, which will alert you when a specific change occurs. e.g: Technology related to your business vertical, key skills, designation or job title.

Competitive insights

• Change alerts help understand strategic shifts organizational leadership and their skillsets.
• Using sitemap on careers pages will help in understand hiring trends based on new job pages created.
• Keyword based conditions will give timely alerts for specific job posts updates.
• You can understand the requirements and qualifications of talent pool and use it as a standard for your hiring purposes
• Understand the compensation, salary structure for key designations and get competitive insights about the benchmark.

5. Monitor review platforms

It’s important to have a complete picture of the players in your market, including new entrants. The information about how customer perceive the other products as compared to yours, what are the features they prefer, the pain points and the utility can be gauged by tracking reviews and ratings on product review platforms like G2, Capterra, Trustradius, Chrome store and app stores.

You can also set up monitors on product announcements platforms like ProductHunt that announce the launch of new products and allow early users to comment and share their feedback.

Steps to set up review monitor:

You can follow the same set of initial steps as in the above section and use the visual selector to make selections, as shown below.

Reference for selections on tracking ratings of a product within a category

Reference for making selections to track new products that have come up for your product category. This will help you keep a pulse on the new entrants and their product offerings.

Competitive insights

• Track the rankings and positions of your products or services on review platforms.
• Receive notifications when there are new reviews or changes in existing reviews for your products as well as your competitors.
• You can get alerts whenever a new product is announced and the public’s sentiments about the product through reviews.
• Through early user reviews you can understand the advantages, unique features and painpoints of the newly launched products and shape your roadmap accordingly.

6. Watch investor relation pages and financial reports

You can use Distill to track financial reports, investor relations pages and understand how your contemporaries in the market are faring. And if you are an investor yourself, you can stay updated on the latest about the company you are interested in by tracking financial PDF reports about quarterly earnings.

Distill can track PDFs for changes. However, this is available on paid plans, please explore our pricing page for more details

Steps to set up investor-page tracking:

You can follow the same set of initial steps as in the above section and use the visual selector to make selections as shown below.

Reference for selections to get the latest updates about new reports and announcements.

You may even choose to track a business’s press page to get the latest news about them.

Here’s a detailed guide with steps on how to track the SEC filing updates.

Competitive insights

• Anytime a new report is announced, updated or modified, you will get an alert.
• Updates from quarterly and annual financial reports can reveal revenue trends, profit margins, and financial stability.
• Get insights about a company’s funding sources, investment strategies, and capital allocation.
• Understand any changes in corporate governance policies or practices.

7. Get the latest updates on research and development

You can set up Distill to track the search feed to get updates about the new patents filed or publications made for a particular research area.

You will get alerts whenever a new publication is made for a particular area of practice. You can even add keywords related to research topics, so you get targeted alerts. The conditions could also include keywords like a the author’s name, company, technology being used etc,.

Sometimes, companies update their RnD page with new research areas.

Databases such as the United States Patent and Trademark Office (USPTO), the European Patent Office (EPO), and similar platforms provide in-depth information on recently filed patents. By setting up monitoring alerts on these websites, you can stay updated on new patents. Additionally, it’s beneficial to closely follow leading scientific journals and conference proceedings in your field of interest. Keeping an eye on these varied sources offers a thorough overview of the research and development environment specific to your business sector and helps in the discovery of new opportunities.

Competitive insights

• Identifying cutting-edge technologies being developed or researched in your business vertical.
• Understanding evolving trends in your industry and related fields.
• Discovering organizations or researchers actively working in areas relevant to your business.
• Identifying potential partners for collaborative projects by identifying key experts, inventors, or researchers in specific fields.
• Identifying potential areas for your own R&D and innovation efforts and assessing the viability of entering new markets or sectors

8. Price monitoring

You can set up monitors on the plans and pricing page of your competitors’ website as shown in the image below. This allows you stay on top of the features they offer against the pricing plan. If they do change their offerings you will get an alert from Distill.

Another important way to use monitors would be to track the offers and discounts your competitors offer. This way you do not lose customers and update your pricing accordingly.

Setting price monitors also helps you understand what is offered as free, what’s paid and if the plans have been offered at a free trial for a fixed duration, if they have a no card policy etc. Here in the image below notice the TRY FOR FREE CTA has been changed to UPGRADE

Competitive insights

• Gain insights into competitors’ pricing models and strategies.
• Understanding prevailing pricing trends within your industry.
• Utilize the change alerts for identifying the best price points for your products or services.
• Understand the frequency of offer cycles and discount strategies.
• The change history of offers and discounts allows you to understand trends in the pricing strategy.

9. Monitor legal, compliance and policy related updates

You can set up monitors to track the privacy policy, or the terms of service, as shown below.

You can track particular sections within the privacy policy. It will be useful to add conditions to get updates when certain keywords are added, removed or modified in the policy.

You can even track regulatory bodies respective to your industry and country. The below image shows selections made to track updates in the FTC (Federal Trade Commission) website. In this set up you will get notified about new announcements and changes in statutes and acts.

Competitive insights

• Understanding how changes in terms of service and privacy policy of your contemporaries impact your business operations or products.
• Comparing your compliance and legal strategies with industry peers and identifying best practices in legal and regulatory compliance.

10. Track roadmaps and product updates

Roadmaps offer a forward-looking view, helping businesses in strategic planning and setting a long-term vision. Tracking roadmaps, particularly in business and technology contexts, is crucial as it provides strategic insights that help organizations stay competitive, adapt to changing market conditions, and innovate effectively.

You can track the timelines of feature launches as shown below.

You can track specific sections of the roadmap, based on your field of interest and category of innovation.

Another important monitor you should set up will be on product launch platforms. For example - ProductHunt. Here you can get notifications of new products entering the market. Understand what early users have to say about the product, what are its unique features and what are the product owners saying about the planned features and roadmaps.

Competitive insights from monitoring roadmaps:

• Stay alerted about estimated release dates for upcoming products or feature updates and identifying where competitors are investing their resources and efforts.
• Understand new markets or regions for expansion, growth strategy and target audiences.
• Prioritizing R&D projects that align with anticipated market developments.
• Aligning your product development cycles with industry and technological trends. And planning upgrades, iterations, or new product introductions at optimal times.
• Spotting market gaps and unmet needs based on projected technology developments.

Essential strategies and tools for comprehensive competitive data gathering

• Google Trends will help in identifying market trends, consumer interests, offering a macro view of public search behaviors. This can be crucial in tapping into user behavior and content relevance.
• Ahrefs can be used in your SEO efforts. It excels in providing detailed insights into backlinks, organic search traffic, and keyword rankings. It’s particularly useful for analyzing your own website against competitors’ to find gaps in keywords.
• Gartner covers extensive market trends, helps understand industry shifts and future directions. You can set up Distill to track Gartner’s website in the following ways:

Additionally, we reccommend that you set up monitors to track the Magic Quadrant Reports for insights into vendor performance and positioning within specific markets. Monitoring Gartner’s Hype Cycles helps understand the potential impact of emerging technologies and industry forecasts. Gartner frequently publishes articles on best practices and case studies these help learn effective strategies and successful implementations across different sectors.

• Bloomberg, Reuters, and Yahoo! Finance are handy tools for competitive intelligence, as they reveal a company’s financial health, stock trends, and investor sentiment. Bloomberg offers in-depth financial analysis, Reuters provides global market insights, and Yahoo! Finance delivers user-friendly financial data and public opinion.

How to use Distill in your competitive intelligence pipeline?

Now that you have a holistic idea of what webpages offer the competitive data you need to build your CI, you can now use Distill web monitors to watch these webpages closely. You can either monitor targeted sections of the page, or you can monitor the full page.

Distill change alerts, which translate to competitive insights, can be easily integrated into your competitive intelligence pipeline. You can use our enterprise API to integrate change alerts into your workflows to efficiently manage and consume timely information about your competitors.

Troubleshooting tips

1. Handling Cookie pop-ups and tracking pages behind login forms: Some websites have a cookie pop up disclaimer that pops up in the front of the webpage contents. This will get in the way of your selectors on the page and the monitor will error out. You can use profiles in Distill to save the cookie information. This way the browser automatically uses the cookies to access the page and monitor its content. Learn more about how to use profiles.. You can also track pages behind a login page using Profiles. Here’s a detailed step-by-step guide for tracking password protected pages.

2. SELECTION_EMPTY: Your monitor will encounter a “SELECTION_EMPTY” error when the selected area or element for tracking on a webpage is no longer available or has changed. This can happen when:

   • The website layout or the HTML structure has been modified, and the selected area doesn’t exist in the new layout.
   • If the selected content is dynamically loaded or rendered, it might not be available at the time when Distill checks the webpage. Your selectors are brittle and have broken. Be sure to check the visual selector and see if there are meaningful selectors picked up.
   • If the website has imposed certain access restrictions, such as CAPTCHA requirements, or IP blocking, Distill might not be able to access the selected area of the webpage.

   Tip: Reselect the section of the page you want to track. If there are layout changes this is the way to fix the monitor. You may also add a delay of a few seconds in the monitor by clicking on “Edit config.” This helps avoid issues related to dynamically loaded sections.

3. EPROXY: Your monitor will show you this error when you are trying to access a page from a particular region but have been flagged by the website. You will also face this issue when you are using a VPN of a different country to access a region specific webpage. In that case, turn your VPN off, or use one of Distill’s proxies which has a shared pool of IPs. Learn more about Distill’s proxies here.

4. EMACRO: Your macros with sequence of pre-recorded actions that you want to perform on webpage might encounter issues when the element on the webpage for interaction is missing or has changed. In this case your monitor associated to the macro will flag an EMACRO error. To fix this, go back to your macro, open the recorder and replay the steps, check where the macro errors out and edit or replace the steps and save your updated macro.

5. Log of checks is empty:

The log of checks might be empty if your monitor has never run. This might be because the monitor was turned off. Turn it on and run it and it should work. If it still does not work, the selections you have have made on the page have changed. In that case set up a full page monitor and check if the section you are monitoring exists. Then set up a fresh selection, save it and re-run your monitor.

Please note that all the monitors in your teams watchlist are cloud monitors, you will need to run them on your webapp. Your log of checks will show an empty record when you try to run it an a local device.

For more details on how to use some of the features and troubleshoot some errors that you might encounter, check out Distill docs.

Conclusion

For more information, visit our solutions page to learn more about how you can use Distill as valuable tool in your competitive intelligence pipeline.

You can download Distill for free today. It is available as a Web App, Desktop App, browser extension and as iOS app and Android app.

Please feel free to reach out to us on Distill Forums in case you have any queries or need assistance while setting up monitors for competitive web data.