Crawl Your Site — Build the Product or Content Catalog From Your Own Pages

When there is no feed to load, Personyze can build your Product Catalog or Article Catalog from your own site. It finds your item pages through your sitemap or your links, reads the details each page already publishes for search engines — name, picture, price, stock — and reads them again on the schedule you choose, so the catalog stays current. Every account has it; there is nothing to switch on first.

When to use it

• No feed. Your platform will not export one, or nobody keeps it up to date.
• A feed with gaps. Keep the feed and let the crawl add only the items it misses — see Add only new product pages.
• A content site. Blogs, magazines and help centres: the crawl reads article pages the same way — see Products and articles.

When you do have a feed or a store, prefer it: it carries every field your system knows, with no page reading in between. Shopify, WooCommerce, Magento and BigCommerce connect under Sync my store on the catalog screen; feeds are on Setting the Product / Content Feed and Uploading Feeds & Data Files.

Where to find it

Three ways in, all to the same dialog:

• Product Catalog → ⋯ (More options) → Crawl my site — discover and queue product pages. On the Article Catalog the entry reads … article pages. On the older catalog screen it is under Data settings.
• Add a source → We read your site → Continue → Open the crawl settings — the catalog’s setup, which then checks the items as a shopper sees them.
• Once a crawl has a schedule or has run, it is listed under Where your products come from as Site crawl. Click the row, or its gear, for the settings.

The settings

There is no separate Save: Crawl now saves what is in the dialog and starts a crawl, and Close leaves without saving.

Under the settings sit three panels, each opened by its bar: Field mapping, Tracking suggestions and Access credentials. Progress is at the bottom.

What arrives on its own, and what to map

Personyze reads the details your pages publish for search engines; it does not guess from how a page looks. For each field it takes the first of these that has it:

1. Structured data — schema.org Product details (articles: Article, NewsArticle or BlogPosting) in the page’s JSON-LD. The most reliable, and what Google reads.
2. Microdata — the same schema.org details written as itemprop attributes in the HTML.
3. Open Graph and meta tags — og:title, og:image, product:price:amount and the like.

Rules it applies:

• No price means out of stock. The item keeps its page — shoppers there still see “also viewed” — but it is never recommended. A price that appears only as text, with no structured data behind it, cannot be read, so such products arrive out of stock. Adding schema.org Product data to your product pages fixes it, and it is the same data Google uses for rich results.
• The SKU is read but not stored as a field. It is used only as the item id, when Use the page’s own SKU as the item id is on.
• Not read: categories, a was-price (the struck-through one), the currency (set how prices print under ⋯ → Price formats), and anything particular to your site. Read those with Grab columns from site (same ⋯ menu), which reads values off each item page with CSS selectors; they then show in Field mapping. A card shows the higher of Price and Sale Price as the price before discount, so a was-price read into Sale Price shows as the discount.
• Your site’s own details — its logo, its name — are never taken as an item’s name, image, brand or price.
• A page marked up as an article is never stored as a product, and a page marked up as a product never as an article.
• A page that answers with a login prompt instead of the item is not added — see Private and members-only sites.

Field mapping

For fields the crawl does not read. Field mapping lists the custom fields set up under Grab columns from site, and points each one at a source:

Two more sources are listed — the page’s JSON-LD and a meta tag — with not read yet, so a field set to it stays empty; do not choose them. Value found and Sample page show what the last crawl found on one sample page. Add field adds a row, and nothing is saved until Apply. A field you set here is never changed by a crawl.

When fields that used to be found come back empty on most pages, the panel says These fields used to be found but came back empty on most pages of the last crawl — the site may have changed, or a rule below may need updating, and names them. Usually the site’s design changed: update the selector.

Tracking: what makes crawled items count

A crawled item is recommended — and anchors “people who viewed this also viewed” — only once Personyze records which item each visitor views. When nothing records that yet, the dialog and the Site crawl row say Visits to these pages are not tracked yet, so they cannot anchor or appear in recommendations., with Set up tracking, which opens Product tracking (Article tracking on a content catalog). See Product Interaction Tracking.

The item id is built from the page address, so visits to a crawled page are matched to its item with no id to map. If your tracking reports a SKU instead, turn on Use the page’s own SKU as the item id.

Tracking suggestions reads a handful of pages your last crawl read, and suggests tracking to add: a product page viewed, added to cart or saved to favourites (articles: an article read, a comment), and goals that count a click or a visit. Each comes with Why and Seen on. Press Get tracking suggestions; every box starts unticked, and nothing is written until you tick one and press Apply. One that already exists is reported as Already set up — nothing changed.

Progress and results

After Crawl now, a line under the buttons says what happened — for example 25 pages queued (found via the sitemap). Watch their progress below., or Nothing new — every page already found is queued or done. It also counts pages skipped because their address is too long to store, and, with Add only new product pages on, pages already in your catalog and left as they are.

Progress, at the bottom of the dialog, shows how many of the queued pages have been read (… of …), how many failed, and Everything read at the end, with a row per page:

It updates by itself every few seconds while pages wait; Refresh looks at once. You can close the dialog — the crawl goes on in the background, reading one page at a time from your site, at most one a second. A few thousand pages take an hour or more.

On the catalog screen, the Site crawl row shows the schedule that runs (Runs daily, Runs weekly, Runs monthly or Paused), how many items it has read, Last run, and a pill: On, Paused or Failing. A paused crawl offers Turn on, which opens the settings to choose a schedule; a failing one opens with the reason and the page list, and offers Check. Its ⋯ menu holds View crawl and settings and Crawl settings.

Schedules

A scheduled crawl that cannot reach your site waits longer before each try, up to a week (or one period, on a monthly schedule). The dialog says so under the schedule: … scheduled attempts in a row could not reach your site (the reason). It is backing off, waiting longer between tries each time.

Limits

The crawl reads other people’s servers from Personyze’s own address, so it runs inside limits the server applies whatever the dialog holds:

• Pages per crawl, by plan. Shown under Page limit; a higher number is saved but not used. 20,000 pages is the most on any plan, and a crawl in a real browser reads 200 pages unless Personyze has raised it for your account.
• The fastest schedule, by plan. Faster choices are greyed out, with Your plan only allows a re-crawl every … days at most. The Site crawl row shows the schedule that actually runs.
• Polite reading. Your robots.txt is obeyed. Each site is read one page at a time, a second apart, and only a set number of sites are crawled at once across all accounts. Only https:// addresses are read.
• Sites that keep refusing. When one site answers 401, 403 or 429 to 100 pages in a row, the rest of that crawl on that site is held — see Sites that block crawlers.

Sites that block crawlers

Many sites put a bot filter in front of their pages — Cloudflare, an AWS firewall, a rule at the hosting company. It answers our crawler 403 (Forbidden) while the same page opens fine in your browser. The crawler names itself and never pretends to be a browser, so the fix is a rule on your site that lets it through. It takes a couple of minutes.

How you see it:

• When a crawl starts: Your site blocked our crawler (403). Allow this user-agent and try again: Personyze catalogue crawler; +https://www.personyze.com/
• On a page in Progress: Blocked by the site’s bot protection (403). Allow our crawler: …
• After 100 refusals in a row: Not requested: the site refused our crawler on 100 pages in a row, so the rest of this crawl was held. Allow our crawler (Crawl settings show how), then click Crawl now. Those pages are tried again on the next crawl.

Crawl blocked or returning 403? What to allow, under Start pages, says what to let through. It opens by itself once any page came back 403.

Cloudflare: Security → WAF → Custom rules → Create rule, match the header, set the action to Skip, then tick All remaining custom rules, All managed rules and All Super Bot Fight Mode rules. The free Bot Fight Mode cannot be skipped by any rule: turn it off, switch to Super Bot Fight Mode, or allow our address with an IP Access rule (Security → WAF → Tools), which is checked before it.

robots.txt

Pages your robots.txt disallows for our crawler are never queued. A group that names the crawler is used instead of the * group, so this lets it read everything:

User-agent: Personyze
Allow: /

Any Disallow you want kept for us goes into that same group.

A 401 — Those pages need a login (401) at … — is a login wall, not a bot filter: see the next sections.

Sites built with JavaScript

Some shops and magazines build their pages in the browser after loading. A plain read then finds an empty shell: the crawl finds no items, or items with no name or price. Read pages in a real browser opens each page in a browser and reads what it renders.

• Try without it first. Many sites built that way still publish structured data in the page itself, and a plain read is many times faster.
• It reads 200 pages per crawl, whatever Page limit says, unless Personyze has raised that for your account; and it is slower page by page.
• The switch shows only on plans that include it.

Private and members-only sites

Access credentials, the third panel in the dialog:

• Cookie or API header — a header name (for example Cookie or X-Api-Key) and its value, for pages behind a login a cookie can get past, or an API that needs a key. It is sent with every crawl request to your own site, and never along a redirect to another domain. The value is never shown again: the panel only says A value is stored. — Replace or Delete it. A session cookie stops working when it expires; paste a fresh one.
• Form login (not used yet) — stores a login page URL, a username and a password for later. Personyze does not sign in to your site with them yet, so nothing is read with them today.

A page that answers with a login prompt instead of the item — a title such as “Login Required” — is not added. Progress shows This page asks for a login, so it is not public and was not added. It is tried again on later crawls, in case it becomes public.

Products and articles

Either way, a page that answers again on a later crawl comes back on its own: the article is published again, the product’s stock is read again.

Troubleshooting

The panel assistant (Ask Personyze) can also report on a crawl: whether it stopped at its page limit, which sections of your site its queue holds, and how many stored titles repeat or look like error pages.

Related

• Setting the Product / Content Feed for Recommendations — the catalog screen and every way to load it.
• Uploading Feeds & Data Files — files, URL feeds, SFTP and the API.
• Product Recommendations Wizard — where the catalog fits in a product recommendations campaign.
• Content Recommendations Wizard — the same for articles.
• Product Interaction Tracking — recording views, cart adds and purchases.