CORS & Crawler Policy

Allowed website origin

The static CDN is configured to send CORS headers for this exact origin:

https://softhoursdiary.pages.dev

This allowlist is applied across static routes served by the Pages project: emoji assets, flag assets, helper files and documentation. An HTTP Origin contains the scheme and hostname, not a trailing slash.

When CORS is needed

Displaying an external image through an ordinary HTML image element usually works without JavaScript CORS access. CORS matters when browser JavaScript needs to read the response through fetch(), or when a canvas needs to read pixels from the image.

const response = await fetch(
  "https://softhoursdiary-emojis.pages.dev/emojis/svg/1f632.svg"
);

if (!response.ok) {
  throw new Error("Asset request failed: " + response.status);
}

const svgText = await response.text();

For ordinary display, prefer setting an image element's src directly rather than fetching the image into JavaScript.

Response headers

The Cloudflare Pages _headers file sets the allowed origin, methods, allowed request headers, preflight cache duration and Vary: Origin on static responses. This repository is a static Pages site; if a Pages Function is added later, its own response must attach the CORS headers.

Documentation crawler policy

The robots.txt file allows general crawling of the site and explicitly allows the documentation tree. It lists several AI crawler tokens and asks them not to crawl the raw asset folders:

Allow: /api/
Disallow: /emojis/
Disallow: /flag/
Disallow: /flags/

Asset file responses also receive X-Robots-Tag: noindex, nofollow, nosnippet. Documentation pages remain indexable and are listed in the sitemap.

What this policy does and does not do

• Does: invite search engines and AI documentation crawlers to read /api/*.
• Does: ask compliant crawlers to skip /emojis/*, /flag/* and /flags/*.
• Does: attach noindex response headers to these asset paths.
• Does not: authenticate requests, hide filenames, or stop a crawler that ignores robots.txt.

Verify the live policy

curl -I https://softhoursdiary-emojis.pages.dev/emojis/svg/1f632.svg
curl -I https://softhoursdiary-emojis.pages.dev/api/
curl -I -H "Origin: https://softhoursdiary.pages.dev" \
  https://softhoursdiary-emojis.pages.dev/emojis/svg/1f632.svg

Check that asset responses include the allowed CORS origin and asset-specific noindex header. Check that documentation pages remain indexable. These rules take effect after the Pages deployment completes.