Staging and test sites: where to put them

Put the non-public copy somewhere an accident cannot cost you anything: behind authentication, on infrastructure that does not point back at the property you are protecting. Everything else trades fidelity to production against exposure. Two things genuinely go wrong. A staging copy gets indexed and starts competing with the site it was cloned from, or a staging copy quietly proves that two properties you present as unrelated belong to the same person.

Failure mode one: the copy enters the index

A crawlable clone is a near-identical site with a different hostname. Search engines will pick one version to show and it is not guaranteed to be the one you sell from. Worse, staging carries content you would never publish: half-finished pages, test orders, placeholder legal text. When one of those ranks for your brand, the damage is reputational before it is technical. Removal afterwards is slow, so this is a problem you prevent rather than clean up.

Failure mode two: the copy joins your properties together

Separation work is undone by the test environment more often than by the live one, because nobody audits staging. The usual giveaways:

• Sites separated carefully in production, with every staging copy on one shared address.
• The clone keeps absolute URLs, canonical tags and image paths pointing at the source site, which is a direct machine-readable join.
• Analytics properties, tag container IDs and verification tokens get copied along with the files.
• A copy of site A is parked under a subdomain of site B, which links the two by hostname alone.
• Anything you issue a publicly trusted certificate for is published to certificate transparency logs, which are searchable. An obscure hostname is not a secret hostname.

The four places to put it

Making exclusion reliable instead of hopeful

1. Require HTTP authentication or an IP allowlist at the server. Unauthenticated requests get 401 or 403, so there is no content to index and no footprint to read. This is the only method that fails safe.
2. If it must be publicly reachable, send X-Robots-Tag: noindex on every response, including assets and PDFs. A header survives template edits.
3. Treat robots.txt as traffic shaping, not as exclusion. It requests that crawling stop; it does not remove a URL that something else has linked to.
4. Verify from outside your network with a fresh session, then re-check a week later.

The classic own goal

A staging site carries <meta name="robots" content="noindex"> and a robots.txt that disallows everything. Those two instructions cancel each other. A crawler that obeys the disallow never fetches the page, so it never reads the noindex, and if the URL was discovered from a link elsewhere it can still surface as a bare listing. Google's own Search Central documentation is explicit that a blocked page cannot have its noindex seen. Pick one lane: either allow crawling and serve noindex, or block access outright with credentials and let the 401 do the work.

Before you clone anything

• Decide the address first. A live site on separated space such as B class separated hosting needs a copy with its own arrangement, not a shared corner; the reasoning under network planning applies to test environments too.
• Strip analytics and verification tokens during the copy, not afterwards.
• Rewrite absolute URLs to the staging hostname so nothing calls home.
• Keep an inventory of live copies with an owner and a delete date. Abandoned staging is how this fails years later.
• Remove the block deliberately at cutover and confirm it is gone, which the launch playbooks treat as its own step.