A Third of Shipstry's URLs Are SEO Debt

Shipstry has 2,882 known URLs in Search Console, and 949 of them are reported as either "Page with redirect" or "Duplicate, Google chose different canonical." That is a third of the site.

There is no penalty and no manual action. Google is doing what I asked, because I kept moving pages around and never cleaned up the map underneath them.

I want to write that down, because the mistake is easy to repeat and the fix is boring. Boring things rarely get written down, which is why everybody keeps making the same mistake.

Three migrations in four months

Each change took an afternoon, and each one was defensible on its own. Together they produced the number at the top of this post.

What I did not understand at the time: a URL change does not replace a URL, it creates a second one. The old shape keeps getting crawled, keeps getting requested, and keeps showing up in reports as a redirect or as a duplicate that Google resolved on its own. Ninety days of small improvements left me with a crawl map where one in three URLs is debris.

The cost shows up in the reports before it shows up in traffic. When a third of your URL inventory is noise, you cannot separate a real indexation problem from migration residue, and you stop trusting the numbers you make decisions with.

The four rules I enforce now

All four live in code, and each one has a test.

1. Page URLs are frozen

Product pages live at /products/{slug}/{id} with no trailing slash, ever. Page paths are slash-free; only file-like paths (/robots.txt, /sitemap.xml, /llms.txt, /rss.xml) keep their extension.

If a URL shape is wrong today, the fix is a redirect, and the new shape stays put. A fourth migration would repeat the 949-URL mess, and I would have nobody to blame but myself.

2. Exactly one module may issue a 308, and it stays one hop

Redirects live in a single place, and that place may only emit one hop. Legacy product path, alias, trailing-slash strip: whichever applies, it resolves in a single response.

The one-hop rule is what keeps "Page with redirect" from multiplying. If the first hop lands on a URL that redirects again, you have two entries in the report and the crawl budget pays for both.

3. The sitemap and the page's own robots tag read the same module

The sitemap and each page's robots directive are two code paths that both claim to know which pages deserve indexing. When they disagree, Search Console reports "Submitted URL marked noindex" and the sitemap loses credibility as a source of truth. That is worse than a missing page, because it puts every other entry in the sitemap in doubt.

This one took me a while to notice, and it is the cheapest of the four to get right. Both read one module now, and the thresholds live in it: a maker profile needs a bio or two approved products, a ship week needs ten, a tag needs three. Below those numbers the page is a restatement of something that already exists.

4. Prerendering is not free

Static prerendering is the standard advice for content sites. On this stack it removes three capabilities, because a prerendered page never reaches the Worker: markdown negotiation for AI agents (Accept: text/markdown), the canonical redirect, and the shared edge cache.

That is why /about, /pricing and /blog are rendered on demand even though they never change. All three have to answer AI agents with Markdown. There is a test that fails if a path appears in both the prerender list and the on-demand list, because a page that is silently in both is a page nobody can reason about.

The inventory those gates protect

The last two rows are the constraint. An average of 707 characters of maker-written notes, and a leader with seven votes, describes a registry with a long tail and a thin head.

That shape is why the gates matter more here than they would on a bigger site. A directory's failure mode is too many pages that are all slightly worse than the page next to them, not too few pages.

What I got wrong: list pages were covered, detail pages were not

I assumed the thin-page risk lived in the archive pages, so that is where I put the editorial work. Every category archive has authored copy at the bottom: what the category is, how to choose inside it, related topics, an FAQ. All 17 of them. Same for topics, where 58 of 58 indexable tag pages carry their own section.

Then I read a product page end to end. The page has a facts rail with the category, tags, ship week, the maker and any awards. It has generated sentences about alternatives and about what shipped the same week. It has the maker's own notes. It has no sentence, written by the site, explaining what the product is, who it is for, or how it compares.

That gap is easy to see once you look, and hard to fix at 820 pages. Two options, both unattractive:

The Wikipedia model everyone cites for this does not survive a long tail. Wikipedia has an article for DeepSeek because thousands of people care about DeepSeek. It has nothing for the 717th launch of the week. A registry that writes encyclopedia entries for its long tail ends up with near-duplicate pages and a year of wasted writing.

So the plan is a curated tier: award winners, and whoever else earns it, get real editorial. Everyone else keeps the facts rail and the maker's notes. That is a smaller claim than "every product gets a page", and it is the one I can defend.

Internal links: what is built, and the thing I refuse to automate

The linking structure is in place and bidirectional. A product page points at its category archive, at each tag whose topic archive clears the gate, at its ship week, at its alternatives, and at whatever shipped the same week. Week archives point at the categories they touch, the monthly awards, and the previous and next week. Topic archives point at related topics, and category archives point at the topics inside them.

Week archives chain in both directions, which is what stops 37 weeks from becoming 37 orphans.

What I refuse to automate is linking inside maker-written copy. The temptation is obvious: notes mention ChatGPT, Figma or Notion, so link those words to the matching product pages. The matching pages mostly do not exist, and where a similar domain does exist it is often a same-name site with nothing to do with the product. That manufactures wrong links and hands authority to junk.

Structural links come from data. A prose link needs someone who read the sentence.

The machine-readable layer

The site serves /llms.txt and /llms-full.txt, negotiates Accept: text/markdown on every server-rendered page, and exposes a read-only MCP server over the launch data. I have no idea yet whether this pays for itself in citations. I built it because it took a few hours, and the alternative was being unreadable to the tools that increasingly answer the questions my pages are trying to rank for.

What I would tell someone starting a directory

Buy depth before page types. Detail page, category archive, tag archive, time archive, winners, profiles, search: I have all of them, and building the taxonomy was the part I would skip if I started again.

The debt lives in moving things that were already indexed, in archive pages that exist because a template rendered them rather than because someone searched, and in detail pages that carry facts where they should carry explanation.

The checklist I would hand to myself a year ago has four items on it, and none of them are "add a page":

1. Freeze every URL you have today. Write the rule down before you need it.
2. Give redirects exactly one owner and exactly one hop.
3. Make the sitemap and the page's robots tag import the same predicate, then never let them drift.
4. Decide what a page has to contain before it may be indexed, and write that number down somewhere the next version of you will find it.

The other half of this is the manual: which page families a directory needs, what belongs on each, and the order to build them. Everything else is content work, and content work has no shortcut. 949 URLs of debris is what the shortcut costs.