Confirm accuracy of Google Search Console alerts

[leelas-pm]

We frequently receive alerts from Google Search Console indicating a number of errors with MobiMart and calitp.org. Some of these are not accurate, but it's possible some of them are.

In order to ensure a smooth transition of Cal-ITP websites when the time comes, we should:

1. Investigate which errors are accurate and in need of attention
2. Resolve said errors
3. Document our resolution and provide guidance on how to review Google Search Console alerts moving forward.

[jgravois]

hey @leelas-pm 👋

if you can give me access to https://search.google.com like you already did for https://analytics.google.com i can take a look at this one. 🤗

[leelas-pm]

hey @leelas-pm 👋

if you can give me access to https://search.google.com like you already did for https://analytics.google.com i can take a look at this one. 🤗

sorry @jgravois , totally missed this! @lalver1 and @angela-tran I updated it so y'all should all have permissions across both sites in Google Search Console now too :)

[Scotchester]

I took a comprehensive look at all of the issues being reported in Google Search Console for calitp.org. (I'll take a look at MobiMart next, but that should take much less time.)

calitp.org findings

The first thing to note is that it's reporting data for ALL subdomains of calitp.org, including:

• analysis.calitp.org (301 redirects to https://analysis.dds.dot.ca.gov/*)
• benefits.calitp.org, dev-benefits, and test-benefits
• dashboards.calitp.org (301 redirects to https://metabase.dds.dot.ca.gov/auth/login)
• dbt-docs.calitp.org (301 redirects to https://dbt-docs.dds.dot.ca.gov/*)
• docs.calitp.org (See: https://github.com/cal-itp/cal-itp.github.io)
• embeddable-maps.calitp.org (see: https://github.com/cal-itp/data-infra/tree/main/apps/maps)
• gtfs.calitp.org (redirects to https://gtfs.dds.dot.ca.gov/)
• monitoring.calitp.org (now decommissioned, results in Cloudflare host error 522)
• notebooks.calitp.org (redirects to https://jupyterhub.dds.dot.ca.gov/hub/login?next=%2Fhub%2F)
• ods.calitp.org (301 redirects to https://tods-transit.org/, which seems to be run by https://github.com/MobilityData)
• reports.calitp.org (301 redirects to https://reports.dds.dot.ca.gov/*)
• resources.calitp.org (which redirects to https://www.camobilitymarketplace.org/go-contactless/ and then /tap-to-pay/)
• sentry.calitp.org (now decommissioned, results in Cloudflare host error 522)

Also, entries are present for both www.calitp.org and the subdomain-less calitp.org.

This means there's a lot in here that we don't care about / isn't our responsibility, but here's a rundown of the identified issues.

Pages not being indexed

• 461 Crawled - not currently indexed
• 448 Duplicate without user-selected canonical
• 83 Page with redirect
• 39 Excluded by 'noindex' tag
• 22 404 Not Found
• 18 Alternate page with proper canonical tag
• 4 403 Forbidden
• 2 Blocked by robots.txt

The subsections below describe what each of these mean and what specific things are being reported in each.

Crawled - not currently indexed

It's not easy to tell why these would not be indexed. Most of them seem like perfectly legitimate indexable URLs. If you look at Google's explanation for the error, it is unhelpful:

The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.

• www.calitp.org results (31) are all PDFs – Doesn't Google index those?
• docs.calitp.org results (48) are various Cal-ITP docs sites – Why wouldn't these be indexed?
• reports.calitp.org results (129) redirect to https://reports.dds.dot.ca.gov/*, which are indexed, so this is fine, but why do these show up here and not in the "Page with redirect" list?
• resources.calitp.org results (27) are all file downloads – Doesn't Google index those?
• embeddable-maps.calitp.org results (193) are all just querystring variations – These it actually makes sense shouldn't be indexed separately, but isn't that just standard Google behavior? Why show up on this report?
  • I guess on some sites, URLs with query strings do make sense to be indexed. I think it's kinda opaque how Google chooses to do this or not, and I'm not sure it's controllable.

Duplicate without user-selected canonical

Google's explanation:

This page is a duplicate of another page, although it doesn't indicate a preferred canonical page. Google has chosen the other page as the canonical for this page, and so will not serve this page in Search. You can Inspect this URL to see which URL Google considers canonical for this page.

This is not an error, but is working as intended, because Google does not serve duplicate pages. However, if you think that Google has chosen the wrong URL as canonical, you can explicitly mark the canonical for this page. Alternately, if you think that this page is not a duplicate of the Google-chosen canonical, you should ensure that the content differs substantially between the two pages.

• Most of these are embeddable-maps URLs (439), which seems right.
• A handful of resources.calitp.org PDF results – all the same doc with different query strings, so this seems "right", but we probably don't want these HubSpot cross-domain query strings in the first place.
• A few Benefits results that seem mostly right
  • Ignoring a query string added by an inbound link to the homepage
  • Ignoring different variations on the user error page (/eligibility selected as canonical over /eligibility/start)
  • a benefits.calitp.org URL selected over test-benefits

Page with redirect

Google's explanation:

This is a non-canonical URL that redirects to another page. As such, this URL will not be indexed. The target URL of the redirect might or might not be indexed, depending on what Google thinks about that target URL.

A canonical URL with a redirect can be indexed.

If you view this URL in the URL Inspection report, the indexed information applies to the tested URL (ignoring any redirects). To see the index status of the canonical URL associated with this URL (the URL that is in the Google Index), click the INSPECT button in the Page indexing > Indexing section of the report.

The live URL Inspection test follows redirects and then tests the final URL, although the live test doesn't indicate that it is following a redirect.

Most of these are expected:

• http -> https
• bare calitp.org -> www.calitp.org
• No trailing slash -> trailing slash on www
• *.calitp.org -> *.dds.dot.ca.gov

Excluded by 'noindex' tag

Google's explanation:

When Google tried to index the page it encountered a 'noindex' directive and therefore did not index it. If you do not want this page indexed, congratulations! If you do want this page to be indexed, you should remove the 'noindex' directive.

There are a bunch of false positives on the Benefits docs where the pages redirect, but the destination does not have a noindex tag at all. Not sure if anything can be done about that.

Example: https://docs.calitp.org/benefits/configuration/environment-variables/ actually redirects to https://docs.calitp.org/benefits/reference/environment-variables/ following Diataxis reorg. It does not have a noindex tag on it, but it is not indexed by Google (see https://docs.calitp.org/benefits/reference/environment-variables/). Because of this false positive or some other reason?

Legit ones:

• test-benefits Django Admin URLs
• dashboards (Metabase)

404 Not Found

Google's explanation:

This page returned a 404 error when requested. Google discovered this URL without any explicit request or sitemap. Google might have discovered the URL as a link from another page, or possibly the page existed before and was deleted. Googlebot will probably continue to try this URL for some period of time; there is no way to tell Googlebot to permanently forget a URL, although it will crawl it less and less often. 404 responses are not necessarily a problem, if the page has been removed without any replacement. If your page has moved, use a 301 redirect to the new location. See Fixing 404 errors

This turned up some legitimate broken links within the site, which I'll fix in a follow-up PR.

There are some curious entries for test-benefits.calitp.org/agency_slug/eligibility, which show referrals from https://test-benefits.calitp.org/ and https://benefits.calitp.org/eligibility/. As far as I know, we've never had URLs for agency_slug/eligibility, so I'm confused as to how these could show up, and especially how prod would be referring to test URLs.

Others we might want to try to correct:

• https://docs.calitp.org/operational-data-standard/ (referred by https://www.calitp.org/press/cal-itp-announces-ods/)
  • Can't find anything on GitHub related to this anymore.
• Benefits docs URLs that are missing redirects:
  • https://docs.calitp.org/benefits/tests/
  • https://docs.calitp.org/benefits/explanation/development/models-migrations/
• A CS docs broken link:
  • https://docs.calitp.org/admin/permissions/invite-a-user (referred by https://docs.calitp.org/customer-success/admin/Permissions/permissions/)

Alternate page with proper canonical tag

Google's explanation:

This page is marked as an alternate of another page (that is, an AMP page with a desktop canonical, or a mobile version of a desktop canonical, or the desktop version of a mobile canonical). This page correctly points to the canonical page, which is indexed, so there is nothing you need to do. Alternate language pages are not detected by Search Console.

Most of these are links to the www.calitp.org homepage with weird query strings, referrer unknown. Not worth worrying about.

403 Forbidden

Google's explanation:

HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.

If you do want Googlebot to index this page, you should either admitting non-signed-in users or explicitly allow Googlebot requests without authentication (though you should verify its identity).

All four of these are URLs that redirect to Metabase. Nothing to do here.

Blocked by robots.txt

Google's explanation:

This page was blocked by your site's robots.txt file. You can verify this using the robots.txt tester. Note that this does not guarantee that the page won't be indexed through some other means. If Google can find other information about this page without loading it, there is a very small chance that the page might still be indexed. To ensure that a page is not indexed by Google, remove the robots.txt block and use a 'noindex' directive.

Only two items here are for the retired Sentry and Monitoring services.

Settings > robots.txt

• http://monitoring.calitp.org/robots.txt: Not Fetched - Server error (5xx)
• https://dashboards.calitp.org/robots.txt: Not Fetched - Blocked due to other 4xx issue (http also redirects to https, both forward to https://metabase.dds.dot.ca.gov/auth/login)

HTTPS > HTTP marked with canonical tag

• https://docs.calitp.org/data-analyses/high_quality_transit_areas/
  • Poking around, I can see that it's true for other pages under https://docs.calitp.org/data-analyses/, as well. Looks to be an issue with the Jekyll SEO plugin.

---

Possible action items

• Determine if we can get Google to ignore subdomains unrelated to the calitp.org website
• Prevent dev-benefits and test-benefits from being crawled at all
• Investigate why files on www and resources are not being indexed
• Investigate why some docs.calitp.org URLs are not being indexed
• Investigate whether we can get Google to ignore querystring variations
• Open PR for broken links on calitp.org (Repair external links that were accidentally made relative #685)
• Consider making other fixes identified in the 404 Not Found section
• Consider trying to eliminate all links to https://calitp.org to eliminate the redirects to www.calitp.org and reduce "not indexed" entries in Google Search Console

[Scotchester]

MobiMart findings

Pages not being indexed

• 27 Crawled - not currently indexed
• 23 Page with redirect
• 4 404 Not Found
• 3 Alternate page with proper canonical tag

The subsections below describe what each of these mean and what specific things are being reported in each.

Crawled - not currently indexed

It's not easy to tell why these would not be indexed. Most of them seem like perfectly legitimate indexable URLs. If you look at Google's explanation for the error, it is unhelpful:

The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.

• 10 results are file downloads – Doesn't Google index those?
• https://www.camobilitymarketplace.org/resources/std-213-tap-to-pay.html is a real 404. Not sure why it shows up here and not in the 404 section.
• Other results all redirect to real pages (mostly removing .html or adding a trailing slash). Not sure why they are showing up here and not in the "page with redirect" section.

Page with redirect

Google's explanation:

This is a non-canonical URL that redirects to another page. As such, this URL will not be indexed. The target URL of the redirect might or might not be indexed, depending on what Google thinks about that target URL.

A canonical URL with a redirect can be indexed.

If you view this URL in the URL Inspection report, the indexed information applies to the tested URL (ignoring any redirects). To see the index status of the canonical URL associated with this URL (the URL that is in the Google Index), click the INSPECT button in the Page indexing > Indexing section of the report.

The live URL Inspection test follows redirects and then tests the final URL, although the live test doesn't indicate that it is following a redirect.

All of these are expected:

• http -> https
• bare camobilitymarketplace.org -> www.camobilitymarketplace.org
• No trailing slash -> trailing slash
• Removing .html extensions
• Explicitly set redirects in our src/_redirects file

404 Not Found

Google's explanation:

This page returned a 404 error when requested. Google discovered this URL without any explicit request or sitemap. Google might have discovered the URL as a link from another page, or possibly the page existed before and was deleted. Googlebot will probably continue to try this URL for some period of time; there is no way to tell Googlebot to permanently forget a URL, although it will crawl it less and less often. 404 responses are not necessarily a problem, if the page has been removed without any replacement. If your page has moved, use a 301 redirect to the new location. See Fixing 404 errors

All four of these are legitimate 404s where old URLs with .html extensions are not being redirected to their correct destination (like the std-213-tap-to-pay.html listed in the "Crawled - not currently indexed" section). We must have misunderstood how the redirects in 11ty work. I'll open a PR to fix these.

Alternate page with proper canonical tag

Google's explanation:

This page is marked as an alternate of another page (that is, an AMP page with a desktop canonical, or a mobile version of a desktop canonical, or the desktop version of a mobile canonical). This page correctly points to the canonical page, which is indexed, so there is nothing you need to do. Alternate language pages are not detected by Search Console.

These are MobiMart homepage URLs with query strings, two from the old contracts viewer URL now redirecting to the homepage, one with UTM campaign parameters from unknown origin. Nothing to worry about.

Sitemaps

• Last read on June 30, 2026, showing 21 errors due to Sitemap and other URLs use netlify.app domain instead of proper one mobility-marketplace#960. Apparently no button to just ask for it to be re-read, so I resubmitted the same URL to try and force it.

---

Possible action items

• Open PR for missing redirects (Clean up redirects to cut down on Google Search Console issues mobility-marketplace#971)
• Investigate why files are not being indexed
• Investigate whether we can get Google to ignore querystring variations
• Ensure all internal links include their trailing slash to cut down on redirects

[Scotchester]

The first thing to note is that it's reporting data for ALL subdomains of calitp.org

…

• Determine if we can get Google to ignore subdomains unrelated to the calitp.org website

Yes, we could re-create the property specifying www.calitp.org and the it would ignore other subdomains.

[Scotchester]

• Investigate why files on www and resources are not being indexed

Spot checking a few of these, Google does see them as indexable, they just haven't (yet?). Doing a live check on them shows this:

URL will be indexed only if certain conditions are met

And the contents of the section it links to there are:

Does a valid result mean that my page will be indexed?

No. The live URL test only confirms if Google-InspectionTool can access your page for indexing. There is no definitive test that can guarantee whether your page will be included in the Google index. Even if you get a valid or warning verdict in the live test, your page must still fulfill other conditions in order to be indexed. For instance:

• The page cannot be subject to any manual actions or legal issues.
• The page cannot be a duplicate of another indexed page; it must either be unique, or selected as the canonical version of a set of similar pages.
• The page quality must be high enough to warrant indexing.

I don't think these are duplicates of other indexed pages, since we are the original source for these documents, and we don't even publish summaries, we just link to them from the main resources page. I wonder if there is some sort of quality issue with how the PDFs are being ouptput that is causing Google not to index them, but not sure how to ascertain that.

There are two PDFs that are getting indexed, though, so it's not a blanket thing.

[Scotchester]

• Investigate why some docs.calitp.org URLs are not being indexed

Similar story for these: Some are being indexed (at least half the Benefits docs site), but some just aren't, for whatever inscrutable reasons that only Google knows.

[Scotchester]

• Investigate whether we can get Google to ignore querystring variations

As most of these are on the embeddable-maps.calitp.org site or outside of our control for other reasons (i.e., how third parties link to us), I don't think we should do anything on this point.