Use raw format to GET data from IPFS gateway#24
Conversation
Bundle Size ReportChunks over 500 KB:
All files
Commit: 91a7c12 |
⚡ Performance Report
|
karolk91
requested review from
BigTava,
Cyr06130,
KarimJedda,
ba1uev,
justraman,
leonardocustodio and
tallesborges
as code owners
|
Looks sensible, thank you @karolk91 . Let's deploy it then we can merge, sharing with the team |
leonardocustodio
left a comment
leonardocustodio
left a comment
There was a problem hiding this comment.
We might need to evaluate whether this will not break other things.
We request the .car file to use the trustless gateway option, as stated here.
Removing that would "remove" this feature, which would not be a problem for applications uploaded through bulletin-deploy, e.g., as they are uploaded as .car already, so getting them as raw would just make it return as a ' .car ' for those; this would actually be very good, as it would get rid of the double-wrap.
The problem is with data that is not uploaded through bulletin-deploy, where it would indeed come as a file rather than a .car.
We have a mechanism that verifies the .car file against the CID to ensure the integrity of the fetched files. I'm not sure if this change will break it.
Were you able to fully test e2e both multiple files uploaded and a .car file to see if everything keeps working correctly?
As far as I'm aware by looking at the code - this change here doesn't modify anything related to CAR handling, these are separate branches in the code dotli-community/packages/content/src/fetch.ts Lines 162 to 164 in 80bedce and dotli-community/packages/content/src/fetch.ts Lines 148 to 150 in 80bedce This change is also in line with trustless gateway design as you linked: https://docs.ipfs.tech/reference/http/gateway/#trusted-vs-trustless I don't expect this impacting "double-wrap" in any way
I tested this change by verifying if various apps in the Playground->Apps are loading properly - and I checked that some of them were just "single file" (raw codec) and some of them were "multi file" (dag-pb codec) based. Any specific user story you have in mind I could go through to confirm all is ok? |
|
This is a Nova-side outage and not a dot.li test failure. Re-run Logs: https://github.com/paritytech/dotli-community/actions/runs/27603606552 |
|
Nice, sounds good to me @karolk91 thanks 😀 |
|
❌ E2E Product suite failed on Failed tests:
Logs: https://github.com/paritytech/dotli-community/actions/runs/27611015323 |
4 participants
Content can fail to load via the IPFS-gateway backend with:
Content hash mismatch for — refusing tampered content (via ipfs-gateway)The gateway sits behind Cloudflare, which rewrites
text/htmlresponses in-flight — Email Address Obfuscation, Auto Minify, Rocket Loader, HTTPS rewrites, beacon injection, and similar. Once the body is modified, the delivered bytes no longer hash to the CID, and content verification correctly rejects them.In the observed case the trigger was Email Obfuscation stripping the
<!--email_off-->…<!--/email_off-->markers around amailto:link.This only affects the IPFS-gateway path. The P2P/bitswap path returns the raw block untouched.
The fix
These transforms only apply to
text/html, so we request the raw block as a binary type instead of a content-negotiableGET:The gateway then returns
application/vnd.ipld.raw, which Cloudflare passes through verbatim.Scope and caveats
fetchFromIpfs(single raw-codec 0x55 blocks).dag-pbalready uses?format=car, which is also binary and unaffected.Example affected site
shawn-hphq24.dot — e.g. https://shawn-hphq24.paseo.li/?chainBackend=rpc-gateway