Avoiding type piracy for custom serialization of type that I do not own

[rasmushenningsson]

If we want custom serialization for a type that I do not own, I currently need to wrap it in my own type to avoid type piracy for JLD2.writeas:

struct MyWrappedType
    x::TypeIDontOwn
end

JLD2.writeas(::Type{MyWrappedType}) = ...

But this only works to a certain degree, if I want to serialize a Vector/Tuple/Pair/etc. that contains TypeIDontOwn, it becomes much harder.

One solution would be if it was possible to supply a "context" object to e.g. jldopen, that was passed along to the relevant functions. Then we could write

JLD2.writeas(::MyContext, ::Type{TypeIDontOwn} = ...`

thus avoiding piracy. (This strategy is used by e.g. StableHashTraits.jl.)
It would naturally fallback to calling without a context, so this is not a breaking change.

Is there any other way to achieve this currently?

[JonasIsensee]

Hi @rasmushenningsson ,

currently, there is no generic solution, as you found out yourself.

I like the idea of the context objects for dispatch.
However, I am a bit unsure about what this will do to the compiler.
JLD2 has always been struggling to hit the right amount of compilation vs runtime dispatching
and I'm not certain, if there is a reasonably performant solution that does not invalidate EVERYTHING when being passed now context types.

If you are keen to try it out, I'm happy to review (even exploratory) PRs.

[rasmushenningsson]

You raise a valid point about compilation, and I don't know all details, but basically I think that:

• If this mechanism isn't used, everything should stay the same as it is today.
• If this mechanism is used, the user could be expected to add precompile statements (if they need to speed up compilation, and we can't get it to be fast without it).

Another questions is whether it is worth complicating the code base for something that most users will never need. If the changes are small, it might be worth it.

I don't have the bandwidth to make a PR right now, but I might do one down the line. I'm glad you like the idea in principle.

[cstjean]

I agree that it shouldn't affect performance for users of the default context, with the caveat that Julia's method table is (was?) indexed on the type of the first argument, so you might still see some performance loss.

But this only works to a certain degree, if I want to serialize a Vector/Tuple/Pair/etc. that contains TypeIDontOwn, it becomes much harder.

Perhaps you're aware, but all of those are parametric, so you can write JLD2.writeas(::Type{Vector{TypeIDontOwn}}, ...) = .... That's not piracy.

[rasmushenningsson]

That's true, we need to measure performance to see if there is some performance loss or not.

Perhaps you're aware, but all of those are parametric, so you can write JLD2.writeas(::Type{Vector{TypeIDontOwn}}, ...) = .... That's not piracy.

I don't know if there is a formal definition of type piracy, but this could definitely cause problems, since I don't own any of the types.
If I do this in a package for e.g. Vector{Float64}, then a user using JLD2 while my package is loaded will save their Vector{Float64} using my custom definition, which is really bad.

[cstjean]

Oh, I'm sorry, I don't know what I was thinking. You're absolutely right.

Out of curiosity, what's your usecase?

[rasmushenningsson]

My use case is that I have a package for analyzing some domain-specific data and it needs to cache/memoize some things.
So I want to:

• Make sure I don't affect any user code dealing with JLD2.
• Have custom serialization of some (external) types to guarantee long-term stability. The argument here is that the exact contents of a struct is often an implementation detail, so if I serialize using one version, it might not work later. Thus I prefer to serialize a more "semantic" representation, e.g. a DataFrame could be serialized as a list of names and a vector per column.