Distinguish between public and internal headers

[kirklandsign]

Right now all headers are in https://github.com/pytorch-labs/tokenizers/tree/main/include/pytorch/tokenizers.

That probably includes internal ones.

Users doesn't know what headers they are expected to use ("exported headers") in their own project.

Say in linux, they put internal headers in src/ like https://github.com/torvalds/linux/blob/master/drivers/accel/qaic/mhi_controller.h, or have some subdir like https://github.com/torvalds/linux/tree/master/include/crypto/internal, whereas exported headers are in https://github.com/torvalds/linux/tree/master/include/crypto

[larryliu0820]

yeah makes sense. We can start to move headers to internal/ I guess. We need to see if this is feasible though, given the code is shared internally