Skip to content

Check performance of hard-coded parameters to export parquet files #277

@ArBridgeman

Description

@ArBridgeman

Summary

In #267, we hard-coded customer-sensible values to prevent easily overlooked mistakes. We noted a slight worsening in performance, and it would be good to identify which of these affect performance. If a particular parameter has a poor performance, we should add an option for user to override these parameters, provide documentation on how to adapt these, and make it clear that is potentially unsafe behavior (how & why).

To Do

  • Run performance tests with different parameter configurations
    • Note that this could need to be executed even more than the current performance tests, as we get different machines, etc. & the difference is O(0.4s) so our sensitivity needs to higher
  • Isolate which parameters customers would potential want to override & expose those via **kwargs
  • Update documentation -> user guide & docstring
  • Create a major release -> this will most likely be a breaking change as **kwargs would potentially need to be split to go to different options (parse_options, dataset_options, etc.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureProduct feature
    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions