streamparse 3.13.0

This tiny release just adds the --overwrite_virtualenv flag to sparse submit and sparse update_virtualenv for the cases where you want to recreate a virtualenv without having to manually delete it from all the worker nodes. (PR #416)

streamparse 3.12.0

This release mostly improves and fixes some CLI option handling.

Features

• Added --config option to all commands to pass custom config.json path. (PR #409, Issue #343)
• Added --options option to sparse update_virtualenv, so it can properly handle customized worker lists. (PR #409)
• Switched from using prettytable to texttable, which has better line wrapping support for wide tables. (PR #413)
• Updated version of storm.thrift used internally to the one from Storm 1.1.1, which made the sparse list tables more informative. We also now verify that Storm believe the topology name you are submitting is a valid name. (PR #414)

Fixes

• Added proper option resolution for sparse update_virtualenv (PR #409)
• Fixed a couple typos in our handling of using custom Java objects for groupings. (PR #405)

streamparse 3.11.0

This release simply makes it possible to override more settings that are in config.json at the Topology level. You can now add config = {'virtualenv_flags': '-p /path/to/python3'} to have some topologies in your project using one version of Python and others using another (Issue #399, PR #402)

streamparse 3.10.0

This release just adds options to pre_submit_hook and post_submit_hook arguments. This is mostly so you can use the storm workers list inside hooks. (PR #396)

streamparse 3.9.0

This release simply adds a new feature where the storm.workers.list topology configuration option is now set when you submit a topology, so if some part of your topology needs to know the list of Storm workers, you do not need to resort to connecting to Nimbus with each executor to find it out. (PR #395)

streamparse 3.8.0

Another small release, but fixes issues with sparse tail and sparse remove_logs.

Features

• Can now specify the number of simultaneous SSH connections to allow when updating virtualenvs or accessing/removing logs from your workers with the --pool_size argument. (PR #393)
• The user that is used to remove logs from the Storm workers with sparse remove_logs can now be specified with --user (PR #393)

Fixes

• sparse tail and sparse remove_logs do a much better job of only finding the logs that relate to your specified topology in Storm 1.x (PR #393)
• sparse run will no longer crash if you have par set to a dict in your topology. (bafb72b)

streamparse 3.7.1

Fix issue where sparse run would not work without Nimbus properly configured in config.json (Issue #391, PR #392)

streamparse 3.7.0

Small release, but a big convenience feature was added.

Features

• No longer need to specify workers in your config.json! They will be looked up dynamically by communicating with your Nimbus server. If for some reason you would like to restrict where streamparse creates virtualenvs to a subset of your workers, you can still specify the worker list in config.json and that will take precedence. (PR #389)
• Added better error messages for when util.get_ui_jsons fails (commit b2a8219)
• Can now pass config_file file-like objects to util.get_config for if you need to do something like retrieve the config at runtime from a wheel. Not very common, but it is now supported. (PR #390)
• Removed the now unused decorators module.

Fixes

• Fixed a documentation issue where the FAQ still referenced the old ext module that was removed a long time ago (Issue #388)

streamparse 3.6.0

This has bunch of bugfixes, but a few new features too.

Features

• Added virtualenv_name as a setting in config.json for users who want to reuse the same virtualenv for multiple topologies. (Issue #371, PR #373)
• Support ruamel.yaml>=0.15 (PR #379)
• You can now inherit Storm Thrift types directly from streamparse.thrift instead of needing the extra from streamparse.thrift import storm_thrift (PR #380)
• Added --timeout option to sparse run, sparse list, and sparse submit so that you can control how long to wait for Nimbus to respond before timing out. This is very useful on slow connections. (Issue #341, PR #381)

Fixes

• Fixed failing fabfile.py and tasks.py imports in get_user_tasks with Python 3. (Issue #376, PR #378)
• Made Storm version parsing a little more lenient to fix some rare crashes (Issue #356)
• Added documentation on how to pass string arguments to Java constructors (Issue #357)
• Added documentation on how to create topologies with cycles in them (Issue #339)
• Fixed issue where we were writing invalid YAML when a string contained a colon. (Issue #361)
• We now pass on the serialized properly with sparse run (Issue #340, PR #382)
• sparse no longer crashes on Windows (Issues #346 and pystorm/pystorm#40, PR pystorm/pystorm#45)

streamparse 3.5.0

This release brings some compatibility fixes for Storm 1.0.3+.

Features

• Propagate topology parallelism settings to Flux when using sparse run (Issue #364, PR #365)
• Pass component configs on to Flux when using sparse run (PR #266)

Fixes

• Make sure output shows up when using sparse run (PR #363)
• Handle new resources nesting added in Storm 1.0.3 (Issue #362, PR #366)