Support state-dependent action space

[zsunberg]

Not sure if it is easy to support action spaces in this solver, but that would be a good modification for someone to do. It is not currently supported.

[dressel]

I'm not entirely sure how we would do this. For example, we take a maximum over a', which is the action you would take at s'. The set of available actions would be different for all your s' in your sum, and I'm not sure how to handle that theoretically. Also, how do you update alpha_a for an action s that doesn't support action a?

[zsunberg]

Oh, ok. I see. Yeah, it's not obvious how to do it.

[zsunberg]

Would it make more sense to have it be observation-dependent? That would have bearing on JuliaPOMDP/POMDPs.jl#226. It does not seem very obvious to me

[dressel]

The more I think about it, do state-dependent action spaces make sense for POMDPs? As you generally don't know your state so it's not like you can limit your actions. I think it makes more sense to just have actions not do anything (have no effect) from certain states.

It seems like observation-dependent action spaces might be feasible. In the summation over observations, you can just take the maximum over the a' that you can take having seen that observation.

I probably won't add this, but if anyone wants to do so (and to take a rigorous theoretical look to make sure it is indeed ok), I'd be ok with that.

[zsunberg]

Yeah - the question of whether state-dependent actions even make sense for POMDPs is key in JuliaPOMDP/POMDPs.jl#226. As you say, you don't know your state, so it makes more sense to make the invalid actions do nothing/ have negative consequences.