[subset] convert subset input flags into bit flags.

Store the flags in a bit set. Updates the public api to work with the bit set directly.
16 files changed