Better support for ?include_attributes and ?exclude_attributes API query param filters #4300
Conversation
__json__ method. This method gets called just before the response back to the user and serializing API object to JSON.
…h API model" This reverts commit 80f1d40.
the router. This way the filtering is performed at the very end, before returning raw response to the user. Filtering on the router makes more sense compared to filtering on the API model because filtering the object might make it an invalid API model (aka object won't pass API model validation anymore) and filtering is purerly a property of the response and not the API model.
|
Another thing to keep in mind - due to how we handle our resource models (actions, sensors, rules, triggers, etc.), fields which form a For resource models those fields are |
All of those models now contain "ref" attribute which is already included in the response so there is no need to have this feature anymore.
attribute are provided. If those attributes are provided, response will likely be partial and fail validation.
|
@enykeev I'm still working on adding support for those query param filters to all the resource API endpoints and improved tests, but the core functionality is ready for review. Main changes are pretty straight forward and include:
|
key (reference) so they need to be always included in the response.
execution.runner.runner_parameters field from the database. This is needed so we can correctly determine and mask secret parameters.
|
On a related note - it's actually a good idea that we do this parameter filtering at the very and don't skip the DB model and API model layer. Secrets masking happens inside the DB and API layer which means skipping it would be a bad idea. Right now the function which handles filtering (including, exclusion) gets passed in an object which has already been masked so there is no way to "fish out" secret parameter. I also plan to spend a good amount of time next week writing test cases for all the various edge cases to make sure this functionality doesn't negatively affect the security (notable secret parameters and related stuff). |
the all resource get all / list API endpoints.
affected controllers.
exclude and include attributes query param filters.
because that would break secrets masking. Also add test cases for various edge cases.
|
On a related note - this pull request also exposed a lot of issues with the current handling of |
Kami
changed the title
get all response when ?include_attributes filter is provided. Those responses don't make sense without model primary keys.
works with ?include_attributes filtering.
include_attributes filtering. This controller doesn't directly return a database object, but a new one based on the database object value so we need to take this into account. Also update code to only retrieve fields we need.
policies API controller.
It breaks it and it's not needed nor wanted.
string, not actually importing from st2stream).
This pull request improves handling of the
?include_attributesand?exclude_attributesAPI query param filters.With those query parameters user can specify which attributes it wants to receive in the API response. Among other things, it also controls which fields we retrieve from the database.
Because of the way our API models and responses work, we convert database document to the API model before serializing that to JSON and returning it as a response. API model performs object validation and sets the default values for the fields which are not provided.
That means, for example, if user only requests
namefield it may get many other fields back, depending on the default values specified on the API model jsonschema (and those fields will use default values from the API model schema which is confusing and invalid).The problem is related to the issue (deficiency in our API model handling) I've already mentioned a long time ago. We are abusing API model for two purposes - input and output. We really should have two models (one for input and one for output) and this way we would avoid issues like that.
Sadly that would require a lot more work so I'm going with the middle of the ground approach to begin with.
As per discussion with @enykeev, I decided to perform this filtering inside the router. This way it happens very last, just before serializing the response to JSON and returning it back to the user.
In the first attempt, I tried to do it inside the API model, but it really doesn't belong there. API model represents data which has been validated and it's complete and valid (based on the model jsonschema). API model with some fields missing could potentially be invalid, hence that data shouldn't be represented by the API model.
Current approach is not the most efficient, because we still convert each database document to the API model before filtering / processing the response, but it should be a start. Future optimization would involve skipping the API model layer all together when either
?exclude_attributesor?include_attributesquery param filter is provided.That involves a lot more work and refactoring though (we would potentially need to touch and change each API controller because a lot of them override parent controller "_get_all" method and handle API model instantiation themselves).
TODO
/v1/actions/v1/actions/views/overview/v1/rules/v1/rules/views/v1/runnertypes/v1/sensortypes/v1/ruleenforcements/v1/ruleenforcements/views/v1/traces/v1/triggertypes/v1/triggerinstances/v1/actionalias/v1/executions/v1/inquiries/v1/packs/v1/policytypes/v1/policies