-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Format error messages for queries that fail validation #312
Comments
Since switching our validation implementation to malli from clojure.spec, we have more options for implementing this. A good low effort first step that probably gets us 90% of what we want could be enabling humanized error messages. If humanized error messages prove not to be good enough, we can then implement custom error formatters that offer more flexibility. |
Moving to |
@cap10morgan I think the example that caused us to resurrect this issue was providing an invalid query key. Like I think @aaj3f tried using
|
that looks like it is humanized though? So if it needs to be better than this, then we may need some more guidance on what that should look like |
@mpoffald @cap10morgan if nothing else, I think we should return consistent error formats on 400s. It seems wrong to me that some errors (captured by the query resolution paths) always have two keys, Error #1 {
"error": "db/invalid-query",
"message": "Invalid predicate: schema:foobar"
} Error #2 {
"value": {
"query": {
"select": [
"?name"
],
"whereFoobar": [
[
"?s",
"schema:name",
"?name"
]
]
},
"ledger": "cookbook/base"
},
"type": "reitit.coercion/request-coercion",
"coercion": "malli",
"in": [
"request",
"body-params"
],
"humanized": {
"query": {
"where": [
"missing required key"
]
}
}
} |
Not only for readability (I think v3 users might scratch their heads at |
@aaj3f On latest #error {
│ :cause "Invalid Query"
│ :data {:status 400, :error :db/invalid-query, :reasons {:where ["missing required key"], :select ["invalid type" "invalid type"], :whereFoobar ["invalid type" "invalid type"]}} ...} I can add a |
The malli "humanized" error messages probably aren't enough for a person using Fluree who didn't take part in implementing it. "Invalid query" for every possible error is also extremely frustrating for a user. I think we should write code that takes malli error maps, filters out the irrelevant info (of which there's a lot), and turns the remaining relevant info into English. So the |
There is a ton more work that could conceptually fall under this issue but would be a huge scope explosion for what this started out as. We should design, write up, and prioritize that additional work in future issues. |
This is going to be an uber-issue to cover the overall effort to improve these validation error messages. We will hopefully link to separate issues that cover various categories of errors (to better scope and parallelize the work) and when all of those are done, this can be closed. |
Sub-issues that fall under the scope of this (please edit this comment to add to the list and strikethrough completed sub-issues rather than adding additional comments): |
I reviewed error messages from v2 vs v3, here's what I found regarding regressions that seemed to be introduced by the move to malli. We will probably never be done improving error messages, so this can't be a truly exhaustive list, but I think by addressing these we can improve our overall approach to reworking/presenting malli's error messages. Inscrutable Some caveats:
Consistent error formatsIn many cases, our error messages were fine, but the keys in the map aren't consistent with other errors. We should use Example: {
"coercion": "malli",
"humanized": {
"history": [
"should be a string",
"should be a keyword",
"invalid type",
"invalid type",
"invalid type"
],
"malli/error": [
"Must supply either a :history or :commit-details key."
],
"t": [
"missing required key"
]
},
"in": [
"request",
"body-params"
],
"type": "reitit.coercion/request-coercion",
"value": {
"from": "fluree/387028092977569",
"history": null
}
} This issue is common to a lot of our error messages and we do this wrapping in one place, so I'll elide pointing this out in the rest of the list. History queries
Slightly different formatting issue, the useful explanation is buried in @(fluree/history ledger {:history nil})
;;=>
#error
{:cause "History query not properly formatted. Provided {:history nil}"
:data {:status 400, :message {:history ["should be a string" "should be a keyword" "invalid type" "invalid type" "invalid type"], :t ["missing required key"], :malli/error ["Must supply either a :history or :commit-details key."]}, :error :db/invalid-query}}
V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/api.clj#L809 (throw (ex-info (str "Please specify an subject for which to search history. Provided: " history)
{:status 400
:error :db/invalid-query}))
This doesn't clearly explain what went wrong: @(fluree/history ledger {:commit-details true :t {:from -1 :to 0}})
;;=>
#error
{:cause "History query not properly formatted. Provided {:commit-details true, :t {:from -1, :to 0}}"
;;This message needs to be improved, and the whole thing reformatted
:data {:status 400, :message {:t {:from ["should be :latest" "should be at least 0" "should match regex"]}}, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "History query not properly formatted. Provided {:commit-details true, :t {:from -1, :to 0}}"
:data {:status 400, :message {:t {:from ["should be :latest" "should be at least 0" "should match regex"]}}, :error :db/invalid-query}
:at [fluree.db.api.query$history$fn__60507$state_machine__7787__auto____60512$fn__60515$fn__60524 invoke "query.cljc" 113]}]} AFAIK this is equivalent to (when (> block-start db-block)
(throw (ex-info (str "Start block is out of range for this ledger. Start block provided: " (pr-str block-start) ". Database block: " (pr-str db-block)) {:status 400 :error :db/invalid-query}))) Select clause
This doesn't tell you much about what went wrong, or how to do it correctly: {:select [+]
:where '[[?s ?p ?o ]]}
;;=>
#error
{:cause "Invalid query: {:select [#function[clojure.core/+]], :where [[?s ?p ?o]], :opts {:issuer nil}} - select: unknown error should be a string should be a keyword unknown error unknown error invalid type"
:data {:status 400, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid query: {:select [#function[clojure.core/+]], :where [[?s ?p ?o]], :opts {:issuer nil}} - select: unknown error should be a string should be a keyword unknown error unknown error invalid type"
:data {:status 400, :error :db/invalid-query}
:at [fluree.db.query.fql.syntax$coerce_query invokeStatic "syntax.cljc" 212]}]} V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L282 (or (every? #(or (string? %) (map? %)) select-smt)
(throw (ex-info (str "Invalid select statement. Every selection must be a string or map. Provided: " select-smt) {:status 400 :error :db/invalid-query}))) Where clauseDoesn't say which part of the query should have at most 1 element, extraneous
{:select ['?name '?email]
:where [['?s :type :ex/User]
['?s :schema/age '?age]
['?s :schema/name '?name]
{:union [[['?s :ex/email '?email]]
[['?s :schema/email '?email]]]
:filter ["(> ?age 30)"]}]}
;;=>
#error
{:cause "Invalid query: {:select [?name ?email], :where [[?s :type :ex/User] [?s :schema/age ?age] [?s :schema/name ?name] {:union [[[?s :ex/email ?email]] [[?s :schema/email ?email]]], :filter [\"(> ?age 30)\"]}], :opts {:issuer nil}} - where: [\"should have at most 1 elements\" \"invalid type\" \"invalid type\"]"
:data {:status 400, :error :db/invalid-query} ,,,} V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L464 (throw (ex-info (str "Where clause maps can only have one key/val, provided: " map-clause)
{:status 400 :error :db/invalid-query}))
The most useful part, Also {:select ['?name '?age]
:where [['?s :type :ex/User]
['?s :schema/age '?age]
['?s :schema/name '?name]
{:foo "(> ?age 45)"}]}
;;=>
#error
{:cause "Invalid query: {:select [?name ?age], :where [[?s :type :ex/User] [?s :schema/age ?age] [?s :schema/name ?name] {:foo \"(> ?age 45)\"}], :opts {:issuer nil}} - where: {:foo [\"should be either :filter, :optional, :union or :bind\"], :malli/error [\"invalid dispatch value\" \"invalid type\" \"invalid type\"]}"
:data {:status 400, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid query: {:select [?name ?age], :where [[?s :type :ex/User] [?s :schema/age ?age] [?s :schema/name ?name] {:foo \"(> ?age 45)\"}], :opts {:issuer nil}} - where: {:foo [\"should be either :filter, :optional, :union or :bind\"], :malli/error [\"invalid dispatch value\" \"invalid type\" \"invalid type\"]}"
:data {:status 400, :error :db/invalid-query}
:at [fluree.db.query.fql.syntax$coerce_query invokeStatic "syntax.cljc" 212]}]
,,,} V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L501 (throw (ex-info (str "Invalid where clause, unsupported where clause operation: " clause-type)
{:status 400 :error :db/invalid-query}))
Doesn't tell you what exactly is the wrong type, or what the type of that thing should be: {:select '[?s ?o]
:where 's}
;;=>
#error
{:cause "Invalid query: {:select [?s ?o], :where s, :opts {:issuer nil}} - where: invalid type"
:data {:status 400, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid query: {:select [?s ?o], :where s, :opts {:issuer nil}} - where: invalid type"
:data {:status 400, :error :db/invalid-query}
:at [fluree.db.query.fql.syntax$coerce_query invokeStatic "syntax.cljc" 212]}]
,,,} V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L685 (when-not (sequential? where)
(throw (ex-info (str "Invalid where clause, must be a vector of tuples and/or maps: " where)
{:status 400 :error :db/invalid-query})))
Doesn't tell you what a valid '{:select [?name (count ?favNums)]
:where [[?s :schema/name ?name]
[?s :ex/favNums ?favNums]]
:group-by {}}
;;=>
#error {
:cause "Invalid query: {:select [?name (count ?favNums)], :where [[?s :schema/name ?name] [?s :ex/favNums ?favNums]], :group-by {}, :opts {:issuer nil}} - group-by: unknown error invalid type"
:data {:status 400, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid query: {:select [?name (count ?favNums)], :where [[?s :schema/name ?name] [?s :ex/favNums ?favNums]], :group-by {}, :opts {:issuer nil}} - group-by: unknown error invalid type"
:data {:status 400, :error :db/invalid-query}
:at [fluree.db.query.fql.syntax$coerce_query invokeStatic "syntax.cljc" 212]}]
,,,
} V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/fql.cljc#L332 (cond (vector? groupBy) [true (map symbol groupBy)]
(string? groupBy) [false [(symbol groupBy)]]
:else (throw (ex-info
(str "Invalid groupBy clause, must be a string or vector. Provided: " groupBy)
The {:select ['?name '?favNums]
:where [['?s :schema/name '?name]
['?s :schema/age '?age]
['?s :ex/favNums '?favNums]]
:orderBy ['(foo ?favNums)]}
;;=>
#error {
:cause "Invalid query: {:context {:schema \"http://schema.org/\", :ex \"http://example.org/ns/\"}, :select [?name ?favNums], :where [[?s :schema/name ?name] [?s :schema/age ?age] [?s :ex/favNums ?favNums]], :orderBy [(foo ?favNums)], :opts {:issuer nil}} - orderBy: unknown error should be a list"
:data {:status 400, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid query: {:context {:schema \"http://schema.org/\", :ex \"http://example.org/ns/\"}, :select [?name ?favNums], :where [[?s :schema/name ?name] [?s :schema/age ?age] [?s :ex/favNums ?favNums]], :orderBy [(foo ?favNums)], :opts {:issuer nil}} - orderBy: unknown error should be a list"
:data {:status 400, :error :db/invalid-query}
:at [fluree.db.query.fql.syntax$coerce_query invokeStatic "syntax.cljc" 212]}]
:trace
,,,
} V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L310 (when-let [orderBy (or (:orderBy opts) orderBy)]
(if (or (string? orderBy)
(and (vector? orderBy)
(#{"DESC" "ASC"} (first orderBy))))
(if (vector? orderBy) orderBy ["ASC" orderBy])
(throw (ex-info (str "Invalid orderBy clause, must be variable or two-tuple formatted ['ASC' or 'DESC', var]. Provided: " orderBy)
{:status 400
:error :db/invalid-query}))))
Needs more explanation of {:select [?firstLetterOfName ?name ?canVote]
:where [[?s :schema/age ?age]
[?s :schema/name ?name]
{:bind [?canVote (>= ?age 18)]}]}
;;=>
#error {:cause "Invalid query: {:select [?firstLetterOfName ?name ?canVote], :where [[?s :schema/age ?age] [?s :schema/name ?name] {:bind [?canVote (>= ?age 18)]}], :opts {:issuer nil}} - where: {:bind [\"invalid type\"], :malli/error [\"invalid type\" \"invalid type\"]}"
:data {:status 400, :error :db/invalid-query}
,,,}
```
V2: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L492
```clj
(throw (ex-info (str "Invalid where clause, 'bind' must be a map with binding vars as keys "
"and binding scalars, or aggregates, as values.")
{:status 400 :error :db/invalid-query}))
More ;;1.
{:select ['?name '?age]
:where [['?s :type :ex/User]
['?s :schema/age '?age]
['?s :schema/name '?name]
{:filter "(> ?age 45)"}]}
;;=>
;;#error
{:cause "Invalid query: {:select [?name ?age], :where [[?s :type :ex/User] [?s :schema/age ?age] [?s :schema/name ?name] {:filter \"(> ?age 45)\"}], :opts {:issuer nil}} - where: {:filter [\"invalid type\"], :malli/error [\"invalid type\" \"invalid type\"]}"
:data {:status 400, :error :db/invalid-query}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid query: {:select [?name ?age], :where [[?s :type :ex/User] [?s :schema/age ?age] [?s :schema/name ?name] {:filter \"(> ?age 45)\"}], :opts {:issuer nil}} - where: {:filter [\"invalid type\"], :malli/error [\"invalid type\" \"invalid type\"]}"
:data {:status 400, :error :db/invalid-query}
:at [fluree.db.query.fql.syntax$coerce_query invokeStatic "syntax.cljc" 212]}] ,,,}
;;2
{:select ['?name '?age]
:where [['?s :type :ex/User]
['?s :schema/age '?age]
['?s :schema/name '?name]
{:filter :foo}]}
;;=>
#error
{:cause "Invalid query: {:select [?name ?age], :where [[?s :type :ex/User] [?s :schema/age ?age] [?s :schema/name ?name] {:filter :foo}], :opts {:issuer nil}} - where: {:filter [\"invalid type\"], :malli/error [\"invalid type\" \"invalid type\"]}"
:data {:status 400, :error :db/invalid-query}}
V2 examples: https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical.cljc#L762 (throw (ex-info (str "Filter must be enclosed in square brackets. Provided: " filters)
{:status 400
:error :db/invalid-query})) https://github.com/fluree/db/blob/maintenance/v2/src/fluree/db/query/analytical_parse.cljc#L426 (if-not (sequential? filter)
(throw (ex-info (str "Filter clause must be a vector/array, provided: " filter)
{:status 400 :error :db/invalid-query})) |
The query engine defined in #304 returns error responses containing the raw parse validation failures under the
reasons
key:We should format these error messages to make them more clear, actionable, and user friendly.
Blocked by #304The text was updated successfully, but these errors were encountered: