Policy-enforced query fails when policy is transacted first

[mpoffald]

Related issues/discussions:

• https://github.com/fluree/core/issues/61 (listed as bug #2)
• Add pending test for missing predicate in policy-enforced query result #694 (comment)

Given the following data:

    (def conn @(fluree/connect {:method :memory}))

    (def context {"ex"     "http://example.org/"
                   "schema" "http://schema.org/"
                   "f"      "https://ns.flur.ee/ledger#"})

    (def ledger @(fluree/create conn "test/policy-txn-order"))

    (def did "did:fluree:Tf5M4L7SNkziB4Q5gC8Hjuqu9WQKCwKpU1Y")

    (def data-txn {"@context" context
                   "insert"   [{"@id"            "ex:betty",
                                "@type"          "ex:Yeti",
                                "schema:name"    "Betty"
                                "schema:age"     55
                                "schema:follows" [{"@id" "ex:freddy"}]},
                               {"@id"         "ex:freddy",
                                "@type"       "ex:Yeti",
                                "schema:name" "Freddy",
                                "schema:age"  1002},
                               {"@id"            "ex:letty",
                                "@type"          "ex:Yeti",
                                "schema:name"    "Leticia",
                                "schema:age"     38
                                "schema:follows" [{"@id" "ex:freddy"}]}
                               {"@id"            "ex:andrew",
                                "@type"          "schema:Person",
                                "schema:age"     35,
                                "schema:name"    "Andrew Johnson",
                                "schema:follows" [{ "@id" "ex:freddy" },
                                                  { "@id" "ex:letty" },
                                                  { "@id" "ex:betty" }]}]})
    (def identity-txn {"@context" context
                       "insert"   {"@id"     did
                                   "ex:user" {"@id" "ex:freddy"}
                                   "f:role"  {"@id" "ex:yetiRole"}}})
    (def policy-txn {"@context" context
                     "insert"
                     {"@id"           "ex:yetiPolicy",
                      "@type"         ["f:Policy"],
                      "f:targetClass" {"@id" "ex:Yeti"},
                      "f:allow"       [{"@id"          "ex:yetiViewAllow",
                                        "f:targetRole" {"@id" "ex:yetiRole"},
                                        "f:action"     [{"@id" "f:view"}]}],
                      "f:property"    [{"@id"     "ex:yetisViewOnlyOwnAge",
                                        "f:path"  {"@id" "schema:age"},
                                        "f:allow" [{"@id"          "ex:ageViewRule",
                                                    "f:targetRole" {"@id" "ex:yetiRole"},
                                                    "f:action"     [{"@id" "f:view"}],
                                                    "f:equals" { "@list" [{"@id" "f:$identity"}
                                                                          {"@id" "ex:user"}] }}]}]}})

Transacted in this order (policy, followed by data):

    (def db-policy-first (-> ledger
                             fluree/db
                             (fluree/stage policy-txn)
                             deref
                             (fluree/stage identity-txn)
                             deref
                             (fluree/stage data-txn)
                             deref))

We do not get back the query results we expect. We expect to see ex:freddy's schema:age (and no one else's), because the query is being performed using that identity, but we don't get any ages back.

    @(fluree/query db-policy-first {"@context" {"schema" "http://schema.org/"}
                                    :where     '{"@id"         ?s
                                                 "schema:name" "?name"}
                                    :select    '{?s ["*"]}
                                    :opts      {:did did}})
    ;;=>
    [{"@id" "http://example.org/betty",
      "@type" "http://example.org/Yeti",
      "schema:follows" {"@id" "http://example.org/freddy"},
      "schema:name" "Betty"}
     {"@id" "http://example.org/freddy",
      "@type" "http://example.org/Yeti",
      ;;Freddy's age is missing
      "schema:name" "Freddy"}
     {"@id" "http://example.org/letty",
      "@type" "http://example.org/Yeti",
      "schema:follows" {"@id" "http://example.org/freddy"},
      "schema:name" "Leticia"}]

In contrast, if we tranasct the data first, before the policy:

    (def db-data-first (-> ledger
                           fluree/db
                           (fluree/stage data-txn)
                           deref
                           (fluree/stage policy-txn)
                           deref
                           (fluree/stage identity-txn)
                           deref))

We get back the expected results:

    @(fluree/query db-data-first {"@context" {"schema" "http://schema.org/"}
                                  :where     '{"@id"         ?s
                                               "schema:name" "?name"}
                                  :select    '{?s ["*"]}
                                  :opts      {:did did}})
    ;;=>
   [{"@id" "http://example.org/betty",
     "@type" "http://example.org/Yeti",
     "schema:follows" {"@id" "http://example.org/freddy"},
     "schema:name" "Betty"}
    {"@id" "http://example.org/freddy",
     "@type" "http://example.org/Yeti",
     ;;Freddy's age is returned, and no one else's
     "schema:age" 1002,
     "schema:name" "Freddy"}
    {"@id" "http://example.org/letty",
     "@type" "http://example.org/Yeti",
     "schema:follows" {"@id" "http://example.org/freddy"},
     "schema:name" "Leticia"}]

Initial investigation findings

The schema:age predicate is not present in the db's cache when we transact the policy first:

    (-> @(fluree/wrap-policy db-policy-first {:did did})
        :schema
        :pred
        (get "http://schema.org/age"))
    ;;=>
    nil

Here it is on the db where we transacted the data first:

    (-> @(fluree/wrap-policy db-data-first {:did did})
        :schema
        :pred
        (get "http://schema.org/age"))
    ;;=>
    {:id 1001,
     :class true,
     :idx? true,
     :ref? false,
     :subclassOf #{},
     :equivalentProperty #{},
     :iri "http://schema.org/age"}

The correct flakes are being returned by query process here: https://github.com/fluree/db/blob/main/src/fluree/db/query/exec/select.cljc#L164, but during the iri lookup process for displaying the query results (https://github.com/fluree/db/blob/main/src/fluree/db/query/json_ld/response.cljc#L87), we are not finding the predicate and therefore failing to return it in the final results.

[JaceRockman]

I couldn't reproduce this issue when following the steps, so I think it has been fixed by other changes that we have made.