Proposal for populates++ hook

[eddyystop]

An attempt at a design to start discussion.

• @ekryski 's proposal is basically used when we get to populating one item with another.
• Populate definitions may be stored together. This way they can easily refer to one another.
• Once an item or group of items is added, they can be further populated using 'expandBy'. This is recursive.
• This may eventually be related to models for feathers services.

I'm hoping for

• Comments on concept and packaging
• Improvements to naming.

const populates = {

  // a straight forward set of populates taken from MichaelErmer/feathers-populate-hook
  message: {
    user: { // what @ekryski proposed. Temp renames to localField & foreignField for clarity.
      service: 'users', localField: '_id', foreignField: 'userId',
      on: 'result', multiple: false, query: {}
    },
    comments: {
      service: 'comments', localField: '_id', foreignField: 'commentIds' /* is array */,
      on: 'result', multiple: true, query: {}
    },
    categories: {},
    creator: {},
  },

  // populate a room with all its messages, further populating each message.
  room: {
    creator: {},
    messages: {
      service: 'messages', localField: '_id', foreignField: 'messageIds',
      on: 'result', multiple: true, query: {},
      expandBy: populates.message, // expand every message with populates.message
    },
    owners: {},
  },

  // populate an organization with its rooms, partially further populating each room
  organization: {
    owners: {},
    rooms: {
      service: 'rooms', localField: '_id', foreignField: 'roomIds',
      on: 'result', multiple: true, query: {},
      expandBy: [populates.room, 'messages', 'owners'], // don't expand using populates.room.creator
    }
  },
};

const organizations = app.service('organizations');

organizations.after({
  find: [
    // the fancy populating
    hook.population(populates.organization),

    // You can continue to populate an item the same way as with the current populate
    hook.population({
      creator: {
         service: 'users', localField: '_id', foreignField: 'creatorId',
         on: 'result', multiple: false, query: {}
      }
    }),
  ]
});

[ekryski]

Looks really good @eddyystop! I would probably keep it as hook.populate instead of hook.population. I like the localField and foreignField terms.

Can you explain what you were thinking this does?

organization: {
    owners: {},
    rooms: {
      service: 'rooms', localField: '_id', foreignField: 'roomIds',
      on: 'result', multiple: true, query: {},
      expandBy: [populates.room, 'messages', 'owners'], // I don't get this part
    }
  },

From my understanding calling hook.populate(populates.organization) would bring back the organization, owners for an org, rooms for an org, messages in a room, and owners of a room. Is that correct?

Maybe I'm getting tripped up by populates.room, would you not just have ['messages', 'owners'] in that case?

I was also thinking that maybe instead of expandBy it should be include?

[corymsmith]

I had the same thought as @ekryski re: expandBy: [populates.room, 'messages', 'owners'], that one confused me, wouldn't that one just be expandBy: [populates.room.messages, populates.room.owners]? Or maybe I'm completed misunderstanding it...

[ekryski]

Ya or what @corymsmith suggested.

[eddyystop]

(1) @ekryski @corymsmith expandBy: [populates.room, 'messages', 'owners'] was intended to mean expand as per populates.room but only using its messages and owners props.

populates.room is essentially a "view" on the DB. My experience is that a reasonably sized app has many views, some similar to one another. You could end up with this populates.room including creators, messages, owners and then having populates.room1 including messages and owners, then having populates.room2 including just messages.

So I'm looking for a syntax which says use the includes only from the following props.

expandBy: [populates.room.messages, populates.room.owners] exposes us to people writing stuff like expandedBy; [populates.room.message.comments, populates.somethingTotallyUnrelated] when I'd have no idea what they want.

Perhaps you can suggest a more intuitive syntax.

(2) This syntax would not be compatible with the present populate syntax. Do we want to totallly deprecate the current one?

(3) I like 'include' more than 'expandBy'.

(4) hook.populate(populates.organization) would add all the owners items and rooms items to an organization item. It would also add creator, messages and owner items to each room item. It would also add all user, comments, categories, creator items to each message item.

[eddyystop]

(5) As an aside, we can have localField: [a1, a2, a3], foreignField: [b1, b2, b3] which would form query: { a1: b1, a2: b2, a3: b3 }.

Perhaps you can suggest a different syntax for these multi-field situations.

(6) The field names need to support dot notation (destination.addressLine.1). We may decide to allow an array notation (desctination.address[]) which would have to loop thru the array. ugh

[ekryski]

Hmmmm 🤔

What we're doing is population + serialization schemes (to some extent). Sort of reminds me of Rails' active model serializers. Think I'm going to have to stew on this a bit. It's definitely headed in the right direction.

(3) I like 'include' more than 'expandBy'.

Ok let's go with include.

This syntax would not be compatible with the present populate syntax. Do we want to totally deprecate the current one?

I'm good with deprecating and inserting a breaking change. It's not a brutal change to migrate I don't think and we can always set the old one to legacyPopulate if need be to make it a simple find and replace for people.

[eddyystop]

Perhaps populate would be even better than include. It would be rather clear what populate does since its used within hook.populate().

[eddyystop]

Ideas from ekryski's Rails link:

• only: []: props to incl
• except: []: props to exclude
• methods: []: each func adds a prop: value where prop is the func name. Is the func passed the base item as a param? What about arrow funcs which have no names? Should we use computed: { propName: (base) => {} } syntax?

The part related to populate:

// to include associations
{ include: :posts }
// Second level and higher order associations
{ include: {
  posts: { 
    only: :title,
    include: { comments: { only: :body } }
  },
}}

I prefer using established keywords and syntax, so I'd go with only, except and the include keyword. I'd rather use computed than methods.

The syntax above makes use of relationships between tables defined elsewhere, sorta like we've proposed.
Our syntax may be more flexible than Rails'.

• Rails posts may have to be the table name while, with messages1: { service: 'messages', we separate the definition name messages1 and the service name messages. So we can define multiple relationships between messages and users.
• The Rails syntax seem to require you to specify all the nested included items every time. Our hook.population(populates.organization), brings in nested items by default.

I considered allowing nested properties in excludeBy but then decided something simpler would be less confusing to people. I have no objection to turtling down.

[eddyystop]

Here is a more Rails flavored syntax.

const populates = {
  post: {
    author: {
      service: 'users',
      // the following 2 lines are mutually exclusive
          localField: '_id', foreignField: 'userId', // can use when relation is a single field
          getItems: (post) => ({ _id: post.userId, ifDeleted: false }), // more complicated relations
      // the following 2 lines are mutually exclusive
          nameItemAs: 'author', // save first item found as an object
          nameItemsAs: 'authors', // save all items found as an array
      placeWhere: 'result',
      query: {},
      // from Rails active model serializer
      only: ['email', 'username', 'birthDate'],
      except: ['password', 'verifyToken', 'verifyTokenShort'],
      computed: {
        // how do we later remove these when we want to 'update' the table?
        // (a) add a _computed_: ['over18'] to item or (b) name the prop _over18
        over18: (user, post) => Date.now() < (user.birthDate + 18 * 100000), // create a prop over18
      },
    },
    comments: {
      service: 'comments',
      localField: 'postId', foreignField: '_id',
      nameItemsAS: 'comments',
    },
    categories: {
      service: 'categories',
      localField: '_id', foreignField: 'postCategories', // categories is an array so use $in
      nameItemsAs: 'categories',
    },
  },
  favorites: {
    posts: {
      service: 'posts',
      localField: '_id', foreignField: 'postIds',
      nameItemsAs: 'posts',
      include: {
        post: {
          // Override part of the post.author definition
          include: { author: { only: ['username', 'birthDate'], } },
          // Ignore post.categories. Don't include categories items for a post.
          exclude: ['categories']
        }
      },
    }
  },
};

favorites.after({
  find: [
    hook.population(populates.favorites),
    // or
    hook.population({
      posts: {
        service: 'posts',
        localField: '_id', foreignField: 'postIds',
        nameItemsAs: 'posts',
        include: { post: {} },
      },
    }),
  ]
});

[ekryski]

Could methods just be an object? I don't think order matters so much in that case. That would force the developer to provide a name for each function. They are pretty much the same as virtuals (virtual attributes which is more similar to mongoose) so maybe that could be an alternative name. I prefer either computed or virtuals over methods.

I think those methods should just be passed the hook object and the base object and would be expected to return a promise (in case you need to do some async in order to set the attribute).

I was also looking at the Sail ORM solution and we already support everything they do. Mongoose's syntax is not super clear and also doesn't support deep populates well. Not much to draw from there.

This last iteration feels like we are venturing into ORM territory. Not entirely because they aren't defined on the models, and may not need to rely on the models in order to do population. In my mind this style of populate is more for "serializing data out" (but could be for populating prior to save).

Just brain dumping but I'm wondering if we are getting a little too complex with this function (trying to do too much). Including and excluding might be better defined as subsequent hooks in the hook chain. However, one thing I have been thinking a lot about is defining different serializations (or views) of the data based on the permissions of the client requesting that data (which this is starting to venture into). Basically "serialization policies". No one has solved that in a nice way yet.

---

A couple notes that are more specific to the above example:

1. I would consolidate nameItemAs and nameItemsAs into just name or nameAs. I don't think it matters to us whether it is an array or not. If they want to pluralize an object they are more than welcome to. Then there is less API surface area.

2. I don't think you would do except alongside only. Maybe that was just you demonstrating all the different keys we might have.

3. I don't think you would have an include block in this instance:

   hook.population({
     posts: {
       service: 'posts',
       localField: '_id', foreignField: 'postIds',
       nameItemsAs: 'posts',
       include: { post: {} },
     },
   }),

4. I'm assuming we are on the same page that by default when an item is populated all fields are returned unless otherwise specified or the service has a hook removing fields.

---

I'm going to think about this and put together an example or two. I already started but my hunch is that we are mixing serialization and population and that really they should be 2 different things and that in general when we populate we should bring in everything and then make a second pass with another hook to serialize data out or sanitize data in appropriately.

[eddyystop]

About mixing serialization and population: Beeplin has made a comment that items could have a lot of fields (I agree) and its better to drop fields on the find than waiting till later. How do you feel about this?

I really like the idea of including permissions in deciding on how to populate as they are one of the big reasons for different populate definitions. How are permissions exposed in feathers-permissions?

I agree we are getting complex with this function. Perhaps the question is whether the function will be very simple to use for the simple cases and for the majority of cases? Further, will it be performant enough for the complex ones?

I prefer computed over virtuals as its more obvious what computed means. They are indeed objects now. I would pass them hook, localObject and foreignObject as well.

About your (3) above. We need include to know what to populate each post item with.

We need to know if we add an array of objects or add an object. Perhaps we have objectName and arrayName. Or name and asArray. What do you think?

[eddyystop]

About this nested included stuff. Its just to temporarily override values within an object.

We could just clone the object and let the user use dot notation to customize it.

[ekryski]

This is my stab at thinking through a realistic example for populating and serialization. This would support having data in different services that are backed by different data stores. By default everything is included so I'm only using exclude but you could also use include if you wanted to only include certain properties.

Example 1: NoSQL Posts + SQL Comments

// Example Data
const Favourites = [
  {
    userId: 'as61389dadhga62343hads6712',
    postId: '1'
  },
  {
    userId: 'as61389dadhga62343hads6712',
    postId: '2'
  },
  {
    userId: '167asdf3689348sdad7312131s',
    postId: '1'
  }
];

const Posts = [
  {
    id: '1',
    title: 'Post 1',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    comments: ['1', '3'],
    createdAt: ''
  },
  {
    id: '2',
    title: 'Post 2',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: '167asdf3689348sdad7312131s',
    comments: ['2'],
    createdAt: ''
  }
];

const Comments = [
  {
    id: '1',
    title: 'Comment 1',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: '2',
    title: 'Comment 2',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: '3',
    title: 'Comment 3',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: '167asdf3689348sdad7312131s',
    createdAt: ''
  }
];

const Users = [
  {
    _id: 'as61389dadhga62343hads6712',
    name: 'Author 1',
    email: '[email protected]',
    password: '2347wjkadhad8y7t2eeiudhd98eu2rygr',
    age: 55
  },
  {
    _id: '167asdf3689348sdad7312131s',
    name: 'Author 2',
    email: '[email protected]',
    password: '2347wjkadhad8y7t2eeiudhd98eu2rygr',
    age: 16
  }
];

const serializers = {
  favorites: {
    exclude: ['userId'] // we don't need the userId as we likely queried for the user's favourites already
    post: {
      // Possibly we can use this as a short hand syntax? In this case comments.createdAt wouldn't do anything because that field wasn't included in the original query
      exclude: ['comments.createdAt']
      author: {
        exclude: ['password', 'createdAt'],
        computed: {
          isChild: (author) => author.age < 18,
        }
      }
    },
  }
};


const populations = {
  // NOTE: We're not populating the user on the favourite object
  favorite: {
    on: 'result', // where to place the populated item on the hook object
    post: {
      service: 'posts', // The service to populate from
      localField: 'postId', // The field local to the favourite. Only needed if different than the key name.
      foreignField: 'id', // The key of the populated object to use for comparison to the id stored
      nameAs: 'post', // Key name to place the populated items on (optional. Defaults to the key name in the object)
      author: {
        service: 'users', // The service to populate from
        foreignField: '_id' // The key of the populated object to use for comparison to the id stored
      },
      comments: {
        service: 'comments', // The service to populate from
        foreignField: 'id', // The key of the populated object to use for comparison to the id stored
        query: { // Normal feathers query syntax. Get the title and body of the last 5 comments
          $limit: 5,
          $select: ['title', 'content'],
          $sort: { createdAt: -1 }
        },
      }
    }
  }
};

favorites.after({
  find: [
    hook.populate(populations.favourite),
    hook.serialize(serializers.favourite)
  ]
});

[daffl]

What if comments are linked with a postId instead of the array in posts?

[ekryski]

About mixing serialization and population: Beeplin has made a comment that items could have a lot of fields (I agree) and its better to drop fields on the find than waiting till later. How do you feel about this?

Personally I don't think we should optimize on that too early. There is definitely going to be a cost to ripping through objects in JS twice to exclude or include items as opposed to getting the DB to do it but I think that is an optimization we can introduce if we are finding it's not fast enough, as opposed to doing it up front and making the populate hook really complex.

The problem with the current hooks was that previously until you added dot notation there wasn't an easy way include/exclude nested properties, so I think people gravitated to that external hook the other guy wrote.

For permissions really they are defined wherever you want. Currently it is assumed in the database associated with the entity that has limitations (ie. user, device, organization, etc.) but I think we'd be able to cook up something that allows different serialization based on permissions. I think that would be easily supported by what I proposed above as it would just be a different serializer definition.

About your (3) above. We need include to know what to populate each post item with. We need to know if we add an array of objects or add an object. Perhaps we have objectName and arrayName. Or name and asArray. What do you think?

I might be missing something here but I think we can look at the object we are going to populate into (ie. favourites and posts in the example above) and check to see if the field we are referencing is an array or singular. There shouldn't be a need for the developer to specify it explicitly I don't think. Maybe something that will flesh out during implementation.

[ekryski]

What if comments are linked with a postId instead of the array in posts?

That might be what @eddyystop is referring to. Let me modify and think about that. my damn nosql brain

[eddyystop]

What if comments are linked with a postId instead of the array in posts?

localField: 'postId', // field name in comments
foreignField: '_id', // field name in posts

If an array in posts

localField: '_id', // field name in comments
foreignField: 'commentIds', // name in posts. since this is an array at run time, a $in is used in the query

[ekryski]

Example 2: SQL Posts + Comments

// Example Data

// Favourites stored in SQL DB
const Favourites = [
  {
    userId: 'as61389dadhga62343hads6712',
    postId: 1
  },
  {
    userId: 'as61389dadhga62343hads6712',
    postId: 2
  },
  {
    userId: '167asdf3689348sdad7312131s',
    postId: 1
  }
];

// Posts stored in SQL DB
const Posts = [
  {
    id: 1,
    title: 'Post 1',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: 2,
    title: 'Post 2',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: '167asdf3689348sdad7312131s',
    createdAt: ''
  }
];

// Comments stored in SQL DB
const Comments = [
  {
    id: 1,
    postId: 1,
    title: 'Comment 1',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: 2,
    postId: 2,
    title: 'Comment 2',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: 3,
    postId: 1,
    title: 'Comment 3',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: '167asdf3689348sdad7312131s',
    createdAt: ''
  }
];

// Users stored in NoSQL DB
const Users = [
  {
    _id: 'as61389dadhga62343hads6712',
    name: 'Author 1',
    email: '[email protected]',
    password: '2347wjkadhad8y7t2eeiudhd98eu2rygr',
    age: 55
  },
  {
    _id: '167asdf3689348sdad7312131s',
    name: 'Author 2',
    email: '[email protected]',
    password: '2347wjkadhad8y7t2eeiudhd98eu2rygr',
    age: 16
  }
];

const serializers = {
  favorites: {
    exclude: ['userId'] // we don't need the userId as we likely queried for the user's favourites already
    post: {
      // Possibly we can use this as a short hand syntax? In this case comments.createdAt wouldn't do anything because that field wasn't included in the original query
      exclude: ['comments.createdAt']
      author: {
        exclude: ['password', 'createdAt'],
        virtuals: {
          isChild: (author) => author.age < 18,
        }
      }
    },
  }
};


const populations = {
  // NOTE: We're not populating the user on the favourite object
  favorite: {
    on: 'result', // where to place the populated item on the hook object
    post: {
      service: 'posts', // The service to populate from
      localField: 'postId', // The field local to the favourite. Only needed if different than the key name.
      foreignField: 'id', // The key of the populated object to use for comparison to the id stored
      author: {
        service: 'users', // The service to populate from
        foreignField: '_id' // The key of the populated object to use for comparison to the id stored
      },
      comment: {
        service: 'comments', // The service to populate from
        localField: 'post.id', // compare the post's id field to the comments postId field
        foreignField: 'postId',
        nameAs: 'comments', // Key name to place the populated items on (optional. Defaults to the key name in the object)
        asArray: true,
        query: { // Normal feathers query syntax. Get the title and body of the last 5 comments
          $limit: 5,
          $select: ['title', 'content'],
          $sort: { createdAt: -1 }
        },
      }
    }
  }
};

favorites.after({
  find: [
    hook.populate(populations.favourite),
    hook.serialize(serializers.favourite)
  ]
});

[eddyystop]

(1) My meaning for localField and foreignField seem to be the reverse of yours. See my comment above yours.

(2) In the post object, do we really want to consider every prop that's not a "reserved name" to define a relationship? Typos, etc?

[ekryski]

@eddyystop I'm not sure your suggestion covers that. Don't we need to be comparing the post's id to the comment's postId field? How would we denote that?

I was thinking it could be:

// parsing the tree is always assumed to start from the root
// so local props you use `this.id` or `./id`. Sort of handlerbars-ish.
// don't really love that idea....
comment: {
   localField: 'post.id'
}

// or this. Assume everything is references local to the object
// scope and you can back out.
// Also don't love this...
comment: {
   localField: '../id'
}

// or this. I personally think this would be much
// easier to parse in the hook.
comment: {
   localField: {
     field: 'id',
     from: 'post'
   },
   foreignField: 'postId'
}

[ekryski]

(1) My meaning for localField and foreignField seem to be the reverse of yours. See my comment above yours.

Ya I must be confused then. I don't think I follow which is which. Maybe you can update the your example with comments spelling out what the foreignField and localField would be in relation to the example?

(2) In the post object, do we really want to consider every prop that's not a "reserved name" to define a relationship? Typos, etc?

Riiiiiiggght. Ya maybe we should keep that include syntax in the population scheme.

[ekryski]

@eddyystop thanks for updating that. I think it might still be a bit confusing. I'd rather keep it so that foreignField is the field to look at when fetching docs from the service, and localField is local to the object you are trying to populate into. That could just be me though...

@feathersjs/core-team suggestions?

[eddyystop]

Since we have

post: {
  comment: {
    ekryskiLocalField: '_id', // this is always from the item that is including comment i.e. post
    ekryskiForeignField: 'postId', // this is alwys from the item be are including i.e. comment
}}

People used to Entity Relation Diagrams, Third Normal Form and SQL will have a firm idea of what foreign key should mean. A foreign key is something you have which is the primary key of another entity. So in our syntax perhaps foreignField -- I avoided using foreignKey -- should mean something that's in another table, not in our table.

BTW, that's what f_key and l_key mean in MichaelErmer/feathers-populate-hook

[ekryski]

@eddyystop yup I follow. Ok final example with those changes:

Example 3: NoSQL Users, SQL Posts + Comments + Favourites

// Example Data

// Favourites stored in SQL DB
const Favourites = [
  {
    userId: 'as61389dadhga62343hads6712',
    postId: 1
  },
  {
    userId: 'as61389dadhga62343hads6712',
    postId: 2
  },
  {
    userId: '167asdf3689348sdad7312131s',
    postId: 1
  }
];

// Posts stored in SQL DB
const Posts = [
  {
    id: 1,
    title: 'Post 1',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: 2,
    title: 'Post 2',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: '167asdf3689348sdad7312131s',
    createdAt: ''
  }
];

// Comments stored in SQL DB
const Comments = [
  {
    id: 1,
    postId: 1,
    title: 'Comment 1',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: 2,
    postId: 2,
    title: 'Comment 2',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: 'as61389dadhga62343hads6712',
    createdAt: ''
  },
  {
    id: 3,
    postId: 1,
    title: 'Comment 3',
    content: 'Lorem ipsum dolor sit amet, consectetur adipisicing elit. Possimus, architecto!',
    author: '167asdf3689348sdad7312131s',
    createdAt: ''
  }
];

// Users stored in NoSQL DB
const Users = [
  {
    _id: 'as61389dadhga62343hads6712',
    name: 'Author 1',
    email: '[email protected]',
    password: '2347wjkadhad8y7t2eeiudhd98eu2rygr',
    age: 55
  },
  {
    _id: '167asdf3689348sdad7312131s',
    name: 'Author 2',
    email: '[email protected]',
    password: '2347wjkadhad8y7t2eeiudhd98eu2rygr',
    age: 16
  }
];

const serializers = {
  favorites: {
    exclude: ['userId'] // we don't need the userId as we likely queried for the user's favourites already
    post: {
      // Possibly we can use this as a short hand syntax? In this case comments.createdAt wouldn't do anything because that field wasn't included in the original query
      exclude: ['comments.createdAt']
      author: {
        exclude: ['password', 'createdAt'],
        virtuals: {
          isChild: (author) => author.age < 18,
        }
      }
    },
  }
};


const populations = {
  // NOTE: We're not populating the user on the favourite object
  favorite: {
    on: 'result', // where to place the populated item on the hook object
    include: {
      post: {
        service: 'posts', // The service to populate from
        localField: 'postId', // The local field to the parent (ie. favourite). Only needed if different than the key name.
        foreignField: 'id', // The field of populated object to use for comparison to the localField
        include: {
          author: {
            service: 'users', // The service to populate from
            localField: 'author', // The local field to the parent (ie. post)
            foreignField: '_id' // The field of populated object to use for comparison to the localField
          },
          comment: {
            service: 'comments', // The service to populate from
            localField: 'id', // The local field to the parent (ie. post)
            foreignField: 'postId', // The field of populated object to use for comparison to the localField
            nameAs: 'comments', // Key name to place the populated items on (optional. Defaults to the key name in the object)
            asArray: true,
            query: { // Normal feathers query syntax. Get the title and body of the last 5 comments
              $limit: 5,
              $select: ['title', 'content'],
              $sort: { createdAt: -1 }
            },
          }
        }
      }
    }
  }
};

favorites.after({
  find: [
    hook.populate(populations.favourite),
    hook.serialize(serializers.favourite)
  ]
});

[eddyystop]

Test of initial populate hook, with trace logs.

https://github.com/eddyystop/feathers-test-populate-etc

[kc-dot-io]

@eddyystop nice! Any chance you can add some performance timing? In the past, I've found a big issue with populating data on json and key value stores is the performance of making multiple queries, especially at the 3rd to Nth levels, as each population requires and additional call. This can be pretty heavy if the data structure is an array of arrays, so it would be ideal for us to have some consideration around bench marks when it comes to this so people are thinking about it up front when they design their data models. Food for thought, but great work here!

[eddyystop]

@slajax that's a good idea and I'll keep it in mind.

On the one hand, the data is presently being populated in place within the
hook hoping for performance.

On the other hand, I've listened to fasinating, but a bit dated, talks
about how V8 starts toiling when the schema of objects changes.

On Wed, Nov 9, 2016 at 3:14 PM, Kyle Campbell [email protected]
wrote:

@eddyystop https://github.com/eddyystop nice! Any chance you can add
some performance timing? In the past, I've found a big issue with
populating data on json and key value stores is the performance of making
multiple queries, especially at the 3rd to Nth levels, as each population
requires and additional call. This can be pretty heavy if the data
structure is an array of arrays, so it would be ideal for us to have some
consideration around bench marks when it comes to this so people are
thinking about it up front when they design their data models. Food for
thought, but great work here!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#38 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABezn38FHryZSMlkB50cce8hqULQbipbks5q8imIgaJpZM4Krox_
.

[daffl]

One optimization for performance when populating an array of items would be to collect all ids and then make one request with { query: { ids: { $in: ids } } }.

It looks like the JSON schema is going to have a lot of options. One thought I had was to provide a more intuitive query builder interface to create a hook. Something like

populate('users')
  .from('userId')
  .withService('/users')
  .withQuery(function(hook) {
    return {
      active: true
    }
  })

Also, can we create a spec document for this? It's a pretty long thread. I almost feel that this is one of those cases I mentioned where it would make sense for it to live in its own module. To me the common hooks repo is kind of like a "Hooks Lodash" and this one needs a lot more context and documentation than the standard Lodash documentation style.

[eddyystop]

@daffl The $in operator is used when the parent item contains an array of child ids, that is parentField evaluates to an array.

Check out https://github.com/eddyystop/feathers-test-populate-etc for the latest specs and implementation of both the populate and serialize hooks

[eddyystop]

https://github.com/eddyystop/feathers-test-populate-etc is where this is being prototyped.