Skip to content

Commit

Permalink
Add hooks for field transformation and for asdict serialization
Browse files Browse the repository at this point in the history
  • Loading branch information
sscherfke committed Sep 9, 2020
1 parent 28d75db commit 77b8edd
Show file tree
Hide file tree
Showing 11 changed files with 397 additions and 19 deletions.
3 changes: 3 additions & 0 deletions changelog.d/653.change.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
``attr.s()`` now has a *field_transformer* hook that is called for all ``Attribute``s and returns a (modified or updated) list of ``Attribute`` instances.
``attr.asdict()`` has a *value_serializer* hook that can change the way values are converted.
Both hooks are meant to help with data (de)serialization workflows.
1 change: 1 addition & 0 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ def pytest_configure(config):
collect_ignore.extend(
[
"tests/test_annotations.py",
"tests/test_hooks.py",
"tests/test_init_subclass.py",
"tests/test_next_gen.py",
]
Expand Down
2 changes: 1 addition & 1 deletion docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Core

.. autodata:: attr.NOTHING

.. autofunction:: attr.s(these=None, repr_ns=None, repr=None, cmp=None, hash=None, init=None, slots=False, frozen=False, weakref_slot=True, str=False, auto_attribs=False, kw_only=False, cache_hash=False, auto_exc=False, eq=None, order=None, auto_detect=False, collect_by_mro=False, getstate_setstate=None, on_setattr=None)
.. autofunction:: attr.s(these=None, repr_ns=None, repr=None, cmp=None, hash=None, init=None, slots=False, frozen=False, weakref_slot=True, str=False, auto_attribs=False, kw_only=False, cache_hash=False, auto_exc=False, eq=None, order=None, auto_detect=False, collect_by_mro=False, getstate_setstate=None, on_setattr=None, field_transformer=None)

.. note::

Expand Down
104 changes: 104 additions & 0 deletions docs/extending.rst
Original file line number Diff line number Diff line change
Expand Up @@ -159,3 +159,107 @@ Here are some tips for effective use of metadata:
... x = typed(int, default=1, init=False)
>>> attr.fields(C).x.metadata[MY_TYPE_METADATA]
<class 'int'>


Automatic field transformation and modification
------------------------------------------------

Attrs allows you to automatically modify or transform the class' fields while the class is being created.
You do this by passing a *field_transformer* hook to :func:`attr.define()` (and its friends).
Its main purpose is to automatically add converters to attributes based on their type to aid the development of API clients and other typed data loaders.

| This hook must have the signature:
| :code:`your_hook(cls: type, fields: List[Attribute]) -> List[Attribute]`
- *cls* is your class right *before* it is being converted into an attrs class.
This means it does not yet have the ``__attrs_attrs__`` attribute.

- *fields* is a list of all :class:`attr.Attribute` instances that will later be set to ``__attrs_attrs__``. You can modify these attributes any way you want:
You can add converters, change types, and even remove attributes completely or create new ones!

For example, let's assume that you really don't like floats:

.. doctest::

>>> def drop_floats(cls, fields):
... return [f for f in fields if f.type not in {float, 'float'}]
...
>>> @attr.frozen(field_transformer=drop_floats)
... class Data:
... a: int
... b: float
... c: str
...
>>> Data(42, "spam")
Data(a=42, c='spam')

A more realistic example would be to automatically convert data that you, e.g., load from JSON:

.. doctest::

>>> from datetime import datetime
>>>
>>> def auto_convert(cls, fields):
... results = []
... for field in fields:
... if field.converter is not None:
... results.append(field)
... continue
... if field.type in {datetime, 'datetime'}:
... converter = (lambda d: datetime.fromisoformat(d) if isinstance(d, str) else d)
... else:
... converter = None
... results.append(field._assoc(converter=converter))
... return results
...
>>> @attr.frozen(field_transformer=auto_convert)
... class Data:
... a: int
... b: str
... c: datetime
...
>>> from_json = {"a": 3, "b": "spam", "c": "2020-05-04T13:37:00"}
>>> Data(**from_json) # ****
Data(a=3, b='spam', c=datetime.datetime(2020, 5, 4, 13, 37))


Change value serialization in ``asdict()``
------------------------------------------

Attrs allows you to serialize instances of attrs classes to dicts using the :func:`attr.asdict()` function.
However, the result can not always be serialized since most data types will remain as they are:

.. doctest::

>>> import json
>>> import datetime
>>>
>>> @attr.frozen
... class Data:
... dt: datetime.datetime
...
>>> data = attr.asdict(Data(datetime.datetime(2020, 5, 4, 13, 37)))
>>> data
{'dt': datetime.datetime(2020, 5, 4, 13, 37)}
>>> json.dumps(data)
Traceback (most recent call last):
...
TypeError: Object of type datetime is not JSON serializable

To help you with this, :attr:`attr.asdict()` allows you to pass a *value_serializer* hook. It has the signature :code:`your_hook(inst: type, field: Attribute, value: Any) -> Any`:

.. doctest::

>>> def serialize(inst, field, value):
... if isinstance(value, datetime.datetime):
... return value.isoformat()
... return value
...
>>> data = attr.asdict(
... Data(datetime.datetime(2020, 5, 4, 13, 37)),
... value_serializer=serialize,
... )
>>> data
{'dt': '2020-05-04T13:37:00'}
>>> json.dumps(data)
'{"dt": "2020-05-04T13:37:00"}'
7 changes: 7 additions & 0 deletions src/attr/__init__.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ _OnSetAttrType = Callable[[Any, Attribute[Any], Any], Any]
_OnSetAttrArgType = Union[
_OnSetAttrType, List[_OnSetAttrType], setters._NoOpType
]
_FieldTransformer = Callable[[type, List[Attribute]], List[Attribute]]
# FIXME: in reality, if multiple validators are passed they must be in a list
# or tuple, but those are invariant and so would prevent subtypes of
# _ValidatorType from working when passed in a list or tuple.
Expand Down Expand Up @@ -274,6 +275,7 @@ def attrs(
auto_detect: bool = ...,
getstate_setstate: Optional[bool] = ...,
on_setattr: Optional[_OnSetAttrArgType] = ...,
field_transformer: Optional[_FieldTransformer] = ...,
) -> _C: ...
@overload
def attrs(
Expand All @@ -297,6 +299,7 @@ def attrs(
auto_detect: bool = ...,
getstate_setstate: Optional[bool] = ...,
on_setattr: Optional[_OnSetAttrArgType] = ...,
field_transformer: Optional[_FieldTransformer] = ...,
) -> Callable[[_C], _C]: ...
@overload
def define(
Expand All @@ -319,6 +322,7 @@ def define(
auto_detect: bool = ...,
getstate_setstate: Optional[bool] = ...,
on_setattr: Optional[_OnSetAttrArgType] = ...,
field_transformer: Optional[_FieldTransformer] = ...,
) -> _C: ...
@overload
def define(
Expand All @@ -341,6 +345,7 @@ def define(
auto_detect: bool = ...,
getstate_setstate: Optional[bool] = ...,
on_setattr: Optional[_OnSetAttrArgType] = ...,
field_transformer: Optional[_FieldTransformer] = ...,
) -> Callable[[_C], _C]: ...

mutable = define
Expand Down Expand Up @@ -382,6 +387,7 @@ def make_class(
eq: Optional[bool] = ...,
order: Optional[bool] = ...,
on_setattr: Optional[_OnSetAttrArgType] = ...,
field_transformer: Optional[_FieldTransformer] = ...,
) -> type: ...

# _funcs --
Expand All @@ -397,6 +403,7 @@ def asdict(
filter: Optional[_FilterType[Any]] = ...,
dict_factory: Type[Mapping[Any, Any]] = ...,
retain_collection_types: bool = ...,
value_serializer: Optional[Callable[[type, Attribute, Any], Any]] = ...,
) -> Dict[str, Any]: ...

# TODO: add support for returning NamedTuple from the mypy plugin
Expand Down
65 changes: 56 additions & 9 deletions src/attr/_funcs.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ def asdict(
filter=None,
dict_factory=dict,
retain_collection_types=False,
value_serializer=None,
):
"""
Return the ``attrs`` attribute values of *inst* as a dict.
Expand All @@ -32,6 +33,9 @@ def asdict(
:param bool retain_collection_types: Do not convert to ``list`` when
encountering an attribute whose type is ``tuple`` or ``set``. Only
meaningful if ``recurse`` is ``True``.
:param callable value_serializer: A hook that is called for every attribute
or dict key/value. It receives the current instance, field and value
and must return the (updated) value.
:rtype: return type of *dict_factory*
Expand All @@ -40,24 +44,36 @@ def asdict(
.. versionadded:: 16.0.0 *dict_factory*
.. versionadded:: 16.1.0 *retain_collection_types*
.. versionadded:: 20.3.0 *value_serializer*
"""
attrs = fields(inst.__class__)
rv = dict_factory()
for a in attrs:
v = getattr(inst, a.name)
if value_serializer is not None:
v = value_serializer(inst, a, v)
if filter is not None and not filter(a, v):
continue
if recurse is True:
if has(v.__class__):
rv[a.name] = asdict(
v, True, filter, dict_factory, retain_collection_types
v,
True,
filter,
dict_factory,
retain_collection_types,
value_serializer,
)
elif isinstance(v, (tuple, list, set)):
cf = v.__class__ if retain_collection_types is True else list
rv[a.name] = cf(
[
_asdict_anything(
i, filter, dict_factory, retain_collection_types
i,
filter,
dict_factory,
retain_collection_types,
value_serializer,
)
for i in v
]
Expand All @@ -67,10 +83,18 @@ def asdict(
rv[a.name] = df(
(
_asdict_anything(
kk, filter, df, retain_collection_types
kk,
filter,
df,
retain_collection_types,
value_serializer,
),
_asdict_anything(
vv, filter, df, retain_collection_types
vv,
filter,
df,
retain_collection_types,
value_serializer,
),
)
for kk, vv in iteritems(v)
Expand All @@ -82,19 +106,36 @@ def asdict(
return rv


def _asdict_anything(val, filter, dict_factory, retain_collection_types):
def _asdict_anything(
val,
filter,
dict_factory,
retain_collection_types,
value_serializer,
):
"""
``asdict`` only works on attrs instances, this works on anything.
"""
if getattr(val.__class__, "__attrs_attrs__", None) is not None:
# Attrs class.
rv = asdict(val, True, filter, dict_factory, retain_collection_types)
rv = asdict(
val,
True,
filter,
dict_factory,
retain_collection_types,
value_serializer,
)
elif isinstance(val, (tuple, list, set)):
cf = val.__class__ if retain_collection_types is True else list
rv = cf(
[
_asdict_anything(
i, filter, dict_factory, retain_collection_types
i,
filter,
dict_factory,
retain_collection_types,
value_serializer,
)
for i in val
]
Expand All @@ -103,13 +144,19 @@ def _asdict_anything(val, filter, dict_factory, retain_collection_types):
df = dict_factory
rv = df(
(
_asdict_anything(kk, filter, df, retain_collection_types),
_asdict_anything(vv, filter, df, retain_collection_types),
_asdict_anything(
kk, filter, df, retain_collection_types, value_serializer
),
_asdict_anything(
vv, filter, df, retain_collection_types, value_serializer
),
)
for kk, vv in iteritems(val)
)
else:
rv = val
if value_serializer is not None:
rv = value_serializer(None, None, rv)
return rv


Expand Down
Loading

0 comments on commit 77b8edd

Please sign in to comment.