Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for de/serializing objects with json and msgpack #230

Merged
merged 22 commits into from
Nov 4, 2022

Conversation

anthrotype
Copy link
Member

@anthrotype anthrotype commented Jul 26, 2022

This adds methods for serializing / deserializing to/from JSON, MessagePack and Pickle to all the ufoLib2.objects (Font, Glyph, etc.).
It's mostly syntactic sugar around the ufoLib2.converters' structure/unstructure which provides convenience one-line methods for serialization to popular formats.

E.g.

font = Font.json_load("path/to/Font1.json")
font2 = Font.json_loads(font.json_dumps())
font2.json_dump("path/to/Font2.json")

or

font = Font.msgpack_load("path/to/Font1.msgpack")
font2 = Font.msgpack_loads(font.msgpack_dumps())
font.msgpack_dump("path/to/Font2.msgpack")

Json serialization uses orjson on CPython as it's faster than built-in json. Optional dependencies (e.g. msgpack) are handled via extras.

It's just a proof-of-concept to solicit feedback, I simply drop it here in case it may be of use in our current effort to break up fontmake into parallel processes in order to speed it up.

@anthrotype anthrotype marked this pull request as draft July 27, 2022 12:49
tox.ini Outdated Show resolved Hide resolved
@anthrotype anthrotype changed the title WIP: add support for de/serializing objects with json, msgpack or pickle add support for de/serializing objects with json, msgpack or pickle Jul 27, 2022
@anthrotype anthrotype marked this pull request as ready for review July 27, 2022 17:16
@anthrotype
Copy link
Member Author

Added some tests, I think this is now ready to review

@anthrotype
Copy link
Member Author

One thing this PR adds which I didn't mention in the description is the ability to pickle/unpickle ufoLib2 objects that are "lazy", i.e. they contain references to filesystem objects (via UFOReader or glifLib.GlyphSet) which aren't pickleable as such (because they hold a threading.RLock and that apparently is by design in pyfilesystem2).
So to make lazy objects pickleable I do like we already do with deepcopy, that is I run unlazify() on them before handling them to pickle.

@anthrotype anthrotype force-pushed the serde branch 2 times, most recently from 350ca77 to 0a8c32c Compare July 27, 2022 18:01
@anthrotype
Copy link
Member Author

I have one question for typing enthusiast like @madig

currently, since I add the json_loads/json_dumps/json_load/json_dump etc. methods dynamically to the ufoLib2 objects using the @serde class decorator, static typecheckers like mypy can't see them hance the deluge of # type: ignore whenever I use them.
I don't see how I can make mypy aware of these. Could we write typing stubs maybe? I have no idea.

So it's a bit inconvenient if one relies on typechecking to having to opt-out every time one uses these new de/serialization methods.

One alternative is to not use the class/instance methods added by the @serde decorators, but directly call the loads/dumps functions defined respectively in ufoLib2.serde.json and ufoLib2.serde.msgpack, which have type hints and don't make mypy angry.

E.g. instead of doing

from ufoLib2 import Font

font = Font.json_load("MyFont.json")  # type: ignore
font.json_dump("MyFont2.json")  # type: ignore

one could do instead:

from pathlib import Path
import ufoLib2.serde.json
from ufoLib2 import Font

font = ufoLib2.serde.json.loads(Path("MyFont.json").read_bytes(), Font)

Path("MyFont2.json").write_bytes(ufoLib2.serde.json.dumps(font, indent=2))

It's a bit more verbose and indirect but makes mypy happy.

we currently don't have any attributes that are init=True and copyable=False, the only copyable=False is Font.path which is also init=False, so simply checking if init=False is enough
and in fact I had forgotten to decorate one of them (Info) ;)
Doing font.pickle_dumps() is exactly the same as doing pickle.dumps(font), same for Font.pickle_loads(s) vs pickle.loads(s), we don't use cattrs for pickling. So it's better to simply remove the extra code, pickling still works out of the box (even with lazy objects now).
Copy link
Collaborator

@madig madig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as far as I can see 👀

So one can do ufoLib2.serde.json.load('MyFont.json', ufoLib2.Font) in alternative to ufoLib2.Font.json_load('MyFont.json')
@anthrotype anthrotype merged commit 7ec6b69 into master Nov 4, 2022
@anthrotype anthrotype deleted the serde branch November 4, 2022 18:05
@anthrotype anthrotype changed the title add support for de/serializing objects with json, msgpack or pickle add support for de/serializing objects with json and msgpack Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants