Add options to exclude objects from dump_session()

[leogama]

Additions and changes in this PR by file

__init__.py

• Rolled back bad reordering of name imported from ._dill
• Added the function is_pickled_module() to dills namespace
• Renamed imported submodule logger to logging
• Moved import reload to inside the function where it's used to clear it from the global namespace

_dill.py

• Added a paragraph to the module's docstring pointing to the main functions (with links in the Sphinx generated document)
• Added from __future__ import annotations
• Use the submodule dill.logging in place of the stdlib logging module due to the new TRACE logging level
• Rolled back the reordering of names as in __init__.py
• Moved the import of dataclasses up, close to other imports, and moved the definition of EXTENSION_SUFFIXES down, close to a new PYTHONPATH_PREFIXES constant and the definition of _is_builtin_module(), which uses both.
• Renamed singletontypes to IPYTHON_SINGLETONS, a more descriptive name
• Joined some exception types from pickles() into a new constant UNPICKLEABLE_ERRORS, which is also used elsewhere for refonfail
• Created a helper function _getopt() to get an option from a named argument, falling back to settings, to reduce the clutter
• Pickler:
  • Added a description to the dispatch attribute and a Sphinx directive to hide the giant and uninformative representation of the dictionary that was included in the documentation page
  • Added flags _refimported, _refonfail and _first_pass with default (False) values to the Pickler class (should this remove the need to test for pickler._session before testing for them? not related to is_dill(pickler) tests)
  • Substituted the _file attribute of Pickler instances by a more specific _file_tell, used by the logger to compute the partial pickle sizes —following the self.file_write pattern from the pickle module (should this also be public?)
• Pickler.save():
  • No need to test for the truthiness of Numpy*Type in each if test with a previous test of NumpyArrayType
  • Implementation of rollback if object fails to pickle with refonfail
  • An unpickleable variable in __main__ or in a module-type object raises an exception
  • An unpickleable variable in another module makes the whole module to be saved by reference with a warning
• Substitute bytes('0', 'UTF-8') instances by the pickle.POP constant
• Moved _module_map() and _lookup_module() back here from session.py, as they are also used by the refonfail feature
• _module_map() now receives a module as argument and doesn't include it in the module map, instead of making _lookup_module() to test if the found object's module is not main
• _lookup_module():
  • Now follows a priority based on the hierarchy of modules discussed in Session: improvements to documentation + handle unpickleable objects #527: Standard Library modules > same package module > installed modules > not installed modules
  • If the module related to the object is the module being saved, it returns not_found
  • If the module related to the object is in the same top-level package as the module being saved but the module where the object is found is not, this reference is discarded
• New _global_string() helper function
• New _save_module_dict() function, currently called by save_module_dict() for the main module at session saving when refonfail is set. Uses the recursive strategy of Pickler.save() to try to save a variable or any of its attributes by reference if there are pickling errors, and use _lookup_module() as a last resort for the whole object. Follows the stratified rules discussed at Session: improvements to documentation + handle unpickleable objects #527.
• save_module_dict():
  • Dictionaries are now compared by identity
  • Added a branch for session module's dict with refonfail
  • Use the constant GLOBAL and bytes literals in opcode strings
• New @_weak_cache decorator that implements a simple cache for results with a weakref.WeakKeyDictionary, to not keep references to the modules passed as argument to the functions that use it
• _is_builtin_module():
  • It now may be called multiple times from _lookup_module() and _save_module_dict(), possibly for the same few modules
  • Because of that, use @_weak_cache to speed things up a bit
  • Moved part of the calculation to the definition of a new constant PYTHONPATH_PREFIXES outside the function
  • Important: it now returns False for __main__ (and __mp_main__), which is necessary for the correct behavior of refonfail. It should not affect any existing code (I can't imagine a real world example that would be changed by this)
• New function _is_stdlib_module(), which is also called multiple times and thus uses @_weak_cache for performance.
• New function _module_package() that returns the top-level package of a module or None if it's not part of a package. Maybe this and the function above should be merged, returning 'stdlib' or '__stdlib__' as the "package" of Standard Library modules
• save_module():
  • Rewrote/formatted/annotated the if-else branch dealing with the main module at session saving with the refonfail option set
  • Check the IPython singletons' __module__ before removing them
  • Save the modified dictionary in the pickler._main_dict_copy attribute, which is used by save_module_dict() later
• save_type(): use stacklevel=3 for warnings to reduce clutter in repeated warning messages
• save_function(): removed the _main_modified flag; _original_main is set conditionally

_utils.py

• New module with utilities used by more tha one submodule, created to circunvent circular import problems
• Defines some public functions and base types, etc. currently used only by dill.session, but which may be used for other tasks in the future:
  • classes: FilterRules, FilterSet, NamedObject
  • functions: size_filter
  • constants (enums): EXCLUDE, INCLUDE
  • type hints: Filter, FilterFunction, Rule, RuleType
• Defines private functions used by other modules (should these be in __all__?):
  • functions: _format_bytes_size, _open
• _open(): function that returns a file/stream context manager and takes care of assuring that it has the necessary methods (e.g. peek or seek)

logging.py (renamed from logger.py)

• Exposes the logging level constants (e.g. INFO) and the getLogger() function from the Standard Library module logging, so that this submodule can be used in place of that in dill's modules
• Defines an intermediary logging level named TRACE between INFO and DEBUG, used by the pickling trace functions
• Now the INFO logging level can be used to display only informative messages, for example, about variables filtered out by dump_module()
• Added capability to roll back the stack trace on pickling failure, used by the refonfail feature
• The logger.trace(pickler, "X: ...", ...) call now must receive the object being pickled as either 1. the last positional argument, or 2. the obj keyword argument. Added examples
• New roll_back() method of TraceAdapter
• Moved bytes formatting to a new _format_bytes_size() function in _utils.py, which is also used by size_filter()
• trace() now accepts a string representation of a logging level (e.g. "INFO") as argument

session.py

• Moved _module_map() and _lookup_module() to _dill.py
• Added the constant BUILTIN_CONSTANTS, used by _stash_modules()
• _stash_modules():
  • Moved code dealing with self-references and the __loader__ attribute to a new function _fix_module_namespace()
  • More special cases: don't save empty strings and small integers by reference, as these may have been cached by the interpreter (CPython) or the compiler
  • Only look up for objects by id if they match IMPORTED_AS_TYPES or IMPORTED_AS_MODULES
  • Log list of variables saved by reference by the refimported feature
• New function _filter_vars() to apply the exclude/include filters passed as arguments to dump_module() or set in dill.session.settings
• New function _discard_added_variables to remove empty variables possibly added by ModuleType()
• New function _fix_module_namespace() joining code that was spread across multiple functions
• dump_module():
  • Added keyword-only parameters refonfail, exclude, include and base_rules; made refimported keyword-only too
  • Added documentation about filters usage
  • Use the helper function _open() as context manager in place of the try-except-finally block
  • Removed the Pickler._main_modified attribute, pickler._original_main is set conditionally, if the main module was modified by _filter_vars() or _stash_modules()
  • If refonfail is True, set the attributes _refonfail, _fille_seek and _file_truncate to allow Pickler.save() to roll back the stream if pickling any object fails
  • Log list of unpickleable variables saved by reference by the refonfail feature
• Moved _PeekableReader class to _utils.py
• _identify_module() now uses a more strict test to determine if a pickle stream is a module pickle
• New function is_pickled_module() that tests whether a pickle file was created by dump_module() and if it's module is importable or not —if the identify argument is True, return the module name instead of True
• load_module():
  • Complemented the documentation about the module parameter and the return value
  • Use the context manage _open() like in dump_module()
  • Removed the attribute Unpickler._main, the only use of this was removed in a previous commit
• load_module_as_dict():
  • Complete rewrite of the function body, it was flawed; now it always load the module
  • Removed the update parmeter
  • Also use the _open() context manager
• New class ModuleFilter derived from FilterRules, that can hold filters for specific modules
• New dictionary dill.session.settings with settings specific for the module
• New "filter factory" function ipython_filter() for excluding IPython hidden variables from __main__

tests/

• New test file test_filtering.py for testing the different of types of filters (input and usage); exclude and include filters interaction; plus filter_size()
• test_session.py:
  • New test for the _lookup_module() prioritization when multiple versions of an object are found in different modules
  • New test for the refonfail behavior according to the proposed stratification of cases between "builtin" (installed) and "user" (not installed) modules
  • New test for ipython_filter() using a simulated IPython interactive namespace

docs/source/

• conf.py:
  • Set default options for the automodule directives in dill.rst
  • Added default exclude-members attributes that are either implementation details or are redundant and just clutter the documentation page: __dict__, __slots__, __module__, __annotations__, __dataclass_fields__ and _abc_impl
• dill.rst: removed automodule options set as default in conf.py

---

Linked issues

1. Resolves should dump_session accept a list/dict of objects to ignore? #66 with new parameters exclude and include in dump_module()*
2. Resolves Failure to dump_session after get_ipython #255 with the use of dill.session.ipython_filter()
3. Resolves dill.dump_session file size #345 with the use of dill.session.size_filter()
4. Resolves some errors with dump_module and load_module #525
5. Solution to (2) can be used as a workaround to Can't dump_session with inline plots #4 and related isssues

(*) renamed from dump_session()

---

Remaining tasks to reach the 0.3.6 milestone

• fix current test error and some errors with dump_module and load_module #525
• unroll the filtering loop
• play with the functions and classes
• write documentation
• create a unpickleable filter(?) -> refonfail
• create a settings/filters file parser with configparser (reference: logging.config) — moved to Config file reader and format specification #545

Originally in #527

Documentation:

• Describe the effects of passing the module argument to load_module()
• Better describe the side effects and the usage of the returned value from load_module()
• Document the function is_pickled_module(filename) (what about its name and signature?)
• Explain the distinction between calling dill.dump(module, file) and dill.dump_module(file, module), and the assumptions and use cases for each alternative — probably in the package's main documentation, or in a separate session.py module's docstring

Code:

• Make is_pickled_module() return False for modules saved by name (and for modules saved by value inside other objects) — saved an empty dict as 'state' for modules saved by reference by refonfail, which also allows load_module_asdict() to read such files.
• Deal with unpickleable objects in modules
• Deal with unpickleable variables in the module's namespace

Review

• Write unit tests — any critical test missing?
• Rewrite/delete some added comments (e.g. _save_module_dict())

---

Original post

This is a proposed feature for excluding objects from the namespace to be saved with dill.dump_session().

Currently implemented are exclusion rules that discard global variables when the variable:

• name matches an exact string
• name matches a regular expression (in a full match, i.e. re.search('^%s$' % regex, name) matches)
• id matches
• is instance of a type
• returns True when passed to a filter function

It adds two new parameters to dump_session(): exclude and filters. The exclude argument can be a mixed list of variable names, regular expressions, object ids and type objects, in any order. The filters argument can be either a single filter function that receives an object and returns True if it should be discarded, or a list of such functions.

For example, if the user has a variable my_pi = 3.14 in the global namespace, any of the following calls would not save my_pi:

dill.dump_session(exclude=["my_pi"])
dill.dump_session(exclude=[re.compile(r"\w+_\w+")])
dill.dump_session(exclude=[id(my_pi)])
dill.dump_session(exclude=[float])
dill.dump_session(filters=lambda x: x.startswith("my_"))

Beyond that, there are two new entries in the settings dictionary called session_exclude and session_filters, in the case the user wants to have the same rules applied to multiple calls to dump_session(). The session_exclude entry is itself a dictionary with four keys: ids, names, regex and types. Its values, and also session_filters, are instances of a subclass of set with an associated constructor function that can transform the object upon addition or do some checks before adding. This is a facility for including/excluding non trivial rules like regular expressions and abstract types.

Important: Rules from settings are merged with those from dump_session() arguments, not overwritten by them.

Here are examples of usage:

exclude = dill.settings.session_exclude

exclude.ids.add(var)              # stores id(var)
exclude.regex.add(r"__.+__")      # stores re.compile(r"__.+__")
exclude.types.add("Function")     # stores types.FunctionType

exclude.regex.discard(r"__.+__")  # does what is expected
exclude.types.clear()             # empties the types list

Finally, if trace(True) is set, dump_session() will list the names and types of the variables excluded before the pickling trace.

---

If accepted, this feature will serve as the base to implement better support for saving IPython sessions (possibly with a filter function that the user can pass to the filters parameter or add to settings).

Related issues: #4, #66, #345

Quote from #4 (this is old! 😄):

@mmckerns commented on Oct 5, 2014

I don't know of any current workaround, but having a list or dict of objects-to-ignore in dump_session is an interesting idea. Hmm.

P.S.: I chose "exclude" because "ignore" was already taken —and it's more faithful to the feature, as "ignore" is more faithful to ignoring objects that fail pickling.

[leogama]

@mmckerns The only new feature I'd like to see in dill 0.3.6 is this one (not necessarily exactly like it is now, I'm open to implementation changes).

By the way, I'm thinking if the hard support drop for Python 2.7-3.6 wouldn't deserve a major version bump to 1.0. (One dot oh, yay! 🥳) It would be a clear signaling for the users. At least I would go read the release notes if one of my dependencies rolled out a new major release.

[anivegesana]

~It would be good for this PR and #424 to have the same interface. Is there a reason that you only filter on name, regex, and type as opposed to filtering using more general functions. In psuedocode, a list of filters could be represented like:

def has_name(name):
  def _filter(obj):
    return name == obj.__name__

def has_type(t):
  def _filter(obj):
    return isinstance(obj, t)

[
  (ACCEPT, has_name("x")),
  (REJECT, has_type(int))
]

Since one argument is already known to be the object itself, we could get rid of the currying and just do this:

def has_name(obj, name):
    return name == obj.__name__

def has_type(obj, t):
    return isinstance(obj, t)

[
  (ACCEPT, has_name, "x"),
  (REJECT, has_type, int)
]

The benefit of the first option is that we can overload the has_name and has_type functions to accept types, strings, and regexes, and ints for the id without degrading runtime performance. We could also do the function call silently to get the best of both worlds.

This allows people to make custom filters based on the contents of the object as opposed to just the name and the type. The names ACCEPT and REJECT draw inspiration from iptables and probably should be changed to something else because this is not a firewall. Maybe INCLUDE and EXCLUDE. Now that Python 2 is deprecated, these can be enum values.

Should we go in this direction?~

Ah I see. That is fine I guess. Why not put the filters and other excluding options under the same name?

[leogama]

Why not put the filters and other excluding options under the same name?

Uhmm... I wrote this code a month ago and I can't remember why I kept filters separated from the rest. I think that I was concerned about both types and filters being callable and that they couldn't be distinguished safely. However, only types are, well, types and I can't think of a type constructor that could be used as a filter —just bool, that would work as a filter for non-empty containers and sequences, and almost everything else, and consequentially would allow saving only empty containers and sequences, zeros and Nones. But that is absurd, a more reasonable use of "bool" as filter would be indirectly with lambda x: not x or operator.not_.

I'll will unify the parameters and also rewrite the commits so that the code moving to session.py is after (or before?) all the changes and additions, so you can review it more easily. If you feel that the API is good enough, let me now so that I can write the documentation. I'll start creating the unit tests right now.

There are more things to discuss from your comment...

[leogama]

About the API in general, accepting only "exclude" filters makes the implementation simpler (but not simple at all). Adding "include" filters have two major implications:

1. With both include and exclude rules, order matters. We have global rules in the settings, we could have module-specific rules in the settings, and we have rules passed to the excludes parameter. All must apply and they must apply in the order they were created (an inclusion can only have effect if it intersects a previous exclusion). Current code stores rules in settings in separated sets by rule type, as duplicated rules don't change anything and the rules can be applied in any order.
2. I'd like the interactive API to be as easy as possible to use, as I think that most people will use this feature to save interactive sessions without ever touching settings. If dump_session() has a single excludes parameter that also accepts filter functions, the rule order is no problem. But there need to be an easy way to pass include rules. Maybe a simple include() function is sufficient. Something like:

import dill, re
from dill.session import include
dill.dump_session(excludes=[re.compile(r'prefix_.+'), include('prefix_keep')])

Internally, this argument would be converted to something like

[
    (EXCLUDE, re.compile(r'prefix_.+'),
    (INCLUDE, 'prefix_keep'),
]

and processed together with the rules from settings.

I agree that this PR and #424 could use the same logic/code.

[anivegesana]

I meant to strikethrough the comment I made (the part in ~.) I guess strikethrough doesn't work across lines.

The include might be useful in some situations, but it makes things more confusing and won't help most people probably and maybe is not worth the added complexity.

I didn’t mean change the interface. I meant change the names a little so it is like

main_globals = dill.settings.main_globals

main_globals.exclude_ids.add(var)              # stores id(var)
main_globals.exclude_regex.add(r"__.+__")      # stores re.compile(r"__.+__")
main_globals.exclude_types.add("Function")     # stores types.FunctionType

main_globals.exclude_regex.discard(r"__.+__")  # does what is expected
main_globals.exclude_types.clear()             # empties the types list

main_globals.filter # filter part of main_globals

main_globals could also be session_globals or just session. We can leave it as is or come up with a less verbose name. Conceptually, both the exclude and filter feel like they control the same setting, just at different granularities.

Also, maybe regexes instead of regex to keep the naming scheme consistent. All plural in this case.

[leogama]

@anivegesana, I got the attempted strikethrough. It seems to work on single paragraphs, you need to add the "~" to each one...

So, I reached a compromise: exclude and include rules, yes, but all the include rules run after the exclude rules. This simplifies a lot the implementation, besides making it more efficient, but still adds some flexibility. As a bonus, while I was finishing the main filtering procedure, I noticed that the include rules could be easily treated as an allowlist if there are no exclude rules.

It still needs tests, but the API is working like this:

import dill, re
from dill.session import ExcludeRules, EXCLUDE, INCLUDE, ipython_filter  # or just import *

dill.settings.session_exclude = ExcludeRules([
    (EXCLUDE, 'some_name'),                         # a rule can be as simple as a 2-tuple with the rule type and a single filter...
    (EXCLUDE, ['another_name', '__.+__', dict]),    # ...or it can have a sequence of filters, even of different types
    (INCLUDE, '__doc__'),                           # the names refer to "exclude", but also accept include rules, in any order
    (EXCLUDE, re.compile(r'ascii_\w+', re.ASCII)),  # regular expressions can be specified by strings or re.Pattern's
    (EXCLUDE, (lambda obj: obj.name.istitle())),    # filter functions receive a namedtuple with 'name' and 'value' attributes
])

# The convenience of adding types from the module types is now in a separated method:
rules = dill.settings.session_exclude
rules.exclude.add_type('Function')

# Clear in different levels (also update()):
rules.include.ids.clear()
rules.include.clear()
rules.clear()

# Filters passed to dump_session() are *added* to the settings' rules:
dill.dump_session(exclude=ipython_filter(), include=['spam', 'eggs'])  # arguments can be a single filter or a sequence of filters

# If settings.exclude_rules is empty and we only pass 'include' rules, just the matched variables will be saved:
dill.dump_session(include=['save', 'only', 'these'])

---

About naming... I liked session_globals, but just session is problematic if we need to add another option to session settings in the future (like byref). With this AttrDict class I'm proposing to use with settings, it also could be session.globals. It's hierarchical and I think I even like it:

dill.settings.session.globals.exclude.names

Of course, if you are using it more than once you can alias it with rules = dill.settings.session.globals and use rules.add(...), etc.

I'm not sure about the classes names by the way. ExcludeRules could just as well be FilterRules, and ExcludeFilters could be FilterSet(s)? And I have no preference for the regular expression set's name, it can be regex, regexes, regexps, regexs... (you find all these scattered through the Standard Library). You and @mmckerns may choose. 😮‍💨

[mmckerns]

I'll start a review shortly. Note that CI is failing for PyPy-3.7.

[leogama]

Yup... And the error message doesn't help. I'll try to install a more recent version of PyPy and debug it.

[mmckerns]

Note that the most recent version that Travis has is pypy3.7-7.3.5, while the latest version released (and tested on my dev boxes) is pypy3.7-7.3.9.

[leogama]

Note that the most recent version that Travis has is pypy3.7-7.3.5, while the latest version released (and tested on my dev boxes) is pypy3.7-7.3.9.

I got no the same error (sorry, I was building the wrong branch) with pypy 3.8.13 (7.3.9+dfsg-1~ppa1~ubuntu20.04)
Isn't this the version you have? I just realized that the pypy micro version has nothing to do with the python minor version...

[leogama]

will try to review this over the next 2-3 days :)

Opening the file diffs and the changelog side-by-side may help (that's how I wrote the changelog, following the order of changes in the files).

[leogama]

Results of trying to pickle and unpickle all Python StdLib modules on my machine:

python3.7

Testing pickling and unpickling of Standard Library modules...
F:::::::::::::::F:::::::::FF:::F:FFF:FF:::::::::::::::::::::F::::::::::::F::::F:
::::::F:F::::::::F::::::::F:::F::F::F:F:::::::::::::::::::::::F:FF::::::::FF::F:
FF::::::F:::::::::F:::::F:F:F:::::F:::::::F::::F:F:::::::FF:::
INFO:root:Success rate (dump_module, refonfail=False): 82.0% [182/222]
INFO:root:Success rate (load_module, refonfail=False): 93.4% [170/182]
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::F:::::::::::::::::::::::::::
INFO:root:Success rate (dump_module, refonfail=True): 99.5% [221/222]
INFO:root:Success rate (load_module, refonfail=True): 92.8% [205/221]
INFO:root:dump_module() FAILURES: unittest.mock
INFO:root:load_module() FAILURES: decimal, enum, http, inspect, ipaddress,
logging, plistlib, py_compile, re, signal, socket, ssl, string, sys, typing,
uuid

python3.11

Testing pickling and unpickling of Standard Library modules...
F::::::::::::::F::::::::F:F::::F::::FF:FF:::::::::::::::::::::F::::::::::F::::::
::F:::::::FF:F:F::::::F:::::::::::::::F:::F:::FF:::::::::::::::::F:::::::::::FF:
F:::::F:F::FFF::F::::::::F::F:::::F::::F::F::::F::::::::F:F:::::FF::::::::F:::F:
INFO:root:Success rate (dump_module, refonfail=False): 82.2% [198/241]
INFO:root:Success rate (load_module, refonfail=False): 91.9% [182/198]
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
INFO:root:Success rate (dump_module, refonfail=True): 100.0% [241/241]  <---
INFO:root:Success rate (load_module, refonfail=True): 92.1% [222/241]
INFO:root:load_module() FAILURES: ast, decimal, encodings, enum, http, inspect,
ipaddress, logging, plistlib, pstats, py_compile, re, signal, socket, ssl, sys,
typing, unittest.mock, uuid

[leogama]

Damn keyboard shortcuts!

[jhallard]

Any way we can get this merged to unblock the 0.3.6 release?

[Pierre-Sassoulas]

Any way we can help unblock this ? python 3.11 release is approaching fast now and we need a release of dill to be able to use python 3.11 in pylint and prospector.