Mandalika's scratchpad | [ Work blog @Oracle | My Music Compositions ] |
Old Posts: 09.04 10.04 11.04 12.04 01.05 02.05 03.05 04.05 05.05 06.05 07.05 08.05 09.05 10.05 11.05 12.05 01.06 02.06 03.06 04.06 05.06 06.06 07.06 08.06 09.06 10.06 11.06 12.06 01.07 02.07 03.07 04.07 05.07 06.07 08.07 09.07 10.07 11.07 12.07 01.08 02.08 03.08 04.08 05.08 06.08 07.08 08.08 09.08 10.08 11.08 12.08 01.09 02.09 03.09 04.09 05.09 06.09 07.09 08.09 09.09 10.09 11.09 12.09 01.10 02.10 03.10 04.10 05.10 06.10 07.10 08.10 09.10 10.10 11.10 12.10 01.11 02.11 03.11 04.11 05.11 07.11 08.11 09.11 10.11 11.11 12.11 01.12 02.12 03.12 04.12 05.12 06.12 07.12 08.12 09.12 10.12 11.12 12.12 01.13 02.13 03.13 04.13 05.13 06.13 07.13 08.13 09.13 10.13 11.13 12.13 01.14 02.14 03.14 04.14 05.14 06.14 07.14 09.14 10.14 11.14 12.14 01.15 02.15 03.15 04.15 06.15 09.15 12.15 01.16 03.16 04.16 05.16 06.16 07.16 08.16 09.16 12.16 01.17 02.17 03.17 04.17 06.17 07.17 08.17 09.17 10.17 12.17 01.18 02.18 03.18 04.18 05.18 06.18 07.18 08.18 09.18 11.18 12.18 01.19 02.19 05.19 06.19 08.19 10.19 11.19 05.20 10.20 11.20 12.20 09.21 11.21 12.22
Pickling or serialization is the process of converting a Python object to a byte stream; and Unpickling or deserialization is the process of re-creating the original in-memory Python object (not necessarily at the same memory address).
Python's pickle
module has the necessary methods to pickle and unpickle Python object hierarchies.
pickle
module:
is part of the Python standard library
Those byte streams can be:
sent over network between Python end-points that expect binary data
It is possible to pickle a variety of data types including built-in types — numeric types (integer, float, complex numbers), sequence types (lists, tuples), text sequence type (strings), binary sequence types (bytes, bytearray), set types (set), mapping types (dictionary), classes and built-in functions defined at the top level of a module.
Any attempt to pickle an unpicklable object may trigger PicklingError
exception.
Couple of gotchas:
When dealing with applications written in different programming languages; or even within Python ecosystem with different versions of Python involved, best to avoid Pickle as non-Python applications may not be able to reconstruct pickled Python objects.
Given the nature of binary data, pickled Python objects are not human-readable unless earlier protocols are used to serialize the data
eg.,
A trivial example demonstrating the calls to pickle (save data to a binary file) and unpickle (load data from the binary file) a Python data structure.
#!/usr/bin/python import pickle EMP = {} EMP['name'] = 'Gary' EMP['id'] = 12345 # pickle with open('employee.db', 'wb') as f: pickle.dump(EMP, f, pickle.HIGHEST_PROTOCOL) print ' Pickled data, EMP ', EMP # unpickle with open('employee.db', 'rb') as f: EMP_REC = pickle.load(f) print 'Unpickled data, EMP_REC ', EMP_REC, '\n' print '(EMP_REC is EMP)? : ', (EMP_REC is EMP) print '(EMP_REC == EMP)? : ', (EMP_REC == EMP)
Running the above code shows the following on stdout.
Pickled data, EMP {'name': 'Gary', 'id': 12345} Unpickled data, EMP_REC {'name': 'Gary', 'id': 12345} (EMP_REC is EMP)? : False (EMP_REC == EMP)? : True
dump()
method takes a serializable Python object as the first argument; and writes pickled representation of the object (serialized object) to a file. Second argument is the file handle that points to an open file. Rest of the arguments are optional.
Third argument, if specified, is the protocol to use. pickle.HIGHEST_PROTOCOL
tells pickle module to use the highest protocol version available. When working with a mix of old and new Python versions, using earlier versions of the protocol may ease or eliminate some of the potential compatibility issues.
As highlighted earlier, the protocol used by pickle module is Python specific so better watch out for cross-language compatibility issues while working in heterogeneous environments.
load()
method reads a pickled object representation (serialized data) from a file and returns the reconstructed object. The protocol version is detected automatically so it is not necessary to specify the protocol version during unpickling process.
If persistence is not a requirement, dumps()
and loads()
methods in pickle
module can be used to serialize (pickle) and deserialize (unpickle) a Python object in memory. This is useful when sending Python objects over network between compatible applications.
#!/usr/bin/python import pickle EMP = {} EMP['name'] = 'Gary' EMP['id'] = 12345 # in-memory pickling x = pickle.dumps(EMP, pickle.HIGHEST_PROTOCOL) print ' Pickled data, EMP ', EMP # in-memory unpickling EMP_REC = pickle.loads(x) print 'Unpickled data, EMP_REC ', EMP_REC, '\n' print '(EMP_REC is EMP)? : ', (EMP_REC is EMP) print '(EMP_REC == EMP)? : ', (EMP_REC == EMP)
Running the above code shows output identical to the output produced by the previous code listing - just that there is no file involved this time.
Pickled data, EMP {'name': 'Gary', 'id': 12345} Unpickled data, EMP_REC {'name': 'Gary', 'id': 12345} (EMP_REC is EMP)? : False (EMP_REC == EMP)? : True
As mentioned earlier, any attempt to pickle or unpickle objects that are not appropriate for serialization fail with an exception. Therefore, it is appropriate to safe guard the code with try-except blocks to handle unexpected failures.
Here is another trivial example demonstrating a pickling exception.
#!/usr/bin/python import sys import pickle try: f = open('dummy.txt', 'a') x = pickle.dumps(f) print 'Pickled file handle' except Exception, e: print 'Caught ', e.__class__.__name__, '-', str(e)
Running the above code throws a TypeError as shown below.
Caught TypeError - can't pickle file objects
(Credit: Various Sources including Python Documentation)
Labels: Python Pickle Unpickle Pickling Unpickling Object Persistence Serialization Tips
2004-2019 |