61248

Python pickle not one-to-one: different pickles give same object

Can someone explain this?

pickle.loads(b'\x80\x03X\x01\x00\x00\x00.q\x00h\x00\x86q\x01.') == pickle.loads(b'\x80\x03X\x01\x00\x00\x00.q\x00X\x01\x00\x00\x00.q\x01\x86q\x02.') >>>True pickle.loads(b'\x80\x03X\x01\x00\x00\x00.q\x00h\x00\x86q\x01.') >>>('.', '.') pickle.loads(b'\x80\x03X\x01\x00\x00\x00.q\x00X\x01\x00\x00\x00.q\x01\x86q\x02.') >>>('.', '.')

There seems to be a long and short pickled version of tuples with the same element repeatedly.

Other examples:

pickle.loads(b'\x80\x03X\x01\x00\x00\x00#q\x00X\x01\x00\x00\x00#q\x01\x86q\x02.') >>>('#', '#') pickle.loads(b'\x80\x03X\x01\x00\x00\x00#q\x00h\x00\x86q\x01.') >>>('#', '#') pickle.loads(b'\x80\x03X\x01\x00\x00\x00$q\x00X\x01\x00\x00\x00$q\x01\x86q\x02.') >>>('$', '$') pickle.loads(b'\x80\x03X\x01\x00\x00\x00$q\x00h\x00\x86q\x01.') >>>('$', '$')

I'm trying to index items by their pickle but I'm not finding the items because their pickles seem to be changing.

I'm using Python 3.3.2 on Ubuntu.

Answer1:

Pickles aren't unique; the pickle format is actually a tiny little programming language, and different programs (pickles) can produce the same output (unpickled object). From the docs:

Since the pickle data format is actually a tiny stack-oriented programming language, and some freedom is taken in the encodings of certain objects, it is possible that the two modules [pickle and cPickle] produce different data streams for the same input objects. However it is guaranteed that they will always be able to read each other’s data streams.

There's even a pickletools.optimize function that will take a pickle and output a better pickle. You're going to need to redesign your program.

Recommend

  • Keras model pyspark error
  • Django mod_wsgi PicklingError while saving object
  • Is it possible to predict in sagemaker without using s3
  • pickling and unpickling user-defined class
  • Idris - map function on custom dependent data type fails
  • What's the difference between [String] vs. [(String)]?
  • in this page hover is not working i dont know why and also footer is not taking 100% width even thou
  • detecting end of http header with \\r\\n\\r\\n
  • sending email using “bcc” without “to” in java application
  • Random access image (picture) file
  • SEO friendly 301 redirect .htm to .aspx
  • Grouping by blank nodes
  • CUDA Debugging - VS on windows workstation, GPUs on Linux server?
  • What is ./.local/share/Trash (Unix) [closed]
  • Can a Collections.shuffle be considered equivalent to a series of Randoms?
  • pymongo replication secondary readreference not work
  • Is it possible to define rest argument in OCaml?
  • Building Qt project for C++11 standard
  • import scipy.sparse failed
  • During installation of Django, why do I keep getting ImportError: No module named django?
  • where do I find the xml.dom python package for the python-2.6.0-8.9.28 and I have a suse/x86_64 vers
  • SAVE attribute needed for Fortran variables when only the C_LOC address is returned to a C program?
  • Adjust width of select element according to selected option's width
  • C: Incompatible pointer type initializing
  • why xml file does not aligned properly after append the string in beginning and end of the file usin
  • Meteor helpers not available in Angular template
  • PHPUnit_Framework_TestCase class is not available. Fix… - Makegood , Eclipse
  • Deserializing XML into class C#
  • Javascript + PHP Encryption with pidCrypt
  • Weird JavaScript statement, what does it mean?
  • Rearranging Cells in UITableView Bug & Saving Changes
  • How to include full .NET prerequisite for Wix Burn installer
  • Proper way to use connect-multiparty with express.js?
  • Understanding cpu registers
  • Qt: Run a script BEFORE make
  • Django query for large number of relationships
  • How to get NHibernate ISession to cache entity not retrieved by primary key
  • costura.fody for a dll that references another dll
  • Conditional In-Line CSS for IE and Others?
  • jQuery Masonry / Isotope and fluid images: Momentary overlap on window resize