Add documentation for handling openpyxl - move troubleshooting into separate page
diff --git a/docs/index.rst b/docs/index.rst index c38273e..883f3bb 100644 --- a/docs/index.rst +++ b/docs/index.rst
@@ -13,6 +13,7 @@ intro usage + troubleshooting autopatch modules api
diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst new file mode 100644 index 0000000..d6024af --- /dev/null +++ b/docs/troubleshooting.rst
@@ -0,0 +1,177 @@ +Troubleshooting +=============== +This is a collection of problems with ``pyfakefs`` and possible solutions. +It will be expanded continuously based on issues and problems found by users. + +Modules not working with pyfakefs +--------------------------------- + +Modules may not work with ``pyfakefs`` for several reasons. ``pyfakefs`` +works by patching some file system related modules and functions, specifically: + +- most file system related functions in the ``os`` and ``os.path`` modules +- the ``pathlib`` module +- the build-in ``open`` function and ``io.open`` +- ``shutil.disk_usage`` + +Other file system related modules work with ``pyfakefs``, because they use +exclusively these patched functions, specifically ``shutil`` (except for +``disk_usage``), ``tempfile``, ``glob`` and ``zipfile``. + +A module may not work with ``pyfakefs`` because of one of the following +reasons: + +- It uses a file system related function of the mentioned modules that is + not or not correctly patched. Mostly these are functions that are seldom + used, but may be used in Python libraries (this has happened for example + with a changed implementation of ``shutil`` in Python 3.7). Generally, + these shall be handled in issues and we are happy to fix them. +- It uses file system related functions in a way that will not be patched + automatically. This is the case for functions that are executed while + reading a module. This case and a possibility to make them work is + documented above under ``modules_to_reload``. +- It uses OS specific file system functions not contained in the Python + libraries. These will not work out of the box, and we generally will not + support them in ``pyfakefs``. If these functions are used in isolated + functions or classes, they may be patched by using the ``modules_to_patch`` + parameter (see the example for file locks in Django above), or by using + ``unittest.patch`` if you don't need to simulate the functions. We + added some of these patches to ``pyfakefs``, so that they are applied + automatically (currently done for some ``pandas`` and ``Django`` + functionality). +- It uses C libraries to access the file system. There is no way no make + such a module work with ``pyfakefs``--if you want to use it, you + have to patch the whole module. In some cases, a library implemented in + Python with a similar interface already exists. An example is ``lxml``, + which can be substituted with ``ElementTree`` in most cases for testing. + +A list of Python modules that are known to not work correctly with +``pyfakefs`` will be collected here: + +`multiprocessing`_ (build-in) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This module has several issues (related to points 1 and 3 above). +Currently there are no plans to fix this, but this may change in case of +sufficient demand. + +`subprocess`_ (build-in) +~~~~~~~~~~~~~~~~~~~~~~~~ +This has very similar problems to ``multiprocessing`` and cannot be used with +``pyfakefs`` to start a process. ``subprocess`` can either be mocked, if +the process is not needed for the test, or patching can be paused to start +a process if needed, and resumed afterwards +(see `this issue <https://github.com/jmcgeheeiv/pyfakefs/issues/447>`__). + +Modules that rely on ``subprocess`` or ``multiprocessing`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This includes a number of modules that need to start other executables to +function correctly. Examples that have shown this problem include `GitPython`_ +and `plumbum`_. + +The `Pillow`_ image library +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This library partly works with ``pyfakefs``, but it is known to not work at +least if writing JPEG files +(see `this issue <https://github.com/jmcgeheeiv/pyfakefs/issues/529>`__) + +`pandas`_ - the Python data analysis library +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This uses its own internal file system access written in C, thus much of +``pandas`` will not work with ``pyfakefs`` out of the box. Having said that, +``pyfakefs`` patches ``pandas`` to use standard file-system access instead, +so that many of the ``read_xxx`` functions, including ``read_csv`` and +``read_excel``, as well as some writer functions, do work with the fake file +system. If you use only these functions, ``pyfakefs`` should work fine with +``pandas``. + +`xlrd`_ +~~~~~~~ +This libary is used by ``pandas`` to read Excel files in the `.xls` format, and +can also be used stand-alone. Similar to ``pandas``, it is by default patched +by ``pyfakefs`` to use normal file system functions that can be patched. + +`openpyxl`_ +~~~~~~~~~~~ +This is another library that reads and writes Excel files, and is also +used by ``pandas`` if installed. ``openpyxl`` uses ``lxml`` for some file-system +access if it is installed - in this case ``pyfakefs`` will not able to patch +it correctly (``lxml`` uses C functions for file system access). It will `not` +use ``lxml`` however, if the environment variable ``OPENPYXL_LXML` is set to +"False" (or anything other than "True"), so if you set this variable `before` +running the tests, it can work fine with ``pyfakefs``. + +Please write a new issue, if you are not sure if a module can be handled, or +how to do it. + +Pyfakefs behaves differently than the real filesystem +----------------------------------------------------- +There are at least the following kinds of deviations from the actual behavior: + +- unwanted deviations that we didn't notice--if you find any of these, please + write an issue and will try to fix it +- behavior that depends on different OS versions and editions--as mentioned + in :ref:`limitations`, ``pyfakefs`` uses the TravisCI systems as reference + system and will not replicate all system-specific behavior +- behavior that depends on low-level OS functionality that ``pyfakefs`` is not + able to emulate; examples are the ``fcntl.ioctl`` and ``fcntl.fcntl`` + functions that are patched to do nothing + +The test code tries to access files in the real filesystem +---------------------------------------------------------- +The loading of the actual Python code from the real filesystem does not use +the filesystem functions that ``pyfakefs`` patches, but in some cases it may +access other files in the packages. An example is loading timezone information +from configuration files. In these cases, you have to map the respective files +or directories from the real into the fake filesystem as described in +:ref:`real_fs_access`. + + +OS temporary directories +------------------------ +Tests relying on a completely empty file system on test start will fail. +As ``pyfakefs`` does not fake the ``tempfile`` module (as described above), +a temporary directory is required to ensure ``tempfile`` works correctly, +e.g., that ``tempfile.gettempdir()`` will return a valid value. This +means that any newly created fake file system will always have either a +directory named ``/tmp`` when running on Linux or Unix systems, +``/var/folders/<hash>/T`` when running on MacOs, or +``C:\Users\<user>\AppData\Local\Temp`` on Windows. + +User rights +----------- +If you run ``pyfakefs`` tests as root (this happens by default if run in a +docker container), ``pyfakefs`` also behaves as a root user, for example can +write to write-protected files. This may not be the expected behavior, and +can be changed. +``Pyfakefs`` has a rudimentary concept of user rights, which differentiates +between root user (with the user id 0) and any other user. By default, +``pyfakefs`` assumes the user id of the current user, but you can change +that using ``fake_filesystem.set_uid()`` in your setup. This allows to run +tests as non-root user in a root user environment and vice verse. +Another possibility to run tests as non-root user in a root user environment +is the convenience argument :ref:`allow_root_user`. + +.. _usage_with_mock_open: + +Pyfakefs and mock_open +---------------------- +If you patch ``open`` using ``mock_open`` before the initialization of +``pyfakefs``, it will not work properly, because the ``pyfakefs`` +initialization relies on ``open`` working correctly. +Generally, you should not need ``mock_open`` if using ``pyfakefs``, because you +always can create the files with the needed content using ``create_file``. +This is true for patching any filesystem functions - avoid patching them +while working with ``pyfakefs``. +If you still want to use ``mock_open``, make sure it is only used while +patching is in progress. For example, if you are using ``pytest`` with the +``mocker`` fixture used to patch ``open``, make sure that the ``fs`` fixture is +passed before the ``mocker`` fixture to ensure this. + +.. _`multiprocessing`: https://docs.python.org/3/library/multiprocessing.html +.. _`subprocess`: https://docs.python.org/3/library/subprocess.html +.. _`GitPython`: https://pypi.org/project/GitPython/ +.. _`plumbum`: https://pypi.org/project/plumbum/ +.. _`Pillow`: https://pypi.org/project/Pillow/ +.. _`pandas`: https://pypi.org/project/pandas/ +.. _`xlrd`: https://pypi.org/project/xlrd/ +.. _`openpyxl`: https://pypi.org/project/openpyxl/ \ No newline at end of file
diff --git a/docs/usage.rst b/docs/usage.rst index 297dace..a5f6a4d 100644 --- a/docs/usage.rst +++ b/docs/usage.rst
@@ -843,150 +843,9 @@ assert os.path.splitdrive(r"C:\foo\bar") == ("C:", r"\foo\bar") assert os.path.ismount("C:") -Troubleshooting ---------------- - -Modules not working with pyfakefs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Modules may not work with ``pyfakefs`` for several reasons. ``pyfakefs`` -works by patching some file system related modules and functions, specifically: - -- most file system related functions in the ``os`` and ``os.path`` modules -- the ``pathlib`` module -- the build-in ``open`` function and ``io.open`` -- ``shutil.disk_usage`` - -Other file system related modules work with ``pyfakefs``, because they use -exclusively these patched functions, specifically ``shutil`` (except for -``disk_usage``), ``tempfile``, ``glob`` and ``zipfile``. - -A module may not work with ``pyfakefs`` because of one of the following -reasons: - -- It uses a file system related function of the mentioned modules that is - not or not correctly patched. Mostly these are functions that are seldom - used, but may be used in Python libraries (this has happened for example - with a changed implementation of ``shutil`` in Python 3.7). Generally, - these shall be handled in issues and we are happy to fix them. -- It uses file system related functions in a way that will not be patched - automatically. This is the case for functions that are executed while - reading a module. This case and a possibility to make them work is - documented above under ``modules_to_reload``. -- It uses OS specific file system functions not contained in the Python - libraries. These will not work out of the box, and we generally will not - support them in ``pyfakefs``. If these functions are used in isolated - functions or classes, they may be patched by using the ``modules_to_patch`` - parameter (see the example for file locks in Django above), or by using - ``unittest.patch`` if you don't need to simulate the functions. We - added some of these patches to ``pyfakefs``, so that they are applied - automatically (currently done for some ``pandas`` and ``Django`` - functionality). -- It uses C libraries to access the file system. There is no way no make - such a module work with ``pyfakefs``--if you want to use it, you - have to patch the whole module. In some cases, a library implemented in - Python with a similar interface already exists. An example is ``lxml``, - which can be substituted with ``ElementTree`` in most cases for testing. - -A list of Python modules that are known to not work correctly with -``pyfakefs`` will be collected here: - -- `multiprocessing`_ has several issues (related to points 1 and 3 above). - Currently there are no plans to fix this, but this may change in case of - sufficient demand. -- `subprocess`_ has very similar problems and cannot be used with - ``pyfakefs`` to start a process. ``subprocess`` can either be mocked, if - the process is not needed for the test, or patching can be paused to start - a process if needed, and resumed afterwards - (see `this issue <https://github.com/jmcgeheeiv/pyfakefs/issues/447>`__). -- Modules that rely on ``subprocess`` or ``multiprocessing`` to work - correctly, e.g. need to start other executables. Examples that have shown - this problem include `GitPython`_ and `plumbum`_. -- the `Pillow`_ image library does not work with pyfakefs at least if writing - JPEG files (see `this issue <https://github.com/jmcgeheeiv/pyfakefs/issues/529>`__) -- `pandas`_ (the Python data analysis library) uses its own internal file - system access written in C. Thus much of ``pandas`` will not work with - ``pyfakefs``. Having said that, ``pyfakefs`` patches ``pandas`` so that many - of the ``read_xxx`` functions, including ``read_csv`` and ``read_excel``, - as well as some writer functions, do work with the fake file system. If - you use only these functions, ``pyfakefs`` will work with ``pandas``. - -If you are not sure if a module can be handled, or how to do it, you can -always write a new issue, of course! - -Pyfakefs behaves differently than the real filesystem -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -There are at least the following kinds of deviations from the actual behavior: - -- unwanted deviations that we didn't notice--if you find any of these, please - write an issue and will try to fix it -- behavior that depends on different OS versions and editions--as mentioned - in :ref:`limitations`, ``pyfakefs`` uses the TravisCI systems as reference - system and will not replicate all system-specific behavior -- behavior that depends on low-level OS functionality that ``pyfakefs`` is not - able to emulate; examples are the ``fcntl.ioctl`` and ``fcntl.fcntl`` - functions that are patched to do nothing - -The test code tries to access files in the real filesystem -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The loading of the actual Python code from the real filesystem does not use -the filesystem functions that ``pyfakefs`` patches, but in some cases it may -access other files in the packages. An example is loading timezone information -from configuration files. In these cases, you have to map the respective files -or directories from the real into the fake filesystem as described in -:ref:`real_fs_access`. - - -OS temporary directories -~~~~~~~~~~~~~~~~~~~~~~~~ -Tests relying on a completely empty file system on test start will fail. -As ``pyfakefs`` does not fake the ``tempfile`` module (as described above), -a temporary directory is required to ensure ``tempfile`` works correctly, -e.g., that ``tempfile.gettempdir()`` will return a valid value. This -means that any newly created fake file system will always have either a -directory named ``/tmp`` when running on Linux or Unix systems, -``/var/folders/<hash>/T`` when running on MacOs, or -``C:\Users\<user>\AppData\Local\Temp`` on Windows. - -User rights -~~~~~~~~~~~ - -If you run ``pyfakefs`` tests as root (this happens by default if run in a -docker container), ``pyfakefs`` also behaves as a root user, for example can -write to write-protected files. This may not be the expected behavior, and -can be changed. -``Pyfakefs`` has a rudimentary concept of user rights, which differentiates -between root user (with the user id 0) and any other user. By default, -``pyfakefs`` assumes the user id of the current user, but you can change -that using ``fake_filesystem.set_uid()`` in your setup. This allows to run -tests as non-root user in a root user environment and vice verse. -Another possibility to run tests as non-root user in a root user environment -is the convenience argument :ref:`allow_root_user`. - -.. _usage_with_mock_open: - -Pyfakefs and mock_open -~~~~~~~~~~~~~~~~~~~~~~ -If you patch ``open`` using ``mock_open`` before the initialization of -``pyfakefs``, it will not work properly, because the ``pyfakefs`` -initialization relies on ``open`` working correctly. -Generally, you should not need ``mock_open`` if using ``pyfakefs``, because you -always can create the files with the needed content using ``create_file``. -This is true for patching any filesystem functions - avoid patching them -while working with ``pyfakefs``. -If you still want to use ``mock_open``, make sure it is only used while -patching is in progress. For example, if you are using ``pytest`` with the -``mocker`` fixture used to patch ``open``, make sure that the ``fs`` fixture is -passed before the ``mocker`` fixture to ensure this. .. _`example.py`: https://github.com/jmcgeheeiv/pyfakefs/blob/master/pyfakefs/tests/example.py .. _`example_test.py`: https://github.com/jmcgeheeiv/pyfakefs/blob/master/pyfakefs/tests/example_test.py .. _`pytest`: https://doc.pytest.org .. _`nose`: https://docs.nose2.io/en/latest/ .. _`all Patcher arguments`: https://jmcgeheeiv.github.io/pyfakefs/master/modules.html#pyfakefs.fake_filesystem_unittest.Patcher -.. _`multiprocessing`: https://docs.python.org/3/library/multiprocessing.html -.. _`subprocess`: https://docs.python.org/3/library/subprocess.html -.. _`GitPython`: https://pypi.org/project/GitPython/ -.. _`plumbum`: https://pypi.org/project/plumbum/ -.. _`Pillow`: https://pypi.org/project/Pillow/ -.. _`pandas`: https://pypi.org/project/pandas/ \ No newline at end of file