top: Replace upip with mip everywhere.

Updates all README.md and docs, and manifests to `require("mip")`.

Also extend and improve the documentation on freezing and packaging.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

16 files changed, 404 insertions, 878 deletions

diff --git a/docs/develop/gettingstarted.rst b/docs/develop/gettingstarted.rst
index 000b7d613..c2d3816d4 100644
--- a/docs/develop/gettingstarted.rst
+++ b/docs/develop/gettingstarted.rst
@@ -322,7 +322,8 @@ tests
 
 tools
 
-  Contains helper tools including the ``upip`` and the ``pyboard.py`` module.
+  Contains scripts used by the build and CI process, as well as user tools such
+  as ``pyboard.py`` and ``mpremote``.
 
 examples
 
diff --git a/docs/develop/optimizations.rst b/docs/develop/optimizations.rst
index d972cde66..7f2c8cbe7 100644
--- a/docs/develop/optimizations.rst
+++ b/docs/develop/optimizations.rst
@@ -25,6 +25,8 @@ into the firmware image as part of the main firmware compilation process, which
 the bytecode will be executed from ROM. This can lead to a significant memory saving, and
 reduce heap fragmentation.
 
+See :ref:`manifest` for more information.
+
 Variables
 ---------
 
diff --git a/docs/esp8266/tutorial/intro.rst b/docs/esp8266/tutorial/intro.rst
index ac46e68b5..75739bd6f 100644
--- a/docs/esp8266/tutorial/intro.rst
+++ b/docs/esp8266/tutorial/intro.rst
@@ -23,7 +23,7 @@ convertor to make the UART available to your PC.
 The minimum requirement for flash size is 1Mbyte. There is also a special
 build for boards with 512KB, but it is highly limited comparing to the
 normal build: there is no support for filesystem, and thus features which
-depend on it won't work (WebREPL, upip, etc.). As such, 512KB build will
+depend on it won't work (WebREPL, mip, etc.). As such, 512KB build will
 be more interesting for users who build from source and fine-tune parameters
 for their particular application.
 
diff --git a/docs/pyboard/tutorial/lcd160cr_skin.rst b/docs/pyboard/tutorial/lcd160cr_skin.rst
index fa0debcb1..a0fe88a2e 100644
--- a/docs/pyboard/tutorial/lcd160cr_skin.rst
+++ b/docs/pyboard/tutorial/lcd160cr_skin.rst
@@ -42,7 +42,7 @@ There is a test program which you can use to test the features of the display,
 and which also serves as a basis to start creating your own code that uses the
 LCD.  This test program is available on GitHub
 `here <https://github.com/micropython/micropython/blob/master/drivers/display/lcd160cr_test.py>`__.
-Copy it to the board over USB mass storage, or by using `mpremote`.
+Copy it to the board over USB mass storage, or by using :ref:`mpremote`.
 
 To run the test from the MicroPython prompt do::
 
diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst
index da951189e..4c66f7031 100644
--- a/docs/reference/glossary.rst
+++ b/docs/reference/glossary.rst
@@ -52,7 +52,7 @@ Glossary
     cross-compiler
         Also known as ``mpy-cross``. This tool runs on your PC and converts a
         :term:`.py file` containing MicroPython code into a :term:`.mpy file`
-        containing MicroPython bytecode. This means it loads faster (the board
+        containing MicroPython :term:`bytecode`. This means it loads faster (the board
         doesn't have to compile the code), and uses less space on flash (the
         bytecode is more space efficient).
 
@@ -128,7 +128,7 @@ Glossary
 
         Unlike the :term:`CPython` stdlib, micropython-lib modules are
         intended to be installed individually - either using manual copying or
-        using :term:`upip`.
+        using :term:`mip`.
 
     MicroPython port
         MicroPython supports different :term:`boards <board>`, RTOSes, and
@@ -151,16 +151,26 @@ Glossary
         machine-independent features. It can also function in a similar way to
         :term:`CPython`'s ``python`` executable.
 
+    mip
+        A package installer for MicroPython (mip - "mip installs packages"). It
+        installs MicroPython packages either from :term:`micropython-lib`,
+        GitHub, or arbitrary URLs.  mip can be used on-device on
+        network-capable boards, and internally by tools such
+        as :term:`mpremote`.
+
+    mpremote
+        A tool for interacting with a MicroPython device. See :ref:`mpremote`.
+
     .mpy file
         The output of the :term:`cross-compiler`. A compiled form of a
-        :term:`.py file` that contains MicroPython bytecode instead of Python
-        source code.
+        :term:`.py file` that contains MicroPython :term:`bytecode` instead of
+        Python source code.
 
     native
         Usually refers to "native code", i.e. machine code for the target
         microcontroller (such as ARM Thumb, Xtensa, x86/x64). The ``@native``
         decorator can be applied to a MicroPython function to generate native
-        code instead of bytecode for that function, which will likely be
+        code instead of :term:`bytecode` for that function, which will likely be
         faster but use more RAM.
 
     port
@@ -193,8 +203,10 @@ Glossary
         as a serial port over USB.
 
     upip
-        (Literally, "micro pip"). A package manager for MicroPython, inspired
+        A now-obsolete package manager for MicroPython, inspired
         by :term:`CPython`'s pip, but much smaller and with reduced
-        functionality.
-        upip runs both on the :term:`Unix port <MicroPython Unix port>` and on
-        :term:`baremetal` ports which offer filesystem and networking support.
+        functionality. See its replacement, :term:`mip`.
+
+    webrepl
+        A way of connecting to the REPL (and transferring files) on a device
+        over the internet from a browser. See https://micropython.org/webrepl
diff --git a/docs/reference/manifest.rst b/docs/reference/manifest.rst
index b756de47e..9bcafd583 100644
--- a/docs/reference/manifest.rst
+++ b/docs/reference/manifest.rst
@@ -1,35 +1,177 @@
+.. _manifest:
+
 MicroPython manifest files
 ==========================
 
-When building firmware for a device the following components are included in
-the compilation process:
+Summary
+-------
+
+MicroPython has a feature that allows Python code to be "frozen" into the
+firmware, as an alternative to loading code from the filesystem.
+
+This has the following benefits:
+
+- the code is pre-compiled to bytecode, avoiding the need for the Python
+  source to be compiled at load-time.
+- the bytecode can be executed directly from ROM (i.e. flash memory) rather than
+  being copied into RAM. Similarly any constant objects (strings, tuples, etc)
+  are loaded from ROM also. This can lead to significantly more memory being
+  available for your application.
+- on devices that do not have a filesystem, this is the only way to
+  load Python code.
+
+During development, freezing is generally not recommended as it will
+significantly slow down your development cycle, as each update will require
+re-flashing the entire firmware. However, it can still be useful to
+selectively freeze some rarely-changing dependencies (such as third-party
+libraries).
+
+The way to list the Python files to be be frozen into the firmware is via
+a "manifest", which is a Python file that will be interpreted by the build
+process. Typically you would write a manifest file as part of a board
+definition, but you can also write a stand-alone manifest file and use it with
+an existing board definition.
+
+Manifest files can define dependencies on libraries from :term:`micropython-lib`
+as well as Python files on the filesystem, and also on other manifest files.
+
+Writing manifest files
+----------------------
+
+A manifest file is a Python file containing a series of function calls. See the
+available functions defined below.
+
+Any paths used in manifest files can include the following variables. These all
+resolve to absolute paths.
+
+- ``$(MPY_DIR)`` -- path to the micropython repo.
+- ``$(MPY_LIB_DIR)`` -- path to the micropython-lib submodule. Prefer to use
+  ``require()``.
+- ``$(PORT_DIR)`` -- path to the current port (e.g. ``ports/stm32``)
+- ``$(BOARD_DIR)`` -- path to the current board
+  (e.g. ``ports/stm32/boards/PYBV11``)
+
+Custom manifest files should not live in the main MicroPython repository. You
+should keep them in version control with the rest of your project.
+
+Typically a manifest used for compiling firmware will need to include the port
+manifest, which might include frozen modules that are required for the board to
+function. If you just want to add additional modules to an existing board, then
+include the board manifest (which will in turn include the port manifest).
+
+Building with a custom manifest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Your manifest can be specified on the ``make`` command line with:
+
+.. code-block:: bash
+
+    $ make BOARD=MYBOARD FROZEN_MANIFEST=/path/to/my/project/manifest.py
+
+This applies to all ports, including CMake-based ones (e.g. esp32, rp2), as the
+Makefile wrapper that will pass this into the CMake build.
+
+Adding a manifest to a board definition
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have a custom board definition, you can make it include your custom
+manifest automatically. On make-based ports (most ports), in your
+``mpconfigboard.mk`` set the ``FROZEN_MANIFEST`` variable.
+
+.. code-block:: makefile
+
+    FROZEN_MANIFEST ?= $(BOARD_DIR)/manifest.py
+
+On CMake-based ports (e.g. esp32, rp2), instead use ``mpconfigboard.cmake``
+
+.. code-block:: cmake
+
+    set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py)
+
+High-level functions
+~~~~~~~~~~~~~~~~~~~~
+
+Note: The ``opt`` keyword argument can be set on the various functions, this controls
+the optimisation level used by the cross-compiler.
+See :func:`micropython.opt_level`.
+
+.. function:: package(package_path, files=None, base_path=".", opt=None)
+
+    This is equivalent to copying the "package_path" directory to the device
+    (except as frozen code).
+
+    In the simplest case, to freeze a package "foo" in the current directory:
+
+    .. code-block:: python3
+
+        package("foo")
+
+    will recursively include all .py files in foo, and will be frozen as
+    ``foo/**/*.py``.
+
+    If the package isn't in the same directory as the manifest file, use ``base_path``:
+
+    .. code-block:: python3
+
+        package("foo", base_path="path/to/libraries")
+
+    You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``.
+
+    To restrict to certain files in the package use ``files`` (note: paths
+    should be relative to the package): ``package("foo", files=["bar/baz.py"])``.
+
+.. function:: module(module_path, base_path=".", opt=None)
+
+    Include a single Python file as a module.
+
+    If the file is in the current directory:
+
+    .. code-block:: python3
+
+        module("foo.py")
+
+    Otherwise use base_path to locate the file:
+
+    .. code-block:: python3
+
+        module("foo.py", base_path="src/drivers")
+
+    You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``.
+
+.. function:: require(name, unix_ffi=False)
+
+    Require a package by name (and its dependencies) from :term:`micropython-lib`.
+
+    Optionally specify unix_ffi=True to use a module from the unix-ffi directory.
+
+.. function:: include(manifest_path)
 
-- the core MicroPython virtual machine and runtime
-- port-specific system code and drivers to interface with the
-  microcontroller/device that the firmware is targeting
-- standard built-in modules, like ``sys``
-- extended built-in modules, like ``json`` and ``machine``
-- extra modules written in C/C++
-- extra modules written in Python
+    Include another manifest.
+
+    Typically a manifest used for compiling firmware will need to include the
+    port manifest, which might include frozen modules that are required for
+    the board to function.
+
+    The *manifest* argument can be a string (filename) or an iterable of
+    strings.
 
-All the modules included in the firmware are available via ``import`` from
-Python code.  The extra modules written in Python that are included in a build
-(the last point above) are called *frozen modules*, and are specified by a
-``manifest.py`` file.  Changing this manifest requires rebuilding the firmware.
+    Relative paths are resolved with respect to the current manifest file.
 
-It's also possible to add additional modules to the filesystem of the device
-once it is up and running.  Adding and removing modules to/from the filesystem
-does not require rebuilding the firmware so is a simpler process than rebuilding
-firmware.  The benefit of using a manifest is that frozen modules are more
-efficient: they are faster to import and take up less RAM once imported.
+    If the path is to a directory, then it implicitly includes the
+    manifest.py file inside that directory.
 
-MicroPython manifest files are Python files and can contain arbitrary Python
-code.  There are also a set of commands (predefined functions) which are used
-to specify the Python source files to include.  These commands are described
-below.
+    You can use the variables above, such as ``$(PORT_DIR)`` in ``manifest_path``.
 
-Freezing source code
---------------------
+.. function:: metadata(description=None, version=None, license=None, author=None)
+
+    Define metadata for this manifest file. This is useful for manifests for
+    micropython-lib packages.
+
+Low-level functions
+~~~~~~~~~~~~~~~~~~~
+
+These functions are documented for completeness, but with the exception of
+``freeze_as_str`` all functionality can be accessed via the high-level functions.
 
 .. function:: freeze(path, script=None, opt=0)
 
@@ -42,9 +184,7 @@ Freezing source code
     module will start after *path*, i.e. *path* is excluded from the module
     name.
 
-    If *path* is relative, it is resolved to the current ``manifest.py``.  Use
-    ``$(MPY_DIR)``, ``$(MPY_LIB_DIR)``, ``$(PORT_DIR)``, ``$(BOARD_DIR)`` if you
-    need to access specific paths.
+    If *path* is relative, it is resolved to the current ``manifest.py``.
 
     If *script* is None, all files in *path* will be frozen.
 
@@ -75,71 +215,48 @@ Freezing source code
     Freeze the input, which must be ``.mpy`` files that are frozen directly.
     See ``freeze()`` for further details on the arguments.
 
-
-Including other manifest files
-------------------------------
-
-.. function:: include(manifest, **kwargs)
-
-    Include another manifest.
-
-    The *manifest* argument can be a string (filename) or an iterable of
-    strings.
-
-    Relative paths are resolved with respect to the current manifest file.
-
-    Optional *kwargs* can be provided which will be available to the included
-    script via the *options* variable.
-
-    For example:
-
-    .. code-block:: python3
-
-        include("path.py", extra_features=True)
-
-    then in path.py:
-
-    .. code-block:: python3
-
-            options.defaults(standard_features=True)
-            # freeze minimal modules.
-            if options.standard_features:
-                # freeze standard modules.
-            if options.extra_features:
-                # freeze extra modules.
-
-
 Examples
 --------
 
-To freeze a single file which is available as ``import mydriver``, use:
+To freeze a single file from the current directory which will be available as
+``import mydriver``, use:
 
 .. code-block:: python3
 
-    freeze(".", "mydriver.py")
+    module("mydriver.py")
 
-To freeze a set of files which are available as ``import test1`` and
-``import test2``, and which are compiled with optimisation level 3, use:
+To freeze a directory of files in a subdirectory "mydriver" of the current
+directory which will be available as ``import mydriver``, use:
 
 .. code-block:: python3
 
-    freeze("/path/to/tests", ("test1.py", "test2.py"), opt=3)
+    package("mydriver")
 
-To freeze a module which can be imported as ``import mymodule``, use:
+To freeze the "hmac" library from :term:`micropython-lib`, use:
 
 .. code-block:: python3
 
-    freeze(
-        "../relative/path",
-        (
-            "mymodule/__init__.py",
-            "mymodule/core.py",
-            "mymodule/extra.py",
-        ),
-    )
+    require("hmac")
 
-To include a manifest from the MicroPython repository, use:
+A more complete example of a custom ``manifest.py`` file for the ``PYBD_SF2``
+board is:
 
 .. code-block:: python3
 
-    include("$(MPY_DIR)/extmod/uasyncio/manifest.py")
+    # Include the board's default manifest.
+    include("$(BOARD_DIR)/manifest.py")
+    # Add a custom driver
+    module("mydriver.py")
+    # Add aiorepl from micropython-lib
+    require("aiorepl")
+
+Then the board can be compiled with
+
+.. code-block:: bash
+
+    $ cd ports/stm32
+    $ make BOARD=PYBD_SF2 FROZEN_MANIFEST=~/src/myproject/manifest.py
+
+Note that most boards do not have their own ``manifest.py``, rather they use the
+port one directly, in which case your manifest should just
+``include("$(PORT_DIR)/boards/manifest.py")`` instead.
diff --git a/docs/reference/packages.rst b/docs/reference/packages.rst
index eb44992ed..0c049d1fb 100644
--- a/docs/reference/packages.rst
+++ b/docs/reference/packages.rst
@@ -1,314 +1,153 @@
 .. _packages:
 
-Distribution packages, package management, and deploying applications
-=====================================================================
-
-Just as the "big" Python, MicroPython supports creation of "third party"
-packages, distributing them, and easily installing them in each user's
-environment. This chapter discusses how these actions are achieved.
-Some familiarity with Python packaging is recommended.
-
-Overview
---------
-
-Steps below represent a high-level workflow when creating and consuming
-packages:
-
-1. Python modules and packages are turned into distribution package
-   archives, and published at the Python Package Index (PyPI).
-2. :term:`upip` package manager can be used to install a distribution package
-   on a :term:`MicroPython port` with networking capabilities (for example,
-   on the Unix port).
-3. For ports without networking capabilities, an "installation image"
-   can be prepared on the Unix port, and transferred to a device by
-   suitable means.
-4. For low-memory ports, the installation image can be frozen as the
-   bytecode into MicroPython executable, thus minimizing the memory
-   storage overheads.
-
-The sections below describe this process in details.
-
-Distribution packages
----------------------
-
-Python modules and packages can be packaged into archives suitable for
-transfer between systems, storing at the well-known location (PyPI),
-and downloading on demand for deployment. These archives are known as
-*distribution packages* (to differentiate them from Python packages
-(means to organize Python source code)).
-
-The MicroPython distribution package format is a well-known tar.gz
-format, with some adaptations however. The Gzip compressor, used as
-an external wrapper for TAR archives, by default uses 32KB dictionary
-size, which means that to uncompress a compressed stream, 32KB of
-contiguous memory needs to be allocated. This requirement may be not
-satisfiable on low-memory devices, which may have total memory available
-less than that amount, and even if not, a contiguous block like that
-may be hard to allocate due to memory fragmentation. To accommodate
-these constraints, MicroPython distribution packages use Gzip compression
-with the dictionary size of 4K, which should be a suitable compromise
-with still achieving some compression while being able to uncompressed
-even by the smallest devices.
-
-Besides the small compression dictionary size, MicroPython distribution
-packages also have other optimizations, like removing any files from
-the archive which aren't used by the installation process. In particular,
-:term:`upip` package manager doesn't execute ``setup.py`` during installation
-(see below), and thus that file is not included in the archive.
-
-At the same time, these optimizations make MicroPython distribution
-packages not compatible with :term:`CPython`'s package manager, ``pip``.
-This isn't considered a big problem, because:
-
-1. Packages can be installed with :term:`upip`, and then can be used with
-   CPython (if they are compatible with it).
-2. In the other direction, majority of CPython packages would be
-   incompatible with MicroPython by various reasons, first of all,
-   the reliance on features not implemented by MicroPython.
-
-Summing up, the MicroPython distribution package archives are highly
-optimized for MicroPython's target environments, which are highly
-resource constrained devices.
-
-
-``upip`` package manager
-------------------------
-
-MicroPython distribution packages are intended to be installed using
-the :term:`upip` package manager. :term:`upip` is a Python application which is
-usually distributed (as frozen bytecode) with network-enabled
-:term:`MicroPython ports <MicroPython port>`. At the very least,
-:term:`upip` is available in the :term:`MicroPython Unix port`.
-
-On any :term:`MicroPython port` providing :term:`upip`, it can be accessed as
-following::
-
-    import upip
-    upip.help()
-    upip.install(package_or_package_list, [path])
-
-Where *package_or_package_list* is the name of a distribution
-package to install, or a list of such names to install multiple
-packages. Optional *path* parameter specifies filesystem
-location to install under and defaults to the standard library
-location (see below).
-
-An example of installing a specific package and then using it::
-
-    >>> import upip
-    >>> upip.install("micropython-pystone_lowmem")
-    [...]
-    >>> import pystone_lowmem
-    >>> pystone_lowmem.main()
-
-Note that the name of Python package and the name of distribution
-package for it in general don't have to match, and oftentimes they
-don't. This is because PyPI provides a central package repository
-for all different Python implementations and versions, and thus
-distribution package names may need to be namespaced for a particular
-implementation. For example, all packages from `micropython-lib`
-follow this naming convention: for a Python module or package named
-``foo``, the distribution package name is ``micropython-foo``.
-
-For the ports which run MicroPython executable from the OS command
-prompts (like the Unix port), `upip` can be (and indeed, usually is)
-run from the command line instead of MicroPython's own REPL. The
-commands which corresponds to the example above are::
-
-    micropython -m upip -h
-    micropython -m upip install [-p <path>] <packages>...
-    micropython -m upip install micropython-pystone_lowmem
-
-[TODO: Describe installation path.]
-
-
-Cross-installing packages
--------------------------
-
-For :term:`MicroPython ports <MicroPython port>` without native networking
-capabilities, the recommend process is "cross-installing" them into a
-"directory image" using the :term:`MicroPython Unix port`, and then
-transferring this image to a device by suitable means.
-
-Installing to a directory image involves using ``-p`` switch to :term:`upip`::
-
-    micropython -m upip install -p install_dir micropython-pystone_lowmem
-
-After this command, the package content (and contents of every dependency
-packages) will be available in the ``install_dir/`` subdirectory. You
-would need to transfer contents of this directory (without the
-``install_dir/`` prefix) to the device, at the suitable location, where
-it can be found by the Python ``import`` statement (see discussion of
-the :term:`upip` installation path above).
-
-
-Cross-installing packages with freezing
----------------------------------------
-
-For the low-memory :term:`MicroPython ports <MicroPython port>`, the process
-described in the previous section does not provide the most efficient
-resource usage,because the packages are installed in the source form,
-so need to be compiled to the bytecome on each import. This compilation
-requires RAM, and the resulting bytecode is also stored in RAM, reducing
-its amount available for storing application data. Moreover, the process
-above requires presence of the filesystem on a device, and the most
-resource-constrained devices may not even have it.
-
-The bytecode freezing is a process which resolves all the issues
-mentioned above:
-
-* The source code is pre-compiled into bytecode and store as such.
-* The bytecode is stored in ROM, not RAM.
-* Filesystem is not required for frozen packages.
-
-Using frozen bytecode requires building the executable (firmware)
-for a given :term:`MicroPython port` from the C source code. Consequently,
-the process is:
-
-1. Follow the instructions for a particular port on setting up a
-   toolchain and building the port. For example, for ESP8266 port,
-   study instructions in ``ports/esp8266/README.md`` and follow them.
-   Make sure you can build the port and deploy the resulting
-   executable/firmware successfully before proceeding to the next steps.
-2. Build :term:`MicroPython Unix port` and make sure it is in your PATH and
-   you can execute ``micropython``.
-3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266).
-4. Run ``make clean-frozen``. This step cleans up any previous
-   modules which were installed for freezing (consequently, you need
-   to skip this step to add additional modules, instead of starting
-   from scratch).
-5. Run ``micropython -m upip install -p modules <packages>...`` to
-   install packages you want to freeze.
-6. Run ``make clean``.
-7. Run ``make``.
-
-After this, you should have the executable/firmware with modules as
-the bytecode inside, which you can deploy the usual way.
-
-Few notes:
-
-1. Step 5 in the sequence above assumes that the distribution package
-   is available from PyPI. If that is not the case, you would need
-   to copy Python source files manually to ``modules/`` subdirectory
-   of the port directory. (Note that upip does not support
-   installing from e.g. version control repositories).
-2. The firmware for baremetal devices usually has size restrictions,
-   so adding too many frozen modules may overflow it. Usually, you
-   would get a linking error if this happens. However, in some cases,
-   an image may be produced, which is not runnable on a device. Such
-   cases are in general bugs, and should be reported and further
-   investigated. If you face such a situation, as an initial step,
-   you may want to decrease the amount of frozen modules included.
-
-
-Creating distribution packages
-------------------------------
-
-Distribution packages for MicroPython are created in the same manner
-as for CPython or any other Python implementation, see references at
-the end of chapter. Setuptools (instead of distutils) should be used,
-because distutils do not support dependencies and other features. "Source
-distribution" (``sdist``) format is used for packaging. The post-processing
-discussed above, (and pre-processing discussed in the following section)
-is achieved by using custom ``sdist`` command for setuptools. Thus, packaging
-steps remain the same as for the standard setuptools, the user just
-needs to override ``sdist`` command implementation by passing the
-appropriate argument to ``setup()`` call::
-
-    from setuptools import setup
-    import sdist_upip
-
-    setup(
-        ...,
-        cmdclass={'sdist': sdist_upip.sdist}
-    )
-
-The sdist_upip.py module as referenced above can be found in
-`micropython-lib`:
-https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py
-
-
-Application resources
----------------------
-
-A complete application, besides the source code, oftentimes also consists
-of data files, e.g. web page templates, game images, etc. It's clear how
-to deal with those when application is installed manually - you just put
-those data files in the filesystem at some location and use the normal
-file access functions.
-
-The situation is different when deploying applications from packages - this
-is more advanced, streamlined and flexible way, but also requires more
-advanced approach to accessing data files. This approach is treating
-the data files as "resources", and abstracting away access to them.
-
-Python supports resource access using its "setuptools" library, using
-``pkg_resources`` module. MicroPython, following its usual approach,
-implements subset of the functionality of that module, specifically
-``pkg_resources.resource_stream(package, resource)`` function.
-The idea is that an application calls this function, passing a
-resource identifier, which is a relative path to data file within
-the specified package (usually top-level application package). It
-returns a stream object which can be used to access resource contents.
-Thus, the ``resource_stream()`` emulates interface of the standard
-`open()` function.
-
-Implementation-wise, ``resource_stream()`` uses file operations
-underlyingly, if distribution package is install in the filesystem.
-However, it also supports functioning without the underlying filesystem,
-e.g. if the package is frozen as the bytecode. This however requires
-an extra intermediate step when packaging application - creation of
-"Python resource module".
-
-The idea of this module is to convert binary data to a Python bytes
-object, and put it into the dictionary, indexed by the resource name.
-This conversion is done automatically using overridden ``sdist`` command
-described in the previous section.
-
-Let's trace the complete process using the following example. Suppose
-your application has the following structure::
-
-    my_app/
-        __main__.py
-        utils.py
-        data/
-            page.html
-            image.png
-
-``__main__.py`` and ``utils.py`` should access resources using the
-following calls::
-
-    import pkg_resources
-
-    pkg_resources.resource_stream(__name__, "data/page.html")
-    pkg_resources.resource_stream(__name__, "data/image.png")
-
-You can develop and debug using the :term:`MicroPython Unix port` as usual.
-When time comes to make a distribution package out of it, just use
-overridden "sdist" command from sdist_upip.py module as described in
-the previous section.
-
-This will create a Python resource module named ``R.py``, based on the
-files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py``
-file will be considered a resource and added to ``R.py``) - before
-proceeding with the normal packaging steps.
-
-Prepared like this, your application will work both when deployed to
-filesystem and as frozen bytecode.
-
-If you would like to debug ``R.py`` creation, you can run::
-
-    python3 setup.py sdist --manifest-only
-
-Alternatively, you can use tools/mpy_bin2res.py script from the
-MicroPython distribution, in which can you will need to pass paths
-to all resource files::
-
-    mpy_bin2res.py data/page.html data/image.png
-
-References
-----------
-
-* Python Packaging User Guide: https://packaging.python.org/
-* Setuptools documentation: https://setuptools.readthedocs.io/
-* Distutils documentation: https://docs.python.org/3/library/distutils.html
+Package management
+==================
+
+Installing packages with ``mip``
+--------------------------------
+
+Network-capable boards include the ``mip`` module, which can install packages
+from :term:`micropython-lib` and from third-party sites (including GitHub).
+
+``mip`` ("mip installs packages") is similar in concept to Python's ``pip`` tool,
+however it does not use the PyPI index, rather it uses :term:`micropython-lib`
+as its index by default. ``mip`` will automatically fetch compiled
+:term:`.mpy file` when downloading from micropython-lib.
+
+The most common way to use ``mip`` is from the REPL::
+
+    >>> import mip
+    >>> mip.install("pkgname")  # Installs the latest version of "pkgname" (and dependencies)
+    >>> mip.install("pkgname", version="x.y")  # Installs version x.y of "pkgname"
+    >>> mip.install("pkgname", mpy=False)  # Installs the source version (i.e. .py rather than .mpy files)
+
+``mip`` will detect an appropriate location on the filesystem by searching
+``sys.path`` for the first entry ending in ``/lib``. You can override the
+destination using ``target``, but note that this path must be in ``sys.path`` to be
+able to subsequently import it.::
+
+    >>> mip.install("pkgname", target="third-party")
+    >>> sys.path.append("third-party")
+
+As well as downloading packages from the micropython-lib index, ``mip`` can also
+install third-party libraries. The simplest way is to download a file directly::
+
+    >>> mip.install("http://example.com/x/y/foo.py")
+    >>> mip.install("http://example.com/x/y/foo.mpy")
+
+When installing a file directly, the ``target`` argument is still supported to set
+the destination path, but ``mpy`` and ``version`` are ignored.
+
+The URL can also start with ``github:`` as a simple way of pointing to content
+hosted on GitHub::
+
+    >>> mip.install("github:org/repo/path/foo.py")  # Uses default branch
+    >>> mip.install("github:org/repo/path/foo.py", version="branch-or-tag")  # Optionally specify the branch or tag
+
+More sophisticated packages (i.e. with more than one file, or with dependencies)
+can be downloaded by specifying the path to their ``package.json``.
+
+    >>> mip.install("http://example.com/x/package.json")
+    >>> mip.install("github:org/user/path/package.json")
+
+If no json file is specified, then "package.json" is implicitly added::
+
+    >>> mip.install("http://example.com/x/")
+    >>> mip.install("github:org/repo")
+    >>> mip.install("github:org/repo", version="branch-or-tag")
+
+
+Using ``mip`` on the Unix port
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On the Unix port, ``mip`` can be used at the REPL as above, and also by using ``-m``::
+
+    $ ./micropython -m mip install pkgname-or-url
+    $ ./micropython -m mip install pkgname-or-url@version
+
+The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
+
+    $ ./micropython -m mip install --target=third-party pkgname
+    $ ./micropython -m mip install --no-mpy pkgname
+    $ ./micropython -m mip install --index https://host/pi pkgname
+
+Installing packages with ``mpremote``
+-------------------------------------
+
+The :term:`mpremote` tool also includes the same functionality as ``mip`` and
+can be used from a host PC to install packages to a locally connected device
+(e.g. via USB or UART)::
+
+    $ mpremote install pkgname
+    $ mpremote install pkgname@x.y
+    $ mpremote install http://example.com/x/y/foo.py
+    $ mpremote install github:org/repo
+    $ mpremote install github:org/repo@branch-or-tag
+
+The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
+
+    $ mpremote install --target=/flash/third-party pkgname
+    $ mpremote install --no-mpy pkgname
+    $ mpremote install --index https://host/pi pkgname
+
+Installing packages manually
+----------------------------
+
+Packages can also be installed (in either .py or .mpy form) by manually copying
+the files to the device. Depending on the board this might be via USB Mass Storage,
+the :term:`mpremote` tool (e.g. ``mpremote fs cp path/to/package.py :package.py``),
+:term:`webrepl`, etc.
+
+Writing & publishing packages
+-----------------------------
+
+Publishing to :term:`micropython-lib` is the easiest way to make your package
+broadly accessible to MicroPython users, and automatically available via
+``mip`` and ``mpremote`` and compiled to bytecode. See
+https://github.com/micropython/micropython-lib for more information.
+
+To write a "self-hosted" package that can be downloaded by ``mip`` or
+``mpremote``, you need a static webserver (or GitHub) to host either a
+single .py file, or a package.json file alongside your .py files.
+
+A typical package.json for an example ``mlx90640`` library looks like::
+
+    {
+      "urls": [
+        ["mlx90640/__init__.py", "github:org/micropython-mlx90640/mlx90640/__init__.py"],
+        ["mlx90640/utils.py", "github:org/micropython-mlx90640/mlx90640/utils.py"]
+      ],
+      "deps": [
+        ["collections-defaultdict", "latest"],
+        ["os-path", "latest"]
+      ],
+      "version": "0.2"
+    }
+
+This includes two files, hosted at a GitHub repo named
+``org/micropython-mlx90640``, which install into the ``mlx90640`` directory on
+the device. It depends on ``collections-defaultdict`` and ``os-path`` which will
+be installed automatically.
+
+Freezing packages
+-----------------
+
+When a Python module or package is imported from the device filesystem, it is
+compiled into :term:`bytecode` in RAM, ready to be executed by the VM. For
+a :term:`.mpy file`, this conversion has been done already, but the bytecode
+still ends up in RAM.
+
+For low-memory devices, or for large applications, it can be advantageous to
+instead run the bytecode from ROM (i.e. flash memory). This can be done
+by "freezing" the bytecode into the MicroPython firmware, which is then flashed
+to the device. The runtime performance is the same (although importing is
+faster), but it can free up significant amounts of RAM for your program to
+use.
+
+The downside of this approach is that it's much slower to develop, because you
+have to flash the firmware each time, but it can be still useful to freeze
+dependencies that don't change often.
+
+Freezing is done by writing a manifest file and using it in the build, often as
+part of a custom board definition. See the :ref:`manifest` guide for more
+information.
diff --git a/examples/hwapi/README.md b/examples/hwapi/README.md
index 1992eb660..df16b4c86 100644
--- a/examples/hwapi/README.md
+++ b/examples/hwapi/README.md
@@ -116,7 +116,7 @@ For example, one may invent a "configuration manager" helper module which will
 try to detect current board (among well-known ones), and load appropriate
 `hwconfig_*.py` - this assumes that a user would lazily deploy them all
 (or that application will be automatically installed, e.g. using MicroPython's
-`upip` package manager). The key point in this case remains the same as
+`mip` package manager). The key point in this case remains the same as
 elaborated above - always assume there can, and will be a custom configuration,
 and it should be well supported. So, any automatic detection should be
 overridable by a user, and instructions how to do so are among the most
diff --git a/ports/esp32/boards/manifest.py b/ports/esp32/boards/manifest.py
index fcc48d721..3f6c8cfde 100644
--- a/ports/esp32/boards/manifest.py
+++ b/ports/esp32/boards/manifest.py
@@ -1,11 +1,10 @@
 freeze("$(PORT_DIR)/modules")
-module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
-module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
 include("$(MPY_DIR)/extmod/uasyncio")
 
 # Require some micropython-lib modules.
 require("dht")
 require("ds18x20")
+require("mip")
 require("neopixel")
 require("ntptime")
 require("onewire")
diff --git a/ports/esp8266/README.md b/ports/esp8266/README.md
index dd50fc1af..b54d8958d 100644
--- a/ports/esp8266/README.md
+++ b/ports/esp8266/README.md
@@ -200,20 +200,22 @@ Python prompt over WiFi, connecting through a browser.
 - GitHub repository https://github.com/micropython/webrepl.
   Please follow the instructions there.
 
-__upip__
+__mip__
 
-The ESP8266 port comes with builtin `upip` package manager, which can
-be used to install additional modules (see the main README for more
-information):
+The ESP8266 port comes with the built-in `mip` package manager, which can
+be used to install additional modules:
 
 ```
->>> import upip
->>> upip.install("micropython-pystone_lowmem")
+>>> import mip
+>>> mip.install("hmac")
 [...]
->>> import pystone_lowmem
->>> pystone_lowmem.main()
+>>> import hmac
+>>> hmac.new(b"1234567890", msg="hello world").hexdigest()
 ```
 
+See [Package management](https://docs.micropython.org/en/latest/reference/packages.html) for more
+information about `mip`.
+
 Downloading and installing packages may requite a lot of free memory,
 if you get an error, retry immediately after the hard reset.
 
diff --git a/ports/esp8266/boards/manifest.py b/ports/esp8266/boards/manifest.py
index e7defd0bb..53975f6a6 100644
--- a/ports/esp8266/boards/manifest.py
+++ b/ports/esp8266/boards/manifest.py
@@ -1,9 +1,8 @@
 freeze("$(PORT_DIR)/modules")
-module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
-module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
-require("ntptime")
 require("dht")
-require("onewire")
 require("ds18x20")
+require("mip")
 require("neopixel")
+require("ntptime")
+require("onewire")
 require("webrepl")
diff --git a/ports/rp2/boards/PICO_W/manifest.py b/ports/rp2/boards/PICO_W/manifest.py
index 4d9eb87f2..8a74006c6 100644
--- a/ports/rp2/boards/PICO_W/manifest.py
+++ b/ports/rp2/boards/PICO_W/manifest.py
@@ -1,7 +1,5 @@
 include("../manifest.py")
 
-module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
-module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
-
+require("mip")
 require("ntptime")
 require("urequests")
diff --git a/ports/unix/README.md b/ports/unix/README.md
index efc68b245..a3a0dba75 100644
--- a/ports/unix/README.md
+++ b/ports/unix/README.md
@@ -24,21 +24,26 @@ Use `CTRL-D` (i.e. EOF) to exit the shell.
 Learn about command-line options (in particular, how to increase heap size
 which may be needed for larger applications):
 
-    $ ./micropython -h
+    $ ./build-standard/micropython -h
 
 To run the complete testsuite, use:
 
     $ make test
 
-The Unix port comes with a builtin package manager called upip, e.g.:
+The Unix port comes with a built-in package manager called `mip`, e.g.:
 
-    $ ./micropython -m upip install micropython-pystone
-    $ ./micropython -m pystone
+    $ ./build-standard/micropython -m mip install hmac
 
-Browse available modules on
-[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython).
-Standard library modules come from the
-[micropython-lib](https://github.com/micropython/micropython-lib) project.
+or
+
+    $ ./build-standard/micropython
+    >>> import mip
+    >>> mip.install("hmac")
+
+Browse available modules at [micropython-lib]
+(https://github.com/micropython/micropython-lib). See
+[Package management](https://docs.micropython.org/en/latest/reference/packages.html)
+for more information about `mip`.
 
 External dependencies
 ---------------------
@@ -65,6 +70,5 @@ or not). If you intend to build MicroPython with additional options
 (like cross-compiling), the same set of options should be passed to `make
 deplibs`. To actually enable/disable use of dependencies, edit the
 `ports/unix/mpconfigport.mk` file, which has inline descriptions of the
-options. For example, to build SSL module (required for the `upip` tool
-described above, and so enabled by default), `MICROPY_PY_USSL` should be set
-to 1.
+options. For example, to build the SSL module, `MICROPY_PY_USSL` should be
+set to 1.
diff --git a/ports/unix/variants/manifest.py b/ports/unix/variants/manifest.py
index bf7ce992a..e7fe747cb 100644
--- a/ports/unix/variants/manifest.py
+++ b/ports/unix/variants/manifest.py
@@ -1,2 +1 @@
-module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
-module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
+require("mip")
diff --git a/tools/upip.py b/tools/upip.py
deleted file mode 100644
index 2932cca50..000000000
--- a/tools/upip.py
+++ /dev/null
@@ -1,351 +0,0 @@
-#
-# upip - Package manager for MicroPython
-#
-# Copyright (c) 2015-2018 Paul Sokolovsky
-#
-# Licensed under the MIT license.
-#
-import sys
-import gc
-import uos as os
-import uerrno as errno
-import ujson as json
-import uzlib
-import upip_utarfile as tarfile
-
-gc.collect()
-
-
-debug = False
-index_urls = ["https://micropython.org/pi", "https://pypi.org/pypi"]
-install_path = None
-cleanup_files = []
-gzdict_sz = 16 + 15
-
-file_buf = bytearray(512)
-
-
-class NotFoundError(Exception):
-    pass
-
-
-def op_split(path):
-    if path == "":
-        return ("", "")
-    r = path.rsplit("/", 1)
-    if len(r) == 1:
-        return ("", path)
-    head = r[0]
-    if not head:
-        head = "/"
-    return (head, r[1])
-
-
-# Expects *file* name
-def _makedirs(name, mode=0o777):
-    ret = False
-    s = ""
-    comps = name.rstrip("/").split("/")[:-1]
-    if comps[0] == "":
-        s = "/"
-    for c in comps:
-        if s and s[-1] != "/":
-            s += "/"
-        s += c
-        try:
-            os.mkdir(s)
-            ret = True
-        except OSError as e:
-            if e.errno != errno.EEXIST and e.errno != errno.EISDIR:
-                raise e
-            ret = False
-    return ret
-
-
-def save_file(fname, subf):
-    global file_buf
-    with open(fname, "wb") as outf:
-        while True:
-            sz = subf.readinto(file_buf)
-            if not sz:
-                break
-            outf.write(file_buf, sz)
-
-
-def install_tar(f, prefix):
-    meta = {}
-    for info in f:
-        # print(info)
-        fname = info.name
-        try:
-            fname = fname[fname.index("/") + 1 :]
-        except ValueError:
-            fname = ""
-
-        save = True
-        for p in ("setup.", "PKG-INFO", "README"):
-            # print(fname, p)
-            if fname.startswith(p) or ".egg-info" in fname:
-                if fname.endswith("/requires.txt"):
-                    meta["deps"] = f.extractfile(info).read()
-                save = False
-                if debug:
-                    print("Skipping", fname)
-                break
-
-        if save:
-            outfname = prefix + fname
-            if info.type != tarfile.DIRTYPE:
-                if debug:
-                    print("Extracting " + outfname)
-                _makedirs(outfname)
-                subf = f.extractfile(info)
-                save_file(outfname, subf)
-    return meta
-
-
-def expandhome(s):
-    if "~/" in s:
-        h = os.getenv("HOME")
-        s = s.replace("~/", h + "/")
-    return s
-
-
-import ussl
-import usocket
-
-warn_ussl = True
-
-
-def url_open(url):
-    global warn_ussl
-
-    if debug:
-        print(url)
-
-    proto, _, host, urlpath = url.split("/", 3)
-    try:
-        port = 443
-        if ":" in host:
-            host, port = host.split(":")
-            port = int(port)
-        ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM)
-    except OSError as e:
-        fatal("Unable to resolve %s (no Internet?)" % host, e)
-    # print("Address infos:", ai)
-    ai = ai[0]
-
-    s = usocket.socket(ai[0], ai[1], ai[2])
-    try:
-        # print("Connect address:", addr)
-        s.connect(ai[-1])
-
-        if proto == "https:":
-            s = ussl.wrap_socket(s, server_hostname=host)
-            if warn_ussl:
-                print("Warning: %s SSL certificate is not validated" % host)
-                warn_ussl = False
-
-        # MicroPython rawsocket module supports file interface directly
-        s.write("GET /%s HTTP/1.0\r\nHost: %s:%s\r\n\r\n" % (urlpath, host, port))
-        l = s.readline()
-        protover, status, msg = l.split(None, 2)
-        if status != b"200":
-            if status == b"404" or status == b"301":
-                raise NotFoundError("Package not found")
-            raise ValueError(status)
-        while 1:
-            l = s.readline()
-            if not l:
-                raise ValueError("Unexpected EOF in HTTP headers")
-            if l == b"\r\n":
-                break
-    except Exception as e:
-        s.close()
-        raise e
-
-    return s
-
-
-def get_pkg_metadata(name):
-    for url in index_urls:
-        try:
-            f = url_open("%s/%s/json" % (url, name))
-        except NotFoundError:
-            continue
-        try:
-            return json.load(f)
-        finally:
-            f.close()
-    raise NotFoundError("Package not found")
-
-
-def fatal(msg, exc=None):
-    print("Error:", msg)
-    if exc and debug:
-        raise exc
-    sys.exit(1)
-
-
-def install_pkg(pkg_spec, install_path):
-    package = pkg_spec.split("==")
-    data = get_pkg_metadata(package[0])
-
-    if len(package) == 1:
-        latest_ver = data["info"]["version"]
-    else:
-        latest_ver = package[1]
-    packages = data["releases"][latest_ver]
-    del data
-    gc.collect()
-    assert len(packages) == 1
-    package_url = packages[0]["url"]
-    print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
-    f1 = url_open(package_url)
-    try:
-        f2 = uzlib.DecompIO(f1, gzdict_sz)
-        f3 = tarfile.TarFile(fileobj=f2)
-        meta = install_tar(f3, install_path)
-    finally:
-        f1.close()
-    del f3
-    del f2
-    gc.collect()
-    return meta
-
-
-def install(to_install, install_path=None):
-    # Calculate gzip dictionary size to use
-    global gzdict_sz
-    sz = gc.mem_free() + gc.mem_alloc()
-    if sz <= 65536:
-        gzdict_sz = 16 + 12
-
-    if install_path is None:
-        install_path = get_install_path()
-    if install_path[-1] != "/":
-        install_path += "/"
-    if not isinstance(to_install, list):
-        to_install = [to_install]
-    print("Installing to: " + install_path)
-    # sets would be perfect here, but don't depend on them
-    installed = []
-    try:
-        while to_install:
-            if debug:
-                print("Queue:", to_install)
-            pkg_spec = to_install.pop(0)
-            if pkg_spec in installed:
-                continue
-            meta = install_pkg(pkg_spec, install_path)
-            installed.append(pkg_spec)
-            if debug:
-                print(meta)
-            deps = meta.get("deps", "").rstrip()
-            if deps:
-                deps = deps.decode("utf-8").split("\n")
-                to_install.extend(deps)
-    except Exception as e:
-        print(
-            "Error installing '{}': {}, packages may be partially installed".format(pkg_spec, e),
-            file=sys.stderr,
-        )
-
-
-def get_install_path():
-    global install_path
-    if install_path is None:
-        # sys.path[0] is current module's path
-        install_path = sys.path[1]
-        if install_path == ".frozen":
-            install_path = sys.path[2]
-    install_path = expandhome(install_path)
-    return install_path
-
-
-def cleanup():
-    for fname in cleanup_files:
-        try:
-            os.unlink(fname)
-        except OSError:
-            print("Warning: Cannot delete " + fname)
-
-
-def help():
-    print(
-        """\
-upip - Simple PyPI package manager for MicroPython
-Usage: micropython -m upip install [-p <path>] <package>... | -r <requirements.txt>
-import upip; upip.install(package_or_list, [<path>])
-
-If <path> isn't given, packages will be installed to sys.path[1], or
-sys.path[2] if the former is .frozen (path can be set from MICROPYPATH
-environment variable if supported)."""
-    )
-    print("Default install path:", get_install_path())
-    print(
-        """\
-
-Note: only MicroPython packages (usually, named micropython-*) are supported
-for installation, upip does not support arbitrary code in setup.py.
-"""
-    )
-
-
-def main():
-    global debug
-    global index_urls
-    global install_path
-    install_path = None
-
-    if len(sys.argv) < 2 or sys.argv[1] == "-h" or sys.argv[1] == "--help":
-        help()
-        return
-
-    if sys.argv[1] != "install":
-        fatal("Only 'install' command supported")
-
-    to_install = []
-
-    i = 2
-    while i < len(sys.argv) and sys.argv[i][0] == "-":
-        opt = sys.argv[i]
-        i += 1
-        if opt == "-h" or opt == "--help":
-            help()
-            return
-        elif opt == "-p":
-            install_path = sys.argv[i]
-            i += 1
-        elif opt == "-r":
-            list_file = sys.argv[i]
-            i += 1
-            with open(list_file) as f:
-                while True:
-                    l = f.readline()
-                    if not l:
-                        break
-                    if l[0] == "#":
-                        continue
-                    to_install.append(l.rstrip())
-        elif opt == "-i":
-            index_urls = [sys.argv[i]]
-            i += 1
-        elif opt == "--debug":
-            debug = True
-        else:
-            fatal("Unknown/unsupported option: " + opt)
-
-    to_install.extend(sys.argv[i:])
-    if not to_install:
-        help()
-        return
-
-    install(to_install)
-
-    if not debug:
-        cleanup()
-
-
-if __name__ == "__main__":
-    main()
diff --git a/tools/upip_utarfile.py b/tools/upip_utarfile.py
deleted file mode 100644
index 21b899f02..000000000
--- a/tools/upip_utarfile.py
+++ /dev/null
@@ -1,95 +0,0 @@
-import uctypes
-
-# http://www.gnu.org/software/tar/manual/html_node/Standard.html
-TAR_HEADER = {
-    "name": (uctypes.ARRAY | 0, uctypes.UINT8 | 100),
-    "size": (uctypes.ARRAY | 124, uctypes.UINT8 | 11),
-}
-
-DIRTYPE = "dir"
-REGTYPE = "file"
-
-
-def roundup(val, align):
-    return (val + align - 1) & ~(align - 1)
-
-
-class FileSection:
-    def __init__(self, f, content_len, aligned_len):
-        self.f = f
-        self.content_len = content_len
-        self.align = aligned_len - content_len
-
-    def read(self, sz=65536):
-        if self.content_len == 0:
-            return b""
-        if sz > self.content_len:
-            sz = self.content_len
-        data = self.f.read(sz)
-        sz = len(data)
-        self.content_len -= sz
-        return data
-
-    def readinto(self, buf):
-        if self.content_len == 0:
-            return 0
-        if len(buf) > self.content_len:
-            buf = memoryview(buf)[: self.content_len]
-        sz = self.f.readinto(buf)
-        self.content_len -= sz
-        return sz
-
-    def skip(self):
-        sz = self.content_len + self.align
-        if sz:
-            buf = bytearray(16)
-            while sz:
-                s = min(sz, 16)
-                self.f.readinto(buf, s)
-                sz -= s
-
-
-class TarInfo:
-    def __str__(self):
-        return "TarInfo(%r, %s, %d)" % (self.name, self.type, self.size)
-
-
-class TarFile:
-    def __init__(self, name=None, fileobj=None):
-        if fileobj:
-            self.f = fileobj
-        else:
-            self.f = open(name, "rb")
-        self.subf = None
-
-    def next(self):
-        if self.subf:
-            self.subf.skip()
-        buf = self.f.read(512)
-        if not buf:
-            return None
-
-        h = uctypes.struct(uctypes.addressof(buf), TAR_HEADER, uctypes.LITTLE_ENDIAN)
-
-        # Empty block means end of archive
-        if h.name[0] == 0:
-            return None
-
-        d = TarInfo()
-        d.name = str(h.name, "utf-8").rstrip("\0")
-        d.size = int(bytes(h.size), 8)
-        d.type = [REGTYPE, DIRTYPE][d.name[-1] == "/"]
-        self.subf = d.subf = FileSection(self.f, d.size, roundup(d.size, 512))
-        return d
-
-    def __iter__(self):
-        return self
-
-    def __next__(self):
-        v = self.next()
-        if v is None:
-            raise StopIteration
-        return v
-
-    def extractfile(self, tarinfo):
-        return tarinfo.subf