all: Bump version to 1.22.2.

Signed-off-by: Damien George <damien@micropython.org>

.git-blame-ignore-revs

@@ -0,0 +1,59 @@
+# all: Update Python formatting to ruff-format.
+bbd8760bd9a2302e5abee29db279102bb11d7732
+
+# all: Fix various spelling mistakes found by codespell 2.2.6.
+cf490a70917a1b2d38ba9b58e763e0837d0f7ca7
+
+# all: Fix spelling mistakes based on codespell check.
+b1229efbd1509654dec6053865ab828d769e29db
+
+# top: Update Python formatting to black "2023 stable style".
+8b2748269244304854b3462cb8902952b4dcb892
+
+# all: Reformat remaining C code that doesn't have a space after a comma.
+5b700b0af90591d6b1a2c087bb8de6b7f1bfdd2d
+
+# ports: Reformat more C and Python source code.
+5c32111fa0e31e451b0f1666bdf926be2fdfd82c
+
+# all: Update Python formatting to latest Black version 22.1.0.
+ab2923dfa1174dc177f0a90cb00a7e4ff87958d2
+
+# all: Update Python formatting to latest Black version 21.12b0.
+3770fab33449a5dadf8eb06edfae0767e75320a6
+
+# tools/gen-cpydiff.py: Fix formatting of doc strings for new Black.
+0f78c36c5aa458a954eed39a46942209107a553e
+
+# tests/run-tests.py: Reformat with Black.
+2a38d7103672580882fb621a5b76e8d26805d593
+
+# all: Update Python code to conform to latest black formatting.
+06659077a81b85882254cf0953c33b27614e018e
+
+# tools/uncrustify: Enable more opts to remove space between func and '('.
+77ed6f69ac35c1663a5633a8ee1d8a2446542204
+
+# tools/codeformat.py: Include extmod/{btstack,nimble} in code formatting.
+026fda605e03113d6e753290d65fed774418bc53
+
+# all: Format code to add space after C++-style comment start.
+84fa3312cfa7d2237d4b56952f2cd6e3591210c4
+
+# tests: Format all Python code with black, except tests in basics subdir.
+3dc324d3f1312e40d3a8ed87e7244966bb756f26
+
+# all: Remove spaces inside and around parenthesis.
+1a3e386c67e03a79eb768cb6e9f6777e002d6660
+
+# all: Remove spaces between nested paren and inside function arg paren.
+feb25775851ba0c04b8d1013716f442258879d9c
+
+# all: Reformat C and Python source code with tools/codeformat.py.
+69661f3343bedf86e514337cff63d96cc42f8859
+
+# stm32/usbdev: Convert files to unix line endings.
+abde0fa2267f9062b28c3c015d7662a550125cc6
+
+# all: Remove trailing spaces, per coding conventions.
+761e4c7ff62896c7d8f8c3dfc3cc98a4cc4f2f6f

.gitattributes

@@ -8,10 +8,12 @@
 
 # These are binary so should never be modified by git.
 *.a binary
+*.ico binary
 *.png binary
 *.jpg binary
 *.dxf binary
 *.mpy binary
+*.der binary
 
 # These should also not be modified by git.
 tests/basics/string_cr_conversion.py -text

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: micropython

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,25 @@
+---
+name: Bug report
+about: Report an issue
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+* Please search existing issues before raising a new issue. For questions about MicroPython or for help using MicroPython, or any sort of "how do I?" requests, please use the Discussions tab or raise a documentation request instead.
+
+* In your issue, please include a clear and concise description of what the bug is, the expected output, and how to replicate it.
+
+* If this issue involves external hardware, please include links to relevant datasheets and schematics.
+
+* If you are seeing code being executed incorrectly, please provide a minimal example and expected output (e.g. comparison to CPython).
+
+* For build issues, please include full details of your environment, compiler versions, command lines, and build output.
+
+* Please provide as much information as possible about the version of MicroPython you're running, such as:
+ - firmware file name
+ - git commit hash and port/board
+ - version information shown in the REPL (hit Ctrl-B to see the startup message)
+
+* Remove all placeholder text above before submitting.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,11 @@
+blank_issues_enabled: false
+contact_links:
+  - name: MicroPython GitHub Discussions
+    url: https://github.com/orgs/micropython/discussions
+    about: Community discussion about all things MicroPython. This is the best place to start if you have questions about using MicroPython or getting started with MicroPython development.
+  - name: MicroPython Documentation
+    url: https://docs.micropython.org/
+    about: Documentation for using and working with MicroPython and libraries.
+  - name: MicroPython Downloads
+    url: https://micropython.org/download/
+    about: Pre-built firmware and information for most supported boards.

.github/ISSUE_TEMPLATE/documentation.md

@@ -0,0 +1,16 @@
+---
+name: Documentation issue
+about: Report areas of the documentation or examples that need improvement
+title: 'docs: '
+labels: documentation
+assignees: ''
+
+---
+
+* Please search existing issues before raising a new issue. For questions about MicroPython or for help using MicroPython, or any sort of "how do I?" requests, please use the Discussions tab instead.
+
+* Describe what was missing from the documentation and/or what was incorrect/incomplete.
+
+* If possible, please link to the relevant page on https://docs.micropython.org/
+
+* Remove all placeholder text above before submitting.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,24 @@
+---
+name: Feature request
+about: Request a feature or improvement
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+* Please search existing issues before raising a new issue. For questions about MicroPython or for help using MicroPython, or any sort of "how do I?" requests, please use the Discussions tab or raise a documentation request instead.
+
+* Describe the feature you'd like to see added to MicroPython. In particular, what does this feature enable and why is it useful. MicroPython aims to strike a balance between functionality and code size, so please consider whether this feature can be optionally enabled and whether it can be provided in other ways (e.g. pure-Python library).
+
+* For core Python features, where possible please include a link to the relevant PEP.
+
+* For new architectures / ports / boards, please provide links to relevant documentation, specifications, and toolchains. Any information about the popularity and unique features about this hardware would also be useful.
+
+* For features for existing ports (e.g. new peripherals or microcontroller features), please describe which port(s) it applies too, and whether this is could be an extension to the machine API or a port-specific module?
+
+* For drivers (e.g. for external hardware), please link to datasheets and/or existing drivers from other sources.
+
+* Who do you expect will implement the feature you are requesting? Would you be willing to sponsor this work?
+
+* Remove all placeholder text above before submitting.

.github/ISSUE_TEMPLATE/security.md

@@ -0,0 +1,16 @@
+---
+name: Security report
+about: Report a security issue or vulnerability in MicroPython
+title: ''
+labels: security
+assignees: ''
+
+---
+
+* If you need to raise this issue privately with the MicroPython team, please email contact@micropython.org instead.
+
+* Include a clear and concise description of what the security issue is.
+
+* What does this issue allow an attacker to do?
+
+* Remove all placeholder text above before submitting.

.github/dependabot.yml

@@ -0,0 +1,7 @@
+version: 2
+updates:
+  # Maintain dependencies for GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "daily"

.github/workflows/code_formatting.yml

@@ -0,0 +1,30 @@
+name: Check code formatting
+
+on: [push, pull_request]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  code-formatting:
+    runs-on: ubuntu-22.04
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
+    - name: Install packages
+      run: source tools/ci.sh && ci_c_code_formatting_setup
+    - name: Run code formatting
+      run: source tools/ci.sh && ci_c_code_formatting_run
+    - name: Check code formatting
+      run: git diff --exit-code
+
+  code-spelling:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
+    - name: Install packages
+      run: source tools/ci.sh && ci_code_spell_setup
+    - name: Run spell checker
+      run: source tools/ci.sh && ci_code_spell_run

.github/workflows/code_size.yml

@@ -0,0 +1,45 @@
+name: Check code size
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'lib/**'
+      - 'ports/bare-arm/**'
+      - 'ports/minimal/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 100
+    - name: Install packages
+      run: source tools/ci.sh && ci_code_size_setup
+    - name: Build
+      run: source tools/ci.sh && ci_code_size_build
+    - name: Compute code size difference
+      run: tools/metrics.py diff ~/size0 ~/size1 | tee diff
+    - name: Save PR number
+      if: github.event_name == 'pull_request'
+      env:
+        PR_NUMBER: ${{ github.event.number }}
+      run: echo $PR_NUMBER > pr_number
+    - name: Upload diff
+      if: github.event_name == 'pull_request'
+      uses: actions/upload-artifact@v4
+      with:
+        name: code-size-report
+        path: |
+          diff
+          pr_number
+        retention-days: 1

.github/workflows/code_size_comment.yml

@@ -0,0 +1,105 @@
+name: Code size comment
+
+on:
+  workflow_run:
+    workflows: [Check code size]
+    types: [completed]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  comment:
+    runs-on: ubuntu-20.04
+    steps:
+      - name: 'Download artifact'
+        id: download-artifact
+        uses: actions/github-script@v7
+        with:
+          result-encoding: string
+          script: |
+            const fs = require('fs');
+
+            const allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
+               owner: context.repo.owner,
+               repo: context.repo.repo,
+               run_id: context.payload.workflow_run.id,
+            });
+
+            const matchArtifact = allArtifacts.data.artifacts.filter((artifact) => {
+              return artifact.name == "code-size-report"
+            });
+
+            if (matchArtifact.length === 0) {
+              console.log('no matching artifact found');
+              console.log('result: "skip"');
+
+              return 'skip';
+            }
+
+            const download = await github.rest.actions.downloadArtifact({
+               owner: context.repo.owner,
+               repo: context.repo.repo,
+               artifact_id: matchArtifact[0].id,
+               archive_format: 'zip',
+            });
+
+            fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/code-size-report.zip`, Buffer.from(download.data));
+
+            console.log('artifact downloaded to `code-size-report.zip`');
+            console.log('result: "ok"');
+
+            return 'ok';
+      - name: 'Unzip artifact'
+        if: steps.download-artifact.outputs.result == 'ok'
+        run: unzip code-size-report.zip
+      - name: Post comment to pull request
+        if: steps.download-artifact.outputs.result == 'ok'
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{secrets.GITHUB_TOKEN}}
+          script: |
+            const fs = require('fs');
+
+            const prNumber = Number(fs.readFileSync('pr_number'));
+            const codeSizeReport = `Code size report:
+
+            \`\`\`
+            ${fs.readFileSync('diff')}
+            \`\`\`
+            `;
+
+            const comments = await github.paginate(
+              github.rest.issues.listComments,
+              {
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: prNumber,
+              }
+            );
+
+            comments.reverse();
+
+            const previousComment = comments.find(comment =>
+              comment.user.login === 'github-actions[bot]'
+            )
+
+            // if github-actions[bot] already made a comment, update it,
+            // otherwise create a new comment.
+
+            if (previousComment) {
+              await github.rest.issues.updateComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                comment_id: previousComment.id,
+                body: codeSizeReport,
+              });
+            } else {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: prNumber,
+                body: codeSizeReport,
+              });
+            }

.github/workflows/commit_formatting.yml

@@ -0,0 +1,18 @@
+name: Check commit message formatting
+
+on: [push, pull_request]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: '100'
+    - uses: actions/setup-python@v5
+    - name: Check commit message formatting
+      run: source tools/ci.sh && ci_commit_formatting_run

.github/workflows/docs.yml

@@ -0,0 +1,23 @@
+name: Build docs
+
+on:
+  push:
+  pull_request:
+    paths:
+      - docs/**
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
+    - name: Install Python packages
+      run: pip install -r docs/requirements.txt
+    - name: Build docs
+      run: make -C docs/ html

.github/workflows/examples.yml

@@ -0,0 +1,25 @@
+name: Check examples
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'examples/**'
+      - 'ports/unix/**'
+      - 'py/**'
+      - 'shared/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  embedding:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: make -C examples/embedding -f micropython_embed.mk && make -C examples/embedding
+    - name: Run
+      run: ./examples/embedding/embed | grep "hello world"

.github/workflows/mpremote.yml

@@ -0,0 +1,29 @@
+name: Package mpremote
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        # Setting this to zero means fetch all history and tags,
+        # which hatch-vcs can use to discover the version tag.
+        fetch-depth: 0
+    - uses: actions/setup-python@v5
+    - name: Install build tools
+      run: pip install build
+    - name: Build mpremote wheel
+      run: cd tools/mpremote && python -m build --wheel
+    - name: Archive mpremote wheel
+      uses: actions/upload-artifact@v4
+      with:
+        name: mpremote
+        path: |
+          tools/mpremote/dist/mpremote*.whl

.github/workflows/mpy_format.yml

@@ -0,0 +1,24 @@
+name: .mpy file format and tools
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'examples/**'
+      - 'tests/**'
+      - 'tools/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-20.04 # use 20.04 to get python2
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_mpy_format_setup
+    - name: Test mpy-tool.py
+      run: source tools/ci.sh && ci_mpy_format_test

.github/workflows/ports.yml

@@ -0,0 +1,22 @@
+name: Build ports metadata
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - ports/**
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build ports download metadata
+      run: mkdir boards && ./tools/autobuild/build-downloads.py . ./boards

.github/workflows/ports_cc3200.yml

@@ -0,0 +1,28 @@
+name: cc3200 port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/cc3200/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_cc3200_setup
+    - name: Build
+      run: source tools/ci.sh && ci_cc3200_build

.github/workflows/ports_esp32.yml

@@ -0,0 +1,57 @@
+name: esp32 port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/esp32/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build_idf:
+    strategy:
+      fail-fast: false
+      matrix:
+        ci_func:  # names are functions in ci.sh
+          - esp32_build_cmod_spiram_s2
+          - esp32_build_s3_c3
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+
+    - id: idf_ver
+      name: Read the ESP-IDF version
+      run: source tools/ci.sh && echo "IDF_VER=$IDF_VER" | tee "$GITHUB_OUTPUT"
+
+    - name: Cached ESP-IDF install
+      id: cache_esp_idf
+      uses: actions/cache@v3
+      with:
+        path: |
+          ./esp-idf/
+          ~/.espressif/
+          !~/.espressif/dist/
+          ~/.cache/pip/
+        key: esp-idf-${{ steps.idf_ver.outputs.IDF_VER }}
+
+    - name: Install ESP-IDF packages
+      if: steps.cache_esp_idf.outputs.cache-hit != 'true'
+      run: source tools/ci.sh && ci_esp32_idf_setup
+
+    - name: ccache
+      uses: hendrikmuhs/ccache-action@v1.2
+      with:
+        key: esp32-${{ matrix.ci_func }}
+
+    - name: Build ci_${{matrix.ci_func }}
+      run: source tools/ci.sh && ci_${{ matrix.ci_func }}

.github/workflows/ports_esp8266.yml

@@ -0,0 +1,28 @@
+name: esp8266 port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/esp8266/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_esp8266_setup && ci_esp8266_path >> $GITHUB_PATH
+    - name: Build
+      run: source tools/ci.sh && ci_esp8266_build

.github/workflows/ports_mimxrt.yml

@@ -0,0 +1,28 @@
+name: mimxrt port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/mimxrt/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_mimxrt_setup
+    - name: Build
+      run: source tools/ci.sh && ci_mimxrt_build

.github/workflows/ports_nrf.yml

@@ -0,0 +1,28 @@
+name: nrf port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/nrf/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_nrf_setup
+    - name: Build
+      run: source tools/ci.sh && ci_nrf_build

.github/workflows/ports_powerpc.yml

@@ -0,0 +1,28 @@
+name: powerpc port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/powerpc/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_powerpc_setup
+    - name: Build
+      run: source tools/ci.sh && ci_powerpc_build

.github/workflows/ports_qemu-arm.yml

@@ -0,0 +1,32 @@
+name: qemu-arm port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/qemu-arm/**'
+      - 'tests/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build_and_test:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_qemu_arm_setup
+    - name: Build and run test suite
+      run: source tools/ci.sh && ci_qemu_arm_build
+    - name: Print failures
+      if: failure()
+      run: grep --before-context=100 --text "FAIL" ports/qemu-arm/build/console.out

.github/workflows/ports_renesas-ra.yml

@@ -0,0 +1,29 @@
+name: renesas-ra port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/renesas-ra/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build_renesas_ra_board:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_renesas_ra_setup
+    - name: Build
+      run: source tools/ci.sh && ci_renesas_ra_board_build
+

.github/workflows/ports_rp2.yml

@@ -0,0 +1,28 @@
+name: rp2 port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/rp2/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_rp2_setup
+    - name: Build
+      run: source tools/ci.sh && ci_rp2_build

.github/workflows/ports_samd.yml

@@ -0,0 +1,28 @@
+name: samd port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/samd/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_samd_setup
+    - name: Build
+      run: source tools/ci.sh && ci_samd_build

.github/workflows/ports_stm32.yml

@@ -0,0 +1,35 @@
+name: stm32 port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'drivers/**'
+      - 'ports/stm32/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build_stm32:
+    strategy:
+      fail-fast: false
+      matrix:
+        ci_func:  # names are functions in ci.sh
+          - stm32_pyb_build
+          - stm32_nucleo_build
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_stm32_setup
+    - name: Build ci_${{matrix.ci_func }}
+      run: source tools/ci.sh && ci_${{ matrix.ci_func }}
+

.github/workflows/ports_unix.yml

@@ -0,0 +1,236 @@
+name: unix port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'examples/**'
+      - 'mpy-cross/**'
+      - 'ports/unix/**'
+      - 'tests/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  minimal:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: source tools/ci.sh && ci_unix_minimal_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_minimal_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  reproducible:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build with reproducible date
+      run: source tools/ci.sh && ci_unix_minimal_build
+      env:
+        SOURCE_DATE_EPOCH: 1234567890
+    - name: Check reproducible build date
+      run: echo | ports/unix/build-minimal/micropython -i | grep 'on 2009-02-13;'
+
+  standard:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: source tools/ci.sh && ci_unix_standard_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_standard_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  standard_v2:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: source tools/ci.sh && ci_unix_standard_v2_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_standard_v2_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  coverage:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_coverage_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_coverage_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_coverage_run_tests
+    - name: Test merging .mpy files
+      run: source tools/ci.sh && ci_unix_coverage_run_mpy_merge_tests
+    - name: Build native mpy modules
+      run: source tools/ci.sh && ci_native_mpy_modules_build
+    - name: Test importing .mpy generated by mpy_ld.py
+      run: source tools/ci.sh && ci_unix_coverage_run_native_mpy_tests
+    - name: Run gcov coverage analysis
+      run: |
+        (cd ports/unix && gcov -o build-coverage/py ../../py/*.c || true)
+        (cd ports/unix && gcov -o build-coverage/extmod ../../extmod/*.c || true)
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v3
+      with:
+        fail_ci_if_error: true
+        verbose: true
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  coverage_32bit:
+    runs-on: ubuntu-20.04 # use 20.04 to get libffi-dev:i386
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_32bit_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_coverage_32bit_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_coverage_32bit_run_tests
+    - name: Build native mpy modules
+      run: source tools/ci.sh && ci_native_mpy_modules_32bit_build
+    - name: Test importing .mpy generated by mpy_ld.py
+      run: source tools/ci.sh && ci_unix_coverage_32bit_run_native_mpy_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  nanbox:
+    runs-on: ubuntu-20.04 # use 20.04 to get python2, and libffi-dev:i386
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_32bit_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_nanbox_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_nanbox_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  float:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: source tools/ci.sh && ci_unix_float_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_float_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  stackless_clang:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_clang_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_stackless_clang_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_stackless_clang_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  float_clang:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_clang_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_float_clang_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_float_clang_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  settrace:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: source tools/ci.sh && ci_unix_settrace_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_settrace_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  settrace_stackless:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: source tools/ci.sh && ci_unix_settrace_stackless_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_settrace_stackless_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  macos:
+    runs-on: macos-11.0
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
+      with:
+        python-version: '3.8'
+    - name: Build
+      run: source tools/ci.sh && ci_unix_macos_build
+    - name: Run tests
+      run: source tools/ci.sh && ci_unix_macos_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  qemu_mips:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_qemu_mips_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_qemu_mips_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_qemu_mips_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures
+
+  qemu_arm:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_unix_qemu_arm_setup
+    - name: Build
+      run: source tools/ci.sh && ci_unix_qemu_arm_build
+    - name: Run main test suite
+      run: source tools/ci.sh && ci_unix_qemu_arm_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures

.github/workflows/ports_webassembly.yml

@@ -0,0 +1,32 @@
+name: webassembly port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'ports/webassembly/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_webassembly_setup
+    - name: Build
+      run: source tools/ci.sh && ci_webassembly_build
+    - name: Run tests
+      run: source tools/ci.sh && ci_webassembly_run_tests
+    - name: Print failures
+      if: failure()
+      run: tests/run-tests.py --print-failures

.github/workflows/ports_windows.yml

@@ -0,0 +1,28 @@
+name: windows port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/*.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'ports/unix/**'
+      - 'ports/windows/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_windows_setup
+    - name: Build
+      run: source tools/ci.sh && ci_windows_build

.github/workflows/ports_zephyr.yml

@@ -0,0 +1,29 @@
+name: zephyr port
+
+on:
+  push:
+  pull_request:
+    paths:
+      - '.github/workflows/ports_zephyr.yml'
+      - 'tools/**'
+      - 'py/**'
+      - 'extmod/**'
+      - 'shared/**'
+      - 'lib/**'
+      - 'ports/zephyr/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install packages
+      run: source tools/ci.sh && ci_zephyr_setup
+    - name: Install Zephyr
+      run: source tools/ci.sh && ci_zephyr_install
+    - name: Build
+      run: source tools/ci.sh && ci_zephyr_build

.github/workflows/ruff.yml

@@ -0,0 +1,11 @@
+# https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+name: Python code lint and formatting with ruff
+on: [push, pull_request]
+jobs:
+  ruff:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - run: pip install --user ruff==0.1.3
+    - run: ruff check --output-format=github .
+    - run: ruff format --diff .

.gitignore

@@ -1,45 +1,25 @@
-# Compiled Sources
-###################
-*.o
-*.a
-*.elf
-*.bin
-*.map
-*.hex
-*.dis
-*.exe
-
-# Packages
-############
-
-# Logs and Databases
-######################
-*.log
-
-# VIM Swap Files
-######################
-*.swp
+# This .gitignore file is intended to be minimal.
+#
+# If you find that you need additional rules, such as IDE temporary
+# files, please do so either via a global .gitignore file (registered
+# with core.excludesFile), or by adding private repository-specific
+# rules to .git/info/exclude.  See https://git-scm.com/docs/gitignore
+# for more information.
 
 # Build directories
-######################
 build/
 build-*/
+docs/genrst/
 
 # Test failure outputs
-######################
-tests/*.exp
-tests/*.out
+tests/results/*
 
 # Python cache files
-######################
 __pycache__/
-*.pyc
 
 # Customized Makefile/project overrides
-######################
 GNUmakefile
 user.props
 
-# Generated rst files
-######################
-genrst/
+# MacOS desktop metadata files
+.DS_Store

.gitmodules

@@ -1,20 +1,19 @@
 [submodule "lib/axtls"]
 	path = lib/axtls
-	url = https://github.com/pfalcon/axtls
-	branch = micropython
+	url = https://github.com/micropython/axtls.git
 [submodule "lib/libffi"]
 	path = lib/libffi
 	url = https://github.com/atgreen/libffi
 [submodule "lib/lwip"]
 	path = lib/lwip
-	url = https://git.savannah.gnu.org/r/lwip.git
+	url = https://github.com/lwip-tcpip/lwip.git
 [submodule "lib/berkeley-db-1.xx"]
 	path = lib/berkeley-db-1.xx
 	url = https://github.com/pfalcon/berkeley-db-1.xx
 [submodule "lib/stm32lib"]
 	path = lib/stm32lib
 	url = https://github.com/micropython/stm32lib
-	branch = work-F4-1.13.1+F7-1.5.0+L4-1.3.0
+	branch = work-F0-1.9.0+F4-1.16.0+F7-1.7.0+G0-1.5.1+G4-1.3.0+H7-1.6.0+L0-1.11.2+L1-1.10.3+L4-1.17.0+WB-1.10.0+WL-1.1.0
 [submodule "lib/nrfx"]
 	path = lib/nrfx
 	url = https://github.com/NordicSemiconductor/nrfx.git
@@ -24,9 +23,39 @@
 [submodule "lib/asf4"]
 	path = lib/asf4
 	url = https://github.com/adafruit/asf4
+	branch = circuitpython
 [submodule "lib/tinyusb"]
 	path = lib/tinyusb
 	url = https://github.com/hathach/tinyusb
 [submodule "lib/mynewt-nimble"]
 	path = lib/mynewt-nimble
-	url = https://github.com/apache/mynewt-nimble.git
+	url = https://github.com/micropython/mynewt-nimble.git
+[submodule "lib/btstack"]
+	path = lib/btstack
+	url = https://github.com/bluekitchen/btstack.git
+[submodule "lib/nxp_driver"]
+	path = lib/nxp_driver
+	url = https://github.com/hathach/nxp_driver.git
+[submodule "lib/libhydrogen"]
+	path = lib/libhydrogen
+	url = https://github.com/jedisct1/libhydrogen.git
+[submodule "lib/pico-sdk"]
+	path = lib/pico-sdk
+	url = https://github.com/raspberrypi/pico-sdk.git
+[submodule "lib/fsp"]
+	path = lib/fsp
+	url = https://github.com/renesas/fsp.git
+[submodule "lib/wiznet"]
+	path = lib/wiznet5k
+	url = https://github.com/andrewleech/wiznet_ioLibrary_Driver.git
+	# Requires https://github.com/Wiznet/ioLibrary_Driver/pull/120
+	# url = https://github.com/Wiznet/ioLibrary_Driver.git
+[submodule "lib/cyw43-driver"]
+	path = lib/cyw43-driver
+	url = https://github.com/georgerobotics/cyw43-driver.git
+[submodule "lib/micropython-lib"]
+	path = lib/micropython-lib
+	url = https://github.com/micropython/micropython-lib.git
+[submodule "lib/protobuf-c"]
+	path = lib/protobuf-c
+	url = https://github.com/protobuf-c/protobuf-c.git

.pre-commit-config.yaml

@@ -0,0 +1,18 @@
+repos:
+  - repo: local
+    hooks:
+      - id: codeformat
+        name: MicroPython codeformat.py for changed C files
+        entry: tools/codeformat.py -v -c -f
+        language: python
+      - id: verifygitlog
+        name: MicroPython git commit message format checker
+        entry: tools/verifygitlog.py --check-file --ignore-rebase
+        language: python
+        verbose: true
+        stages: [commit-msg]
+  - repo: https://github.com/charliermarsh/ruff-pre-commit
+    rev: v0.1.3
+    hooks:
+      - id: ruff
+      - id: ruff-format

.travis.yml

@@ -1,265 +0,0 @@
-# global options
-dist: xenial
-language:
-  - c
-compiler:
-  - gcc
-cache:
-  directories:
-    - "${HOME}/persist"
-env:
-  global:
-    - MAKEOPTS="-j4"
-git:
-  submodules: false
-
-# define the successive stages
-stages:
-  - name: test
-
-# define the jobs for the stages
-# order of the jobs has longest running first to optimise total time
-jobs:
-  include:
-    # stm32 port
-    - stage: test
-      env: NAME="stm32 port build"
-      install:
-        # need newer gcc version for Cortex-M7 support
-        - sudo add-apt-repository -y ppa:team-gcc-arm-embedded/ppa
-        - sudo apt-get update -qq || true
-        - sudo apt-get install gcc-arm-embedded
-        - sudo apt-get install libnewlib-arm-none-eabi
-        - arm-none-eabi-gcc --version
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/stm32 submodules
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=NUCLEO_F091RC
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=PYBV11 MICROPY_PY_WIZNET5K=5200 MICROPY_PY_CC3K=1
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=PYBD_SF2
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=NUCLEO_H743ZI CFLAGS_EXTRA='-DMICROPY_PY_THREAD=1'
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=B_L072Z_LRWAN1
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=STM32L476DISC
-        - make ${MAKEOPTS} -C ports/stm32 BOARD=NUCLEO_WB55
-        - make ${MAKEOPTS} -C ports/stm32/mboot BOARD=PYBD_SF6
-
-    # qemu-arm port
-    - stage: test
-      env: NAME="qemu-arm port build and tests"
-      install:
-        - sudo apt-get install gcc-arm-none-eabi
-        - sudo apt-get install libnewlib-arm-none-eabi
-        - sudo apt-get install qemu-system
-        - arm-none-eabi-gcc --version
-        - qemu-system-arm --version
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/qemu-arm -f Makefile.test test
-      after_failure:
-        - grep "FAIL" ports/qemu-arm/build/console.out
-
-    # unix coverage
-    - stage: test
-      env: NAME="unix coverage build and tests"
-      install:
-        - sudo apt-get install python3-pip
-        - sudo pip install cpp-coveralls
-        - sudo pip3 install setuptools
-        - sudo pip3 install pyelftools
-        - gcc --version
-        - python3 --version
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/unix submodules
-        - make ${MAKEOPTS} -C ports/unix deplibs
-        - make ${MAKEOPTS} -C ports/unix coverage
-        # run the main test suite
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests)
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests -d thread)
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests --emit native)
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests --via-mpy -d basics float micropython)
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests --via-mpy --emit native -d basics float micropython)
-        # test when input script comes from stdin
-        - cat tests/basics/0prelim.py | ports/unix/micropython_coverage | grep -q 'abc'
-        # test building native mpy modules
-        - make -C examples/natmod/features1 ARCH=x64
-        - make -C examples/natmod/features2 ARCH=x64
-        - make -C examples/natmod/btree ARCH=x64
-        - make -C examples/natmod/framebuf ARCH=x64
-        - make -C examples/natmod/uheapq ARCH=x64
-        - make -C examples/natmod/urandom ARCH=x64
-        - make -C examples/natmod/ure ARCH=x64
-        - make -C examples/natmod/uzlib ARCH=x64
-        # test importing .mpy generated by mpy_ld.py
-        - MICROPYPATH=examples/natmod/features2 ./ports/unix/micropython_coverage -m features2
-        - (cd tests && ./run-natmodtests.py extmod/{btree*,framebuf*,uheapq*,ure*,uzlib*}.py)
-        # run coveralls coverage analysis (try to, even if some builds/tests failed)
-        - (cd ports/unix && coveralls --root ../.. --build-root . --gcov $(which gcov) --gcov-options '\-o build-coverage/' --include py --include extmod)
-      after_failure:
-        - (cd tests && for exp in *.exp; do testbase=$(basename $exp .exp); echo -e "\nFAILURE $testbase"; diff -u $testbase.exp $testbase.out; done)
-
-    # standard unix port
-    - stage: test
-      env: NAME="unix port build and tests"
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/unix submodules
-        - make ${MAKEOPTS} -C ports/unix deplibs
-        - make ${MAKEOPTS} -C ports/unix
-        - make ${MAKEOPTS} -C ports/unix test
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython ./run-perfbench.py 1000 1000)
-
-    # unix nanbox (and using Python 2 to check it can run the build scripts)
-    - stage: test
-      env: NAME="unix nanbox port build and tests"
-      install:
-        - sudo apt-get install gcc-multilib libffi-dev:i386
-      script:
-        - make ${MAKEOPTS} -C mpy-cross PYTHON=python2
-        - make ${MAKEOPTS} -C ports/unix submodules
-        - make ${MAKEOPTS} -C ports/unix PYTHON=python2 deplibs
-        - make ${MAKEOPTS} -C ports/unix PYTHON=python2 nanbox
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_nanbox ./run-tests)
-
-    # unix stackless
-    - stage: test
-      env: NAME="unix stackless port build and tests with clang"
-      install:
-        - sudo apt-get install clang
-      script:
-        - make ${MAKEOPTS} -C mpy-cross CC=clang
-        - make ${MAKEOPTS} -C ports/unix submodules
-        - make ${MAKEOPTS} -C ports/unix CC=clang CFLAGS_EXTRA="-DMICROPY_STACKLESS=1 -DMICROPY_STACKLESS_STRICT=1"
-        - make ${MAKEOPTS} -C ports/unix CC=clang test
-
-    # unix with sys.settrace
-    - stage: test
-      env: NAME="unix port with sys.settrace build and tests"
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/unix MICROPY_PY_BTREE=0 MICROPY_PY_FFI=0 MICROPY_PY_USSL=0 CFLAGS_EXTRA="-DMICROPY_PY_SYS_SETTRACE=1" test
-        - make ${MAKEOPTS} -C ports/unix clean
-        - make ${MAKEOPTS} -C ports/unix MICROPY_PY_BTREE=0 MICROPY_PY_FFI=0 MICROPY_PY_USSL=0 CFLAGS_EXTRA="-DMICROPY_STACKLESS=1 -DMICROPY_STACKLESS_STRICT=1 -DMICROPY_PY_SYS_SETTRACE=1" test
-      after_failure:
-        - (cd tests && for exp in *.exp; do testbase=$(basename $exp .exp); echo -e "\nFAILURE $testbase"; diff -u $testbase.exp $testbase.out; done)
-
-    # minimal unix port with tests
-    - stage: test
-      env: NAME="minimal unix port build and tests"
-      script:
-        - make ${MAKEOPTS} -C ports/unix minimal
-        - (cd tests && MICROPY_CPYTHON3=python3 MICROPY_MICROPYTHON=../ports/unix/micropython_minimal ./run-tests -e exception_chain -e self_type_check -e subclass_native_init -d basics)
-
-    # windows port via mingw
-    - stage: test
-      env: NAME="windows port build via mingw"
-      install:
-        - sudo apt-get install gcc-mingw-w64
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/windows CROSS_COMPILE=i686-w64-mingw32-
-
-    # esp32 port
-    - stage: test
-      env: NAME="esp32 port build"
-      install:
-        - sudo apt-get install python3-pip
-        - sudo pip3 install 'pyparsing<2.4'
-        - wget https://dl.espressif.com/dl/xtensa-esp32-elf-linux64-1.22.0-80-g6c4433a-5.2.0.tar.gz
-        - zcat xtensa-esp32-elf-linux64-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar x
-        - export PATH=$(pwd)/xtensa-esp32-elf/bin:$PATH
-        - git clone https://github.com/espressif/esp-idf.git
-        - export IDF_PATH=$(pwd)/esp-idf
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        # IDF v3 build
-        - git -C esp-idf checkout $(grep "ESPIDF_SUPHASH_V3 :=" ports/esp32/Makefile | cut -d " " -f 3)
-        - git -C esp-idf submodule update --init components/json/cJSON components/esp32/lib components/esptool_py/esptool components/expat/expat components/lwip/lwip components/mbedtls/mbedtls components/micro-ecc/micro-ecc components/nghttp/nghttp2
-        - make ${MAKEOPTS} -C ports/esp32 submodules
-        - make ${MAKEOPTS} -C ports/esp32
-        # clean
-        - git -C esp-idf clean -f -f -d components/json/cJSON components/esp32/lib components/expat/expat components/micro-ecc/micro-ecc components/nghttp/nghttp2
-        - make ${MAKEOPTS} -C ports/esp32 clean
-        # IDF v4 build
-        - git -C esp-idf checkout $(grep "ESPIDF_SUPHASH_V4 :=" ports/esp32/Makefile | cut -d " " -f 3)
-        - git -C esp-idf submodule update --init components/bt/controller/lib components/bt/host/nimble/nimble components/esp_wifi/lib_esp32 components/esptool_py/esptool components/lwip/lwip components/mbedtls/mbedtls
-        - make ${MAKEOPTS} -C ports/esp32 submodules
-        - make ${MAKEOPTS} -C ports/esp32
-
-    # esp8266 port
-    - stage: test
-      env: NAME="esp8266 port build"
-      install:
-        - wget https://github.com/jepler/esp-open-sdk/releases/download/2018-06-10/xtensa-lx106-elf-standalone.tar.gz
-        - zcat xtensa-lx106-elf-standalone.tar.gz | tar x
-        - export PATH=$(pwd)/xtensa-lx106-elf/bin:$PATH
-      script:
-        - make ${MAKEOPTS} -C mpy-cross
-        - make ${MAKEOPTS} -C ports/esp8266 submodules
-        - make ${MAKEOPTS} -C ports/esp8266
-        - make ${MAKEOPTS} -C ports/esp8266 BOARD=GENERIC_512K
-
-    # nrf port
-    - stage: test
-      env: NAME="nrf port build"
-      install:
-        - sudo apt-get install gcc-arm-none-eabi
-        - sudo apt-get install libnewlib-arm-none-eabi
-        - arm-none-eabi-gcc --version
-      script:
-        - make ${MAKEOPTS} -C ports/nrf submodules
-        - make ${MAKEOPTS} -C ports/nrf
-
-    # bare-arm and minimal ports
-    - stage: test
-      env: NAME="bare-arm and minimal ports build"
-      install:
-        - sudo apt-get install gcc-arm-none-eabi
-        - sudo apt-get install libnewlib-arm-none-eabi
-        - arm-none-eabi-gcc --version
-      script:
-        - make ${MAKEOPTS} -C ports/bare-arm
-        - make ${MAKEOPTS} -C ports/minimal CROSS=1 build/firmware.bin
-        - ls -l ports/minimal/build/firmware.bin
-        - tools/check_code_size.sh
-        - mkdir -p ${HOME}/persist
-        # Save new firmware for reference, but only if building a main branch, not a pull request
-        - 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then cp ports/minimal/build/firmware.bin ${HOME}/persist/; fi'
-
-    # cc3200 port
-    - stage: test
-      env: NAME="cc3200 port build"
-      install:
-        - sudo apt-get install gcc-arm-none-eabi
-        - sudo apt-get install libnewlib-arm-none-eabi
-      script:
-        - make ${MAKEOPTS} -C ports/cc3200 BTARGET=application BTYPE=release
-        - make ${MAKEOPTS} -C ports/cc3200 BTARGET=bootloader  BTYPE=release
-
-    # samd port
-    - stage: test
-      env: NAME="samd port build"
-      install:
-        - sudo apt-get install gcc-arm-none-eabi
-        - sudo apt-get install libnewlib-arm-none-eabi
-      script:
-        - make ${MAKEOPTS} -C ports/samd submodules
-        - make ${MAKEOPTS} -C ports/samd
-
-    # teensy port
-    - stage: test
-      env: NAME="teensy port build"
-      install:
-        - sudo apt-get install gcc-arm-none-eabi
-        - sudo apt-get install libnewlib-arm-none-eabi
-      script:
-        - make ${MAKEOPTS} -C ports/teensy
-
-    # powerpc port
-    - stage: test
-      env: NAME="powerpc port build"
-      install:
-        - sudo apt-get install gcc-powerpc64le-linux-gnu
-        - sudo apt-get install libc6-dev-ppc64el-cross
-      script:
-        - make ${MAKEOPTS} -C ports/powerpc CROSS_COMPILE=powerpc64le-linux-gnu-

CODECONVENTIONS.md

@@ -11,7 +11,7 @@ It's also ok to drop file extensions.
 
 Besides prefix, first line of a commit message should describe a
 change clearly and to the point, and be a grammatical sentence with
-final full stop. First line should fit within 78 characters. Examples
+final full stop. First line must fit within 72 characters. Examples
 of good first line of commit messages:
 
     py/objstr: Add splitlines() method.
@@ -19,16 +19,17 @@ of good first line of commit messages:
     docs/machine: Fix typo in reset() description.
     ports: Switch to use lib/foo instead of duplicated code.
 
-After the first line, add an empty line and in following lines describe
-a change in a detail, if needed. Any change beyond 5 lines would likely
-require such detailed description.
+After the first line add an empty line and in the following lines describe
+the change in a detail, if needed, with lines fitting within 75 characters
+(with an exception for long items like URLs which cannot be broken). Any
+change beyond 5 lines would likely require such detailed description.
 
 To get good practical examples of good commits and their messages, browse
 the `git log` of the project.
 
-MicroPython doesn't require explicit sign-off for patches ("Signed-off-by"
-lines and similar). Instead, the commit message, and your name and email
-address on it construes your sign-off of the following:
+When committing you must sign-off your commit by adding "Signed-off-by:"
+line(s) at the end of the commit message, e.g. using `git commit -s`.  You
+are then certifying and signing off against the following:
 
 * That you wrote the change yourself, or took it from a project with
   a compatible license (in the latter case the commit message, and possibly
@@ -42,19 +43,137 @@ address on it construes your sign-off of the following:
   copyright for your changes (for smaller changes, the commit message
   conveys your copyright; if you make significant changes to a particular
   source module, you're welcome to add your name to the file header).
-* Your signature for all of the above, which is the 'Author' line in
-  the commit message, and which should include your full real name and
-  a valid and active email address by which you can be contacted in the
-  foreseeable future.
+* Your contribution including commit message will be publicly and
+  indefinitely available for anyone to access, including redistribution
+  under the terms of the project's license.
+* Your signature for all of the above, which is the "Signed-off-by" line,
+  includes your full real name and a valid and active email address by
+  which you can be contacted in the foreseeable future.
+
+Code auto-formatting
+====================
+
+Both C and Python code formatting are controlled for consistency across the
+MicroPython codebase.  C code is formatted using the `tools/codeformat.py`
+script which uses [uncrustify](https://github.com/uncrustify/uncrustify).
+Python code is linted and formatted using
+[ruff & ruff format](https://github.com/astral-sh/ruff).
+After making changes, and before committing, run  `tools/codeformat.py` to
+reformat your C code and `ruff format` for any Python code.  Without
+arguments this tool will reformat all source code (and may take some time
+to run).  Otherwise pass as arguments to the tool the files that changed,
+and it will only reformat those.
+
+uncrustify
+==========
+
+Only [uncrustify](https://github.com/uncrustify/uncrustify) v0.71 or v0.72 can
+be used for MicroPython. Different uncrustify versions produce slightly
+different formatting, and the configuration file formats are often
+incompatible. v0.73 or newer *will not work*.
+
+Depending on your operating system version, it may be possible to install a pre-compiled
+uncrustify version:
+
+Ubuntu, Debian
+--------------
+
+Ubuntu versions 21.10 or 22.04LTS, and Debian versions bullseye or bookworm all
+include v0.72 so can be installed directly:
+
+```
+$ apt install uncrustify
+```
+
+Arch Linux
+----------
+
+The current Arch uncrustify version is too new. There is an [old Arch package
+for v0.72](https://archive.archlinux.org/packages/u/uncrustify/) that can be
+installed from the Arch Linux archive ([more
+information](https://wiki.archlinux.org/title/Downgrading_packages#Arch_Linux_Archive)). Use
+the [IgnorePkg feature](https://wiki.archlinux.org/title/Pacman#Skip_package_from_being_upgraded)
+to prevent it re-updating.
+
+Brew
+----
+
+This command may work, please raise a new Issue if it doesn't:
+
+```
+curl -L https://github.com/Homebrew/homebrew-core/raw/2b07d8192623365078a8b855a164ebcdf81494a6/Formula/uncrustify.rb > uncrustify.rb && brew install uncrustify.rb && rm uncrustify.rb
+```
+
+Automatic Pre-Commit Hooks
+==========================
+
+To have code formatting and commit message conventions automatically checked,
+a configuration file is provided for the [pre-commit](https://pre-commit.com/)
+tool.
+
+First install `pre-commit`, either from your system package manager or via
+`pip`. When installing `pre-commit` via pip, it is recommended to use a
+virtual environment. Other sources, such as Brew are also available, see
+[the docs](https://pre-commit.com/index.html#install) for details.
+
+```
+$ apt install pre-commit       # Ubuntu, Debian
+$ pacman -Sy python-precommit  # Arch Linux
+$ brew install pre-commit      # Brew
+$ pip install pre-commit       # PyPI
+```
+
+Next, install [uncrustify (see above)](#uncrustify). Other dependencies are managed by
+pre-commit automatically, but uncrustify needs to be installed and available on
+the PATH.
+
+Then, inside the MicroPython repository, register the git hooks for pre-commit
+by running:
+
+```
+$ pre-commit install --hook-type pre-commit --hook-type commit-msg
+```
+
+pre-commit will now automatically run during `git commit` for both code and
+commit message formatting.
+
+The same formatting checks will be run by CI for any Pull Request submitted to
+MicroPython. Pre-commit allows you to see any failure more quickly, and in many
+cases will automatically correct it in your local working copy.
+
+To unregister `pre-commit` from your MicroPython repository, run:
+
+```
+$ pre-commit uninstall --hook-type pre-commit --hook-type commit-msg
+```
+
+Tips:
+
+* To skip pre-commit checks on a single commit, use `git commit -n` (for
+  `--no-verify`).
+* To ignore the pre-commit message format check temporarily, start the commit
+  message subject line with "WIP" (for "Work In Progress").
+
+Running pre-commit manually
+===========================
+
+Once pre-commit is installed as per the previous section it can be manually
+run against the MicroPython python codebase to update file formatting on
+demand, with either:
+* `pre-commit run --all-files` to fix all files in the MicroPython codebase
+* `pre-commit run --file ./path/to/my/file` to fix just one file
+* `pre-commit run --file ./path/to/my/folder/*` to fix just one folder
 
 Python code conventions
 =======================
 
-Python code follows [PEP 8](http://legacy.python.org/dev/peps/pep-0008/).
+Python code follows [PEP 8](https://legacy.python.org/dev/peps/pep-0008/) and
+is auto-formatted using [ruff format](https://docs.astral.sh/ruff/formatter)
+with a line-length of 99 characters.
 
 Naming conventions:
 - Module names are short and all lowercase; eg pyb, stm.
-- Class names are CamelCase, with abreviations all uppercase; eg I2C, not
+- Class names are CamelCase, with abbreviations all uppercase; eg I2C, not
   I2c.
 - Function and method names are all lowercase with words separated by
   a single underscore as necessary to improve readability; eg mem_read.
@@ -64,7 +183,12 @@ Naming conventions:
 C code conventions
 ==================
 
-When writing new C code, please adhere to the following conventions.
+C code is auto-formatted using [uncrustify](https://github.com/uncrustify/uncrustify)
+and the corresponding configuration file `tools/uncrustify.cfg`, with a few
+minor fix-ups applied by `tools/codeformat.py`.  When writing new C code please
+adhere to the existing style and use `tools/codeformat.py` to check any changes.
+The main conventions, and things not enforceable via the auto-formatter, are
+described below.
 
 White space:
 - Expand tabs to 4 spaces.
@@ -125,7 +249,7 @@ Braces, spaces, names and comments:
             foo(x + TO_ADD, some_value - 1);
         }
 
-        for (int my_counter = 0; my_counter < x; my_counter++) {
+        for (int my_counter = 0; my_counter < x; ++my_counter) {
         }
     }
 
@@ -140,7 +264,7 @@ Documentation conventions
 =========================
 
 MicroPython generally follows CPython in documentation process and
-conventions. reStructuredText syntax is used for the documention.
+conventions. reStructuredText syntax is used for the documentation.
 
 Specific conventions/suggestions:
 

CONTRIBUTING.md

@@ -3,6 +3,6 @@ make sure that you are acquainted with Contributor Guidelines:
 
 https://github.com/micropython/micropython/wiki/ContributorGuidelines
 
-and Code Conventions:
+as well as the Code Conventions, which includes details of how to commit:
 
 https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2013-2019 Damien P. George
+Copyright (c) 2013-2023 Damien P. George
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -19,3 +19,68 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+Unless specified otherwise (see below), the above license and copyright applies
+to all files in this repository.
+
+Individual files may include additional copyright holders.
+
+The various ports of MicroPython may include third-party software that is
+licensed under different terms. These licenses are summarised in the tree
+below, please refer to these files and directories for further license and
+copyright information. Note that (L)GPL-licensed code listed below is only
+used during the build process and is not part of the compiled source code.
+
+/ (MIT)
+    /drivers
+        /cc3100 (BSD-3-clause)
+    /lib
+        /asf4 (Apache-2.0)
+        /axtls (BSD-3-clause)
+            /config
+                /scripts
+                    /config (GPL-2.0-or-later)
+                /Rules.mak (GPL-2.0)
+        /berkeley-db-1xx (BSD-4-clause)
+        /btstack (See btstack/LICENSE)
+        /cmsis (BSD-3-clause)
+        /crypto-algorithms (NONE)
+        /libhydrogen (ISC)
+        /littlefs (BSD-3-clause)
+        /lwip (BSD-3-clause)
+        /mynewt-nimble (Apache-2.0)
+        /nrfx (BSD-3-clause)
+        /nxp_driver (BSD-3-Clause)
+        /oofatfs (BSD-1-clause)
+        /pico-sdk (BSD-3-clause)
+        /re15 (BSD-3-clause)
+        /stm32lib (BSD-3-clause)
+        /tinytest (BSD-3-clause)
+        /tinyusb (MIT)
+        /uzlib (Zlib)
+        /wiznet5k (MIT)
+    /logo (uses OFL-1.1)
+    /ports
+        /cc3200
+            /hal (BSD-3-clause)
+            /simplelink (BSD-3-clause)
+            /FreeRTOS (GPL-2.0 with FreeRTOS exception)
+        /esp32
+            /ppp_set_auth.* (Apache-2.0)
+        /rp2
+            /mutex_extra.c (BSD-3-clause)
+        /stm32
+            /usbd*.c (MCD-ST Liberty SW License Agreement V2)
+            /stm32_it.* (MIT + BSD-3-clause)
+            /system_stm32*.c (MIT + BSD-3-clause)
+            /boards
+                /startup_stm32*.s (BSD-3-clause)
+                /*/stm32*.h (BSD-3-clause)
+            /usbdev (MCD-ST Liberty SW License Agreement V2)
+            /usbhost (MCD-ST Liberty SW License Agreement V2)
+        /zephyr
+            /src (Apache-2.0)
+    /tools
+        /dfu.py (LGPL-3.0-only)

README.md

@@ -1,4 +1,4 @@
-[![Build Status](https://travis-ci.org/micropython/micropython.png?branch=master)](https://travis-ci.org/micropython/micropython) [![Coverage Status](https://coveralls.io/repos/micropython/micropython/badge.png?branch=master)](https://coveralls.io/r/micropython/micropython?branch=master)
+[![Unix CI badge](https://github.com/micropython/micropython/actions/workflows/ports_unix.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![STM32 CI badge](https://github.com/micropython/micropython/actions/workflows/ports_stm32.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![Docs CI badge](https://github.com/micropython/micropython/actions/workflows/docs.yml/badge.svg)](https://docs.micropython.org/) [![codecov](https://codecov.io/gh/micropython/micropython/branch/master/graph/badge.svg?token=I92PfD05sD)](https://codecov.io/gh/micropython/micropython)
 
 The MicroPython project
 =======================
@@ -15,167 +15,43 @@ code-base, including project-wide name changes and API changes.
 
 MicroPython implements the entire Python 3.4 syntax (including exceptions,
 `with`, `yield from`, etc., and additionally `async`/`await` keywords from
-Python 3.5). The following core datatypes are provided: `str` (including
-basic Unicode support), `bytes`, `bytearray`, `tuple`, `list`, `dict`, `set`,
-`frozenset`, `array.array`, `collections.namedtuple`, classes and instances.
-Builtin modules include `sys`, `time`, and `struct`, etc. Select ports have
-support for `_thread` module (multithreading). Note that only a subset of
-Python 3 functionality is implemented for the data types and modules.
+Python 3.5 and some select features from later versions). The following core
+datatypes are provided: `str`(including basic Unicode support), `bytes`,
+`bytearray`, `tuple`, `list`, `dict`, `set`, `frozenset`, `array.array`,
+`collections.namedtuple`, classes and instances. Builtin modules include
+`os`, `sys`, `time`, `re`, and `struct`, etc. Select ports have support for
+`_thread` module (multithreading), `socket` and `ssl` for networking, and
+`asyncio`. Note that only a subset of Python 3 functionality is implemented
+for the data types and modules.
 
-MicroPython can execute scripts in textual source form or from precompiled
-bytecode, in both cases either from an on-device filesystem or "frozen" into
-the MicroPython executable.
+MicroPython can execute scripts in textual source form (.py files) or from
+precompiled bytecode (.mpy files), in both cases either from an on-device
+filesystem or "frozen" into the MicroPython executable.
 
-See the repository http://github.com/micropython/pyboard for the MicroPython
-board (PyBoard), the officially supported reference electronic circuit board.
+MicroPython also provides a set of MicroPython-specific modules to access
+hardware-specific functionality and peripherals such as GPIO, Timers, ADC,
+DAC, PWM, SPI, I2C, CAN, Bluetooth, and USB.
 
-Major components in this repository:
-- py/ -- the core Python implementation, including compiler, runtime, and
-  core library.
-- mpy-cross/ -- the MicroPython cross-compiler which is used to turn scripts
-  into precompiled bytecode.
-- ports/unix/ -- a version of MicroPython that runs on Unix.
-- ports/stm32/ -- a version of MicroPython that runs on the PyBoard and similar
-  STM32 boards (using ST's Cube HAL drivers).
-- ports/minimal/ -- a minimal MicroPython port. Start with this if you want
-  to port MicroPython to another microcontroller.
-- tests/ -- test framework and test scripts.
-- docs/ -- user documentation in Sphinx reStructuredText format. Rendered
-  HTML documentation is available at http://docs.micropython.org.
+Getting started
+---------------
 
-Additional components:
-- ports/bare-arm/ -- a bare minimum version of MicroPython for ARM MCUs. Used
-  mostly to control code size.
-- ports/teensy/ -- a version of MicroPython that runs on the Teensy 3.1
-  (preliminary but functional).
-- ports/pic16bit/ -- a version of MicroPython for 16-bit PIC microcontrollers.
-- ports/cc3200/ -- a version of MicroPython that runs on the CC3200 from TI.
-- ports/esp8266/ -- a version of MicroPython that runs on Espressif's ESP8266 SoC.
-- ports/esp32/ -- a version of MicroPython that runs on Espressif's ESP32 SoC.
-- ports/nrf/ -- a version of MicroPython that runs on Nordic's nRF51 and nRF52 MCUs.
-- extmod/ -- additional (non-core) modules implemented in C.
-- tools/ -- various tools, including the pyboard.py module.
-- examples/ -- a few example Python scripts.
+See the [online documentation](https://docs.micropython.org/) for API
+references and information about using MicroPython and information about how
+it is implemented.
 
-The subdirectories above may include READMEs with additional info.
+We use [GitHub Discussions](https://github.com/micropython/micropython/discussions)
+as our forum, and [Discord](https://discord.gg/RB8HZSAExQ) for chat. These
+are great places to ask questions and advice from the community or to discuss your
+MicroPython-based projects.
 
-"make" is used to build the components, or "gmake" on BSD-based systems.
-You will also need bash, gcc, and Python 3.3+ available as the command `python3`
-(if your system only has Python 2.7 then invoke make with the additional option
-`PYTHON=python2`).
+For bugs and feature requests, please [raise an issue](https://github.com/micropython/micropython/issues/new/choose)
+and follow the templates there.
 
-The MicroPython cross-compiler, mpy-cross
------------------------------------------
-
-Most ports require the MicroPython cross-compiler to be built first.  This
-program, called mpy-cross, is used to pre-compile Python scripts to .mpy
-files which can then be included (frozen) into the firmware/executable for
-a port.  To build mpy-cross use:
-
-    $ cd mpy-cross
-    $ make
-
-The Unix version
-----------------
-
-The "unix" port requires a standard Unix environment with gcc and GNU make.
-x86 and x64 architectures are supported (i.e. x86 32- and 64-bit), as well
-as ARM and MIPS. Making full-featured port to another architecture requires
-writing some assembly code for the exception handling and garbage collection.
-Alternatively, fallback implementation based on setjmp/longjmp can be used.
-
-To build (see section below for required dependencies):
-
-    $ cd ports/unix
-    $ make submodules
-    $ make
-
-Then to give it a try:
-
-    $ ./micropython
-    >>> list(5 * x + y for x in range(10) for y in [4, 2, 1])
-
-Use `CTRL-D` (i.e. EOF) to exit the shell.
-Learn about command-line options (in particular, how to increase heap size
-which may be needed for larger applications):
-
-    $ ./micropython --help
-
-Run complete testsuite:
-
-    $ make test
-
-Unix version comes with a builtin package manager called upip, e.g.:
-
-    $ ./micropython -m upip install micropython-pystone
-    $ ./micropython -m pystone
-
-Browse available modules on
-[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython).
-Standard library modules come from
-[micropython-lib](https://github.com/micropython/micropython-lib) project.
-
-External dependencies
----------------------
-
-Building MicroPython ports may require some dependencies installed.
-
-For Unix port, `libffi` library and `pkg-config` tool are required. On
-Debian/Ubuntu/Mint derivative Linux distros, install `build-essential`
-(includes toolchain and make), `libffi-dev`, and `pkg-config` packages.
-
-Other dependencies can be built together with MicroPython. This may
-be required to enable extra features or capabilities, and in recent
-versions of MicroPython, these may be enabled by default. To build
-these additional dependencies, in the port directory you're
-interested in (e.g. `ports/unix/`) first execute:
-
-    $ make submodules
-
-This will fetch all the relevant git submodules (sub repositories) that
-the port needs.  Use the same command to get the latest versions of
-submodules as they are updated from time to time. After that execute:
-
-    $ make deplibs
-
-This will build all available dependencies (regardless whether they
-are used or not). If you intend to build MicroPython with additional
-options (like cross-compiling), the same set of options should be passed
-to `make deplibs`. To actually enable/disable use of dependencies, edit
-`ports/unix/mpconfigport.mk` file, which has inline descriptions of the options.
-For example, to build SSL module (required for `upip` tool described above,
-and so enabled by dfeault), `MICROPY_PY_USSL` should be set to 1.
-
-For some ports, building required dependences is transparent, and happens
-automatically.  But they still need to be fetched with the `make submodules`
-command.
-
-The STM32 version
------------------
-
-The "stm32" port requires an ARM compiler, arm-none-eabi-gcc, and associated
-bin-utils.  For those using Arch Linux, you need arm-none-eabi-binutils,
-arm-none-eabi-gcc and arm-none-eabi-newlib packages.  Otherwise, try here:
-https://launchpad.net/gcc-arm-embedded
-
-To build:
-
-    $ cd ports/stm32
-    $ make submodules
-    $ make
-
-You then need to get your board into DFU mode.  On the pyboard, connect the
-3V3 pin to the P1/DFU pin with a wire (on PYBv1.0 they are next to each other
-on the bottom left of the board, second row from the bottom).
-
-Then to flash the code via USB DFU to your device:
-
-    $ make deploy
-
-This will use the included `tools/pydfu.py` script.  If flashing the firmware
-does not work it may be because you don't have the correct permissions, and
-need to use `sudo make deploy`.
-See the README.md file in the ports/stm32/ directory for further details.
+For information about the [MicroPython pyboard](https://store.micropython.org/pyb-features),
+the officially supported board from the
+[original Kickstarter campaign](https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers),
+see the [schematics and pinouts](http://github.com/micropython/pyboard) and
+[documentation](https://docs.micropython.org/en/latest/pyboard/quickref.html).
 
 Contributing
 ------------
@@ -186,3 +62,86 @@ productive, please be sure to follow the
 and the [Code Conventions](https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md).
 Note that MicroPython is licenced under the MIT license, and all contributions
 should follow this license.
+
+About this repository
+---------------------
+
+This repository contains the following components:
+- [py/](py/) -- the core Python implementation, including compiler, runtime, and
+  core library.
+- [mpy-cross/](mpy-cross/) -- the MicroPython cross-compiler which is used to turn scripts
+  into precompiled bytecode.
+- [ports/](ports/) -- platform-specific code for the various ports and architectures that MicroPython runs on.
+- [lib/](lib/) -- submodules for external dependencies.
+- [tests/](tests/) -- test framework and test scripts.
+- [docs/](docs/) -- user documentation in Sphinx reStructuredText format. This is used to generate the [online documentation](http://docs.micropython.org).
+- [extmod/](extmod/) -- additional (non-core) modules implemented in C.
+- [tools/](tools/) -- various tools, including the pyboard.py module.
+- [examples/](examples/) -- a few example Python scripts.
+
+"make" is used to build the components, or "gmake" on BSD-based systems.
+You will also need bash, gcc, and Python 3.3+ available as the command `python3`
+(if your system only has Python 2.7 then invoke make with the additional option
+`PYTHON=python2`). Some ports (rp2 and esp32) additionally use CMake.
+
+Supported platforms & architectures
+-----------------------------------
+
+MicroPython runs on a wide range of microcontrollers, as well as on Unix-like
+(including Linux, BSD, macOS, WSL) and Windows systems.
+
+Microcontroller targets can be as small as 256kiB flash + 16kiB RAM, although
+devices with at least 512kiB flash + 128kiB RAM allow a much more
+full-featured experience.
+
+The [Unix](ports/unix) and [Windows](ports/windows) ports allow both
+development and testing of MicroPython itself, as well as providing
+lightweight alternative to CPython on these platforms (in particular on
+embedded Linux systems).
+
+The ["minimal"](ports/minimal) port provides an example of a very basic
+MicroPython port and can be compiled as both a standalone Linux binary as
+well as for ARM Cortex M4. Start with this if you want to port MicroPython to
+another microcontroller. Additionally the ["bare-arm"](ports/bare-arm) port
+is an example of the absolute minimum configuration, and is used to keep
+track of the code size of the core runtime and VM.
+
+In addition, the following ports are provided in this repository:
+ - [cc3200](ports/cc3200) -- Texas Instruments CC3200 (including PyCom WiPy).
+ - [esp32](ports/esp32) -- Espressif ESP32 SoC (including ESP32S2, ESP32S3, ESP32C3).
+ - [esp8266](ports/esp8266) -- Espressif ESP8266 SoC.
+ - [mimxrt](ports/mimxrt) -- NXP m.iMX RT (including Teensy 4.x).
+ - [nrf](ports/nrf) -- Nordic Semiconductor nRF51 and nRF52.
+ - [pic16bit](ports/pic16bit) -- Microchip PIC 16-bit.
+ - [powerpc](ports/powerpc) -- IBM PowerPC (including Microwatt)
+ - [qemu-arm](ports/qemu-arm) -- QEMU-based emulated target, for testing)
+ - [renesas-ra](ports/renesas-ra) -- Renesas RA family.
+ - [rp2](ports/rp2) -- Raspberry Pi RP2040 (including Pico and Pico W).
+ - [samd](ports/samd) -- Microchip (formerly Atmel) SAMD21 and SAMD51.
+ - [stm32](ports/stm32) -- STMicroelectronics STM32 family (including F0, F4, F7, G0, G4, H7, L0, L4, WB)
+ - [webassembly](ports/webassembly) -- Emscripten port targeting browsers and NodeJS.
+ - [zephyr](ports/zephyr) -- Zephyr RTOS.
+
+The MicroPython cross-compiler, mpy-cross
+-----------------------------------------
+
+Most ports require the [MicroPython cross-compiler](mpy-cross) to be built
+first.  This program, called mpy-cross, is used to pre-compile Python scripts
+to .mpy files which can then be included (frozen) into the
+firmware/executable for a port.  To build mpy-cross use:
+
+    $ cd mpy-cross
+    $ make
+
+External dependencies
+---------------------
+
+The core MicroPython VM and runtime has no external dependencies, but a given
+port might depend on third-party drivers or vendor HALs. This repository
+includes [several submodules](lib/) linking to these external dependencies.
+Before compiling a given port, use
+
+    $ cd ports/name
+    $ make submodules
+
+to ensure that all required submodules are initialised.

docs/Makefile

@@ -3,7 +3,7 @@
 
 # You can set these variables from the command line.
 PYTHON        = python3
-SPHINXOPTS    =
+SPHINXOPTS    = -W --keep-going -j auto
 SPHINXBUILD   = sphinx-build
 PAPER         =
 BUILDDIR      = build/$(MICROPY_PORT)

docs/README.md

@@ -25,6 +25,21 @@ In `micropython/docs`, build the docs:
 
 You'll find the index page at `micropython/docs/build/html/index.html`.
 
+Having readthedocs.org build the documentation
+----------------------------------------------
+
+If you would like to have docs for forks/branches hosted on GitHub, GitLab or
+BitBucket an alternative to building the docs locally is to sign up for a free
+https://readthedocs.org account. The rough steps to follow are:
+1. sign-up for an account, unless you already have one
+2. in your account settings: add GitHub as a connected service (assuming
+you have forked this repo on github)
+3. in your account projects: import your forked/cloned micropython repository
+into readthedocs
+4. in the project's versions: add the branches you are developing on or
+for which you'd like readthedocs to auto-generate docs whenever you
+push a change
+
 PDF manual generation
 ---------------------
 
@@ -32,7 +47,7 @@ This can be achieved with:
 
     make latexpdf
 
-but require rather complete install of LaTeX with various extensions. On
-Debian/Ubuntu, try (500MB+ download):
+but requires a rather complete install of LaTeX with various extensions. On
+Debian/Ubuntu, try (1GB+ download):
 
-    apt-get install texlive-latex-recommended texlive-latex-extra
+    apt install texlive-latex-recommended texlive-latex-extra texlive-xetex texlive-fonts-extra cm-super xindy

docs/conf.py

@@ -19,54 +19,56 @@ import os
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
-sys.path.insert(0, os.path.abspath('.'))
+sys.path.insert(0, os.path.abspath("."))
+
+# The MICROPY_VERSION env var should be "vX.Y.Z" (or unset).
+micropy_version = os.getenv("MICROPY_VERSION") or "latest"
+micropy_all_versions = (os.getenv("MICROPY_ALL_VERSIONS") or "latest").split(",")
+url_pattern = "%s/en/%%s" % (os.getenv("MICROPY_URL_PREFIX") or "/",)
 
 # The members of the html_context dict are available inside topindex.html
-micropy_version = os.getenv('MICROPY_VERSION') or 'latest'
-micropy_all_versions = (os.getenv('MICROPY_ALL_VERSIONS') or 'latest').split(',')
-url_pattern = '%s/en/%%s' % (os.getenv('MICROPY_URL_PREFIX') or '/',)
 html_context = {
-    'cur_version':micropy_version,
-    'all_versions':[
-        (ver, url_pattern % ver) for ver in micropy_all_versions
-    ],
-    'downloads':[
-        ('PDF', url_pattern % micropy_version + '/micropython-docs.pdf'),
+    "cur_version": micropy_version,
+    "all_versions": [(ver, url_pattern % ver) for ver in micropy_all_versions],
+    "downloads": [
+        ("PDF", url_pattern % micropy_version + "/micropython-docs.pdf"),
     ],
+    "is_release": micropy_version != "latest",
 }
 
 
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
+# needs_sphinx = '1.0'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.doctest',
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.todo',
-    'sphinx.ext.coverage',
+    "sphinx.ext.autodoc",
+    "sphinx.ext.doctest",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.todo",
+    "sphinx.ext.coverage",
+    "sphinxcontrib.jquery",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['templates']
+templates_path = ["templates"]
 
 # The suffix of source filenames.
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The encoding of source files.
-#source_encoding = 'utf-8-sig'
+# source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # General information about the project.
-project = 'MicroPython'
-copyright = '2014-2019, Damien P. George, Paul Sokolovsky, and contributors'
+project = "MicroPython"
+copyright = "- The MicroPython Documentation is Copyright © 2014-2023, Damien P. George, Paul Sokolovsky, and contributors"
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -74,45 +76,45 @@ copyright = '2014-2019, Damien P. George, Paul Sokolovsky, and contributors'
 #
 # We don't follow "The short X.Y version" vs "The full version, including alpha/beta/rc tags"
 # breakdown, so use the same version identifier for both to avoid confusion.
-version = release = '1.12'
+version = release = micropy_version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
-#language = None
+# language = None
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
-#today = ''
+# today = ''
 # Else, today_fmt is used as the format for a strftime call.
-#today_fmt = '%B %d, %Y'
+# today_fmt = '%B %d, %Y'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['build', '.venv']
+exclude_patterns = ["build", ".venv"]
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
-default_role = 'any'
+default_role = "any"
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
-#add_function_parentheses = True
+# add_function_parentheses = True
 
 # If true, the current module name will be prepended to all description
 # unit titles (such as .. function::).
-#add_module_names = True
+# add_module_names = True
 
 # If true, sectionauthor and moduleauthor directives will be shown in the
 # output. They are ignored by default.
-#show_authors = False
+# show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
 # A list of ignored prefixes for module index sorting.
-#modindex_common_prefix = []
+# modindex_common_prefix = []
 
 # If true, keep warnings as "system message" paragraphs in the built documents.
-#keep_warnings = False
+# keep_warnings = False
 
 # Global include files. Sphinx docs suggest using rst_epilog in preference
 # of rst_prolog, so we follow. Absolute paths below mean "from the base
@@ -124,153 +126,168 @@ rst_epilog = """
 # -- Options for HTML output ----------------------------------------------
 
 # on_rtd is whether we are on readthedocs.org
-on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+on_rtd = os.environ.get("READTHEDOCS", None) == "True"
 
 if not on_rtd:  # only import and set the theme if we're building docs locally
     try:
         import sphinx_rtd_theme
-        html_theme = 'sphinx_rtd_theme'
-        html_theme_path = [sphinx_rtd_theme.get_html_theme_path(), '.']
+
+        html_theme = "sphinx_rtd_theme"
+        html_theme_path = [sphinx_rtd_theme.get_html_theme_path(), "."]
     except:
-        html_theme = 'default'
-        html_theme_path = ['.']
+        html_theme = "default"
+        html_theme_path = ["."]
 else:
-    html_theme_path = ['.']
+    html_theme_path = ["."]
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
-#html_theme_options = {}
+# html_theme_options = {}
 
 # Add any paths that contain custom themes here, relative to this directory.
 # html_theme_path = ['.']
 
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".
-#html_title = None
+# html_title = None
 
 # A shorter title for the navigation bar.  Default is the same as html_title.
-#html_short_title = None
+# html_short_title = None
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = '../../logo/trans-logo.png'
+# html_logo = '../../logo/trans-logo.png'
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-html_favicon = 'static/favicon.ico'
+html_favicon = "static/favicon.ico"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['static']
+html_static_path = ["static"]
 
+# Add a custom CSS file for HTML generation
+html_css_files = [
+    "custom.css",
+]
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
 # directly to the root of the documentation.
-#html_extra_path = []
+# html_extra_path = []
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
-html_last_updated_fmt = '%d %b %Y'
+html_last_updated_fmt = "%d %b %Y"
 
 # If true, SmartyPants will be used to convert quotes and dashes to
 # typographically correct entities.
-#html_use_smartypants = True
+# html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
-#html_sidebars = {}
+# html_sidebars = {}
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.
 html_additional_pages = {"index": "topindex.html"}
 
 # If false, no module index is generated.
-#html_domain_indices = True
+# html_domain_indices = True
 
 # If false, no index is generated.
-#html_use_index = True
+# html_use_index = True
 
 # If true, the index is split into individual pages for each letter.
-#html_split_index = False
+# html_split_index = False
 
 # If true, links to the reST sources are added to the pages.
-#html_show_sourcelink = True
+# html_show_sourcelink = True
 
 # If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
-#html_show_sphinx = True
+# html_show_sphinx = True
 
 # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#html_show_copyright = True
+# html_show_copyright = True
 
 # If true, an OpenSearch description file will be output, and all pages will
 # contain a <link> tag referring to it.  The value of this option must be the
 # base URL from which the finished HTML is served.
-#html_use_opensearch = ''
+# html_use_opensearch = ''
 
 # This is the file name suffix for HTML files (e.g. ".xhtml").
-#html_file_suffix = None
+# html_file_suffix = None
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'MicroPythondoc'
+htmlhelp_basename = "MicroPythondoc"
 
 
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
-
-# The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
-
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
-# Include 3 levels of headers in PDF ToC
-'preamble': '\setcounter{tocdepth}{2}',
+    # The paper size ('letterpaper' or 'a4paper').
+    #'papersize': 'letterpaper',
+    # The font size ('10pt', '11pt' or '12pt').
+    #'pointsize': '10pt',
+    # Additional stuff for the LaTeX preamble.
+    #'preamble': '',
+    # Include 3 levels of headers in PDF ToC
+    "preamble": r"\setcounter{tocdepth}{2}",
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-  (master_doc, 'MicroPython.tex', 'MicroPython Documentation',
-   'Damien P. George, Paul Sokolovsky, and contributors', 'manual'),
+    (
+        master_doc,
+        "MicroPython.tex",
+        "MicroPython Documentation",
+        "Damien P. George, Paul Sokolovsky, and contributors",
+        "manual",
+    ),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
-#latex_logo = None
+# latex_logo = None
 
 # For "manual" documents, if this is true, then toplevel headings are parts,
 # not chapters.
-#latex_use_parts = False
+# latex_use_parts = False
 
 # If true, show page references after internal links.
-#latex_show_pagerefs = False
+# latex_show_pagerefs = False
 
 # If true, show URL addresses after external links.
-#latex_show_urls = False
+# latex_show_urls = False
 
 # Documents to append as an appendix to all manuals.
-#latex_appendices = []
+# latex_appendices = []
 
 # If false, no module index is generated.
-#latex_domain_indices = True
+# latex_domain_indices = True
 
+# Enable better Unicode support so that `make latexpdf` doesn't fail
+latex_engine = "xelatex"
 
 # -- Options for manual page output ---------------------------------------
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
-    ('index', 'micropython', 'MicroPython Documentation',
-     ['Damien P. George, Paul Sokolovsky, and contributors'], 1),
+    (
+        "index",
+        "micropython",
+        "MicroPython Documentation",
+        ["Damien P. George, Paul Sokolovsky, and contributors"],
+        1,
+    ),
 ]
 
 # If true, show URL addresses after external links.
-#man_show_urls = False
+# man_show_urls = False
 
 
 # -- Options for Texinfo output -------------------------------------------
@@ -279,23 +296,29 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-  (master_doc, 'MicroPython', 'MicroPython Documentation',
-   'Damien P. George, Paul Sokolovsky, and contributors', 'MicroPython', 'One line description of project.',
-   'Miscellaneous'),
+    (
+        master_doc,
+        "MicroPython",
+        "MicroPython Documentation",
+        "Damien P. George, Paul Sokolovsky, and contributors",
+        "MicroPython",
+        "One line description of project.",
+        "Miscellaneous",
+    ),
 ]
 
 # Documents to append as an appendix to all manuals.
-#texinfo_appendices = []
+# texinfo_appendices = []
 
 # If false, no module index is generated.
-#texinfo_domain_indices = True
+# texinfo_domain_indices = True
 
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
-#texinfo_show_urls = 'footnote'
+# texinfo_show_urls = 'footnote'
 
 # If true, do not generate a @detailmenu in the "Top" node's menu.
-#texinfo_no_detailmenu = False
+# texinfo_no_detailmenu = False
 
 
 # Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'python': ('https://docs.python.org/3.5', None)}
+intersphinx_mapping = {"python": ("https://docs.python.org/3.5", None)}

docs/develop/cmodules.rst

@@ -8,7 +8,8 @@ limitations with the Python environment, often due to an inability to access
 certain hardware resources or Python speed limitations.
 
 If your limitations can't be resolved with suggestions in :ref:`speed_python`,
-writing some or all of your module in C is a viable option.
+writing some or all of your module in C (and/or C++ if implemented for your port)
+is a viable option.
 
 If your module is designed to access or work with commonly available
 hardware or libraries please consider implementing it inside the MicroPython
@@ -17,7 +18,11 @@ If however you're targeting obscure or proprietary systems it may make
 more sense to keep this external to the main MicroPython repository.
 
 This chapter describes how to compile such external modules into the
-MicroPython executable or firmware image.
+MicroPython executable or firmware image.  Both Make and CMake build
+tools are supported, and when writing an external module it's a good idea to
+add the build files for both of these tools so the module can be used on all
+ports.  But when compiling a particular port you will only need to use one
+method of building, either Make or CMake.
 
 An alternative approach is to use :ref:`natmod` which allows writing custom C
 code that is placed in a .mpy file, which can be imported dynamically in to
@@ -29,7 +34,7 @@ Structure of an external C module
 
 A MicroPython user C module is a directory with the following files:
 
-* ``*.c`` and/or ``*.h`` source code files for your module.
+* ``*.c`` / ``*.cpp`` / ``*.h`` source code files for your module.
 
   These will typically include the low level functionality being implemented and
   the MicroPython binding functions to expose the functions and module(s).
@@ -44,137 +49,239 @@ A MicroPython user C module is a directory with the following files:
   in your ``micropython.mk`` to a local make variable,
   eg ``EXAMPLE_MOD_DIR := $(USERMOD_DIR)``
 
-  Your ``micropython.mk`` must add your modules C files relative to your
-  expanded copy of ``$(USERMOD_DIR)`` to ``SRC_USERMOD``, eg
-  ``SRC_USERMOD += $(EXAMPLE_MOD_DIR)/example.c``
+  Your ``micropython.mk`` must add your modules source files to the
+  ``SRC_USERMOD_C`` or ``SRC_USERMOD_LIB_C`` variables. The former will be
+  processed for ``MP_QSTR_`` and ``MP_REGISTER_MODULE`` definitions, the latter
+  will not (e.g. helpers and library code that isn't MicroPython-specific).
+  These paths should include your expanded copy of ``$(USERMOD_DIR)``, e.g.::
+
+    SRC_USERMOD_C += $(EXAMPLE_MOD_DIR)/modexample.c
+    SRC_USERMOD_LIB_C += $(EXAMPLE_MOD_DIR)/utils/algorithm.c
+
+  Similarly, use ``SRC_USERMOD_CXX`` and ``SRC_USERMOD_LIB_CXX`` for C++
+  source files.
+
+  If you have custom compiler options (like ``-I`` to add directories to search
+  for header files), these should be added to ``CFLAGS_USERMOD`` for C code
+  and to ``CXXFLAGS_USERMOD`` for C++ code.
+
+* ``micropython.cmake`` contains the CMake configuration for this module.
+
+  In ``micropython.cmake``, you may use ``${CMAKE_CURRENT_LIST_DIR}`` as the path to
+  the current module.
+
+  Your ``micropython.cmake`` should define an ``INTERFACE`` library and associate
+  your source files, compile definitions and include directories with it.
+  The library should then be linked to the ``usermod`` target.
+
+  .. code-block:: cmake
+
+      add_library(usermod_cexample INTERFACE)
+
+      target_sources(usermod_cexample INTERFACE
+          ${CMAKE_CURRENT_LIST_DIR}/examplemodule.c
+      )
+
+      target_include_directories(usermod_cexample INTERFACE
+          ${CMAKE_CURRENT_LIST_DIR}
+      )
+
+      target_link_libraries(usermod INTERFACE usermod_cexample)
 
-  If you have custom ``CFLAGS`` settings or include folders to define, these
-  should be added to ``CFLAGS_USERMOD``.
 
   See below for full usage example.
 
 
-Basic Example
+Basic example
 -------------
 
-This simple module named ``example`` provides a single function
-``example.add_ints(a, b)`` which adds the two integer args together and returns
-the result.
+The ``cexample`` module provides examples for a function and a class. The
+``cexample.add_ints(a, b)`` function adds two integer args together and returns
+the result. The ``cexample.Timer()`` type creates timers that can be used to
+measure the elapsed time since the object is instantiated.
 
-Directory::
+The module can be found in the MicroPython source tree
+`in the examples directory <https://github.com/micropython/micropython/tree/master/examples/usercmodule/cexample>`_
+and has a source file and a Makefile fragment with content as described above::
 
-    example/
-    ├── example.c
-    └── micropython.mk
+    micropython/
+    └──examples/
+       └──usercmodule/
+          └──cexample/
+             ├── examplemodule.c
+             ├── micropython.mk
+             └── micropython.cmake
 
 
-``example.c``
-
-.. code-block:: c
-
-    // Include required definitions first.
-    #include "py/obj.h"
-    #include "py/runtime.h"
-    #include "py/builtin.h"
-
-    // This is the function which will be called from Python as example.add_ints(a, b).
-    STATIC mp_obj_t example_add_ints(mp_obj_t a_obj, mp_obj_t b_obj) {
-        // Extract the ints from the micropython input objects
-        int a = mp_obj_get_int(a_obj);
-        int b = mp_obj_get_int(b_obj);
-
-        // Calculate the addition and convert to MicroPython object.
-        return mp_obj_new_int(a + b);
-    }
-    // Define a Python reference to the function above
-    STATIC MP_DEFINE_CONST_FUN_OBJ_2(example_add_ints_obj, example_add_ints);
-
-    // Define all properties of the example module.
-    // Table entries are key/value pairs of the attribute name (a string)
-    // and the MicroPython object reference.
-    // All identifiers and strings are written as MP_QSTR_xxx and will be
-    // optimized to word-sized integers by the build system (interned strings).
-    STATIC const mp_rom_map_elem_t example_module_globals_table[] = {
-        { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_example) },
-        { MP_ROM_QSTR(MP_QSTR_add_ints), MP_ROM_PTR(&example_add_ints_obj) },
-    };
-    STATIC MP_DEFINE_CONST_DICT(example_module_globals, example_module_globals_table);
-
-    // Define module object.
-    const mp_obj_module_t example_user_cmodule = {
-        .base = { &mp_type_module },
-        .globals = (mp_obj_dict_t*)&example_module_globals,
-    };
-
-    // Register the module to make it available in Python
-    MP_REGISTER_MODULE(MP_QSTR_example, example_user_cmodule, MODULE_EXAMPLE_ENABLED);
-
-
-``micropython.mk``
-
-.. code-block:: make
-
-    EXAMPLE_MOD_DIR := $(USERMOD_DIR)
-
-    # Add all C files to SRC_USERMOD.
-    SRC_USERMOD += $(EXAMPLE_MOD_DIR)/example.c
-
-    # We can add our module folder to include paths if needed
-    # This is not actually needed in this example.
-    CFLAGS_USERMOD += -I$(EXAMPLE_MOD_DIR)
-
-Finally you will need to define ``MODULE_EXAMPLE_ENABLED`` to 1. This
-can be done by adding ``CFLAGS_EXTRA=-DMODULE_EXAMPLE_ENABLED=1`` to
-the ``make`` command, or editing ``mpconfigport.h`` or
-``mpconfigboard.h`` to add
-
-.. code-block:: c
-
-    #define MODULE_EXAMPLE_ENABLED (1)
-
-Note that the exact method depends on the port as they have different
-structures. If not done correctly it will compile but importing will
-fail to find the module.
+Refer to the comments in these files for additional explanation.
+Next to the ``cexample`` module there's also ``cppexample`` which
+works in the same way but shows one way of mixing C and C++ code
+in MicroPython.
 
 
 Compiling the cmodule into MicroPython
 --------------------------------------
 
 To build such a module, compile MicroPython (see `getting started
-<https://github.com/micropython/micropython/wiki/Getting-Started>`_) with an
-extra ``make`` flag named ``USER_C_MODULES`` set to the directory containing
-all modules you want included (not to the module itself). For example:
+<https://github.com/micropython/micropython/wiki/Getting-Started>`_),
+applying 2 modifications:
 
+1. Set the build-time flag ``USER_C_MODULES`` to point to the modules
+   you want to include.  For ports that use Make this variable should be a
+   directory which is searched automatically for modules.  For ports that
+   use CMake this variable should be a file which includes the modules to
+   build.  See below for details.
 
-Directory::
+2. Enable the modules by setting the corresponding C preprocessor macro to
+   1.  This is only needed if the modules you are building are not
+   automatically enabled.
 
-    my_project/
-    ├── modules/
-    │   └──example/
-    │       ├──example.c
-    │       └──micropython.mk
-    └── micropython/
-        ├──ports/
-       ... ├──stm32/
-          ...
+For building the example modules which come with MicroPython,
+set ``USER_C_MODULES`` to the ``examples/usercmodule`` directory for Make,
+or to ``examples/usercmodule/micropython.cmake`` for CMake.
 
-Building for stm32 port:
+For example, here's how the to build the unix port with the example modules:
+
+.. code-block:: bash
+
+    cd micropython/ports/unix
+    make USER_C_MODULES=../../examples/usercmodule
+
+You may need to run ``make clean`` once at the start when including new
+user modules in the build.  The build output will show the modules found::
+
+    ...
+    Including User C Module from ../../examples/usercmodule/cexample
+    Including User C Module from ../../examples/usercmodule/cppexample
+    ...
+
+For a CMake-based port such as rp2, this will look a little different (note
+that CMake is actually invoked by ``make``):
+
+.. code-block:: bash
+
+    cd micropython/ports/rp2
+    make USER_C_MODULES=../../examples/usercmodule/micropython.cmake
+
+Again, you may need to run ``make clean`` first for CMake to pick up the
+user modules.  The CMake build output lists the modules by name::
+
+    ...
+    Including User C Module(s) from ../../examples/usercmodule/micropython.cmake
+    Found User C Module(s): usermod_cexample, usermod_cppexample
+    ...
+
+The contents of the top-level ``micropython.cmake`` can be used to control which
+modules are enabled.
+
+For your own projects it's more convenient to keep custom code out of the main
+MicroPython source tree, so a typical project directory structure will look
+like this::
+
+      my_project/
+      ├── modules/
+      │   ├── example1/
+      │   │   ├── example1.c
+      │   │   ├── micropython.mk
+      │   │   └── micropython.cmake
+      │   ├── example2/
+      │   │   ├── example2.c
+      │   │   ├── micropython.mk
+      │   │   └── micropython.cmake
+      │   └── micropython.cmake
+      └── micropython/
+          ├──ports/
+         ... ├──stm32/
+            ...
+
+When building with Make set ``USER_C_MODULES`` to the ``my_project/modules``
+directory.  For example, building the stm32 port:
 
 .. code-block:: bash
 
     cd my_project/micropython/ports/stm32
-    make USER_C_MODULES=../../../modules CFLAGS_EXTRA=-DMODULE_EXAMPLE_ENABLED=1 all
+    make USER_C_MODULES=../../../modules
+
+When building with CMake the top level ``micropython.cmake`` -- found directly
+in the ``my_project/modules`` directory -- should ``include`` all of the modules
+you want to have available:
+
+  .. code-block:: cmake
+
+      include(${CMAKE_CURRENT_LIST_DIR}/example1/micropython.cmake)
+      include(${CMAKE_CURRENT_LIST_DIR}/example2/micropython.cmake)
+
+Then build with:
+
+.. code-block:: bash
+
+    cd my_project/micropython/ports/esp32
+    make USER_C_MODULES=../../../../modules/micropython.cmake
+
+Note that the esp32 port needs the extra ``..`` for relative paths due to the
+location of its main ``CMakeLists.txt`` file.   You can also specify absolute
+paths to ``USER_C_MODULES``.
+
+All modules specified by the ``USER_C_MODULES`` variable (either found in this
+directory when using Make, or added via ``include`` when using CMake) will be
+compiled, but only those which are enabled will be available for importing.
+User modules are usually enabled by default (this is decided by the developer
+of the module), in which case there is nothing more to do than set ``USER_C_MODULES``
+as described above.
+
+If a module is not enabled by default then the corresponding C preprocessor macro
+must be enabled.  This macro name can be found by searching for the ``MP_REGISTER_MODULE``
+line in the module's source code (it usually appears at the end of the main source file).
+This macro should be surrounded by a ``#if X`` / ``#endif`` pair, and the configuration
+option ``X`` must be set to 1 using ``CFLAGS_EXTRA`` to make the module available.  If
+there is no ``#if X`` / ``#endif`` pair then the module is enabled by default.
+
+For example, the ``examples/usercmodule/cexample`` module is enabled by default so
+has the following line in its source code:
+
+  .. code-block:: c
+
+      MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule);
+
+Alternatively, to make this module disabled by default but selectable through
+a preprocessor configuration option, it would be:
+
+  .. code-block:: c
+
+      #if MODULE_CEXAMPLE_ENABLED
+      MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule);
+      #endif
+
+In this case the module is enabled by adding ``CFLAGS_EXTRA=-DMODULE_CEXAMPLE_ENABLED=1``
+to the ``make`` command, or editing ``mpconfigport.h`` or ``mpconfigboard.h`` to add
+
+  .. code-block:: c
+
+      #define MODULE_CEXAMPLE_ENABLED (1)
+
+Note that the exact method depends on the port as they have different
+structures.  If not done correctly it will compile but importing will
+fail to find the module.
 
 
 Module usage in MicroPython
 ---------------------------
 
-Once built into your copy of MicroPython, the module implemented
-in ``example.c`` above can now be accessed in Python just
-like any other builtin module, eg
+Once built into your copy of MicroPython, the module
+can now be accessed in Python just like any other builtin module, e.g.
 
 .. code-block:: python
 
-    import example
-    print(example.add_ints(1, 3))
+    import cexample
+    print(cexample.add_ints(1, 3))
     # should display 4
+
+.. code-block:: python
+
+    from cexample import Timer
+    from time import sleep_ms
+
+    watch = Timer()
+    sleep_ms(1000)
+    print(watch.time())
+    # should display approximately 1000

docs/develop/compiler.rst

@@ -0,0 +1,324 @@
+.. _compiler:
+
+The Compiler
+============
+
+The compilation process in MicroPython involves the following steps:
+
+* The lexer converts the stream of text that makes up a MicroPython program into tokens.
+* The parser then converts the tokens into an abstract syntax (parse tree).
+* Then bytecode or native code is emitted based on the parse tree.
+
+For purposes of this discussion we are going to add a simple language feature ``add1``
+that can be use in Python as:
+
+.. code-block:: bash
+
+    >>> add1 3
+    4
+    >>>
+
+The ``add1`` statement takes an integer as argument and adds ``1`` to it.
+
+Adding a grammar rule
+----------------------
+
+MicroPython's grammar is based on the `CPython grammar <https://docs.python.org/3.5/reference/grammar.html>`_
+and is defined in `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.
+This grammar is what is used to parse MicroPython source files.
+
+There are two macros you need to know to define a grammar rule: ``DEF_RULE`` and ``DEF_RULE_NC``.
+``DEF_RULE`` allows you to define a rule with an associated compile function,
+while ``DEF_RULE_NC`` has no compile (NC) function for it.
+
+A simple grammar definition with a compile function for our new ``add1`` statement
+looks like the following:
+
+.. code-block:: c
+
+   DEF_RULE(add1_stmt, c(add1_stmt), and(2), tok(KW_ADD1), rule(testlist))
+
+The second argument ``c(add1_stmt)`` is the corresponding compile function that should be implemented
+in ``py/compile.c`` to turn this rule into executable code.
+
+The third required argument can be ``or`` or ``and``. This specifies the number of nodes associated
+with a statement. For example, in this case, our ``add1`` statement is similar to ADD1 in assembly
+language. It takes one numeric argument. Therefore, the ``add1_stmt`` has two nodes associated with it.
+One node is for the statement itself, i.e the literal ``add1`` corresponding to ``KW_ADD1``,
+and the other for its argument, a ``testlist`` rule which is the top-level expression rule.
+
+.. note::
+   The ``add1`` rule here is just an example and not part of the standard
+   MicroPython grammar.
+
+The fourth argument in this example is the token associated with the rule, ``KW_ADD1``. This token should be
+defined in the lexer by editing ``py/lexer.h``.
+
+Defining the same rule without a compile function is achieved by using the ``DEF_RULE_NC`` macro
+and omitting the compile function argument:
+
+.. code-block:: c
+
+   DEF_RULE_NC(add1_stmt, and(2), tok(KW_ADD1), rule(testlist))
+
+The remaining arguments take on the same meaning. A rule without a compile function must
+be handled explicitly by all rules that may have this rule as a node. Such NC-rules are usually
+used to express sub-parts of a complicated grammar structure that cannot be expressed in a
+single rule.
+
+.. note::
+   The macros ``DEF_RULE`` and ``DEF_RULE_NC`` take other arguments. For an in-depth understanding of
+   supported parameters, see `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.
+
+Adding a lexical token
+----------------------
+
+Every rule defined in the grammar should have a token associated with it that is defined in ``py/lexer.h``.
+Add this token by editing the ``_mp_token_kind_t`` enum:
+
+.. code-block:: c
+   :emphasize-lines: 12
+
+   typedef enum _mp_token_kind_t {
+       ...
+       MP_TOKEN_KW_OR,
+       MP_TOKEN_KW_PASS,
+       MP_TOKEN_KW_RAISE,
+       MP_TOKEN_KW_RETURN,
+       MP_TOKEN_KW_TRY,
+       MP_TOKEN_KW_WHILE,
+       MP_TOKEN_KW_WITH,
+       MP_TOKEN_KW_YIELD,
+       MP_TOKEN_KW_ADD1,
+       ...
+   } mp_token_kind_t;
+
+Then also edit ``py/lexer.c`` to add the new keyword literal text:
+
+.. code-block:: c
+   :emphasize-lines: 12
+
+   STATIC const char *const tok_kw[] = {
+       ...
+       "or",
+       "pass",
+       "raise",
+       "return",
+       "try",
+       "while",
+       "with",
+       "yield",
+       "add1",
+       ...
+   };
+
+Notice the keyword is named depending on what you want it to be. For consistency, maintain the
+naming standard accordingly.
+
+.. note::
+   The order of these keywords in ``py/lexer.c`` must match the order of tokens in the enum
+   defined in ``py/lexer.h``.
+
+Parsing
+-------
+
+In the parsing stage the parser takes the tokens produced by the lexer and converts them to an abstract syntax tree (AST) or
+*parse tree*. The implementation for the parser is defined in `py/parse.c <https://github.com/micropython/micropython/blob/master/py/parse.c>`_.
+
+The parser also maintains a table of constants for use in different aspects of parsing, similar to what a
+`symbol table <https://steemit.com/programming/@drifter1/writing-a-simple-compiler-on-my-own-symbol-table-basic-structure>`_
+does.
+
+Several optimizations like `constant folding <http://compileroptimizations.com/category/constant_folding.htm>`_
+on integers for most operations e.g. logical, binary, unary, etc, and optimizing enhancements on parenthesis
+around expressions are performed during this phase, along with some optimizations on strings.
+
+It's worth noting that *docstrings* are discarded and not accessible to the compiler.
+Even optimizations like `string interning <https://en.wikipedia.org/wiki/String_interning>`_ are
+not applied to *docstrings*.
+
+Compiler passes
+---------------
+
+Like many compilers, MicroPython compiles all code to MicroPython bytecode or native code. The functionality
+that achieves this is implemented in `py/compile.c <https://github.com/micropython/micropython/blob/master/py/compile.c>`_.
+The most relevant method you should know about is this:
+
+.. code-block:: c
+
+   mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, bool is_repl) {
+       // Create a context for this module, and set its globals dict.
+       mp_module_context_t *context = m_new_obj(mp_module_context_t);
+       context->module.globals = mp_globals_get();
+
+       // Compile the input parse_tree to a raw-code structure.
+       mp_compiled_module_t cm;
+       cm.context = context;
+       mp_compile_to_raw_code(parse_tree, source_file, is_repl, &cm);
+
+       // Create and return a function object that executes the outer module.
+       return mp_make_function_from_raw_code(cm.rc, cm.context, NULL);
+   }
+
+The compiler compiles the code in four passes: scope, stack size, code size and emit.
+Each pass runs the same C code over the same AST data structure, with different things
+being computed each time based on the results of the previous pass.
+
+First pass
+~~~~~~~~~~
+
+In the first pass, the compiler learns about the known identifiers (variables) and
+their scope, being global, local, closed over, etc. In the same pass the emitter
+(bytecode or native code) also computes the number of labels needed for the emitted
+code.
+
+.. code-block:: c
+
+   // Compile pass 1.
+   comp->emit = emit_bc;
+   comp->emit_method_table = &emit_bc_method_table;
+
+   uint max_num_labels = 0;
+   for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
+       if (s->emit_options == MP_EMIT_OPT_ASM) {
+           compile_scope_inline_asm(comp, s, MP_PASS_SCOPE);
+       } else {
+           compile_scope(comp, s, MP_PASS_SCOPE);
+
+           // Check if any implicitly declared variables should be closed over.
+           for (size_t i = 0; i < s->id_info_len; ++i) {
+               id_info_t *id = &s->id_info[i];
+               if (id->kind == ID_INFO_KIND_GLOBAL_IMPLICIT) {
+                   scope_check_to_close_over(s, id);
+               }
+           }
+       }
+       ...
+   }
+
+Second and third passes
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The second and third passes involve computing the Python stack size and code size
+for the bytecode or native code. After the third pass the code size cannot change,
+otherwise jump labels will be incorrect.
+
+.. code-block:: c
+
+   for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
+       ...
+
+       // Pass 2: Compute the Python stack size.
+       compile_scope(comp, s, MP_PASS_STACK_SIZE);
+
+       // Pass 3: Compute the code size.
+       if (comp->compile_error == MP_OBJ_NULL) {
+           compile_scope(comp, s, MP_PASS_CODE_SIZE);
+       }
+
+       ...
+   }
+
+Just before pass two there is a selection for the type of code to be emitted, which can
+either be native or bytecode.
+
+.. code-block:: c
+
+   // Choose the emitter type.
+   switch (s->emit_options) {
+       case MP_EMIT_OPT_NATIVE_PYTHON:
+       case MP_EMIT_OPT_VIPER:
+           if (emit_native == NULL) {
+               emit_native = NATIVE_EMITTER(new)(&comp->compile_error, &comp->next_label, max_num_labels);
+           }
+           comp->emit_method_table = NATIVE_EMITTER_TABLE;
+           comp->emit = emit_native;
+           break;
+
+       default:
+           comp->emit = emit_bc;
+           comp->emit_method_table = &emit_bc_method_table;
+           break;
+   }
+
+The bytecode option is the default but something unique to note for the native
+code option is that there is another option via ``VIPER``. See the
+:ref:`Emitting native code <emitting_native_code>` section for more details on
+viper annotations.
+
+There is also support for *inline assembly code*, where assembly instructions are
+written as Python function calls but are emitted directly as the corresponding
+machine code. This assembler has only three passes (scope, code size, emit)
+and uses a different implementation, not the ``compile_scope`` function.
+See the `inline assembler tutorial <https://docs.micropython.org/en/latest/pyboard/tutorial/assembler.html#pyboard-tutorial-assembler>`_
+for more details.
+
+Fourth pass
+~~~~~~~~~~~
+
+The fourth pass emits the final code that can be executed, either bytecode in
+the virtual machine, or native code directly by the CPU.
+
+.. code-block:: c
+
+   for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
+       ...
+
+       // Pass 4: Emit the compiled bytecode or native code.
+       if (comp->compile_error == MP_OBJ_NULL) {
+           compile_scope(comp, s, MP_PASS_EMIT);
+       }
+   }
+
+Emitting bytecode
+-----------------
+
+Statements in Python code usually correspond to emitted bytecode, for example ``a + b``
+generates "push a" then "push b" then "binary op add". Some statements do not emit
+anything but instead affect other things like the scope of variables, for example
+``global a``.
+
+The implementation of a function that emits bytecode looks similar to this:
+
+.. code-block:: c
+
+   void mp_emit_bc_unary_op(emit_t *emit, mp_unary_op_t op) {
+       emit_write_bytecode_byte(emit, 0, MP_BC_UNARY_OP_MULTI + op);
+   }
+
+We use the unary operator expressions for an example here but the implementation
+details are similar for other statements/expressions. The method ``emit_write_bytecode_byte()``
+is a wrapper around the main function ``emit_get_cur_to_write_bytecode()`` that all
+functions must call to emit bytecode.
+
+.. _emitting_native_code:
+
+Emitting native code
+---------------------
+
+Similar to how bytecode is generated, there should be a corresponding function in ``py/emitnative.c`` for each
+code statement:
+
+.. code-block:: c
+
+   STATIC void emit_native_unary_op(emit_t *emit, mp_unary_op_t op) {
+        vtype_kind_t vtype;
+        emit_pre_pop_reg(emit, &vtype, REG_ARG_2);
+        if (vtype == VTYPE_PYOBJ) {
+            emit_call_with_imm_arg(emit, MP_F_UNARY_OP, op, REG_ARG_1);
+            emit_post_push_reg(emit, VTYPE_PYOBJ, REG_RET);
+        } else {
+            adjust_stack(emit, 1);
+            EMIT_NATIVE_VIPER_TYPE_ERROR(emit,
+                MP_ERROR_TEXT("unary op %q not implemented"), mp_unary_op_method_name[op]);
+        }
+   }
+
+The difference here is that we have to handle *viper typing*. Viper annotations allow
+us to handle more than one type of variable. By default all variables are Python objects,
+but with viper a variable can also be declared as a machine-typed variable like a native
+integer or pointer. Viper can be thought of as a superset of Python, where normal Python
+objects are handled as usual, while native machine variables are handled in an optimised
+way by using direct machine instructions for the operations. Viper typing may break
+Python equivalence because, for example, integers become native integers and can overflow
+(unlike Python integers which extend automatically to arbitrary precision).

docs/develop/extendingmicropython.rst

@@ -0,0 +1,18 @@
+.. _extendingmicropython:
+
+Extending MicroPython in C
+==========================
+
+This chapter describes options for implementing additional functionality in C, but from code
+written outside of the main MicroPython repository. The first approach is useful for building
+your own custom firmware with some project-specific additional modules or functions that can
+be accessed from Python. The second approach is for building modules that can be loaded at runtime.
+
+Please see the :ref:`library section <internals_library>` for more information on building core modules that
+live in the main MicroPython repository.
+
+.. toctree::
+   :maxdepth: 3
+
+   cmodules.rst
+   natmod.rst

docs/develop/gettingstarted.rst

@@ -0,0 +1,330 @@
+.. _gettingstarted:
+
+Getting Started
+===============
+
+This guide covers a step-by-step process on setting up version control, obtaining and building
+a copy of the source code for a port, building the documentation, running tests, and a description of the
+directory structure of the MicroPython code base.
+
+Source control with git
+-----------------------
+
+MicroPython is hosted on `GitHub <https://github.com/micropython/micropython>`_ and uses
+`Git <https://git-scm.com>`_ for source control. The workflow is such that
+code is pulled and pushed to and from the main repository. Install the respective version
+of Git for your operating system to follow through the rest of the steps.
+
+.. note::
+   For a reference on the installation instructions, please refer to
+   the `Git installation instructions <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`_.
+   Learn about the basic git commands in this `Git Handbook <https://guides.github.com/introduction/git-handbook/>`_
+   or any other sources on the internet.
+
+.. note::
+   A .git-blame-ignore-revs file is included which avoids the output of git blame getting cluttered
+   by commits which are only for formatting code but have no functional changes. See `git blame documentation
+   <https://git-scm.com/docs/git-blame#Documentation/git-blame.txt---ignore-revltrevgt>`_ on how to use this.
+
+Get the code
+------------
+
+It is recommended that you maintain a fork of the MicroPython repository for your development purposes.
+The process of obtaining the source code includes the following:
+
+#. Fork the repository https://github.com/micropython/micropython
+#. You will now have a fork at <https://github.com/<your-user-name>/micropython>.
+#. Clone the forked repository using the following command:
+
+.. code-block:: bash
+
+   $ git clone https://github.com/<your-user-name>/micropython
+
+Then, `configure the remote repositories <https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes>`_ to be able to
+collaborate on the MicroPython project.
+
+Configure remote upstream:
+
+.. code-block:: bash
+
+   $ cd micropython
+   $ git remote add upstream https://github.com/micropython/micropython
+
+It is common to configure ``upstream`` and ``origin`` on a forked repository
+to assist with sharing code changes. You can maintain your own mapping but
+it is recommended that ``origin`` maps to your fork and ``upstream`` to the main
+MicroPython repository.
+
+After the above configuration, your setup should be similar to this:
+
+.. code-block:: bash
+
+   $ git remote -v
+   origin       https://github.com/<your-user-name>/micropython (fetch)
+   origin       https://github.com/<your-user-name>/micropython (push)
+   upstream     https://github.com/micropython/micropython (fetch)
+   upstream     https://github.com/micropython/micropython (push)
+
+You should now have a copy of the source code. By default, you are pointing
+to the master branch. To prepare for further development, it is recommended
+to work on a development branch.
+
+.. code-block:: bash
+
+    $ git checkout -b dev-branch
+
+You can give it any name. You will have to compile MicroPython whenever you change
+to a different branch.
+
+Compile and build the code
+--------------------------
+
+When compiling MicroPython, you compile a specific :term:`port`, usually
+targeting a specific :ref:`board <glossary>`. Start by installing the required dependencies.
+Then build the MicroPython cross-compiler before you can successfully compile and build.
+This applies specifically when using Linux to compile.
+The Windows instructions are provided in a later section.
+
+.. _required_dependencies:
+
+Required dependencies
+~~~~~~~~~~~~~~~~~~~~~
+
+Install the required dependencies for Linux:
+
+.. code-block:: bash
+
+   $ sudo apt-get install build-essential libffi-dev git pkg-config
+
+For the stm32 port, the ARM cross-compiler is required:
+
+.. code-block:: bash
+
+   $ sudo apt-get install gcc-arm-none-eabi libnewlib-arm-none-eabi
+
+See the `ARM GCC
+toolchain <https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads>`_
+for the latest details.
+
+Python is also required. Python 2 is supported for now, but we recommend using Python 3.
+Check that you have Python available on your system:
+
+.. code-block:: bash
+
+   $ python3
+   Python 3.5.0 (default, Jul 17 2020, 14:04:10) 
+   [GCC 5.4.0 20160609] on linux
+   Type "help", "copyright", "credits" or "license" for more information.
+   >>> 
+
+All supported ports have different dependency requirements, see their respective
+`readme files <https://github.com/micropython/micropython/tree/master/ports>`_.
+
+Building the MicroPython cross-compiler
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Almost all ports require building ``mpy-cross`` first to perform pre-compilation
+of Python code that will be included in the port firmware:
+
+.. code-block:: bash
+
+   $ cd mpy-cross
+   $ make
+
+.. note::
+   Note that, ``mpy-cross`` must be built for the host architecture
+   and not the target architecture.
+
+If it built successfully, you should see a message similar to this:
+
+.. code-block:: bash
+
+   LINK mpy-cross
+      text          data    bss     dec     hex filename
+    279328          776     880  280984   44998 mpy-cross
+
+.. note::
+
+   Use ``make -C mpy-cross`` to build the cross-compiler in one statement
+   without moving to the ``mpy-cross`` directory otherwise, you will need
+   to do ``cd ..`` for the next steps.
+
+Building the Unix port of MicroPython
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Unix port is a version of MicroPython that runs on Linux, macOS, and other Unix-like operating systems.
+It's extremely useful for developing MicroPython as it avoids having to deploy your code to a device to test it.
+In many ways, it works a lot like CPython's python binary.
+
+To build for the Unix port, make sure all Linux related dependencies are installed as detailed in the
+required dependencies section. See the :ref:`required_dependencies`
+to make sure that all dependencies are installed for this port. Also, make sure you have a working
+environment for ``gcc`` and ``GNU make``. Ubuntu 20.04 has been used for the example
+below but other unixes ought to work with little modification:
+
+.. code-block:: bash
+
+   $ gcc --version
+   gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
+   Copyright (C) 2019 Free Software Foundation, Inc.
+   This is free software; see the source for copying conditions.  There is NO
+   warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.then build:
+
+.. code-block:: bash
+
+   $ cd ports/unix
+   $ make submodules
+   $ make
+
+If MicroPython built correctly, you should see the following:
+
+.. code-block:: bash
+
+   LINK micropython
+      text         data     bss     dec     hex filename
+    412033         5680    2496  420209   66971 micropython
+
+Now run it:
+
+.. code-block:: bash
+
+   $ ./micropython
+   MicroPython v1.13-38-gc67012d-dirty on 2020-09-13; linux version
+   Use Ctrl-D to exit, Ctrl-E for paste mode
+   >>> print("hello world")
+   hello world
+   >>>
+
+Building the Windows port
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Windows port includes a Visual Studio project file micropython.vcxproj that you can use to build micropython.exe.
+It can be opened in Visual Studio or built from the command line using msbuild. Alternatively, it can be built using mingw,
+either in Windows with Cygwin, or on Linux.
+See `windows port documentation <https://github.com/micropython/micropython/tree/master/ports/windows>`_ for more information.
+
+Building the STM32 port
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Like the Unix port, you need to install some required dependencies
+as detailed in the :ref:`required_dependencies` section, then build:
+
+.. code-block:: bash
+
+   $ cd ports/stm32
+   $ make submodules
+   $ make
+
+Please refer to the `stm32 documentation <https://github.com/micropython/micropython/tree/master/ports/stm32>`_
+for more details on flashing the firmware.
+
+.. note::
+   See the :ref:`required_dependencies` to make sure that all dependencies are installed for this port.
+   The cross-compiler is needed. ``arm-none-eabi-gcc`` should also be in the $PATH or specified manually
+   via CROSS_COMPILE, either by setting the environment variable or in the ``make`` command line arguments.
+
+You can also specify which board to use:
+
+.. code-block:: bash
+
+   $ cd ports/stm32
+   $ make BOARD=<board> submodules
+   $ make BOARD=<board>
+
+See `ports/stm32/boards <https://github.com/micropython/micropython/tree/master/ports/stm32/boards>`_
+for the available boards. e.g. "PYBV11" or "NUCLEO_WB55".
+
+Building the documentation
+--------------------------
+
+MicroPython documentation is created using ``Sphinx``. If you have already
+installed Python, then install ``Sphinx`` using ``pip``. It is recommended
+that you use a virtual environment:
+
+.. code-block:: bash
+
+   $ python3 -m venv env
+   $ source env/bin/activate
+   $ pip install -r docs/requirements.txt
+
+Navigate to the ``docs`` directory:
+
+.. code-block:: bash
+
+   $ cd docs
+
+Build the docs:
+
+.. code-block:: bash
+
+   $ make html
+
+Open ``docs/build/html/index.html`` in your browser to view the docs locally. Refer to the
+documentation on `importing your documentation
+<https://docs.readthedocs.io/en/stable/intro/import-guide.html>`_ to use Read the Docs.
+
+Running the tests
+-----------------
+
+To run all tests in the test suite on the Unix port use:
+
+.. code-block:: bash
+
+   $ cd ports/unix
+   $ make test
+
+To run a selection of tests on a board/device connected over USB use:
+
+.. code-block:: bash
+
+   $ cd tests
+   $ ./run-tests.py --target minimal --device /dev/ttyACM0
+
+See also :ref:`writingtests`.
+
+Folder structure
+----------------
+
+There are a couple of directories to take note of in terms of where certain implementation details
+are. The following is a break down of the top-level folders in the source code.
+
+py
+
+  Contains the compiler, runtime, and core library implementation.
+
+mpy-cross
+
+  Has the MicroPython cross-compiler which pre-compiles the Python scripts to bytecode.
+
+ports
+
+  Code for all the versions of MicroPython for the supported ports.
+
+lib
+
+  Low-level C libraries used by any port which are mostly 3rd-party libraries.
+
+drivers
+
+  Has drivers for specific hardware and intended to work across multiple ports.
+
+extmod
+
+  Contains a C implementation of more non-core modules.
+
+docs
+
+  Has the standard documentation found at https://docs.micropython.org/.
+
+tests
+
+  An implementation of the test suite.
+
+tools
+
+  Contains scripts used by the build and CI process, as well as user tools such
+  as ``pyboard.py`` and ``mpremote``.
+
+examples
+
+  Example code for building MicroPython as a library as well as native modules.

docs/develop/index.rst

@@ -1,14 +1,26 @@
-Developing and building MicroPython
-===================================
+MicroPython Internals
+=====================
 
-This chapter describes some options for extending MicroPython in C. Note
-that it doesn't aim to be a complete guide for developing with MicroPython.
-See the `getting started guide
-<https://github.com/micropython/micropython/wiki/Getting-Started>`_ for further information.
+This chapter covers a tour of MicroPython from the perspective of a developer, contributing
+to MicroPython. It acts as a comprehensive resource on the implementation details of MicroPython
+for both novice and expert contributors.
+
+Development around MicroPython usually involves modifying the core runtime, porting or
+maintaining a new library. This guide describes at great depth, the implementation
+details of MicroPython including a getting started guide, compiler internals, porting
+MicroPython to a new platform and implementing a core MicroPython library.
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 3
 
-   cmodules.rst
+   gettingstarted.rst
+   writingtests.rst
+   compiler.rst
+   memorymgt.rst
+   library.rst
+   optimizations.rst
    qstr.rst
-   natmod.rst
+   maps.rst
+   publiccapi.rst
+   extendingmicropython.rst
+   porting.rst

docs/develop/library.rst

@@ -0,0 +1,86 @@
+.. _internals_library:
+
+Implementing a Module
+=====================
+
+This chapter details how to implement a core module in MicroPython.
+MicroPython modules can be one of the following:
+
+- Built-in module: A general module that is be part of the MicroPython repository.
+- User module: A module that is useful for your specific project that you maintain
+  in your own repository or private codebase.
+- Dynamic module: A module that can be deployed and imported at runtime to your device.
+
+A module in MicroPython can be implemented in one of the following locations:
+
+- py/: A core library that mirrors core CPython functionality.
+- extmod/: A CPython or MicroPython-specific module that is shared across multiple ports.
+- ports/<port>/: A port-specific module.
+
+.. note::
+   This chapter describes modules implemented in ``py/`` or core modules.
+   See :ref:`extendingmicropython` for details on implementing an external module.
+   For details on port-specific modules, see :ref:`porting_to_a_board`.
+
+Implementing a core module
+--------------------------
+
+Like CPython, MicroPython has core builtin modules that can be accessed through import statements.
+An example is the ``gc`` module discussed in :ref:`memorymanagement`.
+
+.. code-block:: bash
+
+   >>> import gc
+   >>> gc.enable()
+   >>>
+
+MicroPython has several other builtin standard/core modules like ``io``, ``array`` etc.
+Adding a new core module involves several modifications.
+
+First, create the ``C`` file in the ``py/`` directory. In this example we are adding a
+hypothetical new module ``subsystem`` in the file ``modsubsystem.c``:
+
+.. code-block:: c
+
+   #include "py/builtin.h"
+   #include "py/runtime.h"
+
+   #if MICROPY_PY_SUBSYSTEM
+
+   // info()
+   STATIC mp_obj_t py_subsystem_info(void) {
+       return MP_OBJ_NEW_SMALL_INT(42);
+   }
+   MP_DEFINE_CONST_FUN_OBJ_0(subsystem_info_obj, py_subsystem_info);
+
+   STATIC const mp_rom_map_elem_t mp_module_subsystem_globals_table[] = {
+       { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_subsystem) },
+       { MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&subsystem_info_obj) },
+   };
+   STATIC MP_DEFINE_CONST_DICT(mp_module_subsystem_globals, mp_module_subsystem_globals_table);
+
+   const mp_obj_module_t mp_module_subsystem = {
+       .base = { &mp_type_module },
+       .globals = (mp_obj_dict_t *)&mp_module_subsystem_globals,
+   };
+
+   MP_REGISTER_MODULE(MP_QSTR_subsystem, mp_module_subsystem);
+
+   #endif
+
+The implementation includes a definition of all functions related to the module and adds the
+functions to the module's global table in ``mp_module_subsystem_globals_table``. It also
+creates the module object with ``mp_module_subsystem``.  The module is then registered with
+the wider system via the ``MP_REGISTER_MODULE`` macro.
+
+After building and running the modified MicroPython, the module should now be importable:
+
+.. code-block:: bash
+
+   >>> import subsystem
+   >>> subsystem.info()
+   42
+   >>>
+
+Our ``info()`` function currently returns just a single number but can be extended
+to do anything.  Similarly, more functions can be added to this new module.

docs/develop/maps.rst

@@ -0,0 +1,63 @@
+.. _maps:
+
+Maps and Dictionaries
+=====================
+
+MicroPython dictionaries and maps use techniques called open addressing and linear probing.
+This chapter details both of these methods.
+
+Open addressing
+---------------
+
+`Open addressing <https://en.wikipedia.org/wiki/Open_addressing>`_ is used to resolve collisions.
+Collisions are very common occurrences and happen when two items happen to hash to the same
+slot or location. For example, given a hash setup as this:
+
+.. image:: img/collision.png
+
+If there is a request to fill slot ``0`` with ``70``, since the slot ``0`` is not empty, open addressing
+finds the next available slot in the dictionary to service this request. This sequential search for an alternate
+location is called *probing*. There are several sequence probing algorithms but MicroPython uses
+linear probing that is described in the next section.
+
+Linear probing
+--------------
+
+Linear probing is one of the methods for finding an available address or slot in a dictionary. In MicroPython,
+it is used with open addressing. To service the request described above, unlike other probing algorithms,
+linear probing assumes a fixed interval of ``1`` between probes. The request will therefore be serviced by
+placing the item in the next free slot which is slot ``4`` in our example:
+
+.. image:: img/linprob.png
+
+The same methods i.e open addressing and linear probing are used to search for an item in a dictionary.
+Assume we want to search for the data item ``33``. The computed hash value will be 2. Looking at slot 2
+reveals ``33``, at this point, we return ``True``. Searching for ``70`` is quite different as there was a
+collision at the time of insertion. Therefore computing the hash value is ``0`` which is currently
+holding ``44``. Instead of simply returning ``False``, we perform a sequential search starting at point
+``1`` until the item ``70`` is found or we encounter a free slot. This is the general way of performing
+look-ups in hashes:
+
+.. code-block:: c
+
+   // not yet found, keep searching in this table
+   pos = (pos + 1) % set->alloc;
+
+   if (pos == start_pos) {
+       // search got back to starting position, so index is not in table
+       if (lookup_kind & MP_MAP_LOOKUP_ADD_IF_NOT_FOUND) {
+           if (avail_slot != NULL) {
+               // there was an available slot, so use that
+               set->used++;
+               *avail_slot = index;
+               return index;
+           } else {
+               // not enough room in table, rehash it
+               mp_set_rehash(set);
+               // restart the search for the new element
+               start_pos = pos = hash % set->alloc;
+           }
+       }
+   } else {
+        return MP_OBJ_NULL;
+   }

docs/develop/memorymgt.rst

@@ -0,0 +1,141 @@
+.. _memorymanagement:
+
+Memory Management
+=================
+
+Unlike programming languages such as C/C++, MicroPython hides memory management
+details from the developer by supporting automatic memory management.
+Automatic memory management is a technique used by operating systems or applications to automatically manage
+the allocation and deallocation of memory. This eliminates challenges such as forgetting to
+free the memory allocated to an object. Automatic memory management also avoids the critical issue of using memory
+that is already released. Automatic memory management takes many forms, one of them being
+garbage collection (GC).
+
+The garbage collector usually has two responsibilities;
+
+#. Allocate new objects in available memory.
+#. Free unused memory.
+
+There are many GC algorithms but MicroPython uses the
+`Mark and Sweep <https://en.wikipedia.org/wiki/Tracing_garbage_collection#Basic_algorithm>`_
+policy for managing memory. This algorithm has a mark phase that traverses the heap marking all
+live objects while the sweep phase goes through the heap reclaiming all unmarked objects.
+
+Garbage collection functionality in MicroPython is available through the ``gc`` built-in
+module:
+
+.. code-block:: bash
+
+   >>> x = 5
+   >>> x
+   5
+   >>> import gc
+   >>> gc.enable()
+   >>> gc.mem_alloc()
+   1312
+   >>> gc.mem_free()
+   2071392
+   >>> gc.collect()
+   19
+   >>> gc.disable()
+   >>>
+
+Even when ``gc.disable()`` is invoked, collection can be triggered with ``gc.collect()``.
+
+The object model
+----------------
+
+All MicroPython objects are referred to by the ``mp_obj_t`` data type.
+This is usually word-sized (i.e. the same size as a pointer on the target architecture),
+and can be typically 32-bit (STM32, nRF, ESP32, Unix x86) or 64-bit (Unix x64).
+It can also be greater than a word-size for certain object representations, for
+example ``OBJ_REPR_D`` has a 64-bit sized ``mp_obj_t`` on a 32-bit architecture.
+
+An ``mp_obj_t`` represents a MicroPython object, for example an integer, float, type, dict or
+class instance. Some objects, like booleans and small integers, have their value stored directly
+in the ``mp_obj_t`` value and do not require additional memory. Other objects have their value
+store elsewhere in memory (for example on the garbage-collected heap) and their ``mp_obj_t`` contains
+a pointer to that memory. A portion of ``mp_obj_t`` is the tag which tells what type of object it is.
+
+See ``py/mpconfig.h`` for the specific details of the available representations.
+
+**Pointer tagging**
+
+Because pointers are word-aligned, when they are stored in an ``mp_obj_t`` the
+lower bits of this object handle will be zero.  For example on a 32-bit architecture
+the lower 2 bits will be zero:
+
+``********|********|********|******00``
+
+These bits are reserved for purposes of storing a tag. The tag stores extra information as
+opposed to introducing a new field to store that information in the object, which may be
+inefficient.  In MicroPython the tag tells if we are dealing with a small integer, interned
+(small) string or a concrete object, and different semantics apply to each of these.
+
+For small integers the mapping is this:
+
+``********|********|********|*******1``
+
+Where the asterisks hold the actual integer value.  For an interned string or an immediate
+object (e.g. ``True``) the layout of the ``mp_obj_t`` value is, respectively:
+
+``********|********|********|*****010``
+
+``********|********|********|*****110``
+
+While a concrete object that is none of the above takes the form:
+
+``********|********|********|******00``
+
+The stars here correspond to the address of the concrete object in memory.
+
+Allocation of objects
+----------------------
+
+The value of a small integer is stored directly in the ``mp_obj_t`` and will be
+allocated in-place, not on the heap or elsewhere.  As such, creation of small
+integers does not affect the heap.  Similarly for interned strings that already have
+their textual data stored elsewhere, and immediate values like ``None``, ``False``
+and ``True``.
+
+Everything else which is a concrete object is allocated on the heap and its object structure is such that
+a field is reserved in the object header to store the type of the object.
+
+.. code-block:: bash
+
+    +++++++++++
+    +         +
+    + type    + object header
+    +         +
+    +++++++++++
+    +         + object items
+    +         +
+    +         +
+    +++++++++++
+
+The heap's smallest unit of allocation is a block, which is four machine words in
+size (16 bytes on a 32-bit machine, 32 bytes on a 64-bit machine).
+Another structure also allocated on the heap tracks the allocation of
+objects in each block. This structure is called a *bitmap*.
+
+.. image:: img/bitmap.png
+
+The bitmap tracks whether a block is "free" or "in use" and use two bits to track this state
+for each block.
+
+The mark-sweep garbage collector manages the objects allocated on the heap, and also
+utilises the bitmap to mark objects that are still in use.
+See `py/gc.c <https://github.com/micropython/micropython/blob/master/py/gc.c>`_
+for the full implementation of these details.
+
+**Allocation: heap layout**
+
+The heap is arranged such that it consists of blocks in pools. A block
+can have different properties:
+
+- *ATB(allocation table byte):* If set, then the block is a normal block
+- *FREE:* Free block
+- *HEAD:* Head of a chain of blocks
+- *TAIL:* In the tail of a chain of blocks
+- *MARK :* Marked head block
+- *FTB(finaliser table byte):* If set, then the block has a finaliser

docs/develop/natmod.rst

@@ -21,7 +21,8 @@ language which can be compiled to stand-alone machine code can be put into a
 
 A native .mpy module is built using the ``mpy_ld.py`` tool, which is found in the
 ``tools/`` directory of the project.  This tool takes a set of object files
-(.o files) and links them together to create a native .mpy files.
+(.o files) and links them together to create a native .mpy files.  It requires
+CPython 3 and the library pyelftools v0.25 or greater.
 
 Supported features and limitations
 ----------------------------------
@@ -33,6 +34,7 @@ options for the ``ARCH`` variable, see below):
 
 * ``x86`` (32 bit)
 * ``x64`` (64 bit x86)
+* ``armv6m`` (ARM Thumb, eg Cortex-M0)
 * ``armv7m`` (ARM Thumb 2, eg Cortex-M3)
 * ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7)
 * ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7)
@@ -67,6 +69,19 @@ The known limitations are:
 So, if your C code has writable data, make sure the data is defined globally,
 without an initialiser, and only written to within functions.
 
+Linker limitation: the native module is not linked against the symbol table of the
+full MicroPython firmware.  Rather, it is linked against an explicit table of exported
+symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware
+build time.  It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system
+function, for example.
+
+New symbols can be added to the end of the table and the firmware rebuilt.
+The symbols also need to be added to ``tools/mpy_ld.py``'s ``fun_table`` dict in the
+same location.  This allows ``mpy_ld.py`` to be able to pick the new symbols up and
+provide relocations for them when the mpy is imported.  Finally, if the symbol is a
+function, a macro or stub should be added to ``py/dynruntime.h`` to make it easy to
+call the function.
+
 Defining a native module
 ------------------------
 
@@ -152,12 +167,12 @@ The file ``Makefile`` contains:
     MPY_DIR = ../../..
 
     # Name of module
-    MOD = features0
+    MOD = factorial
 
     # Source files (.c or .py)
-    SRC = features0.c
+    SRC = factorial.c
 
-    # Architecture to build for (x86, x64, armv7m, xtensa, xtensawin)
+    # Architecture to build for (x86, x64, armv6m, armv7m, xtensa, xtensawin)
     ARCH = x64
 
     # Include to get the rules for compiling and linking the module
@@ -166,6 +181,14 @@ The file ``Makefile`` contains:
 Compiling the module
 --------------------
 
+The prerequisite tools needed to build a native .mpy file are:
+
+* The MicroPython repository (at least the ``py/`` and ``tools/`` directories).
+* CPython 3, and the library pyelftools (eg ``pip install 'pyelftools>=0.25'``).
+* GNU make.
+* A C compiler for the target architecture (if C source is used).
+* Optionally ``mpy-cross``, built from the MicroPython repository (if .py source is used).
+
 Be sure to select the correct ``ARCH`` for the target you are going to run on.
 Then build with::
 
@@ -180,7 +203,7 @@ Module usage in MicroPython
 
 Once the module is built there should be a file called ``factorial.mpy``.  Copy
 this so it is accessible on the filesystem of your MicroPython system and can be
-found in the import path.  The module con now be accessed in Python just like any
+found in the import path.  The module can now be accessed in Python just like any
 other module, for example::
 
     import factorial

docs/develop/optimizations.rst

@@ -0,0 +1,74 @@
+.. _optimizations:
+
+Optimizations
+=============
+
+MicroPython uses several optimizations to save RAM but also ensure the efficient
+execution of programs. This chapter discusses some of these optimizations.
+
+.. note::
+   :ref:`qstr` and :ref:`maps` details other optimizations on strings and
+   dictionaries.
+
+Frozen bytecode
+---------------
+
+When MicroPython loads Python code from the filesystem, it first has to parse the file into
+a temporary in-memory representation, and then generate bytecode for execution, both of which
+are stored in the heap (in RAM). This can lead to significant amounts of memory being used.
+The MicroPython cross compiler can be used to generate
+a ``.mpy`` file, containing the pre-compiled bytecode for a Python module. This will still
+be loaded into RAM, but it avoids the additional overhead of the parsing stage.
+
+As a further optimisation, the pre-compiled bytecode from a ``.mpy`` file can be "frozen"
+into the firmware image as part of the main firmware compilation process, which means that
+the bytecode will be executed from ROM. This can lead to a significant memory saving, and
+reduce heap fragmentation.
+
+See :ref:`manifest` for more information.
+
+Variables
+---------
+
+MicroPython processes local and global variables differently. Global variables
+are stored and looked up from a global dictionary that is allocated on the heap
+(note that each module has its own separate dict, so separate namespace).
+Local variables on the other hand are are stored on the Python value stack, which may
+live on the C stack or on the heap.  They are accessed directly by their offset
+within the Python stack, which is more efficient than a global lookup in a dict.
+
+The length of global variable names also affects how much RAM is used as identifiers
+are stored in RAM. The shorter the identifier, the less memory is used.
+
+The other aspect is that ``const`` variables that start with an underscore are treated as
+proper constants and are not allocated or added in a dictionary, hence saving some memory.
+These variables use ``const()`` from the MicroPython library. Therefore:
+
+.. code-block:: python
+
+    from micropython import const
+
+    X = const(1)
+    _Y = const(2)
+    foo(X, _Y)
+
+Compiles to:
+
+.. code-block:: python
+
+    X = 1
+    foo(1, 2)
+
+Allocation of memory
+--------------------
+
+Most of the common MicroPython constructs are not allocated on the heap.
+However the following are:
+
+- Dynamic data structures like lists, mappings, etc;
+- Functions, classes and object instances;
+- imports; and
+- First-time assignment of global variables (to create the slot in the global dict).
+
+For a detailed discussion on a more user-centric perspective on optimization,
+see `Maximising MicroPython speed <https://docs.micropython.org/en/latest/reference/speed_python.html>`_

docs/develop/porting.rst

@@ -0,0 +1,305 @@
+.. _porting_to_a_board:
+
+Porting MicroPython
+===================
+
+The MicroPython project contains several ports to different microcontroller families and
+architectures. The project repository has a `ports <https://github.com/micropython/micropython/tree/master/ports>`_
+directory containing a subdirectory for each supported port.
+
+A port will typically contain definitions for multiple "boards", each of which is a specific piece of
+hardware that that port can run on, e.g. a development kit or device.
+
+The `minimal port <https://github.com/micropython/micropython/tree/master/ports/minimal>`_ is
+available as a simplified reference implementation of a MicroPython port.  It can run on both the
+host system and an STM32F4xx MCU.
+
+In general, starting a port requires:
+
+- Setting up the toolchain (configuring Makefiles, etc).
+- Implementing boot configuration and CPU initialization.
+- Initialising basic drivers required for development and debugging (e.g. GPIO, UART).
+- Performing the board-specific configurations.
+- Implementing the port-specific modules.
+
+Minimal MicroPython firmware
+----------------------------
+
+The best way to start porting MicroPython to a new board is by integrating a minimal
+MicroPython interpreter.  For this walkthrough, create a subdirectory for the new
+port in the ``ports`` directory:
+
+.. code-block:: bash
+
+   $ cd ports
+   $ mkdir example_port
+
+The basic MicroPython firmware is implemented in the main port file, e.g ``main.c``:
+
+.. code-block:: c
+
+   #include "py/builtin.h"
+   #include "py/compile.h"
+   #include "py/gc.h"
+   #include "py/mperrno.h"
+   #include "py/stackctrl.h"
+   #include "shared/runtime/gchelper.h"
+   #include "shared/runtime/pyexec.h"
+
+   // Allocate memory for the MicroPython GC heap.
+   static char heap[4096];
+
+   int main(int argc, char **argv) {
+       // Initialise the MicroPython runtime.
+       mp_stack_ctrl_init();
+       gc_init(heap, heap + sizeof(heap));
+       mp_init();
+
+       // Start a normal REPL; will exit when ctrl-D is entered on a blank line.
+       pyexec_friendly_repl();
+
+       // Deinitialise the runtime.
+       gc_sweep_all();
+       mp_deinit();
+       return 0;
+   }
+
+   // Handle uncaught exceptions (should never be reached in a correct C implementation).
+   void nlr_jump_fail(void *val) {
+       for (;;) {
+       }
+   }
+
+   // Do a garbage collection cycle.
+   void gc_collect(void) {
+       gc_collect_start();
+       gc_helper_collect_regs_and_stack();
+       gc_collect_end();
+   }
+
+   // There is no filesystem so stat'ing returns nothing.
+   mp_import_stat_t mp_import_stat(const char *path) {
+       return MP_IMPORT_STAT_NO_EXIST;
+   }
+
+   // There is no filesystem so opening a file raises an exception.
+   mp_lexer_t *mp_lexer_new_from_file(const char *filename) {
+       mp_raise_OSError(MP_ENOENT);
+   }
+
+We also need a Makefile at this point for the port:
+
+.. code-block:: Makefile
+
+   # Include the core environment definitions; this will set $(TOP).
+   include ../../py/mkenv.mk
+
+   # Include py core make definitions.
+   include $(TOP)/py/py.mk
+   include $(TOP)/extmod/extmod.mk
+
+   # Set CFLAGS and libraries.
+   CFLAGS += -I. -I$(BUILD) -I$(TOP)
+   LIBS += -lm
+
+   # Define the required source files.
+   SRC_C = \
+       main.c \
+       mphalport.c \
+       shared/readline/readline.c \
+       shared/runtime/gchelper_generic.c \
+       shared/runtime/pyexec.c \
+       shared/runtime/stdout_helpers.c \
+
+   # Define source files containung qstrs.
+   SRC_QSTR += shared/readline/readline.c shared/runtime/pyexec.c
+
+   # Define the required object files.
+   OBJ = $(PY_CORE_O) $(addprefix $(BUILD)/, $(SRC_C:.c=.o))
+
+   # Define the top-level target, the main firmware.
+   all: $(BUILD)/firmware.elf
+
+   # Define how to build the firmware.
+   $(BUILD)/firmware.elf: $(OBJ)
+       $(ECHO) "LINK $@"
+       $(Q)$(CC) $(LDFLAGS) -o $@ $^ $(LIBS)
+       $(Q)$(SIZE) $@
+
+   # Include remaining core make rules.
+   include $(TOP)/py/mkrules.mk
+
+Remember to use proper tabs to indent the Makefile.
+
+MicroPython Configurations
+--------------------------
+
+After integrating the minimal code above, the next step is to create the MicroPython
+configuration files for the port. The compile-time configurations are specified in
+``mpconfigport.h`` and additional hardware-abstraction functions, such as time keeping,
+in ``mphalport.h``.
+
+The following is an example of an ``mpconfigport.h`` file:
+
+.. code-block:: c
+
+   #include <stdint.h>
+
+   // Python internal features.
+   #define MICROPY_ENABLE_GC                       (1)
+   #define MICROPY_HELPER_REPL                     (1)
+   #define MICROPY_ERROR_REPORTING                 (MICROPY_ERROR_REPORTING_TERSE)
+   #define MICROPY_FLOAT_IMPL                      (MICROPY_FLOAT_IMPL_FLOAT)
+
+   // Fine control over Python builtins, classes, modules, etc.
+   #define MICROPY_PY_ASYNC_AWAIT                  (0)
+   #define MICROPY_PY_BUILTINS_SET                 (0)
+   #define MICROPY_PY_ATTRTUPLE                    (0)
+   #define MICROPY_PY_COLLECTIONS                  (0)
+   #define MICROPY_PY_MATH                         (0)
+   #define MICROPY_PY_IO                           (0)
+   #define MICROPY_PY_STRUCT                       (0)
+
+   // Type definitions for the specific machine.
+
+   typedef intptr_t mp_int_t; // must be pointer size
+   typedef uintptr_t mp_uint_t; // must be pointer size
+   typedef long mp_off_t;
+
+   // We need to provide a declaration/definition of alloca().
+   #include <alloca.h>
+
+   // Define the port's name and hardware.
+   #define MICROPY_HW_BOARD_NAME "example-board"
+   #define MICROPY_HW_MCU_NAME   "unknown-cpu"
+
+   #define MP_STATE_PORT MP_STATE_VM
+
+This configuration file contains machine-specific configurations including aspects like if different
+MicroPython features should be enabled e.g. ``#define MICROPY_ENABLE_GC (1)``. Making this Setting
+``(0)`` disables the feature.
+
+Other configurations include type definitions, root pointers, board name, microcontroller name
+etc.
+
+Similarly, an minimal example ``mphalport.h`` file looks like this:
+
+.. code-block:: c
+
+   static inline void mp_hal_set_interrupt_char(char c) {}
+
+Support for standard input/output
+---------------------------------
+
+MicroPython requires at least a way to output characters, and to have a REPL it also
+requires a way to input characters. Functions for this can be implemented in the file
+``mphalport.c``, for example:
+
+.. code-block:: c
+
+   #include <unistd.h>
+   #include "py/mpconfig.h"
+
+   // Receive single character, blocking until one is available.
+   int mp_hal_stdin_rx_chr(void) {
+       unsigned char c = 0;
+       int r = read(STDIN_FILENO, &c, 1);
+       (void)r;
+       return c;
+   }
+
+   // Send the string of given length.
+   void mp_hal_stdout_tx_strn(const char *str, mp_uint_t len) {
+       int r = write(STDOUT_FILENO, str, len);
+       (void)r;
+   }
+
+These input and output functions have to be modified depending on the
+specific board API. This example uses the standard input/output stream.
+
+Building and running
+--------------------
+
+At this stage the directory of the new port should contain::
+
+    ports/example_port/
+    ├── main.c
+    ├── Makefile
+    ├── mpconfigport.h
+    ├── mphalport.c
+    └── mphalport.h
+
+The port can now be built by running ``make`` (or otherwise, depending on your system).
+
+If you are using the default compiler settings in the Makefile given above then this
+will create an executable called ``build/firmware.elf`` which can be executed directly.
+To get a functional REPL you may need to first configure the terminal to raw mode:
+
+.. code-block:: bash
+
+   $ stty raw opost -echo
+   $ ./build/firmware.elf
+
+That should give a MicroPython REPL.  You can then run commands like:
+
+.. code-block:: bash
+
+   MicroPython v1.13 on 2021-01-01; example-board with unknown-cpu
+   >>> import sys
+   >>> sys.implementation
+   ('micropython', (1, 13, 0))
+   >>>
+
+Use Ctrl-D to exit, and then run ``reset`` to reset the terminal.
+
+Adding a module to the port
+---------------------------
+
+To add a custom module like ``myport``, first add the module definition in a file
+``modmyport.c``:
+
+.. code-block:: c
+
+   #include "py/runtime.h"
+
+   STATIC mp_obj_t myport_info(void) {
+       mp_printf(&mp_plat_print, "info about my port\n");
+       return mp_const_none;
+   }
+   STATIC MP_DEFINE_CONST_FUN_OBJ_0(myport_info_obj, myport_info);
+
+   STATIC const mp_rom_map_elem_t myport_module_globals_table[] = {
+       { MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_myport) },
+       { MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&myport_info_obj) },
+   };
+   STATIC MP_DEFINE_CONST_DICT(myport_module_globals, myport_module_globals_table);
+
+   const mp_obj_module_t myport_module = {
+       .base = { &mp_type_module },
+       .globals = (mp_obj_dict_t *)&myport_module_globals,
+   };
+
+   MP_REGISTER_MODULE(MP_QSTR_myport, myport_module);
+
+You will also need to edit the Makefile to add ``modmyport.c`` to the ``SRC_C`` list, and
+a new line adding the same file to ``SRC_QSTR`` (so qstrs are searched for in this new file),
+like this:
+
+.. code-block:: Makefile
+
+   SRC_C = \
+       main.c \
+       modmyport.c \
+       mphalport.c \
+       ...
+
+   SRC_QSTR += modmyport.c
+
+If all went correctly then, after rebuilding, you should be able to import the new module:
+
+.. code-block:: bash
+
+    >>> import myport
+    >>> myport.info()
+    info about my port
+    >>>

docs/develop/publiccapi.rst

@@ -0,0 +1,25 @@
+.. _publiccapi:
+
+The public C API
+================
+
+The public C-API comprises functions defined in all C header files in the ``py/``
+directory. Most of the important core runtime C APIs are exposed in ``runtime.h`` and
+``obj.h``.
+
+The following is an example of public API functions from ``obj.h``:
+
+.. code-block:: c
+
+   mp_obj_t mp_obj_new_list(size_t n, mp_obj_t *items);
+   mp_obj_t mp_obj_list_append(mp_obj_t self_in, mp_obj_t arg);
+   mp_obj_t mp_obj_list_remove(mp_obj_t self_in, mp_obj_t value);
+   void mp_obj_list_get(mp_obj_t self_in, size_t *len, mp_obj_t **items);
+
+At its core, any functions and macros in header files make up the public
+API and can be used to access very low-level details of MicroPython. Static
+inline functions in header files are fine too, such functions will be
+inlined in the code when used.
+
+Header files in the ``ports`` directory are only exposed to the functionality
+specific to a given port.

docs/develop/qstr.rst

@@ -1,3 +1,5 @@
+.. _qstr:
+
 MicroPython string interning
 ============================
 
@@ -51,12 +53,13 @@ Processing happens in the following stages:
    through the C pre-processor.  This means that any conditionally disabled code
    will be removed, and macros expanded.  This means we don't add strings to the
    pool that won't be used in the final firmware.  Because at this stage (thanks
-   to the ``NO_QSTR`` macro added by ``QSTR_GEN_EXTRA_CFLAGS``) there is no
+   to the ``NO_QSTR`` macro added by ``QSTR_GEN_CFLAGS``) there is no
    definition for ``MP_QSTR_Foo`` it passes through this stage unaffected.  This
    file also includes comments from the preprocessor that include line number
    information.  Note that this step only uses files that have changed, which
    means that ``qstr.i.last`` will only contain data from files that have
    changed since the last compile.
+
 2. ``qstr.split`` is an empty file created after running ``makeqstrdefs.py split``
    on qstr.i.last. It's just used as a dependency to indicate that the step ran.
    This script outputs one file per input C file,  ``genhdr/qstr/...file.c.qstr``,
@@ -71,8 +74,8 @@ Processing happens in the following stages:
    data is written to another file (``qstrdefs.collected.h.hash``) which allows
    it to track changes across builds.
 
-4. ``qstrdefs.preprocessed.h`` adds in the QSTRs from qstrdefs*.  It
-   concatenates ``qstrdefs.collected.h`` with ``qstrdefs*.h``, then it transforms
+4. Generate an enumeration, each entry of which maps a ``MP_QSTR_Foo`` to it's corresponding index.
+   It concatenates ``qstrdefs.collected.h`` with ``qstrdefs*.h``, then it transforms
    each line from ``Q(Foo)`` to ``"Q(Foo)"`` so they pass through the preprocessor
    unchanged.  Then the preprocessor is used to deal with any conditional
    compilation in ``qstrdefs*.h``.  Then the transformation is undone back to

docs/develop/writingtests.rst

@@ -0,0 +1,70 @@
+.. _writingtests:
+
+Writing tests
+=============
+
+Tests in MicroPython are located at the path ``tests/``. The following is a listing of
+key directories and the run-tests.py runner script:
+
+.. code-block:: bash
+
+   .
+    ├── basics
+    ├── extmod
+    ├── float
+    ├── micropython
+    ├── run-tests.py
+    ...
+
+There are subfolders maintained to categorize the tests. Add a test by creating a new file in one of the
+existing folders or in a new folder. It's also possible to make custom tests outside this tests folder,
+which would be recommended for a custom port.
+
+For example, add the following code in a file ``print.py`` in the ``tests/unix/`` subdirectory:
+
+.. code-block:: python
+
+   def print_one():
+       print(1)
+
+   print_one()
+
+If you run your tests, this test should appear in the test output:
+
+.. code-block:: bash
+
+   $ cd ports/unix
+   $ make tests
+   skip  unix/extra_coverage.py
+   pass  unix/ffi_callback.py
+   pass  unix/ffi_float.py
+   pass  unix/ffi_float2.py
+   pass  unix/print.py
+   pass  unix/time.py
+   pass  unix/time2.py
+
+Tests are run by comparing the output from the test target against the output from CPython.
+So any test should use print statements to indicate test results.
+
+For tests that can't be compared to CPython (i.e. micropython-specific functionality),
+you can provide a ``.py.exp`` file which will be used as the truth for comparison.
+
+The other way to run tests, which is useful when running on targets other than the Unix port, is:
+
+.. code-block:: bash
+
+   $ cd tests
+   $ ./run-tests.py
+
+Then to run on a board:
+
+.. code-block:: bash
+
+   $ ./run-tests.py --target minimal --device /dev/ttyACM0
+
+And to run only a certain set of tests (eg a directory):
+
+.. code-block:: bash
+
+   $ ./run-tests.py -d basics
+   $ ./run-tests.py float/builtin*.py

docs/differences/index_template.txt

@@ -3,8 +3,22 @@
 MicroPython differences from CPython
 ====================================
 
-The operations listed in this section produce conflicting results in MicroPython when compared to standard Python.
-MicroPython implements Python 3.4 and some select features of Python 3.5.
+MicroPython implements Python 3.4 and some select features of Python 3.5 and
+above.  The sections below describe the current status of these features.
+
+.. toctree::
+
+    ../differences/python_35.rst
+    ../differences/python_36.rst
+    ../differences/python_37.rst
+    ../differences/python_38.rst
+    ../differences/python_39.rst
+    ../differences/python_310.rst
+
+For the features of Python that are implemented by MicroPython, there are
+sometimes differences in their behaviour compared to standard Python.  The
+operations listed in the sections below produce conflicting results in
+MicroPython when compared to standard Python.
 
 .. toctree::
     :maxdepth: 2

docs/differences/python_310.rst

@@ -0,0 +1,238 @@
+.. _python_310:
+
+Python 3.10
+===========
+
+Python 3.10.0 (final) was released on the 4 October 2021. The Features for 3.10 are
+defined in `PEP 619 <https://www.python.org/dev/peps/pep-0619/#features-for-3-10>`_
+and a detailed description of the changes can be found in
+`What's New in Python 3.10 <https://docs.python.org/3/whatsnew/3.10.html>`_.
+
+.. table::
+  :widths: 20 60 20
+
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | **New syntax features**                                                                                     | **Status**   |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 634 <https://www.python.org/dev/peps/pep-0634/>`_ | Structural Pattern Matching: Specification         | [#spm]_      |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 635 <https://www.python.org/dev/peps/pep-0635/>`_ | Structural Pattern Matching: Motivation and        | [#spm]_      |
+  |                                                        | Rationale                                          |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 636 <https://www.python.org/dev/peps/pep-0636/>`_ | Structural Pattern Matching: Tutorial              | [#spm]_      |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `bpo-12782                                             | Parenthesized context managers are now officially  |              |
+  | <https://github.com/python/cpython/issues/56991>`_     | allowed                                            |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | **New features in the standard library**                                                                                   |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 618 <https://www.python.org/dev/peps/pep-0618/>`_ | Add Optional Length-Checking To zip                |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | **Interpreter improvements**                                                                                               |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 626 <https://www.python.org/dev/peps/pep-0626/>`_ | Precise line numbers for debugging and other tools |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | **New typing features**                                                                                                    |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`_ | Allow writing union types as X | Y                 |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 613 <https://www.python.org/dev/peps/pep-0613/>`_ | Explicit Type Aliases                              |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 612 <https://www.python.org/dev/peps/pep-0612/>`_ | Parameter Specification Variables                  |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | **Important deprecations, removals or restrictions**                                                                       |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 644 <https://www.python.org/dev/peps/pep-0644/>`_ | Require OpenSSL 1.1.1 or newer                     |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 632 <https://www.python.org/dev/peps/pep-0632/>`_ | Deprecate distutils module.                        | Not relevant |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 623 <https://www.python.org/dev/peps/pep-0623/>`_ | Deprecate and prepare for the removal of the wstr  | Not relevant |
+  |                                                        | member in PyUnicodeObject.                         |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 624 <https://www.python.org/dev/peps/pep-0624/>`_ | Remove Py_UNICODE encoder APIs                     | Not relevant |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+  | `PEP 597 <https://www.python.org/dev/peps/pep-0597/>`_ | Add optional EncodingWarning                       |              |
+  +--------------------------------------------------------+----------------------------------------------------+--------------+
+
+
+Other Language Changes:
+
+.. table::
+  :widths: 90 10
+
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | The :class:`int` type has a new method :meth:`int.bit_count`, returning the                                 |               |
+  | number of ones in the binary expansion of a given integer, also known                                       |               |
+  | as the population count.                                                                                    |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | The views returned by :meth:`dict.keys`, :meth:`dict.values` and                                            |               |
+  | :meth:`dict.items` now all have a ``mapping`` attribute that gives a                                        |               |
+  | :class:`types.MappingProxyType` object wrapping the original                                                |               |
+  | dictionary.                                                                                                 |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | :pep:`618`: The :func:`zip` function now has an optional ``strict`` flag, used                              |               |
+  | to require that all the iterables have an equal length.                                                     |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Builtin and extension functions that take integer arguments no longer accept                                |               |
+  | :class:`~decimal.Decimal`\ s, :class:`~fractions.Fraction`\ s and other                                     |               |
+  | objects that can be converted to integers only with a loss (e.g. that have                                  |               |
+  | the :meth:`~object.__int__` method but do not have the                                                      |               |
+  | :meth:`~object.__index__` method).                                                                          |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | If :func:`object.__ipow__` returns :const:`NotImplemented`, the operator will                               |               |
+  | correctly fall back to :func:`object.__pow__` and :func:`object.__rpow__` as expected.                      |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Assignment expressions can now be used unparenthesized within set literals                                  |               |
+  | and set comprehensions, as well as in sequence indexes (but not slices).                                    |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Functions have a new ``__builtins__`` attribute which is used to look for                                   |               |
+  | builtin symbols when a function is executed, instead of looking into                                        |               |
+  | ``__globals__['__builtins__']``. The attribute is initialized from                                          |               |
+  | ``__globals__["__builtins__"]`` if it exists, else from the current builtins.                               |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Two new builtin functions -- :func:`aiter` and :func:`anext` have been added                                |               |
+  | to provide asynchronous counterparts to :func:`iter` and :func:`next`,                                      |               |
+  | respectively.                                                                                               |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Static methods (:func:`@staticmethod <staticmethod>`) and class methods                                     |               |
+  | (:func:`@classmethod <classmethod>`) now inherit the method attributes                                      |               |
+  | (``__module__``, ``__name__``, ``__qualname__``, ``__doc__``,                                               |               |
+  | ``__annotations__``) and have a new ``__wrapped__`` attribute.                                              |               |
+  | Moreover, static methods are now callable as regular functions.                                             |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Annotations for complex targets (everything beside ``simple name`` targets                                  |               |
+  | defined by :pep:`526`) no longer cause any runtime effects with ``from __future__ import annotations``.     |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Class and module objects now lazy-create empty annotations dicts on demand.                                 |               |
+  | The annotations dicts are stored in the object’s ``__dict__`` for                                           |               |
+  | backwards compatibility.  This improves the best practices for working                                      |               |
+  | with ``__annotations__``.                                                                                   |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Annotations consist of ``yield``, ``yield from``, ``await`` or named expressions                            |               |
+  | are now forbidden under ``from __future__ import annotations`` due to their side                            |               |
+  | effects.                                                                                                    |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Usage of unbound variables, ``super()`` and other expressions that might                                    |               |
+  | alter the processing of symbol table as annotations are now rendered                                        |               |
+  | effectless under ``from __future__ import annotations``.                                                    |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Hashes of NaN values of both :class:`float` type and                                                        |               |
+  | :class:`decimal.Decimal` type now depend on object identity. Formerly, they                                 |               |
+  | always hashed to ``0`` even though NaN values are not equal to one another.                                 |               |
+  | This caused potentially quadratic runtime behavior due to excessive hash                                    |               |
+  | collisions when creating dictionaries and sets containing multiple NaNs.                                    |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | A :exc:`SyntaxError` (instead of a :exc:`NameError`) will be raised when deleting                           |               |
+  | the :const:`__debug__` constant.                                                                            |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | :exc:`SyntaxError` exceptions now have ``end_lineno`` and                                                   |               |
+  | ``end_offset`` attributes.  They will be ``None`` if not determined.                                        |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+
+Changes to built-in modules:
+
+.. table::
+  :widths: 90 10
+
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `asyncio <https://docs.python.org/3/whatsnew/3.10.html#asyncio>`_                                                             |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add missing :meth:`~asyncio.events.AbstractEventLoop.connect_accepted_socket`                                 |               |
+  | method.                                                                                                       |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `array <https://docs.python.org/3/whatsnew/3.10.html#array>`_                                                                 |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The :meth:`~array.array.index` method of :class:`array.array` now has                                         |               |
+  | optional *start* and *stop* parameters.                                                                       |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `gc <https://docs.python.org/3/whatsnew/3.10.html#gc>`_                                                                       |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add audit hooks for :func:`gc.get_objects`, :func:`gc.get_referrers` and                                      |               |
+  | :func:`gc.get_referents`.                                                                                     |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `hashlib <https://docs.python.org/3/whatsnew/3.10.html#hashlib>`_                                                             |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The hashlib module requires OpenSSL 1.1.1 or newer.                                                           |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The hashlib module has preliminary support for OpenSSL 3.0.0.                                                 |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The pure-Python fallback of :func:`~hashlib.pbkdf2_hmac` is deprecated. In                                    |               |
+  | the future PBKDF2-HMAC will only be available when Python has been built with                                 |               |
+  | OpenSSL support.                                                                                              |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `os <https://docs.python.org/3/whatsnew/3.10.html#os>`_                                                                       |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add :func:`os.cpu_count()` support for VxWorks RTOS.                                                          |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add a new function :func:`os.eventfd` and related helpers to wrap the                                         |               |
+  | ``eventfd2`` syscall on Linux.                                                                                |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add :func:`os.splice()` that allows to move data between two file                                             |               |
+  | descriptors without copying between kernel address space and user                                             |               |
+  | address space, where one of the file descriptors must refer to a                                              |               |
+  | pipe.                                                                                                         |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add :data:`~os.O_EVTONLY`, :data:`~os.O_FSYNC`, :data:`~os.O_SYMLINK`                                         |               |
+  | and :data:`~os.O_NOFOLLOW_ANY` for macOS.                                                                     |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `platform <https://docs.python.org/3/whatsnew/3.10.html#platform>`_                                                           |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add :func:`platform.freedesktop_os_release()` to retrieve operation system                                    |               |
+  | identification from `freedesktop.org os-release                                                               |               |
+  | <https://www.freedesktop.org/software/systemd/man/os-release.html>`_ standard file.                           |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `socket <https://docs.python.org/3/whatsnew/3.10.html#socket>`_                                                               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The exception :exc:`socket.timeout` is now an alias of :exc:`TimeoutError`.                                   |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add option to create MPTCP sockets with ``IPPROTO_MPTCP``.                                                    |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add ``IP_RECVTOS`` option to receive the type of service (ToS) or DSCP/ECN fields.                            |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `ssl <https://docs.python.org/3/whatsnew/3.10.html#ssl>`_                                                                     |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The ssl module requires OpenSSL 1.1.1 or newer.                                                               |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The ssl module has preliminary support for OpenSSL 3.0.0 and new option                                       |               |
+  | :data:`~ssl.OP_IGNORE_UNEXPECTED_EOF`.                                                                        |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Deprecated function and use of deprecated constants now result in                                             |               |
+  | a :exc:`DeprecationWarning`. :attr:`ssl.SSLContext.options` has                                               |               |
+  | :data:`~ssl.OP_NO_SSLv2` and :data:`~ssl.OP_NO_SSLv3` set by default and                                      |               |
+  | therefore cannot warn about setting the flag again.                                                           |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The ssl module now has more secure default settings. Ciphers without forward                                  |               |
+  | secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits                                      |               |
+  | weak RSA, DH, and ECC keys with less than 112 bits of security.                                               |               |
+  | :class:`~ssl.SSLContext` defaults to minimum protocol version TLS 1.2.                                        |               |
+  | Settings are based on Hynek Schlawack's research.                                                             |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer                                          |               |
+  | officially supported. Python does not block them actively. However                                            |               |
+  | OpenSSL build options, distro configurations, vendor patches, and cipher                                      |               |
+  | suites may prevent a successful handshake.                                                                    |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add a *timeout* parameter to the :func:`ssl.get_server_certificate` function.                                 |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The ssl module uses heap-types and multi-phase initialization.                                                |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | A new verify flag :data:`~ssl.VERIFY_X509_PARTIAL_CHAIN` has been added.                                      |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `sys <https://docs.python.org/3/whatsnew/3.10.html#sys>`_                                                                     |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add :data:`sys.orig_argv` attribute: the list of the original command line                                    |               |
+  | arguments passed to the Python executable.                                                                    |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Add :data:`sys.stdlib_module_names`, containing the list of the standard library                              |               |
+  | module names.                                                                                                 |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `_thread <https://docs.python.org/3/whatsnew/3.10.html#_thread>`_                                                             |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | :func:`_thread.interrupt_main` now takes an optional signal number to                                         |               |
+  | simulate (the default is still :data:`signal.SIGINT`).                                                        |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+
+.. rubric:: Notes
+
+.. [#spm] The structural pattern matching feature is discussed in `issue #7847 <https://github.com/micropython/micropython/issues/7847>`_.
+

docs/differences/python_35.rst

@@ -0,0 +1,193 @@
+.. _python_35:
+
+Python 3.5
+==========
+
+Below is a list of finalised/accepted PEPs for Python 3.5 grouped into their impact to MicroPython.
+
+.. table::
+  :widths: 30 50 20
+
+  +--------------------------------------------------------------------------------------------------------------+--------------------+
+  | **Extensions to the syntax**                                                                                 | **Status**         |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 448 <https://www.python.org/dev/peps/pep-0448/>`_ | Additional unpacking generalizations                | Partial            |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 465 <https://www.python.org/dev/peps/pep-0465/>`_ | A new matrix multiplication operator                | Complete           |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 492 <https://www.python.org/dev/peps/pep-0492/>`_ | Coroutines with ``async`` and ``await`` syntax      | Complete           |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | **Extensions and changes to runtime**                                                                                             |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 461 <https://www.python.org/dev/peps/pep-0461/>`_ | % formatting for binary strings                     | Complete           |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 475 <https://www.python.org/dev/peps/pep-0475/>`_ | Retrying system calls that fail with ``EINTR``      | Complete           |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 479 <https://www.python.org/dev/peps/pep-0479/>`_ | Change ``StopIteration`` handling inside generators | Complete           |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | **Standard library changes**                                                                                                      |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 471 <https://www.python.org/dev/peps/pep-0471/>`_ | ``os.scandir()``                                    |                    |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 485 <https://www.python.org/dev/peps/pep-0485/>`_ | ``math.isclose()``, a function for testing          | Complete           |
+  |                                                        | approximate equality                                |                    |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | **Miscellaneous changes**                                                                                                         |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 441 <https://www.python.org/dev/peps/pep-0441/>`_ | Improved Python zip application support             |                    |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 486 <https://www.python.org/dev/peps/pep-0486/>`_ | Make the Python Launcher aware of virtual           | Not relevant       |
+  |                                                        | environments                                        |                    |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 484 <https://www.python.org/dev/peps/pep-0484/>`_ | Type hints (advisory only)                          | Complete [#fth]_   |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 488 <https://www.python.org/dev/peps/pep-0488/>`_ | Elimination of PYO files                            | Not relevant       |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+  | `PEP 489 <https://www.python.org/dev/peps/pep-0489/>`_ | Redesigning extension module loading                |                    |
+  +--------------------------------------------------------+-----------------------------------------------------+--------------------+
+
+Other Language Changes:
+
+.. table::
+  :widths: 90 10
+
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Added the *namereplace* error handlers. The *backslashreplace* error handlers now work with decoding and  |               |
+  | translating.                                                                                              |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Property docstrings are now writable. This is especially useful for collections.namedtuple() docstrings   |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Circular imports involving relative imports are now supported.                                            |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+
+
+New Modules:
+
+* `typing <https://docs.python.org/3/whatsnew/3.5.html#typing>`_
+
+* `zipzap <https://docs.python.org/3/whatsnew/3.5.html#zipapp>`_
+
+
+Changes to built-in modules:
+
+.. table::
+  :widths: 90 10
+
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `collections <https://docs.python.org/3/whatsnew/3.5.html#collections>`_                                                  |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *OrderedDict* class is now implemented in C, which makes it 4 to 100 times faster.                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | *OrderedDict.items()* , *OrderedDict.keys()* , *OrderedDict.values()* views now support reversed()        |               |
+  | iteration.                                                                                                |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The deque class now defines *index()*, *insert()*, and *copy()*, and supports the + and * operators.      |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Docstrings produced by namedtuple() can now be updated.                                                   |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The UserString class now implements the *__getnewargs__()*, *__rmod__()*, *casefold()*, *format_map()*,   |               |
+  | *isprintable()*, and *maketrans()* methods to match the corresponding methods of str.                     |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `heapq <https://docs.python.org/3/whatsnew/3.5.html#heapq>`_                                                              |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Element comparison in *merge()* can now be customized by passing a key function in a new optional key     |               |
+  | keyword argument, and a new optional *reverse* keyword argument can be used to reverse element comparison |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `io <https://docs.python.org/3/whatsnew/3.5.html#io>`_                                                                    |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | A new *BufferedIOBase.readinto1()* method, that uses at most one call to the underlying raw stream's      |               |
+  | *RawIOBase.read()* or *RawIOBase.readinto()* methods                                                      |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `json <https://docs.python.org/3/whatsnew/3.5.html#json>`_                                                |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | JSON decoder now raises JSONDecodeError instead of ValueError to provide better context information about |               |
+  | the error.                                                                                                |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `math <https://docs.python.org/3/whatsnew/3.5.html#math>`_                                                                |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Two new constants have been added to the math module: *inf* and *nan*.                                    | Complete      |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | A new function *isclose()* provides a way to test for approximate equality.                               |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | A new *gcd()* function has been added. The *fractions.gcd()* function is now deprecated.                  |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `os <https://docs.python.org/3/whatsnew/3.5.html#os>`_                                                                    |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The new *scandir()* function returning an iterator of DirEntry objects has been added.                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *urandom()* function now uses the *getrandom()* syscall on Linux 3.17 or newer, and *getentropy()* on |               |
+  | OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file  |               |
+  | descriptor exhaustion.                                                                                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | New *get_blocking()* and *set_blocking()* functions allow getting and setting a file descriptor's blocking|               |
+  | mode (O_NONBLOCK.)                                                                                        |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | There is a new *os.path.commonpath()* function returning the longest common sub-path of each passed       |               |
+  | pathname                                                                                                  |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `re <https://docs.python.org/3/whatsnew/3.5.html#re>`_                                                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | References and conditional references to groups with fixed length are now allowed in lookbehind assertions|               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The number of capturing groups in regular expressions is no longer limited to 100.                        |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *sub()* and *subn()* functions now replace unmatched groups with empty strings instead of raising an  |               |
+  | exception.                                                                                                |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *re.error* exceptions have new attributes, msg, pattern, pos, lineno, and colno, that provide better  |               |
+  | context information about the error                                                                       |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `socket <https://docs.python.org/3/whatsnew/3.5.html#socket>`_                                                            |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Functions with timeouts now use a monotonic clock, instead of a system clock.                             |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | A new *socket.sendfile()* method allows sending a file over a socket by using the high-performance        |               |
+  | *os.sendfile()* function on UNIX, resulting in uploads being from 2 to 3 times faster than when using     |               |
+  | plain *socket.send()*                                                                                     |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *socket.sendall()* method no longer resets the socket timeout every time bytes are received or sent.  |               |
+  | The socket timeout is now the maximum total duration to send all data.                                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The backlog argument of the *socket.listen()* method is now optional. By default it is set to SOMAXCONN or| Complete      |
+  | to 128, whichever is less.                                                                                |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `ssl <https://docs.python.org/3/whatsnew/3.5.html#ssl>`_                                                                  |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Memory BIO Support                                                                                        |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | Application-Layer Protocol Negotiation Support                                                            |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | There is a new *SSLSocket.version()* method to query the actual protocol version in use.                  |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The SSLSocket class now implements a *SSLSocket.sendfile()* method.                                       |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *SSLSocket.send()* method now raises either the *ssl.SSLWantReadError* or *ssl.SSLWantWriteError*     |               |
+  | exception on a non-blocking socket if the operation would block. Previously, it would return 0.           |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *cert_time_to_seconds()* function now interprets the input time as UTC and not as local time, per RFC |               |
+  | 5280. Additionally, the return value is always an int.                                                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | New *SSLObject.shared_ciphers()* and *SSLSocket.shared_ciphers()* methods return the list of ciphers sent |               |
+  | by the client during the handshake.                                                                       |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *SSLSocket.do_handshake()*, *SSLSocket.read()*, *SSLSocket.shutdown()*, and *SSLSocket.write()*       |               |
+  | methods of the SSLSocket class no longer reset the socket timeout every time bytes are received or sent.  |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *match_hostname()* function now supports matching of IP addresses.                                    |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `sys <https://docs.python.org/3/whatsnew/3.5.html#sys>`_                                                                  |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | A new *set_coroutine_wrapper()* function allows setting a global hook that will be called whenever a      |               |
+  | coroutine object is created by an async def function. A corresponding *get_coroutine_wrapper()* can be    |               |
+  | used to obtain a currently set wrapper.                                                                   |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | A new *is_finalizing()* function can be used to check if the Python interpreter is shutting down.         |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | `time <https://docs.python.org/3/whatsnew/3.5.html#time>`_                                                                |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+  | The *monotonic()* function is now always available                                                        |               |
+  +-----------------------------------------------------------------------------------------------------------+---------------+
+
+.. rubric:: Notes
+
+.. [#fth] The MicroPython parser correct ignores all type hints. However, the ``typing`` module is not built-in.

docs/differences/python_36.rst

@@ -0,0 +1,200 @@
+.. _python_36:
+
+Python 3.6
+==========
+
+Python 3.6 beta 1 was released on 12 Sep 2016, and a summary of the new features can be found here:
+
+.. table::
+  :widths: 30 50 20
+
+  +-----------------------------------------------------------------------------------------------------------+-----------------+
+  | **New Syntax Features**                                                                                   | **Status**      |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 498 <https://www.python.org/dev/peps/pep-0498/>`_ | Literal String Formatting                        | Complete        |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 515 <https://www.python.org/dev/peps/pep-0515/>`_ | Underscores in Numeric Literals                  | Complete        |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 525 <https://www.python.org/dev/peps/pep-0525/>`_ | Asynchronous Generators                          |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 526 <https://www.python.org/dev/peps/pep-0526/>`_ | Syntax for Variable Annotations (provisional)    | Complete        |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 530 <https://www.python.org/dev/peps/pep-0530/>`_ | Asynchronous Comprehensions                      |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | **New Built-in Features**                                                                                                   |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 468 <https://www.python.org/dev/peps/pep-0468/>`_ | Preserving the order of *kwargs* in a function   |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 487 <https://www.python.org/dev/peps/pep-0487/>`_ | Simpler customization of class creation          |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 520 <https://www.python.org/dev/peps/pep-0520/>`_ | Preserving Class Attribute Definition Order      |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | **Standard Library Changes**                                                                                                |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 495 <https://www.python.org/dev/peps/pep-0495/>`_ | Local Time Disambiguation                        |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 506 <https://www.python.org/dev/peps/pep-0506/>`_ | Adding A Secrets Module To The Standard Library  |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 519 <https://www.python.org/dev/peps/pep-0519/>`_ | Adding a file system path protocol               |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | **CPython Internals**                                                                                                       |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 509 <https://www.python.org/dev/peps/pep-0509/>`_ | Add a private version to dict                    | Won't do        |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 523 <https://www.python.org/dev/peps/pep-0523/>`_ | Adding a frame evaluation API to CPython         |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | **Linux/Window Changes**                                                                                                    |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 524 <https://www.python.org/dev/peps/pep-0524/>`_ | Make ``os.urandom()`` blocking on Linux          |                 |
+  |                                                        | (during system startup)                          |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_ | Change Windows console encoding to UTF-8         |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+  | `PEP 529 <https://www.python.org/dev/peps/pep-0529/>`_ | Change Windows filesystem encoding to UTF-8      |                 |
+  +--------------------------------------------------------+--------------------------------------------------+-----------------+
+
+Other Language Changes:
+
+.. table::
+  :widths: 90 10
+
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | A *global* or *nonlocal* statement must now textually appear before the first use of the affected name in   |               |
+  | the same scope. Previously this was a SyntaxWarning.                                                        |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | It is now possible to set a special method to None to indicate that the corresponding operation is not      |               |
+  | available. For example, if a class sets *__iter__()* to *None* , the class is not iterable.                 |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Long sequences of repeated traceback lines are now abbreviated as *[Previous line repeated {count} more     |               |
+  | times]*                                                                                                     |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Import now raises the new exception *ModuleNotFoundError* when it cannot find a module. Code that currently |               |
+  | checks for ImportError (in try-except) will still work.                                                     |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Class methods relying on zero-argument *super()* will now work correctly when called from metaclass methods |               |
+  | during class creation.                                                                                      |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+
+Changes to built-in modules:
+
+.. table::
+  :widths: 90 10
+
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `array <https://docs.python.org/3.6/whatsnew/3.6.html#array>`_                                               |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Exhausted iterators of *array.array* will now stay exhausted even if the iterated array is extended.         |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `binascii <https://docs.python.org/3.6/whatsnew/3.6.html#binascii>`_                                         |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline    | Complete       |
+  | character is appended to the return value                                                                    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `cmath <https://docs.python.org/3.6/whatsnew/3.6.html#cmath>`_                                               |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new cmath.tau (τ) constant has been added                                                                |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | New constants: *cmath.inf* and *cmath.nan* to match *math.inf* and *math.nan* , and also *cmath.infj* and    |                |
+  | *cmath.nanj* to match the format used by complex repr                                                        |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `collections <https://docs.python.org/3.6/whatsnew/3.6.html#collections>`_                                                    |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new Collection abstract base class has been added to represent sized iterable container classes          |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new *Reversible* abstract base class represents iterable classes that also provide the *__reversed__()*  |                |
+  | method.                                                                                                      |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new *AsyncGenerator* abstract base class represents asynchronous generators.                             |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The *namedtuple()* function now accepts an optional keyword argument module, which, when specified, is used  |                |
+  | for the *__module__* attribute of the returned named tuple class.                                            |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The verbose and rename arguments for *namedtuple()* are now keyword-only.                                    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Recursive *collections.deque* instances can now be pickled.                                                  |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `hashlib <https://docs.python.org/3.6/whatsnew/3.6.html#hashlib>`_                                                            |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | BLAKE2 hash functions were added to the module. *blake2b()* and *blake2s()* are always available and support |                |
+  | the full feature set of BLAKE2.                                                                              |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The SHA-3 hash functions *sha3_224()*, *sha3_256()*, *sha3_384()*, *sha3_512()*, and *SHAKE* hash functions  |                |
+  | *shake_128()* and *shake_256()* were added.                                                                  |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The password-based key derivation function *scrypt()* is now available with OpenSSL 1.1.0 and newer.         |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `json <https://docs.python.org/3.6/whatsnew/3.6.html#json>`_                                                                  |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | *json.load()* and *json.loads()* now support binary input. Encoded JSON should be represented using either   |                |
+  | UTF-8, UTF-16, or UTF-32.                                                                                    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `math <https://docs.python.org/3.6/whatsnew/3.6.html#math>`_                                                                  |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new math.tau (τ) constant has been added                                                                 | Complete       |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `os <https://docs.python.org/3.6/whatsnew/3.6.html#os>`_                                                                      |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | A new *close()* method allows explicitly closing a *scandir()* iterator. The *scandir()* iterator now        |                |
+  | supports the context manager protocol.                                                                       |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | On Linux, *os.urandom()* now blocks until the system urandom entropy pool is initialized to increase the     |                |
+  | security.                                                                                                    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The Linux *getrandom()* syscall (get random bytes) is now exposed as the new *os.getrandom()* function.      |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `re <https://docs.python.org/3.6/whatsnew/3.6.html#re>`_                                                                      |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Added support of modifier spans in regular expressions. Examples: *'(?i:p)ython'* matches 'python' and       |                |
+  | 'Python', but not 'PYTHON'; *'(?i)g(?-i:v)r'* matches *'GvR'* and *'gvr'*, but not *'GVR'*.                  |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Match object groups can be accessed by *__getitem__*, which is equivalent to *group()*. So *mo['name']* is   |                |
+  | now equivalent to *mo.group('name')*.                                                                        |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Match objects now support index-like objects as group indices.                                               |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `socket <https://docs.python.org/3.6/whatsnew/3.6.html#socket>`_                                                              |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The *ioctl()* function now supports the *SIO_LOOPBACK_FAST_PATH* control code.                               |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The *getsockopt()* constants *SO_DOMAIN* , *SO_PROTOCOL*, *SO_PEERSEC* , and *SO_PASSSEC* are now supported. |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The *setsockopt()* now supports the *setsockopt(level, optname, None, optlen: int)* form.                    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The socket module now supports the address family *AF_ALG* to interface with Linux Kernel crypto API.        |                |
+  | *ALG_*, *SOL_ALG* and *sendmsg_afalg()* were added.                                                          |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | New Linux constants *TCP_USER_TIMEOUT* and *TCP_CONGESTION* were added.                                      |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `ssl <https://docs.python.org/3.6/whatsnew/3.6.html#ssl>`_                                                                    |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2.                                          |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | 3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added.    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | *SSLContext* has better default configuration for options and ciphers.                                       |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | SSL session can be copied from one client-side connection to another with the new *SSLSession* class. TLS    |                |
+  | session resumption can speed up the initial handshake, reduce latency and improve performance.               |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new *get_ciphers()* method can be used to get a list of enabled ciphers in order of cipher priority.     |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | All constants and flags have been converted to *IntEnum* and *IntFlags*.                                     |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Server and client-side specific TLS protocols for *SSLContext* were added.                                   |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | Added *SSLContext.post_handshake_auth* to enable and *ssl.SSLSocket.verify_client_post_handshake()* to       |                |
+  | initiate TLS 1.3 post-handshake authentication.                                                              |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `struct <https://docs.python.org/3.6/whatsnew/3.6.html#struct>`_                                             |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | now supports IEEE 754 half-precision floats via the 'e' format specifier.                                    |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `sys <https://docs.python.org/3.6/whatsnew/3.6.html#sys>`_                                                   |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The new *getfilesystemencodeerrors()* function returns the name of the error mode used to convert between    |                |
+  | Unicode filenames and bytes filenames.                                                                       |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | `zlib <https://docs.python.org/3.6/whatsnew/3.6.html#zlib>`_                                                 |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+
+  | The *compress()* and *decompress()* functions now accept keyword arguments                                   |                |
+  +--------------------------------------------------------------------------------------------------------------+----------------+

docs/differences/python_37.rst

@@ -0,0 +1,108 @@
+.. _python_37:
+
+Python 3.7
+==========
+
+New Features:
+
+.. table::
+  :widths: 20 60 20
+
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | **Feature**                                                                                               | **Status**                           |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 538 <https://www.python.org/dev/peps/pep-0538/>`_ | Coercing the legacy C locale to a UTF-8 based    |                                      |
+  |                                                        | locale                                           |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 539 <https://www.python.org/dev/peps/pep-0539/>`_ | A New C-API for Thread-Local Storage in CPython  |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 540 <https://www.python.org/dev/peps/pep-0540/>`_ | UTF-8 mode                                       |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 552 <https://www.python.org/dev/peps/pep-0552/>`_ | Deterministic pyc                                |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 553 <https://www.python.org/dev/peps/pep-0553/>`_ | Built-in ``breakpoint()``                        |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 557 <https://www.python.org/dev/peps/pep-0557/>`_ | Data Classes                                     |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 560 <https://www.python.org/dev/peps/pep-0560/>`_ | Core support for typing module and generic types |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 562 <https://www.python.org/dev/peps/pep-0562/>`_ | Module ``__getattr__`` and ``__dir__``           | Partial                              |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 563 <https://www.python.org/dev/peps/pep-0563/>`_ | Postponed Evaluation of Annotations              |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 564 <https://www.python.org/dev/peps/pep-0564/>`_ | Time functions with nanosecond resolution        | Partial [#ftimenanosec]_             |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 565 <https://www.python.org/dev/peps/pep-0565/>`_ | Show DeprecationWarning in ``__main__``          |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+  | `PEP 567 <https://www.python.org/dev/peps/pep-0567/>`_ | Context Variables                                |                                      |
+  +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
+
+Other Language Changes:
+
+.. table::
+  :widths: 90 10
+
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | ``async`` and ``await`` are now reserved keywords                                                               | Complete       |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | ``dict`` objects must preserve insertion-order                                                                  |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | More than 255 arguments can now be passed to a function; a function can now have more than 255 parameters       |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | ``bytes.fromhex()`` and ``bytearray.fromhex()`` now ignore all ASCII whitespace, not only spaces                |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | ``str``, ``bytes``, and ``bytearray`` gained support for the new ``isascii()`` method, which can be used to     |                |
+  | test if a string or bytes contain only the ASCII characters                                                     |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | ``ImportError`` now displays module name and module ``__file__`` path when ``from ... import ...`` fails        |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | Circular imports involving absolute imports with binding a submodule to a name are now supported                |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than ``format(str(self), '')``              |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | In order to better support dynamic creation of stack traces, ``types.TracebackType`` can now be                 |                |
+  | instantiated from Python code, and the ``tb_next`` attribute on tracebacks is now writable                      |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | When using the ``-m`` switch, ``sys.path[0]`` is now eagerly expanded to the full starting directory path,      |                |
+  | rather than being left as the empty directory (which allows imports from the current working directory          |                |
+  | at the time when an import occurs)                                                                              |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+  | The new ``-X importtime`` option or the ``PYTHONPROFILEIMPORTTIME`` environment variable can be used to         |                |
+  | show the timing of each module import                                                                           |                |
+  +-----------------------------------------------------------------------------------------------------------------+----------------+
+
+Changes to built-in modules:
+
+.. table::
+  :widths: 90 10
+
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | `asyncio <https://docs.python.org/3/whatsnew/3.7.html#asyncio>`_                                           |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | Too many to list                                                                                           |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | `gc <https://docs.python.org/3/whatsnew/3.7.html#gc>`_                                                     |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | New features include *gc.freeze()*, *gc.unfreeze()*, *gc-get_freeze_count*                                 |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | `math <https://docs.python.org/3/whatsnew/3.7.html#math>`_                                                 |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | math.remainder() added to implement IEEE 754-style remainder                                               |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | `re <https://docs.python.org/3/whatsnew/3.7.html#re>`_                                                     |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | A number of tidy up features including better support for splitting on empty strings and copy support for  |                |
+  | compiled expressions and match objects                                                                     |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | `sys <https://docs.python.org/3/whatsnew/3.7.html#sys>`_                                                   |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | sys.breakpointhook() added. sys.get(/set)_coroutine_origin_tracking_depth() added                          |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | `time <https://docs.python.org/3/whatsnew/3.7.html#time>`_                                                 |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+  | Mostly updates to support nanosecond resolution in PEP564, see above                                       |                |
+  +------------------------------------------------------------------------------------------------------------+----------------+
+
+.. rubric:: Notes
+
+.. [#ftimenanosec] Only :func:`time.time_ns` is implemented. 

docs/differences/python_38.rst

@@ -0,0 +1,127 @@
+.. _python_38:
+
+Python 3.8
+==========
+
+Python 3.8.0 (final) was released on the 14 October 2019. The Features for 3.8
+are defined in `PEP 569 <https://www.python.org/dev/peps/pep-0569/#id9>`_ and
+a detailed description of the changes can be found in `What's New in Python
+3.8. <https://docs.python.org/3/whatsnew/3.8.html>`_
+
+.. table::
+  :widths: 20 60 20
+
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | **Features**                                                                                               | **Status**    |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | `PEP 570 <https://www.python.org/dev/peps/pep-0570/>`_ | Positional-only arguments                         |               |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | `PEP 572 <https://www.python.org/dev/peps/pep-0572/>`_ | Assignment Expressions                            | Complete      |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | `PEP 574 <https://www.python.org/dev/peps/pep-0574/>`_ | Pickle protocol 5 with out-of-band data           |               |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | `PEP 578 <https://www.python.org/dev/peps/pep-0578/>`_ | Runtime audit hooks                               |               |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | `PEP 587 <https://www.python.org/dev/peps/pep-0587/>`_ | Python Initialization Configuration               |               |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | `PEP 590 <https://www.python.org/dev/peps/pep-0590/>`_ | Vectorcall: a fast calling protocol for CPython   |               |
+  +--------------------------------------------------------+---------------------------------------------------+---------------+
+  | **Miscellaneous**                                                                                                          |
+  +------------------------------------------------------------------------------------------------------------+---------------+
+  |  f-strings support = for self-documenting expressions and debugging                                        | Complete      |
+  +------------------------------------------------------------------------------------------------------------+---------------+
+
+Other Language Changes:
+
+.. table::
+  :widths: 90 10
+
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | A *continue* statement was illegal in the *finally* clause due to a problem with the implementation. In    | Complete    |
+  | Python 3.8 this restriction was lifted                                                                     |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | The *bool*, *int* , and *fractions.Fraction* types now have an *as_integer_ratio()* method like that found |             |
+  | in *float* and *decimal.Decimal*                                                                           |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Constructors of *int*, *float* and *complex* will now use the *__index__()* special method, if available   |             |
+  | and the corresponding method *__int__()*, *__float__()* or *__complex__()* is not available                |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added support of *\N{name}* escapes in regular expressions                                                 |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Dict and dictviews are now iterable in reversed insertion order using *reversed()*                         |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | The syntax allowed for keyword names in function calls was further restricted. In particular,              |             |
+  | f((keyword)=arg) is no longer allowed                                                                      |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Generalized iterable unpacking in yield and return statements no longer requires enclosing parentheses     |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | When a comma is missed in code such as [(10, 20) (30, 40)], the compiler displays a SyntaxWarning with a   |             |
+  | helpful suggestion                                                                                         |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Arithmetic operations between subclasses of *datetime.date* or *datetime.datetime* and *datetime.timedelta*|             |
+  | objects now return an instance of the subclass, rather than the base class                                 |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | When the Python interpreter is interrupted by *Ctrl-C (SIGINT)* and the resulting *KeyboardInterrupt*      |             |
+  | exception is not caught, the Python process now exits via a SIGINT signal or with the correct exit code    |             |
+  | such that the calling process can detect that it died due to  a *Ctrl-C*                                   |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Some advanced styles of programming require updating the *types.CodeType* object for an existing function  |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | For integers, the three-argument form of the pow() function now permits the exponent to be negative in the |             |
+  | case where the base is relatively prime to the modulus                                                     |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Dict comprehensions have been synced-up with dict literals so that the key is computed first and the value |             |
+  | second                                                                                                     |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | The *object.__reduce__()* method can now return a tuple from two to six elements long                      |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+
+Changes to built-in modules:
+
+.. table::
+  :widths: 90 10
+
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | `asyncio <https://docs.python.org/3/whatsnew/3.8.html#asyncio>`_                                                         |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | *asyncio.run()* has graduated from the provisional to stable API                                           | Complete    |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Running *python -m asyncio* launches a natively async REPL                                                 |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | The exception *asyncio.CancelledError* now inherits from *BaseException* rather than *Exception* and no    | Complete    |
+  | longer inherits from *concurrent.futures.CancelledError*                                                   |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added *asyncio.Task.get_coro()* for getting the wrapped coroutine within an *asyncio.Task*                 |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Asyncio tasks can now be named, either by passing the name keyword argument to *asyncio.create_task()* or  |             |
+  | the *create_task()* event loop method, or by calling the *set_name()* method on the task object            |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added support for Happy Eyeballs to *asyncio.loop.create_connection()*. To specify the behavior, two new   |             |
+  | parameters have been added: *happy_eyeballs_delay* and interleave.                                         |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | `gc <https://docs.python.org/3/whatsnew/3.8.html#gc>`_                                                                   |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | *get_objects()* can now receive an optional generation parameter indicating a generation to get objects    |             |
+  | from. (Note, though, that while *gc* is a built-in, *get_objects()* is not implemented for MicroPython)    |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | `math <https://docs.python.org/3/whatsnew/3.8.html#math>`_                                                               |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added new function *math.dist()* for computing Euclidean distance between two points                       |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Expanded the *math.hypot()* function to handle multiple dimensions                                         |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added new function, *math.prod()*, as analogous function to *sum()* that returns the product of a "start"  |             |
+  | value (default: 1) times an iterable of numbers                                                            |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added two new combinatoric functions *math.perm()* and *math.comb()*                                       |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Added a new function *math.isqrt()* for computing accurate integer square roots without conversion to      |             |
+  | floating point                                                                                             |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | The function *math.factorial()* no longer accepts arguments that are not int-like                          | Complete    |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | `sys <https://docs.python.org/3/whatsnew/3.8.html#sys>`_                                                                 |
+  +------------------------------------------------------------------------------------------------------------+-------------+
+  | Add new *sys.unraisablehook()* function which can be overridden to control how "unraisable exceptions"     |             |
+  | are handled                                                                                                |             |
+  +------------------------------------------------------------------------------------------------------------+-------------+

docs/differences/python_39.rst

@@ -0,0 +1,134 @@
+.. _python_39:
+
+Python 3.9
+==========
+
+Python 3.9.0 (final) was released on the 5th October 2020. The Features for 3.9 are
+defined in `PEP 596 <https://www.python.org/dev/peps/pep-0596/#features-for-3-9>`_
+and a detailed description of the changes can be found in
+`What's New in Python 3.9 <https://docs.python.org/3/whatsnew/3.9.html>`_
+
+.. table::
+  :widths: 20 60 20
+
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | **Features**                                           |                                                    | **Status**           |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 573 <https://www.python.org/dev/peps/pep-0573/>`_ | Fast access to module state from methods of C      | Not relevant         |
+  |                                                        | extension types                                    |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 584 <https://www.python.org/dev/peps/pep-0584/>`_ | Union operators added to dict                      | Complete [#pep584]_  |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 585 <https://www.python.org/dev/peps/pep-0584/>`_ | Type hinting generics in standard collections      |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 593 <https://www.python.org/dev/peps/pep-0593/>`_ | Flexible function and variable annotations         |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 602 <https://www.python.org/dev/peps/pep-0602/>`_ | CPython adopts an annual release cycle. Instead of | Not relevant         |
+  |                                                        | annual, aiming for two month release cycle         |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 614 <https://www.python.org/dev/peps/pep-0614/>`_ | Relaxed grammar restrictions on decorators         |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 615 <https://www.python.org/dev/peps/pep-0615/>`_ | The IANA Time Zone Database is now present in the  |                      |
+  |                                                        | standard library in the zoneinfo module            |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 616 <https://www.python.org/dev/peps/pep-0616/>`_ | String methods to remove prefixes and suffixes     |                      |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+  | `PEP 617 <https://www.python.org/dev/peps/pep-0617/>`_ | CPython now uses a new parser based on PEG         | Not relevant         |
+  +--------------------------------------------------------+----------------------------------------------------+----------------------+
+
+Other Language Changes:
+
+.. table::
+  :widths: 90 10
+
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | *__import__()* now raises *ImportError* instead of *ValueError*                                             | Complete      |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Python now gets the absolute path of the script filename specified on the command line (ex: *python3*       |               |
+  | *script.py*): the *__file__* attribute of the *__main__* module became an absolute path, rather than a      |               |
+  | relative path                                                                                               |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | By default, for best performance, the errors argument is only checked at the first encoding/decoding error  |               |
+  | and the encoding argument is sometimes ignored for empty strings                                            |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | *"".replace("", s, n)* now returns *s* instead of an empty string for all non-zero n. It is now consistent  |               |
+  | with *"".replace("", s)*                                                                                    |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Any valid expression can now be used as a decorator. Previously, the grammar was much more restrictive      |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Parallel running of *aclose()* / *asend()* / *athrow()* is now prohibited, and *ag_running* now reflects    |               |
+  | the actual running status of the async generator                                                            |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Unexpected errors in calling the *__iter__* method are no longer masked by TypeError in the in operator and |               |
+  | functions contains(), indexOf() and countOf() of the operator module                                        |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+  | Unparenthesized lambda expressions can no longer be the expression part in an if clause in comprehensions   |               |
+  | and generator expressions                                                                                   |               |
+  +-------------------------------------------------------------------------------------------------------------+---------------+
+
+Changes to built-in modules:
+
+.. table::
+  :widths: 90 10
+
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `asyncio <https://docs.python.org/3/whatsnew/3.9.html#asyncio>`_                                                              |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Due to significant security concerns, the reuse_address parameter of *asyncio.loop.create_datagram_endpoint()*|               |
+  | is no longer supported                                                                                        |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added a new coroutine *shutdown_default_executor()* that schedules a shutdown for the default executor that   |               |
+  | waits on the *ThreadPoolExecutor* to finish closing. Also, *asyncio.run()* has been updated to use the new    |               |
+  | coroutine.                                                                                                    |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added *asyncio.PidfdChildWatcher*, a Linux-specific child watcher implementation that polls process file      |               |
+  | descriptors                                                                                                   |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | added a new *coroutine asyncio.to_thread()*                                                                   |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | When cancelling the task due to a timeout, *asyncio.wait_for()* will now wait until the cancellation is       |               |
+  | complete also in the case when timeout is <= 0, like it does with positive timeouts                           |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | *asyncio* now raises *TyperError* when calling incompatible methods with an *ssl.SSLSocket* socket            |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `gc <https://docs.python.org/3/whatsnew/3.9.html#gc>`_                                                                        |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Garbage collection does not block on resurrected objects                                                      |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added a new function *gc.is_finalized()* to check if an object has been finalized by the garbage collector    |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `math <https://docs.python.org/3/whatsnew/3.9.html#math>`_                                                                    |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Expanded the *math.gcd()* function to handle multiple arguments. Formerly, it only supported two arguments    |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added *math.lcm()*: return the least common multiple of specified arguments                                   |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added *math.nextafter()*: return the next floating-point value after x towards y                              |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added *math.ulp()*: return the value of the least significant bit of a float                                  |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `os <https://docs.python.org/3/whatsnew/3.9.html#os>`_                                                                        |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Exposed the Linux-specific *os.pidfd_open()* and *os.P_PIDFD*                                                 |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The *os.unsetenv()* function is now also available on Windows                                                 | Complete      |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | The *os.putenv()* and *os.unsetenv()* functions are now always available                                      | Complete      |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  |  Added *os.waitstatus_to_exitcode()* function: convert a wait status to an exit code                          |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `random <https://docs.python.org/3/whatsnew/3.9.html#random>`_                                                                |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added a new *random.Random.randbytes* method: generate random bytes                                           |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | `sys <https://docs.python.org/3/whatsnew/3.9.html#sys>`_                                                                      |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Added a new *sys.platlibdir* attribute: name of the platform-specific library directory                       |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+  | Previously, *sys.stderr* was block-buffered when non-interactive. Now stderr defaults to always being         |               |
+  | line-buffered                                                                                                 |               |
+  +---------------------------------------------------------------------------------------------------------------+---------------+
+
+.. rubric:: Notes
+
+.. [#pep584] PEP 584 ``dict`` union operator is only available on MicroPython builds with ``MICROPY_CPYTHON_COMPAT`` enabled.

docs/esp32/quickref.rst

@@ -16,7 +16,7 @@ working with this board it may be useful to get an overview of the microcontroll
    :maxdepth: 1
 
    general.rst
-   tutorial/intro.rst
+   tutorial/index.rst
 
 Installing MicroPython
 ----------------------
@@ -57,8 +57,7 @@ The :mod:`esp32` module::
 
     import esp32
 
-    esp32.hall_sensor()     # read the internal hall sensor
-    esp32.raw_temperature() # read the internal temperature of the MCU, in Farenheit
+    esp32.raw_temperature() # read the internal temperature of the MCU, in Fahrenheit
     esp32.ULP()             # access to the Ultra-Low-Power Co-processor
 
 Note that the temperature sensor in the ESP32 will typically read higher than
@@ -68,6 +67,9 @@ by reading the temperature sensor immediately after waking up from sleep.
 Networking
 ----------
 
+WLAN
+^^^^
+
 The :mod:`network` module::
 
     import network
@@ -76,12 +78,13 @@ The :mod:`network` module::
     wlan.active(True)       # activate the interface
     wlan.scan()             # scan for access points
     wlan.isconnected()      # check if the station is connected to an AP
-    wlan.connect('essid', 'password') # connect to an AP
+    wlan.connect('ssid', 'key') # connect to an AP
     wlan.config('mac')      # get the interface's MAC address
     wlan.ifconfig()         # get the interface's IP/netmask/gw/DNS addresses
 
     ap = network.WLAN(network.AP_IF) # create access-point interface
-    ap.config(essid='ESP-AP') # set the ESSID of the access point
+    ap.config(ssid='ESP-AP') # set the SSID of the access point
+    ap.config(max_clients=10) # set how many clients can connect to the network
     ap.active(True)         # activate the interface
 
 A useful function for connecting to your local WiFi network is::
@@ -92,19 +95,76 @@ A useful function for connecting to your local WiFi network is::
         wlan.active(True)
         if not wlan.isconnected():
             print('connecting to network...')
-            wlan.connect('essid', 'password')
+            wlan.connect('ssid', 'key')
             while not wlan.isconnected():
                 pass
         print('network config:', wlan.ifconfig())
 
-Once the network is established the :mod:`socket <usocket>` module can be used
-to create and use TCP/UDP sockets as usual, and the ``urequests`` module for
+Once the network is established the :mod:`socket <socket>` module can be used
+to create and use TCP/UDP sockets as usual, and the ``requests`` module for
 convenient HTTP requests.
 
+After a call to ``wlan.connect()``, the device will by default retry to connect
+**forever**, even when the authentication failed or no AP is in range.
+``wlan.status()`` will return ``network.STAT_CONNECTING`` in this state until a
+connection succeeds or the interface gets disabled.  This can be changed by
+calling ``wlan.config(reconnects=n)``, where n are the number of desired reconnect
+attempts (0 means it won't retry, -1 will restore the default behaviour of trying
+to reconnect forever).
+
+LAN
+^^^
+
+To use the wired interfaces one has to specify the pins and mode ::
+
+    import network
+
+    lan = network.LAN(mdc=PIN_MDC, ...)   # Set the pin and mode configuration
+    lan.active(True)                      # activate the interface
+    lan.ifconfig()                        # get the interface's IP/netmask/gw/DNS addresses
+
+
+The keyword arguments for the constructor defining the PHY type and interface are:
+
+- mdc=pin-object    # set the mdc and mdio pins.
+- mdio=pin-object
+- power=pin-object  # set the pin which switches the power of the PHY device.
+- phy_type=<type>   # Select the PHY device type. Supported devices are PHY_LAN8710,
+  PHY_LAN8720, PH_IP101, PHY_RTL8201, PHY_DP83848 and PHY_KSZ8041
+- phy_addr=number   # The address number of the PHY device.
+- ref_clk_mode=mode # Defines, whether the ref_clk at the ESP32 is an input
+  or output. Suitable values are Pin.IN and Pin.OUT.
+- ref_clk=pin-object  # defines the Pin used for ref_clk.
+
+These are working configurations for LAN interfaces of popular boards::
+
+    # Olimex ESP32-GATEWAY: power controlled by Pin(5)
+    # Olimex ESP32 PoE and ESP32-PoE ISO: power controlled by Pin(12)
+
+    lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), power=machine.Pin(5),
+                      phy_type=network.PHY_LAN8720, phy_addr=0,
+                      ref_clk=machine.Pin(17), ref_clk_mode=machine.Pin.OUT)
+
+    # Wireless-Tag's WT32-ETH01
+
+    lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18),
+                      phy_type=network.PHY_LAN8720, phy_addr=1, power=None)
+
+    # Wireless-Tag's WT32-ETH01 v1.4
+
+    lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18),
+                      phy_type=network.PHY_LAN8720, phy_addr=1,
+                      power=machine.Pin(16))
+
+    # Espressif ESP32-Ethernet-Kit_A_V1.2
+
+    lan = network.LAN(id=0, mdc=Pin(23), mdio=Pin(18), power=Pin(5),
+                      phy_type=network.PHY_IP101, phy_addr=1)
+
 Delay and timing
 ----------------
 
-Use the :mod:`time <utime>` module::
+Use the :mod:`time <time>` module::
 
     import time
 
@@ -117,17 +177,21 @@ Use the :mod:`time <utime>` module::
 Timers
 ------
 
-Virtual (RTOS-based) timers are supported. Use the :ref:`machine.Timer <machine.Timer>` class
-with timer ID of -1::
+The ESP32 port has four hardware timers. Use the :ref:`machine.Timer <machine.Timer>` class
+with a timer ID from 0 to 3 (inclusive)::
 
     from machine import Timer
 
-    tim = Timer(-1)
-    tim.init(period=5000, mode=Timer.ONE_SHOT, callback=lambda t:print(1))
-    tim.init(period=2000, mode=Timer.PERIODIC, callback=lambda t:print(2))
+    tim0 = Timer(0)
+    tim0.init(period=5000, mode=Timer.ONE_SHOT, callback=lambda t:print(0))
+
+    tim1 = Timer(1)
+    tim1.init(period=2000, mode=Timer.PERIODIC, callback=lambda t:print(1))
 
 The period is in milliseconds.
 
+Virtual timers are not currently supported on this port.
+
 .. _Pins_and_GPIO:
 
 Pins and GPIO
@@ -147,6 +211,7 @@ Use the :ref:`machine.Pin <machine.Pin>` class::
 
     p4 = Pin(4, Pin.IN, Pin.PULL_UP) # enable internal pull-up resistor
     p5 = Pin(5, Pin.OUT, value=1) # set pin high on creation
+    p6 = Pin(6, Pin.OUT, drive=Pin.DRIVE_3) # set maximum drive strength
 
 Available Pins are from the following ranges (inclusive): 0-19, 21-23, 25-27, 32-39.
 These correspond to the actual GPIO pin numbers of ESP32 chip.  Note that many
@@ -154,6 +219,23 @@ end-user boards use their own adhoc pin numbering (marked e.g. D0, D1, ...).
 For mapping between board logical pins and physical chip pins consult your board
 documentation.
 
+Four drive strengths are supported, using the ``drive`` keyword argument to the
+``Pin()`` constructor or ``Pin.init()`` method, with different corresponding
+safe maximum source/sink currents and approximate internal driver resistances:
+
+ - ``Pin.DRIVE_0``: 5mA / 130 ohm
+ - ``Pin.DRIVE_1``: 10mA / 60 ohm
+ - ``Pin.DRIVE_2``: 20mA / 30 ohm (default strength if not configured)
+ - ``Pin.DRIVE_3``: 40mA / 15 ohm
+
+The ``hold=`` keyword argument to ``Pin()`` and ``Pin.init()`` will enable the
+ESP32 "pad hold" feature. When set to ``True``, the pin configuration
+(direction, pull resistors and output value) will be held and any further
+changes (including changing the output level) will not be applied. Setting
+``hold=False`` will immediately apply any outstanding pin configuration changes
+and release the pin. Using ``hold=True`` while a pin is already held will apply
+any configuration changes and then immediately reapply the hold.
+
 Notes:
 
 * Pins 1 and 3 are REPL UART TX and RX respectively
@@ -163,96 +245,213 @@ Notes:
 
 * Pins 34-39 are input only, and also do not have internal pull-up resistors
 
-* The pull value of some pins can be set to ``Pin.PULL_HOLD`` to reduce power
-  consumption during deepsleep.
+* See :ref:`Deep_sleep_Mode` for a discussion of pin behaviour during sleep
+
+There's a higher-level abstraction :ref:`machine.Signal <machine.Signal>`
+which can be used to invert a pin. Useful for illuminating active-low LEDs
+using ``on()`` or ``value(1)``.
+
+UART (serial bus)
+-----------------
+
+See :ref:`machine.UART <machine.UART>`. ::
+
+    from machine import UART
+
+    uart1 = UART(1, baudrate=9600, tx=33, rx=32)
+    uart1.write('hello')  # write 5 bytes
+    uart1.read(5)         # read up to 5 bytes
+
+The ESP32 has three hardware UARTs: UART0, UART1 and UART2.
+They each have default GPIO assigned to them, however depending on your
+ESP32 variant and board, these pins may conflict with embedded flash,
+onboard PSRAM or peripherals.
+
+Any GPIO can be used for hardware UARTs using the GPIO matrix, except for
+input-only pins 34-39 that can be used as ``rx``. To avoid conflicts simply
+provide ``tx`` and ``rx`` pins when constructing. The default pins listed
+below.
+
+=====  =====  =====  =====
+\      UART0  UART1  UART2
+=====  =====  =====  =====
+tx     1      10     17
+rx     3      9      16
+=====  =====  =====  =====
 
 PWM (pulse width modulation)
 ----------------------------
 
 PWM can be enabled on all output-enabled pins. The base frequency can
 range from 1Hz to 40MHz but there is a tradeoff; as the base frequency
-*increases* the duty resolution *decreases*. See 
+*increases* the duty resolution *decreases*. See
 `LED Control <https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/peripherals/ledc.html>`_
 for more details.
 
-Use the ``machine.PWM`` class::
+Use the :ref:`machine.PWM <machine.PWM>` class::
 
     from machine import Pin, PWM
 
-    pwm0 = PWM(Pin(0))      # create PWM object from a pin
-    pwm0.freq()             # get current frequency
-    pwm0.freq(1000)         # set frequency
-    pwm0.duty()             # get current duty cycle
-    pwm0.duty(200)          # set duty cycle
-    pwm0.deinit()           # turn off PWM on the pin
+    pwm0 = PWM(Pin(0), freq=5000, duty_u16=32768) # create PWM object from a pin
+    freq = pwm0.freq()         # get current frequency
+    pwm0.freq(1000)            # set PWM frequency from 1Hz to 40MHz
 
-    pwm2 = PWM(Pin(2), freq=20000, duty=512) # create and configure in one go
+    duty = pwm0.duty()         # get current duty cycle, range 0-1023 (default 512, 50%)
+    pwm0.duty(256)             # set duty cycle from 0 to 1023 as a ratio duty/1023, (now 25%)
+
+    duty_u16 = pwm0.duty_u16() # get current duty cycle, range 0-65535
+    pwm0.duty_u16(2**16*3//4)  # set duty cycle from 0 to 65535 as a ratio duty_u16/65535, (now 75%)
+
+    duty_ns = pwm0.duty_ns()   # get current pulse width in ns
+    pwm0.duty_ns(250_000)      # set pulse width in nanoseconds from 0 to 1_000_000_000/freq, (now 25%)
+
+    pwm0.deinit()              # turn off PWM on the pin
+
+    pwm2 = PWM(Pin(2), freq=20000, duty=512)  # create and configure in one go
+    print(pwm2)                               # view PWM settings
+
+ESP chips have different hardware peripherals:
+
+=====================================================  ========  ========  ========
+Hardware specification                                    ESP32  ESP32-S2  ESP32-C3
+-----------------------------------------------------  --------  --------  --------
+Number of groups (speed modes)                                2         1         1
+Number of timers per group                                    4         4         4
+Number of channels per group                                  8         8         6
+-----------------------------------------------------  --------  --------  --------
+Different PWM frequencies (groups * timers)                   8         4         4
+Total PWM channels (Pins, duties) (groups * channels)        16         8         6
+=====================================================  ========  ========  ========
+
+A maximum number of PWM channels (Pins) are available on the ESP32 - 16 channels,
+but only 8 different PWM frequencies are available, the remaining 8 channels must
+have the same frequency.  On the other hand, 16 independent PWM duty cycles are
+possible at the same frequency.
+
+See more examples in the :ref:`esp32_pwm` tutorial.
+
+DAC (digital to analog conversion)
+----------------------------------
+
+On the ESP32, DAC functionality is available on pins 25, 26.
+On the ESP32S2, DAC functionality is available on pins 17, 18.
+
+Use the DAC::
+
+    from machine import DAC, Pin
+
+    dac = DAC(Pin(25))  # create an DAC object acting on a pin
+    dac.write(128)      # set a raw analog value in the range 0-255, 50% now
 
 ADC (analog to digital conversion)
 ----------------------------------
 
-On the ESP32 ADC functionality is available on Pins 32-39. Note that, when
-using the default configuration, input voltages on the ADC pin must be between
-0.0v and 1.0v (anything above 1.0v will just read as 4095).  Attenuation must
-be applied in order to increase this usable voltage range.
+On the ESP32, ADC functionality is available on pins 32-39 (ADC block 1) and
+pins 0, 2, 4, 12-15 and 25-27 (ADC block 2).
 
 Use the :ref:`machine.ADC <machine.ADC>` class::
 
     from machine import ADC
 
-    adc = ADC(Pin(32))          # create ADC object on ADC pin
-    adc.read()                  # read value, 0-4095 across voltage range 0.0v - 1.0v
+    adc = ADC(pin)        # create an ADC object acting on a pin
+    val = adc.read_u16()  # read a raw analog value in the range 0-65535
+    val = adc.read_uv()   # read an analog value in microvolts
 
-    adc.atten(ADC.ATTN_11DB)    # set 11dB input attenuation (voltage range roughly 0.0v - 3.6v)
-    adc.width(ADC.WIDTH_9BIT)   # set 9 bit return values (returned range 0-511)
-    adc.read()                  # read value using the newly configured attenuation and width
+ADC block 2 is also used by WiFi and so attempting to read analog values from
+block 2 pins when WiFi is active will raise an exception.
 
-ESP32 specific ADC class method reference:
+The internal ADC reference voltage is typically 1.1V, but varies slightly from
+package to package. The ADC is less linear close to the reference voltage
+(particularly at higher attenuations) and has a minimum measurement voltage
+around 100mV, voltages at or below this will read as 0. To read voltages
+accurately, it is recommended to use the ``read_uv()`` method (see below).
 
-.. method:: ADC.atten(attenuation)
+ESP32-specific ADC class method reference:
 
-    This method allows for the setting of the amount of attenuation on the
-    input of the ADC. This allows for a wider possible input voltage range,
-    at the cost of accuracy (the same number of bits now represents a wider
-    range). The possible attenuation options are:
+.. class:: ADC(pin, *, atten)
 
-      - ``ADC.ATTN_0DB``: 0dB attenuation, gives a maximum input voltage
-        of 1.00v - this is the default configuration
-      - ``ADC.ATTN_2_5DB``: 2.5dB attenuation, gives a maximum input voltage
-        of approximately 1.34v
-      - ``ADC.ATTN_6DB``: 6dB attenuation, gives a maximum input voltage
-        of approximately 2.00v
-      - ``ADC.ATTN_11DB``: 11dB attenuation, gives a maximum input voltage
-        of approximately 3.6v
+    Return the ADC object for the specified pin. ESP32 does not support
+    different timings for ADC sampling and so the ``sample_ns`` keyword argument
+    is not supported.
+
+    To read voltages above the reference voltage, apply input attenuation with
+    the ``atten`` keyword argument. Valid values (and approximate linear
+    measurement ranges) are:
+
+      - ``ADC.ATTN_0DB``: No attenuation (100mV - 950mV)
+      - ``ADC.ATTN_2_5DB``: 2.5dB attenuation (100mV - 1250mV)
+      - ``ADC.ATTN_6DB``: 6dB attenuation (150mV - 1750mV)
+      - ``ADC.ATTN_11DB``: 11dB attenuation (150mV - 2450mV)
 
 .. Warning::
-   Despite 11dB attenuation allowing for up to a 3.6v range, note that the
-   absolute maximum voltage rating for the input pins is 3.6v, and so going
-   near this boundary may be damaging to the IC!
+   Note that the absolute maximum voltage rating for input pins is 3.6V. Going
+   near to this boundary risks damage to the IC!
 
-.. method:: ADC.width(width)
+.. method:: ADC.read_uv()
 
-    This method allows for the setting of the number of bits to be utilised
-    and returned during ADC reads. Possible width options are:
+    This method uses the known characteristics of the ADC and per-package eFuse
+    values - set during manufacture - to return a calibrated input voltage
+    (before attenuation) in microvolts. The returned value has only millivolt
+    resolution (i.e., will always be a multiple of 1000 microvolts).
+
+    The calibration is only valid across the linear range of the ADC. In
+    particular, an input tied to ground will read as a value above 0 microvolts.
+    Within the linear range, however, more accurate and consistent results will
+    be obtained than using `read_u16()` and scaling the result with a constant.
+
+The ESP32 port also supports the :ref:`machine.ADC <machine.ADCBlock>` API:
+
+.. class:: ADCBlock(id, *, bits)
+
+    Return the ADC block object with the given ``id`` (1 or 2) and initialize
+    it to the specified resolution (9 to 12-bits depending on the ESP32 series)
+    or the highest supported resolution if not specified.
+
+.. method:: ADCBlock.connect(pin)
+            ADCBlock.connect(channel)
+            ADCBlock.connect(channel, pin)
+
+    Return the ``ADC`` object for the specified ADC pin or channel number.
+    Arbitrary connection of ADC channels to GPIO is not supported and so
+    specifying a pin that is not connected to this block, or specifying a
+    mismatched channel and pin, will raise an exception.
+
+Legacy methods:
+
+.. method:: ADC.read()
+
+    This method returns the raw ADC value ranged according to the resolution of
+    the block, e.g., 0-4095 for 12-bit resolution.
+
+.. method:: ADC.atten(atten)
+
+    Equivalent to ``ADC.init(atten=atten)``.
+
+.. method:: ADC.width(bits)
+
+    Equivalent to ``ADC.block().init(bits=bits)``.
+
+For compatibility, the ``ADC`` object also provides constants matching the
+supported ADC resolutions:
+
+  - ``ADC.WIDTH_9BIT`` = 9
+  - ``ADC.WIDTH_10BIT`` = 10
+  - ``ADC.WIDTH_11BIT`` = 11
+  - ``ADC.WIDTH_12BIT`` = 12
 
-      - ``ADC.WIDTH_9BIT``: 9 bit data
-      - ``ADC.WIDTH_10BIT``: 10 bit data
-      - ``ADC.WIDTH_11BIT``: 11 bit data
-      - ``ADC.WIDTH_12BIT``: 12 bit data - this is the default configuration
 
 Software SPI bus
 ----------------
 
-There are two SPI drivers. One is implemented in software (bit-banging)
-and works on all pins, and is accessed via the :ref:`machine.SPI <machine.SPI>`
-class::
+Software SPI (using bit-banging) works on all pins, and is accessed via the
+:ref:`machine.SoftSPI <machine.SoftSPI>` class::
 
-    from machine import Pin, SPI
+    from machine import Pin, SoftSPI
 
-    # construct an SPI bus on the given pins
+    # construct a SoftSPI bus on the given pins
     # polarity is the idle state of SCK
     # phase=0 means sample on the first edge of SCK, phase=1 means the second
-    spi = SPI(baudrate=100000, polarity=1, phase=0, sck=Pin(0), mosi=Pin(2), miso=Pin(4))
+    spi = SoftSPI(baudrate=100000, polarity=1, phase=0, sck=Pin(0), mosi=Pin(2), miso=Pin(4))
 
     spi.init(baudrate=200000) # set the baudrate
 
@@ -271,7 +470,7 @@ class::
 
 .. Warning::
    Currently *all* of ``sck``, ``mosi`` and ``miso`` *must* be specified when
-   initialising Software SPI. 
+   initialising Software SPI.
 
 Hardware SPI bus
 ----------------
@@ -292,38 +491,72 @@ mosi   13           23
 miso   12           19
 =====  ===========  ============
 
-Hardware SPI has the same methods as Software SPI above::
+Hardware SPI is accessed via the :ref:`machine.SPI <machine.SPI>` class and
+has the same methods as software SPI above::
 
     from machine import Pin, SPI
 
+    hspi = SPI(1, 10000000)
     hspi = SPI(1, 10000000, sck=Pin(14), mosi=Pin(13), miso=Pin(12))
     vspi = SPI(2, baudrate=80000000, polarity=0, phase=0, bits=8, firstbit=0, sck=Pin(18), mosi=Pin(23), miso=Pin(19))
 
+Software I2C bus
+----------------
 
-I2C bus
--------
+Software I2C (using bit-banging) works on all output-capable pins, and is
+accessed via the :ref:`machine.SoftI2C <machine.SoftI2C>` class::
 
-The I2C driver has both software and hardware implementations, and the two
-hardware peripherals have identifiers 0 and 1.  Any available output-capable
-pins can be used for SCL and SDA.  The driver is accessed via the
-:ref:`machine.I2C <machine.I2C>` class::
+    from machine import Pin, SoftI2C
+
+    i2c = SoftI2C(scl=Pin(5), sda=Pin(4), freq=100000)
+
+    i2c.scan()              # scan for devices
+
+    i2c.readfrom(0x3a, 4)   # read 4 bytes from device with address 0x3a
+    i2c.writeto(0x3a, '12') # write '12' to device with address 0x3a
+
+    buf = bytearray(10)     # create a buffer with 10 bytes
+    i2c.writeto(0x3a, buf)  # write the given buffer to the peripheral
+
+Hardware I2C bus
+----------------
+
+There are two hardware I2C peripherals with identifiers 0 and 1.  Any available
+output-capable pins can be used for SCL and SDA but the defaults are given
+below.
+
+=====  ===========  ============
+\      I2C(0)       I2C(1)
+=====  ===========  ============
+scl    18           25
+sda    19           26
+=====  ===========  ============
+
+The driver is accessed via the :ref:`machine.I2C <machine.I2C>` class and
+has the same methods as software I2C above::
 
     from machine import Pin, I2C
 
-    # construct a software I2C bus
-    i2c = I2C(scl=Pin(5), sda=Pin(4), freq=100000)
-
-    # construct a hardware I2C bus
     i2c = I2C(0)
     i2c = I2C(1, scl=Pin(5), sda=Pin(4), freq=400000)
 
-    i2c.scan()              # scan for slave devices
+I2S bus
+-------
 
-    i2c.readfrom(0x3a, 4)   # read 4 bytes from slave device with address 0x3a
-    i2c.writeto(0x3a, '12') # write '12' to slave device with address 0x3a
+See :ref:`machine.I2S <machine.I2S>`. ::
 
-    buf = bytearray(10)     # create a buffer with 10 bytes
-    i2c.writeto(0x3a, buf)  # write the given buffer to the slave
+    from machine import I2S, Pin
+
+    i2s = I2S(0, sck=Pin(13), ws=Pin(14), sd=Pin(34), mode=I2S.TX, bits=16, format=I2S.STEREO, rate=44100, ibuf=40000) # create I2S object
+    i2s.write(buf)             # write buffer of audio samples to I2S device
+
+    i2s = I2S(1, sck=Pin(33), ws=Pin(25), sd=Pin(32), mode=I2S.RX, bits=16, format=I2S.MONO, rate=22050, ibuf=40000) # create I2S object
+    i2s.readinto(buf)          # fill buffer with audio samples from I2S device
+
+The I2S class is currently available as a Technical Preview.  During the preview period, feedback from
+users is encouraged.  Based on this feedback, the I2S class API and implementation may be changed.
+
+ESP32 has two I2S buses with id=0 and id=1
 
 Real time clock (RTC)
 ---------------------
@@ -336,6 +569,19 @@ See :ref:`machine.RTC <machine.RTC>` ::
     rtc.datetime((2017, 8, 23, 1, 12, 48, 0, 0)) # set a specific date and time
     rtc.datetime() # get date and time
 
+WDT (Watchdog timer)
+--------------------
+
+See :ref:`machine.WDT <machine.WDT>`. ::
+
+    from machine import WDT
+
+    # enable the WDT with a timeout of 5s (1s is the minimum)
+    wdt = WDT(timeout=5000)
+    wdt.feed()
+
+.. _Deep_sleep_mode:
+
 Deep-sleep mode
 ---------------
 
@@ -355,15 +601,64 @@ Notes:
 * Calling ``deepsleep()`` without an argument will put the device to sleep
   indefinitely
 * A software reset does not change the reset cause
-* There may be some leakage current flowing through enabled internal pullups.
-  To further reduce power consumption it is possible to disable the internal pullups::
 
-    p1 = Pin(4, Pin.IN, Pin.PULL_HOLD)
+Some ESP32 pins (0, 2, 4, 12-15, 25-27, 32-39) are connected to the RTC during
+deep-sleep and can be used to wake the device with the ``wake_on_`` functions in
+the :mod:`esp32` module. The output-capable RTC pins (all except 34-39) will
+also retain their pull-up or pull-down resistor configuration when entering
+deep-sleep.
 
-  After leaving deepsleep it may be necessary to un-hold the pin explicitly (e.g. if
-  it is an output pin) via::
+If the pull resistors are not actively required during deep-sleep and are likely
+to cause current leakage (for example a pull-up resistor is connected to ground
+through a switch), then they should be disabled to save power before entering
+deep-sleep mode::
 
-    p1 = Pin(4, Pin.OUT, None)
+    from machine import Pin, deepsleep
+
+    # configure input RTC pin with pull-up on boot
+    pin = Pin(2, Pin.IN, Pin.PULL_UP)
+
+    # disable pull-up and put the device to sleep for 10 seconds
+    pin.init(pull=None)
+    machine.deepsleep(10000)
+
+Output-configured RTC pins will also retain their output direction and level in
+deep-sleep if pad hold is enabled with the ``hold=True`` argument to
+``Pin.init()``.
+
+Non-RTC GPIO pins will be disconnected by default on entering deep-sleep.
+Configuration of non-RTC pins - including output level - can be retained by
+enabling pad hold on the pin and enabling GPIO pad hold during deep-sleep::
+
+    from machine import Pin, deepsleep
+    import esp32
+
+    opin = Pin(19, Pin.OUT, value=1, hold=True) # hold output level
+    ipin = Pin(21, Pin.IN, Pin.PULL_UP, hold=True) # hold pull-up
+
+    # enable pad hold in deep-sleep for non-RTC GPIO
+    esp32.gpio_deep_sleep_hold(True)
+
+    # put the device to sleep for 10 seconds
+    deepsleep(10000)
+
+The pin configuration - including the pad hold - will be retained on wake from
+sleep. See :ref:`Pins_and_GPIO` above for a further discussion of pad holding.
+
+SD card
+-------
+
+See :ref:`machine.SDCard <machine.SDCard>`. ::
+
+    import machine, os
+
+    # Slot 2 uses pins sck=18, cs=5, miso=19, mosi=23
+    sd = machine.SDCard(slot=2)
+    os.mount(sd, '/sd')  # mount
+
+    os.listdir('/sd')    # list directory contents
+
+    os.umount('/sd')     # eject
 
 RMT
 ---
@@ -377,7 +672,7 @@ The RMT is ESP32-specific and allows generation of accurate digital pulses with
     r = esp32.RMT(0, pin=Pin(18), clock_div=8)
     r   # RMT(channel=0, pin=18, source_freq=80000000, clock_div=8)
     # The channel resolution is 100ns (1/(source_freq/clock_div)).
-    r.write_pulses((1, 20, 2, 40), start=0) # Send 0 for 100ns, 1 for 2000ns, 0 for 200ns, 1 for 4000ns
+    r.write_pulses((1, 20, 2, 40), 0) # Send 0 for 100ns, 1 for 2000ns, 0 for 200ns, 1 for 4000ns
 
 OneWire driver
 --------------
@@ -409,10 +704,10 @@ Be sure to put a 4.7k pull-up resistor on the data line.  Note that
 the ``convert_temp()`` method must be called each time you want to
 sample the temperature.
 
-NeoPixel driver
----------------
+NeoPixel and APA106 driver
+--------------------------
 
-Use the ``neopixel`` module::
+Use the ``neopixel`` and ``apa106`` modules::
 
     from machine import Pin
     from neopixel import NeoPixel
@@ -423,10 +718,12 @@ Use the ``neopixel`` module::
     np.write()              # write data to all pixels
     r, g, b = np[0]         # get first pixel colour
 
-For low-level driving of a NeoPixel::
 
-    import esp
-    esp.neopixel_write(pin, grb_buf, is800khz)
+The APA106 driver extends NeoPixel, but internally uses a different colour order::
+
+    from apa106 import APA106
+    ap = APA106(pin, 8)
+    r, g, b = ap[0]
 
 .. Warning::
    By default ``NeoPixel`` is configured to control the more popular *800kHz*
@@ -434,8 +731,13 @@ For low-level driving of a NeoPixel::
    400kHz) devices by passing ``timing=0`` when constructing the
    ``NeoPixel`` object.
 
+For low-level driving of a NeoPixel see `machine.bitstream`.
+This low-level driver uses an RMT channel by default.  To configure this see
+`RMT.bitstream_channel`.
 
-Capacitive Touch
+APA102 (DotStar) uses a different driver as it has an additional clock pin.
+
+Capacitive touch
 ----------------
 
 Use the ``TouchPad`` class in the ``machine`` module::
@@ -443,11 +745,11 @@ Use the ``TouchPad`` class in the ``machine`` module::
     from machine import TouchPad, Pin
 
     t = TouchPad(Pin(14))
-    t.read()              # Returns a smaller number when touched 
+    t.read()              # Returns a smaller number when touched
 
 ``TouchPad.read`` returns a value relative to the capacitive variation. Small numbers (typically in
-the *tens*) are common when a pin is touched, larger numbers (above *one thousand*) when 
-no touch is present. However the values are *relative* and can vary depending on the board 
+the *tens*) are common when a pin is touched, larger numbers (above *one thousand*) when
+no touch is present. However the values are *relative* and can vary depending on the board
 and surrounding composition so some calibration may be required.
 
 There are ten capacitive touch-enabled pins that can be used on the ESP32: 0, 2, 4, 12, 13

docs/esp32/tutorial/index.rst

@@ -0,0 +1,23 @@
+.. _esp32_tutorial:
+
+MicroPython tutorial for ESP32
+==============================
+
+This tutorial is intended to get you started using MicroPython on the ESP32
+system-on-a-chip.  If it is your first time it is recommended to follow the
+tutorial through in the order below.  Otherwise the sections are mostly self
+contained, so feel free to skip to those that interest you.
+
+The tutorial does not assume that you know Python, but it also does not attempt
+to explain any of the details of the Python language.  Instead it provides you
+with commands that are ready to run, and hopes that you will gain a bit of
+Python knowledge along the way.  To learn more about Python itself please refer
+to `<https://www.python.org>`__.
+
+.. toctree::
+   :maxdepth: 1
+   :numbered:
+
+   intro.rst
+   pwm.rst
+   peripheral_access.rst

docs/esp32/tutorial/intro.rst

@@ -17,7 +17,7 @@ Requirements
 The first thing you need is a board with an ESP32 chip.  The MicroPython
 software supports the ESP32 chip itself and any board should work.  The main
 characteristic of a board is how the GPIO pins are connected to the outside
-world, and whether it includes a built-in USB-serial convertor to make the
+world, and whether it includes a built-in USB-serial converter to make the
 UART available to your PC.
 
 Names of pins will be given in this tutorial using the chip names (eg GPIO2)
@@ -34,8 +34,8 @@ Please refer to the documentation for your board for further details.
 Getting the firmware
 --------------------
 
-The first thing you need to do is download the most recent MicroPython firmware 
-.bin file to load onto your ESP32 device. You can download it from the  
+The first thing you need to do is download the most recent MicroPython firmware
+.bin file to load onto your ESP32 device. You can download it from the
 `MicroPython downloads page <https://micropython.org/download#esp32>`_.
 From here, you have 3 main choices:
 
@@ -59,7 +59,7 @@ bootloader mode, and second you need to copy across the firmware.  The exact
 procedure for these steps is highly dependent on the particular board and you will
 need to refer to its documentation for details.
 
-Fortunately, most boards have a USB connector, a USB-serial convertor, and the DTR
+Fortunately, most boards have a USB connector, a USB-serial converter, and the DTR
 and RTS pins wired in a special way then deploying the firmware should be easy as
 all steps can be done automatically.  Boards that have such features
 include the Adafruit Feather HUZZAH32, M5Stack, Wemos LOLIN32, and TinyPICO
@@ -104,7 +104,7 @@ Serial prompt
 
 Once you have the firmware on the device you can access the REPL (Python prompt)
 over UART0 (GPIO1=TX, GPIO3=RX), which might be connected to a USB-serial
-convertor, depending on your board.  The baudrate is 115200.
+converter, depending on your board.  The baudrate is 115200.
 
 From here you can now follow the ESP8266 tutorial, because these two Espressif chips
 are very similar when it comes to using MicroPython on them.  The ESP8266 tutorial
@@ -124,7 +124,7 @@ after it, here are troubleshooting recommendations:
 
 * The flashing instructions above use flashing speed of 460800 baud, which is
   good compromise between speed and stability. However, depending on your
-  module/board, USB-UART convertor, cables, host OS, etc., the above baud
+  module/board, USB-UART converter, cables, host OS, etc., the above baud
   rate may be too high and lead to errors. Try a more common 115200 baud
   rate instead in such cases.
 

docs/esp32/tutorial/peripheral_access.rst

@@ -0,0 +1,124 @@
+Accessing peripherals directly via registers
+============================================
+
+The ESP32's peripherals can be controlled via direct register reads and writes.
+This requires reading the datasheet to know what registers to use and what
+values to write to them.  The following example shows how to turn on and change
+the prescaler of the MCPWM0 peripheral.
+
+.. code-block:: python3
+
+    from micropython import const
+    from machine import mem32
+
+    # Define the register addresses that will be used.
+    DR_REG_DPORT_BASE = const(0x3FF00000)
+    DPORT_PERIP_CLK_EN_REG = const(DR_REG_DPORT_BASE + 0x0C0)
+    DPORT_PERIP_RST_EN_REG = const(DR_REG_DPORT_BASE + 0x0C4)
+    DPORT_PWM0_CLK_EN = const(1 << 17)
+    MCPWM0 = const(0x3FF5E000)
+    MCPWM1 = const(0x3FF6C000)
+
+    # Enable CLK and disable RST.
+    print(hex(mem32[DPORT_PERIP_CLK_EN_REG] & 0xffffffff))
+    print(hex(mem32[DPORT_PERIP_RST_EN_REG] & 0xffffffff))
+    mem32[DPORT_PERIP_CLK_EN_REG] |= DPORT_PWM0_CLK_EN
+    mem32[DPORT_PERIP_RST_EN_REG] &= ~DPORT_PWM0_CLK_EN
+    print(hex(mem32[DPORT_PERIP_CLK_EN_REG] & 0xffffffff))
+    print(hex(mem32[DPORT_PERIP_RST_EN_REG] & 0xffffffff))
+
+    # Change the MCPWM0 prescaler.
+    print(hex(mem32[MCPWM0])) # read PWM_CLK_CFG_REG (reset value = 0)
+    mem32[MCPWM0] = 0x55      # change PWM_CLK_PRESCALE
+    print(hex(mem32[MCPWM0])) # read PWM_CLK_CFG_REG
+
+Note that before a peripheral can be used its clock must be enabled and it must
+be taken out of reset.  In the above example the following registers are used
+for this:
+
+- ``DPORT_PERI_CLK_EN_REG``: used to enable a peripheral clock
+
+- ``DPORT_PERI_RST_EN_REG``: used to reset (or take out of reset) a peripheral
+
+The MCPWM0 peripheral is in bit position 17 of the above two registers, hence
+the value of ``DPORT_PWM0_CLK_EN``.
+
+Synchronous access to pins directly via registers
+-------------------------------------------------
+
+The following code shows how to access pins directly via registers.  It has been
+tested on a generic ESP32 board.  It configures pins 16, 17, 32 and 33 in output
+mode via registers, and switches pin output values via registers.  Pins 16 and
+17 are switched simultaneously.
+
+.. code-block:: python3
+
+    from micropython import const
+    from machine import mem32, Pin
+
+    GPIO_OUT_REG = const(0x3FF44004)  # GPIO 0-31 output register
+    GPIO_OUT1_REG = const(0x3FF44010)  # GPIO 32-39 output register
+
+    GPIO_ENABLE_REG = const(0x3FF44020)  # GPIO 0-31 output enable register
+    GPIO_ENABLE1_REG = const(0x3FF4402C)  # GPIO 32-39 output enable register
+
+    M16 = 1 << 16  # Pin(16) bit mask
+    M17 = 1 << 17  # Pin(17) bit mask
+
+    M32 = 1 << (32-32)  # Pin(32) bit mask
+    M33 = 1 << (33-32)  # Pin(33) bit mask
+
+    # Enable pin output mode like
+    # p16 = Pin(16, mode=Pin.OUT)
+    # p17 = Pin(17, mode=Pin.OUT)
+    # p32 = Pin(32, mode=Pin.OUT)
+    # p33 = Pin(33, mode=Pin.OUT)
+    mem32[GPIO_ENABLE_REG] = mem32[GPIO_ENABLE_REG] | M16 | M17
+    mem32[GPIO_ENABLE1_REG] = mem32[GPIO_ENABLE1_REG] | M32 | M33
+
+    print(hex(mem32[GPIO_OUT_REG]), hex(mem32[GPIO_OUT1_REG]))
+
+    # Set outputs to 1 like
+    # p16(1)
+    # p17(1)
+    # p32(1)
+    # p33(1)
+    mem32[GPIO_OUT_REG] = mem32[GPIO_OUT_REG] | M16 | M17
+    mem32[GPIO_OUT1_REG] = mem32[GPIO_OUT1_REG] | M32 | M33
+
+    print(hex(mem32[GPIO_OUT_REG]), hex(mem32[GPIO_OUT1_REG]))
+
+    # Set outputs to 0 like
+    # p16(0)
+    # p17(0)
+    # p32(0)
+    # p33(0)
+    mem32[GPIO_OUT_REG] = mem32[GPIO_OUT_REG] & ~(M16 | M17)
+    mem32[GPIO_OUT1_REG] = mem32[GPIO_OUT1_REG] & ~(M32 | M33)
+
+    print(hex(mem32[GPIO_OUT_REG]), hex(mem32[GPIO_OUT1_REG]))
+
+    while True:
+        # Set outputs to 1
+        mem32[GPIO_OUT_REG] = mem32[GPIO_OUT_REG] | M16 | M17
+        mem32[GPIO_OUT1_REG] = mem32[GPIO_OUT1_REG] | M32 | M33
+
+        # Set outputs to 0
+        mem32[GPIO_OUT_REG] = mem32[GPIO_OUT_REG] & ~(M16 | M17)
+        mem32[GPIO_OUT1_REG] = mem32[GPIO_OUT1_REG] & ~(M32 | M33)
+
+
+Output is::
+
+    0x0 0x0
+    0x30000 0x3
+    0x0 0x0
+
+Pins 16 and 17 are switched synchronously:
+
+.. image:: img/mem32_gpio_output.jpg
+
+Same image on pins 32 and 33.
+
+Note that pins 34-36 and 39 are inputs only. Also pins 1 and 3 are Tx, Rx of the REPL UART,
+pins 6-11 are connected to the built-in SPI flash.

docs/esp32/tutorial/pwm.rst

@@ -0,0 +1,115 @@
+.. _esp32_pwm:
+
+Pulse Width Modulation
+======================
+
+Pulse width modulation (PWM) is a way to get an artificial analog output on a
+digital pin.  It achieves this by rapidly toggling the pin from low to high.
+There are two parameters associated with this: the frequency of the toggling,
+and the duty cycle.  The duty cycle is defined to be how long the pin is high
+compared with the length of a single period (low plus high time).  Maximum
+duty cycle is when the pin is high all of the time, and minimum is when it is
+low all of the time.
+
+* More comprehensive example with all 16 PWM channels and 8 timers::
+
+    from machine import Pin, PWM
+    try:
+        f = 100  # Hz
+        d = 1024 // 16  # 6.25%
+        pins = (15, 2, 4, 16, 18, 19, 22, 23, 25, 26, 27, 14 , 12, 13, 32, 33)
+        pwms = []
+        for i, pin in enumerate(pins):
+            pwms.append(PWM(Pin(pin), freq=f * (i // 2 + 1), duty= 1023 if i==15 else d * (i + 1)))
+            print(pwms[i])
+    finally:
+        for pwm in pwms:
+            try:
+                pwm.deinit()
+            except:
+                pass
+
+  Output is::
+
+    PWM(Pin(15), freq=100, duty=64, resolution=10, mode=0, channel=0, timer=0)
+    PWM(Pin(2), freq=100, duty=128, resolution=10, mode=0, channel=1, timer=0)
+    PWM(Pin(4), freq=200, duty=192, resolution=10, mode=0, channel=2, timer=1)
+    PWM(Pin(16), freq=200, duty=256, resolution=10, mode=0, channel=3, timer=1)
+    PWM(Pin(18), freq=300, duty=320, resolution=10, mode=0, channel=4, timer=2)
+    PWM(Pin(19), freq=300, duty=384, resolution=10, mode=0, channel=5, timer=2)
+    PWM(Pin(22), freq=400, duty=448, resolution=10, mode=0, channel=6, timer=3)
+    PWM(Pin(23), freq=400, duty=512, resolution=10, mode=0, channel=7, timer=3)
+    PWM(Pin(25), freq=500, duty=576, resolution=10, mode=1, channel=0, timer=0)
+    PWM(Pin(26), freq=500, duty=640, resolution=10, mode=1, channel=1, timer=0)
+    PWM(Pin(27), freq=600, duty=704, resolution=10, mode=1, channel=2, timer=1)
+    PWM(Pin(14), freq=600, duty=768, resolution=10, mode=1, channel=3, timer=1)
+    PWM(Pin(12), freq=700, duty=832, resolution=10, mode=1, channel=4, timer=2)
+    PWM(Pin(13), freq=700, duty=896, resolution=10, mode=1, channel=5, timer=2)
+    PWM(Pin(32), freq=800, duty=960, resolution=10, mode=1, channel=6, timer=3)
+    PWM(Pin(33), freq=800, duty=1023, resolution=10, mode=1, channel=7, timer=3)
+
+* Example of a smooth frequency change::
+
+    from time import sleep
+    from machine import Pin, PWM
+
+    F_MIN = 500
+    F_MAX = 1000
+
+    f = F_MIN
+    delta_f = 1
+
+    p = PWM(Pin(5), f)
+    print(p)
+
+    while True:
+        p.freq(f)
+
+        sleep(10 / F_MIN)
+
+        f += delta_f
+        if f >= F_MAX or f <= F_MIN:
+            delta_f = -delta_f
+
+  See PWM wave at Pin(5) with an oscilloscope.
+
+* Example of a smooth duty change::
+
+    from time import sleep
+    from machine import Pin, PWM
+
+    DUTY_MAX = 2**16 - 1
+
+    duty_u16 = 0
+    delta_d = 16
+
+    p = PWM(Pin(5), 1000, duty_u16=duty_u16)
+    print(p)
+
+    while True:
+        p.duty_u16(duty_u16)
+
+        sleep(1 / 1000)
+
+        duty_u16 += delta_d
+        if duty_u16 >= DUTY_MAX:
+            duty_u16 = DUTY_MAX
+            delta_d = -delta_d
+        elif duty_u16 <= 0:
+            duty_u16 = 0
+            delta_d = -delta_d
+
+  See PWM wave at Pin(5) with an oscilloscope.
+
+Note: the Pin.OUT mode does not need to be specified.  The channel is initialized
+to PWM mode internally once for each Pin that is passed to the PWM constructor.
+
+The following code is wrong::
+
+    pwm = PWM(Pin(5, Pin.OUT), freq=1000, duty=512)  # Pin(5) in PWM mode here
+    pwm = PWM(Pin(5, Pin.OUT), freq=500, duty=256)  # Pin(5) in OUT mode here, PWM is off
+
+Use this code instead::
+
+    pwm = PWM(Pin(5), freq=1000, duty=512)
+    pwm.init(freq=500, duty=256)

docs/esp8266/general.rst

@@ -117,7 +117,7 @@ Real-time clock
 
 RTC in ESP8266 has very bad accuracy, drift may be seconds per minute. As
 a workaround, to measure short enough intervals you can use
-``utime.time()``, etc. functions, and for wall clock time, synchronize from
+``time.time()``, etc. functions, and for wall clock time, synchronize from
 the net using included ``ntptime.py`` module.
 
 Due to limitations of the ESP8266 chip the internal real-time clock (RTC)
@@ -125,6 +125,16 @@ will overflow every 7:45h.  If a long-term working RTC time is required then
 ``time()`` or ``localtime()`` must be called at least once within 7 hours.
 MicroPython will then handle the overflow.
 
+Simultaneous operation of STA_IF and AP_IF
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Simultaneous operation of STA_IF and AP_IF interfaces is supported.
+
+However, due to restrictions of the hardware, there may be performance
+issues in the AP_IF, if the STA_IF is not connected and searching.
+An application should manage these interfaces and for example
+deactivate the STA_IF in environments where only the AP_IF is used.
+
 Sockets and WiFi buffers overflow
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -153,25 +163,26 @@ SSL/TLS limitations
 ~~~~~~~~~~~~~~~~~~~
 
 ESP8266 uses `axTLS <http://axtls.sourceforge.net/>`_ library, which is one
-of the smallest TLS libraries with the compatible licensing. However, it
+of the smallest TLS libraries with compatible licensing. However, it
 also has some known issues/limitations:
 
 1. No support for Diffie-Hellman (DH) key exchange and Elliptic-curve
-   cryptography (ECC). This means it can't work with sites which force
-   the use of these features (it works ok with classic RSA certificates).
+   cryptography (ECC). This means it can't work with sites which require
+   the use of these features (it works ok with the typical sites that use
+   RSA certificates).
 2. Half-duplex communication nature. axTLS uses a single buffer for both
    sending and receiving, which leads to considerable memory saving and
    works well with protocols like HTTP. But there may be problems with
    protocols which don't follow classic request-response model.
 
-Besides axTLS own limitations, the configuration used for MicroPython is
+Besides axTLS's own limitations, the configuration used for MicroPython is
 highly optimized for code size, which leads to additional limitations
 (these may be lifted in the future):
 
 3. Optimized RSA algorithms are not enabled, which may lead to slow
    SSL handshakes.
-4. Stored sessions are not supported (may allow faster repeated connections
-   to the same site in some circumstances).
+4. Session Reuse is not enabled, which means every connection must undergo
+   the full, expensive SSL handshake.
 
 Besides axTLS specific limitations described above, there's another generic
 limitation with usage of TLS on the low-memory devices:
@@ -185,13 +196,16 @@ limitation with usage of TLS on the low-memory devices:
    accessing various REST APIs, which usually require much smaller messages.
    The buffers size is on the order of 5KB, and is adjusted from time to
    time, taking as a reference being able to access https://google.com .
-   The smaller buffer hower means that some sites can't be accessed using
-   it, and it's not possible to stream large amounts of data.
+   The smaller buffer however means that some sites can't be accessed using
+   it, and it's not possible to stream large amounts of data. axTLS does
+   have support for TLS's Max Fragment Size extension, but no HTTPS website
+   does, so use of the extension is really only effective for local
+   communication with other devices.
 
 There are also some not implemented features specifically in MicroPython's
-``ussl`` module based on axTLS:
+``ssl`` module based on axTLS:
 
-6. Certificates are not validated (this may make connections susceptible
+6. Certificates are not validated (this makes connections susceptible
    to man-in-the-middle attacks).
 7. There is no support for client certificates (scheduled to be fixed in
    1.9.4 release).

docs/esp8266/quickref.rst

@@ -57,13 +57,13 @@ The :mod:`network` module::
     wlan.active(True)       # activate the interface
     wlan.scan()             # scan for access points
     wlan.isconnected()      # check if the station is connected to an AP
-    wlan.connect('essid', 'password') # connect to an AP
-    wlan.config('mac')      # get the interface's MAC adddress
+    wlan.connect('ssid', 'key') # connect to an AP
+    wlan.config('mac')      # get the interface's MAC address
     wlan.ifconfig()         # get the interface's IP/netmask/gw/DNS addresses
 
     ap = network.WLAN(network.AP_IF) # create access-point interface
     ap.active(True)         # activate the interface
-    ap.config(essid='ESP-AP') # set the ESSID of the access point
+    ap.config(ssid='ESP-AP') # set the SSID of the access point
 
 A useful function for connecting to your local WiFi network is::
 
@@ -73,18 +73,18 @@ A useful function for connecting to your local WiFi network is::
         wlan.active(True)
         if not wlan.isconnected():
             print('connecting to network...')
-            wlan.connect('essid', 'password')
+            wlan.connect('ssid', 'key')
             while not wlan.isconnected():
                 pass
         print('network config:', wlan.ifconfig())
 
-Once the network is established the :mod:`socket <usocket>` module can be used
+Once the network is established the :mod:`socket <socket>` module can be used
 to create and use TCP/UDP sockets as usual.
 
 Delay and timing
 ----------------
 
-Use the :mod:`time <utime>` module::
+Use the :mod:`time <time>` module::
 
     import time
 
@@ -138,6 +138,49 @@ Also note that Pin(16) is a special pin (used for wakeup from deepsleep
 mode) and may be not available for use with higher-level classes like
 ``Neopixel``.
 
+There's a higher-level abstraction :ref:`machine.Signal <machine.Signal>`
+which can be used to invert a pin. Useful for illuminating active-low LEDs
+using ``on()`` or ``value(1)``.
+
+UART (serial bus)
+-----------------
+
+See :ref:`machine.UART <machine.UART>`. ::
+
+    from machine import UART
+    uart = UART(0, baudrate=9600)
+    uart.write('hello')
+    uart.read(5) # read up to 5 bytes
+
+Two UARTs are available. UART0 is on Pins 1 (TX) and 3 (RX). UART0 is
+bidirectional, and by default is used for the REPL. UART1 is on Pins 2
+(TX) and 8 (RX) however Pin 8 is used to connect the flash chip, so
+UART1 is TX only.
+
+When UART0 is attached to the REPL, all incoming chars on UART(0) go
+straight to stdin so uart.read() will always return None.  Use
+sys.stdin.read() if it's needed to read characters from the UART(0)
+while it's also used for the REPL (or detach, read, then reattach).
+When detached the UART(0) can be used for other purposes.
+
+If there are no objects in any of the dupterm slots when the REPL is
+started (on hard or soft reset) then UART(0) is automatically attached.
+Without this, the only way to recover a board without a REPL would be to
+completely erase and reflash (which would install the default boot.py which
+attaches the REPL).
+
+To detach the REPL from UART0, use::
+
+    import os
+    os.dupterm(None, 1)
+
+The REPL is attached by default. If you have detached it, to reattach
+it use::
+
+    import os, machine
+    uart = machine.UART(0, 115200)
+    os.dupterm(uart, 1)
+
 PWM (pulse width modulation)
 ----------------------------
 
@@ -175,15 +218,15 @@ Software SPI bus
 ----------------
 
 There are two SPI drivers. One is implemented in software (bit-banging)
-and works on all pins, and is accessed via the :ref:`machine.SPI <machine.SPI>`
+and works on all pins, and is accessed via the :ref:`machine.SoftSPI <machine.SoftSPI>`
 class::
 
-    from machine import Pin, SPI
+    from machine import Pin, SoftSPI
 
     # construct an SPI bus on the given pins
     # polarity is the idle state of SCK
     # phase=0 means sample on the first edge of SCK, phase=1 means the second
-    spi = SPI(-1, baudrate=100000, polarity=1, phase=0, sck=Pin(0), mosi=Pin(2), miso=Pin(4))
+    spi = SoftSPI(baudrate=100000, polarity=1, phase=0, sck=Pin(0), mosi=Pin(2), miso=Pin(4))
 
     spi.init(baudrate=200000) # set the baudrate
 
@@ -219,18 +262,19 @@ I2C bus
 -------
 
 The I2C driver is implemented in software and works on all pins,
-and is accessed via the :ref:`machine.I2C <machine.I2C>` class::
+and is accessed via the :ref:`machine.I2C <machine.I2C>` class (which is an
+alias of :ref:`machine.SoftI2C <machine.SoftI2C>`)::
 
     from machine import Pin, I2C
 
     # construct an I2C bus
     i2c = I2C(scl=Pin(5), sda=Pin(4), freq=100000)
 
-    i2c.readfrom(0x3a, 4)   # read 4 bytes from slave device with address 0x3a
-    i2c.writeto(0x3a, '12') # write '12' to slave device with address 0x3a
+    i2c.readfrom(0x3a, 4)   # read 4 bytes from peripheral device with address 0x3a
+    i2c.writeto(0x3a, '12') # write '12' to peripheral device with address 0x3a
 
     buf = bytearray(10)     # create a buffer with 10 bytes
-    i2c.writeto(0x3a, buf)  # write the given buffer to the slave
+    i2c.writeto(0x3a, buf)  # write the given buffer to the peripheral
 
 Real time clock (RTC)
 ---------------------
@@ -253,6 +297,17 @@ See :ref:`machine.RTC <machine.RTC>` ::
           (using a custom handler), `RTC.init()` and `RTC.deinit()` are
           currently not supported.
 
+WDT (Watchdog timer)
+--------------------
+
+See :ref:`machine.WDT <machine.WDT>`. ::
+
+    from machine import WDT
+
+    # enable the WDT
+    wdt = WDT()
+    wdt.feed()
+
 Deep-sleep mode
 ---------------
 
@@ -319,10 +374,13 @@ Use the ``neopixel`` module::
     np.write()              # write data to all pixels
     r, g, b = np[0]         # get first pixel colour
 
-For low-level driving of a NeoPixel::
+.. Warning::
+   By default ``NeoPixel`` is configured to control the more popular *800kHz*
+   units. It is possible to use alternative timing to control other (typically
+   400kHz) devices by passing ``timing=0`` when constructing the
+   ``NeoPixel`` object.
 
-    import esp
-    esp.neopixel_write(pin, grb_buf, is800khz)
+For low-level driving of a NeoPixel see `machine.bitstream`.
 
 APA102 driver
 -------------
@@ -362,6 +420,20 @@ The DHT driver is implemented in software and works on all pins::
     d.temperature() # eg. 23.6 (°C)
     d.humidity()    # eg. 41.3 (% RH)
 
+SSD1306 driver
+--------------
+
+Driver for SSD1306 monochrome OLED displays. See tutorial :ref:`ssd1306`. ::
+
+    from machine import Pin, I2C
+    import ssd1306
+
+    i2c = I2C(scl=Pin(5), sda=Pin(4), freq=100000)
+    display = ssd1306.SSD1306_I2C(128, 64, i2c)
+
+    display.text('Hello World', 0, 0, 1)
+    display.show()
+
 WebREPL (web browser interactive prompt)
 ----------------------------------------
 

docs/esp8266/tutorial/index.rst

@@ -31,4 +31,5 @@ to `<https://www.python.org>`__.
    neopixel.rst
    apa102.rst
    dht.rst
+   ssd1306.rst
    nextsteps.rst

docs/esp8266/tutorial/intro.rst

@@ -18,12 +18,12 @@ The first thing you need is a board with an ESP8266 chip.  The MicroPython
 software supports the ESP8266 chip itself and any board should work.  The main
 characteristic of a board is how much flash it has, how the GPIO pins are
 connected to the outside world, and whether it includes a built-in USB-serial
-convertor to make the UART available to your PC.
+converter to make the UART available to your PC.
 
 The minimum requirement for flash size is 1Mbyte. There is also a special
 build for boards with 512KB, but it is highly limited comparing to the
 normal build: there is no support for filesystem, and thus features which
-depend on it won't work (WebREPL, upip, etc.). As such, 512KB build will
+depend on it won't work (WebREPL, mip, etc.). As such, 512KB build will
 be more interesting for users who build from source and fine-tune parameters
 for their particular application.
 
@@ -41,8 +41,8 @@ Please refer to the documentation for your board for further details.
 Getting the firmware
 --------------------
 
-The first thing you need to do is download the most recent MicroPython firmware 
-.bin file to load onto your ESP8266 device. You can download it from the  
+The first thing you need to do is download the most recent MicroPython firmware
+.bin file to load onto your ESP8266 device. You can download it from the
 `MicroPython downloads page <http://micropython.org/download#esp8266>`_.
 From here, you have 3 main choices
 
@@ -64,17 +64,21 @@ such, only daily builds for 512kb modules are provided.
 Deploying the firmware
 ----------------------
 
-Once you have the MicroPython firmware (compiled code), you need to load it onto 
+Once you have the MicroPython firmware (compiled code), you need to load it onto
 your ESP8266 device.  There are two main steps to do this: first you
 need to put your device in boot-loader mode, and second you need to copy across
 the firmware.  The exact procedure for these steps is highly dependent on the
 particular board and you will need to refer to its documentation for details.
 
-If you have a board that has a USB connector, a USB-serial convertor, and has
+If you have a board that has a USB connector, a USB-serial converter, and has
 the DTR and RTS pins wired in a special way then deploying the firmware should
 be easy as all steps can be done automatically.  Boards that have such features
 include the Adafruit Feather HUZZAH and NodeMCU boards.
 
+If you do not have such a board, you need keep GPIO0 pulled to ground and reset
+the device by pulling the reset pin to ground and releasing it again to enter
+programming mode.
+
 For best results it is recommended to first erase the entire flash of your
 device before putting on new MicroPython firmware.
 
@@ -105,20 +109,26 @@ PC.  You may also need to reduce the baudrate if you get errors when flashing
 that you have.
 
 For some boards with a particular FlashROM configuration (e.g. some variants of
-a NodeMCU board) you may need to use the following command to deploy
-the firmware (note the ``-fm dio`` option)::
+a NodeMCU board) you may need to manually set a compatible
+`SPI Flash Mode <https://github.com/espressif/esptool/wiki/SPI-Flash-Modes>`_.
+You'd usually pick the fastest option that is compatible with your device, but
+the ``-fm dout`` option (the slowest option) should have the best compatibility::
 
-    esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect -fm dio 0 esp8266-20170108-v1.8.7.bin
+    esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect -fm dout 0 esp8266-20170108-v1.8.7.bin
 
 If the above commands run without error then MicroPython should be installed on
 your board!
 
+If you pulled GPIO0 manually to ground to enter programming mode, release it
+now and reset the device by again pulling the reset pin to ground for a short
+duration.
+
 Serial prompt
 -------------
 
 Once you have the firmware on the device you can access the REPL (Python prompt)
 over UART0 (GPIO1=TX, GPIO3=RX), which might be connected to a USB-serial
-convertor, depending on your board.  The baudrate is 115200.  The next part of
+converter, depending on your board.  The baudrate is 115200.  The next part of
 the tutorial will discuss the prompt in more detail.
 
 WiFi
@@ -127,7 +137,7 @@ WiFi
 After a fresh install and boot the device configures itself as a WiFi access
 point (AP) that you can connect to.  The ESSID is of the form MicroPython-xxxxxx
 where the x's are replaced with part of the MAC address of your device (so will
-be the same everytime, and most likely different for all ESP8266 chips).  The
+be the same every time, and most likely different for all ESP8266 chips).  The
 password for the WiFi is micropythoN (note the upper-case N).  Its IP address
 will be 192.168.4.1 once you connect to its network.  WiFi configuration will
 be discussed in more detail later in the tutorial.
@@ -159,7 +169,7 @@ after it, here are troubleshooting recommendations:
 
 * The flashing instructions above use flashing speed of 460800 baud, which is
   good compromise between speed and stability. However, depending on your
-  module/board, USB-UART convertor, cables, host OS, etc., the above baud
+  module/board, USB-UART converter, cables, host OS, etc., the above baud
   rate may be too high and lead to errors. Try a more common 115200 baud
   rate instead in such cases.
 

docs/esp8266/tutorial/network_basics.rst

@@ -37,7 +37,7 @@ First activate the station interface::
 
 Then connect to your WiFi network::
 
-    >>> sta_if.connect('<your ESSID>', '<your password>')
+    >>> sta_if.connect('<your SSID>', '<your key>')
 
 To check if the connection is established use::
 
@@ -61,7 +61,7 @@ connect to your WiFi network::
         if not sta_if.isconnected():
             print('connecting to network...')
             sta_if.active(True)
-            sta_if.connect('<essid>', '<password>')
+            sta_if.connect('<ssid>', '<key>')
             while not sta_if.isconnected():
                 pass
         print('network config:', sta_if.ifconfig())

docs/esp8266/tutorial/network_tcp.rst

@@ -118,5 +118,6 @@ that contains a table with the state of all the GPIO pins::
                 break
         rows = ['<tr><td>%s</td><td>%d</td></tr>' % (str(p), p.value()) for p in pins]
         response = html % '\n'.join(rows)
+        cl.send('HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n')
         cl.send(response)
         cl.close()

docs/esp8266/tutorial/repl.rst

@@ -13,14 +13,14 @@ REPL over the serial port
 
 The REPL is always available on the UART0 serial peripheral, which is connected
 to the pins GPIO1 for TX and GPIO3 for RX.  The baudrate of the REPL is 115200.
-If your board has a USB-serial convertor on it then you should be able to access
+If your board has a USB-serial converter on it then you should be able to access
 the REPL directly from your PC.  Otherwise you will need to have a way of
 communicating with the UART.
 
 To access the prompt over USB-serial you need to use a terminal emulator program.
-On Windows TeraTerm is a good choice, on Mac you can use the built-in screen
-program, and Linux has picocom and minicom.  Of course, there are many other
-terminal programs that will work, so pick your favourite!
+On Windows TeraTerm is a good choice, on Mac you can use the built-in ``screen``
+program, and Linux has ``picocom`` and ``minicom``.  Of course, there are many
+other terminal programs that will work, so pick your favourite!
 
 For example, on Linux you can try running::
 
@@ -37,9 +37,9 @@ WebREPL allows you to use the Python prompt over WiFi, connecting through a
 browser. The latest versions of Firefox and Chrome are supported.
 
 For your convenience, WebREPL client is hosted at
-`<http://micropython.org/webrepl>`__ . Alternatively, you can install it
+`<http://micropython.org/webrepl>`__. Alternatively, you can install it
 locally from the the GitHub repository
-`<https://github.com/micropython/webrepl>`__ .
+`<https://github.com/micropython/webrepl>`__.
 
 Before connecting to WebREPL, you should set a password and enable it via
 a normal serial connection. Initial versions of MicroPython for ESP8266

docs/esp8266/tutorial/ssd1306.rst

@@ -0,0 +1,93 @@
+.. _ssd1306:
+
+Using a SSD1306 OLED display
+============================
+
+The SSD1306 OLED display uses either a SPI or I2C interface and comes in a variety of
+sizes (128x64, 128x32, 72x40, 64x48) and colours (white, yellow, blue, yellow + blue).
+
+Hardware SPI interface::
+
+    from machine import Pin, SPI
+    import ssd1306
+
+    hspi = SPI(1)  # sck=14 (scl), mosi=13 (sda), miso=12 (unused)
+
+    dc = Pin(4)    # data/command
+    rst = Pin(5)   # reset
+    cs = Pin(15)   # chip select, some modules do not have a pin for this
+
+    display = ssd1306.SSD1306_SPI(128, 64, hspi, dc, rst, cs)
+
+Software SPI interface::
+
+    from machine import Pin, SoftSPI
+    import ssd1306
+
+    spi = SoftSPI(baudrate=500000, polarity=1, phase=0, sck=Pin(14), mosi=Pin(13), miso=Pin(12))
+
+    dc = Pin(4)   # data/command
+    rst = Pin(5)  # reset
+    cs = Pin(15)  # chip select, some modules do not have a pin for this
+
+    display = ssd1306.SSD1306_SPI(128, 64, spi, dc, rst, cs)
+
+I2C interface::
+
+    from machine import Pin, I2C
+    import ssd1306
+
+    # using default address 0x3C
+    i2c = I2C(sda=Pin(4), scl=Pin(5))
+    display = ssd1306.SSD1306_I2C(128, 64, i2c)
+
+Print Hello World on the first line::
+
+    display.text('Hello, World!', 0, 0, 1)
+    display.show()
+
+Basic functions::
+
+    display.poweroff()     # power off the display, pixels persist in memory
+    display.poweron()      # power on the display, pixels redrawn
+    display.contrast(0)    # dim
+    display.contrast(255)  # bright
+    display.invert(1)      # display inverted
+    display.invert(0)      # display normal
+    display.rotate(True)   # rotate 180 degrees
+    display.rotate(False)  # rotate 0 degrees
+    display.show()         # write the contents of the FrameBuffer to display memory
+
+Subclassing FrameBuffer provides support for graphics primitives::
+
+    display.fill(0)                         # fill entire screen with colour=0
+    display.pixel(0, 10)                    # get pixel at x=0, y=10
+    display.pixel(0, 10, 1)                 # set pixel at x=0, y=10 to colour=1
+    display.hline(0, 8, 4, 1)               # draw horizontal line x=0, y=8, width=4, colour=1
+    display.vline(0, 8, 4, 1)               # draw vertical line x=0, y=8, height=4, colour=1
+    display.line(0, 0, 127, 63, 1)          # draw a line from 0,0 to 127,63
+    display.rect(10, 10, 107, 43, 1)        # draw a rectangle outline 10,10 to 117,53, colour=1
+    display.fill_rect(10, 10, 107, 43, 1)   # draw a solid rectangle 10,10 to 117,53, colour=1
+    display.text('Hello World', 0, 0, 1)    # draw some text at x=0, y=0, colour=1
+    display.scroll(20, 0)                   # scroll 20 pixels to the right
+
+    # draw another FrameBuffer on top of the current one at the given coordinates
+    import framebuf
+    fbuf = framebuf.FrameBuffer(bytearray(8 * 8 * 1), 8, 8, framebuf.MONO_VLSB)
+    fbuf.line(0, 0, 7, 7, 1)
+    display.blit(fbuf, 10, 10, 0)           # draw on top at x=10, y=10, key=0
+    display.show()
+
+Draw the MicroPython logo and print some text::
+
+    display.fill(0)
+    display.fill_rect(0, 0, 32, 32, 1)
+    display.fill_rect(2, 2, 28, 28, 0)
+    display.vline(9, 8, 22, 1)
+    display.vline(16, 2, 22, 1)
+    display.vline(23, 8, 22, 1)
+    display.fill_rect(26, 24, 2, 4, 1)
+    display.text('MicroPython', 40, 0, 1)
+    display.text('SSD1306', 40, 12, 1)
+    display.text('OLED 128x64', 40, 24, 1)
+    display.show()

docs/index.rst

@@ -11,4 +11,10 @@ MicroPython documentation and references
     pyboard/quickref.rst
     esp8266/quickref.rst
     esp32/quickref.rst
+    rp2/quickref.rst
+    mimxrt/quickref.rst
     wipy/quickref.rst
+    unix/quickref.rst
+    zephyr/quickref.rst
+    renesas-ra/quickref.rst
+    samd/quickref.rst

docs/library/array.rst

@@ -0,0 +1,81 @@
+:mod:`array` -- arrays of numeric data
+======================================
+
+.. module:: array
+   :synopsis: efficient arrays of numeric data
+
+|see_cpython_module| :mod:`python:array`.
+
+Supported format codes: ``b``, ``B``, ``h``, ``H``, ``i``, ``I``, ``l``,
+``L``, ``q``, ``Q``, ``f``, ``d`` (the latter 2 depending on the
+floating-point support).
+
+Classes
+-------
+
+.. class:: array(typecode, [iterable])
+
+    Create array with elements of given type. Initial contents of the
+    array are given by *iterable*. If it is not provided, an empty
+    array is created.
+
+    .. method:: append(val)
+
+        Append new element *val* to the end of array, growing it.
+
+    .. method:: extend(iterable)
+
+        Append new elements as contained in *iterable* to the end of
+        array, growing it.
+
+    .. method:: __getitem__(index)
+
+        Indexed read of the array, called as ``a[index]`` (where ``a`` is an ``array``).
+        Returns a value if *index* is an ``int`` and an ``array`` if *index* is a slice.
+        Negative indices count from the end and ``IndexError`` is thrown if the index is
+        out of range.
+
+        **Note:** ``__getitem__`` cannot be called directly (``a.__getitem__(index)`` fails) and
+        is not present in ``__dict__``, however ``a[index]`` does work.
+
+    .. method:: __setitem__(index, value)
+
+        Indexed write into the array, called as ``a[index] = value`` (where ``a`` is an ``array``).
+        ``value`` is a single value if *index* is an ``int`` and an ``array`` if *index* is a slice.
+        Negative indices count from the end and ``IndexError`` is thrown if the index is out of range.
+
+        **Note:** ``__setitem__`` cannot be called directly (``a.__setitem__(index, value)`` fails) and
+        is not present in ``__dict__``, however ``a[index] = value`` does work.
+
+    .. method:: __len__()
+
+        Returns the number of items in the array, called as ``len(a)`` (where ``a`` is an ``array``).
+
+        **Note:** ``__len__`` cannot be called directly (``a.__len__()`` fails) and the
+        method is not present in ``__dict__``, however ``len(a)`` does work.
+
+    .. method:: __add__(other)
+
+        Return a new ``array`` that is the concatenation of the array with *other*, called as
+        ``a + other`` (where ``a`` and *other* are both ``arrays``).
+
+        **Note:** ``__add__`` cannot be called directly (``a.__add__(other)`` fails) and
+        is not present in ``__dict__``, however ``a + other`` does work.
+
+    .. method:: __iadd__(other)
+
+        Concatenates the array with *other* in-place, called as ``a += other`` (where ``a`` and *other*
+        are both ``arrays``).  Equivalent to ``extend(other)``.
+
+        **Note:** ``__iadd__`` cannot be called directly (``a.__iadd__(other)`` fails) and
+        is not present in ``__dict__``, however ``a += other`` does work.
+
+    .. method:: __repr__()
+
+        Returns the string representation of the array, called as ``str(a)`` or ``repr(a)```
+        (where ``a`` is an ``array``).  Returns the string ``"array(<type>, [<elements>])"``,
+        where ``<type>`` is the type code letter for the array and ``<elements>`` is a comma
+        separated list of the elements of the array.
+
+        **Note:** ``__repr__`` cannot be called directly (``a.__repr__()`` fails) and
+        is not present in ``__dict__``, however ``str(a)`` and ``repr(a)`` both work.

docs/library/asyncio.rst

@@ -0,0 +1,364 @@
+:mod:`asyncio` --- asynchronous I/O scheduler
+=============================================
+
+.. module:: asyncio
+   :synopsis: asynchronous I/O scheduler for writing concurrent code
+
+|see_cpython_module|
+`asyncio <https://docs.python.org/3.8/library/asyncio.html>`_
+
+Example::
+
+    import asyncio
+
+    async def blink(led, period_ms):
+        while True:
+            led.on()
+            await asyncio.sleep_ms(5)
+            led.off()
+            await asyncio.sleep_ms(period_ms)
+
+    async def main(led1, led2):
+        asyncio.create_task(blink(led1, 700))
+        asyncio.create_task(blink(led2, 400))
+        await asyncio.sleep_ms(10_000)
+
+    # Running on a pyboard
+    from pyb import LED
+    asyncio.run(main(LED(1), LED(2)))
+
+    # Running on a generic board
+    from machine import Pin
+    asyncio.run(main(Pin(1), Pin(2)))
+
+Core functions
+--------------
+
+.. function:: create_task(coro)
+
+    Create a new task from the given coroutine and schedule it to run.
+
+    Returns the corresponding `Task` object.
+
+.. function:: current_task()
+
+    Return the `Task` object associated with the currently running task.
+
+.. function:: run(coro)
+
+    Create a new task from the given coroutine and run it until it completes.
+
+    Returns the value returned by *coro*.
+
+.. function:: sleep(t)
+
+    Sleep for *t* seconds (can be a float).
+
+    This is a coroutine.
+
+.. function:: sleep_ms(t)
+
+    Sleep for *t* milliseconds.
+
+    This is a coroutine, and a MicroPython extension.
+
+Additional functions
+--------------------
+
+.. function:: wait_for(awaitable, timeout)
+
+    Wait for the *awaitable* to complete, but cancel it if it takes longer
+    than *timeout* seconds.  If *awaitable* is not a task then a task will be
+    created from it.
+
+    If a timeout occurs, it cancels the task and raises ``asyncio.TimeoutError``:
+    this should be trapped by the caller.  The task receives
+    ``asyncio.CancelledError`` which may be ignored or trapped using ``try...except``
+    or ``try...finally`` to run cleanup code.
+
+    Returns the return value of *awaitable*.
+
+    This is a coroutine.
+
+.. function:: wait_for_ms(awaitable, timeout)
+
+    Similar to `wait_for` but *timeout* is an integer in milliseconds.
+
+    This is a coroutine, and a MicroPython extension.
+
+.. function:: gather(*awaitables, return_exceptions=False)
+
+    Run all *awaitables* concurrently.  Any *awaitables* that are not tasks are
+    promoted to tasks.
+
+    Returns a list of return values of all *awaitables*.
+
+    This is a coroutine.
+
+class Task
+----------
+
+.. class:: Task()
+
+    This object wraps a coroutine into a running task.  Tasks can be waited on
+    using ``await task``, which will wait for the task to complete and return
+    the return value of the task.
+
+    Tasks should not be created directly, rather use `create_task` to create them.
+
+.. method:: Task.cancel()
+
+    Cancel the task by injecting ``asyncio.CancelledError`` into it.  The task may
+    ignore this exception.  Cleanup code may be run by trapping it, or via
+    ``try ... finally``.
+
+class Event
+-----------
+
+.. class:: Event()
+
+    Create a new event which can be used to synchronise tasks.  Events start
+    in the cleared state.
+
+.. method:: Event.is_set()
+
+    Returns ``True`` if the event is set, ``False`` otherwise.
+
+.. method:: Event.set()
+
+    Set the event.  Any tasks waiting on the event will be scheduled to run.
+
+    Note: This must be called from within a task. It is not safe to call this
+    from an IRQ, scheduler callback, or other thread. See `ThreadSafeFlag`.
+
+.. method:: Event.clear()
+
+    Clear the event.
+
+.. method:: Event.wait()
+
+    Wait for the event to be set.  If the event is already set then it returns
+    immediately.
+
+    This is a coroutine.
+
+class ThreadSafeFlag
+--------------------
+
+.. class:: ThreadSafeFlag()
+
+    Create a new flag which can be used to synchronise a task with code running
+    outside the asyncio loop, such as other threads, IRQs, or scheduler
+    callbacks.  Flags start in the cleared state.  The class does not currently
+    work under the Unix build of MicroPython.
+
+.. method:: ThreadSafeFlag.set()
+
+    Set the flag.  If there is a task waiting on the flag, it will be scheduled
+    to run.
+
+.. method:: ThreadSafeFlag.clear()
+
+    Clear the flag. This may be used to ensure that a possibly previously-set
+    flag is clear before waiting for it.
+
+.. method:: ThreadSafeFlag.wait()
+
+    Wait for the flag to be set.  If the flag is already set then it returns
+    immediately.  The flag is automatically reset upon return from ``wait``.
+
+    A flag may only be waited on by a single task at a time.
+
+    This is a coroutine.
+
+class Lock
+----------
+
+.. class:: Lock()
+
+    Create a new lock which can be used to coordinate tasks.  Locks start in
+    the unlocked state.
+
+    In addition to the methods below, locks can be used in an ``async with`` statement.
+
+.. method:: Lock.locked()
+
+    Returns ``True`` if the lock is locked, otherwise ``False``.
+
+.. method:: Lock.acquire()
+
+    Wait for the lock to be in the unlocked state and then lock it in an atomic
+    way.  Only one task can acquire the lock at any one time.
+
+    This is a coroutine.
+
+.. method:: Lock.release()
+
+    Release the lock.  If any tasks are waiting on the lock then the next one in the
+    queue is scheduled to run and the lock remains locked.  Otherwise, no tasks are
+    waiting an the lock becomes unlocked.
+
+TCP stream connections
+----------------------
+
+.. function:: open_connection(host, port, ssl=None)
+
+    Open a TCP connection to the given *host* and *port*.  The *host* address will be
+    resolved using `socket.getaddrinfo`, which is currently a blocking call.
+    If *ssl* is a `ssl.SSLContext` object, this context is used to create the transport;
+    if *ssl* is ``True``, a default context is used.
+
+    Returns a pair of streams: a reader and a writer stream.
+    Will raise a socket-specific ``OSError`` if the host could not be resolved or if
+    the connection could not be made.
+
+    This is a coroutine.
+
+.. function:: start_server(callback, host, port, backlog=5, ssl=None)
+
+    Start a TCP server on the given *host* and *port*.  The *callback* will be
+    called with incoming, accepted connections, and be passed 2 arguments: reader
+    and writer streams for the connection.
+
+    If *ssl* is a `ssl.SSLContext` object, this context is used to create the transport.
+
+    Returns a `Server` object.
+
+    This is a coroutine.
+
+.. class:: Stream()
+
+    This represents a TCP stream connection.  To minimise code this class implements
+    both a reader and a writer, and both ``StreamReader`` and ``StreamWriter`` alias to
+    this class.
+
+.. method:: Stream.get_extra_info(v)
+
+    Get extra information about the stream, given by *v*.  The valid values for *v* are:
+    ``peername``.
+
+.. method:: Stream.close()
+
+    Close the stream.
+
+.. method:: Stream.wait_closed()
+
+    Wait for the stream to close.
+
+    This is a coroutine.
+
+.. method:: Stream.read(n=-1)
+
+    Read up to *n* bytes and return them.  If *n* is not provided or -1 then read all
+    bytes until EOF.  The returned value will be an empty bytes object if EOF is
+    encountered before any bytes are read.
+
+    This is a coroutine.
+
+.. method:: Stream.readinto(buf)
+
+    Read up to n bytes into *buf* with n being equal to the length of *buf*.
+
+    Return the number of bytes read into *buf*.
+
+    This is a coroutine, and a MicroPython extension.
+
+.. method:: Stream.readexactly(n)
+
+    Read exactly *n* bytes and return them as a bytes object.
+
+    Raises an ``EOFError`` exception if the stream ends before reading *n* bytes.
+
+    This is a coroutine.
+
+.. method:: Stream.readline()
+
+    Read a line and return it.
+
+    This is a coroutine.
+
+.. method:: Stream.write(buf)
+
+    Accumulated *buf* to the output buffer.  The data is only flushed when
+    `Stream.drain` is called.  It is recommended to call `Stream.drain` immediately
+    after calling this function.
+
+.. method:: Stream.drain()
+
+    Drain (write) all buffered output data out to the stream.
+
+    This is a coroutine.
+
+.. class:: Server()
+
+    This represents the server class returned from `start_server`.  It can be used
+    in an ``async with`` statement to close the server upon exit.
+
+.. method:: Server.close()
+
+    Close the server.
+
+.. method:: Server.wait_closed()
+
+    Wait for the server to close.
+
+    This is a coroutine.
+
+Event Loop
+----------
+
+.. function:: get_event_loop()
+
+    Return the event loop used to schedule and run tasks.  See `Loop`.
+
+.. function:: new_event_loop()
+
+    Reset the event loop and return it.
+
+    Note: since MicroPython only has a single event loop this function just
+    resets the loop's state, it does not create a new one.
+
+.. class:: Loop()
+
+    This represents the object which schedules and runs tasks.  It cannot be
+    created, use `get_event_loop` instead.
+
+.. method:: Loop.create_task(coro)
+
+    Create a task from the given *coro* and return the new `Task` object.
+
+.. method:: Loop.run_forever()
+
+    Run the event loop until `stop()` is called.
+
+.. method:: Loop.run_until_complete(awaitable)
+
+    Run the given *awaitable* until it completes.  If *awaitable* is not a task
+    then it will be promoted to one.
+
+.. method:: Loop.stop()
+
+    Stop the event loop.
+
+.. method:: Loop.close()
+
+    Close the event loop.
+
+.. method:: Loop.set_exception_handler(handler)
+
+    Set the exception handler to call when a Task raises an exception that is not
+    caught.  The *handler* should accept two arguments: ``(loop, context)``.
+
+.. method:: Loop.get_exception_handler()
+
+    Get the current exception handler.  Returns the handler, or ``None`` if no
+    custom handler is set.
+
+.. method:: Loop.default_exception_handler(context)
+
+    The default exception handler that is called.
+
+.. method:: Loop.call_exception_handler(context)
+
+    Call the current exception handler.  The argument *context* is passed through and
+    is a dictionary containing keys: ``'message'``, ``'exception'``, ``'future'``.

docs/library/binascii.rst

@@ -0,0 +1,38 @@
+:mod:`binascii` -- binary/ASCII conversions
+===========================================
+
+.. module:: binascii
+   :synopsis: binary/ASCII conversions
+
+|see_cpython_module| :mod:`python:binascii`.
+
+This module implements conversions between binary data and various
+encodings of it in ASCII form (in both directions).
+
+Functions
+---------
+
+.. function:: hexlify(data, [sep])
+
+   Convert the bytes in the *data* object to a hexadecimal representation.
+   Returns a bytes object.
+
+   If the additional argument *sep* is supplied it is used as a separator
+   between hexadecimal values.
+
+.. function:: unhexlify(data)
+
+   Convert hexadecimal data to binary representation. Returns bytes string.
+   (i.e. inverse of hexlify)
+
+.. function:: a2b_base64(data)
+
+   Decode base64-encoded data, ignoring invalid characters in the input.
+   Conforms to `RFC 2045 s.6.8 <https://tools.ietf.org/html/rfc2045#section-6.8>`_.
+   Returns a bytes object.
+
+.. function:: b2a_base64(data, *, newline=True)
+
+   Encode binary data in base64 format, as in `RFC 3548
+   <https://tools.ietf.org/html/rfc3548.html>`_. Returns the encoded data
+   followed by a newline character if newline is true, as a bytes object.

docs/library/bluetooth.rst

@@ -0,0 +1,767 @@
+:mod:`bluetooth` --- low-level Bluetooth
+========================================
+
+.. module:: bluetooth
+   :synopsis: Low-level Bluetooth radio functionality
+
+This module provides an interface to a Bluetooth controller on a board.
+Currently this supports Bluetooth Low Energy (BLE) in Central, Peripheral,
+Broadcaster, and Observer roles, as well as GATT Server and Client and L2CAP
+connection-oriented-channels. A device may operate in multiple roles
+concurrently. Pairing (and bonding) is supported on some ports.
+
+This API is intended to match the low-level Bluetooth protocol and provide
+building-blocks for higher-level abstractions such as specific device types.
+
+.. note:: For most applications, we recommend using the higher-level
+          `aioble library <https://github.com/micropython/micropython-lib/tree/master/micropython/bluetooth/aioble>`_.
+
+.. note:: This module is still under development and its classes, functions,
+          methods and constants are subject to change.
+
+class BLE
+---------
+
+Constructor
+-----------
+
+.. class:: BLE()
+
+    Returns the singleton BLE object.
+
+Configuration
+-------------
+
+.. method:: BLE.active([active], /)
+
+    Optionally changes the active state of the BLE radio, and returns the
+    current state.
+
+    The radio must be made active before using any other methods on this class.
+
+.. method:: BLE.config('param', /)
+            BLE.config(*, param=value, ...)
+
+    Get or set configuration values of the BLE interface.  To get a value the
+    parameter name should be quoted as a string, and just one parameter is
+    queried at a time.  To set values use the keyword syntax, and one or more
+    parameter can be set at a time.
+
+    Currently supported values are:
+
+    - ``'mac'``: The current address in use, depending on the current address mode.
+      This returns a tuple of ``(addr_type, addr)``.
+
+      See :meth:`gatts_write <BLE.gatts_write>` for details about address type.
+
+      This may only be queried while the interface is currently active.
+
+    - ``'addr_mode'``: Sets the address mode. Values can be:
+
+        * 0x00 - PUBLIC - Use the controller's public address.
+        * 0x01 - RANDOM - Use a generated static address.
+        * 0x02 - RPA - Use resolvable private addresses.
+        * 0x03 - NRPA - Use non-resolvable private addresses.
+
+      By default the interface mode will use a PUBLIC address if available, otherwise
+      it will use a RANDOM address.
+
+    - ``'gap_name'``: Get/set the GAP device name used by service 0x1800,
+      characteristic 0x2a00.  This can be set at any time and changed multiple
+      times.
+
+    - ``'rxbuf'``: Get/set the size in bytes of the internal buffer used to store
+      incoming events.  This buffer is global to the entire BLE driver and so
+      handles incoming data for all events, including all characteristics.
+      Increasing this allows better handling of bursty incoming data (for
+      example scan results) and the ability to receive larger characteristic values.
+
+    - ``'mtu'``: Get/set the MTU that will be used during a ATT MTU exchange. The
+      resulting MTU will be the minimum of this and the remote device's MTU.
+      ATT MTU exchange will not happen automatically (unless the remote device initiates
+      it), and must be manually initiated with
+      :meth:`gattc_exchange_mtu<BLE.gattc_exchange_mtu>`.
+      Use the ``_IRQ_MTU_EXCHANGED`` event to discover the MTU for a given connection.
+
+    - ``'bond'``: Sets whether bonding will be enabled during pairing. When
+      enabled, pairing requests will set the "bond" flag and the keys will be stored
+      by both devices.
+
+    - ``'mitm'``: Sets whether MITM-protection is required for pairing.
+
+    - ``'io'``: Sets the I/O capabilities of this device.
+
+      Available options are::
+
+        _IO_CAPABILITY_DISPLAY_ONLY = const(0)
+        _IO_CAPABILITY_DISPLAY_YESNO = const(1)
+        _IO_CAPABILITY_KEYBOARD_ONLY = const(2)
+        _IO_CAPABILITY_NO_INPUT_OUTPUT = const(3)
+        _IO_CAPABILITY_KEYBOARD_DISPLAY = const(4)
+
+    - ``'le_secure'``: Sets whether "LE Secure" pairing is required. Default is
+      false (i.e. allow "Legacy Pairing").
+
+Event Handling
+--------------
+
+.. method:: BLE.irq(handler, /)
+
+    Registers a callback for events from the BLE stack. The *handler* takes two
+    arguments, ``event`` (which will be one of the codes below) and ``data``
+    (which is an event-specific tuple of values).
+
+    **Note:** As an optimisation to prevent unnecessary allocations, the ``addr``,
+    ``adv_data``, ``char_data``, ``notify_data``, and ``uuid`` entries in the
+    tuples are read-only memoryview instances pointing to :mod:`bluetooth`'s internal
+    ringbuffer, and are only valid during the invocation of the IRQ handler
+    function.  If your program needs to save one of these values to access after
+    the IRQ handler has returned (e.g. by saving it in a class instance or global
+    variable), then it needs to take a copy of the data, either by using ``bytes()``
+    or ``bluetooth.UUID()``, like this::
+
+        connected_addr = bytes(addr)  # equivalently: adv_data, char_data, or notify_data
+        matched_uuid = bluetooth.UUID(uuid)
+
+    For example, the IRQ handler for a scan result might inspect the ``adv_data``
+    to decide if it's the correct device, and only then copy the address data to be
+    used elsewhere in the program.  And to print data from within the IRQ handler,
+    ``print(bytes(addr))`` will be needed.
+
+    An event handler showing all possible events::
+
+        def bt_irq(event, data):
+            if event == _IRQ_CENTRAL_CONNECT:
+                # A central has connected to this peripheral.
+                conn_handle, addr_type, addr = data
+            elif event == _IRQ_CENTRAL_DISCONNECT:
+                # A central has disconnected from this peripheral.
+                conn_handle, addr_type, addr = data
+            elif event == _IRQ_GATTS_WRITE:
+                # A client has written to this characteristic or descriptor.
+                conn_handle, attr_handle = data
+            elif event == _IRQ_GATTS_READ_REQUEST:
+                # A client has issued a read. Note: this is only supported on STM32.
+                # Return a non-zero integer to deny the read (see below), or zero (or None)
+                # to accept the read.
+                conn_handle, attr_handle = data
+            elif event == _IRQ_SCAN_RESULT:
+                # A single scan result.
+                addr_type, addr, adv_type, rssi, adv_data = data
+            elif event == _IRQ_SCAN_DONE:
+                # Scan duration finished or manually stopped.
+                pass
+            elif event == _IRQ_PERIPHERAL_CONNECT:
+                # A successful gap_connect().
+                conn_handle, addr_type, addr = data
+            elif event == _IRQ_PERIPHERAL_DISCONNECT:
+                # Connected peripheral has disconnected.
+                conn_handle, addr_type, addr = data
+            elif event == _IRQ_GATTC_SERVICE_RESULT:
+                # Called for each service found by gattc_discover_services().
+                conn_handle, start_handle, end_handle, uuid = data
+            elif event == _IRQ_GATTC_SERVICE_DONE:
+                # Called once service discovery is complete.
+                # Note: Status will be zero on success, implementation-specific value otherwise.
+                conn_handle, status = data
+            elif event == _IRQ_GATTC_CHARACTERISTIC_RESULT:
+                # Called for each characteristic found by gattc_discover_services().
+                conn_handle, end_handle, value_handle, properties, uuid = data
+            elif event == _IRQ_GATTC_CHARACTERISTIC_DONE:
+                # Called once service discovery is complete.
+                # Note: Status will be zero on success, implementation-specific value otherwise.
+                conn_handle, status = data
+            elif event == _IRQ_GATTC_DESCRIPTOR_RESULT:
+                # Called for each descriptor found by gattc_discover_descriptors().
+                conn_handle, dsc_handle, uuid = data
+            elif event == _IRQ_GATTC_DESCRIPTOR_DONE:
+                # Called once service discovery is complete.
+                # Note: Status will be zero on success, implementation-specific value otherwise.
+                conn_handle, status = data
+            elif event == _IRQ_GATTC_READ_RESULT:
+                # A gattc_read() has completed.
+                conn_handle, value_handle, char_data = data
+            elif event == _IRQ_GATTC_READ_DONE:
+                # A gattc_read() has completed.
+                # Note: Status will be zero on success, implementation-specific value otherwise.
+                conn_handle, value_handle, status = data
+            elif event == _IRQ_GATTC_WRITE_DONE:
+                # A gattc_write() has completed.
+                # Note: Status will be zero on success, implementation-specific value otherwise.
+                conn_handle, value_handle, status = data
+            elif event == _IRQ_GATTC_NOTIFY:
+                # A server has sent a notify request.
+                conn_handle, value_handle, notify_data = data
+            elif event == _IRQ_GATTC_INDICATE:
+                # A server has sent an indicate request.
+                conn_handle, value_handle, notify_data = data
+            elif event == _IRQ_GATTS_INDICATE_DONE:
+                # A client has acknowledged the indication.
+                # Note: Status will be zero on successful acknowledgment, implementation-specific value otherwise.
+                conn_handle, value_handle, status = data
+            elif event == _IRQ_MTU_EXCHANGED:
+                # ATT MTU exchange complete (either initiated by us or the remote device).
+                conn_handle, mtu = data
+            elif event == _IRQ_L2CAP_ACCEPT:
+                # A new channel has been accepted.
+                # Return a non-zero integer to reject the connection, or zero (or None) to accept.
+                conn_handle, cid, psm, our_mtu, peer_mtu = data
+            elif event == _IRQ_L2CAP_CONNECT:
+                # A new channel is now connected (either as a result of connecting or accepting).
+                conn_handle, cid, psm, our_mtu, peer_mtu = data
+            elif event == _IRQ_L2CAP_DISCONNECT:
+                # Existing channel has disconnected (status is zero), or a connection attempt failed (non-zero status).
+                conn_handle, cid, psm, status = data
+            elif event == _IRQ_L2CAP_RECV:
+                # New data is available on the channel. Use l2cap_recvinto to read.
+                conn_handle, cid = data
+            elif event == _IRQ_L2CAP_SEND_READY:
+                # A previous l2cap_send that returned False has now completed and the channel is ready to send again.
+                # If status is non-zero, then the transmit buffer overflowed and the application should re-send the data.
+                conn_handle, cid, status = data
+            elif event == _IRQ_CONNECTION_UPDATE:
+                # The remote device has updated connection parameters.
+                conn_handle, conn_interval, conn_latency, supervision_timeout, status = data
+            elif event == _IRQ_ENCRYPTION_UPDATE:
+                # The encryption state has changed (likely as a result of pairing or bonding).
+                conn_handle, encrypted, authenticated, bonded, key_size = data
+            elif event == _IRQ_GET_SECRET:
+                # Return a stored secret.
+                # If key is None, return the index'th value of this sec_type.
+                # Otherwise return the corresponding value for this sec_type and key.
+                sec_type, index, key = data
+                return value
+            elif event == _IRQ_SET_SECRET:
+                # Save a secret to the store for this sec_type and key.
+                sec_type, key, value = data
+                return True
+            elif event == _IRQ_PASSKEY_ACTION:
+                # Respond to a passkey request during pairing.
+                # See gap_passkey() for details.
+                # action will be an action that is compatible with the configured "io" config.
+                # passkey will be non-zero if action is "numeric comparison".
+                conn_handle, action, passkey = data
+
+
+The event codes are::
+
+    from micropython import const
+    _IRQ_CENTRAL_CONNECT = const(1)
+    _IRQ_CENTRAL_DISCONNECT = const(2)
+    _IRQ_GATTS_WRITE = const(3)
+    _IRQ_GATTS_READ_REQUEST = const(4)
+    _IRQ_SCAN_RESULT = const(5)
+    _IRQ_SCAN_DONE = const(6)
+    _IRQ_PERIPHERAL_CONNECT = const(7)
+    _IRQ_PERIPHERAL_DISCONNECT = const(8)
+    _IRQ_GATTC_SERVICE_RESULT = const(9)
+    _IRQ_GATTC_SERVICE_DONE = const(10)
+    _IRQ_GATTC_CHARACTERISTIC_RESULT = const(11)
+    _IRQ_GATTC_CHARACTERISTIC_DONE = const(12)
+    _IRQ_GATTC_DESCRIPTOR_RESULT = const(13)
+    _IRQ_GATTC_DESCRIPTOR_DONE = const(14)
+    _IRQ_GATTC_READ_RESULT = const(15)
+    _IRQ_GATTC_READ_DONE = const(16)
+    _IRQ_GATTC_WRITE_DONE = const(17)
+    _IRQ_GATTC_NOTIFY = const(18)
+    _IRQ_GATTC_INDICATE = const(19)
+    _IRQ_GATTS_INDICATE_DONE = const(20)
+    _IRQ_MTU_EXCHANGED = const(21)
+    _IRQ_L2CAP_ACCEPT = const(22)
+    _IRQ_L2CAP_CONNECT = const(23)
+    _IRQ_L2CAP_DISCONNECT = const(24)
+    _IRQ_L2CAP_RECV = const(25)
+    _IRQ_L2CAP_SEND_READY = const(26)
+    _IRQ_CONNECTION_UPDATE = const(27)
+    _IRQ_ENCRYPTION_UPDATE = const(28)
+    _IRQ_GET_SECRET = const(29)
+    _IRQ_SET_SECRET = const(30)
+
+For the ``_IRQ_GATTS_READ_REQUEST`` event, the available return codes are::
+
+    _GATTS_NO_ERROR = const(0x00)
+    _GATTS_ERROR_READ_NOT_PERMITTED = const(0x02)
+    _GATTS_ERROR_WRITE_NOT_PERMITTED = const(0x03)
+    _GATTS_ERROR_INSUFFICIENT_AUTHENTICATION = const(0x05)
+    _GATTS_ERROR_INSUFFICIENT_AUTHORIZATION = const(0x08)
+    _GATTS_ERROR_INSUFFICIENT_ENCRYPTION = const(0x0f)
+
+For the ``_IRQ_PASSKEY_ACTION`` event, the available actions are::
+
+    _PASSKEY_ACTION_NONE = const(0)
+    _PASSKEY_ACTION_INPUT = const(2)
+    _PASSKEY_ACTION_DISPLAY = const(3)
+    _PASSKEY_ACTION_NUMERIC_COMPARISON = const(4)
+
+In order to save space in the firmware, these constants are not included on the
+:mod:`bluetooth` module. Add the ones that you need from the list above to your
+program.
+
+
+Broadcaster Role (Advertiser)
+-----------------------------
+
+.. method:: BLE.gap_advertise(interval_us, adv_data=None, *, resp_data=None, connectable=True)
+
+    Starts advertising at the specified interval (in **micro**\ seconds). This
+    interval will be rounded down to the nearest 625us. To stop advertising, set
+    *interval_us* to ``None``.
+
+    *adv_data* and *resp_data* can be any type that implements the buffer
+    protocol (e.g. ``bytes``, ``bytearray``, ``str``). *adv_data* is included
+    in all broadcasts, and *resp_data* is send in reply to an active scan.
+
+    **Note:** if *adv_data* (or *resp_data*) is ``None``, then the data passed
+    to the previous call to ``gap_advertise`` will be re-used. This allows a
+    broadcaster to resume advertising with just ``gap_advertise(interval_us)``.
+    To clear the advertising payload pass an empty ``bytes``, i.e. ``b''``.
+
+
+Observer Role (Scanner)
+-----------------------
+
+.. method:: BLE.gap_scan(duration_ms, interval_us=1280000, window_us=11250, active=False, /)
+
+    Run a scan operation lasting for the specified duration (in **milli**\ seconds).
+
+    To scan indefinitely, set *duration_ms* to ``0``.
+
+    To stop scanning, set *duration_ms* to ``None``.
+
+    Use *interval_us* and *window_us* to optionally configure the duty cycle.
+    The scanner will run for *window_us* **micro**\ seconds every *interval_us*
+    **micro**\ seconds for a total of *duration_ms* **milli**\ seconds. The default
+    interval and window are 1.28 seconds and 11.25 milliseconds respectively
+    (background scanning).
+
+    For each scan result the ``_IRQ_SCAN_RESULT`` event will be raised, with event
+    data ``(addr_type, addr, adv_type, rssi, adv_data)``.
+
+    ``addr_type`` values indicate public or random addresses:
+        * 0x00 - PUBLIC
+        * 0x01 - RANDOM (either static, RPA, or NRPA, the type is encoded in the address itself)
+
+    ``adv_type`` values correspond to the Bluetooth Specification:
+
+        * 0x00 - ADV_IND - connectable and scannable undirected advertising
+        * 0x01 - ADV_DIRECT_IND - connectable directed advertising
+        * 0x02 - ADV_SCAN_IND - scannable undirected advertising
+        * 0x03 - ADV_NONCONN_IND - non-connectable undirected advertising
+        * 0x04 - SCAN_RSP - scan response
+
+    ``active`` can be set ``True`` if you want to receive scan responses in the results.
+
+    When scanning is stopped (either due to the duration finishing or when
+    explicitly stopped), the ``_IRQ_SCAN_DONE`` event will be raised.
+
+
+Central Role
+------------
+
+A central device can connect to peripherals that it has discovered using the observer role (see :meth:`gap_scan<BLE.gap_scan>`) or with a known address.
+
+.. method:: BLE.gap_connect(addr_type, addr, scan_duration_ms=2000, min_conn_interval_us=None, max_conn_interval_us=None, /)
+
+    Connect to a peripheral.
+
+    See :meth:`gap_scan <BLE.gap_scan>` for details about address types.
+
+    To cancel an outstanding connection attempt early, call
+    ``gap_connect(None)``.
+
+    On success, the ``_IRQ_PERIPHERAL_CONNECT`` event will be raised. If
+    cancelling a connection attempt, the ``_IRQ_PERIPHERAL_DISCONNECT`` event
+    will be raised.
+
+    The device will wait up to *scan_duration_ms* to receive an advertising
+    payload from the device.
+
+    The connection interval can be configured in **micro**\ seconds using either
+    or both of *min_conn_interval_us* and *max_conn_interval_us*. Otherwise a
+    default interval will be chosen, typically between 30000 and 50000
+    microseconds. A shorter interval will increase throughput, at the expense
+    of power usage.
+
+
+Peripheral Role
+---------------
+
+A peripheral device is expected to send connectable advertisements (see
+:meth:`gap_advertise<BLE.gap_advertise>`). It will usually be acting as a GATT
+server, having first registered services and characteristics using
+:meth:`gatts_register_services<BLE.gatts_register_services>`.
+
+When a central connects, the ``_IRQ_CENTRAL_CONNECT`` event will be raised.
+
+
+Central & Peripheral Roles
+--------------------------
+
+.. method:: BLE.gap_disconnect(conn_handle, /)
+
+    Disconnect the specified connection handle. This can either be a
+    central that has connected to this device (if acting as a peripheral)
+    or a peripheral that was previously connected to by this device (if acting
+    as a central).
+
+    On success, the ``_IRQ_PERIPHERAL_DISCONNECT`` or ``_IRQ_CENTRAL_DISCONNECT``
+    event will be raised.
+
+    Returns ``False`` if the connection handle wasn't connected, and ``True``
+    otherwise.
+
+
+GATT Server
+-----------
+
+A GATT server has a set of registered services. Each service may contain
+characteristics, which each have a value. Characteristics can also contain
+descriptors, which themselves have values.
+
+These values are stored locally, and are accessed by their "value handle" which
+is generated during service registration. They can also be read from or written
+to by a remote client device. Additionally, a server can "notify" a
+characteristic to a connected client via a connection handle.
+
+A device in either central or peripheral roles may function as a GATT server,
+however in most cases it will be more common for a peripheral device to act
+as the server.
+
+Characteristics and descriptors have a default maximum size of 20 bytes.
+Anything written to them by a client will be truncated to this length. However,
+any local write will increase the maximum size, so if you want to allow larger
+writes from a client to a given characteristic, use
+:meth:`gatts_write<BLE.gatts_write>` after registration. e.g.
+``gatts_write(char_handle, bytes(100))``.
+
+.. method:: BLE.gatts_register_services(services_definition, /)
+
+    Configures the server with the specified services, replacing any
+    existing services.
+
+    *services_definition* is a list of **services**, where each **service** is a
+    two-element tuple containing a UUID and a list of **characteristics**.
+
+    Each **characteristic** is a two-or-three-element tuple containing a UUID, a
+    **flags** value, and optionally a list of *descriptors*.
+
+    Each **descriptor** is a two-element tuple containing a UUID and a **flags**
+    value.
+
+    The **flags** are a bitwise-OR combination of the flags defined below. These
+    set both the behaviour of the characteristic (or descriptor) as well as the
+    security and privacy requirements.
+
+    The return value is a list (one element per service) of tuples (each element
+    is a value handle). Characteristics and descriptor handles are flattened
+    into the same tuple, in the order that they are defined.
+
+    The following example registers two services (Heart Rate, and Nordic UART)::
+
+        HR_UUID = bluetooth.UUID(0x180D)
+        HR_CHAR = (bluetooth.UUID(0x2A37), bluetooth.FLAG_READ | bluetooth.FLAG_NOTIFY,)
+        HR_SERVICE = (HR_UUID, (HR_CHAR,),)
+        UART_UUID = bluetooth.UUID('6E400001-B5A3-F393-E0A9-E50E24DCCA9E')
+        UART_TX = (bluetooth.UUID('6E400003-B5A3-F393-E0A9-E50E24DCCA9E'), bluetooth.FLAG_READ | bluetooth.FLAG_NOTIFY,)
+        UART_RX = (bluetooth.UUID('6E400002-B5A3-F393-E0A9-E50E24DCCA9E'), bluetooth.FLAG_WRITE,)
+        UART_SERVICE = (UART_UUID, (UART_TX, UART_RX,),)
+        SERVICES = (HR_SERVICE, UART_SERVICE,)
+        ( (hr,), (tx, rx,), ) = bt.gatts_register_services(SERVICES)
+
+    The three value handles (``hr``, ``tx``, ``rx``) can be used with
+    :meth:`gatts_read <BLE.gatts_read>`, :meth:`gatts_write <BLE.gatts_write>`, :meth:`gatts_notify <BLE.gatts_notify>`, and
+    :meth:`gatts_indicate <BLE.gatts_indicate>`.
+
+    **Note:** Advertising must be stopped before registering services.
+
+    Available flags for characteristics and descriptors are::
+
+        from micropython import const
+        _FLAG_BROADCAST = const(0x0001)
+        _FLAG_READ = const(0x0002)
+        _FLAG_WRITE_NO_RESPONSE = const(0x0004)
+        _FLAG_WRITE = const(0x0008)
+        _FLAG_NOTIFY = const(0x0010)
+        _FLAG_INDICATE = const(0x0020)
+        _FLAG_AUTHENTICATED_SIGNED_WRITE = const(0x0040)
+
+        _FLAG_AUX_WRITE = const(0x0100)
+        _FLAG_READ_ENCRYPTED = const(0x0200)
+        _FLAG_READ_AUTHENTICATED = const(0x0400)
+        _FLAG_READ_AUTHORIZED = const(0x0800)
+        _FLAG_WRITE_ENCRYPTED = const(0x1000)
+        _FLAG_WRITE_AUTHENTICATED = const(0x2000)
+        _FLAG_WRITE_AUTHORIZED = const(0x4000)
+
+    As for the IRQs above, any required constants should be added to your Python code.
+
+.. method:: BLE.gatts_read(value_handle, /)
+
+    Reads the local value for this handle (which has either been written by
+    :meth:`gatts_write <BLE.gatts_write>` or by a remote client).
+
+.. method:: BLE.gatts_write(value_handle, data, send_update=False, /)
+
+    Writes the local value for this handle, which can be read by a client.
+
+    If *send_update* is ``True``, then any subscribed clients will be notified
+    (or indicated, depending on what they're subscribed to and which operations
+    the characteristic supports) about this write.
+
+.. method:: BLE.gatts_notify(conn_handle, value_handle, data=None, /)
+
+    Sends a notification request to a connected client.
+
+    If *data* is ``None`` (the default), then the current local value (as set
+    with :meth:`gatts_write <BLE.gatts_write>`) will be sent.
+
+    Otherwise, if *data* is not ``None``, then that value is sent to the client
+    as part of the notification. The local value will not be modified.
+
+    **Note:** The notification will be sent regardless of the subscription
+    status of the client to this characteristic.
+
+.. method:: BLE.gatts_indicate(conn_handle, value_handle, data=None, /)
+
+    Sends a indication request to a connected client.
+
+    If *data* is ``None`` (the default), then the current local value (as set
+    with :meth:`gatts_write <BLE.gatts_write>`) will be sent.
+
+    Otherwise, if *data* is not ``None``, then that value is sent to the client
+    as part of the indication. The local value will not be modified.
+
+    On acknowledgment (or failure, e.g. timeout), the
+    ``_IRQ_GATTS_INDICATE_DONE`` event will be raised.
+
+    **Note:** The indication will be sent regardless of the subscription
+    status of the client to this characteristic.
+
+.. method:: BLE.gatts_set_buffer(value_handle, len, append=False, /)
+
+    Sets the internal buffer size for a value in bytes. This will limit the
+    largest possible write that can be received. The default is 20.
+
+    Setting *append* to ``True`` will make all remote writes append to, rather
+    than replace, the current value. At most *len* bytes can be buffered in
+    this way. When you use :meth:`gatts_read <BLE.gatts_read>`, the value will
+    be cleared after reading. This feature is useful when implementing something
+    like the Nordic UART Service.
+
+GATT Client
+-----------
+
+A GATT client can discover and read/write characteristics on a remote GATT server.
+
+It is more common for a central role device to act as the GATT client, however
+it's also possible for a peripheral to act as a client in order to discover
+information about the central that has connected to it (e.g. to read the
+device name from the device information service).
+
+.. method:: BLE.gattc_discover_services(conn_handle, uuid=None, /)
+
+    Query a connected server for its services.
+
+    Optionally specify a service *uuid* to query for that service only.
+
+    For each service discovered, the ``_IRQ_GATTC_SERVICE_RESULT`` event will
+    be raised, followed by ``_IRQ_GATTC_SERVICE_DONE`` on completion.
+
+.. method:: BLE.gattc_discover_characteristics(conn_handle, start_handle, end_handle, uuid=None, /)
+
+    Query a connected server for characteristics in the specified range.
+
+    Optionally specify a characteristic *uuid* to query for that
+    characteristic only.
+
+    You can use ``start_handle=1``, ``end_handle=0xffff`` to search for a
+    characteristic in any service.
+
+    For each characteristic discovered, the ``_IRQ_GATTC_CHARACTERISTIC_RESULT``
+    event will be raised, followed by ``_IRQ_GATTC_CHARACTERISTIC_DONE`` on completion.
+
+.. method:: BLE.gattc_discover_descriptors(conn_handle, start_handle, end_handle, /)
+
+    Query a connected server for descriptors in the specified range.
+
+    For each descriptor discovered, the ``_IRQ_GATTC_DESCRIPTOR_RESULT`` event
+    will be raised, followed by ``_IRQ_GATTC_DESCRIPTOR_DONE`` on completion.
+
+.. method:: BLE.gattc_read(conn_handle, value_handle, /)
+
+    Issue a remote read to a connected server for the specified
+    characteristic or descriptor handle.
+
+    When a value is available, the ``_IRQ_GATTC_READ_RESULT`` event will be
+    raised. Additionally, the ``_IRQ_GATTC_READ_DONE`` will be raised.
+
+.. method:: BLE.gattc_write(conn_handle, value_handle, data, mode=0, /)
+
+    Issue a remote write to a connected server for the specified
+    characteristic or descriptor handle.
+
+    The argument *mode* specifies the write behaviour, with the currently
+    supported values being:
+
+        * ``mode=0`` (default) is a write-without-response: the write will
+          be sent to the remote server but no confirmation will be
+          returned, and no event will be raised.
+        * ``mode=1`` is a write-with-response: the remote server is
+          requested to send a response/acknowledgement that it received the
+          data.
+
+    If a response is received from the remote server the
+    ``_IRQ_GATTC_WRITE_DONE`` event will be raised.
+
+.. method:: BLE.gattc_exchange_mtu(conn_handle, /)
+
+    Initiate MTU exchange with a connected server, using the preferred MTU
+    set using ``BLE.config(mtu=value)``.
+
+    The ``_IRQ_MTU_EXCHANGED`` event will be raised when MTU exchange
+    completes.
+
+    **Note:** MTU exchange is typically initiated by the central. When using
+    the BlueKitchen stack in the central role, it does not support a remote
+    peripheral initiating the MTU exchange. NimBLE works for both roles.
+
+
+L2CAP connection-oriented-channels
+----------------------------------
+
+    This feature allows for socket-like data exchange between two BLE devices.
+    Once the devices are connected via GAP, either device can listen for the
+    other to connect on a numeric PSM (Protocol/Service Multiplexer).
+
+    **Note:** This is currently only supported when using the NimBLE stack on
+    STM32 and Unix (not ESP32). Only one L2CAP channel may be active at a given
+    time (i.e. you cannot connect while listening).
+
+    Active L2CAP channels are identified by the connection handle that they were
+    established on and a CID (channel ID).
+
+    Connection-oriented channels have built-in credit-based flow control. Unlike
+    ATT, where devices negotiate a shared MTU, both the listening and connecting
+    devices each set an independent MTU which limits the maximum amount of
+    outstanding data that the remote device can send before it is fully consumed
+    in :meth:`l2cap_recvinto <BLE.l2cap_recvinto>`.
+
+.. method:: BLE.l2cap_listen(psm, mtu, /)
+
+    Start listening for incoming L2CAP channel requests on the specified *psm*
+    with the local MTU set to *mtu*.
+
+    When a remote device initiates a connection, the ``_IRQ_L2CAP_ACCEPT``
+    event will be raised, which gives the listening server a chance to reject
+    the incoming connection (by returning a non-zero integer).
+
+    Once the connection is accepted, the ``_IRQ_L2CAP_CONNECT`` event will be
+    raised, allowing the server to obtain the channel id (CID) and the local and
+    remote MTU.
+
+    **Note:** It is not currently possible to stop listening.
+
+.. method:: BLE.l2cap_connect(conn_handle, psm, mtu, /)
+
+    Connect to a listening peer on the specified *psm* with local MTU set to *mtu*.
+
+    On successful connection, the the ``_IRQ_L2CAP_CONNECT`` event will be
+    raised, allowing the client to obtain the CID and the local and remote (peer) MTU.
+
+    An unsuccessful connection will raise the ``_IRQ_L2CAP_DISCONNECT`` event
+    with a non-zero status.
+
+.. method:: BLE.l2cap_disconnect(conn_handle, cid, /)
+
+    Disconnect an active L2CAP channel with the specified *conn_handle* and
+    *cid*.
+
+.. method:: BLE.l2cap_send(conn_handle, cid, buf, /)
+
+    Send the specified *buf* (which must support the buffer protocol) on the
+    L2CAP channel identified by *conn_handle* and *cid*.
+
+    The specified buffer cannot be larger than the remote (peer) MTU, and no
+    more than twice the size of the local MTU.
+
+    This will return ``False`` if the channel is now "stalled", which means that
+    :meth:`l2cap_send <BLE.l2cap_send>` must not be called again until the
+    ``_IRQ_L2CAP_SEND_READY`` event is received (which will happen when the
+    remote device grants more credits, typically after it has received and
+    processed the data).
+
+.. method:: BLE.l2cap_recvinto(conn_handle, cid, buf, /)
+
+    Receive data from the specified *conn_handle* and *cid* into the provided
+    *buf* (which must support the buffer protocol, e.g. bytearray or
+    memoryview).
+
+    Returns the number of bytes read from the channel.
+
+    If *buf* is None, then returns the number of bytes available.
+
+    **Note:** After receiving the ``_IRQ_L2CAP_RECV`` event, the application should
+    continue calling :meth:`l2cap_recvinto <BLE.l2cap_recvinto>` until no more
+    bytes are available in the receive buffer (typically up to the size of the
+    remote (peer) MTU).
+
+    Until the receive buffer is empty, the remote device will not be granted
+    more channel credits and will be unable to send any more data.
+
+
+Pairing and bonding
+-------------------
+
+    Pairing allows a connection to be encrypted and authenticated via exchange
+    of secrets (with optional MITM protection via passkey authentication).
+
+    Bonding is the process of storing those secrets into non-volatile storage.
+    When bonded, a device is able to resolve a resolvable private address (RPA)
+    from another device based on the stored identity resolving key (IRK).
+    To support bonding, an application must implement the ``_IRQ_GET_SECRET``
+    and ``_IRQ_SET_SECRET`` events.
+
+    **Note:** This is currently only supported when using the NimBLE stack on
+    STM32 and Unix (not ESP32).
+
+.. method:: BLE.gap_pair(conn_handle, /)
+
+    Initiate pairing with the remote device.
+
+    Before calling this, ensure that the ``io``, ``mitm``, ``le_secure``, and
+    ``bond`` configuration options are set (via :meth:`config<BLE.config>`).
+
+    On successful pairing, the ``_IRQ_ENCRYPTION_UPDATE`` event will be raised.
+
+.. method:: BLE.gap_passkey(conn_handle, action, passkey, /)
+
+    Respond to a ``_IRQ_PASSKEY_ACTION`` event for the specified *conn_handle*
+    and *action*.
+
+    The *passkey* is a numeric value and will depend on on the
+    *action* (which will depend on what I/O capability has been set):
+
+        * When the *action* is ``_PASSKEY_ACTION_INPUT``, then the application should
+          prompt the user to enter the passkey that is shown on the remote device.
+        * When the *action* is ``_PASSKEY_ACTION_DISPLAY``, then the application should
+          generate a random 6-digit passkey and show it to the user.
+        * When the *action* is ``_PASSKEY_ACTION_NUMERIC_COMPARISON``, then the application
+          should show the passkey that was provided in the ``_IRQ_PASSKEY_ACTION`` event
+          and then respond with either ``0`` (cancel pairing), or ``1`` (accept pairing).
+
+
+class UUID
+----------
+
+
+Constructor
+-----------
+
+.. class:: UUID(value, /)
+
+    Creates a UUID instance with the specified **value**.
+
+    The **value** can be either:
+
+    - A 16-bit integer. e.g. ``0x2908``.
+    - A 128-bit UUID string. e.g. ``'6E400001-B5A3-F393-E0A9-E50E24DCCA9E'``.

docs/library/btree.rst

@@ -11,8 +11,9 @@ value, a database also supports efficient ordered range scans (retrieval
 of values with the keys in a given range). On the application interface
 side, BTree database work as close a possible to a way standard `dict`
 type works, one notable difference is that both keys and values must
-be `bytes` objects (so, if you want to store objects of other types, you
-need to serialize them to `bytes` first).
+be `bytes`-like objects (so, if you want to store objects of other types, you
+need to first serialize them to `str` or `bytes` or another type that supports
+the buffer protocol).
 
 The module is based on the well-known BerkelyDB library, version 1.xx.
 
@@ -22,7 +23,7 @@ Example::
 
     # First, we need to open a stream which holds a database
     # This is usually a file, but can be in-memory database
-    # using uio.BytesIO, a raw flash partition, etc.
+    # using io.BytesIO, a raw flash partition, etc.
     # Oftentimes, you want to create a database file if it doesn't
     # exist and open if it exists. Idiom below takes care of this.
     # DO NOT open database with "a+b" access mode.
@@ -76,7 +77,7 @@ Example::
 Functions
 ---------
 
-.. function:: open(stream, \*, flags=0, pagesize=0, cachesize=0, minkeypage=0)
+.. function:: open(stream, *, flags=0, pagesize=0, cachesize=0, minkeypage=0)
 
    Open a database from a random-access `stream` (like an open file). All
    other parameters are optional and keyword-only, and allow to tweak advanced
@@ -116,9 +117,9 @@ Methods
    Flush any data in cache to the underlying stream.
 
 .. method:: btree.__getitem__(key)
-            btree.get(key, default=None)
+            btree.get(key, default=None, /)
             btree.__setitem__(key, val)
-            btree.__detitem__(key)
+            btree.__delitem__(key)
             btree.__contains__(key)
 
    Standard dictionary methods.

docs/library/builtins.rst

@@ -1,5 +1,5 @@
-Builtin functions and exceptions
-================================
+:mod:`builtins` -- builtin functions and exceptions
+===================================================
 
 All builtin functions and exceptions are described here. They are also
 available via ``builtins`` module.
@@ -176,10 +176,6 @@ Exceptions
 
 .. exception:: OSError
 
-    |see_cpython| `python:OSError`. MicroPython doesn't implement ``errno``
-    attribute, instead use the standard way to access exception arguments:
-    ``exc.args[0]``.
-
 .. exception:: RuntimeError
 
 .. exception:: StopIteration

docs/library/collections.rst

@@ -0,0 +1,82 @@
+:mod:`collections` -- collection and container types
+====================================================
+
+.. module:: collections
+   :synopsis: collection and container types
+
+|see_cpython_module| :mod:`python:collections`.
+
+This module implements advanced collection and container types to
+hold/accumulate various objects.
+
+Classes
+-------
+
+.. class:: deque(iterable, maxlen[, flags])
+
+    Deques (double-ended queues) are a list-like container that support O(1)
+    appends and pops from either side of the deque.  New deques are created
+    using the following arguments:
+
+        - *iterable* must be the empty tuple, and the new deque is created empty.
+
+        - *maxlen* must be specified and the deque will be bounded to this
+          maximum length.  Once the deque is full, any new items added will
+          discard items from the opposite end.
+
+        - The optional *flags* can be 1 to check for overflow when adding items.
+
+    As well as supporting `bool` and `len`, deque objects have the following
+    methods:
+
+    .. method:: deque.append(x)
+
+        Add *x* to the right side of the deque.
+        Raises IndexError if overflow checking is enabled and there is no more room left.
+
+    .. method:: deque.popleft()
+
+        Remove and return an item from the left side of the deque.
+        Raises IndexError if no items are present.
+
+.. function:: namedtuple(name, fields)
+
+    This is factory function to create a new namedtuple type with a specific
+    name and set of fields. A namedtuple is a subclass of tuple which allows
+    to access its fields not just by numeric index, but also with an attribute
+    access syntax using symbolic field names. Fields is a sequence of strings
+    specifying field names. For compatibility with CPython it can also be a
+    a string with space-separated field named (but this is less efficient).
+    Example of use::
+
+        from collections import namedtuple
+
+        MyTuple = namedtuple("MyTuple", ("id", "name"))
+        t1 = MyTuple(1, "foo")
+        t2 = MyTuple(2, "bar")
+        print(t1.name)
+        assert t2.name == t2[1]
+
+.. class:: OrderedDict(...)
+
+    ``dict`` type subclass which remembers and preserves the order of keys
+    added. When ordered dict is iterated over, keys/items are returned in
+    the order they were added::
+
+        from collections import OrderedDict
+
+        # To make benefit of ordered keys, OrderedDict should be initialized
+        # from sequence of (key, value) pairs.
+        d = OrderedDict([("z", 1), ("a", 2)])
+        # More items can be added as usual
+        d["w"] = 5
+        d["b"] = 3
+        for k, v in d.items():
+            print(k, v)
+
+    Output::
+
+        z 1
+        a 2
+        w 5
+        b 3

docs/library/cryptolib.rst

@@ -0,0 +1,40 @@
+:mod:`cryptolib` -- cryptographic ciphers
+=========================================
+
+.. module:: cryptolib
+   :synopsis: cryptographic ciphers
+
+Classes
+-------
+
+.. class:: aes
+
+    .. classmethod:: __init__(key, mode, [IV])
+
+        Initialize cipher object, suitable for encryption/decryption. Note:
+        after initialization, cipher object can be use only either for
+        encryption or decryption. Running decrypt() operation after encrypt()
+        or vice versa is not supported.
+
+        Parameters are:
+
+            * *key* is an encryption/decryption key (bytes-like).
+            * *mode* is:
+
+                * ``1`` (or ``cryptolib.MODE_ECB`` if it exists) for Electronic Code Book (ECB).
+                * ``2`` (or ``cryptolib.MODE_CBC`` if it exists) for Cipher Block Chaining (CBC).
+                * ``6`` (or ``cryptolib.MODE_CTR`` if it exists) for Counter mode (CTR).
+
+            * *IV* is an initialization vector for CBC mode.
+            * For Counter mode, *IV* is the initial value for the counter.
+
+    .. method:: encrypt(in_buf, [out_buf])
+
+        Encrypt *in_buf*. If no *out_buf* is given result is returned as a
+        newly allocated `bytes` object. Otherwise, result is written into
+        mutable buffer *out_buf*. *in_buf* and *out_buf* can also refer
+        to the same mutable buffer, in which case data is encrypted in-place.
+
+    .. method:: decrypt(in_buf, [out_buf])
+
+        Like `encrypt()`, but for decryption.

docs/library/deflate.rst

@@ -0,0 +1,182 @@
+:mod:`deflate` -- deflate compression & decompression
+=====================================================
+
+.. module:: deflate
+   :synopsis: deflate compression & decompression
+
+This module allows compression and decompression of binary data with the
+`DEFLATE algorithm <https://en.wikipedia.org/wiki/DEFLATE>`_
+(commonly used in the zlib library and gzip archiver).
+
+**Availability:**
+
+* Added in MicroPython v1.21.
+
+* Decompression: Enabled via the ``MICROPY_PY_DEFLATE`` build option, on by default
+  on ports with the "extra features" level or higher (which is most boards).
+
+* Compression: Enabled via the ``MICROPY_PY_DEFLATE_COMPRESS`` build option, on
+  by default on ports with the "full features" level or higher (generally this means
+  you need to build your own firmware to enable this).
+
+Classes
+-------
+
+.. class:: DeflateIO(stream, format=AUTO, wbits=0, close=False, /)
+
+   This class can be used to wrap a *stream* which is any
+   :term:`stream-like <stream>` object such as a file, socket, or stream
+   (including :class:`io.BytesIO`). It is itself a stream and implements the
+   standard read/readinto/write/close methods.
+
+   The *stream* must be a blocking stream. Non-blocking streams are currently
+   not supported.
+
+   The *format* can be set to any of the constants defined below, and defaults
+   to ``AUTO`` which for decompressing will auto-detect gzip or zlib streams,
+   and for compressing it will generate a raw stream.
+
+   The *wbits* parameter sets the base-2 logarithm of the DEFLATE dictionary
+   window size. So for example, setting *wbits* to ``10`` sets the window size
+   to 1024 bytes. Valid values are ``5`` to ``15`` inclusive (corresponding to
+   window sizes of 32 to 32k bytes).
+
+   If *wbits* is set to ``0`` (the default), then for compression a window size
+   of 256 bytes will be used (as if *wbits* was set to 8). For decompression, it
+   depends on the format:
+
+   * ``RAW`` will use 256 bytes (corresponding to *wbits* set to 8).
+   * ``ZLIB`` (or ``AUTO`` with zlib detected) will use the value from the zlib
+     header.
+   * ``GZIP`` (or ``AUTO`` with gzip detected) will use 32 kilobytes
+     (corresponding to *wbits* set to 15).
+
+   See the :ref:`window size <deflate_wbits>` notes below for more information
+   about the window size, zlib, and gzip streams.
+
+   If *close* is set to ``True`` then the underlying stream will be closed
+   automatically when the :class:`deflate.DeflateIO` stream is closed. This is
+   useful if you want to return a :class:`deflate.DeflateIO` stream that wraps
+   another stream and not have the caller need to know about managing the
+   underlying stream.
+
+   If compression is enabled, a given :class:`deflate.DeflateIO` instance
+   supports both reading and writing. For example, a bidirectional stream like
+   a socket can be wrapped, which allows for compression/decompression in both
+   directions.
+
+Constants
+---------
+
+.. data:: deflate.AUTO
+          deflate.RAW
+          deflate.ZLIB
+          deflate.GZIP
+
+    Supported values for the *format* parameter.
+
+Examples
+--------
+
+A typical use case for :class:`deflate.DeflateIO` is to read or write a compressed
+file from storage:
+
+.. code:: python
+
+   import deflate
+
+   # Writing a zlib-compressed stream (uses the default window size of 256 bytes).
+   with open("data.gz", "wb") as f:
+       with deflate.DeflateIO(f, deflate.ZLIB) as d:
+           # Use d.write(...) etc
+
+   # Reading a zlib-compressed stream (auto-detect window size).
+   with open("data.z", "rb") as f:
+       with deflate.DeflateIO(f, deflate.ZLIB) as d:
+           # Use d.read(), d.readinto(), etc.
+
+Because :class:`deflate.DeflateIO` is a stream, it can be used for example
+with :meth:`json.dump` and :meth:`json.load` (and any other places streams can
+be used):
+
+.. code:: python
+
+   import deflate, json
+
+   # Write a dictionary as JSON in gzip format, with a
+   # small (64 byte) window size.
+   config = { ... }
+   with open("config.gz", "wb") as f:
+       with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
+           json.dump(config, f)
+
+   # Read back that dictionary.
+   with open("config.gz", "rb") as f:
+       with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
+           config = json.load(f)
+
+If your source data is not in a stream format, you can use :class:`io.BytesIO`
+to turn it into a stream suitable for use with :class:`deflate.DeflateIO`:
+
+.. code:: python
+
+   import deflate, io
+
+   # Decompress a bytes/bytearray value.
+   compressed_data = get_data_z()
+   with deflate.DeflateIO(io.BytesIO(compressed_data), deflate.ZLIB) as d:
+       decompressed_data = d.read()
+
+   # Compress a bytes/bytearray value.
+   uncompressed_data = get_data()
+   stream = io.BytesIO()
+   with deflate.DeflateIO(stream, deflate.ZLIB) as d:
+       d.write(uncompressed_data)
+   compressed_data = stream.getvalue()
+
+.. _deflate_wbits:
+
+Deflate window size
+-------------------
+
+The window size limits how far back in the stream the (de)compressor can
+reference. Increasing the window size will improve compression, but will require
+more memory and make the compressor slower.
+
+If an input stream was compressed a given window size, then `DeflateIO`
+using a smaller window size will fail mid-way during decompression with
+:exc:`OSError`, but only if a back-reference actually refers back further
+than the decompressor's window size. This means it may be possible to decompress
+with a smaller window size. For example, this would trivially be the case if the
+original uncompressed data is shorter than the window size.
+
+Decompression
+~~~~~~~~~~~~~
+
+The zlib format includes a header which specifies the window size that was used
+to compress the data. This indicates the maximum window size required to
+decompress this stream. If this header value is less than the specified *wbits*
+value (or if *wbits* is unset), then the header value will be used.
+
+The gzip format does not include the window size in the header, and assumes that
+all gzip compressors (e.g. the ``gzip`` utility, or CPython's implementation of
+:class:`gzip.GzipFile`) use the maximum window size of 32kiB. For this reason,
+if the *wbits* parameter is not set, the decompressor will use a 32 kiB window
+size (corresponding to *wbits* set to 15). This means that to be able to
+decompress an arbitrary gzip stream, you must have at least this much RAM
+available. If you control the source data, consider instead using the zlib
+format with a smaller window size.
+
+The raw format has no header and therefore does not include any information
+about the window size. If *wbits* is not set, then it will default to a window
+size of 256 bytes, which may not be large enough for a given stream. Therefore
+it is recommended that you should always explicitly set *wbits* if using the raw
+format.
+
+Compression
+~~~~~~~~~~~
+
+For compression, MicroPython will default to a window size of 256 bytes for all
+formats. This provides a reasonable amount of compression with minimal memory
+usage and fast compression time, and will generate output that will work with
+any decompressor.

docs/library/errno.rst

@@ -0,0 +1,34 @@
+:mod:`errno` -- system error codes
+==================================
+
+.. module:: errno
+   :synopsis: system error codes
+
+|see_cpython_module| :mod:`python:errno`.
+
+This module provides access to symbolic error codes for `OSError` exception.
+A particular inventory of codes depends on :term:`MicroPython port`.
+
+Constants
+---------
+
+.. data:: EEXIST, EAGAIN, etc.
+
+    Error codes, based on ANSI C/POSIX standard. All error codes start with
+    "E". As mentioned above, inventory of the codes depends on
+    :term:`MicroPython port`. Errors are usually accessible as ``exc.errno``
+    where ``exc`` is an instance of `OSError`. Usage example::
+
+        try:
+            os.mkdir("my_dir")
+        except OSError as exc:
+            if exc.errno == errno.EEXIST:
+                print("Directory already exists")
+
+.. data:: errorcode
+
+    Dictionary mapping numeric error codes to strings with symbolic error
+    code (see above)::
+
+        >>> print(errno.errorcode[errno.EEXIST])
+        EEXIST

docs/library/esp.rst

@@ -4,7 +4,7 @@
 .. module:: esp
     :synopsis: functions related to the ESP8266 and ESP32
 
-The ``esp`` module contains specific functions related to both the ESP8266 and 
+The ``esp`` module contains specific functions related to both the ESP8266 and
 ESP32 modules.  Some functions are only available on one or the other of these
 ports.
 
@@ -31,7 +31,7 @@ Functions
 
     The system enters the set sleep mode automatically when possible.
 
-.. function:: deepsleep(time=0)
+.. function:: deepsleep(time_us=0, /)
 
     **Note**: ESP8266 only - use `machine.deepsleep()` on ESP32
 
@@ -62,6 +62,53 @@ Functions
 
 .. function:: flash_erase(sector_no)
 
+.. function:: osdebug(uart_no)
+
+    .. note:: This is the ESP8266 form of this function.
+
+    Change the level of OS serial debug log messages. On boot,
+    OS serial debug log messages are disabled.
+
+    ``uart_no`` is the number of the UART peripheral which should receive
+    OS-level output, or ``None`` to disable OS serial debug log messages.
+
+.. function:: osdebug(uart_no, [level])
+    :no-index:
+
+    .. note:: This is the ESP32 form of this function.
+
+    Change the level of OS serial debug log messages. On boot, OS
+    serial debug log messages are limited to Error output only.
+
+    The behaviour of this function depends on the arguments passed to it. The
+    following combinations are supported:
+
+    ``osdebug(None)`` restores the default OS debug log message level
+    (``LOG_ERROR``).
+
+    ``osdebug(0)`` enables all available OS debug log messages (in the
+    default build configuration this is ``LOG_INFO``).
+
+    ``osdebug(0, level)`` sets the OS debug log message level to the
+     specified value. The log levels are defined as constants:
+
+        * ``LOG_NONE`` -- No log output
+        * ``LOG_ERROR`` -- Critical errors, software module can not recover on its own
+        * ``LOG_WARN`` -- Error conditions from which recovery measures have been taken
+        * ``LOG_INFO`` -- Information messages which describe normal flow of events
+        * ``LOG_DEBUG`` -- Extra information which is not necessary for normal use (values, pointers, sizes, etc)
+        * ``LOG_VERBOSE`` -- Bigger chunks of debugging information, or frequent messages
+          which can potentially flood the output
+
+    .. note:: ``LOG_DEBUG`` and ``LOG_VERBOSE`` are not compiled into the
+              MicroPython binary by default, to save size. A custom build with a
+              modified "``sdkconfig``" source file is needed to see any output
+              at these log levels.
+
+    .. note:: Log output on ESP32 is automatically suspended in "Raw REPL" mode,
+              to prevent communications issues. This means OS level logging is never
+              seen when using ``mpremote run`` and similar tools.
+
 .. function:: set_native_code_location(start, length)
 
     **Note**: ESP8266 only

docs/library/esp32.rst

@@ -18,6 +18,11 @@ Functions
     Configure whether or not a touch will wake the device from sleep.
     *wake* should be a boolean value.
 
+.. function:: wake_on_ulp(wake)
+
+    Configure whether or not the Ultra-Low-Power co-processor can wake the
+    device from sleep. *wake* should be a boolean value.
+
 .. function:: wake_on_ext0(pin, level)
 
     Configure how EXT0 wakes the device from sleep.  *pin* can be ``None``
@@ -30,50 +35,112 @@ Functions
     or a tuple/list of valid Pin objects.  *level* should be ``esp32.WAKEUP_ALL_LOW``
     or ``esp32.WAKEUP_ANY_HIGH``.
 
+.. function:: gpio_deep_sleep_hold(enable)
+
+    Configure whether non-RTC GPIO pin configuration is retained during
+    deep-sleep mode for held pads. *enable* should be a boolean value.
+
 .. function:: raw_temperature()
 
     Read the raw value of the internal temperature sensor, returning an integer.
 
-.. function:: hall_sensor()
+.. function:: idf_heap_info(capabilities)
+
+    Returns information about the ESP-IDF heap memory regions. One of them contains
+    the MicroPython heap and the others are used by ESP-IDF, e.g., for network
+    buffers and other data. This data is useful to get a sense of how much memory
+    is available to ESP-IDF and the networking stack in particular. It may shed
+    some light on situations where ESP-IDF operations fail due to allocation failures.
+
+    The capabilities parameter corresponds to ESP-IDF's ``MALLOC_CAP_XXX`` values but the
+    two most useful ones are predefined as `esp32.HEAP_DATA` for data heap regions and
+    `esp32.HEAP_EXEC` for executable regions as used by the native code emitter.
+
+    The return value is a list of 4-tuples, where each 4-tuple corresponds to one heap
+    and contains: the total bytes, the free bytes, the largest free block, and
+    the minimum free seen over time.
+
+    Example after booting::
+
+        >>> import esp32; esp32.idf_heap_info(esp32.HEAP_DATA)
+        [(240, 0, 0, 0), (7288, 0, 0, 0), (16648, 4, 4, 4), (79912, 35712, 35512, 35108),
+         (15072, 15036, 15036, 15036), (113840, 0, 0, 0)]
+
+    .. note:: Free IDF heap memory in the `esp32.HEAP_DATA` region is available
+       to be automatically added to the MicroPython heap to prevent a
+       MicroPython allocation from failing. However, the information returned
+       here is otherwise *not* useful to troubleshoot Python allocation
+       failures. :func:`micropython.mem_info()` and :func:`gc.mem_free()` should
+       be used instead:
+
+       The "max new split" value in :func:`micropython.mem_info()` output
+       corresponds to the largest free block of ESP-IDF heap that could be
+       automatically added on demand to the MicroPython heap.
+
+       The result of :func:`gc.mem_free()` is the total of the current "free"
+       and "max new split" values printed by :func:`micropython.mem_info()`.
 
-    Read the raw value of the internal Hall sensor, returning an integer.
 
 Flash partitions
 ----------------
 
-This class gives access to the partitions in the device's flash memory.
+This class gives access to the partitions in the device's flash memory and includes
+methods to enable over-the-air (OTA) updates.
 
-.. class:: Partition(id)
+.. class:: Partition(id, block_size=4096, /)
 
     Create an object representing a partition.  *id* can be a string which is the label
     of the partition to retrieve, or one of the constants: ``BOOT`` or ``RUNNING``.
+    *block_size* specifies the byte size of an individual block.
 
-.. classmethod:: Partition.find(type=TYPE_APP, subtype=0xff, label=None)
+.. classmethod:: Partition.find(type=TYPE_APP, subtype=0xff, label=None, block_size=4096)
 
     Find a partition specified by *type*, *subtype* and *label*.  Returns a
-    (possibly empty) list of Partition objects.
+    (possibly empty) list of Partition objects. Note: ``subtype=0xff`` matches any subtype
+    and ``label=None`` matches any label.
+
+    *block_size* specifies the byte size of an individual block used by the returned
+    objects.
 
 .. method:: Partition.info()
 
     Returns a 6-tuple ``(type, subtype, addr, size, label, encrypted)``.
 
 .. method:: Partition.readblocks(block_num, buf)
-.. method:: Partition.readblocks(block_num, buf, offset)
+            Partition.readblocks(block_num, buf, offset)
 .. method:: Partition.writeblocks(block_num, buf)
-.. method:: Partition.writeblocks(block_num, buf, offset)
+            Partition.writeblocks(block_num, buf, offset)
 .. method:: Partition.ioctl(cmd, arg)
 
     These methods implement the simple and :ref:`extended
     <block-device-interface>` block protocol defined by
-    :class:`uos.AbstractBlockDev`.
+    :class:`os.AbstractBlockDev`.
 
 .. method:: Partition.set_boot()
 
     Sets the partition as the boot partition.
 
+    .. note:: Do not enter :func:`deepsleep<machine.deepsleep>` after changing
+       the OTA boot partition, without first performing a hard
+       :func:`reset<machine.reset>` or power cycle. This ensures the bootloader
+       will validate the new image before booting.
+
 .. method:: Partition.get_next_update()
 
     Gets the next update partition after this one, and returns a new Partition object.
+    Typical usage is ``Partition(Partition.RUNNING).get_next_update()``
+    which returns the next partition to update given the current running one.
+
+.. classmethod:: Partition.mark_app_valid_cancel_rollback()
+
+    Signals that the current boot is considered successful.
+    Calling ``mark_app_valid_cancel_rollback`` is required on the first boot of a new
+    partition to avoid an automatic rollback at the next boot.
+    This uses the ESP-IDF "app rollback" feature with "CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE"
+    and  an ``OSError(-261)`` is raised if called on firmware that doesn't have the
+    feature enabled.
+    It is OK to call ``mark_app_valid_cancel_rollback`` on every boot and it is not
+    necessary when booting firmware that was loaded using esptool.
 
 Constants
 ~~~~~~~~~
@@ -81,13 +148,21 @@ Constants
 .. data:: Partition.BOOT
           Partition.RUNNING
 
-    Used in the `Partition` constructor to fetch various partitions.
+    Used in the `Partition` constructor to fetch various partitions: ``BOOT`` is the
+    partition that will be booted at the next reset and ``RUNNING`` is the currently
+    running partition.
 
 .. data:: Partition.TYPE_APP
           Partition.TYPE_DATA
 
-    Used in `Partition.find` to specify the partition type.
+    Used in `Partition.find` to specify the partition type: ``APP`` is for bootable
+    firmware partitions (typically labelled ``factory``, ``ota_0``, ``ota_1``), and
+    ``DATA`` is for other partitions, e.g. ``nvs``, ``otadata``, ``phy_init``, ``vfs``.
 
+.. data:: HEAP_DATA
+          HEAP_EXEC
+
+    Used in `idf_heap_info`.
 
 .. _esp32.RMT:
 
@@ -103,18 +178,22 @@ used to transmit or receive many other types of digital signals::
     from machine import Pin
 
     r = esp32.RMT(0, pin=Pin(18), clock_div=8)
-    r  # RMT(channel=0, pin=18, source_freq=80000000, clock_div=8)
+    r  # RMT(channel=0, pin=18, source_freq=80000000, clock_div=8, idle_level=0)
+
+    # To apply a carrier frequency to the high output
+    r = esp32.RMT(0, pin=Pin(18), clock_div=8, tx_carrier=(38000, 50, 1))
+
     # The channel resolution is 100ns (1/(source_freq/clock_div)).
-    r.write_pulses((1, 20, 2, 40), start=0)  # Send 0 for 100ns, 1 for 2000ns, 0 for 200ns, 1 for 4000ns
+    r.write_pulses((1, 20, 2, 40), 0)  # Send 0 for 100ns, 1 for 2000ns, 0 for 200ns, 1 for 4000ns
 
 The input to the RMT module is an 80MHz clock (in the future it may be able to
 configure the input clock but, for now, it's fixed). ``clock_div`` *divides*
 the clock input which determines the resolution of the RMT channel. The
-numbers specificed in ``write_pulses`` are multiplied by the resolution to
+numbers specified in ``write_pulses`` are multiplied by the resolution to
 define the pulses.
 
 ``clock_div`` is an 8-bit divider (0-255) and each pulse can be defined by
-multiplying the resolution by a 15-bit (0-32,768) number. There are eight
+multiplying the resolution by a 15-bit (1-``PULSE_MAX``) number. There are eight
 channels (0-7) and each can have a different clock divider.
 
 So, in the example above, the 80MHz clock is divided by 8. Thus the
@@ -127,19 +206,27 @@ For more details see Espressif's `ESP-IDF RMT documentation.
 
 .. Warning::
    The current MicroPython RMT implementation lacks some features, most notably
-   receiving pulses and carrier transmit. RMT should be considered a
+   receiving pulses. RMT should be considered a
    *beta feature* and the interface may change in the future.
 
 
-.. class:: RMT(channel, \*, pin=None, clock_div=8)
+.. class:: RMT(channel, *, pin=None, clock_div=8, idle_level=False, tx_carrier=None)
 
     This class provides access to one of the eight RMT channels. *channel* is
     required and identifies which RMT channel (0-7) will be configured. *pin*,
     also required, configures which Pin is bound to the RMT channel. *clock_div*
     is an 8-bit clock divider that divides the source clock (80MHz) to the RMT
-    channel allowing the resolution to be specified.
+    channel allowing the resolution to be specified. *idle_level* specifies
+    what level the output will be when no transmission is in progress and can
+    be any value that converts to a boolean, with ``True`` representing high
+    voltage and ``False`` representing low.
 
-.. method:: RMT.source_freq()
+    To enable the transmission carrier feature, *tx_carrier* should be a tuple
+    of three positive integers: carrier frequency, duty percent (``0`` to
+    ``100``) and the output level to apply the carrier to (a boolean as per
+    *idle_level*).
+
+.. classmethod:: RMT.source_freq()
 
     Returns the source clock frequency. Currently the source clock is not
     configurable so this will always return 80MHz.
@@ -149,32 +236,77 @@ For more details see Espressif's `ESP-IDF RMT documentation.
     Return the clock divider. Note that the channel resolution is
     ``1 / (source_freq / clock_div)``.
 
-.. method:: RMT.wait_done(timeout=0)
+.. method:: RMT.wait_done(*, timeout=0)
 
-    Returns True if `RMT.write_pulses` has completed.
-
-    If *timeout* (defined in ticks of ``source_freq / clock_div``) is specified
-    the method will wait for *timeout* or until `RMT.write_pulses` is complete,
-    returning ``False`` if the channel continues to transmit.
-
-.. Warning::
-    Avoid using ``wait_done()`` if looping is enabled.
+    Returns ``True`` if the channel is idle or ``False`` if a sequence of
+    pulses started with `RMT.write_pulses` is being transmitted. If the
+    *timeout* keyword argument is given then block for up to this many
+    milliseconds for transmission to complete.
 
 .. method:: RMT.loop(enable_loop)
 
-    Configure looping on the channel, allowing a stream of pulses to be
-    indefinitely repeated. *enable_loop* is bool, set to True to enable looping.
+    Configure looping on the channel. *enable_loop* is bool, set to ``True`` to
+    enable looping on the *next* call to `RMT.write_pulses`. If called with
+    ``False`` while a looping sequence is currently being transmitted then the
+    current loop iteration will be completed and then transmission will stop.
 
-.. method:: RMT.write_pulses(pulses, start)
+.. method:: RMT.write_pulses(duration, data=True)
 
-    Begin sending *pulses*, a list or tuple defining the stream of pulses. The
-    length of each pulse is defined by a number to be multiplied by the channel
-    resolution ``(1 / (source_freq / clock_div))``. *start* defines whether the
-    stream starts at 0 or 1.
+    Begin transmitting a sequence. There are three ways to specify this:
 
+    **Mode 1:** *duration* is a list or tuple of durations. The optional *data*
+    argument specifies the initial output level. The output level will toggle
+    after each duration.
 
-The Ultra-Low-Power co-processor
---------------------------------
+    **Mode 2:** *duration* is a positive integer and *data* is a list or tuple
+    of output levels. *duration* specifies a fixed duration for each.
+
+    **Mode 3:** *duration* and *data* are lists or tuples of equal length,
+    specifying individual durations and the output level for each.
+
+    Durations are in integer units of the channel resolution (as
+    described above), between 1 and ``PULSE_MAX`` units. Output levels
+    are any value that can be converted to a boolean, with ``True``
+    representing high voltage and ``False`` representing low.
+
+    If transmission of an earlier sequence is in progress then this method will
+    block until that transmission is complete before beginning the new sequence.
+
+    If looping has been enabled with `RMT.loop`, the sequence will be
+    repeated indefinitely. Further calls to this method will block until the
+    end of the current loop iteration before immediately beginning to loop the
+    new sequence of pulses. Looping sequences longer than 126 pulses is not
+    supported by the hardware.
+
+.. staticmethod:: RMT.bitstream_channel([value])
+
+    Select which RMT channel is used by the `machine.bitstream` implementation.
+    *value* can be ``None`` or a valid RMT channel number.  The default RMT
+    channel is the highest numbered one.
+
+    Passing in ``None`` disables the use of RMT and instead selects a bit-banging
+    implementation for `machine.bitstream`.
+
+    Passing in no argument will not change the channel.  This function returns
+    the current channel number.
+
+Constants
+---------
+
+.. data:: RMT.PULSE_MAX
+
+   Maximum integer that can be set for a pulse duration.
+
+Ultra-Low-Power co-processor
+----------------------------
+
+This class gives access to the Ultra Low Power (ULP) co-processor on the ESP32,
+ESP32-S2 and ESP32-S3 chips.
+
+.. warning::
+
+    This class does not provide access to the RISCV ULP co-processor available
+    on the ESP32-S2 and ESP32-S3 chips.
 
 .. class:: ULP()
 
@@ -200,3 +332,51 @@ Constants
           esp32.WAKEUP_ANY_HIGH
 
    Selects the wake level for pins.
+
+Non-Volatile Storage
+--------------------
+
+This class gives access to the Non-Volatile storage managed by ESP-IDF. The NVS is partitioned
+into namespaces and each namespace contains typed key-value pairs. The keys are strings and the
+values may be various integer types, strings, and binary blobs. The driver currently only
+supports 32-bit signed integers and blobs.
+
+.. warning::
+
+    Changes to NVS need to be committed to flash by calling the commit method. Failure
+    to call commit results in changes being lost at the next reset.
+
+.. class:: NVS(namespace)
+
+    Create an object providing access to a namespace (which is automatically created if not
+    present).
+
+.. method:: NVS.set_i32(key, value)
+
+    Sets a 32-bit signed integer value for the specified key. Remember to call *commit*!
+
+.. method:: NVS.get_i32(key)
+
+    Returns the signed integer value for the specified key. Raises an OSError if the key does not
+    exist or has a different type.
+
+.. method:: NVS.set_blob(key, value)
+
+    Sets a binary blob value for the specified key. The value passed in must support the buffer
+    protocol, e.g. bytes, bytearray, str. (Note that esp-idf distinguishes blobs and strings, this
+    method always writes a blob even if a string is passed in as value.)
+    Remember to call *commit*!
+
+.. method:: NVS.get_blob(key, buffer)
+
+    Reads the value of the blob for the specified key into the buffer, which must be a bytearray.
+    Returns the actual length read. Raises an OSError if the key does not exist, has a different
+    type, or if the buffer is too small.
+
+.. method:: NVS.erase_key(key)
+
+    Erases a key-value pair.
+
+.. method:: NVS.commit()
+
+    Commits changes made by *set_xxx* methods to flash.

docs/library/espnow.rst

@@ -0,0 +1,936 @@
+:mod:`espnow` --- support for the ESP-NOW wireless protocol
+===========================================================
+
+.. module:: espnow
+    :synopsis: ESP-NOW wireless protocol support
+
+This module provides an interface to the `ESP-NOW <https://www.espressif.com/
+en/products/software/esp-now/overview>`_ protocol provided by Espressif on
+ESP32 and ESP8266 devices (`API docs <https://docs.espressif.com/
+projects/esp-idf/en/latest/api-reference/network/esp_now.html>`_).
+
+Table of Contents:
+------------------
+
+    - `Introduction`_
+    - `Configuration`_
+    - `Sending and Receiving Data`_
+    - `Peer Management`_
+    - `Callback Methods`_
+    - `Exceptions`_
+    - `Constants`_
+    - `Wifi Signal Strength (RSSI) - (ESP32 Only)`_
+    - `Supporting asyncio`_
+    - `Broadcast and Multicast`_
+    - `ESPNow and Wifi Operation`_
+    - `ESPNow and Sleep Modes`_
+
+Introduction
+------------
+
+ESP-NOW is a connection-less wireless communication protocol supporting:
+
+- Direct communication between up to 20 registered peers:
+
+  - Without the need for a wireless access point (AP),
+
+- Encrypted and unencrypted communication (up to 6 encrypted peers),
+
+- Message sizes up to 250 bytes,
+
+- Can operate alongside Wifi operation (:doc:`network.WLAN<network.WLAN>`) on
+  ESP32 and ESP8266 devices.
+
+It is especially useful for small IoT networks, latency sensitive or power
+sensitive applications (such as battery operated devices) and for long-range
+communication between devices (hundreds of metres).
+
+This module also supports tracking the Wifi signal strength (RSSI) of peer
+devices.
+
+A simple example would be:
+
+**Sender:** ::
+
+    import network
+    import espnow
+
+    # A WLAN interface must be active to send()/recv()
+    sta = network.WLAN(network.STA_IF)  # Or network.AP_IF
+    sta.active(True)
+    sta.disconnect()      # For ESP8266
+
+    e = espnow.ESPNow()
+    e.active(True)
+    peer = b'\xbb\xbb\xbb\xbb\xbb\xbb'   # MAC address of peer's wifi interface
+    e.add_peer(peer)      # Must add_peer() before send()
+
+    e.send(peer, "Starting...")
+    for i in range(100):
+        e.send(peer, str(i)*20, True)
+    e.send(peer, b'end')
+
+**Receiver:** ::
+
+    import network
+    import espnow
+
+    # A WLAN interface must be active to send()/recv()
+    sta = network.WLAN(network.STA_IF)
+    sta.active(True)
+    sta.disconnect()   # Because ESP8266 auto-connects to last Access Point
+
+    e = espnow.ESPNow()
+    e.active(True)
+
+    while True:
+        host, msg = e.recv()
+        if msg:             # msg == None if timeout in recv()
+            print(host, msg)
+            if msg == b'end':
+                break
+
+class ESPNow
+------------
+
+Constructor
+-----------
+
+.. class:: ESPNow()
+
+    Returns the singleton ESPNow object. As this is a singleton, all calls to
+    `espnow.ESPNow()` return a reference to the same object.
+
+    .. note::
+      Some methods are available only on the ESP32 due to code size
+      restrictions on the ESP8266 and differences in the Espressif API.
+
+Configuration
+-------------
+
+.. method:: ESPNow.active([flag])
+
+    Initialise or de-initialise the ESP-NOW communication protocol depending on
+    the value of the ``flag`` optional argument.
+
+    .. data:: Arguments:
+
+      - *flag*: Any python value which can be converted to a boolean type.
+
+        - ``True``: Prepare the software and hardware for use of the ESP-NOW
+          communication protocol, including:
+
+          - initialise the ESPNow data structures,
+          - allocate the recv data buffer,
+          - invoke esp_now_init() and
+          - register the send and recv callbacks.
+
+        - ``False``: De-initialise the Espressif ESP-NOW software stack
+          (esp_now_deinit()), disable callbacks, deallocate the recv
+          data buffer and deregister all peers.
+
+    If *flag* is not provided, return the current status of the ESPNow
+    interface.
+
+    .. data:: Returns:
+
+        ``True`` if interface is currently *active*, else ``False``.
+
+.. method:: ESPNow.config(param=value, ...)
+            ESPNow.config('param')   (ESP32 only)
+
+    Set or get configuration values of the ESPNow interface. To set values, use
+    the keyword syntax, and one or more parameters can be set at a time. To get
+    a value the parameter name should be quoted as a string, and just one
+    parameter is queried at a time.
+
+    **Note:** *Getting* parameters is not supported on the ESP8266.
+
+    .. data:: Options:
+
+        *rxbuf*: (default=526) Get/set the size in bytes of the internal
+        buffer used to store incoming ESPNow packet data. The default size is
+        selected to fit two max-sized ESPNow packets (250 bytes) with associated
+        mac_address (6 bytes), a message byte count (1 byte) and RSSI data plus
+        buffer overhead. Increase this if you expect to receive a lot of large
+        packets or expect bursty incoming traffic.
+
+        **Note:** The recv buffer is allocated by `ESPNow.active()`. Changing
+        this value will have no effect until the next call of
+        `ESPNow.active(True)<ESPNow.active()>`.
+
+        *timeout_ms*: (default=300,000) Default timeout (in milliseconds)
+        for receiving ESPNow messages. If *timeout_ms* is less than zero, then
+        wait forever. The timeout can also be provided as arg to
+        `recv()`/`irecv()`/`recvinto()`.
+
+        *rate*: (ESP32 only, IDF>=4.3.0 only) Set the transmission speed for
+        ESPNow packets. Must be set to a number from the allowed numeric values
+        in `enum wifi_phy_rate_t
+        <https://docs.espressif.com/projects/esp-idf/en/v4.4.1/esp32/
+        api-reference/network/esp_wifi.html#_CPPv415wifi_phy_rate_t>`_.
+
+    .. data:: Returns:
+
+        ``None`` or the value of the parameter being queried.
+
+    .. data:: Raises:
+
+        - ``OSError(num, "ESP_ERR_ESPNOW_NOT_INIT")`` if not initialised.
+        - ``ValueError()`` on invalid configuration options or values.
+
+Sending and Receiving Data
+--------------------------
+
+A wifi interface (``network.STA_IF`` or ``network.AP_IF``) must be
+`active()<network.WLAN.active>` before messages can be sent or received,
+but it is not necessary to connect or configure the WLAN interface.
+For example::
+
+    import network
+
+    sta = network.WLAN(network.STA_IF)
+    sta.active(True)
+    sta.disconnect()    # For ESP8266
+
+**Note:** The ESP8266 has a *feature* that causes it to automatically reconnect
+to the last wifi Access Point when set `active(True)<network.WLAN.active>` (even
+after reboot/reset). This reduces the reliability of receiving ESP-NOW messages
+(see `ESPNow and Wifi Operation`_). You can avoid this by calling
+`disconnect()<network.WLAN.disconnect>` after
+`active(True)<network.WLAN.active>`.
+
+.. method:: ESPNow.send(mac, msg[, sync])
+            ESPNow.send(msg)   (ESP32 only)
+
+    Send the data contained in ``msg`` to the peer with given network ``mac``
+    address. In the second form, ``mac=None`` and ``sync=True``. The peer must
+    be registered with `ESPNow.add_peer()<ESPNow.add_peer()>` before the
+    message can be sent.
+
+    .. data:: Arguments:
+
+      - *mac*: byte string exactly ``espnow.ADDR_LEN`` (6 bytes) long or
+        ``None``. If *mac* is ``None`` (ESP32 only) the message will be sent
+        to all registered peers, except any broadcast or multicast MAC
+        addresses.
+
+      - *msg*: string or byte-string up to ``espnow.MAX_DATA_LEN`` (250)
+        bytes long.
+
+      - *sync*:
+
+        - ``True``: (default) send ``msg`` to the peer(s) and wait for a
+          response (or not).
+
+        - ``False`` send ``msg`` and return immediately. Responses from the
+          peers will be discarded.
+
+    .. data:: Returns:
+
+      ``True`` if ``sync=False`` or if ``sync=True`` and *all* peers respond,
+      else ``False``.
+
+    .. data:: Raises:
+
+      - ``OSError(num, "ESP_ERR_ESPNOW_NOT_INIT")`` if not initialised.
+      - ``OSError(num, "ESP_ERR_ESPNOW_NOT_FOUND")`` if peer is not registered.
+      - ``OSError(num, "ESP_ERR_ESPNOW_IF")`` the wifi interface is not
+        `active()<network.WLAN.active>`.
+      - ``OSError(num, "ESP_ERR_ESPNOW_NO_MEM")`` internal ESP-NOW buffers are
+        full.
+      - ``ValueError()`` on invalid values for the parameters.
+
+    **Note**: A peer will respond with success if its wifi interface is
+    `active()<network.WLAN.active>` and set to the same channel as the sender,
+    regardless of whether it has initialised it's ESP-NOW system or is
+    actively listening for ESP-NOW traffic (see the Espressif ESP-NOW docs).
+
+.. method:: ESPNow.recv([timeout_ms])
+
+    Wait for an incoming message and return the ``mac`` address of the peer and
+    the message. **Note**: It is **not** necessary to register a peer (using
+    `add_peer()<ESPNow.add_peer()>`) to receive a message from that peer.
+
+    .. data:: Arguments:
+
+        - *timeout_ms*: (Optional): May have the following values.
+
+          - ``0``: No timeout. Return immediately if no data is available;
+          - ``> 0``: Specify a timeout value in milliseconds;
+          - ``< 0``: Do not timeout, ie. wait forever for new messages; or
+          - ``None`` (or not provided): Use the default timeout value set with
+            `ESPNow.config()`.
+
+    .. data:: Returns:
+
+      - ``(None, None)`` if timeout is reached before a message is received, or
+
+      - ``[mac, msg]``: where:
+
+        - ``mac`` is a bytestring containing the address of the device which
+          sent the message, and
+        - ``msg`` is a bytestring containing the message.
+
+    .. data:: Raises:
+
+      - ``OSError(num, "ESP_ERR_ESPNOW_NOT_INIT")`` if not initialised.
+      - ``OSError(num, "ESP_ERR_ESPNOW_IF")`` if the wifi interface is not
+        `active()<network.WLAN.active>`.
+      - ``ValueError()`` on invalid *timeout_ms* values.
+
+    `ESPNow.recv()` will allocate new storage for the returned list and the
+    ``peer`` and ``msg`` bytestrings. This can lead to memory fragmentation if
+    the data rate is high. See `ESPNow.irecv()` for a memory-friendly
+    alternative.
+
+
+.. method:: ESPNow.irecv([timeout_ms])
+
+    Works like `ESPNow.recv()` but will reuse internal bytearrays to store the
+    return values: ``[mac, msg]``, so that no new memory is allocated on each
+    call.
+
+    .. data:: Arguments:
+
+        *timeout_ms*: (Optional) Timeout in milliseconds (see `ESPNow.recv()`).
+
+    .. data:: Returns:
+
+      - As for `ESPNow.recv()`, except that ``msg`` is a bytearray, instead of
+        a bytestring. On the ESP8266, ``mac`` will also be a bytearray.
+
+    .. data:: Raises:
+
+      - See `ESPNow.recv()`.
+
+    **Note:** You may also read messages by iterating over the ESPNow object,
+    which will use the `irecv()` method for alloc-free reads, eg: ::
+
+      import espnow
+      e = espnow.ESPNow(); e.active(True)
+      for mac, msg in e:
+          print(mac, msg)
+          if mac is None:   # mac, msg will equal (None, None) on timeout
+              break
+
+.. method:: ESPNow.recvinto(data[, timeout_ms])
+
+    Wait for an incoming message and return the length of the message in bytes.
+    This is the low-level method used by both `recv()<ESPNow.recv()>` and
+    `irecv()` to read messages.
+
+    .. data:: Arguments:
+
+        *data*: A list of at least two elements, ``[peer, msg]``. ``msg`` must
+        be a bytearray large enough to hold the message (250 bytes). On the
+        ESP8266, ``peer`` should be a bytearray of 6 bytes. The MAC address of
+        the sender and the message will be stored in these bytearrays (see Note
+        on ESP32 below).
+
+        *timeout_ms*: (Optional) Timeout in milliseconds (see `ESPNow.recv()`).
+
+    .. data:: Returns:
+
+      - Length of message in bytes or 0 if *timeout_ms* is reached before a
+        message is received.
+
+    .. data:: Raises:
+
+      - See `ESPNow.recv()`.
+
+    **Note:** On the ESP32:
+
+    - It is unnecessary to provide a bytearray in the first element of the
+      ``data`` list because it will be replaced by a reference to a unique
+      ``peer`` address in the **peer device table** (see `ESPNow.peers_table`).
+    - If the list is at least 4 elements long, the rssi and timestamp values
+      will be saved as the 3rd and 4th elements.
+
+.. method:: ESPNow.any()
+
+    Check if data is available to be read with `ESPNow.recv()`.
+
+    For more sophisticated querying of available characters use `select.poll()`::
+
+      import select
+      import espnow
+
+      e = espnow.ESPNow()
+      poll = select.poll()
+      poll.register(e, select.POLLIN)
+      poll.poll(timeout)
+
+    .. data:: Returns:
+
+       ``True`` if data is available to be read, else ``False``.
+
+.. method:: ESPNow.stats() (ESP32 only)
+
+    .. data:: Returns:
+
+      A 5-tuple containing the number of packets sent/received/lost:
+
+      ``(tx_pkts, tx_responses, tx_failures, rx_packets, rx_dropped_packets)``
+
+    Incoming packets are *dropped* when the recv buffers are full. To reduce
+    packet loss, increase the ``rxbuf`` config parameters and ensure you are
+    reading messages as quickly as possible.
+
+    **Note**: Dropped packets will still be acknowledged to the sender as
+    received.
+
+Peer Management
+---------------
+
+On ESP32 devices, the Espressif ESP-NOW software requires that other devices
+(peers) must be *registered* using `add_peer()` before we can
+`send()<ESPNow.send()>` them messages (this is *not* enforced on ESP8266
+devices). It is **not** necessary to register a peer to receive an
+un-encrypted message from that peer.
+
+**Encrypted messages**: To receive an *encrypted* message, the receiving device
+must first register the sender and use the same encryption keys as the sender
+(PMK and LMK) (see `set_pmk()` and `add_peer()`.
+
+.. method:: ESPNow.set_pmk(pmk)
+
+    Set the Primary Master Key (PMK) which is used to encrypt the Local Master
+    Keys (LMK) for encrypting messages. If this is not set, a default PMK is
+    used by the underlying Espressif ESP-NOW software stack.
+
+    **Note:** messages will only be encrypted if *lmk* is also set in
+    `ESPNow.add_peer()` (see `Security
+    <https://docs.espressif.com/projects/esp-idf/en/latest/
+    esp32/api-reference/network/esp_now.html#security>`_ in the Espressif API
+    docs).
+
+    .. data:: Arguments:
+
+      *pmk*: Must be a byte string, bytearray or string of length
+      `espnow.KEY_LEN` (16 bytes).
+
+    .. data:: Returns:
+
+      ``None``
+
+    .. data:: Raises:
+
+      ``ValueError()`` on invalid *pmk* values.
+
+.. method:: ESPNow.add_peer(mac, [lmk], [channel], [ifidx], [encrypt])
+            ESPNow.add_peer(mac, param=value, ...)   (ESP32 only)
+
+    Add/register the provided *mac* address as a peer. Additional parameters may
+    also be specified as positional or keyword arguments (any parameter set to
+    ``None`` will be set to it's default value):
+
+    .. data:: Arguments:
+
+        - *mac*: The MAC address of the peer (as a 6-byte byte-string).
+
+        - *lmk*: The Local Master Key (LMK) key used to encrypt data
+          transfers with this peer (unless the *encrypt* parameter is set to
+          ``False``). Must be:
+
+          - a byte-string or bytearray or string of length ``espnow.KEY_LEN``
+            (16 bytes), or
+
+          - any non ``True`` python value (default= ``b''``), signifying an
+            *empty* key which will disable encryption.
+
+        - *channel*: The wifi channel (2.4GHz) to communicate with this peer.
+          Must be an integer from 0 to 14. If channel is set to 0 the current
+          channel of the wifi device will be used. (default=0)
+
+        - *ifidx*: (ESP32 only) Index of the wifi interface which will be
+          used to send data to this peer. Must be an integer set to
+          ``network.STA_IF`` (=0) or ``network.AP_IF`` (=1).
+          (default=0/``network.STA_IF``). See `ESPNow and Wifi Operation`_
+          below for more information.
+
+        - *encrypt*: (ESP32 only) If set to ``True`` data exchanged with
+          this peer will be encrypted with the PMK and LMK. (default =
+          ``True`` if *lmk* is set to a valid key, else ``False``)
+
+        **ESP8266**: Keyword args may not be used on the ESP8266.
+
+        **Note:** The maximum number of peers which may be registered is 20
+        (`espnow.MAX_TOTAL_PEER_NUM`), with a maximum of 6
+        (`espnow.MAX_ENCRYPT_PEER_NUM`) of those peers with encryption enabled
+        (see `ESP_NOW_MAX_ENCRYPT_PEER_NUM <https://docs.espressif.com/
+        projects/esp-idf/en/latest/esp32/api-reference/network/
+        esp_now.html#c.ESP_NOW_MAX_ENCRYPT_PEER_NUM>`_ in the Espressif API
+        docs).
+
+    .. data:: Raises:
+
+        - ``OSError(num, "ESP_ERR_ESPNOW_NOT_INIT")`` if not initialised.
+        - ``OSError(num, "ESP_ERR_ESPNOW_EXIST")`` if *mac* is already
+          registered.
+        - ``OSError(num, "ESP_ERR_ESPNOW_FULL")`` if too many peers are
+          already registered.
+        - ``ValueError()`` on invalid keyword args or values.
+
+.. method:: ESPNow.del_peer(mac)
+
+    Deregister the peer associated with the provided *mac* address.
+
+    .. data:: Returns:
+
+        ``None``
+
+    .. data:: Raises:
+
+        - ``OSError(num, "ESP_ERR_ESPNOW_NOT_INIT")`` if not initialised.
+        - ``OSError(num, "ESP_ERR_ESPNOW_NOT_FOUND")`` if *mac* is not
+          registered.
+        - ``ValueError()`` on invalid *mac* values.
+
+.. method:: ESPNow.get_peer(mac) (ESP32 only)
+
+    Return information on a registered peer.
+
+    .. data:: Returns:
+
+        ``(mac, lmk, channel, ifidx, encrypt)``: a tuple of the "peer
+        info" associated with the given *mac* address.
+
+    .. data:: Raises:
+
+        - ``OSError(num, "ESP_ERR_ESPNOW_NOT_INIT")`` if not initialised.
+        - ``OSError(num, "ESP_ERR_ESPNOW_NOT_FOUND")`` if *mac* is not
+          registered.
+        - ``ValueError()`` on invalid *mac* values.
+
+.. method:: ESPNow.peer_count() (ESP32 only)
+
+    Return the number of registered peers:
+
+    - ``(peer_num, encrypt_num)``: where
+
+      - ``peer_num`` is the number of peers which are registered, and
+      - ``encrypt_num`` is the number of encrypted peers.
+
+.. method:: ESPNow.get_peers() (ESP32 only)
+
+    Return the "peer info" parameters for all the registered peers (as a tuple
+    of tuples).
+
+.. method:: ESPNow.mod_peer(mac, lmk, [channel], [ifidx], [encrypt]) (ESP32 only)
+            ESPNow.mod_peer(mac, 'param'=value, ...) (ESP32 only)
+
+    Modify the parameters of the peer associated with the provided *mac*
+    address. Parameters may be provided as positional or keyword arguments
+    (see `ESPNow.add_peer()`). Any parameter that is not set (or set to
+    ``None``) will retain the existing value for that parameter.
+
+Callback Methods
+----------------
+
+.. method:: ESPNow.irq(callback) (ESP32 only)
+
+  Set a callback function to be called *as soon as possible* after a message has
+  been received from another ESPNow device. The callback function will be called
+  with the `ESPNow` instance object as an argument. For more reliable operation,
+  it is recommended to read out as many messages as are available when the
+  callback is invoked and to set the read timeout to zero, eg: ::
+
+        def recv_cb(e):
+            while True:  # Read out all messages waiting in the buffer
+                mac, msg = e.irecv(0)  # Don't wait if no messages left
+                if mac is None:
+                    return
+                print(mac, msg)
+        e.irq(recv_cb)
+
+  The `irq()<ESPNow.irq()>` callback method is an alternative method for
+  processing incoming messages, especially if the data rate is moderate
+  and the device is *not too busy* but there are some caveats:
+
+  - The scheduler stack *can* overflow and callbacks will be missed if
+    packets are arriving at a sufficient rate or if other MicroPython components
+    (eg, bluetooth, machine.Pin.irq(), machine.timer, i2s, ...) are exercising
+    the scheduler stack. This method may be less reliable for dealing with
+    bursts of messages, or high throughput or on a device which is busy dealing
+    with other hardware operations.
+
+  - For more information on *scheduled* function callbacks see:
+    `micropython.schedule()<micropython.schedule>`.
+
+Constants
+---------
+
+.. data:: espnow.MAX_DATA_LEN(=250)
+          espnow.KEY_LEN(=16)
+          espnow.ADDR_LEN(=6)
+          espnow.MAX_TOTAL_PEER_NUM(=20)
+          espnow.MAX_ENCRYPT_PEER_NUM(=6)
+
+Exceptions
+----------
+
+If the underlying Espressif ESP-NOW software stack returns an error code,
+the MicroPython espnow module will raise an ``OSError(errnum, errstring)``
+exception where ``errstring`` is set to the name of one of the error codes
+identified in the
+`Espressif ESP-NOW docs
+<https://docs.espressif.com/projects/esp-idf/en/latest/
+api-reference/network/esp_now.html#api-reference>`_. For example::
+
+    try:
+        e.send(peer, 'Hello')
+    except OSError as err:
+        if len(err.args) < 2:
+            raise err
+        if err.args[1] == 'ESP_ERR_ESPNOW_NOT_INIT':
+            e.active(True)
+        elif err.args[1] == 'ESP_ERR_ESPNOW_NOT_FOUND':
+            e.add_peer(peer)
+        elif err.args[1] == 'ESP_ERR_ESPNOW_IF':
+            network.WLAN(network.STA_IF).active(True)
+        else:
+            raise err
+
+Wifi Signal Strength (RSSI) - (ESP32 only)
+------------------------------------------
+
+The ESPNow object maintains a **peer device table** which contains the signal
+strength and timestamp of the last received message from all hosts. The **peer
+device table** can be accessed using `ESPNow.peers_table` and can be used to
+track device proximity and identify *nearest neighbours* in a network of peer
+devices. This feature is **not** available on ESP8266 devices.
+
+.. data:: ESPNow.peers_table
+
+    A reference to the **peer device table**: a dict of known peer devices
+    and rssi values::
+
+        {peer: [rssi, time_ms], ...}
+
+    where:
+
+    - ``peer`` is the peer MAC address (as `bytes`);
+    - ``rssi`` is the wifi signal strength in dBm (-127 to 0) of the last
+      message received from the peer; and
+    - ``time_ms`` is the time the message was received (in milliseconds since
+      system boot - wraps every 12 days).
+
+    Example::
+
+      >>> e.peers_table
+      {b'\xaa\xaa\xaa\xaa\xaa\xaa': [-31, 18372],
+       b'\xbb\xbb\xbb\xbb\xbb\xbb': [-43, 12541]}
+
+    **Note**: the ``mac`` addresses returned by `recv()` are references to
+    the ``peer`` key values in the **peer device table**.
+
+    **Note**: RSSI and timestamp values in the device table are updated only
+    when the message is read by the application.
+
+Supporting asyncio
+------------------
+
+A supplementary module (`aioespnow`) is available to provide
+:doc:`asyncio<asyncio>` support.
+
+**Note:** Asyncio support is available on all ESP32 targets as well as those
+ESP8266 boards which include the asyncio module (ie. ESP8266 devices with at
+least 2MB flash memory).
+
+A small async server example::
+
+    import network
+    import aioespnow
+    import asyncio
+
+    # A WLAN interface must be active to send()/recv()
+    network.WLAN(network.STA_IF).active(True)
+
+    e = aioespnow.AIOESPNow()  # Returns AIOESPNow enhanced with async support
+    e.active(True)
+    peer = b'\xbb\xbb\xbb\xbb\xbb\xbb'
+    e.add_peer(peer)
+
+    # Send a periodic ping to a peer
+    async def heartbeat(e, peer, period=30):
+        while True:
+            if not await e.asend(peer, b'ping'):
+                print("Heartbeat: peer not responding:", peer)
+            else:
+                print("Heartbeat: ping", peer)
+            await asyncio.sleep(period)
+
+    # Echo any received messages back to the sender
+    async def echo_server(e):
+        async for mac, msg in e:
+            print("Echo:", msg)
+            try:
+                await e.asend(mac, msg)
+            except OSError as err:
+                if len(err.args) > 1 and err.args[1] == 'ESP_ERR_ESPNOW_NOT_FOUND':
+                    e.add_peer(mac)
+                    await e.asend(mac, msg)
+
+    async def main(e, peer, timeout, period):
+        asyncio.create_task(heartbeat(e, peer, period))
+        asyncio.create_task(echo_server(e))
+        await asyncio.sleep(timeout)
+
+    asyncio.run(main(e, peer, 120, 10))
+
+.. module:: aioespnow
+    :synopsis: ESP-NOW :doc:`asyncio` support
+
+.. class:: AIOESPNow()
+
+    The `AIOESPNow` class inherits all the methods of `ESPNow<espnow.ESPNow>`
+    and extends the interface with the following async methods.
+
+.. method:: async AIOESPNow.arecv()
+
+    Asyncio support for `ESPNow.recv()`. Note that this method does not take a
+    timeout value as argument.
+
+.. method:: async AIOESPNow.airecv()
+
+    Asyncio support for `ESPNow.irecv()`. Note that this method does not take a
+    timeout value as argument.
+
+.. method:: async AIOESPNow.asend(mac, msg, sync=True)
+            async AIOESPNow.asend(msg)
+
+    Asyncio support for `ESPNow.send()`.
+
+.. method:: AIOESPNow._aiter__() / async AIOESPNow.__anext__()
+
+    `AIOESPNow` also supports reading incoming messages by asynchronous
+    iteration using ``async for``; eg::
+
+      e = AIOESPNow()
+      e.active(True)
+      async def recv_till_halt(e):
+          async for mac, msg in e:
+              print(mac, msg)
+              if msg == b'halt':
+                break
+      asyncio.run(recv_till_halt(e))
+
+Broadcast and Multicast
+-----------------------
+
+All active ESPNow clients will receive messages sent to their MAC address and
+all devices (**except ESP8266 devices**) will also receive messages sent to the
+*broadcast* MAC address (``b'\xff\xff\xff\xff\xff\xff'``) or any multicast
+MAC address.
+
+All ESPNow devices (including ESP8266 devices) can also send messages to the
+broadcast MAC address or any multicast MAC address.
+
+To `send()<ESPNow.send()>` a broadcast message, the broadcast (or
+multicast) MAC address must first be registered using
+`add_peer()<ESPNow.add_peer()>`. `send()<ESPNow.send()>` will always return
+``True`` for broadcasts, regardless of whether any devices receive the
+message. It is not permitted to encrypt messages sent to the broadcast
+address or any multicast address.
+
+**Note**: `ESPNow.send(None, msg)<ESPNow.send()>` will send to all registered
+peers *except* the broadcast address. To send a broadcast or multicast
+message, you must specify the broadcast (or multicast) MAC address as the
+peer. For example::
+
+    bcast = b'\xff' * 6
+    e.add_peer(bcast)
+    e.send(bcast, "Hello World!")
+
+ESPNow and Wifi Operation
+-------------------------
+
+ESPNow messages may be sent and received on any `active()<network.WLAN.active>`
+`WLAN<network.WLAN()>` interface (``network.STA_IF`` or ``network.AP_IF``), even
+if that interface is also connected to a wifi network or configured as an access
+point. When an ESP32 or ESP8266 device connects to a Wifi Access Point (see
+`ESP32 Quickref <../esp32/quickref.html#networking>`__) the following things
+happen which affect ESPNow communications:
+
+1. Wifi Power-saving Mode (`network.WLAN.PM_PERFORMANCE`)
+   is automatically activated and
+2. The radio on the esp device changes wifi ``channel`` to match the channel
+   used by the Access Point.
+
+**Wifi Power-saving Mode:** (see `Espressif Docs <https://docs.espressif.com/
+projects/esp-idf/en/latest/esp32/api-guides/
+wifi.html#esp32-wi-fi-power-saving-mode>`_) The power saving mode causes the
+device to turn off the radio periodically (typically for hundreds of
+milliseconds), making it unreliable in receiving ESPNow messages. This can be
+resolved by either of:
+
+1. Disabling the power-saving mode on the STA_IF interface;
+
+   - Use ``sta.config(pm=sta.PM_NONE)``
+
+2. Turning on the AP_IF interface, which will disable the power saving mode.
+   However, the device will then be advertising an active wifi access point.
+
+   - You **may** also choose to send your messages via the AP_IF interface, but
+     this is not necessary.
+   - ESP8266 peers must send messages to this AP_IF interface (see below).
+
+3. Configuring ESPNow clients to retry sending messages.
+
+**Receiving messages from an ESP8266 device:** Strangely, an ESP32 device
+connected to a wifi network using method 1 or 2 above, will receive ESPNow
+messages sent to the STA_IF MAC address from another ESP32 device, but will
+**reject** messages from an ESP8266 device!!!. To receive messages from an
+ESP8266 device, the AP_IF interface must be set to ``active(True)`` **and**
+messages must be sent to the AP_IF MAC address.
+
+**Managing wifi channels:** Any other ESPNow devices wishing to communicate with
+a device which is also connected to a Wifi Access Point MUST use the same
+channel. A common scenario is where one ESPNow device is connected to a wifi
+router and acts as a proxy for messages from a group of sensors connected via
+ESPNow:
+
+**Proxy:** ::
+
+  import network, time, espnow
+
+  sta, ap = wifi_reset()  # Reset wifi to AP off, STA on and disconnected
+  sta.connect('myssid', 'mypassword')
+  while not sta.isconnected():  # Wait until connected...
+      time.sleep(0.1)
+  sta.config(pm=sta.PM_NONE)  # ..then disable power saving
+
+  # Print the wifi channel used AFTER finished connecting to access point
+  print("Proxy running on channel:", sta.config("channel"))
+  e = espnow.ESPNow(); e.active(True)
+  for peer, msg in e:
+      # Receive espnow messages and forward them to MQTT broker over wifi
+
+**Sensor:** ::
+
+  import network, espnow
+
+  sta, ap = wifi_reset()   # Reset wifi to AP off, STA on and disconnected
+  sta.config(channel=6)    # Change to the channel used by the proxy above.
+  peer = b'0\xaa\xaa\xaa\xaa\xaa'  # MAC address of proxy
+  e = espnow.ESPNow(); e.active(True);
+  e.add_peer(peer)
+  while True:
+      msg = read_sensor()
+      e.send(peer, msg)
+      time.sleep(1)
+
+Other issues to take care with when using ESPNow with wifi are:
+
+- **Set WIFI to known state on startup:** MicroPython does not reset the wifi
+  peripheral after a soft reset. This can lead to unexpected behaviour. To
+  guarantee the wifi is reset to a known state after a soft reset make sure you
+  deactivate the STA_IF and AP_IF before setting them to the desired state at
+  startup, eg.::
+
+    import network, time
+
+    def wifi_reset():   # Reset wifi to AP_IF off, STA_IF on and disconnected
+      sta = network.WLAN(network.STA_IF); sta.active(False)
+      ap = network.WLAN(network.AP_IF); ap.active(False)
+      sta.active(True)
+      while not sta.active():
+          time.sleep(0.1)
+      sta.disconnect()   # For ESP8266
+      while sta.isconnected():
+          time.sleep(0.1)
+      return sta, ap
+
+    sta, ap = wifi_reset()
+
+  Remember that a soft reset occurs every time you connect to the device REPL
+  and when you type ``ctrl-D``.
+
+- **STA_IF and AP_IF always operate on the same channel:** the AP_IF will change
+  channel when you connect to a wifi network; regardless of the channel you set
+  for the AP_IF (see `Attention Note 3
+  <https://docs.espressif.com/
+  projects/esp-idf/en/latest/esp32/api-reference/network/esp_wifi.html
+  #_CPPv419esp_wifi_set_config16wifi_interface_tP13wifi_config_t>`_
+  ). After all, there is really only one wifi radio on the device, which is
+  shared by the STA_IF and AP_IF virtual devices.
+
+- **Disable automatic channel assignment on your wifi router:** If the wifi
+  router for your wifi network is configured to automatically assign the wifi
+  channel, it may change the channel for the network if it detects interference
+  from other wifi routers. When this occurs, the ESP devices connected to the
+  wifi network will also change channels to match the router, but other
+  ESPNow-only devices will remain on the previous channel and communication will
+  be lost. To mitigate this, either set your wifi router to use a fixed wifi
+  channel or configure your devices to re-scan the wifi channels if they are
+  unable to find their expected peers on the current channel.
+
+- **MicroPython re-scans wifi channels when trying to reconnect:** If the esp
+  device is connected to a Wifi Access Point that goes down, MicroPython will
+  automatically start scanning channels in an attempt to reconnect to the
+  Access Point. This means ESPNow messages will be lost while scanning for the
+  AP. This can be disabled by ``sta.config(reconnects=0)``, which will also
+  disable the automatic reconnection after losing connection.
+
+- Some versions of the ESP IDF only permit sending ESPNow packets from the
+  STA_IF interface to peers which have been registered on the same wifi
+  channel as the STA_IF::
+
+    ESPNOW: Peer channel is not equal to the home channel, send fail!
+
+ESPNow and Sleep Modes
+----------------------
+
+The `machine.lightsleep([time_ms])<machine.lightsleep>` and
+`machine.deepsleep([time_ms])<machine.deepsleep>` functions can be used to put
+the ESP32 and peripherals (including the WiFi and Bluetooth radios) to sleep.
+This is useful in many applications to conserve battery power. However,
+applications must disable the WLAN peripheral (using
+`active(False)<network.WLAN.active>`) before entering light or deep sleep (see
+`Sleep Modes <https://docs.espressif.com/
+projects/esp-idf/en/latest/esp32/api-reference/system/sleep_modes.html>`_).
+Otherwise the WiFi radio may not be initialised properly after wake from
+sleep. If the ``STA_IF`` and ``AP_IF`` interfaces have both been set
+`active(True)<network.WLAN.active()>` then both interfaces should be set
+`active(False)<network.WLAN.active()>` before entering any sleep mode.
+
+**Example:** deep sleep::
+
+  import network, machine, espnow
+
+  sta, ap = wifi_reset()            # Reset wifi to AP off, STA on and disconnected
+  peer = b'0\xaa\xaa\xaa\xaa\xaa'   # MAC address of peer
+  e = espnow.ESPNow()
+  e.active(True)
+  e.add_peer(peer)                  # Register peer on STA_IF
+
+  print('Sending ping...')
+  if not e.send(peer, b'ping'):
+    print('Ping failed!')
+  e.active(False)
+  sta.active(False)                 # Disable the wifi before sleep
+  print('Going to sleep...')
+  machine.deepsleep(10000)          # Sleep for 10 seconds then reboot
+
+**Example:** light sleep::
+
+  import network, machine, espnow
+
+  sta, ap = wifi_reset()            # Reset wifi to AP off, STA on and disconnected
+  sta.config(channel=6)
+  peer = b'0\xaa\xaa\xaa\xaa\xaa'   # MAC address of peer
+  e = espnow.ESPNow()
+  e.active(True)
+  e.add_peer(peer)                  # Register peer on STA_IF
+
+  while True:
+    print('Sending ping...')
+    if not e.send(peer, b'ping'):
+      print('Ping failed!')
+    sta.active(False)               # Disable the wifi before sleep
+    print('Going to sleep...')
+    machine.lightsleep(10000)       # Sleep for 10 seconds
+    sta.active(True)
+    sta.config(channel=6)           # Wifi loses config after lightsleep()
+

docs/library/framebuf.rst

@@ -1,4 +1,4 @@
-:mod:`framebuf` --- Frame buffer manipulation
+:mod:`framebuf` --- frame buffer manipulation
 =============================================
 
 .. module:: framebuf
@@ -11,24 +11,24 @@ class FrameBuffer
 -----------------
 
 The FrameBuffer class provides a pixel buffer which can be drawn upon with
-pixels, lines, rectangles, text and even other FrameBuffer's. It is useful
-when generating output for displays.
+pixels, lines, rectangles, ellipses, polygons, text and even other
+FrameBuffers. It is useful when generating output for displays.
 
 For example::
 
     import framebuf
 
     # FrameBuffer needs 2 bytes for every RGB565 pixel
-    fbuf = framebuf.FrameBuffer(bytearray(10 * 100 * 2), 10, 100, framebuf.RGB565)
+    fbuf = framebuf.FrameBuffer(bytearray(100 * 10 * 2), 100, 10, framebuf.RGB565)
 
     fbuf.fill(0)
     fbuf.text('MicroPython!', 0, 0, 0xffff)
-    fbuf.hline(0, 10, 96, 0xffff)
+    fbuf.hline(0, 9, 96, 0xffff)
 
 Constructors
 ------------
 
-.. class:: FrameBuffer(buffer, width, height, format, stride=width)
+.. class:: FrameBuffer(buffer, width, height, format, stride=width, /)
 
     Construct a FrameBuffer object.  The parameters are:
 
@@ -77,12 +77,37 @@ The following methods draw shapes onto the FrameBuffer.
     methods draw horizontal and vertical lines respectively up to
     a given length.
 
-.. method:: FrameBuffer.rect(x, y, w, h, c)
-.. method:: FrameBuffer.fill_rect(x, y, w, h, c)
+.. method:: FrameBuffer.rect(x, y, w, h, c[, f])
 
-    Draw a rectangle at the given location, size and color. The `rect`
-    method draws only a 1 pixel outline whereas the `fill_rect` method
-    draws both the outline and interior.
+    Draw a rectangle at the given location, size and color.
+
+    The optional *f* parameter can be set to ``True`` to fill the rectangle.
+    Otherwise just a one pixel outline is drawn.
+
+.. method:: FrameBuffer.ellipse(x, y, xr, yr, c[, f, m])
+
+    Draw an ellipse at the given location. Radii *xr* and *yr* define the
+    geometry; equal values cause a circle to be drawn. The *c* parameter
+    defines the color.
+
+    The optional *f* parameter can be set to ``True`` to fill the ellipse.
+    Otherwise just a one pixel outline is drawn.
+
+    The optional *m* parameter enables drawing to be restricted to certain
+    quadrants of the ellipse. The LS four bits determine which quadrants are
+    to be drawn, with bit 0 specifying Q1, b1 Q2, b2 Q3 and b3 Q4. Quadrants
+    are numbered counterclockwise with Q1 being top right.
+
+.. method:: FrameBuffer.poly(x, y, coords, c[, f])
+
+    Given a list of coordinates, draw an arbitrary (convex or concave) closed
+    polygon at the given x, y location using the given color.
+
+    The *coords* must be specified as a :mod:`array` of integers, e.g.
+    ``array('h', [x0, y0, x1, y1, ... xn, yn])``.
+
+    The optional *f* parameter can be set to ``True`` to fill the polygon.
+    Otherwise just a one pixel outline is drawn.
 
 Drawing text
 ------------
@@ -103,16 +128,25 @@ Other methods
     Shift the contents of the FrameBuffer by the given vector. This may
     leave a footprint of the previous colors in the FrameBuffer.
 
-.. method:: FrameBuffer.blit(fbuf, x, y[, key])
+.. method:: FrameBuffer.blit(fbuf, x, y, key=-1, palette=None)
 
     Draw another FrameBuffer on top of the current one at the given coordinates.
     If *key* is specified then it should be a color integer and the
     corresponding color will be considered transparent: all pixels with that
-    color value will not be drawn.
+    color value will not be drawn. (If the *palette* is specified then the *key*
+    is compared to the value from *palette*, not to the value directly from
+    *fbuf*.)
 
-    This method works between FrameBuffer instances utilising different formats,
-    but the resulting colors may be unexpected due to the mismatch in color
-    formats.
+    The *palette* argument enables blitting between FrameBuffers with differing
+    formats. Typical usage is to render a monochrome or grayscale glyph/icon to
+    a color display. The *palette* is a FrameBuffer instance whose format is
+    that of the current FrameBuffer. The *palette* height is one pixel and its
+    pixel width is the number of colors in the source FrameBuffer. The *palette*
+    for an N-bit source needs 2**N pixels; the *palette* for a monochrome source
+    would have 2 pixels representing background and foreground colors. The
+    application assigns a color to each pixel in the *palette*. The color of the
+    current pixel will be that of that *palette* pixel whose x position is the
+    color of the corresponding source pixel.
 
 Constants
 ---------
@@ -130,7 +164,7 @@ Constants
 
     Monochrome (1-bit) color format
     This defines a mapping where the bits in a byte are horizontally mapped.
-    Each byte occupies 8 horizontal pixels with bit 0 being the leftmost.
+    Each byte occupies 8 horizontal pixels with bit 7 being the leftmost.
     Subsequent bytes appear at successive horizontal locations until the
     rightmost edge is reached. Further bytes are rendered on the next row, one
     pixel lower.
@@ -139,7 +173,7 @@ Constants
 
     Monochrome (1-bit) color format
     This defines a mapping where the bits in a byte are horizontally mapped.
-    Each byte occupies 8 horizontal pixels with bit 7 being the leftmost.
+    Each byte occupies 8 horizontal pixels with bit 0 being the leftmost.
     Subsequent bytes appear at successive horizontal locations until the
     rightmost edge is reached. Further bytes are rendered on the next row, one
     pixel lower.

docs/library/gc.rst

@@ -24,7 +24,7 @@ Functions
 
 .. function:: mem_alloc()
 
-   Return the number of bytes of heap RAM that are allocated.
+   Return the number of bytes of heap RAM that are allocated by Python code.
 
    .. admonition:: Difference to CPython
       :class: attention
@@ -33,8 +33,8 @@ Functions
 
 .. function:: mem_free()
 
-   Return the number of bytes of available heap RAM, or -1 if this amount
-   is not known.
+   Return the number of bytes of heap RAM that is available for Python
+   code to allocate, or -1 if this amount is not known.
 
    .. admonition:: Difference to CPython
       :class: attention

docs/library/gzip.rst

@@ -0,0 +1,106 @@
+:mod:`gzip` -- gzip compression & decompression
+===============================================
+
+.. module:: gzip
+   :synopsis: gzip compression & decompression
+
+|see_cpython_module| :mod:`python:gzip`.
+
+This module allows compression and decompression of binary data with the
+`DEFLATE algorithm <https://en.wikipedia.org/wiki/DEFLATE>`_ used by the gzip
+file format.
+
+.. note:: Prefer to use :class:`deflate.DeflateIO` instead of the functions in this
+   module as it provides a streaming interface to compression and decompression
+   which is convenient and more memory efficient when working with reading or
+   writing compressed data to a file, socket, or stream.
+
+**Availability:**
+
+* This module is **not present by default** in official MicroPython firmware
+  releases as it duplicates functionality available in the :mod:`deflate
+  <deflate>` module.
+
+* A copy of this module can be installed (or frozen)
+  from :term:`micropython-lib` (`source <https://github.com/micropython/micropython-lib/blob/master/python-stdlib/gzip/gzip.py>`_).
+  See :ref:`packages` for more information. This documentation describes that module.
+
+* Compression support will only be available if compression support is enabled
+  in the built-in :mod:`deflate <deflate>` module.
+
+Functions
+---------
+
+.. function:: open(filename, mode, /)
+
+   Wrapper around built-in :func:`open` returning a GzipFile instance.
+
+.. function:: decompress(data, /)
+
+   Decompresses *data* into a bytes object.
+
+.. function:: compress(data, /)
+
+   Compresses *data* into a bytes object.
+
+Classes
+-------
+
+.. class:: GzipFile(*, fileobj, mode)
+
+   This class can be used to wrap a *fileobj* which is any
+   :term:`stream-like <stream>` object such as a file, socket, or stream
+   (including :class:`io.BytesIO`). It is itself a stream and implements the
+   standard read/readinto/write/close methods.
+
+   When the *mode* argument is ``"rb"``, reads from the GzipFile instance will
+   decompress the data in the underlying stream and return decompressed data.
+
+   If compression support is enabled then the *mode* argument can be set to
+   ``"wb"``, and writes to the GzipFile instance will be compressed and written
+   to the underlying stream.
+
+   By default the GzipFile class will read and write data using the gzip file
+   format, including a header and footer with checksum and a window size of 512
+   bytes.
+
+   The **file**, **compresslevel**, and **mtime** arguments are not
+   supported. **fileobj** and **mode** must always be specified as keyword
+   arguments.
+
+Examples
+--------
+
+A typical use case for :class:`gzip.GzipFile` is to read or write a compressed
+file from storage:
+
+.. code:: python
+
+   import gzip
+
+   # Reading:
+   with open("data.gz", "rb") as f:
+       with gzip.GzipFile(fileobj=f, mode="rb") as g:
+           # Use g.read(), g.readinto(), etc.
+
+    # Same, but using gzip.open:
+   with gzip.open("data.gz", "rb") as f:
+        # Use f.read(), f.readinto(), etc.
+
+   # Writing:
+   with open("data.gz", "wb") as f:
+       with gzip.GzipFile(fileobj=f, mode="wb") as g:
+           # Use g.write(...) etc
+
+   # Same, but using gzip.open:
+   with gzip.open("data.gz", "wb") as f:
+       # Use f.write(...) etc
+
+   # Write a dictionary as JSON in gzip format, with a
+   # small (64 byte) window size.
+   config = { ... }
+   with gzip.open("config.gz", "wb") as f:
+       json.dump(config, f)
+
+For guidance on working with gzip sources and choosing the window size see the
+note at the :ref:`end of the deflate documentation <deflate_wbits>`.