diff options
author | Damien George <damien.p.george@gmail.com> | 2017-08-14 12:15:54 +1000 |
---|---|---|
committer | Damien George <damien.p.george@gmail.com> | 2017-08-14 12:15:54 +1000 |
commit | 5e76ea4affd0bd46e67b456496818803010a2d24 (patch) | |
tree | 2be81a889e765b8e24614bcd8bc80c0a4063e2f3 /py/gc.c | |
parent | b4078cbbf3354229fc81ecfd26295237d808caa9 (diff) | |
parent | 1459a8d5c9b29c78da2cf5c7cf3c37ab03b34b8e (diff) |
Merge tag 'v1.8.2' into parse-bytecode
Thread support, ESP8266 memory improvements, btree module, improved docs
This release brings multi-threading support in the form of the _thread
module, which closely matches the semantics of the corresponding CPython
module. There is support for GIL and non-GIL builds; without the GIL
enabled one must protect concurrent access to mutable Python state at the
Python level using Lock objects. Threading with the GIL is enabled in
the cc3200 port on the WiPy. Threading without the GIL is enabled on the
unix port. The ESP8266 port has support for frozen bytecode (for scripts
in the modules/ subdirectory), as well as optimisations for reduced
memory usage and decreased memory fragmentation. The ESP8266 RTC also
resumes correctly after a deepsleep. An initial "btree" module has been
implemented for database support (unix port only), and the documentation
has been further improved, with pre-built PDF versions of the docs now
available.
py core:
- parse: treat constants that start with underscore as private
- objdict: implemented OrderedDict equality check
- support to build berkeley db 1.85 and "btree" module
- mpconfig.h: MP_NOINLINE is universally useful, move from unix port
- makeqstrdefs.py: remove restriction that source path can't be absolute
- mkrules.mk: define "lib" outside conditional block
- rename __QSTR_EXTRACT flag to NO_QSTR
- objtype: instance: inherit protocol vtable from a base class
- mphal.h: if virtpin API is used, automagically include its header
- objtype: inherit protocol vtable from base class only if it exists
- add MP_STATE_THREAD to hold state specific to a given thread
- add basic _thread module, with ability to start a new thread
- modthread: properly cast concrete exception pointer to an object
- modthread: add stack_size() function
- modthread: add exit() function
- modthread: implement lock object, for creating a mutex
- modthread: add with-context capabilities to lock object
- gc: make memory manager and garbage collector thread safe
- modthread: satisfy unused-args warning
- gc: fix 2 cases of concurrent access to ATB and FTB
- modthread: be more careful with root pointers when creating a thread
- modthread: call mp_thread_start/mp_thread_finish around threads
- gc: fix GC+thread bug where ptr gets lost because it's not computed
- make interning of qstrs thread safe
- implement a simple global interpreter lock (GIL)
- don't use gc or qstr mutex when the GIL is enabled, there is no need
- modthread: make Lock objects work when GIL is enabled
- mpthread.h: move GIL macros outside MICROPY_PY_THREAD block
- modthread: allow to properly set the stack limit of a thread
- nlrthumb: convert NLR thumb funcs from asm to C with inline-asm
- nlrsetjmp: update to take into account new location of nlr_top
- mpthread: include mpstate.h when defining GIL macros
- gc: be sure to count last allocated block at heap end in stats
- gc: calculate (and report) maximum contiguous free block size
- builtinimport: disable "imported as namespace package" warning
- mpconfig.h: mention MICROPY_PY_BTREE config option
- objarray: split out header to allow direct access to object
- rename mp_obj_type_t::stream_p to "protocol"
extmod:
- modbtree: initial implementation of "btree" module based on BerkeleyDB
- modbtree: handle default value and error check
- modbtree: implement .items() iterator
- modbtree: actually implement end key support for .items()
- modbtree: items(): Implement "end key inclusive" flag
- modbtree: items(): Implement DESC flag
- modbtree: __getitem__() should raise KeyError for non-existing key
- modbtree: open(): Support "in-memory" database with filename=None
- machine_pinbase: implementation of PinBase class
- machine_pinbase: fix nanbox build
- modlwip: store a chain of incoming pbufs, instead of only one
- modbtree: implement keys(), values(), items() iterators
- modbtree: cleverly implement "for key in btree:" syntax
- modwebsocket: add readinto() method
- modwebrepl: add readinto() method
- modbtree: fix unused argument warning
- modbtree: fixes for nanbox build
- moduos_dupterm: reserve buffer bytearray object for dupterm
- moduos_dupterm: reuse dupterm_arr_obj for write operations
lib:
- berkeley-db-1.xx: add Berkeley DB 1.85 as a submodule
- berkeley-db-1.xx: update to upstream which builds for uPy
- fatfs/option/ccsbcs: follow uPy optional features model
- libm: format code to pass gcc v6.1.1 warning
- libm: remove unused definition of "one"
drivers:
- display/ssd1306: add width arg and support 64px wide displays
tests:
- recursive_iternext: clang/Linux is even more stack-frugal than MacOS
- bench: add testcase to compare bytes(N) vs b"\0" * N
- add testcase for OrderedDict equality
- add a testcase for machine.PinBase class
- extmod: add "btree" module test
- btree1: add testcase for iterating btree object directly
- add tests for _thread module
- add 3 more tests for _thread module
- thread: remove need to sleep to wait for completion in some tests
- thread: add tests for running GC within a thread, and heap stress
- thread: rename thread_stress_XXX.py to stress_XXX.py
- thread: add tests that mutate shared objects
- thread: add test for concurrent interning of strings
- thread: add test for concurrent mutating of user instance
- thread: make stack-size test run correctly and reliable on uPy
- thread: make sure that thread tests don't rely on floating point
- thread: allow thread_sleep1 to run without floating point
- thread: allow some tests to run on ports with not much heap
- meminfo: update for addition of "max free sz" output
- run-tests: If running thread tests on unix, don't run mutate ones
unix port:
- unbreak "minimal" target by disabling FatFs
- mpconfigport_minimal.h: clearly mark where user-define config ends
- deprecate support for GNU Readline (MICROPY_USE_READLINE=2)
- time to build with --gc-sections
- move "utime" module config to C level instead of make level
- Makefile: libffi: Build with -Os
- alloc: add option to use uPy's alloc-exec implementation even for libffi
- alloc: make coverage build and its overzealous warnings happy
- disable FatFs VFS for normal build, keep enabled for coverage
- modmachine: include PinBase class
- modmachine: enable time_pulse_us() function
- fix Makefile to handle gc-sections linker flags on Mac OS
- add basic thread support using pthreads
- file: if write syscall returns because of EINTR then try again
- implement garbage collection with threading
- mpthreadport: use SA_SIGINFO for GC signal handler
- gccollect: provide declaration of exported function
- mpthreadport: suppress compiler warning about unused arguments
- modtime: release the GIL when sleeping
- enable btree module
- Makefile: make "minimal" build be minimal again
- mpconfigport_minimal.h: allow to print a string within 1KB of heap
- main: error out on unknown value of suffix in -X heapsize= option
- main: improve help for -X options a bit
- main: when preparing sys.path, allocate exact strings on uPy heap
- disable the GIL to improve performance of non-thread code
windows port:
- follow unix port changes regarding "utime" module
- msvc: include machine_pinbase.c in build and enable umachine module
stmhal port:
- add board definition files for Olimex STM32-E407
- port of f4 hal commit 1d7fb82 to l4 hal (SD card modifications)
- correct DMA to allow SD card on L4 MCUs
- enable SD card on L4 MCUs
cc3200 port:
- update FreeRTOS to v9.0.0
- define our own FreeRTOS heap so it can go in a special segment
- add basic threading capabilities
- fix call to new exception to be _msg instead of _arg1
- use xTaskCreateStatic instead of osi_TaskCreate
- mpthreadport: make mutex statically allocated
- mpthreadport: properly initialise the main thread's data
- gccollect: use MP_STATE_THREAD(stack_top) to get top of stack
- mpthreadport: scan more root pointers from thread data
- mpthreadport: move mem alloc outside the thread_mutex lock
- enable the GIL
- in FreeRTOSConfig.h, comment on configSUPPORT_STATIC_ALLOCATION
- remove .boot section attribute from osi_TaskCreate function
- start the simplelink spawn task using the static task creator
- shrink the FreeRTOS heap and place TCB+stack in freed location
- allow to compile bootloader with threading enabled
- remove obsolete singleton heart-beat object
teensy port:
- update the README with OSX specific information and tips
esp8266 port:
- modnetwork: use struct bss_info::ssid_len for ESSID length
- let RTC work correctly after deepsleep
- switch floating-point arith routines to BootROM
- Makefile: enable --verify option for esptool.py write_flash
- use RTC to set date & time stamps for files
- main.c: clear the command line history when (re)booting
- enable frozen bytecode, with scripts in modules/ subdir
- README: describe how to build mpy-cross
- websocket_helper.py: avoid extra string allocations
- websocket_helper.py: fix typo in debug output
- modpybuart: allow setting baudrate and other params
- explicitly collect garbage in bootstrap scripts
- switch webrepl_setup to use frozen bytecode
- switch webrepl to use frozen bytecode
- README: promote from "highly experimental" to "experimental"
- main: init recently added dupterm_arr_obj port state var
- esp_mphal: call_dupterm_read: Use readinto() method
- esp_mphal: properly handle dupterm EOF after switching to readinto()
qemu-arm port:
- disable gcc LTO option for nlrthumb.c
docs:
- esp8266/tutorial/network_basics: minor typo fix, ap becomes ap_if
- add link to PDF version of docs in sidebar
- library/network: wipy: fix literal block indentation
- esp.socket: remove unused file
- ure: typo fix
- usocket: there must be empty line after .only::
- library/machine.Pin: mention GPIO and cross-link .value() function
- machine.*: add "currentmodule:: machine" directive
- pyb.*: add "currentmodule:: pyb" directive
- machine.Pin: remove explicit target
- machine.UART: cleanup pyboard section
- machine.*: use proper class case in method headers
- machine.Pin: add class designator to all constants
- pyb.CAN: mark CAN.initfilterbanks() as classmethod explicitly
- pyb.Pin: af_list() is a normal method, not a class method
- pyb.ExtInt,pyb.Pin: mark up class methods as such
- pyb.*: use proper class case in method headers
- pyb.Pin: sort .af() and .af_list() methods together
- builtins: enumerate all builtin functions implemented
- library/index: add builtins.rst
- machine.Pin: disambiguate object call method
- machine*: remove explicit targets and "machine." prefixes on classes
- uctypes: improve documentation
- sys: detailed description of print_exception() diff from traceback module
- add sphinx_selective_exclude extension suite
- conf.py: active sphinx_selective_exclude extensions
- rebuild docs from scratch, as required for proper only:: handling
- select: add an article
- sys: print_exception: Fixes/clarifications
- conf.py: exclude cmath from modindex for wipy
- library: fix typo in docs for usocket.listen()
examples:
- unix/ffi_example.py: update for current "ffi" module API
- unix/ffi_example.py: remove TODO, make output more clear
Diffstat (limited to 'py/gc.c')
-rw-r--r-- | py/gc.c | 139 |
1 files changed, 107 insertions, 32 deletions
@@ -94,6 +94,14 @@ #define FTB_CLEAR(block) do { MP_STATE_MEM(gc_finaliser_table_start)[(block) / BLOCKS_PER_FTB] &= (~(1 << ((block) & 7))); } while (0) #endif +#if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL +#define GC_ENTER() mp_thread_mutex_lock(&MP_STATE_MEM(gc_mutex), 1) +#define GC_EXIT() mp_thread_mutex_unlock(&MP_STATE_MEM(gc_mutex)) +#else +#define GC_ENTER() +#define GC_EXIT() +#endif + // TODO waste less memory; currently requires that all entries in alloc_table have a corresponding block in pool void gc_init(void *start, void *end) { // align end pointer on block boundary @@ -144,6 +152,10 @@ void gc_init(void *start, void *end) { // allow auto collection MP_STATE_MEM(gc_auto_collect_enabled) = 1; + #if MICROPY_PY_THREAD + mp_thread_mutex_init(&MP_STATE_MEM(gc_mutex)); + #endif + DEBUG_printf("GC layout:\n"); DEBUG_printf(" alloc table at %p, length " UINT_FMT " bytes, " UINT_FMT " blocks\n", MP_STATE_MEM(gc_alloc_table_start), MP_STATE_MEM(gc_alloc_table_byte_len), MP_STATE_MEM(gc_alloc_table_byte_len) * BLOCKS_PER_ATB); #if MICROPY_ENABLE_FINALISER @@ -153,11 +165,15 @@ void gc_init(void *start, void *end) { } void gc_lock(void) { + GC_ENTER(); MP_STATE_MEM(gc_lock_depth)++; + GC_EXIT(); } void gc_unlock(void) { + GC_ENTER(); MP_STATE_MEM(gc_lock_depth)--; + GC_EXIT(); } bool gc_is_locked(void) { @@ -236,6 +252,10 @@ STATIC void gc_sweep(void) { case AT_HEAD: #if MICROPY_ENABLE_FINALISER if (FTB_GET(block)) { + #if MICROPY_PY_THREAD + // TODO need to think about reentrancy with finaliser code + assert(!"finaliser with threading not implemented"); + #endif mp_obj_base_t *obj = (mp_obj_base_t*)PTR_FROM_BLOCK(block); if (obj->type != NULL) { // if the object has a type then see if it has a __del__ method @@ -272,14 +292,15 @@ STATIC void gc_sweep(void) { } void gc_collect_start(void) { - gc_lock(); + GC_ENTER(); + MP_STATE_MEM(gc_lock_depth)++; MP_STATE_MEM(gc_stack_overflow) = 0; MP_STATE_MEM(gc_sp) = MP_STATE_MEM(gc_stack); // Trace root pointers. This relies on the root pointers being organised // correctly in the mp_state_ctx structure. We scan nlr_top, dict_locals, // dict_globals, then the root pointer section of mp_state_vm. void **ptrs = (void**)(void*)&mp_state_ctx; - gc_collect_root(ptrs, offsetof(mp_state_ctx_t, vm.stack_top) / sizeof(void*)); + gc_collect_root(ptrs, offsetof(mp_state_ctx_t, vm.qstr_last_chunk) / sizeof(void*)); } void gc_collect_root(void **ptrs, size_t len) { @@ -294,31 +315,26 @@ void gc_collect_end(void) { gc_deal_with_stack_overflow(); gc_sweep(); MP_STATE_MEM(gc_last_free_atb_index) = 0; - gc_unlock(); + MP_STATE_MEM(gc_lock_depth)--; + GC_EXIT(); } void gc_info(gc_info_t *info) { + GC_ENTER(); info->total = MP_STATE_MEM(gc_pool_end) - MP_STATE_MEM(gc_pool_start); info->used = 0; info->free = 0; + info->max_free = 0; info->num_1block = 0; info->num_2block = 0; info->max_block = 0; - for (size_t block = 0, len = 0; block < MP_STATE_MEM(gc_alloc_table_byte_len) * BLOCKS_PER_ATB; block++) { + bool finish = false; + for (size_t block = 0, len = 0, len_free = 0; !finish;) { size_t kind = ATB_GET_KIND(block); - if (kind == AT_FREE || kind == AT_HEAD) { - if (len == 1) { - info->num_1block += 1; - } else if (len == 2) { - info->num_2block += 1; - } - if (len > info->max_block) { - info->max_block = len; - } - } switch (kind) { case AT_FREE: info->free += 1; + len_free += 1; len = 0; break; @@ -336,23 +352,51 @@ void gc_info(gc_info_t *info) { // shouldn't happen break; } + + block++; + finish = (block == MP_STATE_MEM(gc_alloc_table_byte_len) * BLOCKS_PER_ATB); + // Get next block type if possible + if (!finish) { + kind = ATB_GET_KIND(block); + } + + if (finish || kind == AT_FREE || kind == AT_HEAD) { + if (len == 1) { + info->num_1block += 1; + } else if (len == 2) { + info->num_2block += 1; + } + if (len > info->max_block) { + info->max_block = len; + } + if (finish || kind == AT_HEAD) { + if (len_free > info->max_free) { + info->max_free = len_free; + } + len_free = 0; + } + } } info->used *= BYTES_PER_BLOCK; info->free *= BYTES_PER_BLOCK; + GC_EXIT(); } void *gc_alloc(size_t n_bytes, bool has_finaliser) { size_t n_blocks = ((n_bytes + BYTES_PER_BLOCK - 1) & (~(BYTES_PER_BLOCK - 1))) / BYTES_PER_BLOCK; DEBUG_printf("gc_alloc(" UINT_FMT " bytes -> " UINT_FMT " blocks)\n", n_bytes, n_blocks); - // check if GC is locked - if (MP_STATE_MEM(gc_lock_depth) > 0) { + // check for 0 allocation + if (n_blocks == 0) { return NULL; } - // check for 0 allocation - if (n_blocks == 0) { + GC_ENTER(); + + // check if GC is locked + if (MP_STATE_MEM(gc_lock_depth) > 0) { + GC_EXIT(); return NULL; } @@ -372,6 +416,7 @@ void *gc_alloc(size_t n_bytes, bool has_finaliser) { if (ATB_3_IS_FREE(a)) { if (++n_free >= n_blocks) { i = i * BLOCKS_PER_ATB + 3; goto found; } } else { n_free = 0; } } + GC_EXIT(); // nothing found! if (collected) { return NULL; @@ -379,6 +424,7 @@ void *gc_alloc(size_t n_bytes, bool has_finaliser) { DEBUG_printf("gc_alloc(" UINT_FMT "): no free mem, triggering GC\n", n_bytes); gc_collect(); collected = 1; + GC_ENTER(); } // found, ending at block i inclusive @@ -406,9 +452,12 @@ found: } // get pointer to first block + // we must create this pointer before unlocking the GC so a collection can find it void *ret_ptr = (void*)(MP_STATE_MEM(gc_pool_start) + start_block * BYTES_PER_BLOCK); DEBUG_printf("gc_alloc(%p)\n", ret_ptr); + GC_EXIT(); + // zero out the additional bytes of the newly allocated blocks // This is needed because the blocks may have previously held pointers // to the heap and will not be set to something else if the caller @@ -421,7 +470,9 @@ found: // clear type pointer in case it is never set ((mp_obj_base_t*)ret_ptr)->type = NULL; // set mp_obj flag only if it has a finaliser + GC_ENTER(); FTB_SET(start_block); + GC_EXIT(); } #else (void)has_finaliser; @@ -447,8 +498,10 @@ void *gc_alloc_with_finaliser(mp_uint_t n_bytes) { // force the freeing of a piece of memory // TODO: freeing here does not call finaliser void gc_free(void *ptr) { + GC_ENTER(); if (MP_STATE_MEM(gc_lock_depth) > 0) { // TODO how to deal with this error? + GC_EXIT(); return; } @@ -471,18 +524,25 @@ void gc_free(void *ptr) { block += 1; } while (ATB_GET_KIND(block) == AT_TAIL); + GC_EXIT(); + #if EXTENSIVE_HEAP_PROFILING gc_dump_alloc_table(); #endif } else { + GC_EXIT(); assert(!"bad free"); } } else if (ptr != NULL) { + GC_EXIT(); assert(!"bad free"); + } else { + GC_EXIT(); } } size_t gc_nbytes(const void *ptr) { + GC_ENTER(); if (VERIFY_PTR(ptr)) { size_t block = BLOCK_FROM_PTR(ptr); if (ATB_GET_KIND(block) == AT_HEAD) { @@ -491,11 +551,13 @@ size_t gc_nbytes(const void *ptr) { do { n_blocks += 1; } while (ATB_GET_KIND(block + n_blocks) == AT_TAIL); + GC_EXIT(); return n_blocks * BYTES_PER_BLOCK; } } // invalid pointer + GC_EXIT(); return 0; } @@ -529,10 +591,6 @@ void *gc_realloc(void *ptr, mp_uint_t n_bytes) { #else // Alternative gc_realloc impl void *gc_realloc(void *ptr_in, size_t n_bytes, bool allow_move) { - if (MP_STATE_MEM(gc_lock_depth) > 0) { - return NULL; - } - // check for pure allocation if (ptr_in == NULL) { return gc_alloc(n_bytes, false); @@ -554,8 +612,16 @@ void *gc_realloc(void *ptr_in, size_t n_bytes, bool allow_move) { // get first block size_t block = BLOCK_FROM_PTR(ptr); + GC_ENTER(); + // sanity check the ptr is pointing to the head of a block if (ATB_GET_KIND(block) != AT_HEAD) { + GC_EXIT(); + return NULL; + } + + if (MP_STATE_MEM(gc_lock_depth) > 0) { + GC_EXIT(); return NULL; } @@ -590,6 +656,7 @@ void *gc_realloc(void *ptr_in, size_t n_bytes, bool allow_move) { // return original ptr if it already has the requested number of blocks if (new_blocks == n_blocks) { + GC_EXIT(); return ptr_in; } @@ -605,6 +672,8 @@ void *gc_realloc(void *ptr_in, size_t n_bytes, bool allow_move) { MP_STATE_MEM(gc_last_free_atb_index) = (block + new_blocks) / BLOCKS_PER_ATB; } + GC_EXIT(); + #if EXTENSIVE_HEAP_PROFILING gc_dump_alloc_table(); #endif @@ -620,6 +689,8 @@ void *gc_realloc(void *ptr_in, size_t n_bytes, bool allow_move) { ATB_FREE_TO_TAIL(bl); } + GC_EXIT(); + // zero out the additional bytes of the newly allocated blocks (see comment above in gc_alloc) memset((byte*)ptr_in + n_bytes, 0, new_blocks * BYTES_PER_BLOCK - n_bytes); @@ -630,19 +701,21 @@ void *gc_realloc(void *ptr_in, size_t n_bytes, bool allow_move) { return ptr_in; } + #if MICROPY_ENABLE_FINALISER + bool ftb_state = FTB_GET(block); + #else + bool ftb_state = false; + #endif + + GC_EXIT(); + if (!allow_move) { // not allowed to move memory block so return failure return NULL; } // can't resize inplace; try to find a new contiguous chain - void *ptr_out = gc_alloc(n_bytes, -#if MICROPY_ENABLE_FINALISER - FTB_GET(block) -#else - false -#endif - ); + void *ptr_out = gc_alloc(n_bytes, ftb_state); // check that the alloc succeeded if (ptr_out == NULL) { @@ -661,11 +734,12 @@ void gc_dump_info(void) { gc_info(&info); mp_printf(&mp_plat_print, "GC: total: %u, used: %u, free: %u\n", (uint)info.total, (uint)info.used, (uint)info.free); - mp_printf(&mp_plat_print, " No. of 1-blocks: %u, 2-blocks: %u, max blk sz: %u\n", - (uint)info.num_1block, (uint)info.num_2block, (uint)info.max_block); + mp_printf(&mp_plat_print, " No. of 1-blocks: %u, 2-blocks: %u, max blk sz: %u, max free sz: %u\n", + (uint)info.num_1block, (uint)info.num_2block, (uint)info.max_block, (uint)info.max_free); } void gc_dump_alloc_table(void) { + GC_ENTER(); static const size_t DUMP_BYTES_PER_LINE = 64; #if !EXTENSIVE_HEAP_PROFILING // When comparing heap output we don't want to print the starting @@ -713,7 +787,7 @@ void gc_dump_alloc_table(void) { } if (c == 'h') { ptrs = (void**)&c; - len = ((mp_uint_t)MP_STATE_VM(stack_top) - (mp_uint_t)&c) / sizeof(mp_uint_t); + len = ((mp_uint_t)MP_STATE_THREAD(stack_top) - (mp_uint_t)&c) / sizeof(mp_uint_t); for (mp_uint_t i = 0; i < len; i++) { mp_uint_t ptr = (mp_uint_t)ptrs[i]; if (VERIFY_PTR(ptr) && BLOCK_FROM_PTR(ptr) == bl) { @@ -771,6 +845,7 @@ void gc_dump_alloc_table(void) { mp_printf(&mp_plat_print, "%c", c); } mp_print_str(&mp_plat_print, "\n"); + GC_EXIT(); } #if DEBUG_PRINT |