Changelog

2021.09.1

Released on September 21, 2021

2021.09.0

Released on September 3, 2021

2021.08.1

Released on August 20, 2021

2021.08.0

Released on August 13, 2021

2021.07.2

Released on July 30, 2021

2021.07.1

Released on July 23, 2021

2021.07.0

Released on July 9, 2021

2021.06.2

Released on June 22, 2021

2021.06.1

Released on June 18, 2021

2021.06.0

Released on June 4, 2021

2021.05.1

Released on May 28, 2021

2021.05.0

Released on May 14, 2021

2021.04.1

Released on April 23, 2021

2021.04.0

Released on April 2, 2021

2021.03.1

Released on March 26, 2021

2021.03.0

Released on March 5, 2021

Note

This is the first release with support for Python 3.9 and the last release with support for Python 3.6

2021.02.0

Released on February 5, 2021

2021.01.1

Released on January 22, 2021

2021.01.0

Released on January 15, 2021

2020.12.0

Released on December 10, 2020

Highlights

  • Switched to CalVer for versioning scheme.

  • The scheduler can now receives Dask HighLevelGraph s instead of raw dictionary task graphs. This allows for a much more efficient communication of task graphs from the client to the scheduler.

  • Added support for using custom Layer-level annotations like priority, retries, etc. with the dask.annotations context manager.

  • Updated minimum supported version of Dask to 2020.12.0.

  • Added many type annotations and updates to allow for gradually Cythonizing the scheduler.

All changes

2.30.1 - 2020-11-03

2.30.0 - 2020-10-06

2.29.0 - 2020-10-02

2.28.0 - 2020-09-25

2.27.0 - 2020-09-18

2.26.0 - 2020-09-11

2.25.0 - 2020-08-28

2.24.0 - 2020-08-22

  • Move toolbar to above and fix y axis (#4043) Julia Signell

  • Make behavior clearer for how to get worker dashboard (#4047) Julia Signell

  • Worker dashboard clean up (#4046) Julia Signell

  • Add a default argument to the datasets and a possibility to override datasets (#4052) Nils Braun

  • Discover HTTP endpoints (#3744) Martin Durant

2.23.0 - 2020-08-14

2.22.0 - 2020-07-31

2.21.0 - 2020-07-17

2.20.0 - 2020-07-02

2.19.0 - 2020-06-19

2.18.0 - 2020-06-05

2.17.0 - 2020-05-26

2.16.0 - 2020-05-08

2.15.2 - 2020-05-01

2.15.1 - 2020-04-28

2.15.0 - 2020-04-24

2.14.0 - 2020-04-03

2.13.0 - 2020-03-25

2.12.0 - 2020-03-06

2.11.0 - 2020-02-19

2.10.0 - 2020-01-28

2.9.3 - 2020-01-17

2.9.2 - 2020-01-16

2.9.1 - 2019-12-27

2.9.0 - 2019-12-06

2.8.1 - 2019-11-22

2.8.0 - 2019-11-14

2.7.0 - 2019-11-08

This release drops support for Python 3.5

2.6.0 - 2019-10-15

2.5.2 - 2019-10-04

2.5.1 - 2019-09-27

2.5.0 - 2019-09-27

2.4.0 - 2019-09-13

2.3.2 - 2019-08-23

2.3.1 - 2019-08-22

2.3.0 - 2019-08-16

2.2.0 - 2019-07-31

2.1.0 - 2019-07-08

2.0.1 - 2019-06-26

We neglected to include python_requires= in our setup.py file, resulting in confusion for Python 2 users who erroneously get packages for 2.0.0. This is fixed in 2.0.1 and we have removed the 2.0.0 files from PyPI.

2.0.0 - 2019-06-25

1.28.1 - 2019-05-13

This is a small bugfix release due to a config change upstream.

1.28.0 - 2019-05-08

1.27.1 - 2019-04-29

1.27.0 - 2019-04-12

1.26.1 - 2019-03-29

1.26.0 - 2019-02-25

1.25.3 - 2019-01-31

1.25.2 - 2019-01-04

1.25.1 - 2018-12-15

1.25.0 - 2018-11-28

1.24.2 - 2018-11-15

1.24.1 - 2018-11-09

1.24.0 - 2018-10-26

1.23.3 - 2018-10-05

1.23.2 - 2018-09-17

1.23.1 - 2018-09-06

1.23.0 - 2018-08-30

1.22.1 - 2018-08-03

1.22.0 - 2018-06-14

1.21.8 - 2018-05-03

1.21.7 - 2018-05-02

1.21.6 - 2018-04-06

1.21.5 - 2018-03-31

1.21.4 - 2018-03-21

1.21.3 - 2018-03-08

1.21.2 - 2018-03-05

1.21.1 - 2018-02-22

1.21.0 - 2018-02-09

1.20.2 - 2017-12-07

1.20.1 - 2017-11-26

1.20.0 - 2017-11-17

1.19.3 - 2017-10-16

  • Handle None case in profile.identity (GH#1456)

  • Asyncio rewrite (GH#1458)

  • Add rejoin function partner to secede (GH#1462)

  • Nested compute (GH#1465)

  • Use LooseVersion when comparing Bokeh versions (GH#1470)

1.19.2 - 2017-10-06

  • as_completed doesn’t block on cancelled futures (GH#1436)

  • Notify waiting threads/coroutines on cancellation (GH#1438)

  • Set Future(inform=True) as default (GH#1437)

  • Rename Scheduler.transition_story to story (GH#1445)

  • Future uses default client by default (GH#1449)

  • Add keys= keyword to Client.call_stack (GH#1446)

  • Add get_current_task to worker (GH#1444)

  • Ensure that Client remains asynchornous before ioloop starts (GH#1452)

  • Remove “click for worker page” in bokeh plot (GH#1453)

  • Add Client.current() (GH#1450)

  • Clean handling of restart timeouts (GH#1442)

1.19.1 - September 25th, 2017

  • Fix tool issues with TaskStream plot (GH#1425)

  • Move profile module to top level (GH#1423)

1.19.0 - September 24th, 2017

  • Avoid storing messages in message log (GH#1361)

  • fileConfig does not disable existing loggers (GH#1380)

  • Offload upload_file disk I/O to separate thread (GH#1383)

  • Add missing SSLContext (GH#1385)

  • Collect worker thread information from sys._curent_frames (GH#1387)

  • Add nanny timeout (GH#1395)

  • Restart worker if memory use goes above 95% (GH#1397)

  • Track workers memory use with psutil (GH#1398)

  • Track scheduler delay times in workers (GH#1400)

  • Add time slider to profile plot (GH#1403)

  • Change memory-limit keyword to refer to maximum number of bytes (GH#1405)

  • Add cancel(force=) keyword (GH#1408)

1.18.2 - September 2nd, 2017

  • Silently pass on cancelled futures in as_completed (GH#1366)

  • Fix unicode keys error in Python 2 (GH#1370)

  • Support numeric worker names

  • Add dask-mpi executable (GH#1367)

1.18.1 - August 25th, 2017

  • Clean up forgotten keys in fire-and-forget workloads (GH#1250)

  • Handle missing extensions (GH#1263)

  • Allow recreate_exception on persisted collections (GH#1253)

  • Add asynchronous= keyword to blocking client methods (GH#1272)

  • Restrict to horizontal panning in bokeh plots (GH#1274)

  • Rename client.shutdown to client.close (GH#1275)

  • Avoid blocking on event loop (GH#1270)

  • Avoid cloudpickle errors for Client.get_versions (GH#1279)

  • Yield on Tornado IOStream.write futures (GH#1289)

  • Assume async behavior if inside a sync statement (GH#1284)

  • Avoid error messages on closing (GH#1297), (GH#1296) (GH#1318) (GH#1319)

  • Add timeout= keyword to get_client (GH#1290)

  • Respect timeouts when restarting (GH#1304)

  • Clean file descriptor and memory leaks in tests (GH#1317)

  • Deprecate Executor (GH#1302)

  • Add timeout to ThreadPoolExecutor.shutdown (GH#1330)

  • Clean up AsyncProcess handling (GH#1324)

  • Allow unicode keys in Python 2 scheduler (GH#1328)

  • Avoid leaking stolen data (GH#1326)

  • Improve error handling on failed nanny starts (GH#1337), (GH#1331)

  • Make Adaptive more flexible

  • Support --contact-address and --listen-address in worker (GH#1278)

  • Remove old dworker, dscheduler executables (GH#1355)

  • Exit workers if nanny process fails (GH#1345)

  • Auto pep8 and flake (GH#1353)

1.18.0 - July 8th, 2017

1.17.1 - June 14th, 2017

  • Remove Python 3.4 testing from travis-ci (GH#1157)

  • Remove ZMQ Support (GH#1160)

  • Fix memoryview nbytes issue in Python 2.7 (GH#1165)

  • Re-enable counters (GH#1168)

  • Improve scheduler.restart (GH#1175)

1.17.0 - June 9th, 2017

  • Reevaluate worker occupancy periodically during scheduler downtime (GH#1038) (GH#1101)

  • Add AioClient asyncio-compatible client API (GH#1029) (GH#1092) (GH#1099)

  • Update Keras serializer (GH#1067)

  • Support TLS/SSL connections for security (GH#866) (GH#1034)

  • Always create new worker directory when passed --local-directory (GH#1079)

  • Support pre-scattering data when using joblib frontent (GH#1022)

  • Make workers more robust to failure of sizeof function (GH#1108) and writing to disk (GH#1096)

  • Add is_empty and update methods to as_completed (GH#1113)

  • Remove _get coroutine and replace with get(..., sync=False) (GH#1109)

  • Improve API compatibility with async/await syntax (GH#1115) (GH#1124)

  • Add distributed Queues (GH#1117) and shared Variables (GH#1128) to enable inter-client coordination

  • Support direct client-to-worker scattering and gathering (GH#1130) as well as performance enhancements when scattering data

  • Style improvements for bokeh web dashboards (GH#1126) (GH#1141) as well as a removal of the external bokeh process

  • HTML reprs for Future and Client objects (GH#1136)

  • Support nested collections in client.compute (GH#1144)

  • Use normal client API in asynchronous mode (GH#1152)

  • Remove old distributed.collections submodule (GH#1153)

1.16.3 - May 5th, 2017

  • Add bokeh template files to MANIFEST (GH#1063)

  • Don’t set worker_client.get as default get (GH#1061)

  • Clean up logging on Client().shutdown() (GH#1055)

1.16.2 - May 3rd, 2017

  • Support async with Client syntax (GH#1053)

  • Use internal bokeh server for default diagnostics server (GH#1047)

  • Improve styling of bokeh plots when empty (GH#1046) (GH#1037)

  • Support efficient serialization for sparse arrays (GH#1040)

  • Prioritize newly arrived work in worker (GH#1035)

  • Prescatter data with joblib backend (GH#1022)

  • Make client.restart more robust to worker failure (GH#1018)

  • Support preloading a module or script in dask-worker or dask-scheduler processes (GH#1016)

  • Specify network interface in command line interface (GH#1007)

  • Client.scatter supports a single element (GH#1003)

  • Use blosc compression on all memoryviews passing through comms (GH#998)

  • Add concurrent.futures-compatible Executor (GH#997)

  • Add as_completed.batches method and return results (GH#994) (GH#971)

  • Allow worker_clients to optionally stay within the thread pool (GH#993)

  • Add bytes-stored and tasks-processing diagnostic histograms (GH#990)

  • Run supports non-msgpack-serializable results (GH#965)

1.16.1 - March 22nd, 2017

  • Use inproc transport in LocalCluster (GH#919)

  • Add structured and queryable cluster event logs (GH#922)

  • Use connection pool for inter-worker communication (GH#935)

  • Robustly shut down spawned worker processes at shutdown (GH#928)

  • Worker death timeout (GH#940)

  • More visual reporting of exceptions in progressbar (GH#941)

  • Render disk and serialization events to task stream visual (GH#943)

  • Support async for / await protocol (GH#952)

  • Ensure random generators are re-seeded in worker processes (GH#953)

  • Upload sourcecode as zip module (GH#886)

  • Replay remote exceptions in local process (GH#894)

1.16.0 - February 24th, 2017

  • First come first served priorities on client submissions (GH#840)

  • Can specify Bokeh internal ports (GH#850)

  • Allow stolen tasks to return from either worker (GH#853), (GH#875)

  • Add worker resource constraints during execution (GH#857)

  • Send small data through Channels (GH#858)

  • Better estimates for SciPy sparse matrix memory costs (GH#863)

  • Avoid stealing long running tasks (GH#873)

  • Maintain fortran ordering of NumPy arrays (GH#876)

  • Add --scheduler-file keyword to dask-scheduler (GH#877)

  • Add serializer for Keras models (GH#878)

  • Support uploading modules from zip files (GH#886)

  • Improve titles of Bokeh dashboards (GH#895)

1.15.2 - January 27th, 2017

  • Fix a bug where arrays with large dtypes or shapes were being improperly compressed (GH#830 GH#832 GH#833)

  • Extend as_completed to accept new futures during iteration (GH#829)

  • Add --nohost keyword to dask-ssh startup utility (GH#827)

  • Support scheduler shutdown of remote workers, useful for adaptive clusters (:pr: 811 GH#816 GH#821)

  • Add Client.run_on_scheduler method for running debug functions on the scheduler (GH#808)

1.15.1 - January 11th, 2017

  • Make compatibile with Bokeh 0.12.4 (GH#803)

  • Avoid compressing arrays if not helpful (GH#777)

  • Optimize inter-worker data transfer (GH#770) (GH#790)

  • Add –local-directory keyword to worker (GH#788)

  • Enable workers to arrive to the cluster with their own data. Useful if a worker leaves and comes back (GH#785)

  • Resolve thread safety bug when using local_client (GH#802)

  • Resolve scheduling issues in worker (GH#804)

1.15.0 - January 2nd, 2017

  • Major Worker refactor (GH#704)

  • Major Scheduler refactor (GH#717) (GH#722) (GH#724) (GH#742) (GH#743

  • Add check (default is False) option to Client.get_versions to raise if the versions don’t match on client, scheduler & workers (GH#664)

  • Future.add_done_callback executes in separate thread (GH#656)

  • Clean up numpy serialization (GH#670)

  • Support serialization of Tornado v4.5 coroutines (GH#673)

  • Use CPickle instead of Pickle in Python 2 (GH#684)

  • Use Forkserver rather than Fork on Unix in Python 3 (GH#687)

  • Support abstract resources for per-task constraints (GH#694) (GH#720) (GH#737)

  • Add TCP timeouts (GH#697)

  • Add embedded Bokeh server to workers (GH#709) (GH#713) (GH#738)

  • Add embedded Bokeh server to scheduler (GH#724) (GH#736) (GH#738)

  • Add more precise timers for Windows (GH#713)

  • Add Versioneer (GH#715)

  • Support inter-client channels (GH#729) (GH#749)

  • Scheduler Performance improvements (GH#740) (GH#760)

  • Improve load balancing and work stealing (GH#747) (GH#754) (GH#757)

  • Run Tornado coroutines on workers

  • Avoid slow sizeof call on Pandas dataframes (GH#758)

1.14.3 - November 13th, 2016

  • Remove custom Bokeh export tool that implicitly relied on nodejs (GH#655)

  • Clean up scheduler logging (GH#657)

1.14.2 - November 11th, 2016

  • Support more numpy dtypes in custom serialization, (GH#627), (GH#630), (GH#636)

  • Update Bokeh plots (GH#628)

  • Improve spill to disk heuristics (GH#633)

  • Add Export tool to Task Stream plot

  • Reverse frame order in loads for very many frames (GH#651)

  • Add timeout when waiting on write (GH#653)

1.14.0 - November 3rd, 2016

  • Add Client.get_versions() function to return software and package information from the scheduler, workers, and client (GH#595)

  • Improved windows support (GH#577) (GH#590) (GH#583) (GH#597)

  • Clean up rpc objects explicitly (GH#584)

  • Normalize collections against known futures (GH#587)

  • Add key= keyword to map to specify keynames (GH#589)

  • Custom data serialization (GH#606)

  • Refactor the web interface (GH#608) (GH#615) (GH#621)

  • Allow user-supplied Executor in Worker (GH#609)

  • Pass Worker kwargs through LocalCluster

1.13.3 - October 15th, 2016

  • Schedulers can retire workers cleanly

  • Add Future.add_done_callback for concurrent.futures compatibility

  • Update web interface to be consistent with Bokeh 0.12.3

  • Close streams explicitly, avoiding race conditions and supporting more robust restarts on Windows.

  • Improved shuffled performance for dask.dataframe

  • Add adaptive allocation cluster manager

  • Reduce administrative overhead when dealing with many workers

  • dask-ssh --log-directory . no longer errors

  • Microperformance tuning for the scheduler

1.13.2

  • Revert dask_worker to use fork rather than subprocess by default

  • Scatter retains type information

  • Bokeh always uses subprocess rather than spawn

1.13.1

  • Fix critical Windows error with dask_worker executable

1.13.0

  • Rename Executor to Client (GH#492)

  • Add --memory-limit option to dask-worker, enabling spill-to-disk behavior when running out of memory (GH#485)

  • Add --pid-file option to dask-worker and --dask-scheduler (GH#496)

  • Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (GH#494)

  • Add workers= keyword argument to Client.compute and Client.persist, supporting location-restricted workloads with Dask collections (GH#484)

  • Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (GH#494)

    • Add optional dask_worker= keyword to client.run functions that gets provided the worker or nanny object

    • Add nanny=False keyword to Client.run, allowing for the execution of arbitrary functions on the nannies as well as normal workers

1.12.2

This release adds some new features and removes dead code

  • Publish and share datasets on the scheduler between many clients (GH#453). See Publish Datasets.

  • Launch tasks from other tasks (experimental) (GH#471). See Launch Tasks from Tasks.

  • Remove unused code, notably the Center object and older client functions (GH#478)

  • Executor() and LocalCluster() is now robust to Bokeh’s absence (GH#481)

  • Removed s3fs and boto3 from requirements. These have moved to Dask.

1.12.1

This release is largely a bugfix release, recovering from the previous large refactor.

  • Fixes from previous refactor
    • Ensure idempotence across clients

    • Stress test losing scattered data permanently

  • IPython fixes
    • Add start_ipython_scheduler method to Executor

    • Add %remote magic for workers

    • Clean up code and tests

  • Pool connects to maintain reuse and reduce number of open file handles

  • Re-implement work stealing algorithm

  • Support cancellation of tuple keys, such as occur in dask.arrays

  • Start synchronizing against worker data that may be superfluous

  • Improve bokeh plots styling
    • Add memory plot tracking number of bytes

    • Make the progress bars more compact and align colors

    • Add workers/ page with workers table, stacks/processing plot, and memory

  • Add this release notes document

1.12.0

This release was largely a refactoring release. Internals were changed significantly without many new features.

  • Major refactor of the scheduler to use transitions system

  • Tweak protocol to traverse down complex messages in search of large bytestrings

  • Add dask-submit and dask-remote

  • Refactor HDFS writing to align with changes in the dask library

  • Executor reconnects to scheduler on broken connection or failed scheduler

  • Support sklearn.external.joblib as well as normal joblib