nbdime – diffing and merging of Jupyter Notebooks¶

Installing nbdime¶

To install the latest stable release using pip:

pip install --upgrade nbdime

Installing latest development version¶

Installing a development version of nbdime requires Node.js.

Installing nbdime using pip will install the Python package dependencies and will automatically run npm to install the required Node.js packages.

python3 -m venv myenv          # For Python 2: python2 -m virtualenv myenv
source myenv/bin/activate
pip install nodeenv
nodeenv -p

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install nodeenv
Collecting nodeenv
  Downloading nodeenv-1.0.0.tar.gz
Installing collected packages: nodeenv
  Running setup.py install for nodeenv ... done
Successfully installed nodeenv-1.0.0
(myenv) $ nodeenv -p
 * Install prebuilt node (7.2.0) ..... done.
 * Appending data to /Users/username/myenv/bin/activate
(myenv) $

$ python2 -m pip install virtualenv
Collecting virtualenv
  Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 600kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0
$ python2 -m virtualenv myenv
New python executable in /Users/username/myenv/bin/python
Installing setuptools, pip, wheel...done.
$ source myenv/bin/activate
(myenv) $ pip install nodeenv
Collecting nodeenv
  Downloading nodeenv-1.0.0.tar.gz
Installing collected packages: nodeenv
  Running setup.py install for nodeenv ... done
Successfully installed nodeenv-1.0.0
(myenv) $ nodeenv -p
 * Install prebuilt node (7.2.0) ..... done.
 * Appending data to /Users/username/myenv/bin/activate
(myenv) $

pip install -e git+https://github.com/jupyter/nbdime

git clone https://github.com/jupyter/nbdime
cd nbdime
pip install -e .

nbshow¶

nbshow gives you a nice, terminal-optimized summary view of a notebook. You can use it to quickly peek at notebooks without launching the full notebook web application.

Diffing¶

nbdime offers two commands for viewing the diff between two notebooks:

• nbdiff for command-line diffing
• nbdiff-web for rich web-based diffing of notebooks

nbdiff¶

nbdiff does a terminal-optimized rendering of notebook diffs. Pass it the two notebooks you would like to compare, and it returns a nice, readable presentation of the changes in the notebook.

nbdiff-web¶

Like nbdiff, nbdiff-web compares two notebooks.

Instead of a terminal rendering, nbdiff-web opens a web browser, compares the two notebooks, and displays the rich rendered diff of images and other outputs.

Merging¶

Merging notebook changes and dealing with merge conflicts are important parts of a development workflow. With notebooks, merging changes is a non-trivial technical task. Traditional, line-based tools can produce invalid notebooks that you have to fix by hand, which is no fun at all, or can risk unintended data loss.

nbdime provides some improved tools for merging notebooks, taking into account knowledge of the notebook file format to ensure that a valid notebook is always produced. Further, by understanding details of the notebook format, nbdime can automatically resolve conflicts on generated fields.

nbmerge¶

nbmerge merges two notebooks with a common parent. If there are conflicts, they are stored in metadata of the destination file. nbmerge will exit with nonzero status if there are any unresolved conflicts.

nbmerge writes the output to stdout by default, so you can use pipes to send the result to a file, or the -o, --output argument to specify a file in which to save the merged notebook.

Because there are several categories of data in a notebook (such as input, output, and metadata), nbmerge has several ways to deal with conflicts, and can take different actions based on the type of data with the conflict.

Important

Conflict-resolution in nbmerge is under active development and is subject to change.

The -m, --merge-strategy option lets you select a global strategy to use. The following options are currently implemented:

inline

This is the default. Conflicts in input and output are recorded with conflict markers, while conflicts on metadata are stored in the appropriate metadata (actual values are kept as their base values).

This gives you a valid notebook that you can open in your usual notebook editor and resolve conflicts by hand, just like you might for a regular source file in your text editor.

use-base

When a conflict is encountered, use the value from the base notebook.

use-local

When a conflict is encountered, use the value from the local notebook.

use-remote

When a conflict is encountered, use the value from the remote notebook.

union

When a conflict is encountered, include both the local and the remote value, in that order (local then remote). Conflicts on non-sequence types (anything not list or string) are left unresolved.

Note

The union strategy might resolve to nonsensical values, while still marking conflicts as resolved, so use this carefully.

The --input-strategy and --output-strategy options lets you specify a strategy to use for conflicts on inputs and outputs, respecively. They accept the same values as the --merge-strategy option. If these are set, they will take precedence over --merge-strategy for inputs and/or outputs. --output-strategy takes two additional options: remove and clear-all:

remove

When a conflict is encountered on a single output, remove that output.

clear-all

When a conflict is encountered on any output in a given code cell, clear all outputs for that cell.

To use nbmerge, pass it three notebooks:

• base: the base, common parent notebook
• local: your local changes to base
• remote: other changes to base that you want to merge with yours

For example:

nbmerge base.ipynb local.ipynb remote.ipynb > merged.ipynb

nbmerge-web¶

nbmerge-web is just like nbmerge above, but instead of automatically resolving or failing on conflicts, a webapp for manually resolving conflicts is displayed:

nbmerge-web base.ipynb local.ipynb remote.ipynb -o merged.ipynb

Git integration¶

Git integration of nbdime is supported in two ways:

• through drivers for diff and merge operations, where nbdime takes on the responsibility for performing the diff/merge:

  • Diff driver
  • Merge driver

• through defining nbdime as diff and merge tools, which allow nbdime to display the diff/merge to the user without having to actually depend on git:

  • Diff web tool
  • Merge web tool

Configure git integration by editing the .gitconfig (or .git/config) and .gitattributes in each git repository or in the home directory for global effect. Read on for commands that edit these files and execute nbdime through git.

git-nbdiffdriver config --enable [--global]

git-nbmergedriver config --enable [--global]

git-nbdifftool config --enable [--global]

git difftool --tool=nbdime [<commit> [<commit>]] [--] [<path>…​]

git-nbdifftool config --enable [--global] --set-default

git difftool -g [<commit> [<commit>]] [--] [<path>…​]

git-nbmergetool config --enable [--global]

git mergetool --tool=nbdime [<file>…​]

git-nbmergetool config --enable --set-default [--global]

git mergetool [<file>…​]

Dependencies¶

Install the test dependencies:

pip install "nbdime[test]"

Running tests locally¶

To run python tests, locally, enter:

pytest

from the project root. If you have Python 2 and Python 3 installed, you may need to enter:

python3 -m pytest

to run the tests with Python 3. See the pytest documentation for more options.

To run javascript/typescript tests, enter:

npm test

from the nbdime-web folder.

Submitting test cases¶

If you have notebooks with interesting merge challenges, please consider contributing them to nbdime as test cases!

Basic diffing use cases¶

While developing basic correct diffing is fairly straightforward, there are still some issues to discuss.

Other tasks (issues will be created for these):

• Plugin framework for mime type specific diffing.
• Diffing of common output types (png, svg, etc.)
• Improve fundamental sequence diff algorithm. Current algorithm is based on a brute force O(N^2) longest common subsequence (LCS) algorithm. This will be rewritten in terms of a faster algorithm such as Myers O(ND) LCS based diff algorithm, optionally using Python’s difflib for some use cases where it makes sense.

Version control use cases¶

Most commonly, cell source is the primary content, and output can presumably be regenerated. Indeed, it is not possible to guarantee that merged sources and merged output is consistent or makes any kind of sense.

Some tasks:

• Merge of output cell content is not planned.
• Is it important to track source lines moving between cells?

Regression testing use cases¶

Basics¶

A diff object represents the difference B-A between two objects, A and B, as a list of operations (ops) to apply to A to obtain B. Each operation is represented as a dict with at least two items:

{ "op": <opname>, "key": <key> }

The objects A and B are either mappings (dicts) or sequences (lists or strings). A different set of ops are legal for mappings and sequences. Depending on the op, the operation dict usually contains an additional argument, as documented below.

The diff objects in nbdime are:

• json-compatible nested structures of dicts (with string keys) and
• lists of values with heterogeneous datatypes (strings, ints, floats).

The difference between these input objects is represented by a json-compatible results object. A JSON schema for validating diff entries is available in diff_format.schema.json.

Diff format for mappings¶

For mappings, the key is always a string.

Valid operations (ops) are:

• remove - delete existing value at key:

  { "op": "remove", "key": <string> }
  

• add - insert new value at key not previously existing:

  { "op": "add", "key": <string>, "value": <value> }
  

• replace - replace existing value at key with new value:

  { "op": "replace", "key": <string>, "value": <value> }
  

• patch - patch existing value at key with another diffobject:

  { "op": "patch", "key": <string>, "diff": <diffobject> }
  

Diff format for sequences¶

For sequences (list and string) the key is always an integer index. This index is relative to object A of length N.

Valid operations (ops) are:

• removerange - delete the values A[key:key+length]:

  { "op": "removerange", "key": <string>, "length": <n>}
  

• addrange - insert new items from valuelist before A[key], at end if key=len(A):

  { "op": "addrange", "key": <string>, "valuelist": <values> }
  

• patch - patch existing value at key with another diffobject:

  { "op": "patch",   "key": <string>, "diff": <diffobject> }
  

Relation to JSONPatch¶

The above described diff representation format has similarities with the JSONPatch standard but is also different in a few ways:

operations

• JSONPatch contains operations move, copy, test not used by nbdime.
• nbdime contains operations addrange, removerange, and patch not in JSONPatch.

patch

• JSONPatch uses a deep JSON pointer based path item in each operation instead of providing a recursive patch op.
• nbdime uses a key item in its patch op.

diff object

• JSONPatch can represent the diff object as a single list.
• nbdime uses a tree of lists.

To convert a nbdime diff object to the JSONPatch format, use the to_json_patch function:

from nbdime.diff_format import to_json_patch
jp = to_json_patch(diff_obj)

Examples¶

For examples of diffs using nbdime, see test_patch.py.

Merge Results¶

A merge operation with a shared origin object base and modified objects, local and remote, outputs these merge results:

• a fully or partially merged object
• a set of merge decision objects that describe the merge operation

Merge decision format¶

Each three-way notebook merge is based on the differences between the base version and the two changed versions – local and remote. These differences,``base`` with local and base with remote, are then compared, and for each change a set of decisions are made. A merge decision object represents such a decision, and is represented as a dict with the following entries:

{
    "local_diff": <diff object>,
    "remote_diff": <diff object>,
    "conflict": <boolean>,
    "action": <action taken/suggested>,
    "common_path": <JSON path>,
    "custom_diff": <diff object>
}

patch "cells"
┗━┓ patch index 0
  ┣━┓ patch "source"
  ┃ ┗━ addrange <some lines of source to add>
  ┗━┓ patch "outputs"
    ┗━ addrange <a new output added>

patch "cells"
┗━┓ patch index 0
  ┗━┓ patch "outputs"
    ┗━ removerange <all outputs removed>

Definitions¶

json_* always a JSON object

json_notebook a full Jupyter notebook

json_diff_args arguments to control nbdiff behaviour

json_merge_args arguments to control nbmerge behaviour

json_diff_object diff result in nbdime diff format

**json_merge_object merge result in nbdime merge format

/diff¶

Compute diff of two notebooks provided in full JSON format.

Request:

{
  "base":   json_notebook,
  "remote": json_notebook,
  "args": json_diff_args
}

Response:

{
  "diff": json_diff_object
}

/merge¶

Compute merge of three notebooks provided in full JSON format.

Request:

{
  "base":   json_notebook,
  "local":  json_notebook,
  "remote": json_notebook,
  "args": json_merge_args
}

Response:

{
  "merged": json_notebook,
  "localconflicts": json_diff_object,
  "remoteconflicts": json_diff_object,
}

/localdiff¶

Compute diff of notebooks known to the server by name.

Request:

{
  "base":   "filename.ipynb",
  "remote": "filename.ipynb",
  "args": json_diff_args
}

Response:

{
  "base": json_notebook,
  "diff": json_diff_object
}

/localmerge¶

Compute merge of notebooks known to the server by name.

Request:

{
  "base":   "filename.ipynb",
  "local":  "filename.ipynb",
  "remote": "filename.ipynb",
  "args": json_merge_args
}

Response:

{
  "merged": json_notebook,
  "localconflicts": json_diff_object,
  "remoteconflicts": json_diff_object,
}