Skip to content
Projects

Ditto

Declarative JSON-to-JSON mapper: shape-shift JSON files with Ditto

When dealing with data integration, translating JSON from external formats into an internal representation is a constant chore: every new data source arrives with its own field names and nesting. Ditto solves this declaratively. Instead of writing a bespoke transformation script per source, you write a mapping file that describes the output shape, and Ditto produces JSON from the input data to match it.

Status: Stable but dormant. The repo was last pushed in September 2024 and sees little activity today; the library is published on npm as json-ditto.

Ditto parses the mapping file (see mapping rules) and runs three stages (_preMap, _map, _postMap), where the output of each stage is fed as input to the next:

How to use Ditto

Ditto exposes a class that can be instantiated with a mapping file and an optional plugins list. Call unify with the document you want transformed:

const Ditto = require('json-ditto');
// This is a valid JSON mapping file that maps to the mapping rules belowconst myCustomMappings = require('./myMappingFile');
// Create a new mapper that will always use the "myCustomMappings" fileconst myCustomMapper = new Ditto(myCustomMappings);
// Call the unify function that will read the "documentToBeUnified" and transform it via the mapping filemyCustomMapper.unify(documentToBeUnified).then((result) => {    .....});

Alternatively, create a default instance and pass both the document and the mappings to unify itself (its signature is unify(document, mappings)):

const Ditto = require('json-ditto');
const myCustomMappings = require('./myMappingFile');
// Pass the document and the mapping file to unify directlynew Ditto().unify(documentToBeUnified, myCustomMappings).then((result) => {    .....});

Built-in plugins are available out of the box. Extra plugins can be registered either through the addPlugins(plugins) method or by passing them to the constructor directly via new Ditto(myCustomMappings, myCustomPlugins). Added plugins extend, and never overwrite, the default set.

Plugins: transformations in the mapping file

The heart of Ditto's expressiveness is plugins: plain JavaScript functions invoked from the mapping file with the @ symbol. A plugin is nothing more than a function that takes 0+ arguments:

'use strict';
const _ = require('lodash');
module.exports = function aggregateExperience(experience) {
    let totalExperience = 0;
    _.each(experience.values, function(entry){        if (!!entry.experienceDuration) totalExperience += entry.experienceDuration    })
    return totalExperience === 0 ? null : totalExperience;};

Arguments in the mapping file are separated by | and can be paths into the input document, hard-coded values, or defaults:

  • @concatName(firstName|lastName): call concatName with the values extracted from the firstName and lastName paths in the input file
  • @concatName(firstName|>>this_is_my_last_name): the >> prefix passes a hard-coded string instead of a path lookup
  • @concatName(firstName|lastName|*>>default): the * prefix marks a default value, used when the other lookups come up empty

If a plugin might receive a variable number of arguments, the JavaScript arguments object handles it cleanly:

function concatName() {	return _.flatten(_.values(arguments)).join(' ').replace(/\s\s+/g,' ').trim();}

Functions in functions

A plugin's value can be fed into another plugin as a parameter by nesting the call inside the arguments. One catch: parameters of inner function calls are separated by a comma , rather than the usual |:

"type"   : "@getLinkType(value|@getLinkService(value,service))","value"  : "@cleanURI(value|@getLinkType(value,@getLinkService(value,service)))",

Plugin activation

Plugins are activated in the /ditto/plugins/plugins.js file by adding an entry for the plugin (keyed by the .js filename of the definition) to the exported object; each one is required and exported for the main mapping function:

'use strict';
module.exports = {	aggregateExperience              : require('./aggregateExperience'),	assignIds                        : require('./assignIds'),	assignUniqueIds                  : require('./assignUniqueIds')	....

The full list of built-in plugins lives in the ditto/plugins folder, and the complete mapping-rule syntax is documented in the repo README.