@coolgk/mongo

A javascript / typescript MongoDB modelling library which enables joins in collections, simplifies CRUD operations for sub / nested documents and implements schema based data validation.

npm install @coolgk/mongo

Feature Hightlights
Documentation

Feature Hightlights

Join

SQL to @coolgk/mongo

Left Join

SELECT * FROM a LEFT JOIN b ON a.b_id = b.id

becomes

model.find({}, {
    join: [ { on: 'b_id' } ]
})

Result:

[{
    _id: '5a8bde4ae2ead929f89f3c42',
    a_name: 'aname1',
    b_id: {
        _id: '5a8bde4ae2ead929f89f3c41',
        b_name: 'bname1'
    }
}, { ... }, ... ]

Inner Join with Constraints

SELECT * FROM a, b WHERE a.b_id = b.id AND b.b_name = 'bname1'

becomes

model.find({}, {
    join: [ { on: 'b_id', filters: { b_name: 'bname1' } } ]
})

Result:

[{
    _id: '5a8bdfb05d44ea2a08fa8a4c',
    a_name: 'aname2',
    b_id: {
        _id: '5a8bdfb05d44ea2a08fa8a4b',
        b_name: 'bname2'
    }
}]

Inner Join on Mulitple Collections

SELECT * FROM a, b, c WHERE a.b_id = b.id AND b.c_id = c.id AND c.c_name = 'cname3'

modela.find({}, {
    join: [{
        on: 'b_id',
        join: [{
            on: 'c_id',
            filters: { c_name: 'cname3' }
        }]
    }]
})

Result:

[{
    _id: '5a8bdfc1b07af22a12cb1f0b',
    a_name: 'aname3',
    b_id: {
        _id: '5a8bdfc1b07af22a12cb1f0a',
        b_name: 'bname3',
        c_id: {
            _id: '5a8bdfc1b07af22a12cb1f09',
            c_name: 'cname3'
        }
    }
}]

Data Validation

Data Type Check
Required Field
Enum Values
String Regex Pattern
String Minimum and Maximum Length
Minimum and Maximum Values For Numbers
Minimum and Maximum Number of Array Items
Unique Array Values

Default Value, Setter Function, Last Modified Date

Schema:

{
    name: {
        type: DataType.STRING,
        default: 'abc' // default value
    },
    category: {
        type: DataType.STRING,
        setter: (value, document) => { // setter callback
            document.tags = [value];
            return `cat: ${value}`;
        }
    },
    tags: {
        type: DataType.STRING,
        array: true
    }
}

model.insertOne({ category: 'game' });

Result

{
    _id: '5a8c097fb14dc72b8401c773',
    category: 'cat: game', // setter applied
    name: 'abc', // default value
    tags: ['game'], // added by cateegory's setter
    _dateModified: '2018-02-20T11:43:45.612Z' // automatically added by insertOne
}

Sub Document CRUD

Updating documents in arrays could be annoying especially when there are 100+ or even 50+ of them in an array. This library makes it very easy to update documents in arrays.

Example Data

const data = {
    title: 'Support Ticket 1',
    messages: [
        {
            user: 'customer',
            message: 'I found a bug'
        },
        {
            user: 'support',
            message: `Restart your computer`
        },
        {
            user: 'developer',
            message: `That's not a bug, it's a feature`
        }
    ]
}

Data Preparation: `_id` and `_dateModified`

This library automatically adds _id and _dateMofidifed values to each document in the array, and uses the generated _id values for CRUD operations.

model.insertOne(data);

data becomes

{
    _id: '5a8c16f3c452fd2c0d3687c6',
    _dateModified: '2018-02-20T12:39:15.258Z', // auto generated
    title: 'Support Ticket 1',
    messages: [{
            _id: '5a8c16f3c452fd2c0d3687c9', // auto generated
            user: 'customer',
            message: 'I found a bug',
            _dateModified: '2018-02-20T12:39:15.259Z' // auto generated
        },
        {
            _id: '5a8c16f3c452fd2c0d3687c8', // auto generated
            user: 'support',
            message: 'Restart your computer',
            _dateModified: '2018-02-20T12:39:15.259Z' // auto generated
        },
        {
            _id: '5a8c16f3c452fd2c0d3687c7', // auto generated
            user: 'developer',
            message: 'That\'s not a bug, it\'s a feature',
            _dateModified: '2018-02-20T12:39:15.259Z' // auto generated
        }
    ]
}

Update A Sub Document

Similar to InsertOne() but with _id values in data. The script below will update the value of the "message" field of the seconnd document in the "messages" array.

model.updateOne({
    _id: '5a8c16f3c452fd2c0d3687c6', // find the document by _id in the collection
    messages: [
        {
            _id: '5a8c16f3c452fd2c0d3687c8', // find the document by _id in the array
            message: 'Turn on your computer' // update only the message field, other fields will not change
        }
    ]
});

data in the collection becomes

{
    _id: '5a8c16f3c452fd2c0d3687c6',
    title: 'Support Ticket 1',
    messages: [{
            _id: '5a8c16f3c452fd2c0d3687c9',
            user: 'customer',
            message: 'I found a bug',
            _dateModified: '2018-02-20T12:39:15.259Z',
        },
        {
            _id: '5a8c16f3c452fd2c0d3687c8',
            user: 'support',
            message: 'Turn on your computer', // new value
            _dateModified: '2018-02-20T12:53:55.890Z' // new modified date
        },
        {
            _id: '5a8c16f3c452fd2c0d3687c7',
            user: 'developer',
            message: 'That\'s not a bug, it\'s a feature',
            _dateModified: '2018-02-20T12:39:15.259Z'
        }
    ],
    _dateModified: '2018-02-20T12:53:55.889Z' // new modified date
}

Delete A Document

The script below will delete the seconnd document in the "messages" array.

model.updateOne({
    _id: '5a8c16f3c452fd2c0d3687c6', // find the document by _id in the collection
    messages: {
        $delete: [ '5a8c16f3c452fd2c0d3687c8' ] // $delete operator: delete by _id
    }
});

data in the collection becomes

{
    _id: '5a8c16f3c452fd2c0d3687c6',
    title: 'Support Ticket 1',
    messages: [{
            _id: '5a8c16f3c452fd2c0d3687c9',
            user: 'customer',
            message: 'I found a bug',
            _dateModified: '2018-02-20T12:39:15.259Z'
        },
        {
            _id: '5a8c16f3c452fd2c0d3687c7',
            user: 'developer',
            message: 'That\'s not a bug, it\'s a feature',
            _dateModified: '2018-02-20T12:39:15.259Z'
        }
    ],
    _dateModified: '2018-02-20T12:59:05.602Z' // new modified date
}

Replace or Delete All

model.updateOne({
    _id: '5a8c16f3c452fd2c0d3687c6', // find the document by _id in the collection
    messages: {
        $replace: [] // replace the entire array with a new value
    }
});

Add A Sub Document

Similar to Update, but without _id in sub documents. The script below will add a new document into the messages array.

model.updateOne({
    _id: '5a8c16f3c452fd2c0d3687c6', // find the document by _id in the collection
    messages: [
        { // documents without _id = insert new
            user: 'Support',
            message: 'Please Ctrl + F5'
        }
    ]
});

data in the collection becomes

{
    _id: '5a8c16f3c452fd2c0d3687c6',
    title: 'Support Ticket 1',
    messages: [{
            _id: '5a8c16f3c452fd2c0d3687c9',
            user: 'customer',
            message: 'I found a bug',
            _dateModified: '2018-02-20T12:39:15.259Z'
        },
        {
            _id: '5a8c16f3c452fd2c0d3687c7',
            user: 'developer',
            message: 'That\'s not a bug, it\'s a feature',
            _dateModified: '2018-02-20T12:39:15.259Z'
        },
        { // new document
            _id: '5a8c1d6b082a652c35eb17d6', // auto generated
            user: 'Support',
            message: 'Please Ctrl + F5',
            _dateModified: '2018-02-20T13:06:51.244Z' // auto generated
        }
    ],
    _dateModified: '2018-02-20T13:06:51.243Z' // new modified date
}

Multiple Operations

Add, Update and Delete can happen in one single query.

model.updateOne({
    _id: '5a8c16f3c452fd2c0d3687c6', // find the document by _id in the collection
    messages: {
        $update: [
            { // doc contains _id value, this is an update
                _id: '5a8c1d6b082a652c35eb17d6', // find the document by _id in the array
                message: 'Clear Your Cache' // update only the message field, other fields will not change
            },
            { // doc has no _id value, this is an insert
                user: 'developer',
                message: 'cannot replicate, not a bug!'
            }
        ],
        $delete: [ '5a8c16f3c452fd2c0d3687c9' ] // delete by _id (the first doc)
    }
});

Final Result

{
    _id: '5a8c16f3c452fd2c0d3687c6',
    title: 'Support Ticket 1',
    messages: [{
            _id: '5a8c16f3c452fd2c0d3687c7'，
            user: 'developer',
            message: 'That\'s not a bug, it\'s a feature',
            _dateModified: '2018-02-20T12:39:15.259Z'
        },
        {
            _id: '5a8c1d6b082a652c35eb17d6'，
            user: 'Support',
            message: 'Clear Your Cache', // new value
            _dateModified: '2018-02-20T13:30:07.123Z' // new modified date
        },
        { // new doc
            _id: '5a8c22df4656722c3fd787fa'，// auto generated
            user: 'developer',
            message: 'cannot replicate, not a bug!',
            _dateModified: '2018-02-20T13:30:07.123Z' // auto generated
        }
    ],
    _dateModified: '2018-02-20T13:30:07.121Z' // new modified date
}

Documentation

Basics

Model Class

The model class must extend the Mongo property of this library and implement the getCollectionName and getSchema static methods.

getCollectionName() - must return a collection name
getSchema() - must return the schema of the collection (see the Schema section below)

const { Mongo, DataType } = require('@coolgk/mongo');
// OR import { Mongo, DataType } from '@coolgk/mongo';

class ModelA extends Mongo {
    static getCollectionName () {
        return 'a';
    }
    static getSchema () {
        return {
            a_name: {
                type: DataType.STRING
            }
        }
    }
}

Class Instantiation

The model class must be initiated with a Db instance from mongo node client. e.g. the db variable in v3.x MongoClient.connect(url, (err, client) => { const db = client.db(dbName); … }) or in v2.x MongoClient.connect(url, (err, db) => { … })

const { MongoClient } = require('mongodb');

MongoClient.connect('mongodb://localhost', (error, client) => {
    const model = new ModelA({
        db: client.db('test')
    });
});

const { MongoClient } = require('mongodb');

(async () => {
    const globalDb = await new Promise((resolve) => {
        MongoClient.connect('mongodb://localhost', async (error, client) => {
            const db = client.db('test');
            resolve(db);
        });
    });

    // ...
    // ...
    // ...

    const modela = new ModelA({ db: globalDb });
    const modelb = new ModelB({ db: globalDb });
    ...
})()

Schema

Schema is defined in static getSchema() of the model class.

Schema Format

{
    [fieldName]: {
        type: '...',
        ... // other properties
    },
    ...
}

Shared Schema Properties: `type`, `array`, `default`, `setter`, `required`

These properties are valid for all data types.

`type` - Data type of the field. Supported types are in the `DataType` property of the library

const { DataType } = require('@coolgk/mongo');

DataType.STRING
DataType.NUMBER
DataType.ENUM
DataType.OBJECTID
DataType.DOCUMENT
DataType.BOOLEAN
DataType.DATE

`array` - a boolean value that defines if values are arrays

const document = {
    tags: ['game', 'shooter', 'sale']; // array of strings
};

const schema = {
    tags: {
        type: DataType.STRING, // string type
        array: true // array of string
    }
}

`default` - defines the default value of a field

const schema = {
    group: {
        type: DataType.STRING, // string type
        default: 'generic'
    }
}

`setter` - a callback function that transforms the value before insert and update

(value, document) => { return newValue; }

value - original value
document - all new values (to be saved) in the same document
return - a new value

const schema = {
    tags: {
        type: DataType.STRING, // string type
        setter: (value, document) => {
            return value + '-tag'
        }
    }
}

`required` - a boolean value to define if this field is a manditory field

const schema = {
    email: {
        type: DataType.STRING, // string type
        required: true
    }
}

`default`, `setter`, `array`, `required` Example

const schema = {
    name: {
        type: DataType.STRING,
        default: 'abc' // default value
    },
    category: {
        type: DataType.STRING,
        required: true, // manditory field
        setter: (value, document) => { // setter callback
            document.tags = [value];
            return `cat: ${value}`;
        }
    },
    tags: {
        type: DataType.STRING,
        array: true
    }
}

model.insertOne({ category: 'game' });

Result

{
    _id: '5a8c097fb14dc72b8401c773',
    category: 'cat: game', // setter applied
    name: 'abc', // default value
    tags: ['game'], // added by cateegory's setter
    _dateModified: '2018-02-20T11:43:45.612Z' // automatically added by insertOne
}

Shared Array Validation Properties: `maxItems`, `minItems`, `uniqueItems`

for fields that have array: true

maxItems - the maximum number of items in array
minItems - the minimum number of items in array
uniqueItems - a boolean value to define if each item in the array must be unique

{
    secureQuestionAnswers: {
        type: DataType.STRING,
        array: true,
        minItems: 1,
        maxItems: 3,
        uniqueItems: true
    }
}

Type Specific Properties

`DataType.ENUM`

enum - array of enum values

{
    logLevel: {
        type: DataType.ENUM,
        enum: [ 'notice', 'warn', 'error' ]
    }
}

`DataType.STRING`

minLength - minimum length of the string value
maxLength - maximum length of the string value
pattern - string containing a regex, the string value must match the regular expression

{
    email: {
        type: DataType.STRING,
        minLength: 10,
        maxLength: 200,
        pattern: '@\w+\.com$'
    }
}

`DataType.NUMBER`

minimum - minimum value of a number
maximum - minimum value of a number

{
    rating: {
        type: DataType.NUMBER,
        minimum: 0,
        maximum: 10
    }
}

`DataType.OBJECTID`

model - the model class that the object id references to. This is a required property for DataType.OBJECTID type and is required by the join option in find()

const { Mongo, DataType } = require('@coolgk/mongo');

class Category extends Mongo {
    static getCollectionName () {
        return 'Category';
    }
    static getSchema () {
        return {
            name: {
                type: DataType.STRING
            }
        }
    }
}

class Product extends Mongo {
    static getCollectionName () {
        return 'Product';
    }
    static getSchema () {
        return {
            name: {
                type: DataType.STRING
            },
            category: {
                type: DataType.OBJECTID,
                model: Category // category class, in real code, this could be require(../models/category.js)
            }
        }
    }
}

`DataType.DOCUMENT`

schema - the schema of the sub document which uses the same format as the main schema. This is a required property for DataType.DOCUMENT type

{
    address: {
        type: DataType.DOCUMENT,
        schema: { // schema of the address sub document
            street: {
                type: DataType.STRING
            },
            postcode: {
                type: DataType.STRING
            },
            country: {
                type: DataType.OBJECTID,
                model: Country
            }
        }
    }
}

Find & Join

Tested in MongoDB >= 3.x

An augmented version of mongo’s find() method

`find(query, options)`

Parameters

query - same as the query parameter in mongo’s find()
options - all options from mongo’s find() plus two extra properties join and cursor

options.join

array of join definitions

{
    join: [ // array of joins
        {
            on: ['name_of_an_object_id_field'],
            projection: { // optional
                [field_in_the_referenced_collection]: 1 or 0,
                ...
            },
            filters: { // optional
                // normal mongo query
            },
            join: { // optional
                on: ['name_of_an_object_id_field_from_the_referenced_collection'],
                ... // same format as the main join
            }
        },
        ...
    ]
}

join.on - array of object id fields that reference to a same collection. There can be multiple joins in one join array but when there are multiple fields reference to a same collection, these fields could be defined in the same block. For example, createdBy and modifiedBy fields both reference to the user collection, the on value would be ['createdBy', 'modifiedBy']. You can still put them in separate blocks if you need to filter them differently.
join.projection - same as the projection option in find(). Fields to select from the referenced collection, 1 = select, 0 = deselect.
join.filters - same as the query parameter in find() for filtering docs in the referenced collection
join.join - recursively join other collections

Example

// model a schema
{
    a_name: {
        type: DataType.STRING
    },
    b_id: {
        type: DataType.OBJECTID,
        model: B
    },
    c_id: {
        type: DataType.OBJECTID,
        model: C
    }
}
// model b schema
{
    b_name: {
        type: DataType.STRING
    },
    c_id: {
        type: DataType.OBJECTID,
        model: C
    }
}
// model c schema
{
    c_name: {
        type: DataType.STRING
    },
    c_group: {
        type: DataType.STRING
    }
}

modelA.find({}, {
    join: [
        {
            on: ['b_id'], // join a.b_id to b._id
            join: [{
                on: 'c_id', // join b.c_id to c._id
                filters: {
                    c_name: 'cname3'
                }
            }]
        },
        {
            on: 'c_id', // join a.c_id with c._id
            projection: {
                c_group: 1
            }
        }
    ]
});

Result:

[{
    _id: '5a8bdfc1b07af22a12cb1f0b',
    a_name: 'aname3',
    b_id: {
        _id: '5a8bdfc1b07af22a12cb1f0a',
        b_name: 'bname3',
        c_id: {
            _id: '5a8bdfc1b07af22a12cb1f09',
            c_name: 'cname3',
            c_group: 'group3'
        }
    },
    c_id: {
        _id: '5a8bdfc1b07af22a12cb1f09',
        c_group: 'group2'
    }
}]

The query above is similar to SQL:

SELECT
    A.*, B.*, CB.*, CA.c_group
FROM
    A
JOIN
    B ON A.b_id = B._id
JOIN
    C as CB ON B.c_id = CB._id
JOIN
    C as CA ON A.c_id = CA._id
WHERE
    CB.c_name = 'cname3'

options.cursor

Boolean. The default value is false. By default, the results are returned as an array. If cursor is true, the items in the cursor are promises instead documents.

const cursor = modelA.find({}, {
    join: {
        on: 'b_id'
    },
    cursor: true
});

cursor.forEach((documentPromise) => {
    documentPromise.then((document) => {
        ...
    });
});

// OR

cursor.forEach(async (documentPromise) => {
    const document = await documentPromise;
    ...
});

Insert

`insertOne(document, options)`

Parameters

document - same as the doc param in mongo’s insertOne()
options - same as the options param in mongo’s insertOne()
return - a promise, same as the return value of mongo’s insertOne()

`insertMany(document, options)`

Parameters

documents - same as the docs param of mongo’s insertMany()
options - same as the options param of mongo’s insertMany()
return - a promise, same as the return value of mongo’s insertMany()

`insertOne()` and `insertMany()` behaviours

add _dateModified (Constant: GeneratedField.DATE_MODIFIED) in the main doc and docs in arrays
add _id in docs in arrays
set default values defined in schema
apply setter functions defined in schema
convert valid numbers (!isNaN()) to numbers ‘123’ => 123
convert string 'false' or '0' to boolean false and cast other values to boolean (!!value) for DataType.BOOLEAN
convert DataType.OBJECTID strings and _id strings to ObjectID object if they are valid ObjectID
convert valid date string to Date object for DataType.DATE

Update

`updateOne(data, options)`

require MongoDB >= 3.6

data - document data with or without _id values in sub documents
options - all options in mongo’s findOneAndUpdate() plus a new revertOnError option
- options.revertOnError - see below
return - { value: ..., raw: ... }
- value - the updated document rather than the original if options.returnOriginal is false, otherwise the original document
- raw - raw outputs from mongo

Behaviour

Update a single document.
updateOne() updates data in three steps: set, add, delete.
This method is not atomic if more than one type of actions are executed e.g. set+add set+delete or set+add+delete etc.
updateOne() is atomic if only one type of actions is executed e.g. only adding new values

`options.revertOnError`

Boolean. default = false

Restore the document back to the original value before the update. If an error happens in one of the set, add, delete steps, the data is stored as at where the action stopped. e.g. if an error happens in the delete step, data set and added in the previous steps are stored in db. To stop this from happening, the “revertOnerror” option reverts the document back to the status before the updateOne() is executed. This action is NOT atomic. If a document is updated by a different source while updateOne() is still running, the “revertOnError” action will overwrite the changes made by the other source.

Manipulating Sub Documents With `UpdateOne()`

see Sub Document CRUD

Validation

require MongoDB >= 3.6

`setDbValidationSchema()`

Validations are done at database level. setDbValidationSchema() sets validation rules in db and only need to be called once per schema change. insertOne(), insertMany() and updateOne() will return a rejected promise if validation fails at db level. Validation errors are return from Mongo’s node client. This library does not control the contents of the validation error object.

Implementation

Add a “set validation” script to your deployment process.

e.g.

const db = mongoClientDB;
const modelFiles = findAllModelFiles();

(async () => {
    for (file of modelFiles) {
        const model = new (require(file))({ db });
        await model.setDbValidationSchema();
    }
})();

db.close();

Native Mongo Functions

`getDB()`

returns the Db object of the mongo client http://mongodb.github.io/node-mongodb-native/3.0/api/Db.html

// use native db methods
model.getDb().dropDatabase();

`getCollection()`

returns the Collection object of the mongo client http://mongodb.github.io/node-mongodb-native/3.0/api/Collection.html

// use native mongo collection methods
model.getCollection().aggregate( ... )

Utility Method

getObjectID(id)

id - a string or an ObjectID object
return - an ObjectID or undefined if id is not a valid ObjectID string

Error Types

MongoError and SchemaError

const { MongoError, SchemaError } = require('@coolgk/mongo');

Constants

Name of the auto-generated date modified field

GeneratedField.DATE_MODIFIED

Data Types:

DataType.STRING
DataType.NUMBER
DataType.ENUM
DataType.OBJECTID
DataType.DOCUMENT
DataType.BOOLEAN
DataType.DATE

const { GeneratedField, DataType } = require('@coolgk/mongo');

Also see

@coolgk/mvc

A simple, lightweight javascript / typescript MxC framework that helps you to create object oriented, modular and testable code.

@coolgk/utils

A javascript / typescript utility library. Modules: array, string, base64, ampq, bcrypt, cache, captcha, csv, email, jwt, number, pdf, tmp, token, unit conversion, url params, session, form data

Report bugs here: https://github.com/coolgk/node-mongo/issues

=============================== Coverage summary ===============================
Statements   : 96.22% ( 280/291 )
Branches     : 90.45% ( 199/220 )
Functions    : 95.56% ( 43/45 )
Lines        : 96.19% ( 278/289 )
================================================================================

coolgk.github.io

@coolgk/mongo

Feature Hightlights

Join

Left Join

Inner Join with Constraints

Inner Join on Mulitple Collections

Data Validation

Default Value, Setter Function, Last Modified Date

Sub Document CRUD

Data Preparation: _id and _dateModified

Update A Sub Document

Delete A Document

Replace or Delete All

Add A Sub Document

Multiple Operations

Documentation

Basics

Model Class

Class Instantiation

Schema

Schema Format

Shared Schema Properties: type, array, default, setter, required

type - Data type of the field. Supported types are in the DataType property of the library

array - a boolean value that defines if values are arrays

default - defines the default value of a field

setter - a callback function that transforms the value before insert and update

required - a boolean value to define if this field is a manditory field

default, setter, array, required Example

Shared Array Validation Properties: maxItems, minItems, uniqueItems

Type Specific Properties

DataType.ENUM

DataType.STRING

DataType.NUMBER

DataType.OBJECTID

DataType.DOCUMENT

Find & Join

find(query, options)

Insert

insertOne(document, options)

insertMany(document, options)

insertOne() and insertMany() behaviours

Update

updateOne(data, options)

Behaviour

options.revertOnError

Manipulating Sub Documents With UpdateOne()

Validation

setDbValidationSchema()

Implementation

Native Mongo Functions

getDB()

getCollection()

Utility Method

Error Types

Constants

Also see

@coolgk/mvc

@coolgk/utils

Data Preparation: `_id` and `_dateModified`

Shared Schema Properties: `type`, `array`, `default`, `setter`, `required`

`type` - Data type of the field. Supported types are in the `DataType` property of the library

`array` - a boolean value that defines if values are arrays

`default` - defines the default value of a field

`setter` - a callback function that transforms the value before insert and update

`required` - a boolean value to define if this field is a manditory field

`default`, `setter`, `array`, `required` Example

Shared Array Validation Properties: `maxItems`, `minItems`, `uniqueItems`

`DataType.ENUM`

`DataType.STRING`

`DataType.NUMBER`

`DataType.OBJECTID`

`DataType.DOCUMENT`

`find(query, options)`

`insertOne(document, options)`

`insertMany(document, options)`

`insertOne()` and `insertMany()` behaviours

`updateOne(data, options)`

`options.revertOnError`

Manipulating Sub Documents With `UpdateOne()`

`setDbValidationSchema()`

`getDB()`

`getCollection()`