Wednesday, January 17, 2018

Darwino makes OpenNTF mobile!



[Latest News] The iOS app works perfectly, but we are struggling with the great fruity company to get it accepted to the store, because they claim there is not enough mobile content on OpenNTF... We'll do our best with the OpenNTF team to get more content leveraging mobile capabilities.

Darwino 2.0 features a mobile micro application architecture well integrated with the IBM Domino server. It can render any existing Domino web application (classic & XPages) on a mobile device while taking care of the painful details:
  • Real mobile applications available from both Google Play and Apple Store
  • Authentication management, to avoid prompting the user when the server requires credentials
  • Opening external pages in a separate browser window to provide the best user experience
  • Mobile notifications, to alert users when something new arrived and let him/her act appropriately
  • Plus many implementation details, like error management and recovery
To quickly experiment with the capability, we are providing a generic mobile client that one can use to connect to any Darwino enabled Domino server. This client is available on both application stores and is fully configurable to point to any server. As a default configuration, we make it point to OpenNTF!

As most of you know, the main OpenNTF applications, like Collaboration Today, are XPages based with a responsive design. This make them very easy to integrate with the micro application architecture. Moreover, Darwino extends Domino Designer with a new design element to detect database changes and send mobile notifications. With that in place on the OpenNTF servers, we are able to send new notifications when a new entry is available in Collaboration Today. By the way, these notifications use the mobile first IBM Cloud service (a.k.a. Bluemix).

The implementation on the OpenNTF server was straight forward:
  • Get the Darwino OSGi runtime installed
  • Set some configuration properties
  • Create a new design element for mobile notifications. To leave the existing databases untouched, we did that in brand new, separate database
Voila, in less than an hour it can be all setup. The same can happen to your own applications!

What are the next steps then? Well, a few things:
  • We need to work with the OpenNTF team to mobile enable other parts of the site. Ideally, the whole site should be mobile enabled!
  • Why not white labeling the generic client with a pure OpenNTF branding (icon, title, ...)? Darwino offers this options.
  • Move forward and create Darwino single page applications, using web or mobile native technologies, to get the data offline and provide the best user experience without connectivity!
Stay tuned, OpenNTF becomes mobile!


Tuesday, January 9, 2018

Dynamic document based security

Domino provides document based security through the use of readers and authors fields. This is a great capability that is fairly unique in the NoSQL industry heavily used by Domino developer. One caveat though: you have to explicitly manage the list of users/group/roles in each document, generally based on the document state or content. How many times did you write an agent (background task) to update these fields nightly?

Since the beginning, Darwino supports readers and authors (as well as excluded readers and authors to remove specific users from the authorized list). It is implemented by a hook in the SQL generator that systematically adds a condition to the queries. Here is how it looks like in pseudo SQL:
SELECT D.JSON FROM DOC D WHERE (....) AND (D.READERS IN (:userid, :userroles, :usergroups...)

userid, userroles and usergroups are values related to the user making the request.
All of this is implemented (and by the way optimized) at the core Darwino level. So it applies regardless of API you use: Java, JavaScript, REST services, GraphQL...

But in 2.1, it goes beyond this simple capability: it allows an application to insert its own SQL filter:
SELECT D.JSON FROM DOC D WHERE (....) AND (<custom condition>)

For example, the custom condition can be based on fields within the document. Let's suppose that the current user object has an attribute 'country' set in LDAP. Let's also suppose that the documents in a JSON store also have a field name 'country'. From a security standpoint, we only want to allow the users to see the documents tagged with their country. With classic readers fields, you'll need an agent that periodically updates the document readers, or use groups and assign users to groups. In both cases, there is some maintenance to do.

With Darwino, it becomes as easy as implementing a runtime extension that returns a condition for the current user (again, simplified pseudo code):

String getDynamicSecurity() {
    User u = getCurrentUser();
    return "{country: '"+u.getCountryAttribute()+"'}";

}

There is no need to maintain a list of readers here. If a user is moved to a different country, then the query will use the latest user country (or any other needed value), immediately!

The dynamic security condition can be expressed using the built-in, MongoDB like, query language as above. This platform independent query is transpiled to SQL by the core Darwino code.
But it can also be expressed as raw SQL, allowing the filter to be as complex as desired. For example, it can lookup in another SQL table, view or stored procedure for a list of authorized ids for the current user:
... AND (D.DOCID IN SELECT ID FROM SECURITY SEC WHERE SEC.USERID='currentuserid')
This is limitless. Let's suppose that you have a users database where each user document has the name of its manager. Now suppose that user documents can only be accessed by the user and his managers. Well, just get the list of authorized IDs from a recursive SELECT statement
... AND (D.USERID IN WITH RECURSIVE SELECT ...)
If the subquery becomes too complex and leads to performance issues, you can create materialized views that will be managed by the database server.

Come on MongoDB, you cannot compete against the power of relational databases :-)




Thursday, January 4, 2018

Offline your data with React-Native applications


In a previous post, I mentioned how we are leveraging ReactJS to create state-of-the-art web applications. We are now moving a step forward by also leveraging its sister technology, React-Native, to create portable, yet native, mobile applications. With all the benefits of Darwino, which includes a local JSON store that replicates for the best offline experience, and all the other services provided by Darwino!

To make this as easy as possible, we are providing a simple service based API, that can be called from any native language (Java, Objective-C, Swift).

1- Darwino service API

The API is basically made of one class, exposed bellow in pseudo code:

class DarwinoApis {
  ...
  Object fetch(url, params, headers, content)
  ...
}

The fetch method calls the service identified by the URL and associated parameters, and returns the result. Behind the hood, such the API call will be either translated to a remote HTTP service call (online mode), or to a local API call (offline mode). This is done transparently by a dispatcher, so the developer does not have to worry about the current mode. It just works! This API can be called from *any* native application, regardless of the library/framework it uses.

This shows the beauty of the Darwino architecture: the services are all written independently from the HTTP stack, and thus can be executed locally without the need of HTTP calls. It makes them truly available offline, while providing the best possible performance as it also avoids the data serialization/deserialization needed by the HTTP transport.

Here is a basic architecture picture:



2- ReactNative integration

As the API is exposed using the platform native languages, it is easy to bridge it with ReactNative. Moreover, the bridge enforces the use of promises, thus enabling the new aync & await capability for the best developer experience.

Here is, for example, how to retrieve the number of documents from a database, regardless if it is local or remote:

async databaseCount() {     let url = ["$darwino-jstore","databases",this.getDatabase(),"documentscount"]     let count = await darwinoApis.fetch(url);     this.println("This database contains {0} documents",count);
}
Easy, isn't it?

Triggering the database replication isn't more difficult, see:

let result = await apis.replicate(this.getDatabase(),options); this.println("{0}",result);

3- Darwino API distribution

The APIs is provided as a library to be added to any existing application. On Android, it is a maven artifact that you can reference from your gradle build file. On iOS, it is a Cocoa framework that you can add to your XCode project. Just add these dependencies and you're ready to go!

Android:

iOS:

4- Under the hood

Wait, Darwino is Java based, so how can we provide a native Objective-C or Swift API? How can we get all the Darwino services, built-in and customs, run natively on any device? As the goal is to have one single Java source code to ensure the consistency between all the platforms, so how is is exposed as an Objective-C API?

The magic comes from J2OBJC. Right from the beginning, we have been very careful to write "portable" Java, meaning Java that can be compiled or transpiled to many different environments (JRE, Dalvik/Art, RoboVM, Multi-OS Engine...). As J2OBJC made a lot of progress these past years, it became easy to get our code transpiled to J2OBJC. The result is a tiny, efficient, native Objective-C library while sharing the same Java source code.
With the potential to target other platforms in the future, like .NET: what about having your Domino data replicated natively to .NET? :-)

Tuesday, January 2, 2018

Darwino 2.0 is out!

I'm very exited to announce that the official Darwino 2.0 has just been released!



It had been a long journey since the previous release but most of our customers are already using the pre 2.0 release, aka 2.0.0-SNAPSHOT in maven terms. This means that this release has already been tested in many contexts and is thus very stable.

The main topics for this major release are:

  • Creation of new web and mobile applications
    - Provide the best user experience even when the device is offline
    - Leverage of the latest technologies, including ReactJS
  • Modernization of existing applications, in particular Notes & Domino
    - Liberate the Domino data and take advantage of any data reporting/analytics tool
    - Get your existing Notes/Domino apps on mobile device with notifications
    - Create modern mobile applications connected to your IBM Domino data
  • Migration of IBM Notes/Domino applications
    - Incremental migration to minimize the risk
    - Use of modern technologies and development tools
    - Integrate with MS Sharepoint, Office 365 or other platforms, like Oracle Cloud
  • Many additions and enhancements to the existing APIs

I bet that the mobile client is going to be popular in our community because... Well, read my next post :-)

Bellow is a list of the most noticeable features in 2.0:

Darwino for mobile devices


Darwino mobile client
Darwino provides a generic client that renders server side web applications and handles IBM Cloud notifications. This client is available for both iOS and Android, in their respective application stores.
This client can also be fully customized as the full source code is available. For example, one can white label the application with a custom icon, look&feel, options...

Multi-OS-Engine instead of RoboVM
Darwino switched to MOE as RoboVM is no longer available since it has been acquired by Microsoft. Darwino 2.0 supports the latest version of MOE. Thanks Migeran for doing this great product!

Integration


Notification framework
Darwino provides a notification framework that executes actions based on events. The framework can detect data changes in a database, including domino databases, and run actions like a mobile push notifications, notifying IBM Workspace, sending an email or whatever a piece of code can do...

Microsoft Azure active directory support
Darwino features a driver to consume Microsoft Azure active directory services

Microsoft Sharepoint integration
Darwino integration with Sharepoint, including directory access and single sign-on.

More database support
The MS SQL server support is complete, as well as MySQL (the full JSQL requires MySQL 8+).

Darwino DB


JSQL
JSQL is an SQL dialect dedicated to JSON databases. It allows the full power of SQL queries on top of JSON documents: joins, unions, groups, recursive queries, ... while honoring the document level security. It eventually translates to the native database SQL for maximum performance.
https://playground.darwino.com/playground.nsf/JsqlSnippets.xsp

Registered queries
Database queries can be defined on the server side and consumed by name from the client. This is also true for JSQL queries.
https://playground.darwino.com/playground.nsf/JavaSnippets.xsp#snippet=Json_Store_Cursor_Run_Predefined_Query

Domino @formula
An extension to the query language uses the new keyword $atFormula and allows the use of a subset of the Notes/Domino @formula language in Darwino queries. That makes the migration of Notes/Domino views a lot easier!
https://playground.darwino.com/playground.nsf/DarwinoDbSnippets.xsp#snippet=@Formula_Simple


REST services


JSON store services
Added some new services to retrieve document content without the meta-data.
https://playground.darwino.com/playground.nsf/OpenApiExplorer.xsp#openApi=Json_Store_Document_CRUD

GraphQL

Darwino comes with a innovative implementation of FaceBook's GraphQL, allowing both schema based and schema less queries. Moreover, GraphQL queries can be pre-defined on the server side to make them easier to consume
https://playground.darwino.com/playground.nsf/GraphqlSnippets.xsp

Microservices
Darwino provides a easy to use JSON microservice infrastructure. The services can be written in Java, Groovy, JavaScript, Darwino Script or any language supported by the JVM.
https://playground.darwino.com/playground.nsf/ServicesSnippets.xsp


Darwino Studio


Application Wizard
The application creation wizard can now use templates, also known as boilerplate, to get the developer started quickly. A new template using ReactJS and Bootstrap is provided as part of the studio. The architecture is open and any developer can contribute new templates.

Android studio
The wizard can also leverage Android studio for the mobile applications, instead of Eclipse. This allows the developers to benefit from the latest features provided by Google and Multi-OS-Engine.

UI Builder
Darwino studio features an easy form builder that targets mobile devices. This builder is fully extensible to support more targets in the future.


Darwino for IBM Notes/Domino


Domino Designer
Darwino provides a new Domino Designer plugin that adds a new design element dealing with the notifications. It uses a groovy based DSL (Domain Specific Language) .

Darwino client
The Darwino mobile client detects if the server is an IBM Domino server and adapts some behaviors accordingly, like error handling.

Darwino application
Darwino provides new runtime capabilities to supplement IBM Domino missing features, like JEE filters. Moreover, Darwino applications can be auto-started, for example to monitor events.

IBM Domino import
The Darwino studio can generate a basic UI based on existing IBM Domino NSFs. This feature is a preview, but this is something we'd like to enhance moving forward, to get the mobilization or migration of Notes/Domino import easier.
It also generates JSON Schemas based on the Form definition in an NSF.


Developer tools


Commands framework
Darwino features a command line framework that developer can add to any application to execute administration commands or debug an application.
It also includes a built-in application profiler.

Playground
The Playground now show cases the new features like JSQL or GraphQL, but also exposes the REST apis using OpenAPI/Swagger: https://playground.darwino.com/playground.nsf/OpenApiExplorer.xsp

Friday, December 15, 2017

Ubiquity of Java and relational databases


I'm sometimes being asked about our platform choices of Java and RDBMS for Darwino. More precisely, I'm getting questions about Node.js or existing JSON document databases, like MongoDB or CouchDB.
Well, I can certainly expand on the technical value of our choices, and I'll probably will. But, if the technical aspect is important, this is not the only driver.  It is not even the main one: what made us choose these technologies can be summarized in 3 words: ubiquity, diversity and skill and assets.

Ubiquity

Most of our customers have a Java application server and a relational database already in place. Both are almost everywhere. If small companies can be flexible with technology, bigger companies tend to be more conservative. And not to talk about administrations. With Java and RDBMS you can get in right away without going through technology validation processes.

The cloud is not different: whenever you want to use MS Azure, IBM Bluemix, Oracle Cloud, Amazon AWS, they all offer you PAAS services with a Java application server and an RDBMS. As a result, your not locked with a vendor. For example, and despite its value, if you choose IBM Cloudant, you're locked with IBM Cloud. For the good and the bad... Did I say the bad?

It similarly expands to mobile devices. There is no MongoDB implementation running on these devices, providing offline capabilities with the same set of features. Ok, Cloudant/CouchDB has TouchDB or PouchDB for offline, but these are totally different implementations with different APIs and capabilities. On the other hand, SQLite can be run on any mobile devices from Java.

With Darwino encapsulating the platform differences, like SQL dialects, you write the application once and you can run everywhere, including on mobile devices.

Diversity

There is not a single implementation of Java and RDBMS. Rather, you can find several vendors providing their own JVM, web application servers or RDBMS, with different characteristics. You pick and choose the one that better fits your need, based on both technical or business decisions.

Diversity fosters innovation because of the competition: the vendors have to innovate to keep the leadership. Moreover, when a great innovation is brought by one vendor, the other vendors will follow shortly. This is exactly what happened recently with the support of JSON data type, or the in-memory RDBMS. Same with Java processing optimization, like async/actor programming (Akka, Vert.x), or distributed VMs, ... In both spaces, the innovation has been rocking these past years, for our own benefit!

On the other hand, if you choose Node.js, well you're locked to one vendor, and even one JS engine (v8). I'm not saying that these are bad, just that you' don't have any choice. Same, and even worse, with MongoDB: do you want to put your business critical data into the hand of a single database vendor? If something goes wrong with these guys, what are your options beyond re-coding your applications to another API/vendor? This is about risk management, a point that every CIO office has in mind.

Of course, diversity implies differences. But it is pretty easy to overcome them or, better, to leverage the strength of a particular implementation when it makes sense.

Skills and assets

Because of their ubiquity, both RDBMS and Java are already well known. This is not only about development skills, but administration, management, backups, and even procurement. The processes are in place. The IT teams are trained: they have to figure day to day issues like performance, network congestion, server availability, quick backup/restore... And they know how to deal with that.

Finally, the Java ecosystem is far more mature than the Node.js one. Yes, Node is vibrant but go to npm: how do you "separate the wheat from the chaff"? You can find many libraries made by individuals, staled at version 0.x and asking for a new owner. There are some open source organizations around this technology but nothing yet that matches Apache, Eclipse or even OpenNTF. You might find licenses coming with the code but this is not enough: what about the certificates of originality? What guarantees you that the code you're using has not been stolen?
As bigger companies are chiming in, this will certainly evolve to a more controlled world. IBM, for example, has it own internal npm server. But I bet that many smaller companies, or companies not in the IT business cannot afford doing the same.


In short, choosing the technologies that power a framework goes way beyond comparing their technical characteristics. There are other factors that can be even more important, and that's the reason why we chose Java and RDBMS. Now, on the pure technical side, both also have very competitive advantage, but I'll expand on that later!


Sunday, November 12, 2017

Develop a Domino application using any modern tools

In my previous post, I talked about a ReactJS library of components that can be used to create Notes like UI. It allows developers to create modern web and mobile UIs in a flash. Fine but, if Domino Designer is a great tool for rapidly developing applications, it is unfortunately not designed to create such modern applications, mostly because:

  • It does not integrate with the modern tool chain required by these libraries, like maven, npm,  babel, webpack...
  • It has a very limited JavaScript editor that cannot compete against Atom or MS Visual Code. Not talking about the latest standards support, like ES6.
  • Source control and continuous build/integration are very limited
  • Well, it is Windows only

We've been thinking about these problems for a while and we worked to provide a solution. Part of the problem comes from the fact that the NSF is both the development and the deployment container, with no third party development tool in the market being able deal with the NSF content. The on-disk projects were an attempt to fix that issue, but this is for source control only.

With Darwino, we go a step further: we allow developers to create projects using the tools they love (Eclipse, IDEAJ, Ms Visual Studio Code, Atom, ...), then use any of these tools during development and deploy the final result into an NSF.

Ok, better to first see it in action:


In this example, I used the Darwino studio to create a new ReactJS application within Eclipse and I'm using MS Visual Studio Code to develop the client side JavaScript. I can use source control, maven, npm, without any limitation, as I'm outside of Domino Designer. Moreover, the Darwino ReactJS library allows me to seamlessly call the Domino REST services, even though I'm running the Webpack dev server. Ah, you might have noticed, I did all the development on a Mac...

When I build the project by running maven, the relevant built files (JavaScript files, Jar files for the business logic...) are seamlessly deployed to the NSF, making the application instantly available from the Domino server. The source files are by default *not* deployed to the NSF, making it smaller and not triggering a build process when opened within Designer. Finally, your IP is also protected.

The magic behind that is a design element server installed right into Domino Designer, as part of the Darwino extension for Darwino Designer. This server exposes some REST services on port 8800: list of projects, access to design elements, ... These services are called by a client, like the Darwino studio in Eclipse, to deploy the runtime files when desired. Because it is all based on standard REST services, other clients, like pure JavaScript ones, can be developed as well. Seems easy right? Well, the evil is as always in the implementation details, but hopefully this is all written using the Darwino runtime, which drastically simplified the development.

The cherry on the cake: the code for these services is isolated from Domino Designer itself, so it could be installed directly within a Domino server. This would remove the requirement of using Domino Designer for development, at least when not developing"classic" notes applications -> it can enable a pure Web based Designer for citizen developers.

Let me know what you  think about it!

Darwino Studio Screenshots

Configuring the Design Server

Configuring the project files to synchronize:

Thursday, November 9, 2017

Modern Notes like UI using ReactJS

The world is evolving fast, and so technologies are. Today, for now a little while, when we talk about building a new web UI or a mobile hybrid one, we think about using pure client technologies without markup generation on the server side. JSF, JSP, ASP.NET are being replaced by Angular, ReactJS, VueJS and services...  I personally think this is a great evolution. But, are these technologies easy enough to use? Can a developer be as productive as he/she is with XPages, for example? Well, the quick answer is no, at least without any addition to these core libraries, in term of components and tooling.

This is the problem we are currently tackling with Darwino: let's make it easy to consume these new JavaScript technologies without compromising their power. For these, Darwino 2.0 is featuring a brand new ReactJS component libraries as well as some new templates (or boilerplates, or pods, ...) that make the creation of a project easy.

The components should give you all the necessary UI constructs to create a form based application. Of course, all of this is responsive to adapt to mobile devices. In short, everything you need to build Notes/Domino like applications in minutes, but with pure standard ReactJS.
Curious? Have a look a running instance here http://playground.darwino.com:8080/contacts-react (use demo/demo to log in)

The applications should also be data source agnostic, in a sense that the same application should,  almost seamlessly, connect the Darwino JSON store, or to Notes/Domino backend, or even other DB like LiveGrid data store. With this architecture in mind, these applications can work online, connecting to any server data, but also offline, connecting to the locally Darwino replicated data in your mobile device.

There are different steps in this journey. Let's go through them.

Choosing the JS library/framework

There is nothing wrong picking up one of the major library currently available. In a previous post, I explained why we choose VueJS for ProjExec and how our developers quickly became productive using this library. It was great with this goal in mind.
Darwino has different goals. As a development tool, it should target the broader set of use cases. This implies using a library with a bigger eco-systems, a large set of third party additions, several rendering libraries (bootstrap, material design...). For these reasons, we choose ReactJS, as it is adopted by the many vendors like IBM or Microsoft. The later is interesting, as we can quickly create apps with the Office look & feel, thanks to the Office UI Fabric

ReactJS it is.

Choosing the right ReactJS components

ReactJS by itself is low-level: let's consider it as a powerful, core engine that can be extended through components. For example, ReactJS has *no* built-in data binding capabilities like the one you'll find in Angular or VueJS. You should either implement them yourself, or use a third party library.
To get a decent set of capabilities without reinventing the wheel, you have to pick-up libraries from one of the large JavaScript repositories. Nowadays, I would advise you to use npm (or eventually yarn) and leave tools like Bower to the historian.
Gasp, this is where the troubles come in: the npm repository features a plethora of libraries designed for the same job. How would you pickup one over the others? Number of downloads? Stars on Github? Recent activity? Return on experience from other developers? Making the right choices takes time as there is no obvious ones.

For Darwino, we choose the following:
  • react-redux
  • react-router
  • redux-form
  • react-intl
  • react-data-grid
  • react-bootstrap (or eqv for other UI targets)
  • ... and a few others

Project boilerplates

If, for you, building an app if about saving a design element in an NSF, then your life is good. The JavaScript world is way more complex! Even if JavaScript is an interpreted language, you'll quickly feel the need of running build tools to use the latest JS specifications, consume libraries, produce an optimized/compressed version of the code... Again, you 'll have to make some choices between Grunt, Gulp, Webpack, Browserify, ... and many other tools. This is another time consuming research.
Once your choice is made, you have to configure the tools, generally using a JavaScript based DSL (e.g. a JavaScript piece of code driving the tool). This is fairly complex as the DSLs feature many options. To get started, developers created templates, also called boilerplates. They contains the configuration files with default behaviors that you then have to customize. But again you have many available, with some clearly developed by people without a deep knowledge of the technology. Furthermore, they are generally tailored for a particular version of the tool and will not function properly with yours...
Ok, we spent many days of try/fail/redo and we are coming with our own template. Well, it is even easier: go to the Darwino studio, create a new project, select the target technology and voila, you're ready to go. Pfuuuu...

Component library

As we'd like to make the developer experience as smooth as possible, we are providing a series of ready to use components covering most of the developer needs. What started as a demo library became a very useful set of components, covering:

  • Forms with document access (load, save, validation)
  • Data binding controls, including read-only mode, multiple values, pickers, rich text, attachments upload & download
  • Computed fields (yes, not that obvious with ReactJS!)
  • Data grid with Notes view like behaviors: categorization, totals, response doc
  • Navigators, action bars, sub-forms
  • Dialogs, messages
  • Easy access to server side business logic using micro services

And more... But again, these components are just regular ReactJS components, that any ReactJS developer would consume without fear. They do not carry any idiosyncrasies from the original Notes ones. They exist to create Notes like UI for Web and Mobile, not to replicate 1-1 what the Notes UI does.

Ah, I foresee a question: can you automatically convert an existing Notes UI to this technology? The answer is... yes! Feel interested?