Monday, September 3, 2018

Give a new life to your old Android devices

As many mobile developers, I own a fair amount of devices that now start to show their age. They are still ok from an hardware standpoint, but Android is stuck at a 5 yrs old version. In particular, I own a few Samsung tablets (SM-T520, SM-T900...) blocked at 4.4 or 5.1. Samsung is really not good at upgrading their devices which is a shame. Yeah, I don't like Apple a lot, but my venerable iPad is smoothly running the latest iOS.

I took the labor day week-end opportunity to see if there is a way to upgrade these devices. Cyanogenmod is now gone, but the team started LineageOS, which is an up-to-date Android distribution based on the former one. The T-520 being 4.4, I decided to give it a try as I have almost nothing to loose. I also found that we can always recover from a bricked device by downloading the original ROM from Samsung. Ok, let's take the risk!

I used the instructions from here: https://wiki.lineageos.org/devices/n2awifi. There are good so there is no need for me to repeat them, but just throw in a few findings:

1- I started using a Windows 10 machine, but it always failed installing the Heimdall driver. I made different attempts, got different errors until I gave up. Maybe because I already had several Android related drivers installed on that machine? On the other hand, I did the same using my Macbook and it worked like a charm! Conclusion: use a a Mac, my friend!

2- The link to download TWRP is broken. I finally found mine here: https://dl.twrp.me/picassowifi/. The name of the device was different as n2awifi became picassowifi.

3- Have an SD card ready on the device so you can copy the OS images on /sdcard. Not sure if it works with the main storage, as all the instructions refer to /sdcard.

4- The recovery UI is a bit tricky. To get the Google Apps installed, you have to flash them at the same time than the OS, before you even boot to LineageOS. If you don't do that, they will be unstable. The Android initial setup process is different with or without the Google apps, I tried both :-)
When you flash the OS, first add the LineageOS image then, before flashing, click 'Add a new File' and select the GApps files for you architecture. They will be installed in order.

5- I got multiple crashes while flashing the OS. Yes this is scary, but the tablet crashed and rebooted during that operation at different flashing steps. It happened to me 4 times before it finally completed successfully. Also, after a crash, the special reboot key sequence (home+volume up/down) did not always worked. Try to shutdown the tablet often restart it it, without offefing the opportunity to enter the recovery mode. Be patient, try as many times as needed, and you're eventually get there.

The tablet is now working Android 7.1 smoothy, with all the apps I need. I have not seen any issue so far. I have not tried 8 yet as it is not officially supported (you can find the ROM anyway if you're adventurous). Maybe my next jump will be Lineage 16 (Android 9). I'll try that with the SM-T900!

LineageOS is a great way to revitalize your own devices, or buy some old but good enough hardware and upgrade the software version. The SM-T520 has a great 10.1 bright screen (2560x1600) at nowadays a cheap price!


Wednesday, January 17, 2018

Darwino makes OpenNTF mobile!



[Latest News] The iOS app works perfectly, but we are struggling with the great fruity company to get it accepted to the store, because they claim there is not enough mobile content on OpenNTF... We'll do our best with the OpenNTF team to get more content leveraging mobile capabilities.

Darwino 2.0 features a mobile micro application architecture well integrated with the IBM Domino server. It can render any existing Domino web application (classic & XPages) on a mobile device while taking care of the painful details:
  • Real mobile applications available from both Google Play and Apple Store
  • Authentication management, to avoid prompting the user when the server requires credentials
  • Opening external pages in a separate browser window to provide the best user experience
  • Mobile notifications, to alert users when something new arrived and let him/her act appropriately
  • Plus many implementation details, like error management and recovery
To quickly experiment with the capability, we are providing a generic mobile client that one can use to connect to any Darwino enabled Domino server. This client is available on both application stores and is fully configurable to point to any server. As a default configuration, we make it point to OpenNTF!

As most of you know, the main OpenNTF applications, like Collaboration Today, are XPages based with a responsive design. This make them very easy to integrate with the micro application architecture. Moreover, Darwino extends Domino Designer with a new design element to detect database changes and send mobile notifications. With that in place on the OpenNTF servers, we are able to send new notifications when a new entry is available in Collaboration Today. By the way, these notifications use the mobile first IBM Cloud service (a.k.a. Bluemix).

The implementation on the OpenNTF server was straight forward:
  • Get the Darwino OSGi runtime installed
  • Set some configuration properties
  • Create a new design element for mobile notifications. To leave the existing databases untouched, we did that in brand new, separate database
Voila, in less than an hour it can be all setup. The same can happen to your own applications!

What are the next steps then? Well, a few things:
  • We need to work with the OpenNTF team to mobile enable other parts of the site. Ideally, the whole site should be mobile enabled!
  • Why not white labeling the generic client with a pure OpenNTF branding (icon, title, ...)? Darwino offers this options.
  • Move forward and create Darwino single page applications, using web or mobile native technologies, to get the data offline and provide the best user experience without connectivity!
Stay tuned, OpenNTF becomes mobile!


Tuesday, January 9, 2018

Dynamic document based security

Domino provides document based security through the use of readers and authors fields. This is a great capability that is fairly unique in the NoSQL industry heavily used by Domino developer. One caveat though: you have to explicitly manage the list of users/group/roles in each document, generally based on the document state or content. How many times did you write an agent (background task) to update these fields nightly?

Since the beginning, Darwino supports readers and authors (as well as excluded readers and authors to remove specific users from the authorized list). It is implemented by a hook in the SQL generator that systematically adds a condition to the queries. Here is how it looks like in pseudo SQL:
SELECT D.JSON FROM DOC D WHERE (....) AND (D.READERS IN (:userid, :userroles, :usergroups...)

userid, userroles and usergroups are values related to the user making the request.
All of this is implemented (and by the way optimized) at the core Darwino level. So it applies regardless of API you use: Java, JavaScript, REST services, GraphQL...

But in 2.1, it goes beyond this simple capability: it allows an application to insert its own SQL filter:
SELECT D.JSON FROM DOC D WHERE (....) AND (<custom condition>)

For example, the custom condition can be based on fields within the document. Let's suppose that the current user object has an attribute 'country' set in LDAP. Let's also suppose that the documents in a JSON store also have a field name 'country'. From a security standpoint, we only want to allow the users to see the documents tagged with their country. With classic readers fields, you'll need an agent that periodically updates the document readers, or use groups and assign users to groups. In both cases, there is some maintenance to do.

With Darwino, it becomes as easy as implementing a runtime extension that returns a condition for the current user (again, simplified pseudo code):

String getDynamicSecurity() {
    User u = getCurrentUser();
    return "{country: '"+u.getCountryAttribute()+"'}";

}

There is no need to maintain a list of readers here. If a user is moved to a different country, then the query will use the latest user country (or any other needed value), immediately!

The dynamic security condition can be expressed using the built-in, MongoDB like, query language as above. This platform independent query is transpiled to SQL by the core Darwino code.
But it can also be expressed as raw SQL, allowing the filter to be as complex as desired. For example, it can lookup in another SQL table, view or stored procedure for a list of authorized ids for the current user:
... AND (D.DOCID IN SELECT ID FROM SECURITY SEC WHERE SEC.USERID='currentuserid')
This is limitless. Let's suppose that you have a users database where each user document has the name of its manager. Now suppose that user documents can only be accessed by the user and his managers. Well, just get the list of authorized IDs from a recursive SELECT statement
... AND (D.USERID IN WITH RECURSIVE SELECT ...)
If the subquery becomes too complex and leads to performance issues, you can create materialized views that will be managed by the database server.

Come on MongoDB, you cannot compete against the power of relational databases :-)




Thursday, January 4, 2018

Offline your data with React-Native applications


In a previous post, I mentioned how we are leveraging ReactJS to create state-of-the-art web applications. We are now moving a step forward by also leveraging its sister technology, React-Native, to create portable, yet native, mobile applications. With all the benefits of Darwino, which includes a local JSON store that replicates for the best offline experience, and all the other services provided by Darwino!

To make this as easy as possible, we are providing a simple service based API, that can be called from any native language (Java, Objective-C, Swift).

1- Darwino service API

The API is basically made of one class, exposed bellow in pseudo code:

class DarwinoApis {
  ...
  Object fetch(url, params, headers, content)
  ...
}

The fetch method calls the service identified by the URL and associated parameters, and returns the result. Behind the hood, such the API call will be either translated to a remote HTTP service call (online mode), or to a local API call (offline mode). This is done transparently by a dispatcher, so the developer does not have to worry about the current mode. It just works! This API can be called from *any* native application, regardless of the library/framework it uses.

This shows the beauty of the Darwino architecture: the services are all written independently from the HTTP stack, and thus can be executed locally without the need of HTTP calls. It makes them truly available offline, while providing the best possible performance as it also avoids the data serialization/deserialization needed by the HTTP transport.

Here is a basic architecture picture:



2- ReactNative integration

As the API is exposed using the platform native languages, it is easy to bridge it with ReactNative. Moreover, the bridge enforces the use of promises, thus enabling the new aync & await capability for the best developer experience.

Here is, for example, how to retrieve the number of documents from a database, regardless if it is local or remote:

async databaseCount() {     let url = ["$darwino-jstore","databases",this.getDatabase(),"documentscount"]     let count = await darwinoApis.fetch(url);     this.println("This database contains {0} documents",count);
}
Easy, isn't it?

Triggering the database replication isn't more difficult, see:

let result = await apis.replicate(this.getDatabase(),options); this.println("{0}",result);

3- Darwino API distribution

The APIs is provided as a library to be added to any existing application. On Android, it is a maven artifact that you can reference from your gradle build file. On iOS, it is a Cocoa framework that you can add to your XCode project. Just add these dependencies and you're ready to go!

Android:

iOS:

4- Under the hood

Wait, Darwino is Java based, so how can we provide a native Objective-C or Swift API? How can we get all the Darwino services, built-in and customs, run natively on any device? As the goal is to have one single Java source code to ensure the consistency between all the platforms, so how is is exposed as an Objective-C API?

The magic comes from J2OBJC. Right from the beginning, we have been very careful to write "portable" Java, meaning Java that can be compiled or transpiled to many different environments (JRE, Dalvik/Art, RoboVM, Multi-OS Engine...). As J2OBJC made a lot of progress these past years, it became easy to get our code transpiled to J2OBJC. The result is a tiny, efficient, native Objective-C library while sharing the same Java source code.
With the potential to target other platforms in the future, like .NET: what about having your Domino data replicated natively to .NET? :-)

Tuesday, January 2, 2018

Darwino 2.0 is out!

I'm very exited to announce that the official Darwino 2.0 has just been released!



It had been a long journey since the previous release but most of our customers are already using the pre 2.0 release, aka 2.0.0-SNAPSHOT in maven terms. This means that this release has already been tested in many contexts and is thus very stable.

The main topics for this major release are:

  • Creation of new web and mobile applications
    - Provide the best user experience even when the device is offline
    - Leverage of the latest technologies, including ReactJS
  • Modernization of existing applications, in particular Notes & Domino
    - Liberate the Domino data and take advantage of any data reporting/analytics tool
    - Get your existing Notes/Domino apps on mobile device with notifications
    - Create modern mobile applications connected to your IBM Domino data
  • Migration of IBM Notes/Domino applications
    - Incremental migration to minimize the risk
    - Use of modern technologies and development tools
    - Integrate with MS Sharepoint, Office 365 or other platforms, like Oracle Cloud
  • Many additions and enhancements to the existing APIs

I bet that the mobile client is going to be popular in our community because... Well, read my next post :-)

Bellow is a list of the most noticeable features in 2.0:

Darwino for mobile devices


Darwino mobile client
Darwino provides a generic client that renders server side web applications and handles IBM Cloud notifications. This client is available for both iOS and Android, in their respective application stores.
This client can also be fully customized as the full source code is available. For example, one can white label the application with a custom icon, look&feel, options...

Multi-OS-Engine instead of RoboVM
Darwino switched to MOE as RoboVM is no longer available since it has been acquired by Microsoft. Darwino 2.0 supports the latest version of MOE. Thanks Migeran for doing this great product!

Integration


Notification framework
Darwino provides a notification framework that executes actions based on events. The framework can detect data changes in a database, including domino databases, and run actions like a mobile push notifications, notifying IBM Workspace, sending an email or whatever a piece of code can do...

Microsoft Azure active directory support
Darwino features a driver to consume Microsoft Azure active directory services

Microsoft Sharepoint integration
Darwino integration with Sharepoint, including directory access and single sign-on.

More database support
The MS SQL server support is complete, as well as MySQL (the full JSQL requires MySQL 8+).

Darwino DB


JSQL
JSQL is an SQL dialect dedicated to JSON databases. It allows the full power of SQL queries on top of JSON documents: joins, unions, groups, recursive queries, ... while honoring the document level security. It eventually translates to the native database SQL for maximum performance.
https://playground.darwino.com/playground.nsf/JsqlSnippets.xsp

Registered queries
Database queries can be defined on the server side and consumed by name from the client. This is also true for JSQL queries.
https://playground.darwino.com/playground.nsf/JavaSnippets.xsp#snippet=Json_Store_Cursor_Run_Predefined_Query

Domino @formula
An extension to the query language uses the new keyword $atFormula and allows the use of a subset of the Notes/Domino @formula language in Darwino queries. That makes the migration of Notes/Domino views a lot easier!
https://playground.darwino.com/playground.nsf/DarwinoDbSnippets.xsp#snippet=@Formula_Simple


REST services


JSON store services
Added some new services to retrieve document content without the meta-data.
https://playground.darwino.com/playground.nsf/OpenApiExplorer.xsp#openApi=Json_Store_Document_CRUD

GraphQL

Darwino comes with a innovative implementation of FaceBook's GraphQL, allowing both schema based and schema less queries. Moreover, GraphQL queries can be pre-defined on the server side to make them easier to consume
https://playground.darwino.com/playground.nsf/GraphqlSnippets.xsp

Microservices
Darwino provides a easy to use JSON microservice infrastructure. The services can be written in Java, Groovy, JavaScript, Darwino Script or any language supported by the JVM.
https://playground.darwino.com/playground.nsf/ServicesSnippets.xsp


Darwino Studio


Application Wizard
The application creation wizard can now use templates, also known as boilerplate, to get the developer started quickly. A new template using ReactJS and Bootstrap is provided as part of the studio. The architecture is open and any developer can contribute new templates.

Android studio
The wizard can also leverage Android studio for the mobile applications, instead of Eclipse. This allows the developers to benefit from the latest features provided by Google and Multi-OS-Engine.

UI Builder
Darwino studio features an easy form builder that targets mobile devices. This builder is fully extensible to support more targets in the future.


Darwino for IBM Notes/Domino


Domino Designer
Darwino provides a new Domino Designer plugin that adds a new design element dealing with the notifications. It uses a groovy based DSL (Domain Specific Language) .

Darwino client
The Darwino mobile client detects if the server is an IBM Domino server and adapts some behaviors accordingly, like error handling.

Darwino application
Darwino provides new runtime capabilities to supplement IBM Domino missing features, like JEE filters. Moreover, Darwino applications can be auto-started, for example to monitor events.

IBM Domino import
The Darwino studio can generate a basic UI based on existing IBM Domino NSFs. This feature is a preview, but this is something we'd like to enhance moving forward, to get the mobilization or migration of Notes/Domino import easier.
It also generates JSON Schemas based on the Form definition in an NSF.


Developer tools


Commands framework
Darwino features a command line framework that developer can add to any application to execute administration commands or debug an application.
It also includes a built-in application profiler.

Playground
The Playground now show cases the new features like JSQL or GraphQL, but also exposes the REST apis using OpenAPI/Swagger: https://playground.darwino.com/playground.nsf/OpenApiExplorer.xsp

Friday, December 15, 2017

Ubiquity of Java and relational databases


I'm sometimes being asked about our platform choices of Java and RDBMS for Darwino. More precisely, I'm getting questions about Node.js or existing JSON document databases, like MongoDB or CouchDB.
Well, I can certainly expand on the technical value of our choices, and I'll probably will. But, if the technical aspect is important, this is not the only driver.  It is not even the main one: what made us choose these technologies can be summarized in 3 words: ubiquity, diversity and skill and assets.

Ubiquity

Most of our customers have a Java application server and a relational database already in place. Both are almost everywhere. If small companies can be flexible with technology, bigger companies tend to be more conservative. And not to talk about administrations. With Java and RDBMS you can get in right away without going through technology validation processes.

The cloud is not different: whenever you want to use MS Azure, IBM Bluemix, Oracle Cloud, Amazon AWS, they all offer you PAAS services with a Java application server and an RDBMS. As a result, your not locked with a vendor. For example, and despite its value, if you choose IBM Cloudant, you're locked with IBM Cloud. For the good and the bad... Did I say the bad?

It similarly expands to mobile devices. There is no MongoDB implementation running on these devices, providing offline capabilities with the same set of features. Ok, Cloudant/CouchDB has TouchDB or PouchDB for offline, but these are totally different implementations with different APIs and capabilities. On the other hand, SQLite can be run on any mobile devices from Java.

With Darwino encapsulating the platform differences, like SQL dialects, you write the application once and you can run everywhere, including on mobile devices.

Diversity

There is not a single implementation of Java and RDBMS. Rather, you can find several vendors providing their own JVM, web application servers or RDBMS, with different characteristics. You pick and choose the one that better fits your need, based on both technical or business decisions.

Diversity fosters innovation because of the competition: the vendors have to innovate to keep the leadership. Moreover, when a great innovation is brought by one vendor, the other vendors will follow shortly. This is exactly what happened recently with the support of JSON data type, or the in-memory RDBMS. Same with Java processing optimization, like async/actor programming (Akka, Vert.x), or distributed VMs, ... In both spaces, the innovation has been rocking these past years, for our own benefit!

On the other hand, if you choose Node.js, well you're locked to one vendor, and even one JS engine (v8). I'm not saying that these are bad, just that you' don't have any choice. Same, and even worse, with MongoDB: do you want to put your business critical data into the hand of a single database vendor? If something goes wrong with these guys, what are your options beyond re-coding your applications to another API/vendor? This is about risk management, a point that every CIO office has in mind.

Of course, diversity implies differences. But it is pretty easy to overcome them or, better, to leverage the strength of a particular implementation when it makes sense.

Skills and assets

Because of their ubiquity, both RDBMS and Java are already well known. This is not only about development skills, but administration, management, backups, and even procurement. The processes are in place. The IT teams are trained: they have to figure day to day issues like performance, network congestion, server availability, quick backup/restore... And they know how to deal with that.

Finally, the Java ecosystem is far more mature than the Node.js one. Yes, Node is vibrant but go to npm: how do you "separate the wheat from the chaff"? You can find many libraries made by individuals, staled at version 0.x and asking for a new owner. There are some open source organizations around this technology but nothing yet that matches Apache, Eclipse or even OpenNTF. You might find licenses coming with the code but this is not enough: what about the certificates of originality? What guarantees you that the code you're using has not been stolen?
As bigger companies are chiming in, this will certainly evolve to a more controlled world. IBM, for example, has it own internal npm server. But I bet that many smaller companies, or companies not in the IT business cannot afford doing the same.


In short, choosing the technologies that power a framework goes way beyond comparing their technical characteristics. There are other factors that can be even more important, and that's the reason why we chose Java and RDBMS. Now, on the pure technical side, both also have very competitive advantage, but I'll expand on that later!


Sunday, November 12, 2017

Develop a Domino application using any modern tools

In my previous post, I talked about a ReactJS library of components that can be used to create Notes like UI. It allows developers to create modern web and mobile UIs in a flash. Fine but, if Domino Designer is a great tool for rapidly developing applications, it is unfortunately not designed to create such modern applications, mostly because:

  • It does not integrate with the modern tool chain required by these libraries, like maven, npm,  babel, webpack...
  • It has a very limited JavaScript editor that cannot compete against Atom or MS Visual Code. Not talking about the latest standards support, like ES6.
  • Source control and continuous build/integration are very limited
  • Well, it is Windows only

We've been thinking about these problems for a while and we worked to provide a solution. Part of the problem comes from the fact that the NSF is both the development and the deployment container, with no third party development tool in the market being able deal with the NSF content. The on-disk projects were an attempt to fix that issue, but this is for source control only.

With Darwino, we go a step further: we allow developers to create projects using the tools they love (Eclipse, IDEAJ, Ms Visual Studio Code, Atom, ...), then use any of these tools during development and deploy the final result into an NSF.

Ok, better to first see it in action:


In this example, I used the Darwino studio to create a new ReactJS application within Eclipse and I'm using MS Visual Studio Code to develop the client side JavaScript. I can use source control, maven, npm, without any limitation, as I'm outside of Domino Designer. Moreover, the Darwino ReactJS library allows me to seamlessly call the Domino REST services, even though I'm running the Webpack dev server. Ah, you might have noticed, I did all the development on a Mac...

When I build the project by running maven, the relevant built files (JavaScript files, Jar files for the business logic...) are seamlessly deployed to the NSF, making the application instantly available from the Domino server. The source files are by default *not* deployed to the NSF, making it smaller and not triggering a build process when opened within Designer. Finally, your IP is also protected.

The magic behind that is a design element server installed right into Domino Designer, as part of the Darwino extension for Darwino Designer. This server exposes some REST services on port 8800: list of projects, access to design elements, ... These services are called by a client, like the Darwino studio in Eclipse, to deploy the runtime files when desired. Because it is all based on standard REST services, other clients, like pure JavaScript ones, can be developed as well. Seems easy right? Well, the evil is as always in the implementation details, but hopefully this is all written using the Darwino runtime, which drastically simplified the development.

The cherry on the cake: the code for these services is isolated from Domino Designer itself, so it could be installed directly within a Domino server. This would remove the requirement of using Domino Designer for development, at least when not developing"classic" notes applications -> it can enable a pure Web based Designer for citizen developers.

Let me know what you  think about it!

Darwino Studio Screenshots

Configuring the Design Server

Configuring the project files to synchronize: