How could applications look like in the future? #2

This is in continuation to my previous blog with the same topic… https://julianfrank.wordpress.com/2018/01/21/how-could-future-apps-look-like/ … So why again??

One of my pet project was going complicated beyond control to the extent that I hardly remember why I wrote the code in a particular way :sigh: … Hence I’ve been looking for converting my tiny yet complicated monolithic application into a micro service application! Looked around and found istio a few months back and love the level of policed flexibility it provides. This is great for enterprise environments with relatively large teams handling the whole new stack… Sadly for my project that was way too much overhead and complexity for a relatively few features that I was trying to build…

Then I discovered moleculer. This is a nice framework that has all the fundamental features of micro-service ecosystem ticked and yet relatively small and less complicated for my weekend project needs…You can read about its features at a glance at http://moleculer.services/docs/0.13/ … I moved some services -> got into trouble -> cried out for help -> got support from @icebobcsi and others in its gitter.im/moleculerjs/moleculer channel -> now my app is back on track…. Very impressed!

To add to the benefits, the capability to merge and separate services across nodes means that I can code in a system with all services in one node… When deploying to production, I can split the services into separate nodes and make then run in separate container or vm instances… The Framework cares nothing about where you host except that they are able to reach each other over the network. To make things simple you can start with service discovery over TCP Broadcast! Of course for large scale deployment, other mechanisms like NATS, MQTT, Redis etc would be a better fit. My only quip was that this is locked to NodeJS. I Hope that in future, other cloud friendly languages like Golang is also supported…

Back to the topic… My idea of apps being Universally Discover-able is already a possibility Now!

What is left is the service (which I referred to as app in my previous blog) to be self-aware and adapt to work conditions as per needed….

Just imagine a situation where we got five services as part of our application: A, B, C, D, E

Now imagine a situation that the app is bombarded with two users with one user targeting A while the second user using a mix of services

A self-Aware application would in this situation split itself into multiple nodes, letting the first node focus on user 2 while creating copies to handle the high demand from user 1…

When the load is removed, the services move back to a minimal node instance back to original size….

Currently this is facilitated by K8S with careful engineering from IT. But in future I believe, the framework would take care of deciding how the service behaves and the process of mitosis by itself… Am I just day-dreaming?? I don’t know !!

Rendezvous with Picroft

This weekend I decided to try out Picroft which happens to be the Simplified Software Only version of the Mycroft Project that is ready to be run on any Raspberry Pi 3 hardware…. Great! So I read through the instructions provided in their website … There were two deviations that I wanted to use

  1. Use the HDMI Audio for Audio instead of the hard coded Analog Stereo Output in the RasPi3
  2. Use Microphone outside the list of mics mentioned in their site…. I’m referring to the microphone in my ancient Microsoft Lifecam VX-6000 to be precise!

So I installed and connected per instruction… The microphone got detected without any problem and Mycroft learnt ‘my’ voice so well that it wouldn’t respond to my son!

Unfortunately the Audio Output is hardcoded to the on-board analog stereo jack which I did test and was working well but I wanted out put on my HDMI display which has much better acoustics! A bit of googling and I found the solution in this site: https://mycroft.ai/documentation/picroft/picroft-audio/#how-to-output-audio-via-hdmi … Its simple … Just

'Control-C'

out of the debug screen and then open the auto_run.sh file… In the line that says sudo amixer cset numid 3 “1” … Change it to

sudo amixer cset numid 3 "2"

…thats it … on the cli type

exit

and picroft will reload with audio output on the screen instead of the audio jack! Yipee!

Unfortunately looks like the software is still under heavy development on the picroft side… I found the response time to be slower than AVS and also the accuracy is good but needs improvement….

Next I need to find if it can work in a pure LAN environment without Internet! But … Weekend almost over … So that’s for some time later… Peace Out till then

PoPriCl –> My POrtable PRIvate CLoud

Wow …time is running fast! I can’t believe it’s been three months and no blogs in 2017! …Well the wait is over…

So, what am I up to recently… Well you must have read about my ‘Hobby
Stack
’ last year… Now this stack is great if I want to host a tiny Hobby app but becomes problematic for development as it is too reliant on Internet…
Hobby Stack unfortunately does not help me much when I’m in a 10-hour flight? I could run Docker in my laptop but what if I’m having a laptop where I don’t have admin rights and no Hyper-V/VirtualBox? To add to the problems, now Laptops are getting banned as carry-ons. That means, at best tablets and mobiles is all we will have from user side.

I may be taking up some long travel soon and the thought of building my Private Cloud ‘Server’ that I could easily carry in flight is what prompted me to build one and write this blog…

The Requirements

So here are what I would state as the ‘Requirements’ for PoPriCl…

  • The Box Needs to be ‘Portable’ meaning small enough to fit without any problem in a laptop bag and ready for check-in. Also, when in ‘working’ state it should be able to fit comfortably within the confined space of the worst economy seats in a plane.
  • Portable Power’ for >10 Hours -> I don’t think I can code for more than 10 hours on long haul flights…So this looks reasonable. Also, It is reasonable to look for external power but not all flights have USB/AC Ports available in all seats.
  • Portable Wireless Network -> Capability to host its own Wireless Network would eliminate the need for a wireless router and need for congesting Ethernet cables
  • Inexpensive -> The whole Hardware + Software should come in a budget of around ₹5000 … No Monthly recurring costs
  • Currency with my ‘Hobby Stack’ software features to the extent possible…
  • Hosted WebIDE so that I don’t need any Software…even Portable Software on my laptop… Only a bunch of Modern Browsers

That’s it…I don’t ask for too much, right!

The Hardware

The Board

There are many Options here but to satisfy the Portable Power criteria I needed kits that can be powered using Power Banks that supply power over USB. This is usually 5V/2A …That rules out the Intel Edison based x86 IOT Kits which need 12V supply…The rest are just too slow and specifically meant for Low Power IOT and not desktop grade…Outside the x86 world, comes the arm based kits and are powered by mini-USB. Of these I found the Raspberry Pi 3 Model B to be the best fit in my budget and still being desktop grade to handle at least 2-3 small containers easily. This comes with A 1.2GHz 64-bit quad-core ARMv8 CPU,802.11n Wireless LAN and Bluetooth 4.1 / Bluetooth Low Energy (BLE).

Storage

I got a simple 32GB micro SD Card for use as both the OS/Firmware and the Storage.

Power

For Portable power, I was not sure how much would be good enough so I got a 20AH Power Pack… As per my Test it easily lasts for more than 24 hours…. So, I guess I could have used a lower end Power Pack as well.

Total Hardware BoM

Item

Regular Market
Price

Comments

Raspberry Pi 3
Model B

₹2,890

With Case/SD-Card/Power Supply the kits may go to about ₹4,000

SD-Card

₹700+

Better Deals as time passes by. By the time you read this the rates may have gone
down further. Also Lower Capacity is also fine….

Portable Power

₹1500+

The Price Ranges from ₹1500 to ₹2500 …Also considering the
24Hour+ Capacity for 20AH, Lower capacity could be opted for as well

Total

₹5090+

Its slightly over 5K Budget but not too much so excused 😉

How do I make it work for me?

Now that I have the gear, let’s set it up for action…

The Boot Image Mystery

The most common boot image used in Raspberry is the Raspbian. It is small and efficient but I wanted more. I chose to go with Hypriot OS which is fundamentally a Raspbian modified to work well with the latest Docker, swarm and Kubernetes configurations. I found this appealing as I would like to move straight to Developing stuff that tinker with
the knobs in Docker/Kubernetes. I have tried other images but ultimately Hypriot wins.

I used etcher to burn the ISO downloaded from https://blog.hypriot.com/downloads/. Just remember that in the default image the username/password is pirate/hypriot (all small).

Plug the micro SD card into the slot in RasPi and power up the RasPi. Ensure you have a LAN connect, HDMI monitor and USB keyboard attached.

On Login used the pirate/Hypriot combination and login. Now we need to setup the wireless.

Wireless Setup

Power up your AP provider which in my case was my mobile phone with hotspot switched ‘on’.

Scan the local wlan networks using

iwlist wlan0 scan | grep SSID

Ensure that the AP is listed…if yes then ‘Great!’ , else find some other AP that is detectable by RasPi.

 Do not Proceed if your AP is not visible in the scan.

This method below for wlan may change in future versions…(Please keep track of the latest instructions in https://github.com/hypriot/flash#occidentalistxt) to enable Wireless Accesspoint

Open the Hypriot config file

Sudo nano /boot/occidentalis.txt

Change the hostname, wifi ssid and password(plaintext) as needed

# hostname for your Hypriot Raspberry Pi:
hostname= < Your new hostname>
# basic wireless networking options:
wifi_ssid= < correct SSID Name>
wifi_password= <correct password>

Save and exit

Now install adafruit-Occidentalis…

curl -SLs https://apt.adafruit.com/bootstrap | bash

Now try ip addr and you shoud have connected to the AP. If not try rebooting.

Once connected, you are ready with a letting go off, the hdmi screen, keyboard and LAN cable. Now onward you can connect to your RasPi over ssh or using WebIDE….

Congratulations, your device is now a Headless Device 😀

Portable DevOps

Install the excellent Adafruit WebIDE using instructions from https://learn.adafruit.com/webide/installation Or in short download using the
following command

curl https://raw.githubusercontent.com/adafruit/Adafruit-WebIDE/alpha/scripts/install.sh
 | sudo sh

Then access the WebIDE by opening the IP address of the RasPi from chrome on mobile.

[Updated 18th May2017]

Another good WebIDE is the Codiad….And the URL has a very detailed a nd self-explanatory install procedure…

https://www.techcoil.com/blog/how-i-setup-codiad-web-ide-on-my-raspberry-pi-3/

I find Codiad to be better than AdaFruit WebIDE… Check it out…

SSH client on mobile

WebIDE is fine but if you want a quick and dirty SSH connection, then try TerMux client or Terminal IDE on android. Its small and extremely fast.

I hope this helps in your long travels also to be more fun and productive…Bon Voyage!

RQ4D – RQLite for Docker

Its been some time since I last blogged  and in the meantime I’ve been experimenting on the various possibilities / patterns that have been in the realm of fantasy a few years back but now are available to every developer who is interested to put in some effort… On of these is my quest to find a Database that can be maintained in ‘Cattle-Mode’ which is in stark contrast to how enterprises manage Database …’Pet-Mode’

What I was looking was a Distributed Database solution that can be configured by ‘developers’ directly, moved to production environment easily and scaled up -AND- down like other cattleclass application like web servers. This obviously is a strange ask as Database servers have ‘content’ and ‘State’ relevent to the ‘session’ with the user and expecting them to move around and scale up/down like any stateless component is ridiculous….but then we are living in ridiculous times and having such expectations is nothing wrong as long as the limitations are clearly understood and not used for use-cases where they are not applicable….

I did check out Apache Cassandra and Crate.io but wasnt very impressed as they still are either too ‘configuration’ intensive or too heavy…as in my book ‘small is big’ and ‘less configuration means more reliability’…This is where I encountered RQLite which is an open source project by Philip O’Toole. What I found Interesting is that he has managed to use SQLIte which is a tiny (<1mb) SQL Embedded Database, Added Distribution Logic using the RAFT Algorithm and voila , we have a Distributed Database that is less than 15MB per Instance! The Executable works happily in a scratch based image container, reads and writes are very fast, but hits problems when you try to scale in Docker…It needs to be treated like cattle class as everything is based on IP Address and an overlay Intelligence for the Instance to know if it is the Seed (First  Instance) or the Follower is not available…Service Discovery on Docker Network further makes things impossible and Non-Determinate….

So Here is what I did … I wrote a quick wrapper (https://github.com/julianfrank/rq4d) that takes care of this intelligence. Now the developer can Spin up the Container and scale and simply build the application that connects using the docker provided service discovery. All this for a few (<5MB) of code extra…

This setup treats the Database like Cattle …Obviously this is not the right solution for critical Data but Volatile Data like Persistent Caching can use this very well without worrying about distribution Logic available for free…Enjoy!