3 years in and processing 10 million messages: a retrospective

Our company is built around and dedicated to enabling customers to access data they haven’t previously been able to using sensors.  Three years into this journey finds us now processing over 10 million sensor messages per day from more than 100 companies in Europe and North America!  

What have we learnt so far?
– Multi disciplinary teams; In IoT we believe the strength of a team is in building experitise in project management, hardware, networks, software, data and open APIs.   

– Minimising sensor installation headaches are key; Our latest sensor deployment in London last week saw us reduce installation time to only 20 seconds per sensor.  We believe sensors should be easy to place and simple to move which is why we also have training schemes available to our clients so that they can manage the physical repositioning of their own sensors post deployment should they choose to. Flexibility and control are as important to us as they are to you.

We are always investigating and building partnerships to find the best solution for our clients and this has focused us on standardising around open network protocols such as LoRa.  We now have a large number of happy customers using the following types of LoRa sensors.

  • Office utilisation monitoring: desks, meeting rooms and breakout areas
  • Air quality
  • Environmental sensing
  • Parking sensors
  • Footfall monitors
  • Asset management sensors such as vibration, temperature monitoring

Recent clients include; Zaha Hadid Architects, 360 Workgroup, Fourfront group, TripAdvisor and many others.

As much as possible we have tried to share tangible learnings and experience reports with the wider community. Some examples of our publications and webcasts that have proven hugely popular include:

  • An IoT university – with over 1,500 registered students

https://university.opensensors.io/university/

  • Webcasts with Multi Tech about LoRa and the pros and cons

https://www.youtube.com/watch?v=M1jMheYnRq4&t=326s

Thank you to all our customers, partners, advisors and the wider IoT community.  We look forward to the next 3 years of learning, growing and being part of this incredible ecosystem.

Lessons Learned From First Generation IoT Installations

At first glance, Wi-Fi-based sensors seems like a good choice for a non consumer facing sensor network, however, we have discovered that Wi-Fi has some significant drawbacks.

Access

One of the biggest drawbacks to Wi-Fi enabled sensors in a corporate environment at many of the companies is gaining access. Corporate IT often has valid security concerns of hundreds if not thousands of sensors joining the network and have deployed corporate firewalls that block any access. Often this means that we are not allowed to spin up our own Wi-Fi network in order to have a gateway for a customer’s IoT sensor network. If IT has already deployed a Wi-Fi network they are rarely willing to provide the passwords to allow the IoT network devices and gateways to take advantage of it. Relying on corporate Wi-Fi can make on-site installations and maintenance extremely complex and painful. The whole project becomes dependent on the goodwill of a network administrator for access every time maintenance needs to be performed.

Power

Wi-Fi has good transmission range but that comes with a cost of high power usage. With a short battery life, maintenance costs for Wi-Fi sensors are higher than low-power alternatives. One wireless protocol that is we see in many successful deployments is LoRa because it offers long transmission range at a much lower battery usage than Wi-Fi.

Moving to LoRa and other long range protocols

If you follow our blog and publications, you will notice we have been talking a lot about network technologies, this isn’t a coincidence. We have spent a long time evaluating and piloting these stacks with our community.

Network access and battery constraints are driving the move to long range networks and off WiFi for many IoT installations. LoRa is working well for us so far for a number of use cases most of our customer spin up a private network. The ecosystem of providers is maturing and we are finding a lot of companies who are adopting existing sensors for their networks Gateway providers such as Multi Tech provide good support for the long tail of small scale (> 250 sensor installs) hardware providers to thrive.

LoRa is a wireless protocol that promises between two and five kilometers transmission range between sensors and gateway, if you haven’t already done so please read our introduction to what it is. With a separate LoRa network, facilities and/or operations can install and manage the whole operation without the access and security issues of using the corporate Wi-Fi network. A typical network will have hundreds of sensor devices sending messages to a gateway. The LoRa gateway is a self contained system, we can have the LoRa network sit completely outside of the corporate firewall (GSM) and minimize IT security concerns.

One LoRA gateway can normally cover an entire real estate. This can significantly reduce infrastructure, deployment, and administration costs compared to other shorter range wireless options like Zigbee or Bluetooth that requires complex installs. Our aim is to have a system that non technical engineers can roll out and support, more on how to do this on later blog posts, but in most cases the OpenSensors team is the equivalent of ‘2nd line support’ to the onsite team who have intergrated our apis to their helpdesk ticketing systems etc.

LoRa networks can be public or private. An example of a public network is The Things Network, we continue to work with and support that community. Most current commercial projects are running private networks at this time but will be interesting to see how that evolves over time.

To conclude, LoRa is working well for us at the moment but we will keep researching other networks to enable us to understand the pros and cons of all the network providers. Sigfox is a very interesting offering that we will properly test over the next few months, for example.

Getting to Grips With IoT Network Technologies

How sensors communicate with the internet is a fundamental consideration when conceiving of a connected project. There are many different ways to connect your sensors to the web, but how to know which are best for your project?

Having just spent the better part of a week researching these new network technologies, this brief guide outlines the key aspects to focus on for an optimal IoT deployment:

Advanced radio technology

  • Deep indoor performance – networks utilising sub-GHz ISM (industrial-scientific-medical) frequency bands such as LoRaWAN, NWave and Sigfox are able to penetrate the core of large structures and even subsurface deployments.
  • Location aware networking – a variety of networks are able to track remote sensors even without the use of embedded GPS modules. Supporting sensors moving between hubs – with advanced handoff procedures and innovative network topologies mobile sensors can move around an area and remain in contact with core infrastructure without disrupting data transmission. Intelligent node handoff is also crucial for reducing packet loss, if the connection to one hub is hampered by passing through particularly chatty radiowaves, the node can switch to a better placed hub to relay it’s crucial payload.
  • Interference resistance – the capability of a network to cleave through radio traffic and interference that would ordinarily risk data loss.

Low energy profiling

  • Device modes – LoRaWAN is a great case and point with three classes of edge node: the first, Class A, allows a brief downlink window after each uplink upload i.e after having sent a message, the sensor listens in for new instructions; a Class A node appoints a scheduled downlink slot, the device checks in at a certain point; and the last, Class C type nodes, listen for downlink messages from LoRaWAN hubs at all times. The latter burns considerably more power.
  • Asynchronous communication – this enables sensors to communicate data in dribs and drabs where possible, services do not need to wait for eachother thereby reducing power consumption.
  • Adaptive data rates (ADR) – depending on the quality of signal and attenuation, modern networks are able to dynamically allocate data rate depending on interference, distance to hub etc. This delivers real scalability benefits, frees up space on the radio spectrum (spectrum optimisation) and improves overall network reliability.

security

  • Authentication – maintains data integrity by ensuring the sensor which is publishing that mission critical data really is that sensor and not an impostor node. Ensures information privacy.
  • End to end encryption (E2E) – prevents tampering and maintains system integrity.
  • Integrated security – good network security avoids potential breaches and doesn’t place the onus on costly, heavily encrypted message payloads.
  • Secure management of security keys – either written remotely on the initial install or embedded at manufacture, security keys are fundamental to system security. ZigBee’s recent security issue shows how not to manage security keys, by sending them unencrypted over-the-air to devices on an initial install.
  • Receipt acknowledgement – ensures mission critical data is confirmed received by network or device.

Advanced network design

  • Full bidirectional comms – enables over the air (OTA) updates, enabling operators to push new firmware or system updates to thousands of remotely deployed sparse sensors at the push of a button. This is critical to a dynamic and responsive network. As with device modes mentioned previously, bidirectionality allows deployed devices to function as actuators and take action (close a gate, set off a fire alarm etc) rather than just one-way sensors publishing to a server.
  • Embedded scalability and consistent QoS – as load increases on a network so too does the capacity of the network. This takes the form of adaptive data rates, prevention of packet loss by interference and channel-blocking, the ability to deploy over-the-air updates and ensuring the capability to add nodes, hubs and maintain existing assets without impacting on overall network service, perhaps through automatic adaptation.

There are also a number of legal, cost, market and power focused aspects worth considering that I shall not cover here. But, critically, it’s worth mentioning that the majority of these technologies operate on ISM (industrial – scientific – medical) frequency bands, and as a result are unlicensed. These bands are regulated and there are rules, however anyone operating on these bands can do so without purchasing a license. Notably, you don’t have sole ownership of a slice of the spectrum, you don’t get exclusive access. Therefore, with a variety of other vendors blasting away across the radio waves, these technologies encounter significantly more interference than the licensed spectrum. However, the new networks, LoRa, Sigfox, NWave etc are based on protocols and technologies designed to better sort through this noisy environment, grab a channel and send a message.

Understanding that the airwaves are a chaotic mess underlines the importance placed on features such as adaptive data rates, node handoff and power saving methods such as asynchronous communication. Wired networks do not have to consider such things. But for most it’s not just a case of who shouts loudest wins. The majority of wireless protocols ‘play nice’ opting for a polite “listen, then talk” approach, waiting for a free slot in the airwaves before sending their message.

Some protocols such as Sigfox forego such niceties and adopt a shout loud, shout longer approach, broadcasting without listening. A typical LoRaWAN payload takes a fraction of a second to transmit, Sigfox by comparison sends messages 3-4 seconds in length. However, if you just broadcast without listening, Sigfox must therefore operate with severe cycle duty limitations, which translate into a limited number of messages sent per device per day and severe data rate limitations.

These choices also translate into varying costs, and critically, into battery life limitations and gains, the crux of any remote deployment.

See this link for a matrix of the major technologies currently vying for network domination.

Monitoring for Earthquakes With Node-red

OpenSensors now capture seismic data from the Euro-Med Seismic Centre (EMSC) and the United States Geological Survey (USGS). Every ten minutes we are polling the latest information of major and minor earthquakes around the globe and make this information available via our programming interface (API) or as MQTT feed. In this short tutorial, we’re showing you how to use OpenSensors together with Node-RED to receive email alerts whenever there’s a major incident in a region of interest. You can use this guide as starting point for further experiments with Node-RED and develop your own earthquake-triggered workflows. Let’s shake it.

On OpenSensors

  • First, you need to login to your account on OpenSensor or sign up for one if you haven’t done so already at https://OpenSensors.io.
  • Next, it’s good practice to have a new ‘device’ for this application, i.e. a dedicated set of credentials you’re going to use to log in to OpenSensors for this particular set of MQTT feeds.
    • In the panel on the left, click My Devices in the Devices menu.
    • Click the yellow Create New Device button at the top of the page.
    • Optional: Add some optional descriptions and press the disk icon to save your new device.
    • Take a note of your ‘Client id’ and ‘Password’ as you’re going to need them in your Node-RED workflow.

For Node-RED

Install node.js and Node-RED on your system. There’s a very good guide for this on the Node-RED website. Follow the instructions, including the separate section on Running Node-RED.

Once you’re ready, open a web browser and direct it to localhost:1880, the default address and port of the Node-RED GUI on your system.

(A very basic description of the Node-RED vocabulary can also be found at SlideShare.)

Developing a workflow

  • From the input panel of your nodes library on the left side, drag and drop a pink mqtt input node into the work area named Sheet 1.
  • Double-click the mqtt node. A window with configuration details opens.
    • Click the pen symbol next to ‘Add new mqtt-broker…’. Your Broker is opensensors.io, your Client ID and Password those you generated in the previous step on the OpenSensors website, and User is your OpenSensor user name.
  • Once the Broker is defined, enter /orgs/EMSC/+ into the Topic field. This is going to instruct Node-RED to subscribe to all MQTT topics generated by the EMSC.
  • Optional: Set the Name of this node to ‘EMSC’.
  • Drag and drop a second mqtt input node. When you double-click the node, you will realise that the Broker settings default to the ones you previously entered.
    • Enter /orgs/USGS/+ in the Topics field and ‘USGS’ as optional Name.
  • Drag and drop a dark green debug node from the output panel on the left. While debugging has the connotation of fixing a problem, in Node-RED it’s the default way of directly communicating messages to the user.
  • Draw connection lines (“pipes”) from both mqtt nodes to the debug node.
  • Press the red Deploy button in the upper right corner. This starts your Node-RED flow. If everything worked, you should see ‘connected’ underneath the mqtt nodes and your debug panel (on the right) should soon produce the following JSON-formatted output if there’s an event (which may take a while!):

While it is pleasing to be informed about every time the earth shakes, it soon becomes tedious staring at the debug panel in expectation of an earthquake. Also, you may not be interested in events in remote areas of the world, or exactly in those – whatever interests you.

We are going to extend our flow with some decision making:

First, we need to parse the information from the EMSC and USGS. For this example, we’re going to be particularly interested in the fields region and magnitude. There are plenty more fields in their records, and you may want to adjust this flow to your needs.

  • Drag and drop a pale orange function node from the functions panel into your flow. Connect both mqtt nodes to the input side (the left side) of your function node. Function nodes allow you directly interact with your data using JavaScript.
  • Enter the following code (or download the OpenSensors workflow).

Here be a JavaScript course… :–) In a nutshell, this code takes data from the ‘payload’ of the incoming message (read up on the topic and payload concept of Node-RED in the SlideShare article suggested earlier). The payload is then parsed for the region and magnitude fields using standard regular expressions. If we can successfully extract information (in this case: the region containing ‘ia’ somewhere in it’s name), we’re going to set the outgoing message’s payload to the magnitude, its topic to ‘EVENT in ‘ plus the name of the region and pass it on (‘return msg’) to the next node.

  • Drag and drop a lime green switch node from the function panel into your workflow. Connect the output of the function node to the input of the switch node. Configure (by double-clicking) the switch node to assert if the payload (being the magnitude of the earthquake) is greater than 2. Only then the message is going to be passed on
  • Last, we’re going to drag and drop a light green e-mail output node from the social panel and configure it like an e-mail client, but with a default recipient: here in this case, ohmygodithappend@gmail.com.
  • Connect the output of the switch node to our debug node, as well as to the outgoing e-mail node.
  • We can then deploy the new workflow and should see something like this after a while:

In this case, an event was detected ‘off the coast of Northern California’ with a magnitude of 4.4 and at the same time, you should receive an e-mail with the region as subject and the magnitude in the body of the e-mail.

We hope that this flow is getting you started! Remember that Node-RED is superbly suited to interact with hardware… …imagine LEDs and buzzers indicating an earthquake.

The flow JSON: [{“id”:“e9024ae0.16fdb8”,“type”:“mqtt-broker”,“broker”:“opensensors.io”,“port”:“1883”,“clientid”:“1646”},{“id”:“2952b879.d6ad48”,“type”:“mqtt in”,“name”:“EMSC”,“topic”:“/orgs/EMSC/+”,“broker”:“e9024ae0.16fdb8”,“x”:127,“y”:104,“z”:“82a1c632.7d5e38”,“wires”:[[“490a140f.b6f5ec”,“163677af.e9c988”]]},{“id”:“54239d6.fabdc64”,“type”:“mqtt in”,“name”:“USGS”,“topic”:“/orgs/USGS/+”,“broker”:“e9024ae0.16fdb8”,“x”:128,“y”:159,“z”:“82a1c632.7d5e38”,“wires”:[[“490a140f.b6f5ec”,“163677af.e9c988”]]},{“id”:“490a140f.b6f5ec”,“type”:“debug”,“name”:“”,“active”:true,“console”:“false”,“complete”:“false”,“x”:538,“y”:86,“z”:“82a1c632.7d5e38”,“wires”:[]},{“id”:“163677af.e9c988”,“type”:“function”,“name”:“parse”,“func”:“// uppercase the payload (different centres report in mixed formats)nmsg.payload = msg.payload.toUpperCase();nn// extracting interesting fields with regular expressions,n// instead of using JSON.parse which fails with null fieldsnvar places_with_ia_regex = new RegExp(“REGION”:“(.IA.)”,“UPDATED”);nvar result1 = places_with_ia_regex.exec(msg.payload);nnvar magnitude_regex = new RegExp(“MAGNITUDE”:([0-9].[0-9]+)“);nvar result2 = magnitude_regex.exec(msg.payload);nn// if successful, sets topic to the region and payload to the magnitudenif (result1 && result2) {n msg.topic = ‘EVENT in ’+result1[1];n msg.payload = result2[1];n return msg;n}”,“outputs”:1,“noerr”:0,“x”:296,“y”:251,“z”:“82a1c632.7d5e38”,“wires”:[[“64f4f2ea.9b0b0c”]]},{“id”:“64f4f2ea.9b0b0c”,“type”:“switch”,“name”:“at least magnitude 2”,“property”:“payload”,“rules”:[{“t”:“gte”,“v”:“2”}],“checkall”:“true”,“outputs”:1,“x”:428,“y”:179,“z”:“82a1c632.7d5e38”,“wires”:[[“490a140f.b6f5ec”,“f7bcc59c.084338”]]},{“id”:“f7bcc59c.084338”,“type”:“e-mail”,“server”:“smtp.gmail.com”,“port”:“465”,“name”:“ohmygodithappened@gmail.com”,“dname”:“”,“x”:581,“y”:256,“z”:“82a1c632.7d5e38”,“wires”:[]}]