MQTT SMS Gateway Part 2

I cleaned up the code, mostly re-factoring the code for INI file configuration by removing the boost program_options version and substituting my own. Works out a lot simpler. Here is the relevant INI file sections –

Time for testing – with the SIM card installed this time. Here is the new log, with a few added bits for debugging – name of the serial port, test mode status, etc.

I sent two messages out to different numbers and all is working as expected.

Second Test Run

Next up – I only have one SIM card so I need to figure out a way for the alarm panel interface to publish MQTT messages.

MQTT SMS Gateway

A while back I bought one of these quad band GPRS modems for a job I was working on. The task was simple enough – connect to a data source, wait for outgoing messages, send them out via SMS, report back with success/fail.

The software was installed and “presumably” all is working as expected – as is the usual case you don’t typically hear any feedback if it works, only if it doesn’t work.

Anyway I have this module gathering dust at home and thought I’d write an MQTT to SMS gateway that can run on my Linux server. Once its up and running any other MQTT application could use it to send SMS notifications very simply.

Easy mode would have been to just use the C# code I already had modify to source data from MQTT and run it via Mono on the Linux box. I could have used the Java version in a similar fashion. However the final plan was to run the application on a Raspberry Pi and having the Mono/Java run times really kills you from a memory usage perspective.

I still have plenty of spare time on my hands so I decided to port the application to C++ and make use of the boost libraries. In theory I can compile it to a single executable (+ the shared libraries I guess) and all will be good when running a dozen or so of these little applications.

All GSM Modules are not the same

There are so many of these things around these days. The standard one the Arduino community seems to use is is based on the SIM900 modules. We used one of these in our Alarm Panel device.

It’s really easy to send SMS messages as the library code will do it all for you. The AT commands are simple enough if you want to do it yourself though, as the SIM900 supports “text mode”, so something similar to this.

The module I have contains a HUAWEI MG323B module. Unfortunately it does not support the same “text mode” to send SMS messages, instead you need to use “PDU mode”, something more like this.

Ugly! There are many useful sites on the interwebs that explain PDU mode in great detail. I found a simple library to do this and ported it to C#, Java and now C++ (cant remember the original source or implementation language at the moment – bad).

Boost me up

I haven’t used or looked at the boost libraries for a (very) long time. But a lot of the things you might want to do are pretty much covered so why reinvent. In particular I needed threads, logging, locking schemes, configuration files and serial ports to name a few.

How did it work out? Well mostly fine – once you get going with how each of the libraries work you can get some nice things done quite quickly.

The good – The threads, logging, synchronization schemes and serial port code was reasonably straight forward.

The not so good – configuration files. Here I wanted to reuse the same INI files my windows applications use. A single INI file contains an assortment of settings for system wide settings and for specific applications and components used within these applications.

I might port my configuration file code to C++ just to maintain the simplicity – the boost method just seemed too complex for my liking.

Initial Testing

Here is the log of the application running. Basic sequence is start up, read configuration file, create and start the MQTT interface and F1103 interface, connect everything all up with an event queue. MQTT subscribes to the topic, while F1103 starts sending AT commands to get the GPRS device into the correct mode/state.

NOTE: The AT commands are echoed back so the first RX line is actually what I sent.

Later I publish an MQTT message to the topic via the command line.

F1103 tries to initiate the SMS send but fails – thee error indicates that the SIM card is not present.

Next up – Add SIM card and test again.

Test Run

Wireless Problems vs Wired Alternatives

Wireless Problems

Anyone looking at my sensor data will notice that there are a lot of gaps where data has not been recorded. It’s was hard to know exactly what the issue is as the Arduino is running headless. If I see the data has stopped I just rebooted the Arduino by unplugging the USB and plugging it back it. Yes it fixes the problem but it is not a real solution.

I ran it locally the other day and noticed that it was stuck in a loop thinking it had received a new message, but of invalid type, over and over again. I didn’t catch the moment of transition from normal operation to error, maybe another logged test next time. Anyways it was just printing this

All the type numbers seemed random. Here is the code snippet. Nothing looks like it should cause a a problem. And the problem occurs at random times after reboot. Maybe a problem with the nRF library perhaps.

Update – I got tired of trying to find the problem so I changed the code to force a software reset when the unknown type message was detected.

 Wired Alternatives

In a previous post I went through all the iterations of a CAN Input Module design. The latest design includes 8 analog input channels. The inputs can be left floating or internally biased with 10K resistors to GND or VCC.

Input Schematic

That’s handy because if I use the internal 10K pull up I get the same circuit I use in the Arduino sensors. Time to test that theory. I’m using the first three inputs, set the channel IDs (1, 2, 3), set them to analog mode, and added a delay of 10 seconds.

Input Module Programming

With this setup (and the current firmware) I get a new analog reading for each channel at most once every 10 seconds – if the reading has actually changed. The delay is really to stop a deluge of data. The PIC has a 12 bit A2D so the readings range from 0 to 4095. The firmware does 32 sample averaging.

The device programmer software has a logging mode so I have extended the logging to print the analog reading and convert it to a temperature. Here are the logging results – slightly verbose but you get the idea. The sensors are all withing a few centimeters of each other and the values returned seem to be within spec for components I’m using.

  • The 10K bias resistors are probably +/- 5%
  • Channel 1 is a small NTC resistor from Element 14.
  • Channel 2 & 3 are cheapy Aliexpress sensors.

Logging Results

Those paying attention will note that my anticipated 10 second delay seems to evaluate to only 8 seconds … oops … the RTC timer based code in the PIC is probably running a tad too fast.

I have had this running for the last week, logging away, all is good.

CAN/MQTT Gateway

Next step is to write a CAN/MQTT gateway application. My CAN bus is accessed via a CAN to USB device so it should be a simple matter of writing some code to open a connection to the CAN bus, wait for incoming CAN messages and publish a message to an MQTT topic, similar to what the wireless sensors do.

This will also serve as a foundation for working in the opposite direction, subscribing to an MQTT topic, receiving messages and sending them out via CAN.

(more to come)

MQTT and RRDTool

I mentioned in one of my earlier posts that Xively was nice as a quick test setup, but in the end I would like to have my data locally.

To this end I have installed Mosquitto as an MQTT server on a spare Linux machine in the office.  I changed the nRF receiver Arduino to send the sensor data to topics in Mosquitto. I used the PubSubClient for the MQTT connection.

The format of the topics are

“/sensors/radio/<device-id>/<sensor-number>/<type>”

device-id is the address of the remote sensor device

sensor-number is the sensor on the device 1, 2, 3 … etc

type is the type of the sensor temperature, light, battery, etc

This is working as expected, but the data is more or less thrown away unless we have something to subscribe to it!

So on the subscription side i wanted something up and running quick, so I found this snippet of perl code that subscribes to a topic extracts data from the message content and updates an RRD database. For the moment it just does the temperature data.

I then have a script that runs every so often to create some graphs from the rrd data.

The results can be seen here.

You Can Never Have Enough Sensors

Day 1

What to do today now the lawns weeds have been mowed? The solar powered sensor is back out in the garden and seems to be working and recharging its battery in the sun today. The receiver is pumping the data into Xively nicely. I have these bits laying about …

  • 2 x Arduino TwentyTens
  • 2 x nRF24L01 modules
  • 2 x ProtoShields
  • 2 x 10K NTC sensors
  • 2 x Dallas 18B20 temperature sensors
  • 2 x Light sensors
  • 1 x DHT11 humidity sensor
  • 1 x BMP085 pressure sensor
  • Plenty of time

Parts Please

Yes its pretty obvious – more RF sensors. Might also be handy to do some indoor range testing and a network/mesh of the nRF modules.

Day 2

Today I partially assembled the two prototype boards. So far the LDR and DS18B20 are fully functional, and the nRF module is soldered in place but not yet wired up. Looks neat and tidy.

Partially Assembled

Of course now it is partially assembled its always nice to know if everything is working. So here is some simple test code. Same story and usual, Google for some existing libraries, no need to write your own.

I ended up with these OneWire and Dallas Temperature

And the output (including other debugging not in the code)

Day 3

Another day with progress. Today I wired up the nRF modules and added some more code to send the readings to the receiver. Other than a snafu with one incorrectly wired pin, it all went without other issues.

Here is what the protoshield looks like now. Still pretty neat. I added the 10uF cap to the 3.3V line, cant remember why or if it is really needed, but my receiver has one so I figured it can’t hurt to have it installed. Yes the wires I have used (single cores from security cable) are really too think for anything too elegant but it was all I had laying about.

Radio Installed

Latest is now live on the Xively Stream.

  • 1000 is the external sensor in the garden
  • 07f3 & d1ad are the new internal sensors

Next up, Add the screw terminal and send readings for the remote NTC temperature sensor …

RF Sensor Ethernet receiver

Feeling inspired by writing up the last post I decided it cant be that hard to convert my USB receiver into an Ethernet based one.

My USB implementation was using an old Freetronics Twenty Ten (the Eleven seems to be the latest equivalent) with one of the nRF modules soldered onto a ProtoShield Short.

I didn’t want add more shields as things start to look ugly so I happened to have an EtherTen in my Arduino parts box so in theory its just swap the shield to the new base board, change the code to add the ethernet connection and all done! That’s the theory anyway.

Like all good theories it wasn’t that simple. The pins I happened to choose for the nRF module conflicted with the pins used for the Wiznet chip for Ethernet connectivity. No matter, I rewired the offending pins and changed the code to suit – fixed.

So the firmware connects to network and the sensor readings are being received, all working great – but where to send the data to?

Xively

While I’d eventually like to have the data stored and processed locally, Xively seems like a choice to get things up and running with little fuss. I can let the receiver run 24×7 for testing while I sort out something more permanent locally – or if Xively does all I need that it can stay there – meh at this stage who knows.

Anyway, the developer section of the Xively site is pretty simple, find and download the Arduino library from git add it the the Arduino libraries folder and your away.

Total integration time was less that one hour including some optimizations.

Here is the public link for the prototype.

  • 1000.01 Temperature Sensor 1
  • 1000.02 Temperature Sensor 2
  • 1000.03 Light Sensor
  • 1000.08 Battery Voltage

The format is of the channel is XXXX.YY, where XXXX is the transmitters serial number, and YY is the sensor input number on that particular device.

Xively Test Data RF Ethernet Receiver

 

You can see the setup is nice and compact but I don’t like that I cant use the POE adapter board and a shield on the EtherTen at the same time so I still need to power the device from USB or the DC jack.

The nRF module is pretty much sitting over the shielded RJ45 connector which is probably acting as a ground plane – so not ideal.

Anyway for now I have a handy 12VDC source readily available so I’ll hook up the DC jack. Then I don’t need to have it connected to my desktop PC all the time – except for debugging.

Update: Things have gone fine expect for an outage at exactly 9 PM one night. Not sure why – but the timing seems to perfect, perhaps the DHCP lease expired?

I changed the code to renew the DHCP lease and left the debugging output running to hopefully catch something. Could have been a random power issue resetting the Wiznet chip – who knows.

 

Whats The Temperature?

What is the temperature outside now? In the pool? Inside the garden bed? How much light did we get today?

To answer these marginally useful questions the obvious solution is to build a solar powered, RF transmitting sensor – paired up with an RF receiver to act as an interface to the computer.

First thing some basic requirements

  • Multiple temperature sensors
  • Simple light sensor
  • RF transmitter/receiver pair or transceivers
  • Solar powered – implying battery powered
  • Able to run 24×7
  • Weather proof/resistent
Sourcing the Parts

Again better not reinvent the wheel so after a few Google searches and numerous distractions I found the Seeeduino Stalker.

Seeeduino Stalker

A nice starting point to get the main things I needed. In particular the Arduino board with solar input, lipo battery charger circuits, a cheap solar panel, a lipo battery and importantly a waterproof case ready to go.

I got a Bluetooth XBee module for quick testing as I already had a USB Bluetooth dongle so I could get something up and running pretty quickly without needed to prototype the receiver.

Now some cheap temperature sensors – Aliexpress has many options here. I went with some cheap 10K NTC resistor based items. I didn’t hold much hope about sticking these particular ones in the pool (salt water / chlorine) on inside the garden bed (moisture in general) but its a start and at the price  they were cheap enough to be disposable.

The light sensor was just an LDR I already had in my parts draws.

Cheap 10K NTC Sensor

 Initial Setup and Results

I put some quick firmware together and as it was summer time with plenty of sun around I stuck with using Bluetooth as the communications put it all together and installed in by the pool with one temperature sensor dangling in the pool and the other just in free air.

Setup 1

 

Unfortunately I don’t have an image of the sensor that was in the pool – it lasted maybe a month before the temperature readings starting dropping – it appeared that the potting compound wasn’t really adhered to the wires and eventually the salt / chlorine / heat got into it – not unexpected.

The Bluetooth was marginal from the pool to the office – maybe 17 meters away and through a few walls and metal window shutters – not unexpected.

The battery recharging worked quite well – it was full Adelaide summer after all, weeks of 35+ degree days with no clouds. The graphs of battery voltage did indicate that the battery was pretty much marginal as is each night it almost expired only to be saved by the sun the next day.

Happy with the initial results none the less as I hadn’t done any power optimization at all.

 Autumn Radio Upgrades

The initial prototype was retired after summer as the battery drain from the Arduino and Bluetooth modules running 24×7 was just too much. Plus the pool isn’t heated and there was little interest in the water temperature – you already know its cold. Naturally it was time for an upgrade.

I have no idea how much power the Bluetooth module required as I didn’t measure it properly but my assumption of power usage and the range being quite poor I decided to upgrade the to some RF modules. An obvious and cheap choice was one of the numerous nRF24L01 modules you can source just about anywhere these days. $10 for 10 transceivers is very hard to beat so its worth a try.

nRF24L01

There are plenty of Arduino libraries out there for these modules so changing the code over was also pretty straight forward. I also needed to make a receiver in the office to get the data sent from the sensor.

Here is the sensor at the bottom and receiver at the top as it is now.

The winter kit

It was now autumn and there was still enough sun during the day to keep things running overnight even though the Arduino was still running 24×7 at full speed (8Mhz). At this point I was already powering down the nRF module in between transmissions to save power.

Winter Firmware Upgrades

Winter comes along and the amount of sun during the day – assuming there is any at all – is just way too short. So time to finally get the firmware to do proper power management.

Lucky for me the Stalker board had an on board RTC. The firmware basically uses the RTC to set an interrupt/alarm every minute, wakes up, takes the readings, sends the message, and goes back to sleep.

The results were much better – lasting over a week with little sun and the panel flat of the ground – less than ideal. Here is the graph for the last week.

Weekly data

Something Still Obviously Wrong

You will notice the graph is quite sparsely populated – to say the least. The sensor is pretty much outside my office window about 4 meters in the garden bed. The receiver is on my desk. At night the metal window shutters get closed and the sensor readings tend to disappear – not unexpected but annoying.

I think this is a problem with using the nRF modules, at least these ones with the PCB antenna. Some problems I have seen

  • The angle at which I have the transmitter and receiver matters. Flat seems worse than upright.
  • The range I get from the modules, while good, just isn’t good enough to reliably have communications from outside to inside, let alone from one side of the house to the other.
Future Upgrades

For now it will suffice but some future upgrades will include …

  • Using a receiver with the nRF module that has the built in amplifier and normal antenna.
  • Mounting the receiver on an outside wall to eliminate the metal window shutter problem.
  • Using an POE powered + Ethernet enabled Arduino for the receiver and upload the data to a server rather than having the receiver connected to my desktop PC.
  • Creating a few more sensors for around the yard
  • Finding a solution for a pool water temperature sensor.
  • Having the data available and graphed on a website so I can check whats happening from anywhere.
  • Add a soil moisture sensor
  • Add a humidity sensor
  • Integration with my watering system

 

Alarm Panel – Programming settings

With the interface installed I needed an easier way to program the settings. Having to recompile the Arduino code each time you want to change something is pretty poor.

The V2 firmware supports

  • 16 zones, each with
    • Name (15 characters)
    • Enabled flag
    • Alarm flag
    • Cancel flag
  • 4 phone numbers, each with
    • Number (15 characters)
    • Enabled flag

Nothing too complicated but enough to get the job done. All the settings are stored in the EEPROM of the Arduino (atmega2560).

First thing I wanted was a simple command line interface. Worst case all you need to program the interface would be a terminal (something like Putty, TeraTerm, HyperTerminal, etc).

Being lazy its always easier to see if something already exists before you roll your own. Even if its not exactly what you want you can get some ideas. Lucky for me, with a few Google searches, I found the serial command library.

So simple, install the library, include the header, add a few lines and a few callback functions – done!

So now via the command line we can

  • enable/disable debug mode
  • update zone data
  • update phone data
  • display the current information
  • enter setup mode
  • turn on/of SMS message sending

This is enough really for the prototype – but – if you want to make it a bit easier for the average Joe to use then it’s time to a make a simple UI.

Behold – the alarm interface programming tool. Of course it looks like crap but it gets the job done. Simple operation Read the data, modify the data, and then write the data back.

Under the covers it uses the serial commands we defined earlier to to read and write settings. The hardware version and firmware versions do work but not in this early screen shot.

Program some settings

Cool – simple programming tools done – now to customize for my alarm panel and see if we get some SMS messages out!