Ubuntu and USB Serial Devices

So I had to reboot the Linux server today, and I ran into the problem of it randomly assigning /dev/ttyUSB? devices to my two USB serial cables. With randomly assigned devices my CAN and SMS interface serial port names happened to get reversed – grrr.

So the task is to get Linux to assign the same name when rebooting.

First find out the device serial numbers, use this command and serach for the devices. Luckily in my situation the two cables I have are both FTDI cables with different serial numbers. Here is one

Next is to create or edit /etc/udev/rules.d/50-serial-devices.rules, and add a line for each device using the serial number, and in this case create a symlink to ttynasco0/1.

Restart udev, (reload?,  and trigger?) and magically the new files /dev/ttynasco0 and /dev/ttynasco1 will get created and assigned to whatever /dev/ttyUSB? file the kernel assigns.

Lastly I changed my config.ini file to point to the new nasco tty files.

Done!

Update 26/12/2014 – I replaced a cable and needed to update the configuration – It didn’t seem to take effect until I rebooted the machine for some reason.

 

Ubuntu Services

Now that the CAN and SMS MQTT gateways are working on the proper Linux server I figured it would be better to run them as services instead of having to log in and run them both manually.

In the end it was pretty simple (yay for Google). Here is the script template I have used, modified for each app with proper description, paths and user/group settings.

I think it will run the app, if it does crash then only try to restart it after 60 seconds.

So far so good.

Next up is to start decoding the other CAN messages (calls and cancels) and something like a messaging/allocation app that can use the call/cancel messages and push message out to SMS.

Example Use Case

  • Door bell gets pressed
  • CAN app takes the CAN message and puts it into /messages/can/incoming
  • Allocation app takes that message and figures that a message should be sent as an SMS, then SMS message put into /messages/sms/outgoing
  • SMS app takes that message and sends the message

 

MQTT CAN Gateway Issues

I have been developing my code on in a virtual machine on my development box. But the intention was to deploy the apps to the same Linux server MQTT is running on.

To this end I checked all the code into SVN and checked it out on the other machine. After the pain of remembering all the libraries to install again, and discovering the pain of hard coded paths and command line builds of Eclipse C++ projects – I finally got it running – yay!

Issues

After running the app for a little while it would get into a state where I wasn’t publishing any more messages. I added some extra debug logging which indicated that my publishing flag was not getting set back to false. The callback is just a logging a message and setting the variable to false – cant be that. So some changes …

  • I changed the code to make sure I only attempted to publish one message at a time – no idea if you can have more than one outstanding at any given time, but to simplify the issues just have one – nope still problems.
  • I changed the code again to only attempt publishing messages on the main thread – I was publishing the next message using the callback thread, maybe the library didn’t like this tactic? Nope still problems.
  • I changed the code again to use an atomic<bool> thinking inter-thread issues – nope, still problems.

<hair pulling> – Arrg – It can’t be that hard – the code is really simple!

Possible Problem

The problem appears to thread related, and in particular the order in which I set the publishing flag in the main thread, a slight change and all seems to work again.

The code below shows the line I moved.

You can see that, as it was, if the callback function was called before the main thread had a chance to set the publishing flag the sequence would be

main thread(false)->callback(false)->main thread(true)

instead of my intended sequence

main thread(false)->main thread(true)->callback(false).

 

 

MQTT SMS Gateway Part 2

I cleaned up the code, mostly re-factoring the code for INI file configuration by removing the boost program_options version and substituting my own. Works out a lot simpler. Here is the relevant INI file sections –

Time for testing – with the SIM card installed this time. Here is the new log, with a few added bits for debugging – name of the serial port, test mode status, etc.

I sent two messages out to different numbers and all is working as expected.

Second Test Run

Next up – I only have one SIM card so I need to figure out a way for the alarm panel interface to publish MQTT messages.

MQTT SMS Gateway

A while back I bought one of these quad band GPRS modems for a job I was working on. The task was simple enough – connect to a data source, wait for outgoing messages, send them out via SMS, report back with success/fail.

The software was installed and “presumably” all is working as expected – as is the usual case you don’t typically hear any feedback if it works, only if it doesn’t work.

Anyway I have this module gathering dust at home and thought I’d write an MQTT to SMS gateway that can run on my Linux server. Once its up and running any other MQTT application could use it to send SMS notifications very simply.

Easy mode would have been to just use the C# code I already had modify to source data from MQTT and run it via Mono on the Linux box. I could have used the Java version in a similar fashion. However the final plan was to run the application on a Raspberry Pi and having the Mono/Java run times really kills you from a memory usage perspective.

I still have plenty of spare time on my hands so I decided to port the application to C++ and make use of the boost libraries. In theory I can compile it to a single executable (+ the shared libraries I guess) and all will be good when running a dozen or so of these little applications.

All GSM Modules are not the same

There are so many of these things around these days. The standard one the Arduino community seems to use is is based on the SIM900 modules. We used one of these in our Alarm Panel device.

It’s really easy to send SMS messages as the library code will do it all for you. The AT commands are simple enough if you want to do it yourself though, as the SIM900 supports “text mode”, so something similar to this.

The module I have contains a HUAWEI MG323B module. Unfortunately it does not support the same “text mode” to send SMS messages, instead you need to use “PDU mode”, something more like this.

Ugly! There are many useful sites on the interwebs that explain PDU mode in great detail. I found a simple library to do this and ported it to C#, Java and now C++ (cant remember the original source or implementation language at the moment – bad).

Boost me up

I haven’t used or looked at the boost libraries for a (very) long time. But a lot of the things you might want to do are pretty much covered so why reinvent. In particular I needed threads, logging, locking schemes, configuration files and serial ports to name a few.

How did it work out? Well mostly fine – once you get going with how each of the libraries work you can get some nice things done quite quickly.

The good – The threads, logging, synchronization schemes and serial port code was reasonably straight forward.

The not so good – configuration files. Here I wanted to reuse the same INI files my windows applications use. A single INI file contains an assortment of settings for system wide settings and for specific applications and components used within these applications.

I might port my configuration file code to C++ just to maintain the simplicity – the boost method just seemed too complex for my liking.

Initial Testing

Here is the log of the application running. Basic sequence is start up, read configuration file, create and start the MQTT interface and F1103 interface, connect everything all up with an event queue. MQTT subscribes to the topic, while F1103 starts sending AT commands to get the GPRS device into the correct mode/state.

NOTE: The AT commands are echoed back so the first RX line is actually what I sent.

Later I publish an MQTT message to the topic via the command line.

F1103 tries to initiate the SMS send but fails – thee error indicates that the SIM card is not present.

Next up – Add SIM card and test again.

Test Run

Wireless Problems vs Wired Alternatives

Wireless Problems

Anyone looking at my sensor data will notice that there are a lot of gaps where data has not been recorded. It’s was hard to know exactly what the issue is as the Arduino is running headless. If I see the data has stopped I just rebooted the Arduino by unplugging the USB and plugging it back it. Yes it fixes the problem but it is not a real solution.

I ran it locally the other day and noticed that it was stuck in a loop thinking it had received a new message, but of invalid type, over and over again. I didn’t catch the moment of transition from normal operation to error, maybe another logged test next time. Anyways it was just printing this

All the type numbers seemed random. Here is the code snippet. Nothing looks like it should cause a a problem. And the problem occurs at random times after reboot. Maybe a problem with the nRF library perhaps.

Update – I got tired of trying to find the problem so I changed the code to force a software reset when the unknown type message was detected.

 Wired Alternatives

In a previous post I went through all the iterations of a CAN Input Module design. The latest design includes 8 analog input channels. The inputs can be left floating or internally biased with 10K resistors to GND or VCC.

Input Schematic

That’s handy because if I use the internal 10K pull up I get the same circuit I use in the Arduino sensors. Time to test that theory. I’m using the first three inputs, set the channel IDs (1, 2, 3), set them to analog mode, and added a delay of 10 seconds.

Input Module Programming

With this setup (and the current firmware) I get a new analog reading for each channel at most once every 10 seconds – if the reading has actually changed. The delay is really to stop a deluge of data. The PIC has a 12 bit A2D so the readings range from 0 to 4095. The firmware does 32 sample averaging.

The device programmer software has a logging mode so I have extended the logging to print the analog reading and convert it to a temperature. Here are the logging results – slightly verbose but you get the idea. The sensors are all withing a few centimeters of each other and the values returned seem to be within spec for components I’m using.

  • The 10K bias resistors are probably +/- 5%
  • Channel 1 is a small NTC resistor from Element 14.
  • Channel 2 & 3 are cheapy Aliexpress sensors.

Logging Results

Those paying attention will note that my anticipated 10 second delay seems to evaluate to only 8 seconds … oops … the RTC timer based code in the PIC is probably running a tad too fast.

I have had this running for the last week, logging away, all is good.

CAN/MQTT Gateway

Next step is to write a CAN/MQTT gateway application. My CAN bus is accessed via a CAN to USB device so it should be a simple matter of writing some code to open a connection to the CAN bus, wait for incoming CAN messages and publish a message to an MQTT topic, similar to what the wireless sensors do.

This will also serve as a foundation for working in the opposite direction, subscribing to an MQTT topic, receiving messages and sending them out via CAN.

(more to come)

MQTT and RRDTool

I mentioned in one of my earlier posts that Xively was nice as a quick test setup, but in the end I would like to have my data locally.

To this end I have installed Mosquitto as an MQTT server on a spare Linux machine in the office.  I changed the nRF receiver Arduino to send the sensor data to topics in Mosquitto. I used the PubSubClient for the MQTT connection.

The format of the topics are

“/sensors/radio/<device-id>/<sensor-number>/<type>”

device-id is the address of the remote sensor device

sensor-number is the sensor on the device 1, 2, 3 … etc

type is the type of the sensor temperature, light, battery, etc

This is working as expected, but the data is more or less thrown away unless we have something to subscribe to it!

So on the subscription side i wanted something up and running quick, so I found this snippet of perl code that subscribes to a topic extracts data from the message content and updates an RRD database. For the moment it just does the temperature data.

I then have a script that runs every so often to create some graphs from the rrd data.

The results can be seen here.

Alarm Panel – Programming settings

With the interface installed I needed an easier way to program the settings. Having to recompile the Arduino code each time you want to change something is pretty poor.

The V2 firmware supports

  • 16 zones, each with
    • Name (15 characters)
    • Enabled flag
    • Alarm flag
    • Cancel flag
  • 4 phone numbers, each with
    • Number (15 characters)
    • Enabled flag

Nothing too complicated but enough to get the job done. All the settings are stored in the EEPROM of the Arduino (atmega2560).

First thing I wanted was a simple command line interface. Worst case all you need to program the interface would be a terminal (something like Putty, TeraTerm, HyperTerminal, etc).

Being lazy its always easier to see if something already exists before you roll your own. Even if its not exactly what you want you can get some ideas. Lucky for me, with a few Google searches, I found the serial command library.

So simple, install the library, include the header, add a few lines and a few callback functions – done!

So now via the command line we can

  • enable/disable debug mode
  • update zone data
  • update phone data
  • display the current information
  • enter setup mode
  • turn on/of SMS message sending

This is enough really for the prototype – but – if you want to make it a bit easier for the average Joe to use then it’s time to a make a simple UI.

Behold – the alarm interface programming tool. Of course it looks like crap but it gets the job done. Simple operation Read the data, modify the data, and then write the data back.

Under the covers it uses the serial commands we defined earlier to to read and write settings. The hardware version and firmware versions do work but not in this early screen shot.

Program some settings

Cool – simple programming tools done – now to customize for my alarm panel and see if we get some SMS messages out!

Code Generation Tool

Having to write a lot of simple code once can be tedious and error prone – having to write it again for another language is even worse. Having to maintain both is just plain stupid. It would be quite handy to use a tool to generate all this basic code for us.

In my case I want the tool to create my standard code for my home automation software. I started with Java code but wanted to do an application in .NET and I didn’t want to port* or maintain both code bases – especially if the exact specification was still work-in-progress.

* Porting of the underlying base CDEF library was needed of course.

  • plain old data objects
    • fields can be
      • basic types (int, long, string) or other objects
      • lists of basic types or objects
  • messages
    • requests and responses
      • fields can be
        • basic types (int, long, string) or other objects
        • lists of basic types or other objects
      • encoding and decoding of data into byte buffers
  • languages
    • Java
    • .NET

The input to the tool is a form of specification defining all of the details and I chose to use XML to represent this data. Below is a very basic example of a simple class and message.

Running the tool and generating the code gives me the new source files. Note that the CdefMessage class is essentially a byte buffer. There is a constructor that pulls the data out of the message and an encode function to do the reverse.

In Java we get the same thing more or less. Ordering of encoding and decoding each field is important so we can send the objects between applications written in either language.

Here is the .NET Error message code, the Java is much the same. You can see how I handle lists of data objects and how they get encoded and decoded as required.

There are a few other helper classes that get created to make everything work but already we can create all this basic code with very little effort, no mistakes and knowing that both language libraries are in sync. This suits me as I’m very lazy and the less I have think about the better.

Information Displays – Part 2

Time to code … I suspect it wont take too long to get something up and running. For this I will use Visual Studio 2013 Express and code in C#, that way anyone can grab the code from Github and use it for themselves.

Get The Screen Up

LCD screens come in all sorts of resolutions and sizes, so to make this useful we need to able support different configurations. For development and testing we probably want it to run in window mode with smaller fonts, if you have a multi-screen setup maybe you want to to run full screen on your second screen, for installation full screen with larger fonts sounds good, portrait mode maybe?

I chose to have the configuration in an XML file so its relatively easy to edit and parse.

I added the ability to specify the background image at this point. Nothing particularly tricky so far but it loads and re-sizes the form,  and goes full screen on the required screen if needed. 

Adding Some Labels

We could just add Label controls to the Form inside VS2013 but that doesn’t really allow us to change the layout for our different setups. The simplest way to add text to the screen will be to define labels in the XML – It will be slightly painful to do x, y, width and height maths in your head but it will do for now. Hint: Someone could write an WYSIWYG editor I guess or something else?

Initial Label Types

  • static – Just static text – well duh
  • time – Using a timer update the label and print the current time in using the given format.
  • file – Update the label with the contents of a file with configurable refresh rate.

Check out the screen shot so far in window mode. The temperature, light and voltage data is read from files created by another application.

Next up well add code to connect to the server, and display the currently active events in a configurable “list like” view.

 Display Part 1

Configuration Part 1