How Cutie manages my measurements

Fig.1: DSreader application.

Or rather: how does a QT based application acquire measured data, display it and export it into database for further analysis.

What does this do?

Long story short:

How does it work?

It's a chain built of few links. The above graph (simple as it is) describes whole procedure quite good. I'll start the detailed description from the QT application, as it more or less manages whole operation.

The cutie app.

A simple application is running on my PC. It uses QT library with additional serial interface module (Abstract Serial) and MySQL database driver module (QtSql/QMYSQLDriver). Figure 1 shows the application window. Main part of it are two displayed temperature values measured inside my room, and outside, on my balcony. Below are three buttons: the temperature button, the database button and the serial link button. The icons are also serving the role of status controls for the whole app: background colour tells the user what's going on at the moment: red means an error, yellow means action taking place and green one means all is tip-top good.

The temperature button.

One can request a measurement with this button. Once pressed, changes its background colour to yellow and sends a measurement request to the MCU module. When module replies with data, it updates the displayed values and turns the background colour back to green. I call this measurement a "quick" one. It isn't stored anywhere, just displayed on my desktop.

The database button.

Clicking this one also causes it flip to yellow. Then a measurement read request is sent to the MCU module. The reply causes the application to fetch a SQL query with the acquired data and upload it to a remote MySQL database. If there's any error with that - the button's background is being changed to red. If there's no errors and the SQL query is properly executed, the QT application sends a confirmation frame to the MCU board, telling it that this measurement has been stored properly. Then the MCU looks for another unsent data and the procedure is repeated till all of the data in the database is up to date. Of course the MCU module informs the reader application of such fact with a specific command, telling it to set the button's background to green again. A "quick" temperature measurement is performed at this point, to update the application's display.

The serial link button.

Pressing this button causes the application to re-initialize the serial link. This button is called automaticaly on the reader application's startup. Then the transmission parametters are set and the application tries to use the communication port. If an error occurs (i.e. port is already used), button's background turns red. If the connection is made, the application sends current datetime stamp to the module. Then it automaticaly presses the database button.

The MCU module.

Power supply and continous operation.

I've used one of the universal can modules for this project. It requires 4.5VDC minimum voltage supply to work completely, and even about 3.5VDC will let it work without the CAN transceiver operational. This allows me to easily keep it ON all the time, by just adding a set of three AA cells and two schottky junctions to it. By default it is powered from my PC's USB 5VDC supply. The (lower) battery voltage source is then cut-off by the schottky diode. The USB sources supply even while being in standby mode. But if it is turned off, the main voltage drops below about 4.6VDC, and then the batteries source the device. As soon as the 5VDC shows back at the USB port, it once again cuts the batteries off and becomes the voltage source. This allows continous operation. The RTC can be only once set with the true datastamp information and then continue to work supposingly forever. :-)

Temperatures measurement.

I have two DS18B20 sensors attached to this MCU board. One of them is in my room, measuring the indoor temperature. The other one is outside, on my balcony, in the almost a stevenson screen box that I've built some time ago. The module is programmed to respond with a quick measurement's values to just update the display. Although, the main operation is to perform scheduled measurements. This data will be used to draw a temperature VS time graph later on. When starting up the module, it falls into a "do nothing" loop. The client application running on the PC sets the real datetime in the internal RTC, and then the application begins collecting data.

Managing data collection.

It is not a problem to send the measured data when I have the PC turned on. I began to wonder how would I prevent this device from losing values measured when the PC is turned off. Or when I simply have no internet connection available due to some technical difficulties. Answer is easy: store it in the MCU's internal memory. Luckily, the STR712 happens to have a whole 16kB flash memory bank, that I am not (yet) using. I write my application data into the 256kB bank0, so the 16kB bank1 is waiting for me to store my data in it. One can write 32 bits long data writes to the memory, which is exactly what I need to store two sensors' measurements. I'm also storing the datestamp and some other meta data in another 32 bits, so a single complete mesurement takes 64 bits (8 bytes) in the flash. This leaves me with possibility to store 2048 measurements. Measuring every 15 minutes, the module could stay offline even for 3 weeks and 8 hours. Me gusta!

Having the whole flash memory in bank1 arranged in 8B blocks, the application reffers to every measurement by an index value (0-2047). After uploading a measurement's value to the database successfully, the client application can return the index value to the MCU. Now the measurement is marked as "already sent". This prevents from uploading the same measurement twice. Some simple mechanisms are responsible for finding an unused cell in the flash memory, to write next measurement's values, too. Also the whole bank is split into two sectors, 8kB each. I can erase sector 0 when writing last cell in sector 1 and vice versa.

Note: Of course the maximum offline period varies depending on what number of measurements has been already stored. This is caused by the fact that one has to erase whole sector (8kB) of flash at a time. In worse case scenario, when putting module offline while it's at index 1047, it can only write indexes 1047-2027. This is still 10 days and 16h of operation w/o losing any data though.

The charts generator.

Aim is to have a huge amount of data to work with and huge ammounts of data are best managed by databases. MySQL is what everybody knows. Simple, popular, reliable, free of charge. The QT application I wrote for this project uploads measured data to a remote server. On the same server I have a PHP based page running: http://pit.cba.si/temperatures. Few simple SQL queries are used to get the data required by the pChart graphs drawing library (for PHP). I currently draw two graphs. First one is showing data for the last 24 hours' measurements, every 15 minutes. The second graph shows the maximum and minium values' ranges for last 31 days (temperature amplitudes per day).

Charts are automaticaly generated for every single visit on the page, however the values in the database are only updated when my PC is running. This may cause some lags sometimes, especialy since I don't keep my PC running at nights. Checking back in few hours should give updated results.

Conclusions.

First of all, why would I do this? Well a temperature sensor is a good thing to have in every house. Since I have the hardware already done, and the PC is running quite often, too... The question should be why NOT to make it.

Then there's the QT (cutie) app. Yes one doesn't need that to just simply display temperature. But as far as I remember, I wanted to learn a way to quickly develop a good looking GUI desktop/clicked application that would work with my projects. Seems like QT does just that. Nokia nowdays doesn't sound like quality, but I actualy like this one a lot! Within few minutes one can have a simple app running. Within few hours one can have an app that actualy does something. Within few days one can learn from the scratch, how to make pretty much everything a PC app could do. There are classes available for a really wide set of elements/functions already. Then one can write his own classes, too. Also worth mentioning - I've used the official QT creator IDE. Works great too IMHO. Thumbs up. To the QT, to the STR712's bank1 flash, to the whole project.

Gallery.

Few other pics, to have a look at...

• 

  Hardware during developement.

• 

  Cannot setup serial link error: Right on start, busy port. Didn't even acquire any data yet.

• 

  Quick measurement in progress.

• 

  Last 24h chart for the time that i write this article.

FAILLOG

Yup. What we all love the most. "Nothing makes one happier than seeing his brother struggle" once my Analog Circuits class teacher said, before an exam failed by most of the group. Here's what happened with the application, I hope you get your doze of happiness:

2012-01 dataloss

"Let's make this app even better" was my thought when I have been playing with the database one day of the cold january of 2012. I remember my joy after discovering the "IGNORE" mysql instruction for easier altering table. Who would want to waste time playing with errors! Also this task was so easy, that making a DB backup would be waste of time as well. Few miliseconds later my joy has been substituted with regret. About two weeks of measurements have been lost from the database. Learn by mistakes, ok. No biggie. Also, I've managed to recover the data by adding "dump all flash memory via UART" feature to the firmware. All remains cool!

2012-02-11 corrupted measurements

I've been playing with some relay to automatically enable/disable my speakers set when I turn the PC ON/OFF. Even though this is one of the simplest circuits one can think off, I am not sure if it actually caused my temperature logger to fail. Around 16:30 the MCU began to log strange measurements. At the time I was playing around with the mentioned relay addon - the problem might be related to this (couldn't yet repeat the failure by now, so I am not so sure about the real reason of failure). MCU didn't freeze. It kept on logging bad measurements like "0" and/or "0.06" all the time for both indoor and outdoor sensors. After disconnecting the power supply (both main USB one and battery/backup one as well), the device returned to functional state. Maybe one day I will manage to figure this one out, so far all I can do is to simply delete the corrupted measurements' timeframe from the database. Here's some part of the memory dump log, the broken measurements set:

.25M1202111615 1993-07120729
.25M1202111630 2000-07430730
.25M1202111645 2000 00000731
.25M1202111700 2000 00000732
.25M1202111715 2000 00000733
.25M1202111730 2000 00000734
.25M1202111745 2000 00000735
.25M1202111800 2000 00000736
.25M1202111815 2000 00000737
.25M1202111830 2000 00000738
.25M1202111845-0006 00000739
.25M1202111900-0006 00000740
.25M1202111915-0006 00000741
.25M1202111930-0006 00000742
.25M1202111945-0006 00000743
.25M1202112000-0006 00000744
.25M1202112015-0006 00000745
.25M1202112030-0006 00000746
.25M1202112045-0006 00000747
.25M1202112100-0006 00000748
.25M1202112115-0006 00000749
.25M1202112130-0006 00000750
.25M1202112145-0006 00000751
.25M1202112200-0006 00000752
.25M1202112215-0006 00000753
.25M1202112230-0006 00000754
.25M1202112245-0006 00000755
.25M1202112300 2093-12430756

Also I have no idea what happened between 18:30 and 18:45. SNAFU.

Yours truly,

pit, 2011-12-21