IoT Hackathon Part IV : Using Web Services to send Sensordata

In the previous 3 posts, building towards the eProseed IoT Hackathon, I described how to setup your Raspberry Pi, and how to use the GrovePi sensors. The used example is a small weather-station that read temperature and humidity and shows the readings on a display. That is all very nice, however, the data remains local on the Raspberry Pi so there is nothing that we can do with this information from an 'enterprise' perspective. In this post I will explain how easy it is to send the data to whatever 'end point' by using a REST-JSON web-service.

The Database Tables
For this use case I decided I needed 2 tables. One to hold all my available sensors (yes, I know, I have only one) and one to store the measurements per sensor.

The Webservice

In order to store the data in the database tables I need to send it from the Raspberry Pi to the database. For that I use a simple REST/JSON webservice that has a POST method. I also implemented PUT, GET and DELETE methods, because they might come in handy later.

The Webserivce is described by the following WADL:

 <ns0:application xmlns:ns0="http://wadl.dev.java.net/2009/02">  
   <ns0:doc xmlns:ns1="http://jersey.java.net/" ns1:generatedBy="Jersey: 2.5.1 2014-01-02 13:43:00"/>  
   <ns0:doc xmlns:ns2="http://jersey.java.net/"  
        ns2:hint="This is simplified WADL with user and core resources only. To get full WADL with extended resources use the query parameter detail.  Link: http://yourhost.com:7101/IoTRestJsonService/resources/application.wadl?detail=true"/>  
   <ns0:grammars>  
     <ns0:include href="application.wadl/xsd0.xsd">  
       <ns0:doc title="Generated" xml:lang="en"/>  
     </ns0:include>  
   </ns0:grammars>  
   <ns0:resources base="http://yourhost.com:7101/IoTRestJsonService/resources/">  
     <ns0:resource path="iot">  
       <ns0:resource path="/sensordata">  
         <ns0:method id="createSensorData" name="POST">  
           <ns0:request>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:request>  
         </ns0:method>  
         <ns0:method id="updateSensorData" name="PUT">  
           <ns0:request>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:request>  
         </ns0:method>  
         <ns0:method id="findAllSensorData" name="GET">  
           <ns0:response>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:response>  
         </ns0:method>  
       </ns0:resource>  
       <ns0:resource path="/sensordata/{id}">  
         <ns0:param xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="id" style="template" type="xsd:int"/>  
         <ns0:method id="deleteSensorData" name="DELETE"/>  
         <ns0:method id="getSensorDataById" name="GET">  
           <ns0:response>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:response>  
         </ns0:method>  
       </ns0:resource>  
     </ns0:resource>  
   </ns0:resources>  
 </ns0:application>  

The service can be tested from any REST client. You can use 'Test Webserice' from within JDeveloper, or use a tool such as postman to POST a test message.

Calling Webservices from Python

Database is in place, Webservice is up and running, so the only remaining thing is to call the service from my Python code. By now it starting to get pretty obvious that this also cannot be very difficult with Python. And indeed, it is very simple again. Python comes with two libraries that can be used for this purpose: The request library and the json library. When you use those two, it takes only a couple of minutes to implement the webservice call. The code sample below shows you how to add the import of the two libraries and then how to construct the request; The request of course needs a resource URL in our case the relevant part is'iot/sensordata'. We also need to tell the request that is a json type request. This is part of the header. Finally we need to construct our data string that contains the measurement data. Here we can construct the exact JSON string that is expected by the service. The data that is sent in the request is simply dumped as JSON string, by calling "json.dumps()". This is the exact same string as can be seen from the Postman screenshot, whit other data ofcourse.

Note that I use sensorid =1 because this is the unique identifier of this sensor. To be 100% dynamic we could also use the ip adres of the Raspberry Pi as sensorid. We could call out to http://httpbin.org/ip to get the ip adres. For now I keep it simple and stick with the hardcoded value of 1.

 # import for the ws  
 import requests  
 import json  
 # end import for the ws  

 # here we prepare and call the ws  
  url = "http://yourhost.com:7101/IoTRestJsonService/resources/iot/sensordata"  
  headers = {'Content-Type':'application/json'}  
  sensordata = {"dataDate":i.isoformat(),"dataType":"T", "dataValue":t,"dataValueNumber":t,"sensorid":"1" }  
  r=requests.post(url, data=json.dumps(sensordata),headers=headers)  

Finally I share the complete code of the Python script with you so you can see and learn how to do this.

 # grovepi_lcd_dht.py  
 #  
 # This is an project for using the Grove LCD Display and the Grove DHT Sensor from the GrovePi starter kit  
 #   
 # In this project, the Temperature and humidity from the DHT sensor is printed on the DHT sensor  
 from grovepi import *  
 from grove_rgb_lcd import *  
 import logging  
 import datetime  
 # for the ws  
 import requests  
 import json  
 # end for the ws  
 dht_sensor_port = 7          # Connect the DHt sensor to port 7  
 # lets log to file  
 logger = logging.getLogger('weather.logger')  
 logger.setLevel('DEBUG')  
 file_log_handler = logging.FileHandler('/home/pi/Desktop/Lucs_projects/weatherstation/weather.log')  
 logger.addHandler(file_log_handler)  
 stderr_log_handler = logging.StreamHandler()  
 logger.addHandler(stderr_log_handler)  
 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')  
 file_log_handler.setFormatter(formatter)  
 stderr_log_handler.setFormatter(formatter)  
 while True:  
      try:  
           [ temp,hum ] = dht(dht_sensor_port,0)          #Get the temperature and Humidity from the DHT sensor  
           t = '{0:.2f}'.format(temp)  
           h = '{0:.2f}'.format(hum)  
           i = datetime.datetime.now()  
           logger.info("temp ="+ t + "C\thumidity ="+ h + "%")   
           setRGB(0,128,64)  
           setRGB(0,255,0)  
           setText("Temp:" + t + "C   " + "Humidity:" + h + "%")  
         # here we prepare and call the ws  
           url = "http://yourhost.com:7101/IoTRestJsonService/resources/iot/sensordata"  
           headers = {'Content-Type':'application/json'}  
         #prep the temp message  
           sensordata = {"dataDate":i.isoformat(), "dataType":"T", "dataValue":t,"dataValueNumber":t,"sensorid":"1"}  
           r=requests.post(url, data=json.dumps(sensordata),headers=headers)  
         #prep the humidity message  
           sensordata = {"dataDate":i.isoformat(), "dataType":"H", "dataValue":h,"dataValueNumber":h,"sensorid":"1"}  
           r=requests.post(url, data=json.dumps(sensordata),headers=headers)  
         # end ws call  
            time.sleep(60)                 
      except (IOError,TypeError) as e:  
           logger.error('Error')  
           logger.error(r)  

Note that, almost at the end of the script, I use a time.sleep(60). This actually puts the script to sleep for one minute. This means, that every minute the new data is measured and sent to the webservice and saved in the database. It is very simple to build an ADF Page to display the data in some graphs. That is beyond the scope of this blogpost, however, here is an image of the result.

Note
The core purpose of this post is to describe how to use webservices from python, to forward sensor data to a (cloud) server. In a later post I describe how to use the better suited MQTT protocol. An example of this is CloudMQTT. CloudMQTT are managed Mosquitto servers in the cloud.Mosquitto implements the MQ Telemetry Transport protocol, MQTT, which provides lightweight methods of carrying out messaging using a publish/subscribe message queueing model.

Resources
1) RESTful Web Service in JDeveloper 12c
2) Calling REST/JSON from Python