Tuesday, September 08, 2015

ADF 12c : Using Jasper Reports en JasperSoft Studio 6.1; What Libraries do you need?

Over the last couple of years, or better in the last decade I have implemented several reporting solutions with Jasper Reports in ADF. I did that in ADF 10g, ADF 11.1.1.x, ADF 11.1.2.x and ADF 12.1.x
I also used several version of Jasper Reports. There is a whole lot of documentation, blogposts and presentations available. So when today I got a request from one of my customers to make a setup for the implementation of Jasper Reports 6.1 in ADF 12.1.3 I did not expect any problems. Boy was I wrong.

Here is the Story
With all the knowledge from the past, I decided to follow the known steps.
1) Download iReport Designer,
2) Build a report in iReport
3) Create an ADF application
4) Add the necessary libraries to use the report
5) Call the report from a button via a Managed Bean

Step 1
In the past I used iReport designer to build the reports. When you go to the download site of iReport designer you now see an interesting message.


So I took this serious and decided not to use iReport Designer, but to use JasperSoft Studio. It proves to work pretty much the same, so I was able to create a simple sample report.

Step 2
The simple sample report uses a Data Adapter to an Oracle Database (HR) that can be created as in step 5 of this tutorial. Take special note of the classpath of Database Library Driver. I use the one that is shipped with JDeveloper. That one can be found in:
1:  <jdeveloperhome>/oracle_common/rda/da/lib/ojdbc14.jar  

Now the report can be created using a simple query, for instance

 select * from EMPLOYEES  
Or you can use this tutorial to build your own report.

Step 3 and 4 and 5
In the new ADF Application we need to make sure to add the jasperReports Library to your viewController project. I used jasperreports-6.1.0.jar

With that in place I can now use my code from the past that needs to be in the backing bean to call the report from a button. That code should still work and is exactly as I show here.
1:  public class ReportingBean {  
2:    public ReportingBean() {  
3:    }  
4:    public void startReport(ActionEvent actionEvent) {  
5:      // Add event code here...  
6:      String filepath = "C:/JDeveloper/mywork/reports/MyReports/";  
7:      String reportname = "dummy.txt";  
8:      InputStream is;  
9:      try {  
10:        OutputStream os = new FileOutputStream(new File(filepath + reportname));  
11:        Map parameters = new HashMap();  
12:        // parameters.put("P_DEPARTMENT_ID", getDepartmentId());  
13:        Connection conn = getConnectionDS("java:comp/env/jdbc/MYHRDS");  
14:        JasperReport jasperReport;  
15:        //Note; we have two options. EIther we use the .jasper file and run the compiled report  
16:        //    or we use the .jrxml file that needs to be compiled at runtime.  
17:        if(runNoComplile){       
18:        // from .jasper file, so without compiling        
19:         InputStream jasperStream = new FileInputStream(new     
20:                 File("C:/JDeveloper/mywork/reports/MyReports/EMP_A4.jasper"));  
21:         jasperReport=   (JasperReport) JRLoader.loadObject(jasperStream);  
22:        // end from .jasper  
23:        }  
24:        else{  
25:        // from jrxml, so we do a runtime compile  
26:         is = new FileInputStream(new  
27:             File("C:/JDeveloper/mywork/reports/MyReports/EMP_A4.jrxml"));  
28:         JasperDesign jasperDesign = JRXmlLoader.load(is);  
29:         jasperReport = JasperCompileManager.compileReport(jasperDesign);  
30:       // end from jrxml  
31:        }  
32:        JasperPrint jasperPrint = JasperFillManager.fillReport(jasperReport, parameters, conn);  
33:        JasperExportManager.exportReportToPdfStream(jasperPrint, os);  
34:        JasperViewer.viewReport(jasperPrint, false);  
35:      } catch (FileNotFoundException e) {  
36:        e.printStackTrace();  
37:      } catch (JRException e) {  
38:        e.printStackTrace();  
39:      } catch (NamingException e) {  
40:        e.printStackTrace();  
41:      } catch (SQLException e) {  
42:        e.printStackTrace();  
43:      }  
44:    }  
45:    public static Connection getConnectionDS(String dsName) throws NamingException, SQLException {  
46:      Connection connection = null;  
47:      try {  
48:       javax.naming.Context initialContext = new javax.naming.InitialContext();  
49:       javax.sql.DataSource dataSource =   
50:            (javax.sql.DataSource)initialContext.lookup(dsName);  
51:       connection = dataSource.getConnection();  
52:       } catch(Exception e){  
53:         e.printStackTrace();  
54:         //or handle more gracefully   
55:       }  
56:      return connection;  
57:    }  
58:  }  

The necessary imports are added to the class from the jasperreports-6.1.0.jar library, the class compiles, the project builds and the application runs. However, at runtime, you will probably run into trouble. You will see several ClassNotFound Exceptions. At least I did run into these. It took me a couple of hours before I was able to fix these issues. It all had to with the library dependencies in the ViewController project.

So what exactly should you add, and where can you find it
All that the documentation tells you is that you need to add the jasperreports-6.1.0.jar library, however there is more that should be added to make this all work.
From the past I know that we should add the following:


So I decided to add whatever I could find that resembles those libs that made my reports work in the past. In the iReport solution these libraries were located in:
1:  <reportshome>\iReport-4.0.1\ireport\modules\ext  

In the new situation, with JasperSoft Studio, I was unable to find these libraries (or the new versions of them) in the reportshome location. So where have they gone?

I finally managed to find them in the download of jasperReports-6.1.0-project.zip. If you download the file and unzip it you will find all necessary libraries in:
1:  jasperreports-6.1.0/lib  

You can now copy the libraries from there to a folder that you use to bundle those libraries into your ADF Project. So what I did was the following: I created a folder called:
1:  /extralibs/jasper  

And I copied the following libs to it:
  • groovy-all-2.0.1.jar
  • iText-2.1.7.js2.jar
  • jasperreports-6.1.0.jar
  • poi-3.10.1.jar
  • jfreechart-1.0.12.jar
  • jcommon-1.0.15.jar
  • ant-1.7.1.jar (not sure if I need this one)
This resembles pretty much what I used to have in previous versions of my reports implementation. I was convinced that it would work now, so I gave it a try. Almost everything worked...

Except for the report that needs to be compiled (line 29 in the code sample above). Too bad, compilation failed and I was almost giving up. After a lot of googling and trying, I finally found a solution: Apparently Jasper reports uses the ecj library to do the compilation of reports. This library happens to be available from the same location as the ones that I added previously. Once I added this library to my project, all worked perfectly. The image below shows what libs were added by me in order to make it work.


Final note
I added the libraries to the ViewController project via the Jdeveloper Library Dependencies. You can (or should) use Maven or ANT to manage the library dependencies, as this is much more flexible.

Thursday, September 03, 2015

IoT Hackathon Part IV : Using Web Services to send Sensordata

In the previous 3 posts, building towards the eProseed IoT Hackathon, I described how to setup your Raspberry Pi, and how to use the GrovePi sensors. The used example is a small weather-station that read temperature and humidity and shows the readings on a display. That is all very nice, however, the data remains local on the Raspberry Pi so there is nothing that we can do with this information from an 'enterprise' perspective. In this post I will explain how easy it is to send the data to whatever 'end point' by using a REST-JSON web-service.

The Database Tables
For this use case I decided I needed 2 tables. One to hold all my available sensors (yes, I know, I have only one) and one to store the measurements per sensor.


The Webservice

In order to store the data in the database tables I need to send it from the Raspberry Pi to the database. For that I use a simple REST/JSON webservice that has a POST method. I also implemented PUT, GET and DELETE methods, because they might come in handy later.

The Webserivce is described by the following WADL:
 <ns0:application xmlns:ns0="http://wadl.dev.java.net/2009/02">  
   <ns0:doc xmlns:ns1="http://jersey.java.net/" ns1:generatedBy="Jersey: 2.5.1 2014-01-02 13:43:00"/>  
   <ns0:doc xmlns:ns2="http://jersey.java.net/"  
        ns2:hint="This is simplified WADL with user and core resources only. To get full WADL with extended resources use the query parameter detail.  Link: http://yourhost.com:7101/IoTRestJsonService/resources/application.wadl?detail=true"/>  
   <ns0:grammars>  
     <ns0:include href="application.wadl/xsd0.xsd">  
       <ns0:doc title="Generated" xml:lang="en"/>  
     </ns0:include>  
   </ns0:grammars>  
   <ns0:resources base="http://yourhost.com:7101/IoTRestJsonService/resources/">  
     <ns0:resource path="iot">  
       <ns0:resource path="/sensordata">  
         <ns0:method id="createSensorData" name="POST">  
           <ns0:request>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:request>  
         </ns0:method>  
         <ns0:method id="updateSensorData" name="PUT">  
           <ns0:request>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:request>  
         </ns0:method>  
         <ns0:method id="findAllSensorData" name="GET">  
           <ns0:response>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:response>  
         </ns0:method>  
       </ns0:resource>  
       <ns0:resource path="/sensordata/{id}">  
         <ns0:param xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="id" style="template" type="xsd:int"/>  
         <ns0:method id="deleteSensorData" name="DELETE"/>  
         <ns0:method id="getSensorDataById" name="GET">  
           <ns0:response>  
             <ns0:representation element="iotSensorData" mediaType="application/json"/>  
           </ns0:response>  
         </ns0:method>  
       </ns0:resource>  
     </ns0:resource>  
   </ns0:resources>  
 </ns0:application>  

The service can be tested from any REST client. You can use 'Test Webserice' from within JDeveloper, or use a tool such as postman to POST a test message.
Calling Webservices from Python

Database is in place, Webservice is up and running, so the only remaining thing is to call the service from my Python code. By now it starting to get pretty obvious that this also cannot be very difficult with Python. And indeed, it is very simple again. Python comes with two libraries that can be used for this purpose: The request library and the json library. When you use those two, it takes only a couple of minutes to implement the webservice call. The code sample below shows you how to add the import of the two libraries and then how to construct the request; The request of course needs a resource URL in our case the relevant part is'iot/sensordata'. We also need to tell the request that is a json type request. This is part of the header. Finally we need to construct our data string that contains the measurement data. Here we can construct the exact JSON string that is expected by the service. The data that is sent in the request is simply dumped as JSON string, by calling "json.dumps()". This is the exact same string as can be seen from the Postman screenshot, whit other data ofcourse.

Note that I use sensorid =1 because this is the unique identifier of this sensor. To be 100% dynamic we could also use the ip adres of the Raspberry Pi as sensorid. We could call out to http://httpbin.org/ip to get the ip adres. For now I keep it simple and stick with the hardcoded value of 1.

 # import for the ws  
 import requests  
 import json  
 # end import for the ws  

 # here we prepare and call the ws  
  url = "http://yourhost.com:7101/IoTRestJsonService/resources/iot/sensordata"  
  headers = {'Content-Type':'application/json'}  
  sensordata = {"dataDate":i.isoformat(),"dataType":"T", "dataValue":t,"dataValueNumber":t,"sensorid":"1" }  
  r=requests.post(url, data=json.dumps(sensordata),headers=headers)  
Finally I share the complete code of the Python script with you so you can see and learn how to do this.
 # grovepi_lcd_dht.py  
 #  
 # This is an project for using the Grove LCD Display and the Grove DHT Sensor from the GrovePi starter kit  
 #   
 # In this project, the Temperature and humidity from the DHT sensor is printed on the DHT sensor  
 from grovepi import *  
 from grove_rgb_lcd import *  
 import logging  
 import datetime  
 # for the ws  
 import requests  
 import json  
 # end for the ws  
 dht_sensor_port = 7          # Connect the DHt sensor to port 7  
 # lets log to file  
 logger = logging.getLogger('weather.logger')  
 logger.setLevel('DEBUG')  
 file_log_handler = logging.FileHandler('/home/pi/Desktop/Lucs_projects/weatherstation/weather.log')  
 logger.addHandler(file_log_handler)  
 stderr_log_handler = logging.StreamHandler()  
 logger.addHandler(stderr_log_handler)  
 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')  
 file_log_handler.setFormatter(formatter)  
 stderr_log_handler.setFormatter(formatter)  
 while True:  
      try:  
           [ temp,hum ] = dht(dht_sensor_port,0)          #Get the temperature and Humidity from the DHT sensor  
           t = '{0:.2f}'.format(temp)  
           h = '{0:.2f}'.format(hum)  
           i = datetime.datetime.now()  
           logger.info("temp ="+ t + "C\thumidity ="+ h + "%")   
           setRGB(0,128,64)  
           setRGB(0,255,0)  
           setText("Temp:" + t + "C   " + "Humidity:" + h + "%")  
         # here we prepare and call the ws  
           url = "http://yourhost.com:7101/IoTRestJsonService/resources/iot/sensordata"  
           headers = {'Content-Type':'application/json'}  
         #prep the temp message  
           sensordata = {"dataDate":i.isoformat(), "dataType":"T", "dataValue":t,"dataValueNumber":t,"sensorid":"1"}  
           r=requests.post(url, data=json.dumps(sensordata),headers=headers)  
         #prep the humidity message  
           sensordata = {"dataDate":i.isoformat(), "dataType":"H", "dataValue":h,"dataValueNumber":h,"sensorid":"1"}  
           r=requests.post(url, data=json.dumps(sensordata),headers=headers)  
         # end ws call  
            time.sleep(60)                 
      except (IOError,TypeError) as e:  
           logger.error('Error')  
           logger.error(r)  
Note that, almost at the end of the script, I use a time.sleep(60). This actually puts the script to sleep for one minute. This means, that every minute the new data is measured and sent to the webservice and saved in the database. It is very simple to build an ADF Page to display the data in some graphs. That is beyond the scope of this blogpost, however, here is an image of the result.

Note
The core purpose of this post is to describe how to use webservices from python, to forward sensor data to a (cloud) server. In a later post I describe how to use the better suited MQTT protocol. An example of this is CloudMQTT. CloudMQTT are managed Mosquitto servers in the cloud.Mosquitto implements the MQ Telemetry Transport protocol, MQTT, which provides lightweight methods of carrying out messaging using a publish/subscribe message queueing model.

Resources
1) RESTful Web Service in JDeveloper 12c
2) Calling REST/JSON from Python