HBase get column qualifiers by column family

To get column qualifiers by column family we can use following code

Above code will outputs all column qualifer names of family map “basicInfo” in table “merchants”.

 

And following code shows different ways to retrieve HBase cell value

 

  • Get Cell by Family Map and Column Qualifier

This line of code retrives cell value by family map “merchantNo” and column qualifier “string”

 

  • Get All Cells by Faimily Map

result.getFamilyMap(Bytes.toBytes("merchantNo"))  will return column family map with name “merchantNo”, it contains column qualifier name as key and cell value as value.

Above code will iterate the column family map, and outputs each column qualifier and its corresponding cell value.

mongodb InternalError: failed to create service entry worker thread

Following is content from /var/log/mongodb/mongod.log:

2017-12-18T01:21:41.953+0800 W EXECUTOR [conn4444] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.953+0800 I NETWORK [listener] end connection 127.0.0.1:36868 (1001 connections now open)
2017-12-18T01:21:41.953+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36870 #4445 (1002 connections now open)
2017-12-18T01:21:41.954+0800 I – [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.954+0800 W EXECUTOR [conn4445] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.954+0800 I NETWORK [listener] end connection 127.0.0.1:36870 (1001 connections now open)
2017-12-18T01:21:41.979+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36872 #4446 (1002 connections now open)
2017-12-18T01:21:41.979+0800 I – [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.979+0800 W EXECUTOR [conn4446] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.979+0800 I NETWORK [listener] end connection 127.0.0.1:36872 (1001 connections now open)

 

This problem occured after web request (which will create new MongoClient instance) started for a while, it’s not happened immediately, so it’s likely caused by the connection limit is reached.

Through the log we can find the connection count limit is about 1000.

 

Every time I open Mongo Console, I got startup warnings

MongoDB shell version v3.6.0
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.0
Server has startup warnings:
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten]
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
We can see the connection count limit is actually 1000.

 

So to fix this issue, either is to increase the connecton count limit, or to close the created MongoClient explicitly in code.
In my program every request a new MongoClient is created, I didn’t explicitly close it after using it. The solution is calling close method of MongoClient or using singleton MongoClient (the MongoClient library recommended this way for using connection pooling).

By using this method, the problem is solved.

Java HTML Encoding (HTML Entities)

Some XSS attacks can be prevented by using HTML Encoding.

HTML encoding function is built into many languages, In .NET WebUtility.HtmlEncode  can do it, in PHP we can use htmlentites  function, in Python cgi.escape  can be used.

But there is no built-in function to do HTML Encode (or HTML Entities) in Java.

We can use Apache Commons Lang library to do this work.

 

Above code will output following result using HTML Encoding

<>

 

Note that not all XSS attacks can be prevented by HTML encoding (https://stackoverflow.com/questions/53728/will-html-encoding-prevent-all-kinds-of-xss-attacks).

Apache POI Sheet.getPhysicalNumberOfRows()

In the past I use following code to display first cell string of all rows.

But above code will be incorrect if there is empty row.

 

Following is explanation from POI official documentation

Sheet.getPhysicalNumberOfRows()

Returns the number of physically defined rows (NOT the number of rows in the sheet)

This method will ignore empty rows, e.g. if a sheet last row is at row 7, but second row is empty, getPhysicalNumberOfRows()  will return 6.

 

Solution

To get row count of a sheet (no matter the row is empty or not), we should use getLastRowNum()  method.

So above code can be changed to

Because getLastRowNum()  method returns 0-based row index, so we use i<=sheet.getLastRowNum()  as the loop condition.

Dom4j Writing File Not Working

Dom4j is a XML processing library which supports XPath, DOM, SAX,  JAXP. In this post we will see a problem that dom4j could not write document to file.

 

Following XML is sample data which we will save to file using dom4j.

To generate above XML and write it to file, following code will be used

Above code may generate an empty file with no content.

 

This is because the content is stored in buffer and not really written to file when calling write  method. To write to file, we should call flush  method or close  method.

So above code should be changed to

Or

 

Hibernate ConstraintViolationException

In following code (a JUnit test method) I’m trying to delete specified users from database

 

But when running above code, I got following exception:

org.hibernate.exception.ConstraintViolationException: could not execute batch

at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:132)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:111)
at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:163)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:226)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:484)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:351)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222)
at com.dfweb.service.impl.SysUserServiceImplTest.testAddPermissions(SysUserServiceImplTest.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:233)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:87)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:176)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign key constraint fails (db_df.permission_sys_user, CONSTRAINT FK_2l5pbq5juqkbrg6rtqes6w1el FOREIGN KEY (users_id) REFERENCES sys_user (id))
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2055)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1467)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1135)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:123)
… 41 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (db_df.permission_sys_user, CONSTRAINT FK_2l5pbq5juqkbrg6rtqes6w1el FOREIGN KEY (users_id) REFERENCES sys_user (id))
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2441)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2007)
… 44 more

 

Reason

This is because there is an RESTRICT reference for foreign key of permission_sys_user that references id column of sys_user table.

So deleting the SysUser entity and keep its related permissions in permission_sys_user table will break the foreign key constraint of permission_sys_user table (references sys_user table with default RESTRICT reference option).

 

Solution

Following is SysUser class related source code

 

One possible solution is to remove its related records in permission_sys_user table when deleting SysUser entity, so change SysUser to following will fix this issue:

Above code adds CascadeType.REMOVE  for ManyToMany  annotation’s cascade  attribute. This will make deleting SysUser entity will trigger deleting related Permission entities.

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet

Recently I tried to use Selenium for Web UI auto testing, but when running driver.get()  function, I got following error

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet()Ljava/util/stream/Collector;

 

Following is full stacktrace

Starting ChromeDriver 2.29.461591 (62ebf098771772160f391d75e589dc567915b233) on port 20686
Only local connections are allowed.

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet()Ljava/util/stream/Collector;

at org.openqa.selenium.remote.ProtocolHandshake.streamW3CProtocolParameters(ProtocolHandshake.java:238)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:104)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:141)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:604)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:244)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:131)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:144)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:178)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:167)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:124)
at com.dfweb.service.impl.PaymentLogServiceImplTest.testCalcuateProfitWithDifferentAgentAndUserOfInventory(PaymentLogServiceImplTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:233)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:87)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:176)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Suppressed: java.io.IOException: Incomplete document
at com.google.gson.stream.JsonWriter.close(JsonWriter.java:527)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:121)
… 37 more

 

Cause

I look through the com.google.guava jar source code, and found there is com.google.common.collect.ImmutableSet class existed, but there is no toImmutableSet() method defined.

It’s because ImmutableSet.toImmutableSet() method is introduced in Guava version 21.0, while I’m using version 20.0.

 

Solution

My current pom.xml

 

Change version to 21.0 and run mvn install  will fix this issue.

 

DeferredResult not working in Tomcat

The DeferredResult works on my local machine, but not work on the server machine.

On my local machine when a DeferredResult called setResult method, the response will return immediately. But on remote server machine, it only return the response to client after 30s (default timeout is 30s).

According to above posts, it may be caused by earlier version Tomcat 8 issue.

Then I tried jetty on remote server machine, it works. Next I checked the version by running CATALINA_BASE/bin/version.sh, it’s displaying

Server version: Apache Tomcat/8.0.14 (Debian)

And my local Tomcat version is 8.0.36

 

Then I upgrade the remote Tomcat version from 8.0.14 to 8.0.36 (upload my local tomcat to remote server) and try DeferredResult again, it works!

 

http://stackoverflow.com/questions/24619445/spring-async-deferredresult-not-working-in-tomcat-8

Apache Derby Database Browser

Background

Apache Derby is an embed database implemented in pure Java. It’s implemented in pure Java, thus it’s more compatible with Java.

There are mainly three database browsers for Derby

Since the first two softwares are commercial, in this tutorial we will introduce how to use SQuirreL.

 

Prerequisites

  • JRE (or JDK)

Download JRE (or JDK) from java.com and install it. Don’t forget to add its bin path to environment variable.

  • SQuirreL

Download SQuirreL from installation page and install it using following command

java -jar squirrel-sql-<version>-install.jar

(or you can double click the jar file if you’re under Windows)

  • Derby Client and Derby Embedded Driver jar files

Download the zip file from this link, then extract derby.jar and derbyclient.jar from the zip file. Next copy these two files into SQuirreL’s lib folder.

If the two jar files are copied to correct path, you will see in SQuirreL’s Drivers panel Apache Derby Client and Apache Derby Embedded are enabled.

SQuirreL Drivers Panel

SQuirreL Drivers Panel

 

 

Create a connection to Derby database

In Alias Panel, click Add Alias button

SQuirrel add alias

SQuirrel add alias

The Name field is not important, and you can fill in with anything. In URL field you enter the server address and database name, e.g. jdbc:derby://localhost/derby_test

 

And if you want to connect to embedded file database,  you need set JDBC Connection URL like following:

jdbc:derby:PATH

Assume the database is located at D:\derby_test, then the JDBC URL should be

jdbc:derby:D:\derby_test

SQuirreL add embed derby database

SQuirreL add embed derby database

 

 

IKVM.NET Convert JAR to .NET (C#, VB.NET) DLL

Why Convert JAR to .NET DLL

Imagine your company builds an online file converter, and its backend programming language is Java because of its position in web development. And in this project you build a file parsing library (jar format certainly) which is used to parse different types of files.

After this project is done, your boss decides to create a Windows desktop version for this software. Now the best choice is .NET, but how can you reuse your java parsing library? That’s why IKVM.NET existed.

IKVM.NET Introduction

IKVM.NET is a .NET implementation of Java which allows us to run Java code in .NET (C#, VB.NET, etc.) easily. It includes following three components:

  • JVM in .NET (which means you can run java byte code in .NET)
  • Java class library implementation
  • A compiler converting java byte code to .NET IL

There are many famous Java projects converted to .NET using IKVM.NET already, such as Tika, Curator, etc. Even Mono has include it as a built-in component.

Start Converting

This will generate a DLL file named libpst.dll from libpst.jar in current directory.

Without -target:library option, it will generate an exe file.