HBase get column qualifiers by column family

To get column qualifiers by column family we can use following code

Above code will outputs all column qualifer names of family map “basicInfo” in table “merchants”.

 

And following code shows different ways to retrieve HBase cell value

 

  • Get Cell by Family Map and Column Qualifier

This line of code retrives cell value by family map “merchantNo” and column qualifier “string”

 

  • Get All Cells by Faimily Map

result.getFamilyMap(Bytes.toBytes("merchantNo"))  will return column family map with name “merchantNo”, it contains column qualifier name as key and cell value as value.

Above code will iterate the column family map, and outputs each column qualifier and its corresponding cell value.

mongodb InternalError: failed to create service entry worker thread

Following is content from /var/log/mongodb/mongod.log:

2017-12-18T01:21:41.953+0800 W EXECUTOR [conn4444] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.953+0800 I NETWORK [listener] end connection 127.0.0.1:36868 (1001 connections now open)
2017-12-18T01:21:41.953+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36870 #4445 (1002 connections now open)
2017-12-18T01:21:41.954+0800 I – [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.954+0800 W EXECUTOR [conn4445] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.954+0800 I NETWORK [listener] end connection 127.0.0.1:36870 (1001 connections now open)
2017-12-18T01:21:41.979+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36872 #4446 (1002 connections now open)
2017-12-18T01:21:41.979+0800 I – [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.979+0800 W EXECUTOR [conn4446] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.979+0800 I NETWORK [listener] end connection 127.0.0.1:36872 (1001 connections now open)

 

This problem occured after web request (which will create new MongoClient instance) started for a while, it’s not happened immediately, so it’s likely caused by the connection limit is reached.

Through the log we can find the connection count limit is about 1000.

 

Every time I open Mongo Console, I got startup warnings

MongoDB shell version v3.6.0
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.0
Server has startup warnings:
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten]
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
We can see the connection count limit is actually 1000.

 

So to fix this issue, either is to increase the connecton count limit, or to close the created MongoClient explicitly in code.
In my program every request a new MongoClient is created, I didn’t explicitly close it after using it. The solution is calling close method of MongoClient or using singleton MongoClient (the MongoClient library recommended this way for using connection pooling).

By using this method, the problem is solved.

Python TypeError: Error when calling the metaclass bases

When running the code, I got following error:

TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)

 

The source code is like following:

AboutWindow.py

 

GitFtpWindow.py

 

This is because in AboutWindow.py file, the GitFtpWindow identifier is a package, but not a class.

To fix this problem, we need modify AboutWindow.py file like following:

 

 

Java HTML Encoding (HTML Entities)

Some XSS attacks can be prevented by using HTML Encoding.

HTML encoding function is built into many languages, In .NET WebUtility.HtmlEncode  can do it, in PHP we can use htmlentites  function, in Python cgi.escape  can be used.

But there is no built-in function to do HTML Encode (or HTML Entities) in Java.

We can use Apache Commons Lang library to do this work.

 

Above code will output following result using HTML Encoding

<>

 

Note that not all XSS attacks can be prevented by HTML encoding (https://stackoverflow.com/questions/53728/will-html-encoding-prevent-all-kinds-of-xss-attacks).

Apache POI Sheet.getPhysicalNumberOfRows()

In the past I use following code to display first cell string of all rows.

But above code will be incorrect if there is empty row.

 

Following is explanation from POI official documentation

Sheet.getPhysicalNumberOfRows()

Returns the number of physically defined rows (NOT the number of rows in the sheet)

This method will ignore empty rows, e.g. if a sheet last row is at row 7, but second row is empty, getPhysicalNumberOfRows()  will return 6.

 

Solution

To get row count of a sheet (no matter the row is empty or not), we should use getLastRowNum()  method.

So above code can be changed to

Because getLastRowNum()  method returns 0-based row index, so we use i<=sheet.getLastRowNum()  as the loop condition.

Dom4j Writing File Not Working

Dom4j is a XML processing library which supports XPath, DOM, SAX,  JAXP. In this post we will see a problem that dom4j could not write document to file.

 

Following XML is sample data which we will save to file using dom4j.

To generate above XML and write it to file, following code will be used

Above code may generate an empty file with no content.

 

This is because the content is stored in buffer and not really written to file when calling write  method. To write to file, we should call flush  method or close  method.

So above code should be changed to

Or

 

Ext JS Image Reload

In Ext JS, to reload or refresh an image, we can call updateSrc() method, and also setSrc() method is working as well.

 

And if the requested image is cached, we need use cache buster to avoid loading cached image.

 

Note that I found another post saying to use doComponentLayout method to refresh an image, but this method is deprecated since version 4.1.

Hibernate ConstraintViolationException

In following code (a JUnit test method) I’m trying to delete specified users from database

 

But when running above code, I got following exception:

org.hibernate.exception.ConstraintViolationException: could not execute batch

at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:132)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:111)
at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:163)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:226)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:484)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:351)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222)
at com.dfweb.service.impl.SysUserServiceImplTest.testAddPermissions(SysUserServiceImplTest.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:233)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:87)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:176)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign key constraint fails (db_df.permission_sys_user, CONSTRAINT FK_2l5pbq5juqkbrg6rtqes6w1el FOREIGN KEY (users_id) REFERENCES sys_user (id))
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2055)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1467)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1135)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:123)
… 41 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (db_df.permission_sys_user, CONSTRAINT FK_2l5pbq5juqkbrg6rtqes6w1el FOREIGN KEY (users_id) REFERENCES sys_user (id))
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2441)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2007)
… 44 more

 

Reason

This is because there is an RESTRICT reference for foreign key of permission_sys_user that references id column of sys_user table.

So deleting the SysUser entity and keep its related permissions in permission_sys_user table will break the foreign key constraint of permission_sys_user table (references sys_user table with default RESTRICT reference option).

 

Solution

Following is SysUser class related source code

 

One possible solution is to remove its related records in permission_sys_user table when deleting SysUser entity, so change SysUser to following will fix this issue:

Above code adds CascadeType.REMOVE  for ManyToMany  annotation’s cascade  attribute. This will make deleting SysUser entity will trigger deleting related Permission entities.

CakePHP Get Version

To find out current version of CakePHP, we can use following methods:

  • CakePHP 2.x

Open lib/Cake/VERSION.txt file, the version number is at bottom of file.

 

  • CakePHP 3.x

Open vendor/cakephp/cakephp/VERSION.txt file, the version number is at bottom of file.

 

And if you want to get version in code, we can use following code

 

Under the Hood

The Configure class is defined at lib/Cake/Core/Configuration.php file, reading the code we will find it’s actually reading the VERSION.txt file (the file we mentioned above) to find out current version as well.

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet

Recently I tried to use Selenium for Web UI auto testing, but when running driver.get()  function, I got following error

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet()Ljava/util/stream/Collector;

 

Following is full stacktrace

Starting ChromeDriver 2.29.461591 (62ebf098771772160f391d75e589dc567915b233) on port 20686
Only local connections are allowed.

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet()Ljava/util/stream/Collector;

at org.openqa.selenium.remote.ProtocolHandshake.streamW3CProtocolParameters(ProtocolHandshake.java:238)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:104)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:141)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:604)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:244)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:131)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:144)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:178)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:167)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:124)
at com.dfweb.service.impl.PaymentLogServiceImplTest.testCalcuateProfitWithDifferentAgentAndUserOfInventory(PaymentLogServiceImplTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:233)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:87)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:176)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Suppressed: java.io.IOException: Incomplete document
at com.google.gson.stream.JsonWriter.close(JsonWriter.java:527)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:121)
… 37 more

 

Cause

I look through the com.google.guava jar source code, and found there is com.google.common.collect.ImmutableSet class existed, but there is no toImmutableSet() method defined.

It’s because ImmutableSet.toImmutableSet() method is introduced in Guava version 21.0, while I’m using version 20.0.

 

Solution

My current pom.xml

 

Change version to 21.0 and run mvn install  will fix this issue.