Spring Cloud Zuul Ribbon custom rule not working

I need a custom Rule which can route requests to different service according to weight option (actually I want to deploy an A/B test service and only small percent of user will hit it), the code is like following

 

And following is application.yml configuration file

 

Adding breakpoints at constructor of ZoneAwareLoadBalancer and DynamicServerListLoadBalancer, then access the Zuul service via browser and I found ZoneAwareLoadBalancer constructor breakpoint is reached first.

Inspect the value clientConfig->properties->NFLoadBalancerClassName, its value is still “com.netflix.loadbalancer.ZoneAwareLoadBalancer”, and the passed in rule is not my configured rule as well.

This means my configuration didn’t work.

 

Even I changed the class name to an invalid value, there is no error

 

In Spring Cloud documentation, under Customizing the Ribbon Client using properties section, it’s saying

Starting with version 1.2.0, Spring Cloud Netflix now supports customizing Ribbon clients using properties to be compatible with the Ribbon documentation.

Checking maven pom file I found it’s using 1.1.0

 

Changing them to 1.2.0 and run again, following exception is thrown

 

It’s because NFLoadBalancerRuleClassName is aaa, changing it to a valid name and try it again, it works now

 

 

Intellij IDEA plugin development get selected text

To get selected text in a Intellij Platform Plugin project, we can use following code

 

The first line is used to get current editor object

 

Then call getSelectionModel and getSelectedText method to get selected text

 

The last two lines is to to some operation for selected text, it’s unrelated to our goal.

 

 

HBase get column qualifiers by column family

To get column qualifiers by column family we can use following code

Above code will outputs all column qualifer names of family map “basicInfo” in table “merchants”.

 

And following code shows different ways to retrieve HBase cell value

 

  • Get Cell by Family Map and Column Qualifier

This line of code retrives cell value by family map “merchantNo” and column qualifier “string”

 

  • Get All Cells by Faimily Map

result.getFamilyMap(Bytes.toBytes("merchantNo"))  will return column family map with name “merchantNo”, it contains column qualifier name as key and cell value as value.

Above code will iterate the column family map, and outputs each column qualifier and its corresponding cell value.

mongodb InternalError: failed to create service entry worker thread

Following is content from /var/log/mongodb/mongod.log:

2017-12-18T01:21:41.953+0800 W EXECUTOR [conn4444] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.953+0800 I NETWORK [listener] end connection 127.0.0.1:36868 (1001 connections now open)
2017-12-18T01:21:41.953+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36870 #4445 (1002 connections now open)
2017-12-18T01:21:41.954+0800 I – [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.954+0800 W EXECUTOR [conn4445] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.954+0800 I NETWORK [listener] end connection 127.0.0.1:36870 (1001 connections now open)
2017-12-18T01:21:41.979+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36872 #4446 (1002 connections now open)
2017-12-18T01:21:41.979+0800 I – [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.979+0800 W EXECUTOR [conn4446] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.979+0800 I NETWORK [listener] end connection 127.0.0.1:36872 (1001 connections now open)

 

This problem occured after web request (which will create new MongoClient instance) started for a while, it’s not happened immediately, so it’s likely caused by the connection limit is reached.

Through the log we can find the connection count limit is about 1000.

 

Every time I open Mongo Console, I got startup warnings

MongoDB shell version v3.6.0
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.0
Server has startup warnings:
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten]
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
We can see the connection count limit is actually 1000.

 

So to fix this issue, either is to increase the connecton count limit, or to close the created MongoClient explicitly in code.
In my program every request a new MongoClient is created, I didn’t explicitly close it after using it. The solution is calling close method of MongoClient or using singleton MongoClient (the MongoClient library recommended this way for using connection pooling).

By using this method, the problem is solved.

Java HTML Encoding (HTML Entities)

Some XSS attacks can be prevented by using HTML Encoding.

HTML encoding function is built into many languages, In .NET WebUtility.HtmlEncode  can do it, in PHP we can use htmlentites  function, in Python cgi.escape  can be used.

But there is no built-in function to do HTML Encode (or HTML Entities) in Java.

We can use Apache Commons Lang library to do this work.

 

Above code will output following result using HTML Encoding

<>

 

Note that not all XSS attacks can be prevented by HTML encoding (https://stackoverflow.com/questions/53728/will-html-encoding-prevent-all-kinds-of-xss-attacks).

Apache POI Sheet.getPhysicalNumberOfRows()

In the past I use following code to display first cell string of all rows.

But above code will be incorrect if there is empty row.

 

Following is explanation from POI official documentation

Sheet.getPhysicalNumberOfRows()

Returns the number of physically defined rows (NOT the number of rows in the sheet)

This method will ignore empty rows, e.g. if a sheet last row is at row 7, but second row is empty, getPhysicalNumberOfRows()  will return 6.

 

Solution

To get row count of a sheet (no matter the row is empty or not), we should use getLastRowNum()  method.

So above code can be changed to

Because getLastRowNum()  method returns 0-based row index, so we use i<=sheet.getLastRowNum()  as the loop condition.

Dom4j Writing File Not Working

Dom4j is a XML processing library which supports XPath, DOM, SAX,  JAXP. In this post we will see a problem that dom4j could not write document to file.

 

Following XML is sample data which we will save to file using dom4j.

To generate above XML and write it to file, following code will be used

Above code may generate an empty file with no content.

 

This is because the content is stored in buffer and not really written to file when calling write  method. To write to file, we should call flush  method or close  method.

So above code should be changed to

Or

 

Hibernate ConstraintViolationException

In following code (a JUnit test method) I’m trying to delete specified users from database

 

But when running above code, I got following exception:

org.hibernate.exception.ConstraintViolationException: could not execute batch

at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:132)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:111)
at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:163)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:226)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:484)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:351)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222)
at com.dfweb.service.impl.SysUserServiceImplTest.testAddPermissions(SysUserServiceImplTest.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:233)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:87)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:176)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign key constraint fails (db_df.permission_sys_user, CONSTRAINT FK_2l5pbq5juqkbrg6rtqes6w1el FOREIGN KEY (users_id) REFERENCES sys_user (id))
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2055)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1467)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1135)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:123)
… 41 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (db_df.permission_sys_user, CONSTRAINT FK_2l5pbq5juqkbrg6rtqes6w1el FOREIGN KEY (users_id) REFERENCES sys_user (id))
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2441)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2007)
… 44 more

 

Reason

This is because there is an RESTRICT reference for foreign key of permission_sys_user that references id column of sys_user table.

So deleting the SysUser entity and keep its related permissions in permission_sys_user table will break the foreign key constraint of permission_sys_user table (references sys_user table with default RESTRICT reference option).

 

Solution

Following is SysUser class related source code

 

One possible solution is to remove its related records in permission_sys_user table when deleting SysUser entity, so change SysUser to following will fix this issue:

Above code adds CascadeType.REMOVE  for ManyToMany  annotation’s cascade  attribute. This will make deleting SysUser entity will trigger deleting related Permission entities.

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet

Recently I tried to use Selenium for Web UI auto testing, but when running driver.get()  function, I got following error

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet()Ljava/util/stream/Collector;

 

Following is full stacktrace

Starting ChromeDriver 2.29.461591 (62ebf098771772160f391d75e589dc567915b233) on port 20686
Only local connections are allowed.

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.toImmutableSet()Ljava/util/stream/Collector;

at org.openqa.selenium.remote.ProtocolHandshake.streamW3CProtocolParameters(ProtocolHandshake.java:238)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:104)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:141)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:604)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:244)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:131)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:144)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:178)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:167)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:124)
at com.dfweb.service.impl.PaymentLogServiceImplTest.testCalcuateProfitWithDifferentAgentAndUserOfInventory(PaymentLogServiceImplTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:233)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:87)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:176)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Suppressed: java.io.IOException: Incomplete document
at com.google.gson.stream.JsonWriter.close(JsonWriter.java:527)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:121)
… 37 more

 

Cause

I look through the com.google.guava jar source code, and found there is com.google.common.collect.ImmutableSet class existed, but there is no toImmutableSet() method defined.

It’s because ImmutableSet.toImmutableSet() method is introduced in Guava version 21.0, while I’m using version 20.0.

 

Solution

My current pom.xml

 

Change version to 21.0 and run mvn install  will fix this issue.

 

DeferredResult not working in Tomcat

The DeferredResult works on my local machine, but not work on the server machine.

On my local machine when a DeferredResult called setResult method, the response will return immediately. But on remote server machine, it only return the response to client after 30s (default timeout is 30s).

According to above posts, it may be caused by earlier version Tomcat 8 issue.

Then I tried jetty on remote server machine, it works. Next I checked the version by running CATALINA_BASE/bin/version.sh, it’s displaying

Server version: Apache Tomcat/8.0.14 (Debian)

And my local Tomcat version is 8.0.36

 

Then I upgrade the remote Tomcat version from 8.0.14 to 8.0.36 (upload my local tomcat to remote server) and try DeferredResult again, it works!

 

http://stackoverflow.com/questions/24619445/spring-async-deferredresult-not-working-in-tomcat-8