Go deep into servlet3

Deep into Servlet3

Servlet is the most widely used technology in j2ee platform, many developers know it, but not really understand it. In this article I would
like to share some more detailed understanding of it.

Response and socket connection.
----------------------------------
The response object encapsultes all information to be returned from the server to the client. A socket connection is a physical connection
between client and server. In HTTP 1.0, each reqeust/response must establish a new connection, when servelt return the response to the client,
the connection will be closed wa well. But HTTP 1.1 introduces persistent connection, in this case, when servlet return the response to
the client, the connection will not be closed, the client can send a new request on this connection until either side closes the connection
explicitly. Anyway we must know that resposne is'nt a abstract of connection in the servlet context.

- Response.flush(): will force the content in the buffer to be written to the client, the response will be considered as committed.
- Response.reset()/resetBuffer(): the reset() method clears the data in buffer when the response isn't committed. Headers and status code
set by the servlet prior to the reset call must be cleared as well. resetBuffer() will only reset the data in buffer.
- Response.setBufferSize(): The servlet can request a preferred buffer size, but the buffer assigned is not required to be the sieze requested
by the servlet, but must be at least as large as the size requested.

Committed??
----------------------------------
The isCommitted() method return a boolean value indicating whether any response bytes have been returned to client. A servlet container is
allowed, but not required, to buffer output going to the client for efficiency purpose.When using a buffer, the container must immediately
flush the contents of a filled buffer to the client. If this is the first data that is sent to client, the response is considered as committed.

To be successfully transmitted back to the client, headers must be set before the response is committed. Headers set after the response is
committed will be ignored by the servlet container.
Below methods will make the response to be committed:
- Response.flushBuffer()
- Response.getWriter().flush();
- Response.sendError(): The sendErro() method will set the appropriate headers and content body for an erro message to return to the client.
- Response.sendRedirect(): The sendRedirect() method will set the appropriate headers and content body to redirect the client to a different URL.
NOTE: sendErro() and sendRedirect() will have the side effect of committing the response, if it has not already committed,and terminating it.
No further outpu to the client should be made by the servlet after these methods are called. If data is written to the response after these
methods are called, the data will be ignored.

Closure??
----------------------------------
When a response is closed,the container must immediately flush all remaining content in the response buffer to the client. the following events
indicate that the servlet has satisfied the request and that the response object i to be closed:
- The termination of the service() method os the servlet.
- The amount of content specified in the setContentLength() method of the response has been greater than zero and has been written to the response.
- The sendError() method is called.
- The sendRedirect() method is called.
- The complete() method on AsyncContext is called.

Each response object is valid only within the scope of a servlet's service() method, or within the scope of a filter's doFilter() method, unless
the associated request object has asynchronous processing enabled for the component. If asynchronous processing on the associated reqeust is started,
then the reqeust object remains valid until complete() method on AsyncContext is called. Containers commonly recycle response objects in order to
avoid the performance overhead of response object creation.

Dispatcher forward/include.
----------------------------------
The Container Provider should ensure that the dispatch of the request to a target servlet occurs in the same thread of the same JVM as the original
request.
- RequestDispatcher.include(): It can ben called at any time. The target servlet of the include method has access to all aspects of the request
object, but its use of the response object is more limited.
It can only write information to the ServletOuputStream or Writer of the response object and commit a response by writing content past the end of
the response buffer, or by ecplicitly calling the flushBuffer() method of the ServletResponse interface. It cannot set headers or call any method
that affects the ehaders of the response.
When return from target servlet, you can return content in the buffer to the client if the response object isn't closed, or even set headers if the
response object isnot committed.
- RequestDispatcher.forward(): This method may be called by the calling servlet only when no output has been committed to the client. If outpu data
exists in the response buffer that has not been committed, the content must be cleared before the target servlet's service() method is called. If
the response has been committed, all IllegalStateException must be thrown.
When return from the target servlet, the response object is closed, you cannot apply any modification to it any more.

Asynchronous processing.
----------------------------------
Refer to http://www.javaworld.com/cgi-bin/mailto/x_java.cgi?pagetosend=/export/home/httpd/javaworld/javaworld/jw-02-2009/jw-02-servlet3.html&pagename=/javaworld/jw-02-2009/jw-02-servlet3.html&pageurl=http://www.javaworld.com/javaworld/jw-02-2009/jw-02-servlet3.html&site=jw_core
But when check the sourcecode of class SlowWebService which is a ServletContextListener, I found that there is a infinite loop in the contextInitiate()
method, that means when container try to lunch a web application, it will never be successful. I really don't understand why put a infinite loop block
there, my tests in jetty and tomcat both demonstrate that container will fall into the infinite loop and cannot go forward to finish the launch and cannot
be ready to serve any request from the client.

Using TOP more effectively (Repring)

Disclaimer: this article is published by Mulyadi Santosa, I find it from http://www.linuxforums.org/articles/using-top-more-efficiently_89.html

For desktop users, monitoring resource usage is an important task. By doing this, we can locate system bottleneck, planning what to do to optimize our system, identifying memory leak and so on. The problem is, which software one should use and how to use it according to our need.
Among many monitoring tools that available, most people use "top" (a part of procps package). Top provide almost everything we need to monitor our system's resource usage within single shot. In this article, all the information are based on procps 3.2.5 running on top of Linux kernel 2.6.x

Here, we assume that procps package is already installed and run well in your Linux system. No previous experience with top is needed here, but if you had given it a try briefly, that would be an advantage.

Here are some challenges:

A. Interactive or batch mode?
By default, top is invoked using interactive mode. In this mode, top runs indefinitely and accepts keypress to redefine how top works. But, sometimes you need to post-process the top's output and this is hardly achieved using this mode. The solution? Use batch mode.

$ top -b

You will get output like below:

top - 15:22:45 up 4:19, 5 users, load average: 0.00, 0.03, 0.00
Tasks: 60 total, 1 running, 59 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.8% us, 2.9% sy, 0.0% ni, 89.6% id, 3.3% wa, 0.4% hi, 0.0% si
Mem: 515896k total, 495572k used, 20324k free, 13936k buffers
Swap: 909676k total, 4k used, 909672k free, 377608k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 16 0 1544 476 404 S 0.0 0.1 0:01.35 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0
3 root 10 -5 0 0 0 S 0.0 0.0 0:00.11 events/0
Uh, wait, it runs repeatedly, just like interactive mode does. Don't worry, limit its repetition with -n. So, if you just want single shot, type:

$ top -b -n 1
The real advantage of this mode is you can easily combine in with at or cron. Together, top can snapshot resource usage at certain time unattendedly. For example, using at, we can schedule top to run 1 minute later.

$ cat ./test.at
TERM=linux top -b -n 1 >/tmp/top-report.txt
$ at -f ./test.at now+1minutes
Careful reader might ask "why do I need to set TERM environment before invoking top when creating new at job?". The answer is, top needs this variable set but unfortunately "at" isn't retained it from the time of invocation. Simply set it like above and top will work smoothly.

B. How to monitor certain processes only?
Sometimes, we are only interested on several processes only, maybe just 4 or 5 of the whole existing processes. For example, if you want monitor process identifier (PID) 4360 and 4358, you type:

$ top -p 4360,4358
OR
$ top -p 4360 -p 4358
Seems easy, just use -p and list all the PIDs you need, each separated with comma or simply use -p multiple times coupled with the target PID.

Another possibility is just monitoring process with certain user identifier (UID). For this need, you can use -u or -U option. Assuming user "johndoe" has UID 500, you can type:

$ top -u johndoe
OR
$ top -u 500
OR
$ top -U johndoe
The conclusion is, you can either use the plain user name or the numeric UID. "-u, -U? Those two are different?" Yes. Like almost any other GNU tools, options are case sensitive. -U means top will find matching effective, real, saved and filesystem UIDs, while -u just find matching effective user id. Just for reminder, every *nix process runs using effective UID and sometimes it isn't equal with real user ID. Most likely, one is interested in effective UID as filesystem permission and operating system capability are checked against it, not real UID.

While -p is just command-line option only, both -U and -u can be used inside interactive mode. Like you guess, press 'U' or 'u' to filter the processes based on their user name. Same rule is applied, 'u' for effective UID and 'U' for real/effective/saved/filesystem user name. You will be asked to enter the user name or the numeric UID.

{mospagebreak title=Fast or slow update?}
C. Fast or slow update?
Before we answer this question, let's take a short look on how top really works. Strace is your friend here:

$ strace -o /tmp/trace.txt top -b -n 1
Use you favourite text editor and load /tmp/trace.txt. What do you think? A lot of jobs for single invocation, that is what I think and maybe you'll agree. One of the jobs top must do in every iteration is opening many files and parsing their contents, as shown by the number:

$ grep open( /tmp/hasil.txt | wc -l
Just for illustration, in my Linux system, it yields 304. Closer look reveals that top iterates inside /proc directory to gather processes information. /proc itself is pseudo filesystem, meaning it doesn't exist on real disk but is created on the fly by the Linux kernel and live on RAM. Within directory such as /proc/2097 (2097 is a PID), Linux kernel exports information about related process and this is where top gathers processes information along with resource usage.

Also try these:

$ time top -b -n 1
This will give you illustration how fast top works on single round. In my system, this yields around 0.5-0.6 seconds. Look at "real" field, not the "user" or "system" field because "real" reflects the total time top needs to work.

So, realizing this fact, it will be wise to use moderate update interval. Browsing RAM based filesystem takes time too, so be wise. As rule of thumb, 1 to 3 seconds interval is enough for most users. Use -d in command line option or press "s" inside interactive mode to set it. You can use fractional number as interval, e.g: 2.5, 4.1 and so on

When we should faster than 1 seconds?

You need more samples during a time. For this need, better use batch mode and redirect standart output to a file so you can analyze it better.
You don't mind with extra CPU load carried by top. Yes, it is small but it is still a load. If your Linux system is relatively idle, feel free to use short interval, but if not, better preserve your CPU time for more important task.
One way to reduce top's work is by monitoring certain PIDs only. This way, top won't need to traverse all the /proc sub-directory. How about user name filtering? It won't do any good. User name filtering brings extra work for top, thus combining it with very short interval will increase CPU load.

Of course, whenever you need to force the update, just press Space and top will refresh the statistic right away.

{mospagebreak title=Fields we need}
D. Fields that we need
By default, top starts by showing the following task's property:

Field Description
PID : Process ID
USER : Effective User ID
PR : Dynamic priority
NI : Nice value, also known as base priority
VIRT : Virtual Size of the task. This includes the size of process's executable binary, the data area and all the loaded shared libraries.
RES : The size of RAM currently consumed by the task. Swapped out portion of the task is not included.
SHR : Some memory areas could be shared between two or more task, this field reflects that shared areas. The example of shared area are shared library and SysV shared memory.
S : Task status
%CPU : The percentage of CPU time dedicated to run the task since the last top's screen update.
%MEM : The percentage of RAM currently consumed by the task.
TIME+ : The total CPU time the task has been used since it started. "+" sign means it is displayed with hundreth of a second granularity. By default, TIME/TIME+ doesn't account the CPU time used by the task's dead children.
COMMAND : Showing program names.
But, there are more. Here, I will just explain fields that might interest you:

Field Description
nFLT (key 'u')
Number of major page fault since the process is started. Technically, page fault happens when the task access a non existant page in its address space. A page fault is said as "major" if kernel needs to access the disk to make the page available. On the contrary, soft minor page fault means the kernel only need to allocate pages in RAM without reading anything from disk.

For illustration, consider the size of program ABC is 8 kB and assume the page size is 4 kB. When the program is fully loaded to RAM, there will be 2 times major page fault (2 * 4 kB). The program itself allocates another 8 kB for temporary data storage in RAM. Thus, there will be 2 minor page fault.

A high number of nFLT could mean:

The task is aggressively load some portions of its executable or library from the disk.
The task is accessing a page that is swapped ou
It is normal if you see a high number of major page fault when a program is run for first time. On the next invocations, buffer is utilized so likely you will see "0" or low number of nFLT. But, if a program is continously triggerring major page fault, big chance your program needs larger RAM size than currently installed.

nDRT (key 'v')
The number of dirty pages since they are written back to the disk.

Maybe you wonder, what is dirty page? First, a little bac ground. As you know, Linux employ caching mechanism, so everything that is read from disk is also cached in RAM. The advantage of this action is, subsequent read to the same disk block can be served from RAM thus reading completes faster.

But it also costs something. If the buffer's content is modified, it needs to be synchronized. Thus, sooner or la this modified buffer (dirty page) must be written back. The failure on the synchronization might cause data inconsistency on related disk.

On mostly idle to fairly loaded system, nDRT is usually below 10 (this is just a raw prediction)or mostly zero. If it is constantly bigger than that:

The task is aggresively write something to file(s). It is so often that disk I/O can't keep up with it
The disk suffers I/O congestion, thus even the task only modifies small portion of file(s), it must wait a bit longer to be synchronized. Congestion happens when many processes access the disk at a time but cache hit is low.
These days, (1) unlikely happens because I/O speed is getting faster and less CPU demanding (thanks to DMA). So (2) has bigger probability.

Note: On 2.6.x, this field is always zero without unknown reason.

P (key 'j')
Last used CPU. This field only has meaning in SMP environment. SMP here refers to Hyperthread, multi core or true multi processor. If you just have one processor (non multi core, not HT), this field will always show '0'.

In SMP system, don't be surprised if this field change sometimes. That means, the Linux kernel tries to move your task to the other CPU which is considered less loaded.

CODE (key 'r') and DATA (key 's')
CODE simply reflects the size of your application code, while DATA reflects the size of data segment (stack, heap, variables but not shared libraries). Both are measured in kilobyte.

DATA is useful to show how much your application allocates memory. Sometimes, it can also reveal memory leaks. Of course, you need better tool such as valgrind to differentiate between repetitive memory allocation or growing memory leaks if DATA continously climbs up.

Note: DATA, CODE, SHR, SWAP, VIRT, RES are all measured in page size (4KB in Intel architecture). Read only data section is also calculated as CODE size, thus sometimes it is larger than the actual text (executable) segment.

SWAP (key 'p')
The size of swapped out portion of a task's virtual memory image. This field is sometimes confusing, here is why:

Logically, you would expect this field really shows whether your program is partially swapped out and how much. But the reality shows otherwise. Even the "Swap used" field shows 0, you will be surprised that SWAP field of each tasks show greater than zero number. So, what's wrong?

This comes from the fact that top use this formula:


VIRT = SWAP + RES or equal
SWAP = VIRT - RES
As explained previously, VIRT includes anything inside task's address space, no matter it is in RAM, swapped out or still not loaded from disk. While RES represents total RAM consumed by this task. So, SWAP here means it represents the total amount of data being swapped out OR still not loaded from disk. Don't be fooled by the name, it doesn't just represent the swapped out data.

To display the above fields, press 'f' inside the interactive mode. Then press the related key (mentioned above inside the parentheses). Those keys toggle the related fields, so press once to show it, press again to hide it. To find out whether the fields are displayed or not, simply watch the series of letters on the first line (at the right of "Current Fields"). Uppercase means the fields is shown, lower case means the opposite. Press Enter after you are satisfied with the selection.

Sorting use similar way. Press 'O' (upper case) followed by a key representing the field. Don't worry if you don't remember the key map, top will show it. The new sort key will be marked with asterisk and the letter will change to upper case, so you can notice it easily. Press Enter after you are finished

{mospagebreak title=Multi view are better than one?}
E. Multi view are better than one?
In different situations, sometimes we want to monitor different system property. For example, at one time you want to monitor %CPU and cpu time spent by all tasks. At another time, you want to monitor resident size and total page faults of all tasks. Rapidly press 'f' and change the visible fields? I don't think this is a smart choice.

Why don't you use Multiple Windows mode? Press 'A' (upper case) to switch to multi windows view. By default, you will see 4 different set of field groups. Each field groups has a default label/name:

1st field group: Def
2nd field group: Job
3rd field group: Mem
4th field group: Usr

1st field group is the usual group you see in single window view, while the rest are hidden. Inside multi window mode, press 'a' or 'w' to cycle through all the available windows. Pay attention, switching to another window also change the active window (also known as current window). If you are not sure which one is currently the active one, just look at the first line of top's display (at the left of current time field). Another way to change active window is by pressing 'G' followed by windows number (1 to 4).

Active window is the one which react to user input, so make sure to select your preferred window first before doing anything. After that, you can do anything exactly like you do in single window mode. Usually, what you want to do here is customizing field display, so just press 'f' and start customizing.

If you think 4 is too much, just switch to a field group and press '-' to hide it. Please note, even you hide current field group, that doesn't mean you also change the active group. Press '-' once again to make current group visible.

If you are done with multi window mode, press 'A' again. That also make active group as the new field group of single window mode.

F. "How come there is only so few free memory on my Linux PC?"
Come to same question? No matter how much you put RAM in your motherboard, you quickly notice the free RAM is reduced so fast. Free RAM miscalculation? No!

Before answering this, first check the memory summary located on the upper side of top's display (you may need to press 'm' to unhide it). There, you will find two fields: buffers and cached. "Buffers" represent how much portion of RAM is dedicated to cache disk block. "Cached" is similar like "Buffers", only this time it caches pages from file reading. For thorough understanding of those terms, refer to Linux kernel book like Linux Kernel Development by Robert M. Love.

It is enough to understand that both "buffers" and "Cached" repre- sent the size of system cache. They dynamically grow or shrink as requested by internal Linux kernel mechanism.

Besides consumed by cache, the RAM itself is also occupied by application data and code. So, to conclude, free RAM size here means RAM area that isn't occupied by cache nor application data/code. Generally, you can consider cache area as another "free" RAM since it will be shrunk gradually if the application demands more memory.

On the task point of view, you might wonder which field truly represent memory consumption. VIRT field? certainly not! Recall that this field represent everything inside task address space, including the related shared libraries. After reading top source code and proc.txt (inside Documentation/filesystem folder of kernel source's tree), I conclude that RSS field is the best field describing task's memory consumption. I said "best" because you should consider it as approximation and isn't 100% accurate on all time.

G. Working with many saved configurations
Wanna keep several different configuration of top so you can easily switch between preconfigured display? Just create symbolic link to the top binary (name it anything you like:

# ln -s /usr/bin/top /usr/bin/top-a
Then run the new "top-a". Do the tweak and press 'W' to save the configuration. It will be saved under ~/.top-arc (the format is your top alias name+'rc').

Now run the original top to load your first display alternative, top-a for the second one and so on.

{mospagebreak title=Conclusion}
H. Conclusion
There are numerous tricks to use top more efficiently. The key is by knowing what you really need and possibly a little good understanding of Linux low level mechanism. The statistics isn't always correct, but at least it is helpful as a overall measurement. All these numbers are gathered from /proc, so make sure it is mounted first!

Reference:

Understanding The Linux Kernel, 2nd edition.
Documentation/filesystems/proc.txt inside kernel source tree.
Linux kernel source.

Dive into Spring test framework - Part1

I don't wanna repeat that how important is unittest to a developer, but in my expirence I found developers around me seldom write unit test, to say nothing of the spring test framework.
I would like to share my knowledge about how to perform tests based on spring test framework, also it is a reminder to myself to keep walking.

Spring framework
In current world, almost all running or developing systems will follow MVC architectural style, in java world we will separate a system into about 4 layers: 'controller', 'service', 'domain', 'DAO'.
Let me ask you a question, what is the biggest challenge when testing a 'DAO'? It is how to make database stay same state with before runnig test. If the state changed from the initial state of before running test, the tests will influence each other. When you run a single test again and again, you will get diffrent result, absolutely we should avoid it.
By adopting spring test framework, we can easily rollback the transaction when finish a test. Actually spring will automatically rollback it(of course you can ask spring commit it), and each testcase will be the boundary of a transaction.
One more question, do we need Mocker? it depends. When we test a service, do we need to mock a DAO instance then no need to access real database? no, I would like to always run tests against real database. OK, maybe you will say that's integration test, not unit test...the difference here isn't important, actually if the integration test passed, of course the unittest test(a Mock DAO) will pass.

import java.math.BigDecimal;
import java.sql.Connection;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import java.util.UUID;

import javax.sql.DataSource;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.dbunit.database.DatabaseConfig;
import org.dbunit.database.DatabaseConnection;
import org.dbunit.database.IDatabaseConnection;
import org.dbunit.ext.oracle.OracleDataTypeFactory;
import org.springframework.test.AbstractTransactionalDataSourceSpringContextTests;

import com.mpos.lottery.te.common.encrypt.RsaCipher;
import com.mpos.lottery.te.config.MLotteryContext;
import com.mpos.lottery.te.config.dao.OperationParameterDao;
import com.mpos.lottery.te.config.dao.SysConfigurationDao;
import com.mpos.lottery.te.config.domain.LottoOperationParameter;
import com.mpos.lottery.te.config.domain.SysConfiguration;
import com.mpos.lottery.te.config.domain.logic.GameTypeBeanFactory;
import com.mpos.lottery.te.draw.dao.FunTypeDao;
import com.mpos.lottery.te.draw.dao.GameDrawDao;
import com.mpos.lottery.te.draw.domain.Game;
import com.mpos.lottery.te.draw.domain.GameDraw;
import com.mpos.lottery.te.draw.domain.LottoFunType;
import com.mpos.lottery.te.hasplicense.HASPManage;
import com.mpos.lottery.te.settlement.domain.SettlementReport;
import com.mpos.lottery.te.test.unittest.BaseUnitTest;
import com.mpos.lottery.te.ticket.domain.LottoEntry;
import com.mpos.lottery.te.ticket.domain.Ticket;
import com.mpos.lottery.te.ticket.domain.logic.lotto.BankerStrategy;
import com.mpos.lottery.te.ticket.domain.logic.lotto.BetOptionStrategy;
import com.mpos.lottery.te.ticket.domain.logic.lotto.MultipleStrategy;
import com.mpos.lottery.te.ticket.domain.logic.lotto.RollStrategy;
import com.mpos.lottery.te.ticket.domain.logic.lotto.SelectedNumber;
import com.mpos.lottery.te.ticket.domain.logic.lotto.SingleStrategy;
import com.mpos.lottery.te.trans.domain.Transaction;
import com.mpos.lottery.te.workingkey.domain.WorkingKey;

/**
 * Looks like the test framework of spring for JPA only support JUnit 3.8...not
 * ready for Junit4.X
 */
public class BaseTransactionTest extends AbstractTransactionalDataSourceSpringContextTests {
 protected Log logger = LogFactory.getLog(BaseTransactionTest.class);
 public static final String DATE_FORMAT = "yyyyMMddHHmmss";

 /**
  * About onXXX inherited from AbstractTransactionSprintContextTests for
  * onSetUp() and onTearDown(), AbstractTransactionSprintContextTests has
  * implemented logic which adopted template method pattern to invoke
  * onSetUpXXX and onTearDownXXX. It means if you want override onSetUp() and
  * onTearDown(), super.OnSetUp() and super.OnTearDown() must be invoked,
  * otherwise spring won't invoke onSetUpXXX and onTearDownXXX(). You can
  * completely override onSetUpXXX() and onTearDownXXX(), they are template
  * methods.
  * @see org.springframework.test.AbstractTransactionalSpringContextTests#onSetUp
  * @see org.springframework.test.AbstractTransactionalSpringContextTests#onSetUpBeforeTransaction
  * @see org.springframework.test.AbstractTransactionalSpringContextTests#onSetUpInTransaction
  * @see org.springframework.test.AbstractTransactionalSpringContextTests#onTearDownInTransaction
  * @see org.springframework.test.AbstractTransactionalSpringContextTests#onTearDownAfterTransaction
  * @see org.springframework.test.AbstractTransactionalSpringContextTests#onTearDown
  */
 public void onSetUp() throws Exception {
  logger.info("------------------- onSetUp -------------------");
  HASPManage.isChecked = false;
  logger.info("Disable HASP key...");
  this.initializeMLotteryContext();
  // must invoke super.onSetUp(), new transaction will be created here,
  // also will invoke template methods in order
  super.onSetUp();
 }

 public void onTearDown() throws Exception {
  logger.info("------------------- onTearDown -------------------");
  // must invoke super.onTearDown(), transaction will be rolled back
  // here, also will invoke template methods in order.
  super.onTearDown();
 }

 public void onSetUpBeforeTransaction() throws Exception {
  logger.info("------------------- onSetUpBeforeTransaction -------------------");
  // Oops, i found use a sql file to load data into database is simpler
  // that DBUnit,
  // as we can query database directlly. If by DBUnit, we can query
  // database only when committing a trasaction.

  // use DBUnit to cleanup data first, or we can use
  // this.deleteFromTables(String[] tableNames);
  // IDatabaseConnection conn = this.getDataBaseConnection();
  // try {
  // // when delete, DBUnit will execute from last table to first table,
  // // be
  // // opposed to INSERT.
  // DatabaseOperation.DELETE.execute(conn, new
  // FlatXmlDataSetBuilder().build(this
  // .getClass().getResourceAsStream("/testdata/oracle_test_union.xml")));
  // DatabaseOperation.DELETE.execute(conn, new FlatXmlDataSetBuilder()
  // .build(new InputSource(this.getClass().getResourceAsStream(
  // "/testdata/oracle_test_common.xml"))));
  // DatabaseOperation.INSERT.execute(conn, new FlatXmlDataSetBuilder()
  // .build(new InputSource(this.getClass().getResourceAsStream(
  // "/testdata/oracle_test_common.xml"))));
  // DatabaseOperation.INSERT.execute(conn, new
  // FlatXmlDataSetBuilder().build(this
  // .getClass().getResourceAsStream("/testdata/oracle_test_union.xml")));
  // } finally {
  // // return the connection to pool
  // DataSourceUtils.releaseConnection(conn.getConnection(),
  // this.getJdbcTemplate()
  // .getDataSource());
  // }
 }

 public void onTearDownAfterTransaction() throws Exception {
  this.logger.info("------------------- onTearDownAfterTransaction -------------------");
  // // use DBUnit to cleanup data, or we can use
  // // this.deleteFromTables(String[] tableNames);
  // IDatabaseConnection conn = this.getDataBaseConnection();
  // try {
  // // when delete, DBUnit will execute from last table to first table,
  // // be
  // // opposed to INSERT.
  // DatabaseOperation.DELETE.execute(conn, new
  // FlatXmlDataSetBuilder().build(this
  // .getClass().getResourceAsStream("/testdata/oracle_test_union.xml")));
  // DatabaseOperation.DELETE.execute(conn, new FlatXmlDataSetBuilder()
  // .build(new InputSource(this.getClass().getResourceAsStream(
  // "/testdata/oracle_test_common.xml"))));
  // } finally {
  // // return the connection to pool
  // DataSourceUtils.releaseConnection(conn.getConnection(),
  // this.getJdbcTemplate()
  // .getDataSource());
  // }
  logger.info("*** Finished cleanup test data ***");
 }

 public void onSetUpInTransaction() throws Exception {
  logger.info("------------------- onSetUpInTransaction -------------------");
  // this.executeSqlScript("testdata/oracle_masterdata.sql", false);
  // this.executeSqlScript("/testdata/oracle_testdata.sql", false);
  // this.executeSqlScript("/testdata/oracle_testdata_union.sql", false);

  /**
   * NOTE: In the original implementation, I invoke DBUnit.INSERT in
   * onSetUpInTrransaction() and DBUnit.DELETE in
   * onTearDownInTransaction(), it means DBUnit.INSERT and DBUnit.DELETE
   * will be executed in the lifecycle of test transaction managed by
   * Spring, then there is a chance that Spring test transaction will
   * conflict with DBUnit transaction. Here is a case: 1) Sprint create a
   * new transaction for testcase. 2) DBUnit.INSERT test data in
   * onSetUpInTransaction(a new auto-commit transaction) 3)
   * "this.getJdbcTemplate().execute('update GPE_KEY...'" which will
   * update GPE_KEY in test transaction. 4) run test case. 5)
   * DBUnit.DELETE test data in onTearDownInTransaction(a new auto-commit
   * transaction). when DBUnit try to delete from GPE_KEY, it will be
   * blocker forever, as the test transaction has hold the exclusive lock
   * of row of GPE_KEY, and only will release after DBUnit.DELETE.... My
   * conclusion is if we plan to use DBUnit in separated transaction, it
   * is better to invoke DBUnit in onSetUpBeforeTransaction() and
   * onTearDownAfterTransaction(). By this mean all transactions won't
   * influence one another. The other solution is if use DBUnit in
   * onTearDownInTransaction, we should invoke "this.endTransaction()"
   * first which will rollback/commit transaction.
   */
  SimpleDateFormat sdf = new SimpleDateFormat(WorkingKey.DATE_PATTERN);
  this.getJdbcTemplate().execute(
          "update GPE_KEY set create_time=sysdate,update_time=sysdate,create_date='"
                  + sdf.format(new Date()) + "'");

  logger.info("*** Finished preparing test data ***");
 }

 public void onTearDownInTransaction() throws Exception {
  logger.info("------------------- onTearDownInTransaction -------------------");
 }

 protected IDatabaseConnection getDataBaseConnection() throws Exception {
  DataSource ds = this.getJdbcTemplate().getDataSource();

  /**
   * Will retrieve connection from current transaction context, but due to
   * TE will query te_sequence in a new connection, and at the same time
   * DBUnit doesn't commit transaction(managed by Spring) yet, TE will
   * fail to get sequence record. So DBUnit use a new connection which is
   * different from the connection associated with spring transaction
   * context to manipulate data. The disadvantage is you have to delete
   * all those data when finish a test case and close connection manually.
   */
  // Connection connection = DataSourceUtils.getConnection(ds);
  Connection connection = ds.getConnection(); // a auto commit connection

  // must set schema if the database user is DBA.
  // Refer to com.mpos.lottery.te.test.util.DBUnitUtils
  IDatabaseConnection conn = new DatabaseConnection(connection, "RAMONAL");
  DatabaseConfig dbConfig = conn.getConfig();
  dbConfig.setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY, new OracleDataTypeFactory());

  return conn;
 }

 @Override
 protected String[] getConfigLocations() {
  /**
   * If there are beans with same name in different configuration files,
   * the last bean definition will overwrite the previous one. When do
   * integration test, this feature will be a good facility. By defining a
   * separated test spring configuration file, we can get a test
   * environment, but no need to modify normal spring configuration file
   * which will manage the production environment.
   */
  return new String[] { "spring-service.xml", "spring-dao.xml", "spring-eig.xml",
          "spring-raffle.xml" };
 }

 /**
  * Convert java.util.Date to string, then compare the string of date. Due to
  * the long value of java.util.Date is different from the long value of
  * java.util.Date retrieved from database.
  */
 protected String date2String(Date date) {
  assert date != null : "Argument 'date' can not be null.";
  SimpleDateFormat sdf = new SimpleDateFormat(DATE_FORMAT);
  return sdf.format(date);
 }

 protected String uuid() {
  UUID uuid = UUID.randomUUID();
  String uuidStr = uuid.toString();
  return uuidStr.replace("-", "");
 }

 protected SysConfiguration getSysConfiguration() {
  SysConfigurationDao dao = this.getBean(SysConfigurationDao.class, "sysConfigurationDao");
  return dao.getSysConfiguration();
 }

 protected String encryptSerialNo(String serialNo) {
  String tmp = serialNo;
  if (this.getSysConfiguration().isEncryptSerialNo()) {
   tmp = RsaCipher.encrypt(BaseUnitTest.RSA_PUBLIC_KEY, serialNo);
  }
  return tmp;
 }

 protected void printMethod() {
  StringBuffer lineBuffer = new StringBuffer("+");
  for (int i = 0; i < 80; i++) {
   lineBuffer.append("-");
  }
  lineBuffer.append("+");
  String line = lineBuffer.toString();

  // Get the test method. If index=0, it means get current method.
  StackTraceElement eles[] = new Exception().getStackTrace();
  // StackTraceElement eles[] = new Exception().getStackTrace();
  // for (StackTraceElement ele : eles){
  // System.out.println("class:" + ele.getClassName());
  // System.out.println("method:" + ele.getMethodName());
  // }
  String className = eles[1].getClassName();
  int index = className.lastIndexOf(".");
  className = className.substring((index == -1 ? 0 : (index + 1)));

  String method = className + "." + eles[1].getMethodName();
  StringBuffer padding = new StringBuffer();
  for (int i = 0; i < line.length(); i++) {
   padding.append(" ");
  }
  System.out.println(line);
  String methodSig = (method + padding.toString()).substring(0, line.length() - 3);
  System.out.println("| " + methodSig + "|");
  System.out.println(line);
 }

 protected  T getBean(Class c, String beanName) {
  return (T) this.getApplicationContext().getBean(beanName, c);
 }

 protected void initializeMLotteryContext() {
  MLotteryContext.getInstance().setBeanFactory(this.getApplicationContext());
 }

 protected GameDraw getGameInstance(String drawNo, String gameId) {
  GameDrawDao drawDao = this.getBean(GameDrawDao.class, "gameDrawDao");
  GameDraw draw = drawDao.getByNumberAndGame(drawNo, gameId);
  FunTypeDao funTypeDao = this.getBean(FunTypeDao.class, "lottoFunTypeDao");
  LottoFunType funType = (LottoFunType) funTypeDao.getById(draw.getGame().getFunTypeId());
  draw.getGame().setFunType(funType);
  return draw;
 }

 protected BigDecimal calculateTicketAmount(Ticket ticket) throws Exception {
  LottoFunType funType = (LottoFunType) ticket.getGameDraw().getGame().getFunType();
  List entries = ticket.getEntries();
  long totalBets = 0;
  for (LottoEntry entry : entries) {
   int betOption = entry.getBetOption();
   String numberFormat = MLotteryContext.getInstance().getLottoNumberFormat(betOption);

   BetOptionStrategy strategy = null;
   if (betOption == LottoEntry.BETOPTION_SINGLE) {
    strategy = new SingleStrategy(numberFormat, funType);
   } else if (betOption == LottoEntry.BETOPTION_MULTIPLE) {
    strategy = new MultipleStrategy(ticket, numberFormat, funType);
   } else if (betOption == LottoEntry.BETOPTION_BANKER) {
    strategy = new BankerStrategy(numberFormat, funType);
   } else if (betOption == LottoEntry.BETOPTION_ROLL) {
    strategy = new RollStrategy(numberFormat, funType);
   }

   SelectedNumber sNumber = new SelectedNumber();
   String numberParts[] = entry.getSelectNumber().split(SelectedNumber.DELEMETER_BASE);
   sNumber.setBaseNumber(numberParts[0]);
   if (numberParts.length == 2) {
    sNumber.setSpecialNumber(numberParts[1]);
   }
   sNumber.setBaseNumbers(parseNumberPart(sNumber.getBaseNumber()));
   sNumber.setSpecialNumbers(parseNumberPart(sNumber.getSpecialNumber()));
   totalBets += strategy.getTotalBets(sNumber);
  }
  // get base amount
  Game game = ticket.getGameDraw().getGame();
  OperationParameterDao opDao = GameTypeBeanFactory.getOperatorParameterDao(game.getType());
  LottoOperationParameter lop = (LottoOperationParameter) opDao.getById(game
          .getOperatorParameterId());
  return lop.getBaseAmount().multiply(new BigDecimal(totalBets));
 }

 protected int[] parseNumberPart(String numberPart) {
  if (numberPart == null)
   return null;
  String strNumbers[] = numberPart.split(SelectedNumber.DELEMETER_NUMBER);
  int numbers[] = new int[strNumbers.length];
  for (int i = 0; i < numbers.length; i++) {
   numbers[i] = Integer.parseInt(strNumbers[i]);
  }
  // Sorts the specified array of integers into ascending numerical order.
  Arrays.sort(numbers);
  return numbers;
 }
}

As we know, in a web application, Spring context will be stored in Servlet context, so we override the createApoplicationContext() method to meet our requirement, please check the method's comment.
Now it is time to show a example, let's there is a servlet named HttpDispatchServlet, and we will write test case for it. Look at the sprint-service.xml first, as our test case will extend from BaseServletTest which extends from BaseTransactionTest, we must define a DataSource typed bean in spring context(spring will inject the DataSource instance into BaseTransactionTest automatically)

1 <beans ...=""> 2 <bean class="net.mpos.lottery.httprmi.DefaultBookService" id="bookService"></bean> 3 <bean class="org.springframework.jdbc.datasource.DriverManagerDataSource" id="dataSourceTarget"> 4 <property name="driverClassName" value="oracle.jdbc.driver.OracleDriver"> 5 <property name="url" value="jdbc:oracle:thin:@192.168.2.9:1521/orcl"> 6 <property name="username" value="ramonal"> 7 <property name="password" value="ramonal"> 8 </property></property></property></property></bean> 9 <bean class="net.mpos.lottery.spring.MyDelegatingDataSource" id="dataSource"> 10 <property name="targetDataSource"> 11 <ref local="dataSourceTarget"></ref> 12 </property> 13 </bean> 14 <bean class="org.springframework.jdbc.datasource.DataSourceTransactionManager" id="transactionManager"> 15 <property name="dataSource"> 16 <ref bean="dataSource"></ref> 17 </property> 18 </bean> 19 </beans>

When initialize spring context, you will get a exception: two intances of DataSource type...Here I will override the setDataSource() method in BaseTransactionTest class to fix it.

/**
 * As there are two DataSource typed instance in Spring context, we must
 * override the parent method to set the Qualifier. 
 */
public void setDataSource(@Qualifier("dataSource") DataSource dataSource){
    super.setDataSource(dataSource);
}

Let's look at HttpDispatchServletTest, how to implement it...

public class HttpDispatchServletTest extends BaseServletTest {
    private HttpDispatchServlet servlet;
    
    public void mySetUp() throws Exception{
        super.mySetUp();
        servlet = new HttpDispatchServlet();
        servlet.init(config);
    }

    @Test
    public void testDoPost_BookService_add_Encryption() throws Exception {
        printMethod();
        GSonUtils gson = new GSonUtils();
        String reqContent = gson.toJson(DomainMocker.mockBook());
        request.addHeader(HttpDispatchServlet.HEADER_RMI_TAG, "bookService.add");
        request.addHeader(EncryptionHttpPackInterceptor.HEADER_MAC, HMacMd5Cipher.doDigest(
        reqContent, HMacMd5CipherTest.MAC_KEY));
        reqContent = TriperDESCipher.encrypt(TriperDesCipherTest.DES_KEY, reqContent);
        request.setContent(reqContent.getBytes());
        servlet.doPost(request, response);
        // assert response
        int status = response.getStatus();
        assertEquals(200, status);
    } 

    public void myTearDown(){
        super.myTearDown();
        servlet = null;
    }
}

OK, now you can run it from eclipse by 'run as junit test', No need tomcat.

---

JMX, JVisualVM

监控对系统来说是个非常重要的功能，比如如果你的系统能够显示一个动态变化的每10秒刷新一次的实时销量趋势图，客户一定会非常开心，事实上这个功能也是非常重要的，不是一个玩具。
我最近一直在想这个问题，因为这段时间维护系统开始从一个新的角度去看系统开发，开始更多的考虑系统可维护性的问题（对于多数开发者他们只关注开发阶段的问题，但是一个软件系统的生命周期中90％都处于维护阶段），其中监控维护者（系统监控）和客户（偏重业务监控）都需要的工具。
自然而然的我就想到了JMX，这个技术已经出现很长时间了，有成熟的规范，有丰富的资料。但是我仍然再想为什么要用JMX来实现监控呢？ 有什么优势？我自己定义web service，也一样可以暴露监控和管理接口，如果这样来做那就是轻车熟路，技术上没有门槛。而JMX，对于一套专属的系统来说，规范有多少意义，这些接口只有管理系统可以看到，我宁愿采用定制的，轻型的，一切自己掌控的方案。 如果说因为这是JCP的规范，所以我们应该采用JMX，老实说，理由很牵强。
但是我还是看了一遍sun的（O，oracle的）JMX tutorial，现在我想我有了其他的理由，这些理由比JCP更有说服力。
1）与JVisualVM的直接整合。
2）JDK自带的，无需第三方包
3）很容易，学习曲线非常低

1。 与JVisualVM的直接整合
这是一个很酷的特性，我直接开始想到真实系统的应用场景。系统管理员可以用JVisualVM来监控和管理系统，而定制开发的web系统来管理业务数据，这样，我们只需要写服务器端的代码，而客户端就直接是JVisualVM,简单轻松。看看我实验的代码（来自sun jmx tutorial）

* 定义一个StandardMBean（JMX规范中，接口名以MBean结束，比如XXXMBean,而对应的实现类直接去掉MBean就可以了，约定优于配置）
package org.jmx;
public interface HelloMBean {
public void sayHello();
public int add(int x, int y);
public String getName();
public int getCacheSize();
public void setCacheSize(int size);
}

package org.jmx;
import javax.management.AttributeChangeNotification;
import javax.management.MBeanNotificationInfo;
import javax.management.Notification;
import javax.management.NotificationBroadcasterSupport;
public class Hello extends NotificationBroadcasterSupport implements HelloMBean {
public void sayHello() {
System.out.println("hello, world");
}
public int add(int x, int y) {
return x + y;
}
public String getName() {
return this.name;
}
public int getCacheSize() {
return this.cacheSize;
}
public synchronized void setCacheSize(int size) {
int oldSize = this.cacheSize;
this.cacheSize = size;
System.out.println("Cache size now " + this.cacheSize);
Notification n = new AttributeChangeNotification(this, sequenceNumber++, System
.currentTimeMillis(), "CacheSize changed", "CacheSize", "int", oldSize,
this.cacheSize);
sendNotification(n);
}

@Override
public MBeanNotificationInfo[] getNotificationInfo() {
String[] types = new String[] { AttributeChangeNotification.ATTRIBUTE_CHANGE };
String name = AttributeChangeNotification.class.getName();
String description = "An attribute of this MBean has changed";
MBeanNotificationInfo info = new MBeanNotificationInfo(types, name, description);
return new MBeanNotificationInfo[] { info };
}

private final String name = "Reginald";
private int cacheSize = DEFAULT_CACHE_SIZE;
private static final int DEFAULT_CACHE_SIZE = 200;
private long sequenceNumber = 1;
}

在实现类中继承了NotificationBroadcasterSupport ，这样可以向注册的listener发送通知，后来可以看到我从客户端注册了一个listener，这样服务器端发送通知之后，客户端就可以收到（好像在JVisualVM的场景中没有什么作用，但是自定义客户端则非常重要）。

* 启动Server端
package org.jmx;

import java.lang.management.ManagementFactory;
import javax.management.MBeanServer;
import javax.management.ObjectName;
public class Main {
public static void main(String[] args) throws Exception {

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); //1
ObjectName name = new ObjectName("com.example.mbeans:type=Hello");
Hello mbean = new Hello();
mbs.registerMBean(mbean, name); //2
System.out.println("Waiting forever..."); //3
Thread.sleep(Long.MAX_VALUE);
}
}

其中 //1 是获得平台platformMBeanServer, JVisualVM会连接到这个MBeanServer，当然也可以通过MBeanServerFactory来创建自己的MBeanServer。
//2 注册MBean
//3 main thread进入sleep状态，RMI connector server应该是有一个新的线程

* 客户端实现
package org.jmx;

import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

public class MyJConsole {
public static void main(String args[]) {
try {
// "jmxrmi" is the default service name registered by PlatformMBeanServer, which can connected
// by JVisualVM and JConsole directly.
JMXServiceURL url = new JMXServiceURL(
"service:jmx:rmi:///jndi/rmi://localhost:4949/jmxrmi"); //1
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();

// register a listener
System.out.println("Register a remote listener.");
ObjectName name = new ObjectName("com.example.mbeans:type=Hello");
mbsc.addNotificationListener(name, new HelloListener(), null, null); //2

System.out.println("Waiting forever...");
Thread.sleep(Long.MAX_VALUE);
} catch (Exception e) {
e.printStackTrace();
}
}
}

//1 实例化一个JMXServiceURL ， 这个url的格式可以参考JMXServiceURL 的javadoc。
//2 注册客户端listener到远程MBeanServer。 注意所有的类如果实现了Remote和NotificationListener接口，那么在远程传输的时候是传引用而不是值，所以这里listener的逻辑都会在client执行，不是在server端执行。

* client listener
package org.jmx;

import javax.management.AttributeChangeNotification;
import javax.management.Notification;
import javax.management.NotificationListener;

public class HelloListener implements NotificationListener {

@Override
public void handleNotification(Notification notification, Object handback) {
if (notification instanceof AttributeChangeNotification){
AttributeChangeNotification attNotification = (AttributeChangeNotification)notification;
System.out.println("change CacheSize from " + attNotification.getOldValue()
+ " to " + attNotification.getNewValue());
}
else {
System.out.println(notification);
}
}
}

开始执行这些代码。
1） 启动server端
>> java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=4949 -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=10.40.0.101
-Dcom.sun.management.jmxremote.authenticate=false org.jmx.Main (还有制定classpath一类的)
* 注意这里的-Djava.rmi.server.hostname=10.40.0.101， 实际上client跑在32bit的win7操作系统上，而server是64bit的linux操作系统， 如果不指定这个属性来绑定4949端口到network interface 10.40.0.101上，那么client可能无法连接到server。 因为client的机器配置为只能访问server的内网ip(10.40.0.101)，所以必须绑定server端的VM在10.40.0.101上监听4949端口。
Waiting forever...

2)启动客户端
>> java org.jmx.MyJConsole
Register a remote listener.
Waiting forever...

3) 启动JVisualVM，从左边窗口可以看到一个‘Local’节点，右键点击，选择‘add JMX connection’, 在conntion中输入‘service:jmx:rmi:///jndi/rmi://localhost:4949/jmxrmi’。你会看到新建的这个节点，双击来展开右边的tab pane， 选择‘MBeans，然后你就可以看到自己注册的MBean ’com.example.mbeans:type=Hello‘。 双击节点’Hello‘，在右边选择’Attributes‘ ，并且编辑’cacheSize‘的值为444。
然后来检查server和client的控制台有什么输出。
Server： Cache size now 444
Client： change CacheSize from 333 to 444

怎么样？ 还是很值得在项目中采用JMX的吧！


Now I finished a example which use Spring to expose JMX to JvisualVM, my test environment: tomcat7.0.29 + spring2.5.6
1) Define Spring in web.xml:

xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
id="WebApp_ID" version="3.0">

Tomcat7





contextConfigLocation
classpath:spring-service.xml



org.springframework.web.context.ContextLoaderListener



2) Implement a simple POJO:
package org.tomcat7;

public class HelloBean {
private int age;
private String name;

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public void append(String suffix) {
this.name += suffix;
}
}

3) Export HelloBean by Spring(spring-service.xml):

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:context="http://www.springframework.org/schema/context"
xmlns:jee="http://www.springframework.org/schema/jee" xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd
http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd">


lazy-init="false">











4) Deploy web application to tomcat and launch it.
NOTE: modify catelina.sh to enable remote JMX connection(-Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false )

5) Start JvisualVM, and connect to localhost:3333.
Now from 'MBean' tab, you will find our HelloBean, then you can moniter states and invoke operations of it.

The flush mode and behaviour of Hibernate

A few days ago, I found some strange problem from TE's log. There is a service names 'sell ticket', which implement below logic，I will present them in pseudocode.

1. TransactionDao.getByDevAndTraceMessageId(X,Y)
2. TransactionDao.insert(trans)
3. SettlementReportDao.getByOperatorAndBatchNoAndMerchant(X,Y,Z)
4. GameInstanceDao.getByGameIdAndDrawNo(X,Y)
5. MerchantDao.update(X) //1
6. MerchantDao.update(Y) //2
7. TicketDao.insert(X)
8. LottoEntryDao.insert(X) //1
9. LottoEntryDao.insert(Y) //2
10. LottoEntryDao.insert(Z) //3
11. print log 'finish saving entries'
12. TransactionDao.update(trans)
13. commit transaction.
14. print log 'Start to sync transaction log'

Some long transaction spend about 30 seconds from stop#11 to step#14. As my understanding, from step#11 to step#14 only update te_transaction and then commit transaction, why take so long time?
I have below two considerations:

1. when update transaction, this transaction is blocked.
2. when commit transaction, it will cost much

For point#1, it should not be blocked, cause at step#2, this transaction should already captured the lock. For point#2, if all sql statements have been sent to database, then committing should be very quick.
Why?

I understand hibernate in a wrong way which cause I get wrong conclusion. Let's think about the flush behaviour of hibernate. Each time when we call a Dao to insert,update,delete a entry, it won't hit database immediately, hibernate will record these actions, update entity states. When do flush, hiberate will send all these sql to database. Remember below points:

• A select statement will always hit database to retrieve latest data from database, and update the related entity in cache(it should be 2nd level cache, which is managed by hibernate, so hibernate runtime is responsible for ensuring that all updates to the state of all entities in the persistence context which could potentially affect the result of the query are visible to the processing of the query.
   ..actually it isn't always true, there are 2 exceptions. #1)Query entity by id(entity identifier). for example if insert a entity first, then query that entity by ID, hibernate won't hit the underlying database, as the inserted entity is completedly managed by hibernate runtime. #2)If insert a entity, and then query that entity from outside of hibernate runtime, such as direct JDBC. In this case, as hibernate runtime has no awareness of the querying, it won't flush the insert/update/delete SQL to underlying database. ).
• When execute a select query, hibernate will check if some previous actions(save, update,delete) have changed the status of queried entity(if changed, hibernate marked it as dirty). If so, hibernate will flush all update actions of entity to underlying database.
• Hibernate will organize all actions into 'insert', 'update','delete' category. When do flushing, hibernate will perform all actions in order 'insert', 'update', 'delete'. It means maybe you invoke a update method before a insert, but in fact, insert will be performed first.

Then what is flush? when hibernate will flush? Flush will sync the entity state between hibernate cache and database. In case of below condition, hibernate will perform flush(We are talking about FlushMode.AUTO).

• Before select, and some previous actions have changed the queried entity state. Image it, if a transaction query a entity(E1), then change some fields(E2), then query this entity again. As query will always hit database, if doesn't flush, this query will get E1. Obviously it is incorrect, that is why hibernate will perform flush before the second query.
• Before committing transaction.
• Manualy invoke flush() method.

....

"When execute a select enquiry, hibernate will check if some previous actions(save, update,delete) will change the status of queried entity." I will further explain what does this statement means. Lets say there are 2 entities: WorkingKey and Payout(these 2 entities have no any relationship), and:
1) insert a new Payout entity
2) query WorkingKey by some criteria
3) commit transaction.
The test code as below:

@Test
public void testHibernateFlush() throws Exception{
  Payout p = DomainMocker.mockPayout();
  p.getTransaction().setId("TRANS-8111");
  logger.debug("before insert payout...");
  this.getPayoutDao().insert(p);
  logger.debug("before querying working key...");
  this.getWorkingKeyDao().getWorkingKey("20101216", "GPE-111");
  this.setComplete();
}

Ok, what will be printed out? -------------

[2011-01-17 15:39:08,984][main][DEBUG][BaseTransactionTest] before insert payout...
[2011-01-17 15:39:09,046][main][DEBUG][BaseTransactionTest] before querying working key...
[2011-01-17 15:39:09,359][main][INFO ][sqltiming] select workingkey0_.ID as ID24_, workingkey0_.CREATE_TIME as CREATE2_24_, workingkey0_.UPDATE_TIME as UPDATE3_24_, workingkey0_.VERSION as VERSION24_, workingkey0_.CREATE_DATE as CREATE5_24_, workingkey0_.DATA_KEY as DATA6_24_, workingkey0_.GPE_ID as GPE7_24_, workingkey0_.MAC_KEY as MAC8_24_ from GPE_KEY workingkey0_ where workingkey0_.CREATE_DATE='20101216' and workingkey0_.GPE_ID='GPE-111' {executed in 0 msec}
[2011-01-17 15:39:09,375][main][INFO ][sqltiming] insert into PAYOUT (CREATE_TIME, UPDATE_TIME, VERSION, TOTAL_AMOUNT_B4_TAX, DEV_ID, GAME_INSTANCE_ID, IS_BY_MANUAL, IS_VALID, MERCHANT_ID, OPERATOR_ID, SETTLEMENT_FLAG, SETTLEMENT_TIME, STATUS, TICKET_SERIALNO, TOTAL_AMOUNT, TRANSACTION_ID, TYPE, ID) values (2011-01-17 15:39:08.984, 2011-01-17 15:39:08.984, 0, NULL, 0, 'GII-111', 0, 1, 0, NULL, NULL, NULL, 0, '123456', 1245.01, 'TRANS-8111', 1, '97d660a332774c5fa8a434aa9a18c0bf'[BR] {executed in 16 msec}
[2011-01-17 15:39:09,375][main][DEBUG][WorkingKeyDaoImplTest] Committed transaction after execution of test [testHibernateFlush].

-------------
Surprise! You will see 'insert payout...' was executed after 'select workingkey...'. Why? yes, it is due to 'insert Payout' won't change the state of entity 'Workingkey'.

Flush in Spring test framework

I wrote a testcase extending from Spring AbstractTransactionalDataSourceSpringContextTests, by spring test framework, the transaction will be rollbacked automatically when finish testcase.

@Test
 /** the persistence technology is JPA(hibernate) */
public void testInsert() {
int init_countOfSP = this.countRowsInTable("RA_TE_TICKET_SCHEMA");
int init_countOfEntry = this.countRowsInTable("RA_TE_ENTRY");
     // generate a SalePackage instance which map to table 'RA_TE_TICKET_SCHEMA'
SalePackage sp = mock();
this.getSalePackageDao().insert(sp); int countOfSP = this.countRowsInTable("RA_TE_TICKET_SCHEMA");
int countOfEntry = this.countRowsInTable("RA_TE_ENTRY");
assertEquals(init_countOfSP + 1, countOfSP);
assertEquals(init_countOfEntry + 3, countOfEntry);
}

As my understanding, when countRowsInTable("RA_TE_TICKET_SCHEMA") (the underlying SQL is 'select count(0) from RA_TE_TICKET_SCHEMA) hibernate will issue a 'insert salepackage...' to database....but actual result is no any 'insert...' SQL...why??

It is due to countRowsInTable("RA_TE_TICKET_SCHEMA") is executed from jdbcTemplate(spring + jdbc), JPA implementation has no information of countRowsInTable. Finaly this test case will fail, as no salepackage is inserted into database.

Select will always trigger flush??

Continue with above testcase. As countRowsInTable is executed by JDBC directly, i changed my testcase as below:

@Test
public void testInsert() {
int init_countOfSP = this.countRowsInTable("RA_TE_TICKET_SCHEMA");
int init_countOfEntry = this.countRowsInTable("RA_TE_ENTRY");
SalePackage sp = mock();
this.getSalePackageDao().insert(sp);
sp = this.getSalePackageDao().findById(SalePackage.class, sp.getId());
assertNotNull(sp);
int countOfSP = this.countRowsInTable("RA_TE_TICKET_SCHEMA"); int countOfEntry = this.countRowsInTable("RA_TE_ENTRY");
assertEquals(init_countOfSP + 1, countOfSP);
assertEquals(init_countOfEntry + 3, countOfEntry);
}

I will query SalePackage immediately after insert, and the implementation of findById:

public T findById(Class clazz, String id){ 

return this.getJpaTemplate().find(clazz, id);
}

The testcase failed again...why? I think it is caused by this.getJpaTemplate().find(clazz, id);. As JPA implementation(hibernate) has its own cache, when findById hibernate will find this entity from its cache, no need to hit database, and the most important is hibernate knows that Entity with id(sp.getid()) is inserted in current transaction, it means no other transactions can see this entity. So hibernate even no need to issue a 'select...' SQL to database, it will retrieve this entity from its local cache directly.

But how about if we implement findById() like this:

public T findById(Class clazz, String id){ 

List sp = this.getJpaTemplate().find("from SalePackage p where p.id='" + id + "'");

if (sp.size() > 0) return (T)sp.get(0);
return null;
}

The test result is hibernate will issue two SQLs: 'insert salepackage...' and 'select ...with id=..'. As when you call this.getJpaTemplate().find(clazz, id), JPA runtime definitely know that you are query entity by identifier, while if this.getJpaTemplate().find("from SalePackage p where p.id='" + id + "'") hibernate runtime doesn't know it :).

Reference
Wow, I found a nice post about hibernate flush, "http://blog.krecan.net/2008/01/05/flushing-hibernate/"