Monday, March 12, 2007
Wednesday, February 28, 2007
软件整合测试阶段的版本管理。
想起前段时间组织的对安徽通彩网项目的测试。很混乱,测试团队和开发团队缺乏有效的沟通,很大原因上也是因为版本管理的混乱,当大家在说到一个bug的时候,这个bug并没有定义在一个有效的软件版本下,我们总是说最新版本, 但是很显然很多bug并不是在最新的版本中。如果我们总是说最新版本, 这其实就表示了我们根本没有版本管理。 此外,虽然在测试团队里面采用了跌代的测试方式,但是开发团队里面却并没有一个明确的开发过程定义。实际上,测试团队采用什么样的测试过程应当取决于开发团队的开发过程。 如果开发团队也是一个跌代的开发过程,这表示每个跌代开发团队都会发布一个可测试的软件版本,很自然的,测试团队应该也采用跌代的测试方式。 在安徽通彩网的测试中,我定义了每三天为一个测试迭代,在这个迭代中所有的测试用例需要被重新执行,测试环境需要被重新构建,而且软件版本实际上是由测试团队来定义的。。。我现在能说的就是,好歹测试团队遵循了一个测试过程,有这个过程比没有这个过程要好。
昨天看了一份文档‘Revision Control with Subversion’,其中关于branch的那一段里面定义了一个软件发布阶段的版本管理过程, 在这个过程中branch扮演重要角色(在这份文档中定义了几种branch pattern,其中包括release branch, feature branch),这种branch被定义为release branch,现在直接摘录。
Here's where version control can help. The typical procedure looks like this:
• Developers commit all new work to the trunk. Day-to-day changes are committed to /trunk: new features, bugfixes, and so on.
• The trunk is copied to a “release” branch. When the team thinks the software is ready for release (say, a 1.0 release), then /trunk might be copied to /branches/1.0.
• Teams continue to work in parallel. One team begins rigorous testing of the release branch, while another team continues new work (say, for version 2.0) on /trunk. If bugs are discovered in either location, fixes are ported back and forth as necessary(不管是在trunk或者branch中发现了bug,这个bug的解决都应该被merge到另外一方). At some point, however, even that process stops. The branch is “frozen” for final testing right before a release.
• The branch is tagged and released. When testing is complete, /branches/1.0 is copied to /tags/1.0.0 as a reference snapshot. The tag is packaged and released to customers(版本的发布应该由开发团队和测试团队共同定义,比如测试团队声明所有测试都通过了,那么开发组就可以发布一个新版本).
• The branch is maintained over time. While work continues on /trunk for version 2.0, bugfixes continue to be ported from /trunk to /branches/1.0. When enough bugfixes have accumulated, management may decide to do a 1.0.1 release: /branches/1.0 is copied to /tags/1.0.1, and the tag is packaged and released(就算软件已经正式发布,仍然有可能发现bug,这个bug被解决以后会发布一个bugfix的版本).
Sunday, February 25, 2007
关于性能测试
前些天找到了一个专门讨论软件测试的论坛www.sqaforum.com,里面有一个关于性能测试的目录,看来有很多人和我一样对于性能测试的很多东西都不清楚, 尤其是纠缠于performance test, load test, stress test之间的关系,我转贴了一段得到认同的回帖:
BASIC DEFINITIONS
This is an excerpt from my forthcoming book on performance and load testing.
While there is no universal consistency in how people use terms like performance test and robustness test, I can say that the definitions provided here are as much in the mainstream as any others.
The Definition of Performance Testing
The purpose of performance testing is to measure a system’s performance under load. As Humpty Dumpty said, a word can mean whatever one chooses it to mean, so it is worth our time to examine what we mean by the words “measure”, “performance” and “load”.
Performance testing is a measurement of performance characteristics, although sometimes the use of the word “testing” confuses people. Some performance professionals feel strongly that it is important to not use the term “performance testing”, but to call it performance measurement instead. They are concerned that this measurement will get confused with feature testing and debugging, which it is not. They point out that measurement is only testing if the collected measurements are checked against pre-established goals for performance, and that measurement is often done without preconceptions of required performance.
These people have a good point: clarity of terminology is important. But since most people use the term “performance testing” we will go with the majority and use it too.
The term performance can mean response time, throughput, availability, error rate, resource utilization, or another system characteristic (or group of them), which we are interested in measuring. “All promise outruns performance.” Ralph Waldo Emerson
Performance testing simulates the typical user experience under normal working conditions. The load is a typical, representative mix of demands on the system. (And, of course, there can be several different representative loads -- the work load at 2 p.m., at 2 a.m., etc.) Another name sometimes used for a performance test is a capacity test, though there is a minor difference in these terms as we will see later.
First, the performance testers need to define what the term performance means in a specific test situation -- that is, what the objectives are and what we need to measure in the test. The answer to this question is that we measure performance usually as a weighted mix of three characteristics of a system: throughput, response time and availability. In real-time systems, for example, the users need a guarantee that a task will always be completed within a fixed time limit. Performing a task correctly but a millisecond too late could literally be fatal.
The term load simply means the mix of demands placed on a system while we measure its performance and robustness characteristics. In practice, most loads vary continually, so later we will address the challenge of determining the most appropriate load(s) for testing. The terms work load and benchmark are sometimes used as synonyms for load. A benchmark usually means a standard load, one used to compare the performance of systems, system versions, or hardware environments, but the benchmark is not necessarily the actual mix of demands at any one user installation. The term work load is a synonym for a load, and you see both of the terms in this book: they are interchangeable.
Definition of Load Testing
In contrast to a performance test, a load test is a measurement of performance under heavy load: the peak or worst-case conditions. Because loads can have various sizes, more precise terms for this type of testing are peak-load testing or worst-case-load testing.
A performance test usually is done with a typical, representative load, but this measurement may not tell us much about the system’s behavior under heavy load. For example, let’s assume that the peak load on a system is only 15% more than the average load. The system performance may degrade gracefully – the system runs 15% slower at peak load. Often, though, the performance under load is non-linear: as the load increases by a moderate amount (in this case, 15%), the response time does not increase by a comparable percentage but instead becomes infinite because the system fails under the increased load.
Definition of Stress Testing
A stress test is one which deliberately stresses a system by pushing it beyond its specified limits. The idea is to impose an unreasonable load on the system, an overload, without providing the resources which the system needs to process that load.
In a stress test, one or more of the system resources, such as the processor, memory, or database I/O access channel, often “maxes out” and reaches saturation. (Practically, saturation can happen at less than 100% of the theoretical usable amount of the resource, for many reasons.)
This means that the testware (the test environment, test tools, etc.) must be sufficiently robust to support the stress test. We do not want the testware to fail before we have been able to adequately stress the system.
Many bugs found in stress testing are feature bugs which we cannot see with normal loads but are triggered under stress. This can lead to confusion about the difference between a feature bug and a stress bug. We will address this issue in the upcoming section entitled: “Testing Performance and Robustness versus Features”.
Some testers prize stress testing because it is so fruitful in finding bugs. Others think it is dangerous because it misdirects projects to fix irrelevant bugs. Stress testing often finds many bugs, and fixing these bugs leads to significant delays in the system delivery, which in turn leads to resistance to fixing the bugs. If we find a bug with a test case or in a test environment which we can’t connect to actual use, people are likely to dismiss it with comments like: "The users couldn’t do that.", “.. wouldn’t do that” or “... shouldn’t do that.”
Stress, Robustness and Reliability
Although stress, robustness and reliability are similar, the differences among them mean that we test them in related but different ways.
We stress a system when we place a load on it which exceeds its planned capacity. This overload may cause the system to fail, and it is the focus of stress testing.
Systems can fail in many ways, not just from overloading. We define the robustness of a system by its ability to recover from problems; its survivability. Robustness testing tries to make a system fail, so we can observe what happens and whether it recovers. Robustness testing includes stress testing but is broader, since there are many ways in which a system can fail as well as from overloading.
Reliability is most commonly defined as the mean time between failure (MTBF) of a system in operation, and as such it is closely related to availability. Reliability testing measures MTBF in test mode and predicts what the system reliability will be in live operation.
Robustness and reliability testing are discussed in the companion volume to this book, entitled “System Robustness Testing”.
Ross
---------------
这里基本上是说load test, stress test都是performance test的特例,更多的信息可以参考http://www.qaforums.com/Forum2/HTML/000724.html, 还有一个很有个实际指导意义的资源:http://www.codeplex.com/PerfTesting
Wednesday, January 31, 2007
Understanding Transaction Isolation
昨天, 当我现在在这里写这些文字的时候,我一直在回想, 昨天是什么促使我突然又开始关注事务隔离级别这个话题. 想不起来了, 昨天晚上公司年夜饭, 喝了不少红酒, 还喝了一杯白酒, 回家之后倒头就睡,过去的记忆仿佛都在昨天被抛弃了. 就是说想不起来了, 想不起来也无所谓, 这件事情还是要做的, 所以今天又google了一把, 关键字是transaction isolation level, 嘿嘿,找到了一些资源, 先看看. 写这些文字的时候, 我还根本没看那些找到的文章, 有点颠倒是非. 先写道这里, 看了之后, 然后再说我是否已经真正理解了事务隔离级别这个概念. 理解了我便要在这里写下我的理解.
下面这篇文章不错, 我决定直接copy过来.
--------------------------
Understanding Transaction Isolation
http://www.expresscomputeronline.com/20040426/techspace01.shtml
Article summary
This is a topic related to SQL Server. However, the concept is applicable to any RDBMS. While auditing many applications, I have found that incomplete or no understanding of ‘Isolation Level’ leads to a lot of real life problems. These include performance degradation, blocking, locking as well as major deadlocks.
This article provides an easy-to-understand view of what ‘Isolation Levels’ really mean and when to use which level.
‘Isolation Level’ is a setting that decides how data which is a part of an ongoing transaction is made visible to other transactions.
Transactions are well understood. So what is the problem?
As all of us know, it is a unit of work. The work may contain many steps but either all steps happen or all steps don’t happen. This is fairly well known. Nothing new here. But consider how transactions occur in real life.
- I am updating one row in a Begin Tran and Commit section. I have issued Begin Tran as well as the Update statement. Commit is not yet issued – because I want to perform some more actions in the same transactions on some other tables. If some other user wants to read this row that I have updated but not yet committed, what should happen?
- Consider another scenario. I start a transaction. I calculate a total of a field based upon all rows in a table. Now, I need to add a new record in another table which contains this total. Now, can the original table be changed by some other user after I calculate the total? In which case, there could be a mismatch. Do you want to take such chances?
- Another scenario. I am working on some transaction table between a range of keys within a transaction—say 10 and 20. There were only 5 records when I read the range – 10, 12, 14, 16, 20. Now I am working on other things in the transaction. Before I could commit the transaction, someone added another row with a key value of 11. Now, my base assumption about what records I read between 10 and 20 and further work upon them itself is wrong. Problem!!! Is it not?
- I start reading a long table. It is not a transaction at all. But other users want to refer to that table for updating some fields in specific rows. The query takes 20 minutes to read all the rows. What happens to other users who are trying to update the rows? Do they wait for 20 minutes? Or they are allowed to update the rows even when the rows are being read in a large query? What if the query was used to generate a summary report containing grand totals? The total would be wrong because after the summation started, some rows have changed. Some of these rows could have changed after the summation occurred. What’s to be done now?
As you can see all these situations are confusing and prone to inaccuracies. To avoid such problems we have a feature called “Isolation Levels”.
Isolation Levels are applicable to transactions. These decide the visibility of information which is a part of an ongoing transaction.
In very simple terms Isolation Levels decide “What happens when some data being referred to (for reading or writing) within an incomplete transaction is also being referred to (for reading or writing) from another connection (or user – actually it is called another ‘transaction lock space’)?”
To make things sound technical, all these problems have been given nice and complex sounding names.
Data visibility problems that can occur during a transaction
Let us understand some jargon.
- Transaction Lock Space:
Each transaction performs certain operations (select / insert / update / delete) on one or more rows of one or more tables. Transaction starts with Begin Tran command and ends with Commit. If unsuccessful, it ends with Rollback command. Now, after Begin Tran is issues and before Commit is issued, multiple tables may participate in the transaction related commands. Specific parts of these tables need to be locked during this phase to ensure that other users do not interfere with this transaction. This is called Transaction Lock Space.
- Uncommitted (dirty) data:
Consider this code snippet (Blue text is code; green indicates comments):
1. Begin Transaction
—we want to change the customer status from active = “yes” to active = “no”
2. Update customer Set active = “No”
where CustomerID = 2324
—some more commandss
3. Select * from customer where CustomerID = 2324
4. Commit Transaction
Now, if the transaction commits, the value will be “No”. If it does not commit (for whatever reason), the value will remain “Yes”.
Consider that the value has already been changed to “No”. But there are more commands to be executed before the entire transaction commits. These commands are time consuming and take, say, 3 minutes. During these three minutes, if some user outside the transaction lock space reads the value of Active field in the Customer table for ID 2324, what should they see? “Yes” or “No”? The answer is simple. The value is changed but not committed. Therefore, external queries should still show “Yes”.
Now what would happen if a Select active from Customer where CustomerID – 2324 returns “No” to another user? What happens if the transaction rolls back? This is called Uncommitted Data. Ideally this should not be visible outside the transaction.
However, consider the same command executed within the transaction (line 4). It should – and it will return “No”.
Now let us consider various problems that can occur. The problems can be of three types:
1. Dirty Read
2. Non-repeatable read
3. Phantom rows
- Dirty reads
This is when connections outside the transaction space can read the uncommitted data. You don’t want this to happen in most cases. The only reason when you may want to allow this is when you are creating a report which assumes a certain amount of inaccuracy. You think this situation is rare? Not really. Suppose the report compares the current month sale with last month sale and returns a percentage difference. Consider that the report takes 5 minutes to generate. During these 5 minutes, more transactions may be getting added to the sales data. Typically 5 transactions would get added to the sales table. The average transaction value is Rs. 1000. The total sale for the month is typically 30-40 lacs. In such cases, you really don’t need to worry about absolute accuracy. You can intentionally allow dirty reads because the impact is really not significant from a business perspective.
- Non-repeatable read
Consider the above code. We change the value of Active to “No” in line 2. Now in line 4 you expect the value to be “No” because we are querying the row ‘within’ the transaction.
Now what would happen if some other transaction was allowed to change the same value to, say, “Maybe”? In line 4, we would expect value of “No” but actually get value of “Maybe”. This problem is called non-repeatable read. The idea is that within a transaction, the same data read any number of times should yield consistent results. If it is not same then the reads are ‘non-repeatable’.
- Phantom Rows
This was explained in the introduction (range 10 to 20 example). The concept is simple. If you have already read a range of data based upon a key value, another transaction inserts a new row which happens to have a value between this range. The original range which was referred to would now become invalid. This is because a “Phantom row” was added. This is also a major problem.
Important learning
Although the above three issues are listed as problems, they may not always be considered as problems! Paradoxical? Not really. As explained for Dirty Read, whether it is a problem or not depends entirely upon the business context. Technology exists to allow these problems to occurs as well as prevent them. The choice is yours. I know this complicates matters. But if these matters were not complicated, what are IT professionals paid for!
Now let us see how to prevent these problems? The answer is “using Isolation Levels”.
Types of Isolation Levels
| To solve the problem of … | The Isolation Level should be… |
| Dirty Read | Read Committed (Default of SQL Server)
|
| Dirty Read and Non-Repeatable Read | Repeatable Read Non-Repeatable Read
|
| Dirty Read and Non-Repeatable Read and Phantom Rows
| Serializable |
| To retain all three problems | Read Uncommitted |
Before we discuss each isolation level, let us understand how to set or change it in T-SQL.
Syntax
SET TRANSACTION ISOLATION LEVEL
{ READ COMMITTED
| READ UNCOMMITTED
| REPEATABLE READ
| SERIALIZABLE
}
Remember:
- This setting applies to all transactions which are happening within a single connection.
- Multiple connections can have different Isolation Levels.
- Within a single connection, you can change the Isolation Level for specific transactions.
- You must know the kind of transaction you are performing and its business requirements before you can decide the right isolation level.
- Isolation level does not just affect the current transactions in a given connection. It also affects how other transactions issued within other connections will behave.
- Therefore, you should have a good idea of what kind of concurrent transactions occur in your application when the system is live and many users are active.
This is the single biggest problem which leads to low performance, locking and blocking, as well as deadlocks
- Most developers think of individual transactions as though they are occurring in a single user system.
- It is difficult to envisage all permutations of possible transactions which can occur together. But it is very much possible to plan for it in a structured and informed manner. This requires additional effort, monitoring live systems and tweaking of specific transactions and so on.
- Unfortunately, this additional effort is rarely a part of system deployment! Such problems surface only after the system load increases in such a way that small, unnoticed issues become amplified due to large volume of data and / or large number of concurrent users.
- Only one of the options can be set at a time.
- It remains in effect till you explicitly change the option by issuing another command. This is an important aspect to consider.
Best Practice : When you change the Isolation Level for a transaction, remember to set it back to the original level after the transaction is completed.
Isolation levels
- Read Uncommitted
This is as good (or bad) as not having any isolation. All data which is uncommitted is readable from any connection. This should not be used unless you have a very good reason to do so.
- Read Committed
This prevents dirty reads. This does not prevent phantoms or non-repeatable reads. This is the default. Although it is the default, it does not mean that this isolation level is ideal for all transactions. One of the regular tendencies amongst techies is to simply use default values without giving it a second thought! I cannot imagine the number of phantom and non-repeatable reads that must be occurring in the world because someone simply used the ‘default’ value. It is a scary thought to say the least.
This level is obviously more restrictive than the ‘Read Uncommitted’ level.
- Repeatable read
This prevents dirty reads as well as non-repeatable reads. It does not prevent phantom rows. This is more restrictive than Read Committed level. When I say restrictive what does it mean? It means that the chances of other transactions having to wait for this one to finish are INCREASED. Another way of saying this is – Repeatable Read reduces concurrency compared to Read Committed level.
- Serializable
This is the most restrictive of the options. This should never be used as the default level. If you use this one as the default, it will most probably create a single user system!
This prevents all three problems.
How do Isolation Levels work?
You will notice that we have not yet addressed this issue at all in this article. The answer to this question is another Pandora’s box. The answer is simple. It uses the appropriate types of locks to achieve the desired effects. We will see what locks are and how they are implemented using Isolation Levels in the next article.
In the meantime, please go through your code and see if you are simply using the default level or are there some transactions which merit a different isolation level. If you feel like changing a level, never do it in production system. It can create havoc. Try it out in a test environment first, satisfy yourself that it has no side effects and then implement it in production system.
Don’t stop there. Monitor the production system. Some problems are never detected in test systems. They may occur in production. Handle them as required.
If you have a packaged product which handles large amount of data and many concurrent users, you must analyse the appropriate usage of Isolation Levels.
Friday, December 01, 2006
[Window]配置sqlcmd远程连接sqlserver2005.
我不是数据库管理员,对sqlserver不怎么熟悉,尤其是这些涉及到管理员的活儿. 因为sqlcmd是一个命令行的工具,可以用来做一些自动化的工作,比如自动创建数据库表模式, 自动装载数据. 本身ant又个sql的task是可以用来做这些事情的, 但是由于项目中的数据库脚本都是直接从sqlserver中导出的,完全是sqlserver的一套语法, 'sql' task不支持这些语法, 原来我是手工把数据库脚本中一些sqlserver特定的语句给删掉,比如 'GO', 'if exist...'等,但那得建立在数据库脚本比较少的基础上,而且得没有存储过程, 在现在的项目中,这些前提全被打破了, 实在没有办法,只好借助'sqlcmd'.
但是sqlcmd怎么都无法连接到到远程数据库服务器, google了一把, 也到msdn上看了不少资料, 知道关于远程连接的配置主要集中在' sqlserver外围应用配置器'上.
------------------------------------------------
1. 打开'Sqlserver 2005 Surface Area Configuration'->'按实例查看'->'Database Engine', 确保远程连接已经被设置.
2. 打开'Sqlserver Configuraion manager', 保证tcp/ip已经被激活了(服务器端与客户端).
NOTE: 上面两步都是在数据库服务器上进行的操作
3. 在客户端上, 打开'Sqlserver Configuraion manager', 保证tcp/ip已经被激活.
------------------------------------------------
在命令行窗口执行命令:
>sqlcmd -S 192.168.0.94 -U hp_ante -P hp_ante -d AnteProxy
可以连接到数据库服务器.
>sqlcmd -S hp-server-94 -U hp_ante -P hp_ante -d AnteProxy
连接也能成功.
>sqlcmd -s tcp:192.168.0.84,1433
如果'database engine'的远程连接配置的为'同时使用name piple和tcp/ip':
------------
HResult 0x2, Level 16, State 1
命名管道提供程序: 无法打开与 SQL Server 的连接 [2].
Sqlcmd: Error: Microsoft SQL Native Client : 建立到服务器的连
到 SQL Server 2005 时,默认设置 SQL Server 不允许远程连接这个
。.
Sqlcmd: Error: Microsoft SQL Native Client : 登录超时已过期.
------------
如果'database engine'的远程连接配置的为'仅使用tcp/ip':
------------
HResult 0x274D, Level 16, State 1
TCP 提供程序: No connection could be made because the target machine actively re
fused it.
Sqlcmd: Error: Microsoft SQL Native Client : 建立到服务器的连接时发生错误。连接
到 SQL Server 2005 时,默认设置 SQL Server 不允许远程连接这个事实可能会导致失败
。.
Sqlcmd: Error: Microsoft SQL Native Client : 登录超时已过期.
------------
我知道为什么最后那个命令不能成功, 后来又试了几种办法:
- 启动'Sqlserver 2005 Surface Area Configuration'中的'sqlserver browser'服务: 无法连接
唉,不管怎么样,至少有方法可以连接成功了. 如果你的sqlcmd连接不能成功, 除了上面说的配置以外,还要检查以下部分:
1. SQL Express is running (hey, gotta check)
2. SQL Browser is running
3. TCP/IP is enabled as a server protocol
4. You've created an Exception in your firewall for SQL Express
5. You've created an Exception in your firewall for SQL Browser
参考资料:
- http://blogs.msdn.com/sqlexpress/archive/2005/05/05/415084.aspx
- http://msdn2.microsoft.com/zh-cn/library/ms162773.aspx