SSD Quality and Performance Comparison - A Testing and Evaluation Report for SSDs in the Market Today (2)

Since the last article “Exploring the Quality of SSD – A Testing and Evaluation Report for SSDs in the Market Today” was released in the end of 2010, we have received a large amount of positive feedback from readers and followers. In order to response to all the requests from our followers, we are releasing a follow up test report with latest SSD products. Hoping this report could provide valuable information when you purchasing a SSD drive.

Allion's interest is to perform a fair test and acquire valuable test results. We have pre-defined some criteria of test environment and test condition.

● SSD drives selection

There are four requirements when we select SSD drives for test:

1. Products can be bought in the beginning of 2011
2. Products have either 250 GB or 256 GB of storage space
3. Price range is between US$ 500 to US$ 650
4. The declared read/write speed of products can reach between 200 MB/s and 300 MB/s

Base on these requirement, we have selected eight SSDs from market for test: (Please refer to Table 1 for more drive information)

Table 1: DUT

● Test Environment

1. Test system. In this test experiment, we use same test platform for all tests.
2. Test tool. We have utilize several benchmark test tools from the market and Allion's self-developed benchmark test tool to perform the tests.
3. Test scenarios. Average Read/Write Performance, Sequential 64K Read, Sequential 64K Write, Random 16K Read, Random 16K Write, Random 4K Read and Random 4K Write. Test was run for seven days.
4. Test work load. In our test, we adopt several file formats for SSD read/write test.
5. Test time. We have performed a continuous 7 days test.

● Evaluation Items

1. Reliability
2. Performance Evaluation
3. Performance Stability
4. Performance Degradation

Result Analysis:

1. Reliability

The purpose of reliability test is to check if data access error will happen during data read/write, i.e., incomplete read/write or data lose. The main task of SSD is to accurately store and retain data, any data loss will result in serious damage to users. The reliability of SSD is a very basic evaluation item in this test experiment.

In our test, we examined if the SSD's SATA connector meet the SATA I-O's requirement. We also verified that the devices detected by test system without problem. After the basic spec checked, a serious of data read/write commands were sent to the SSD drives. Our experts compared and analyzed both data sets to see if there is any data lose. We used Allion's in house SSD stress tool pro to access all SSD drives for quite a long time. All tested SSD drives pass this test.

2: Performance Evaluation

As we know, high speed transmission is one of the SSD highlights. To check if the actual performance of these SSDs match their announced results, we stimulate real user behavior (i.e. different file size, various file formats) to access those SSDs and record the test result. We can look at the result from two different aspects.

1) Performance Test Results Comparison

Table 3 is the result of the SSD performance test on pure read/write test.

Table 3: Performance Test Results (MB/s)

The performance of each SSD is different. Under the condition of same test time (7 days), the total read/write count for each SSD were different. The SSD which has better performance had more total read/write count, vice versa. The test result that we present in Table 3 is the average number of the read/write result over all read/write count.

Picture A: Performance Test Results

Table 3 test result also indicate result from different test scenarios. Turning Table 3 number into chart, we can observe that SSD tend to have better performance when dealing with larger files (64K reading process) compared to smaller files (16K and 4K reading process) on average. In general, the speed of data reading is faster than data writing for storage media, Samsung does have similar read/write performance on all file size. On Patriot SSD drive, we can see an obvious performance drops when dealing with 16K and 4K data file [drops down from 160 ~170MB/s (64K) to 22MB/s (16K), even close to 3MB/s (4K)].

Among these tested SSDs, Adata, Intel and Samsung are having better performance than others.

2) Comparison of Declared Performance vs. Actual Achievement

Most of the SSD product will declare “max read/write” and “average read/write” number on the product specification. What is the actual read/write speed that users will get when they buy the SSD from the market? We also compare the test result from this perspective. We ran a 7 days long read/write stress test. We selected “64K sequential read” and “64K sequential write” test data for comparison since 64K read/write tend to have better performance than other file sizes.

Table 4: 64K Sequential Read/Write Maximum and Declared Performance Comparison

We summarize the best read/write performance result we got during the test from our test data in Table 4. By looking at this table, we notice that OCZ SSD can reach their declared “read” performance. The rests only can reach 83% to 97% of the declared performance. Most of the test drives can reach the declared “write” performance.

If we look at the result from “average read/write” point of view (we average all the read/write performance result to get this number), we will see slightly different result. Table 5 is the summary of test result. We notice that no SSD could reach their declared “read performance.” On the declared“write performance,”only 4 out of 8 SSDs could reach that number.

Table 5: 64K Sequential Read/Write Average and Declared Performance Comparison

Compared the actual results with declared performance from vendor, OCZ is the most outstanding one among these SSDs.

3: Performance Stability

The purpose of this test is to evaluate if tested SSD can still remains steady performance after a long time access. Two factors might affect SSD drive's stability in our observation: long time access to the drive by using sequential and random data type and use up all the storage capacity recursively. We analyze the performance stability from following perspectives.

• Performance Normal Distribution:
  We use 95% as our analysis criteria. 95% (and above) of the test results need to fall into the range of +/–10% of average read/write performance range. 
• High/Low Gap over Mean:
  The difference of the leading group (mean of the top 10% performance result) and the lagging group (mean of the last 10% performance result) must be lower than 20% of overall average performance result.
• Standard Deviation over Mean:
  The standard deviation over mean need to be lower than 10%.

1) Performance Normal Distribution

We will observe the overall test result distribution. If all results are close to the µ, it means the drive is relatively stable. If the result distribution fall widely that means the drive is relatively not stable.

We use “95% and above of the read/write performance results need to be in the range of +/-10% of the average read/write performance results” as our analyze criteria. We summarize the read/write performance result in table 6. We can see that four out of eight SSDs can reach the criteria. OCZ is the one has the best normal distribution performance result.

Table 6: Performance Normal Distribution Test Results (Overall Read/Write Performance Test)

If we look at the distribution result by graph, we can see that in best case (graphic B), all test results (blue dots) fall into the upper/lower green lines area (+/–10%). In bad case (graphic C) most of results fell out of upper/lower green line area.

Picture B Case: Good Performance Point Distribution (Read/Write Performance Test)	Picture C Case: Bad Performance Point Distribution (Read/Write Performance Test)

2) High/Low Gap over Mean

High/Low Gap over Mean is an indicator of the stability of a SSD drive. We will compare the difference of the best and worst performance for the designated SSD. The closer the gap is, the more stable it is. In our analysis, we use this criteria: The difference of the top 10% best performed SSDs and the last 10% least performed ones must be lower than 20% of overall average performance.

Table 7 High/Low Gap over Mean Test Results (Overall Read/Write Performance Test)

We do use that 4 out of 8 SSDs perform as expected while read/write performance test is undertaking, OCZ and Corsair are two of the bests among these four devices. On the contrary, 2 out of 4 drives were having even larger than 60%of the result. Even though the gap of ADATA and Patriot are close (7.63 and 8.22), there is a big difference between means (72.71 and 37.98), therefore, the evaluation results are significantly different (10.49% and 21.64%).

3) Standard Deviation over Mean

Standard deviation over mean is another good indicator to evaluate the SSD drives' stability. In this analysis, we observe every SSD drive's test result against their average result. Out analysis criteria: The dispersion of all test results must be lower than 10% of overall average performance to meet the expectation under this section.

Table 8 Standard Deviation over Mean Test Results (Overall Read/Write Performance Test)

Let's take OCZ's result for example. It has σ=1.04, μ=78.03MB/s and σ/μ=1.34%. That means most of the read/write test result fell within +/- 1.04MB/s. The σ/μ=1.34% indicate a very stable performance. 5 out of 8 SSDs meet our criteria.

4: Performance Degradation

Performance degradation means SSD's performance declines after certain time of use and wear out. Performance degradation will also impact the usable storage size. Lack of TRIM, Garbage Collection function not well-designed or insufficient Space Provisioning will affect overall performance measurement. In our test, we run a 7 days complete read/write test. The drive is having performance degradation if the difference between first and last cycle of the seven-day test is greater than 10% of average performance (to write data completely into all SSD storage area as a cycle).

We did observe 3 tested SSD has performance degradation. Picture D and E are shown as an example of performance degradation. From picture D we can see that this SSD is able to reach up to 80~90 MB/s of performance in first cycle rounds, then reduce to 70MB/s. The average performance between first (pink) and last cycle (blue) has 11MB/s difference as shown in picture E.

Picture D Case: Performance Point Distribution (Read/Write Performance Test)	Picture E Case: First and Last Cycle Rounds Average Performance Comparison (Read/Write Performance Test)

 

Picture F SSD Key Point of Product Testing

Even though the features of these SSD devices that we selected to test are similar, we can still see different result in different test conditions. SSD still needs a long-term development process; market acceptance will stimulate product technique to keep improving and growing to the next level. With years of research and product test experience, Allion expects this evaluation index would be a useful reference for readers. Along with the growing of technology, we hope to see more SSD products with the better quality and technique in the near future.