Performance of Benchmarks on the Cell Processor

Transcription

1 Performance of Benchmarks on the Cell Processor Kerry Creeron Michael Smith Dan Perron CS/ECE 752 Advanced Computer Architecture I The University of Wisconsin-Madison ktcreeron at wisc dot edu, masmith27 at wisc dot edu, djperron at wisc dot edu Abstract The Cell processor represents a $4 million dollar investment on the part of Sony, Toshiba, and IBM, which is poised to make its way into a slew of embedded devices, from High Definition TV s, to cellular phones. Given that the Cell is relatively new, and its performance is currently unknown, we propose to better gauge its capabilities by modifying portions of the MiBench suite for use on the Cell. In the end, we find that performance speedup using the cell is possible, but that the processor s architectural limitations and thread creation overhead make reaping such benefits difficult. 1. Introduction New processor architectures are nothing unusual in this day in age. The past few years have seen a large number of new architectures, including Sun s Niagara and Intel s Core architectures. And with the introduction of these new architectures, one sees rather obviously, the trend that processors are becoming increasingly parallel in nature. The name of the game seems to more cores. The Cell processor, developed by Sony, Toshiba, and IBM (STI), is a new take on the multicore approach. STI s implementation features a simple, inorder PowerPC core, known as the PowerPC Processing Element (PPE) with 8 Synergistic Processing Elements (SPE s) designed to perform computationally intensive tasks, such as floating point operations and vector operations. The PPE is connected to the 8 SPE s via a ring interconnect, known as the Element Interconnect Bus (EIB). Transfers to and from the PPE and SPE s are performed using DMA, the benefit being that no computation is wasted coordinating memory transfers among the various elements of the chip. The twist in the Cell architecture lies in the fact that while the memory is visible to all the elements on the chip, memory transfers must be explicitly scheduled between the various elements. Thus, memory must be carefully managed to ensure coherence is maintained. Given the departure of the Cell from a truly shared memory architecture, its rather explicit take on parallelizing code, and fact these three computing giants intend to use the Cell in a wide variety of applications, from the Playstation 3, to High Definition TV s, to cellular phones, we believe that it is important to gauge the suitability of the Cell in such embedded applications so that systems designers can make more informed decisions, as opposed to the current cloud of uncertainty hanging over the Cell s performance. To benchmark the Cell, we propose to use the freely available, open source, MiBench suite, a suite designed for embedded applications to gauge the performance and scalability of the Cell. Of this suite, we selected the automotive area as a prime candidate for running on the

2 Cell, due to the parallel nature of the code found in these individual programs, and the lack of data dependencies. In order to more accurately characterize the performance of the Cell s unique architecture, we extensively modify the MiBench code so that computation is farmed out to each of the SPE s. In addition, we are aware of the fact that yield is currently an issue with the Cell, and as a result, the Playstation 3 s Cell chips come with one of the SPE s disabled. To measure the performance impact of having reduced numbers of SPE s, we vary the number of SPE s the benchmarks are allowed to use by limiting SPE thread creation. Finally, we compare the performance of a Cell Blade server versus an Intel x86 system, and use IBM s systemsim simulator to collect more detailed performance statistics about the SPE s. After modifying the automotive section of MiBench and running our test programs on our test systems, we found that in some instances, typically those where the SPE s had large data sets and each SPE spent a large amount of time performing computationally expensive tasks, the Cell Blade outperformed our baseline x86 system. When data sets were limited and the SPE s performed a smaller number of iterations however, using a larger number of SPE s resulted in decreased performance as compared to using a number of SPE s. This degraded performance is due primarily to the overhead associated with creating a new thread on each SPE, and the DMA transfers involved therein. The remainder of this paper is organized as follows: Section 2 describes our benchmark design. Section 3 describes our benchmark implementation and methodology. Section 4 discusses our performance results. Section 5 discusses related work, and Section 6 concludes the paper. 2. Design To test the cell s performance while running an embedded systems workload we needed in appropriate benchmarking suite to use as a template for our modified versions designed to run on the cell. The performance metrics we chose to look at were both wall clock time and CPI. Wall clock time was chosen in order to get a measurement of cross architecture performance. We also wanted to see how big of a difference there was in changing the number of SPE s used at a time. We chose to do this by analyzing the CPI of the SPE s Benchmark Suite Initially when searching for benchmarking suites we came across EEMBC which is the standard for embedded benchmarking programs. The problem with EEMBC is that there are licensing issues involved in using it and we did not have the proper access. We then came across an open source alternative known as MiBench from the University of Michigan. MiBench consists of several different categories of benchmarks. Those categories include automotive, consumer, network, office, security, and telecomm. The automotive suite was chosen for its small code size, and ease of parallelization. Inside the automotive suite there were 4 programs. These four programs were titled basicmath, bitcount, qsort, and susan. We chose to implement the first three since we were unsure on how best to parallelize susan which is an image recognition

3 program. Basicmath consisted of four functions, a cubic solver, square root solver using bit operations, a degree to radians function, and radians to degrees function, each of these functions were called in a series of for loops with varying pieces of data. Bitcount implemented 7 different bit counting algorithms, each were run on large sets of data. Qsort ran c s qsort function on large amounts of data Performance Analysis We wanted to compare performance both against other architectures as well as on the number of SPE s used. In order to judge performance across architectures we used wall clock time of the entire program using. We tested the code on an x86 based machine as well as on a cell blade server which we had limited access to. In order to test the performance with varying SPE s the IBM cell simulator was used. The metric we chose to compare was CPI. The performance monitor was started right before the program was run and observed immediately after. 3. Implementation In order run the MiBench benchmarks on the Cell processor several changes had to be made to the code. The MiBench suite was eventually treated more like a guideline for writing embedded benchmarks then anything else. Various versions of the benchmarks were created in order to analyze performance on the x86, as well as 1 SPE, 2 SPE, 4 SPE, and 8 SPE versions. 3.1 Benchmark Modifications In order to efficiently parallelize these benchmarks some modifications had to be made. In basicmath the number of loop iterations was changed to be a multiple of 8. This allowed for us to easily split up the program to run on 8, 4, 2 or 1 SPE. From here code was added in to be able to create SPE threads instead of function calls. Each for loop was split up into a certain number of partitions depending on the number of SPE s used. These partitions then sent both a starting number and a number of iterations to the SPE or the entire set of data for that particular partition to the SPE. Bitcount was split up in a similar way. Each bit counting algorithm was ran for a large number of iterations. These iterations were split up amongst the number of SPE s and the data and number of iterations was sent to each SPE. Qsort was parallelized by sending a partition of the data to each SPE to be quick sorted. These sorted partitions were then sent back to the PPE where they were merge sorted. 3.2 Performance Analysis Methods In order to judge performance based on wall clock time two systems were used. Wall clock time was found by calling get time of day at the beginning of the code running in the PPE in order to get an overall execution time. Two different cpu architectures were analyzed. The cpu specifications can be found in table 3.1 Table 3.1 System Specifications System 1 System 2 CPU Pentium 4 Two Cell Processors Frequency 3.ghz 3.2 GHz Ram 1gb 512mb per processor

4 The wall clock time for the cell programs also changed the number of SPE s used. The x86 version of our code did not use threads but instead just normal function calls. In order to compare performance of just changing the number of SPE s the IBM cell simulator was used. The simulator allowed us to get detailed statistics about individual SPE s we chose to look at CPI specifically as well as looking at where the stalls are in the SPE execution. The CPI was only found for 8 and 1 SPE s total, also we had to use modified versions of the benchmarks we ran to get wall clock time. Specifically we had to modify them to only create one thread in the case of 1 SPE or 8 when using the 8 SPE version. This is because the simulator did not have an option to turn off certain SPE s and chose to send threads to different SPE s each time. This made getting CPI on anything less then 8 SPE s almost impossible. 4. Results The first series of results that we present are comparisons of wall clock time of the Cell Blade server, using differing numbers of SPE s, to the wall clock time of the Intel Pentium 4 baseline system. Benchmarks are for the Bit Count Program, Basic Math program, and the Quick Sort program. Time (seconds) Time (Microseconds) Bit Count Bit Count Times 1 SPE 2 SPE 4 SPE 8 SPE x86 Figure 1.1 Basic Math Basicmath Times 1 SPE 2 SPE 8 SPE x86 Figure 1.2 Quick Sort Wall Time of qsort on Pentium 4 vs Cell Blade Time (usecc) Cell Blade Pentium 4 Figure 1.3 Processor Our second series of data compares the number of cycles that each SPE

5 performs, scaling the number of active SPE s from 1 to 8. This data was obtained using IBM s systemsim simulator. In this series, the benchmarks run are decomposed to some extent. While the bit count program is run in its entirety, the basic math is broken down into two of its constituent programs, usqrt, which performs an inverse binary square root, and deg2rad, which performs a radian to degree conversion, to provide a more detailed look of the program, and also to allow for the simulation to complete in a more expedient fashion. Finally, we were unable to obtain reliable numbers from the systemsim simulator for the quicksort benchmark and as such, it was omitted. Number of Cycles Cycles Bit Count Bit Count Cycles 1 SPE 8 SPEs Figure 2.1 Usqrt USqrt 1 SPE 8 SPE Figure 2.2 Cycles Deg2rad Deg2Rad 1 SPE 8 SPE Figure 2.3 The first series of data is very telling. The first figure, 1.1, shows that significant performance gains are possible over our baseline Pentium 4 system for the bit count program when 4 or 8 SPE s are used. This is an encouraging sign, showing both that the Cell can scale, and also that it can outperform a more general purpose CPU in some situations. Figures 1.2 and 1.3 paint a very different picture. Here, we not only see the number of SPE s hurting performance, but we also see the Cell losing to a Pentium 4 system with a lower clock speed. The increase in wall clock time can be easily attributed to the overhead associated with thread creation and the additional data transfers that must take place on the EIB in order to move the data to a larger number of SPE s. Additionally, the huge discrepancy between the Cell s performance and the Pentium 4 s can be attributed to the fact that our naïve Pentium 4 implementation was single threaded, so many of the functions could be inlined and no thread overhead was introduced, giving the Pentium 4 a huge boost in performance. Moving on to the second data series, figures 2.1 and 2.2 illustrate the kinds of gains that can be reaped by using multiple SPE s. Clearly the bit count

6 program is amenable to parallelization, as it evinces a 9 times decrease in the number of cycles run on each SPE. The same goes for Usqrt, in which a 5 times decrease in clock occurs between the 1 SPE version and the 8 SPE version. Figure 2.3 illustrates that using more SPE s can incur additional overhead, seeing as the 8 SPE version of the deg2rad program incurs a 2.5 times increase in cycle count. 5. Related Work The key piece of previous work that we relied on for this paper was MiBench: A free, commercially representative embedded benchmark suite. [3] We used this paper and the source code provided with it to create our benchmarks. This paper details the various areas of the MiBench suite, and gives motivation for its suitability as an embedded benchmark suite, as well as for its replacing the EEMBC suite. Additionally, our paper follows in the footsteps of a number of other papers that have worked on testing the Cell s ability to handle various algorithms. The Feldkamp algorithm [1], OpenSSL [2], and Molecular Dynamics [4] have all been successful implementations of algorithms on the Cell. [1] managed to realize a speed gain of more than 2 times using 1 SPE as compared to just the PPE. In [2] a 7.5 times speedup was achieved using 6 SPE s rather than just the PPE. And finally, [4] netted a speedup of 6.1 using 16 SPE s versus 1 SPE. These papers are in line with some of our results, and show that large amounts performance speedup and scalability are achievable using the Cell architecture. CPU. We find this to be necessary because while comparing the cell to itself is useful in determining how well code can be parallelized, it does not indicate with any sort of certainty, the relative merits of the Cell s architecture for a given application. 6. Conclusion At the end of the day, our experience with the Cell processor was something of a mixed bag. While we were able to achieve modest performance increases and demonstrate the scalability of the SPE s, we were unable to do this on a consistent basis, owing to the overhead of thread creation, and lack of computational intensity in some of these programs. Furthermore, the Cell lost in most benchmarks to a Pentium 4 system that costs significantly less. 6.1 Problems We encountered a number of problems throughout the course of our research. First off, the Cell simulator is rather inadequate. It primarily serves as a functional simulator, as opposed to a performance simulator. As such, we based much of our research off of the results output by the simulator. Once we actually moved our code from the simulator onto the production blade server we had access to however, our techniques and strategies for coding on the Cell proved to be inappropriate for the real world. Had we had more time on the Cell blade, we feel confident that we could have tuned our code to realize gains in performance that other research has been able to glean. The main distinction that we make in this paper is to compare the performance of the Cell against another

7 6.2 Future Work Much work is left to be done on the Cell in the embedded arena. There are a number of areas that we would have liked to have touched upon, but did not have the time to do so. Intrinsics, the c wrappers for vector instructions, may provide the performance boost we were hoping to see. After seeing the impact of DMA overhead on our benchmarks, it would also be interesting to see how implementing double buffering would help to reduce memory transfer penalties. Finally, after viewing the overhead associated with thread creation, there is much work to be done in investigating how the thread scheduler on the Cell performs. One option avenue to investigate would be seeing how the SPE s perform when fewer thread joins were performed. Currently, the SPE s run one function, and then a thread_wait is called. What could be done instead is to call thread_wait only at the end of the entire program, testing to see how much amortization can be achieved through more infrequent use of threading commands. 6.3 Final Words The Cell processor presents many unique challenges in the programming space, while at the same time providing great opportunities to see massive performance gains. We believe that our research shows these possibilities, but also provides a realistic characterization of the pitfalls that programmers will face as the Cell continues to mature as an architecture. Even with our ambivalent results, we feel that our insight has been to compare the Cell s performance against dissimilar architectures, and also to demonstrate the scalability and limitations of the architecture, where so many others have taken a blind affection toward this new architecture. In the end, the programming model is a difficult one, the Cell s performance benefits can be difficult to achieve, but given enough time and careful consideration invested, the merits of the Cell can shine through. 6.4 References [1] M. Sakamoto, H. Nishiyama, H. Satoh, S. Shimizu, T. Sanuki, K. Kamijoh, A. Watanabe, A. Asahara, An Implementation of the Feldkamp Algorithm for Medical Imaging On Cell. IBM White Paper. May, 25. [2] N. Costigan, M. Scott, Accelerating SSL using the Vector processors in IBM s Cell Broadband Engine for Sony s Playstation 3TM. Cryptology eprint Archive, Report 27/61. Febuary, 27. [3] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge, R. B. Brown, MiBench: A free, commercially representative embedded benchmark suite. IEEE 4th Annual Workshop on Workload Characterization. December, 21. [4] G. De Fabritiis, Performance of the Cell processor for biomolecular simulations. Comp. Phys. Commun. (27). March, 27.