Benchmarking Telecommunication Systems

The world is full of benchmarks. In January 2006 Google provided 2.9 million hits on phrase "computer performance benchmark". IEEE Xplore found 426 articles published since 1.1.2000 in that category. ACM Digital Library had 11,519 entries in that category. In this paper we review some existing benchmarks that are useful in analyzing performance of telecommunication sytems. The focus is on networking and database benchmarks, but also other benchmarks are briefly summarized.

Telecommunication systems of today are quite complicated networked and distributed computing and communication environments. A telecommunication system involves networking devices and protocols, databases, network management and control software, operations support systems, accounting and charging facilities, billing systems and various kinds on servers. It is evident that a single benchmark cannot give enough insight to understand performance of a telecommunication system.

In telecommunications we divide the functionality into user (or data), control, and management planes. The user plane concerns transmitting data between the end-points. The control plane is involved in establishing, maintaining, and tearing down connectivity between the end-points. The management plane covers management operations of the systems. Efficiency of packet handling is the primary factor in performance of user plane functionality. On the control-plane we have timeliness requirements of setting-up a service session, preparing accounting and charging, executing handoff, reserving and releasing resources, etc. The performance of control-plane operations is affected by databases, authentication, communication between network elemnts, among others. Although processing efficiency is important, performance is not so critical on the management plane. The main concerns include correctness, reliability, security, and availability.