Search this Blog

Monday, March 31, 2014

Top 5 Tech Support questions for Cisco System's products - Weekly Update Mar 24th

The most actively discussed Tech Support questions on the web for Cisco System's products (Week of March 24 /2014)
  1. What are the functions provided by Phone Security with CTL ?
  2.  How does Ethernet Virctual Circuit (EVC) works?
  3. How does a CUCM IP Phone keepalive works?
  4. How to configure paging and intercom features in Cisco CallManager?
  5. Will the Cisco 857 autoconnect after a dropped connection? 
Citation - This blog post does not reflect original content from the author. Rather it summarizes content that are relevant to the topic from different sources in the web. The sources might include any online discussion boards, forums, websites and others.

Friday, March 28, 2014

How to configure paging and intercom features in Cisco CallManager?

What are the steps involved in cofiguring paging and intercom features in Cisco CallManager? Also , what are the restrictions involved?

  1. Go to Call Routing > Intercom > Intercom Partition and create a new partition.
  2. Go to Call Routing > Intercom > Intercom CSS and create a new CSS that containst the partition created above.
  3. Go to Call Routing > Intercom >Intercom Directory Number and create 2 different directory numbers that we will use to communicate. Also associate the CSS created above, and choose a Default Device to which you will assign this particular number.
  4. Go to the first phone we will configure under Device > Phone and modify the phone button template as shown below:button-template-config.jpg It might be necessary to remove one of the options from the left square to make room for the new option; just select an unwanted option and clic on the arrow pointing down to the bottom square. Now, on the right square choose "Intercom [1] - Add a new Intercom", and click on the arrow pointing left to the left square. The option might appear on the left, at the bottom of the list.
  5. Save the changes everywhere and reset the phones or apply configuration (depends on the Callmanager version).
  6. Click on the Intercom option at the left side of the phone configuration page, this will take you to the Intercom Directory Number page.
  7. Look for the "Speed Dial" field, and input the target number. This is the Intercom DN that will go off-hook in speaker mode when you press the Intercom number we are currently configuring. ***NOTE*** If you do fill the Speed Dial field, when pushing your Intercom DN button will open the line and prompt you to enter any Intercom number. Only an Intercom number will be accepted, any regular DN won't work and most likely will ring busy if it doesn't perfectly match an Intercom number you don't know about.
  8. Do the same configuration to the other IP Phone assigning it the second Intercom DN we created.
  9. With the correct CSS and partitions, and after filling the Speed Dial field on each device, we should now be able to press the button and go off-hook with the other device. You can choose if you want your Intercom line to go off-hook in headset mode or in speaker mode from the line configuration page.
  • The following models support the feature in both SCCP and SIP mode:
The 7931 only supports it in SCCP.
  • One Intercom DN, say 200, can't be associated to multiple devices, or in other words, it doest allow sharing. You will get an error similar to "Update failed. Intercom line is not shareable"
  • Intercom calls do not follow a coverage path.
  • Hold - The system does not allow intercom calls to be placed on hold.
  • Call Forwarding - The system does not allow intercom calls cannot be forwarded.
  • Transfer - The system does not allow an intercom call to be transferred.
  • iDivert - The system does not allow an intercom call to be diverted.
  • Call Pickup/Directed Call Pickup - The call pickup groups do not include intercom calls.
  • DND - Intercom overrides Do Not Disturb (DND).
  • If sufficient bandwidth does not exist, the intercom call fails.
  • If two intercom calls are directed to a target, the first one goes through; the second fails with a busy tone.
  • Barge and cBarge - Intercom does not work with Barge and cBarge features.
  • Conferencing - The system does not allow intercom calls to be conferenced.
  • When an active call is being monitored or recorded, the user cannot receive nor place intercom calls.

Callmanager 4.x and below 

Cisco CallManager does not have a dedicated intercom feature. However, you can use the Auto Answer feature in Cisco CallManager. Activatving this option or button causes the speaker phone to go off hook automatically when an incoming call is received.
To configure Auto Answer, perform these steps:
  1. On the Cisco CallManager Admin page, go to  Device > Phones > Select the Extension. Select Enable the Auto Answer feature.
  2. Choose one of these options to activate the Auto Answer feature for the  directory number:       
    • Auto Answer Off
    • Auto Answer with headset
    • Auto Answer with speakerphone (intercom)
Many sites with an existing PBX also have a paging system, allowing users to call  an extension on the PBX that forwards the audio broadcast to overhead  loudspeakers. This concept is useful in workshops, parking lots, and open plan  areas where a called party is not near a telephone handset. PBX manufacturers  may provide dedicated line cards that interface with external paging amplifiers.
The Cisco CallManager needs the Cisco 2610 router to be configured as an H.323  gateway device. The extension number for the paging port is defined under the  Cisco CallManager Route Pattern configuration page, pointing to the Cisco 2610  H.323 gateway.
When the number for the paging system is dialed, a VoIP call is made between the  IP handset to the E&M port on the gateway router. The voice port goes off hook.  This is indicated by the E lead on pin 7 going from open circuit to closed  circuit (with respect to the ground on pin 8). This off hook condition activates  the pager system's control input, and the audio is sent on pins 4 and 5 of the  voice port

Citation - This blog post does not reflect original content from the author. Rather it summarizes content that are relevant to the topic from different sources in the web. The sources might include any online discussion boards, forums, websites and others.

How does a CUCM IP Phone keepalive works?

What is CUCUM IP Phone(SCCP) keeplive and why do we use them?

These keepalives are used by the phone for a couple of different reasons. First, the keepalives ensure that the TCP link to the CUCM node(s) is still viable. Second, the keepalive ensures that the Cisco Call Manager (CCM) Service is still functional, and able to process the phone's call control needs, and requests. While these may seem to be one in the same, they are actually slightly different in functionality, but both are obviously important with the SCCP connection to the CCM Service being reliant upon the TCP connection being connected for success. The implication of keepalive failure to either of these processes will be discussed in greater detail later on in this document.

By default, SCCP phones send a keepalive to their primary CUCM server every 30 seconds and to their failover node, which is the second node listed in the phone's Call Manager (CM) Group, every 60 seconds. The primary node will respond with a keepalive ACK confirming that both the TCP connection and the SCCP connection are both still valid. Alternatively, if the CCM Service on the primary node is down, the TCP connection may be ACK'd, but the SCCP aspect of the keepalive would not. This type of a response would signal to the phone that the TCP stack on the CUCM is still able to respond to inbound traffic, however the CUCM does not appear to be able to process calls at this time. Additionally, if the TCP connection fails to respond, then the phone quickly recognizes that the link is broken and the failover process begins.

Cisco IP phones also send a SCCP keepalive to their secondary node. This is done to maintain and monitor a TCP connection between the phone and the secondary CUCM in order to facilitate a prompt and reliable failover should the need arise. The secondary CUCM, however, does not have a SCCP connection (as the phone has not registered to the secondary node at this point) and will therefore only ACK the TCP connection in response to the SCCP keepalive sent by the phone.
In the packet capture (pcap) below, you can see an example of the keepalive transaction between a phone and it's CUCM nodes.
Frame 3232 - Phone ( is sending a SCCP KeepAliveMessage to it's primary CUCM server (
Frame 3233 - CUCM ( responds with a KeepAliveAck to the phone acknowledging that both the TCP and CCM connections are still valid.
Frame 3234 - Phone ( sends a TCP ACK to the CUCM ACK received in Frame 3233.
Frame 3237 - Phone ( sends a SCCP KeepAliveMessage to it's secondary CUCM server (
Frame 3238 - CUCM ( responds with a TCP ACK to the  phone, but notice that the SCCP aspect is missing from the TCP frame as the secondary CUCM does not ACK a SCCP connection, only the TCP connection to maintain a failover connection to this node if needed.

PCAP Keepalive1.png
From a CUCM perspective, when keepalive tracing is enabled, this is what would be seen in the CCM SDI traces as an example of inbound SCCP keepalive messages to CUCM:
08:48:56.386 |InboundStim - KeepAliveMessage - Send KeepAlive to Device Controller. DeviceName=SEP000000000000, TCPPid = [], IPAddr=, Port=17994, Device Controller=[1,50,1369]|1,100,49,1.7224140^^
08:48:56.389 |InboundStim - KeepAliveMessage - Send KeepAlive to Device Controller. DeviceName=SEP000000000001, TCPPid = [], IPAddr=, Port=6830, Device Controller=[1,50,1652]|1,100,49,1.7224141^^SEP000000000001
While the primary CUCM does log the inbound keepalive message from the phone, there is no logging of the response. From a troubleshooting perspective it is assumed that the CUCM responds to the keepalive when the above message is printed in the traces, but for solid evidence of a response we would need to get a packet capture from the CUCM interface or port.
Conversely, viewing the same keepalive tracing on a secondary node yields slightly different results. Notice below how the CUCM logs the keepalive event, but also states that it is dropping the KeepAliveAck rather than responding. Remember, this is only at the CCM process/SCCP level as the secondary does respond to at a TCP level to maintain that connection to the phone.
12:04:25.892 |KeepAliveMessage received on backup CM link. Dropping KeepAliveAck. DeviceName=, TCPPid = [], IPAddr=, Port=24527, Device Controller=[0,0,0]|2,100,49,1.5783617^^*
12:04:26.048 |KeepAliveMessage received on backup CM link. Dropping KeepAliveAck. DeviceName=, TCPPid = [], IPAddr=, Port=32877, Device Controller=[0,0,0]|2,100,49,1.5783618^^*

 When the keepalives don't get to where they need to go:

In an ideal situation, Cisco IP phones will pass keepalives to their primary and secondary CUCM servers and receive an ACK back for each one maintaining a TCP connection to each and registration to the primary. However, the main reasons for the keepalive system is not only to ensure current connectivity, but to also ensure that the backup link is still available to facilitate a prompt failover.
There are three basic reasons why a phone may failover from one server to another: unresponsive CCM Service on the CUCM node where the phone is currently registered, TCP socket break with the currently registered CUCM node, or availability of a higher priority CUCM server than the one to which the phone is currently registered.
The unavailability of the CCM service (sometimes due to CCM process crashing or high CPU utilization causing response delays) is often confused with the TCP socket break, but they are actually quite different. In a case where the phone is sending keepalives to its registering CUCM server and is receiving TCP ACKs, but is not receiving SCCP ACKs, the phone will wait for three failed SCCP keepalives in a row before considering that node unavailable. After the third failed SCCP keepalive, the phone will fail over to its secondary CUCM server.
Conversely, the second and likely the most common failover reason, is due to a TCP timeout. TCP timeout is almost exclusively indicative of a network problem between the phone and the CUCM server(s). While the CCM Service on the registered CUCM must fail to respond to three consecutive SCCP keepalives before the phone will fail over, TCP keepalive behavior is dependant upon the configuration settings of the phone and is generally much more sensitive to missed keepalives.
By default, CUCM uses Normal failover behavior as set by the "Detect Unified CM Connection Failure" parameter on the device configuration page. The Normal setting was introduced as an enhancement in phone load 7.2(1) and is also known as Geometric TCP Phone Failover, or adaptive failover. Geometric Failover was introduced as an improvement to the previous default configuration (now refererred to under the same parameter as "Delayed") as it allows for a much faster conversion to the secondary CUCM node when the TCP connection to the primary CUCM fails to respond to the keepalives sent from the phone.

"Delayed" failover for SCCP phones.

First, let's review the traditional functionality, or the "Normal" setting, for the "Detect Unified CM Connection Failure" parameter. The original behavior uses a TCP back-off timer that is used to detect TCP connection failure. If a phone sends a SCCP keepalive to CUCM and does not receive a response within 300ms, the phone retransmits the keepalive and again 300ms later if the first retransmit is also unacknowledged. After missing the three initial TCP keepalives, the phone then begins its back-off algorithm sending subsequent TCP retransmissions at approximate intervals of  400ms, 800ms, 1.5 seconds, 2.5 seconds, 5 seconds, 10 seconds and finally resetting the TCP connection after an additional 15 seconds if an ACK still has not been received from the CUCM. 
This, as you can calculate, can lead to a minimum time frame of over 35 seconds simply to declare the connection to be link-dead. Add to this 35 seconds the amount of time required to re-register to the secondary node and the worst case scenario of the link loss occurring immediately after the last successful keepalive ACK from CUCM (giving an additional 30 seconds until the next keepalive is sent from the phone) and you can understand why this is now referred to as "Delayed" behavior. See the image below for an example pcap for a Delayed failover scenario.
PCAP Lab Failover Delayed.png

"Normal" failover behavior (Geometric TCP Phone Failover).

Given the above example of the initial failover timing and behavior of Cisco SCCP IP phones, it's easy to see why there was a call for a faster failover mechanism to address scenarios where link-loss occurred. Any administrator of a large network will tell you that a five minute outage during which users are unable to use their phones while they are missing keepalives and then registering to their secondary nodes will tell you that it's too long. This was the catalyst for the introduction of Geometric TCP Phone Failover.
The Normal setting for "Detect Unified CM Connection Failure" implements an algorithm that monitors each keepalive response from the CUCM to the phone and measures the transit times to create a baseline, expected transit time to which it will compare all future keepalive attempts. A very basic example of this would be that if the phone has received 10,000 keepalive ACK messages and the expected transit time is X, the algorithm will then determine a retransmit time based upon X and then retransmit the keepalive messages accordingly. Generally speaking, the retransmit time under the Normal setting is much faster and at shorter intervals between retransmission than under the Delayed setting. Additionally, the Normal setting will only allow for six missed keepalive ACKs before the connection is reset. As is expected, this drastically reduces the failover time for phones from what could be as much as minutes to as little as seconds in the event of a network outage, between the phones and their primary CUCM server. See the image below for an example pcap for a Normal failover scenario.
PCAP Lab Failover Normal.png

Under most situations, the default setting of Normal should be used for phones in a CUCM environment. However, in the event that phones are traversing a less-than-reliable connections or in bandwidth-starved networks (specifically extreme examples where all available bandwidth is given to high-priority voice/RTP traffic and all other traffic is scavenging) to connect to their CUCM server, the Delayed setting may be useful. If the connection is sufficient to support voice in a reliable manner, but may miss keepalives periodically, the algorithm used by the Normal setting may be too sensitive and cause phones to unregister (and subsequently, re-register) from their primary CUCM node without just cause. Under these conditions, setting the value to Delayed may allow enough time for the phone to receive keepalives before failing over. The important thing to remember is that this is not "fixing" the problem, but merely increasing the failover tolerance and allowing the phones to exercise more flexibility within the keepalive process.
Here are two examples of ping tests from a CUCM server to a phone with two very different network responses. In Ping Test 1, the Normal setting would be the correct setting. In Ping Test 2, however, we may wish to set our "Detect Unified CM Connection Failure" parameter to Delayed in order to account for the massive discrepancies in transit time from packet to packet.
Ping Test 1
64 bytes from icmp_seq=0 ttl=63 time=0.407 ms
64 bytes from icmp_seq=1 ttl=63 time=0.396 ms
64 bytes from icmp_seq=2 ttl=63 time=0.391 ms
64 bytes from icmp_seq=3 ttl=63 time=0.469 ms
64 bytes from icmp_seq=4 ttl=63 time=0.419 ms
64 bytes from icmp_seq=5 ttl=63 time=0.392 ms
64 bytes from icmp_seq=6 ttl=63 time=0.383 ms
64 bytes from icmp_seq=7 ttl=63 time=0.399 ms
64 bytes from icmp_seq=8 ttl=63 time=0.427 ms
64 bytes from icmp_seq=9 ttl=63 time=0.400 ms
64 bytes from icmp_seq=10 ttl=63 time=0.389 ms
64 bytes from icmp_seq=11 ttl=63 time=0.388 ms
64 bytes from icmp_seq=12 ttl=63 time=0.391 ms
64 bytes from icmp_seq=13 ttl=63 time=0.395 ms
64 bytes from icmp_seq=14 ttl=63 time=0.469 ms
64 bytes from icmp_seq=15 ttl=63 time=0.422 ms
64 bytes from icmp_seq=16 ttl=63 time=0.392 ms
64 bytes from icmp_seq=17 ttl=63 time=0.394 ms
64 bytes from icmp_seq=18 ttl=63 time=0.405 ms
64 bytes from icmp_seq=19 ttl=63 time=0.399 ms
64 bytes from icmp_seq=20 ttl=63 time=0.392 ms
64 bytes from icmp_seq=21 ttl=63 time=0.397 ms
64 bytes from icmp_seq=22 ttl=63 time=0.416 ms
64 bytes from icmp_seq=23 ttl=63 time=0.432 ms
64 bytes from icmp_seq=24 ttl=63 time=0.387 ms
64 bytes from icmp_seq=25 ttl=63 time=0.397 ms
Ping Test 2
64 bytes from icmp_seq=1 ttl=63 time=28.1 ms
64 bytes from icmp_seq=2 ttl=63 time=156 ms
64 bytes from icmp_seq=3 ttl=63 time=0.410 ms
64 bytes from icmp_seq=4 ttl=63 time=0.423 ms
64 bytes from icmp_seq=5 ttl=63 time=0.409 ms
64 bytes from icmp_seq=6 ttl=63 time=272 ms
64 bytes from icmp_seq=7 ttl=63 time=224 ms
64 bytes from icmp_seq=8 ttl=63 time=118 ms
64 bytes from icmp_seq=9 ttl=63 time=0.410 ms
64 bytes from icmp_seq=10 ttl=63 time=148 ms
64 bytes from icmp_seq=11 ttl=63 time=82.9 ms
64 bytes from icmp_seq=12 ttl=63 time=412 ms
64 bytes from icmp_seq=13 ttl=63 time=0.413 ms
64 bytes from icmp_seq=14 ttl=63 time=213 ms
64 bytes from icmp_seq=15 ttl=63 time=142 ms
64 bytes from icmp_seq=16 ttl=63 time=226 ms
64 bytes from icmp_seq=17 ttl=63 time=0.400 ms
64 bytes from icmp_seq=18 ttl=63 time=0.404 ms
64 bytes from icmp_seq=19 ttl=63 time=83.1 ms
64 bytes from icmp_seq=20 ttl=63 time=2.61 ms
64 bytes from icmp_seq=21 ttl=63 time=417 ms
64 bytes from icmp_seq=22 ttl=63 time=206 ms
64 bytes from icmp_seq=23 ttl=63 time=284 ms
64 bytes from icmp_seq=25 ttl=63 time=172 ms
Related Defects
CSCed01179 - AVVID: Phone TCP backoff algorithm slow to return to initial timeout
CSCsm81227 - Allow Geometric TCP phone failover to be configurable
CSCtc43246 - Disable fast TCP connection loss detection for wireless networks
CSCtl03634 - Detect Connection Unified CM Failure DELAY does not work properly

Citation - This blog post does not reflect original content from the author. Rather it summarizes content that are relevant to the topic from different sources in the web. The sources might include any online discussion boards, forums, websites and others.

Thursday, March 27, 2014

What are the functions provided by Phone Security with CTL ?

What are the configurations steps of CTL . What are the function of phone secuirty with the CTL?

Phone Security and CTL Overview
Phone Security with CTL provides the following functions:
  1. Authentication of TFTP downloaded files (configuration, locale, ringlist, etc) using a signing key.
  2. Encryption of TFTP configuration files using a signing key.
  3. Encrypted call signaling for IP Phones.
  4. Encrypted call audio (media) for IP Phones.
Note that the first two functions can also be provided by Security By Default using ITL. The second functions of encrypted signaling and media can only be provided by using CTL files. Refer to the Security By Default document for more information on Authenticated and Encrypted configuration files.


1. Obtain USB eTokens

At least two USB eTokens are required for turning on Phone Security. These tokens are the key to signing the CTL file, and must not be lost. Multiple tokens can be used in a CTL file for redundancy since they are so important. They should be stored in secure, separate locations with their current passwords also stored safely.
In case a single token is lost or destroyed, the other tokens used at the initial signing of the CTL file can be used instead.
A token will self destruct after 15 failed password attempts, so remembering the token password and having backup tokens is extremely important.

2. Activate CTL Provider and CAPF Services

CTL Provider accepts connections from the CTL Client to generate the CTL File and collect certificates from all nodes. The CAPF (Certificate Authority Proxy Function) service is responsible for signing and storing LSCs (Locally Significant Certificates) from phones.

3. Download and Install the CTL Client

Starting in CUCM 8.6 Windows 7 is finally supported with the CTL Client. Make sure to download the correct CTL Client for the OS in use on the client PC.

4. Run the CTL Client using eTokens

The CTL Client will present the following options for a brand new install:
Set Cisco Unified CallManager Cluster to Mixed Mode:
This turns off auto registration and creates a CTL file.
Set Cisco Unified CallManager Cluster to Non-Secure Mode:
This allows auto registration to be enabled and leaves any existing CTL file in place. This is the default mode so cannot be selected unless the cluster is already in Mixed Mode.
Update CTL File:
This allows any new certificates or servers to be added to the CTL file.
Choosing any one of these options will require a USB eToken to be inserted in the client PC:
Once inserted, information about the token is displayed:
At this point the CTL Client performs connections to all CUCM servers in the cluster on TCP port 2444 to retrieve existing CallManager and CAPF certificates. This requires proper name resolution if using host names under "CCMAdministration > System > Server".
The list of all servers and certificates is displayed, along with all tokes in the existing CTL file.
If only one eToken is displayed, the "Add Tokens" option must be used to add another token before the cluster can be set to Mixed Mode.
Once Finish is selected, the CTL Client will ask for the private key password of the USB eToken. This allows the eToken to be used to sign the newly created CTL file, which contains all of the certs and tokens displayed above.
Note here that the password has been incorrectly entered once. The eToken software warns that only 14 more attempts are allowed before the token is permanently locked (destroyed). A successful password entry resets this counter back to 15.
If the correct password is entered, the CTL Client unlocks the private key from the eToken and uses it to sign the CTL File. This newly signed CTL File then gets written to every server on the cluster using another connection to the CTL Provider on TCP 2444. Again this requires network connectivity and name resolution from the CTL Client PC to each server in the cluster.

5. Restart Required Servers

The recommended procedure is to restart all TFTP servers, followed by all servers running the CallManager process. Restarting the TFTP servers allows the TFTP process to load in the newly generated CTLFile.tlv. Restarting the nodes running CallManager causes the phones to reset and download the new CTL file from the configured TFTP server.
admin:utils system restart
Do you really want to restart ?
Enter (yes/no)?

6. Install LSCs on Phones

After the CAPF service is activated and the phones obtain the CAPF certificate by downloading the CTL File, phones can connect to CAPF to obtain LSC files.
Set the phone CAPF Certificate Operation to Install Upgrade using Device > Phone, or Bulk Administration Tool > Phones > Update Phones > Query.
After setting the Certificate Operation, reset the phones.
If Null Sting or Existing Certificate have been chosen as the authentication mode no further action will be required.
If a string was chosen for the Authentication Mode then this will need to be entered manually into the phone console.
Settings > Security Configuration > **# > LSC > Update

7. Create and Apply Phone Security Profiles

Now that all of the underlying pieces are in place, phones can have security enabled via the Phone Security Profile. These profiles are specific to the model of phone being configured. A profile will need to be created for each model of phone in use.
CCMAdministration > System > Security > Phone Security Profile
Device Security Mode controls the primary phone security settings with the following options:
Non Secure - unencrypted signaling and unencrypted media (voice / RTP / Real Time Protocol)
Authenticated - encrypted signaling and unencrypted media
Encrypted - encrypted signaling and encrypted media
The separate checkbox for TFTP Encrypted Config controls whether or not the CUCM server sends an encrypted TFTP configuration file to the phone. The encryption of the TFTP file is independent of the Device Security Mode settings, but an encrypted config file is recommended on phones that support it.
The Security Profile needs to be applied at the Device level, so the Bulk Administration Tool is the most appropriate method to apply this profile to a larger number of phones.


Adding Phone Security to a CM cluster brings an additional layer that must be considered when planning and performing administrative tasks. Items such a certificates and certificate expiration dates should be taken into consideration. Certain administrative operations like changing host names may require regenerating certificates and CTL files.
The troubleshooting section here supplements the official Troubleshooting Guide and will provide steps to identify the current state of a cluster and recommend any corrective action necessary.

Verification and Repair Checklist

1. 1. Verify all certificates on all servers.

Collect serial numbers, Common Names, and expiration dates of current CAPF.pem and CallManager.pem certificates on all servers. The certificates loaded onto the CM servers are extremely important. Any mismatch in certificates on the servers could cause phone LSC download failures, configuration file authentication failures, or phone registration failures.
Here is the CAPF.pem certificate. Note the easily identifiable random string in the Common Name. This comes in handy as a quick verification tool. The CAPF.pem is used to sign LSCs (Locally Significant Certificates) and for the SSL handshake between the phone and the CAPF process.
This CAPF.pem expires in 2016, and was generated on April 5, 2011. These pieces of information tell us what dates to watch for in the future as well as what operations happened in the past.
This becomes more obvious when the CallManager.pem certificate is shown. Note that the CallManager.pem certificate also expires in 2016, but was generated on August 23rd, 2011. Some certificate regeneration operation must have been performed on the cluster on Aug 23rd. Remember that this certificate is used in the SSL handshake between phones and the CallManager, as well as by the TFTP process to sign files.

2. 2. Verify CTL contents match current certificates.

After checking the certificate contents, the next item to view is the CTL file on all TFTP servers. The OS Administration SSH CLI provides a simple command called "show ctl". The header of the CTL file contains the date the CTL file was last generated, the CN (Common Name) of the USB eToken used to sign the CTL, and the eToken serial number.
Note that the CTL was generated AFTER the CallManager.pem certificate generation date above. This is good because the CTL file should contain the latest version of the CallManager.pem. If the CTL file had a date that was BEFORE the CallManager.pem or CAPF.pem file generation dates, the CTL Client would need to be run again to get the latest certificates.
admin:show ctl
Length of CTL file: 4712
The CTL File was last modified on Wed Aug 31 13:28:03 EDT 2011
Parse CTL File
Version: 1.2
HeaderLength: 304 (BYTES)
------- --- ------ -----
3 SIGNERID 2 117
4 SIGNERNAME 56 cn="SAST-ADN4e31f914 ";ou=IPCBU;o="Cisco Systems
5 SERIALNUMBER 10 BD:A3:02:00:00:00:D8:88:64:1F
6 CANAME 42 cn=Cisco Manufacturing CA;o=Cisco Systems
The first entry inside the CTL is the full certificate of the eToken. This eToken with a serial number of "ADN4e31f914" was the eToken used to sign the CTL file. The serial number is printed on the token packaging and on the token itself, so the serial number in the Subject CN (Common Name) can be helpful to match the tokens used during signing.
CTL Record #:1
------- --- ------ -----
3 SUBJECTNAME 56 cn="SAST-ADN4e31f914 ";ou=IPCBU;o="Cisco Systems
4 FUNCTION 2 System Administrator Security Token
5 ISSUERNAME 42 cn=Cisco Manufacturing CA;o=Cisco Systems
6 SERIALNUMBER 10 BD:A3:02:00:00:00:D8:88:64:1F
9 CERTIFICATE 894 67 EB 23 F8 F5 16 55 A9 8E C8 CB 8A 4F 9E A2 0A AB 45 B6 E6 (SHA1 Hash HEX)
This etoken was used to sign the CTL file.
Multiple tokens can also be included inside the CTL file. At least 2 eTokens must be present in a CTL file. One token will be used for signing, and the other token is simply present as a backup trust point. The phone will trust any CTL file signed by other of these two tokens.
This output shows that the following eToken wasn't used to sign the CTL file, it's just the backup eToken. To update the CTL file at least one of the tokens inside the current CTL file must be found.
CTL Record #:2
------- --- ------ -----
3 SUBJECTNAME 56 cn="SAST-ADN5bbd7b14 ";ou=IPCBU;o="Cisco Systems
4 FUNCTION 2 System Administrator Security Token
5 ISSUERNAME 42 cn=Cisco Manufacturing CA;o=Cisco Systems
6 SERIALNUMBER 10 AA:C9:20:00:00:00:78:C4:2E:22
9 CERTIFICATE 895 A4 A3 8D 11 57 5A B8 E2 60 6E AF 4A 54 0A 20 B8 CA 0B D3 40 (SHA1 Hash HEX)
This etoken was not used to sign the CTL file.
The next record after the eTokens is the CallManager.pem certificate (denoted by function CCM+TFTP). This certificate is used by CM to sign configuration files and establish SSL connections between phones and the CM server if a Secure Profile is used on the phone.
Note that the serial number here matches the serial number in the CallManager.pem in the OS Admin page above. If this serial number differed between the two places, the CTL Client would need to be run to bring the CTL file in sync with what CM is actually using for a certificate.
CTL Record #:3
------- --- ------ -----
3 SUBJECTNAME 63 CN=CUCM8-Publisher.bbbburns.lab;OU=AS;O=Cisco;L=RTP;ST=NC;C=US
5 ISSUERNAME 63 CN=CUCM8-Publisher.bbbburns.lab;OU=AS;O=Cisco;L=RTP;ST=NC;C=US
9 CERTIFICATE 738 00 7A DE F4 25 26 7A FC 5E 02 B4 D2 BB A4 14 42 2B A5 A0 9C (SHA1 Hash HEX)
The final entry in the CTL file is the CAPF certificate. The serial number here also must match the OS Admin CAPF.pem, so phones are allowed to connect to the CAPF service. If there is a mismatch the same step of re-running the CTL Client must be performed.
CTL Record #:4
------- --- ------ -----
3 SUBJECTNAME 61 CN=CAPF-9c4cba7d;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
5 ISSUERNAME 61 CN=CAPF-9c4cba7d;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
6 SERIALNUMBER 8 0A:DC:6E:77:42:91:4A:53
9 CERTIFICATE 674 C7 3D EA 77 94 5E 06 14 D2 90 B1 A1 43 7B 69 84 1D 2D 85 2E (SHA1 Hash HEX)
The CTL file was verified successfully.
At the completion of this step, the CTL file will be in sync with the certificates loaded onto the CM servers.

3. 3. Verify CM serves TFTP CTL file

The next item to check in the troubleshooting process is whether or not the CM server is providing a CTL file via TFTP. A quick way is to take a packet capture at the IP Phone or the CM TFTP server. Here the phone requests the CTL file as the first file it downloads at boot.
The phone requested a CTL file, and if the filter on the previous capture is removed the transfer of that file can be viewed in detail.
Another method to verify the CTL file is downloaded is to look at the Phone Console logs under the web page of the phone. This requires the setting "Web Access" under "CCMAdmin > Device > Phone > Product Specific Configuration" to be "Enabled".
Here the console logs show the CTL file was downloaded:
837: NOT 09:13:17.561856 SECD: tlRequestFile: Request CTLSEP0011215A1AE3.tlv 846: NOT 09:13:17.670439 TFTP: [27]:Requesting CTLSEP0011215A1AE3.tlv from 847: NOT 09:13:17.685264 TFTP: [27]:Finished --> rcvd 4762 bytes
In addition to packet captures and phone console logs, the TFTP traces also show TFTP file transfers. Here is a shortcut to view the current TFTP trace in real time as a phone resets.
admin:file tail activelog cm/trace/ctftp/sdi recent
11:08:20.766 |TFTPEngine::isReadRequest[0x9950830~188~], [CTLSEP0011215A1AE3.tlv] opcode(1), Mode(octet), Serving Count(0)
11:08:20.779 |TID[a5a4fba0] TFTPEngine::processMessage[0x9950830~188~], Transferred[CTLFile.tlv] Socket[5]

4. 4. Verify phone properly validates and accepts the CTL file.

After verifying the CUCM server is presenting a valid CTL file, the next step is for the phone to validate that CTL file.
Phone console logs also show that the CTL file signature (the eToken signer) was trusted:
877: NOT 09:13:17.925249 SECD: validate_file_envelope: File sign verify SUCCESS; header length <296>

Status Messages displayed on the phone can also be helpful to verify a CTL file was downloaded successfully.
Either in the Status Messages web page of a phone, or under the phone itself "Settings > Status > Status Messages", the following line means a CTL file (or ITL file) has been successfully downloaded and verified:
16:01:16 Trust List Updated
If the phone could not validate the new CTL file the error message would be
Trust List Update Failed
Trust List Verification Failed
If the phone fails to validate the CTL file, that means the phone's existing Certificate Trust List does not have the same eTokens inside of it that the newly downloaded CTL was signed with, or that the newly downloaded CTL was corrupt.
A corrupt CTL can be checked with "show ctl", looking for the output "The CTL File was verified successfully", or the error condition "Verification of the CTL File Failed". Generally a corrupt CTL file can be repaired by running the CTL Client.
If the phone's old CTL file contains only eTokens that are no longer available, the CTL File will need to be deleted from the phone manually.
At this point the dilemma is "Did the phone download the latest CTL File?". The Status Messages and Phone Console logs can be used for verification, but other methods also exist.
The simplest method for verifying if a number of phones have the correct file is to compare the file sizes of the CTL file on the phone with the file size on the TFTP server.
First, create an SSH username and password for the IP Phone under CCMAdministration and enable SSH on the phone. Reset the phone.
Next SSH to the phone with the configured username and password. When prompted for the second login use "default / user". This example discusses 7961 and similar model phones. Use the following debug guide for 89XX and 99XX model phones.
From the phone
$ ls -l /tmp/*.tlv
-rw-r--r-- 1 default usr 4712 Oct 04 19:15 /tmp/CTLFile.tlv
-rw-r--r-- 1 default usr 3899 Oct 04 19:15 /tmp/ITLFile.tlv
From the TFTP Server
admin:file list tftp *.tlv detail
31 Aug,2011 13:28:03 4,712 CTLFile.tlv
16 Sep,2011 11:15:45 3,899 ITLFile.tlv
That method is a close approximation based on the size of the file. For an exact comparison of the contents, first look at the IP Phone to view the md5sum of the current CTL and ITL files
Settings > Security Settings > Trust List
In 8.6(2) and later versions of Communications Manager this hash should be visible on the server with "show ctl" and "show itl". Prior to 8.6(2), use TFTP and md5sum to verify the hash of this file as it exists on the server. This example checks the hash of the ITL file. Just replace ITL with CTL and the example will work for both files.
jasburns@jasburns-gentoo /data/trace/jasburns/certs/SBD $ tftp tftp> get ITLSEP0011215A1AE3.tlv 
Received 5438 bytes in 0.0 seconds 
tftp> quit 

jasburns@jasburns-gentoo /data/trace/jasburns/certs/SBD $ md5sum ITLSEP0011215A1AE3.tlv 
b61910bb01d8d3a1c1b36526cc9f2ddc ITLSEP0011215A1AE3.tlv

5. 5. Verify CTL contents on phone

A shortcut to verifying that the CTL file on the phone matches exactly byte for byte with the file on the server is just to quickly look at the phone's Trust List.
Settings > Security Settings > Trust List > CTL File
The name of the CM / TFTP server does match with the name of this server's CallManager.pem file.
The CAPF- also matches the CAPF.pem certificate that is currently in use.

6. 6. Verify SSL connection between phone and CAPF service if using LSCs

Now that the phone has a CAPF (Certificate Authority Proxy Function) certificate via the CTL, the phone can connect to CAPF to download a certificate.
A packet capture on the CM server can be used to verify the CAPF SSL handshake completed. Here the filter captures all traffic from the IP of the phone. Then the file is uploaded to another server using SFTP and the "file get" command.
admin::utils network capture host ip size ALL count 10000 file CAPF-Install
admin:file get activelog platform/cli/CAPF-Install.cap
The packet capture shows the SSL handshake that happens on the CAPF port, TCP 3804. To view this exchange, right click on any packet in the TCP port 3804 stream and go to "Decode As".
Here the certificate presented by the CAPF server matches the certificate in the CTL, and the certificate the phone displayed earlier. This SSL handshake succeeded because it started sending "Application Data", which would be the CAPF exchange.
Subsequently, the phone has an LSC installed:
If that process had failed despite the SSL handshake success, the next spot to examine would be the CAPF traces. If the SSL handshake failed, it would be time to check the CAPF certificate and update the CTL file again with the CTL Client.
The CAPF traces show that the phone connects, generates a key (which takes some time as seen by the gap in traces), then the CAPF server generates a certificate for the phone.
10:03:06.983 | debug 3:UNKNOWN:Got a new ph conn on 15
10:03:08.060 | debug TLS HS Done for ph_conn .
10:03:08.065 | debug MsgType : CAPF_MSG_AUTH_REQ
10:03:08.341 | debug MsgType : CAPF_MSG_REQ_IN_PROGRESS
10:03:08.341 | debug 3:SEP0011215A1AE3:CAPF CORE: Rcvd Event: CAPF_EV_REQUEST_IN_PROGRESS in State: CAPF_STATE_AWAIT_KEY_GEN_RES
10:03:21.148 | debug 3:SEP0011215A1AE3:Incoming Phone Msg:
10:03:21.158 | debug MsgType : CAPF_MSG_KEY_GEN_RES
10:03:21.162 | debug Generated the cert
10:03:21.724 | debug 3:SEP0011215A1AE3:Certificate upgrade successful

7. 7. Verify SSL connection between phone and CM server for registration

Now that the phone has a CTL and LSC, the next step is secure phone registration. For SCCP phones this happens on TCP Port 2443.
Use the same steps as before to capture all packets from this specific phone.
The first thing that's different is the phone downloading SEP.cnf.xml.enc.sgn. This signifies and encrypted TFTP configuration file as set under the Device Security Profile.
Here the phone connects on TCP port 2443, so this port must be Decoded As SSL in Wireshark. The CUCM presents the CallManager.pem certificate (verifiable by serial number and common name) and then asks for the certificate of the phone. As before the SSL handshake completed successfully since the Application Data phase is reached.
In addition to the packet capture, phone registration via Secure SCCP is also visible in the Cisco CallManager SDI traces:
10:38:26.621 |SdlSSLTCPListener::verify_cb pre-verified=1,cert verification errno=0,depth=0
10:38:26.626 |New connection accepted. DeviceName=, TCPPid = [], IPAddr=, Port=51948,
10:38:29.051 |StationD: (0000048) ClusterSecurityMode = (1) DeviceSecurityMode = (3)
10:38:29.051 |StationD: (0000048) TLS Connection Cipher - INFO:deviceName=SEP0011215A1AE3, Cipher=AES128-SHA, Security Mode=3

8. 8. Verify Encrypted Calls

Calls that have both encrypted signaling and encrypted media will show the lock icon in the lower right corner of the call window. This corresponds to Device Security Mode: Encrypted.
Calls that use encrypted signaling between both ends, but that do not use encrypted media will show the shield icon. This corresponds to Device Security Mode: Authenticated.
The security status of the least secure party is used to determine which icon is displayed and what call security is used. Take a look at the following table for examples:
Phone APhone BIcon Displayed
EncryptedEncryptedLock - Encrypted Audio
EncryptedAuthenticatedShield - Unencrypted Audio
EncryptedNoneNone - Unencrypted Audio

Citation - This blog post does not reflect original content from the author. Rather it summarizes content that are relevant to the topic from different sources in the web. The sources might include any online discussion boards, forums, websites and others.
/* Google Analytics begin ----------------------------------------------- */ /* Google Analytics end ----------------------------------------------- */